System and method for synthesizing pre-acquired image data into an occlusion region of a real-time image stream

Abstract

To provide systems and methods for composing pre-acquired image data into occluded areas of a real-time image stream.SOLUTION: Various aspects of methods, systems, and use cases may be used to generate and display a real-time composite image, e.g., in a manner of extracting a non-occluded portion of a first image corresponding to an occluded portion of a second image and replacing the occluded portion with the non-occluded portion. In some examples, a graphical representation of an instrument may be generated on the composite image.SELECTED DRAWING: Figure 1A

Description

Background Art

[0001] Imaging can be used to observe a part of a patient's body before or during a medical treatment. For example, imaging can be used to observe a patient's internal organs, tissues, and other anatomical structures. Imaging techniques and devices can include, for example, ultrasonic imaging or computed tomography (CT) imaging. Ultrasonic imaging uses high-frequency sound waves to generate an image of the inside of a patient's body. In CT imaging, a series of X-ray images taken from multiple angles around the body are combined to generate a cross-sectional image of the inside of the patient's body.

[0002] Such imaging techniques can be used for patients who may have a solitary pulmonary nodule (SPN) that can be present just outside the airway wall and identified on CT images. Often, when an SPN is identified in a CT image set, a pulmonologist performs a biopsy to obtain a tissue sample for pathology. This is because most SPNs are benign, but some SPNs correspond to early-stage lung cancer and can be fatal if left untreated.

[0003] Patients with an SPN may undergo an endobronchial ultrasound (EBUS) procedure to obtain a biopsy sample. The EBUS procedure can be used to diagnose various lung problems such as lung inflections, diseases, and cancers. During the EBUS procedure, a needle can be used to collect a tissue or body fluid sample while the clinician views real-time ultrasound images to confirm that the needle is entering and sampling the SPN. Due to the dynamic nature of the lungs throughout the respiratory cycle and the small size of the SPN, real-time imaging via an EBUS device is often required. For example, the SPN may be less than 2 cm in diameter or may move more than 2 cm throughout the respiratory cycle. Thus, a preoperative CT scan may reveal the presence of the SPN, but such a static scan does not provide confirmation of the SPN's current real-time location.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] U.S. Patent No. 10,617,324 [Summary of the Invention] [Means for Solving the Problems]

[0005] The present invention discloses a system and method for synthesizing pre-acquired image data into occluded regions of a real-time image stream. In one aspect, the system identifies occluded portions in real-time ultrasound data and fills or replaces the occluded portions using image data from sources other than the real-time ultrasound data.

[0006] For example, previously acquired ultrasound images captured immediately prior to an occlusion event may be used to enhance the real-time stream. The system can analyze one or more ultrasound images captured several seconds before the current occluded portion is not obscured. Subsequently, the matching non-occluded ultrasound data is synthesized into the occluded region. This enables the clinician to observe a complete display of the tissue as if no occlusion had occurred.

[0007] Occlusions may be caused by instruments such as biopsy needles or ablation devices entering the field of view. They may also be caused by bubbles that impede the transmission of ultrasound. The system detects the occluded portion by analyzing the pixel data and identifying regions lacking tissue characteristic information. Known instrument characteristics or bubble characteristics can also be used to identify the occluded region.

[0008] In another embodiment, the system synthesizes pre-acquired CT or other modality data into the occluded region after processing to conform it to the real-time ultrasound data. For example, CT data representing the occluded region can be converted to look like ultrasound data before being synthesized into the live stream.

[0009] The system can further track the position and orientation of the image using techniques such as electromagnetic tracking or respiratory gating. This navigation data helps to align the unobstructed portions of past images with the live occluded regions. Synthetic data can also be warped and deformed to fit the real-time anatomical structure.

[0010] With the disclosed systems and methods, a clinician can observe a complete and rich image containing valuable tissue information during a procedure, despite the real-time occlusion that normally obscures part of the image.

[0011] In the drawings, which are not necessarily drawn to scale, like numerals may in different views represent like components. Like numerals with different suffix letters may represent different instances of like components. The drawings generally show, by way of example and not limitation, the various embodiments discussed in this specification.

Brief Description of the Drawings

[0012]

Figure 1A

Figure 1B

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 6

[0013] Ultrasound (US) imaging can display both tissue and instruments (e.g., biopsy needles or sheaths) in real time, so ultrasound imaging is used in the respiratory region to reliably identify the target area from which a biopsy sample is to be taken. However, the instrument often blocks the transmission of ultrasound energy beyond the instrument in relation to the ultrasound energy source (e.g., a US transducer). Thus, when the instrument is within the field of view of the ultrasound image, there may be a lack of information indicating tissue characteristics in the occluded region beyond the instrument in relation to the transducer while the physician is directing the instrument towards the target area for biopsy. Accordingly, the inventors have recognized that an imaging system that can continue to display a rich image of the tissue within such an occluded region is desirable.

[0014] In addition to the instrument shielding ultrasonic transmission and affecting image quality, the inventors recognized that certain disease states can block ultrasonic transmission and potentially affect the imaging during treatment. For example, in COPD patients, the flow of air from the lungs is blocked. When the lungs are blocked, air bubbles may be trapped within the tissue. When imaging the lungs of COPD patients, the air bubbles degrade the quality of the image. When imaging using an ultrasonic device, the sound waves hit the air bubbles, preventing imaging of the tissue beyond the air bubbles.

[0015] To mitigate these drawbacks, the inventors developed a system that can identify occluded portions within real-time ultrasonic data and fill or replace those occluded portions using image data from some source other than the real-time ultrasonic data. For example, pre-acquired image data can be utilized. Previously acquired image data can be used to highlight portions of an image that are being acquired in real-time, are occluded, or contain occluded portions. Such occlusions can be caused by air bubbles, instruments (such as biopsy needles or ablation devices) that enter the field of view of the imaging device (such as an ultrasonic transducer), or combinations thereof.

[0016] More specifically, the concepts described herein are designed to utilize previously acquired image data to emphasize portions of an image that are occluded (e.g., by an instrument that has entered the field of view of an imaging device such as an ultrasound transducer) and are acquired in real time. An exemplary scenario in which the proposed concepts are beneficial is in ultrasound-guided biopsy. During such a procedure, a physician may navigate an endobronchial ultrasound (EBUS) tissue sampling device towards a target nodule (e.g., tissue previously identified from a CT scan where a biopsy sample is needed). Once the target nodule is reliably identified within the ultrasound image, the physician can advance a sheath that encloses a sampling needle from the working channel of the EBUS tissue sampling device. The sheath can be a flexible plastic sleeve that prevents the sampling needle from damaging the EBUS tissue sampling device at the lateral exit where the sampling needle has a ramp (e.g., deflecting the sampling needle away from the longitudinal axis of the sampling device towards the target nodule beyond the airway wall).

[0017] It will be appreciated that when an instrument (e.g., a sampling needle and / or sheath) extends from the lateral exit, the instrument enters the field of view of the ultrasound transducer. This is a design to enable the physician to visually confirm the instrument in real time within the generated ultrasound image, which is desirable, but one drawback is that the instrument may occlude an area of the image that is behind the instrument with respect to the transducer. The proposed concepts are aimed at reducing this drawback by identifying this occluded portion and filling it using image data from some source other than the real-time ultrasound data.

[0018] In some embodiments, the system may identify an occlusion and then utilize previously acquired ultrasonic images that were acquired immediately before the instrument caused the occlusion. For example, the system may analyze one or more ultrasonic images captured several seconds before the occlusion was yet to be occluded, and may synthesize a portion of these ultrasonic images that matches the occlusion onto the occlusion substantially in real-time. In this way, even though there is an occlusion region in the real-time ultrasonic image, a graphical representation of the tissue within the entire ultrasonic field of view is still displayed to the physician as if there was no effect of the instrument occluding the image.

[0019] In some embodiments, the system may synthesize image information from another imaging modality onto the occlusion. For example, previously acquired CT data representing the occlusion can be processed to look like ultrasonic data and then synthesized onto the occlusion of the real-time ultrasonic image.

[0020] The above description is intended to provide an overview of the subject matter of this patent application. It is not intended to provide an exclusive or exhaustive description of the invention. The following description is included to provide further information regarding this patent application.

[0021] FIG. 1A shows a schematic diagram 100a of an exemplary internal medical imaging system 104 having an image sensor 106. In one example, the internal medical imaging system 104 is inserted into a target region 102 of a patient.

[0022] In one example, the internal medical imaging system 104 is an endobronchial ultrasound (EBUS) tissue sampling device. In one example, the internal medical imaging system 104 is used to navigate towards a target nodule and image it prior to tissue sampling. In one example, the target nodule is a pre-identified tissue for which a biopsy sample is desired. In one example, ultrasound (US) imaging is used in the respiratory region to reliably identify the target area for taking a biopsy sample, as US imaging displays both the tissue and the instrument (e.g., a needle or a sheath) in real time.

[0023] In one example, the image sensor 106 is disposed at the distal end of the internal medical imaging system 104. In one example, the image sensor 106 extends along a portion of the distal end. In one example, the image sensor 106 is substantially flat. In one example, the image sensor 106 is an ultrasound transducer. In one example, the image sensor 106 provides a field of view 108 to an imaging display.

[0024] During imaging, occlusion 110 within the field of view 108 causes an occluded portion 112 to occur within the field of view 108. The field of view 108 extends from a proximal boundary 108a to a distal boundary 108b. In one example, the occlusion 110 is a bubble that causes the occluded portion 112. As shown in FIG. 1A (and even more so in FIG. 1B), the occluded portion extends from a proximal boundary 112a to a distal boundary 112b. The occluded portion 112 is caused by the interruption of the transmission of ultrasonic energy to tissue beyond the occlusion 110 in relation to the image sensor 106 (e.g., a US transducer). For example, ultrasonic waves transmitted from the image sensor 106 (e.g., an ultrasonic transducer) can propagate through the tissue until they reach the boundary between the tissue and the bubble (which is the occlusion 110 in this example), where a significant amount of the ultrasonic energy is reflected back towards the image sensor 106, and a small amount continues to propagate away from the image sensor 106 and into the bubble. The occluded portion 112 may lack information indicating the characteristics of the tissue. Such a lack of information can potentially affect the ability of a physician to direct the internal medical imaging system 104 towards the target region of a patient. Thus, as will be described in more detail below, a display device or its computing device connected to the internal medical imaging system 104 uses previously acquired image data from the field of view to create a composite image that includes both previously acquired images and images collected in real time. The computing device uses a portion of a previously acquired image that is aligned with the occluded portion 112 to replace the occluded portion, thereby providing the clinician with a complete image of the field of view even if a portion is occluded.

[0025] FIG. 1B shows a schematic diagram 100b of an exemplary internal medical imaging system 104 having an instrument 120 extending from the internal medical imaging system 104 within the target region 102 of a patient.

[0026] In one example, the instrument 120 advances through the internal medical imaging system 104 via the working channel. In one example, the instrument 120 extends from the outlet port 114 of the internal medical imaging system 104. In one example, the outlet port 114 includes an inclined path such that the instrument 120 extends at an angle from the internal medical imaging system 104. In one example, the instrument 120 is a sheath. In one example, the sheath is a flexible plastic sleeve. In another example, the instrument 120 is a needle. In one example, the needle is a sampling needle for capturing a biopsy of a target tissue (which can be pre-identified via a pre-operative image source such as a CT scan).

[0027] As another example, the instrument 120 includes both a sheath and a needle. As one example, the sheath covers the needle. As another example, the sheath extends partially outside the internal medical imaging system 104 before the needle extends from the outlet port such that the sheath protects the outlet port 114.

[0028] During imaging, when the instrument 120 extends from the internal medical imaging system 104, an occlusion portion 122 occurs within the field of view 108. The occlusion portion 122 is caused by blocking the transmission of ultrasonic energy to the tissue beyond the instrument 120 in relation to the image sensor 106 (e.g., a US transducer). While it may be advantageous for the user to see the instrument 120 within the field of view 108, the occlusion portion 122 caused by the instrument 120 may lack information indicating the characteristics of the tissue beyond the instrument 120. In this example, the field of view 108 and the occlusion portion 122 share a common proximal boundary (proximal boundary 108a and proximal boundary 122a), while the distal boundary 122b of the occlusion portion 122 will advance distally as the instrument 120 extends into the field of view 108. Such a lack of information can potentially affect the physician's ability to direct the internal medical imaging system 104 towards the patient's target area and / or the physician's ability to direct the instrument 120 towards a target object within the target area.

[0029] In one example, the internal medical imaging system 104 of FIGS. 1A and 1B includes or is connected to a display device for displaying an image of the field of view 108 obtained by the image sensor 106. In one example, the internal medical imaging system 104 is connected to a computing device. The computing device is capable of image processing (e.g., the machine of FIG. 6).

[0030] FIG. 2A is a diagram showing an example of a method 200a for generating a composite image. FIG. 2B shows an exemplary set of images 200b that are processed into a composite image using the method of FIG. 2A. In one example, the method 200a can be executed by a computer device (e.g., the machine of FIG. 6) communicatively connected to an EBUS. The computer device includes a processing circuit for performing various image processing tasks described in connection with the method 200a. Further, the computer device can include, among other things, an output display such as a monitor for displaying ultrasonic images and various additional user interface elements.

[0031] At the start of the method 200a, such as at the start of a procedure, the imaging device (e.g., the internal medical imaging system 104) is positioned in the desired field of view. In one example, when the imaging device is positioned at a predetermined location, the imaging device is rotated to collect a plurality of images at a plurality of tracking positions. In some embodiments, the imaging device 104 is an EBUS device with navigation capabilities, such as an embedded sensor coil that provides signals to a navigation system that enables accurate tracking of the position and orientation within the patient when placed within an electromagnetic field generated, for example, during a procedure. The navigation system of the imaging device can utilize electromagnetic (EM) tracking techniques that enable tracking of the imaging device within six degrees of freedom. In one example, the device rotates between 15 degrees and 360 degrees. In one example, interpolation is used to create images between images collected at different degrees. In one example, the tracking positions include the position of the imaging device and the orientation of the imaging device.

[0032] In one example, an imaging system, or another connected system, monitors and collects respiratory information associated with an image. For example, a portion of the respiratory cycle is captured. In one example, the respiratory information is received via a respiratory gating tracking device. Since the respiratory gating tracking device can indicate where a patient is in the respiratory cycle, the system can adjust the shape and location of a 3D model of the patient's lungs taking respiration into account. An example of a respiratory gating tracking device is discussed in U.S. Patent No. 10,617,324, titled "Apparatuses and methods for endobronchial navigation to and confirmation of the location of a target tissue and percutaneous interception of the target tissue", which is hereby incorporated by reference in its entirety. The respiratory gating tracking device can include a plurality of markers that can move, change direction and shape during patient movement caused by different stages of the respiratory cycle. In one example, a computing device can use the distance between a set of multiple markers at a given time to determine the stage of the respiratory cycle. In one example, the markers are displayed within an image. For example, the computing device uses an image of the markers to determine the distance between the markers and determine the current stage of the respiratory cycle. Thus, in one example, the respiratory cycle is incorporated into the navigation data to further enhance the direction and position information used for imaging within a particular portion of an anatomical structure.

[0033] In 202a, the computing device receives a first image 202b of the field of view. The computing device receives the first image 202b captured from the image sensor 106 of the internal medical imaging system 104. In one example, the first image 202b is of a target region of a patient. For example, the first image 202b may include in the field of view a target tissue, such as a solitary pulmonary nodule (SPN) located adjacent to and just outside the airway, for which a biopsy sample and / or ablation treatment is desired. The first image 202b may not have an occlusion. In one example, the first image 202b includes a tracking position. In one example, a plurality of first images 202b are received. The EBUS device is a real-time imaging device that provides a stream of images. Thus, in these examples, the description of the "first image" is generally used to refer to a first imaging state (e.g., a first imaging state without an occlusion). Thus, when method 200a is discussed as "receiving a first image", this can be interpreted as an abbreviation for a stream of images in a first imaging state. In a particular scenario, the computing device may operate on a single image or a stream of images while updating a display device in real-time or near real-time. For convenience, most of the method is described with respect to individual images, but the operations are performed in real-time on a stream of images.

[0034] In 204a, the computing device receives a second image 204b of the field of view. The second image 204b is received after the first image 202b. The second image 204b includes an occlusion region 206b. In one example, the second image 204b is received in real time. In one example, the computing device receives a plurality of second images 204b. In one example, the second image 204b is received immediately after the first image 202b. In one example, the second image 204b includes a tracking position. The second image 204b that includes the occlusion region 206b may be a consecutive image from the same image stream from which the first image 202b was received. For example, each of the first image 202b and the second image 204b may be provided from an EBUS sampling device to an ultrasonic image processor within the same image stream during a single procedure.

[0035] In 206a, the computing device identifies one or more occluded portions 206b of the second image 204b. In one example, the computing device identifies one or more occluded portions 206b by evaluating the pixels of the second image 204b. In one example, the computing device determines that all the pixels within the occluded portion 206b are black or contain data that is not clearly from the imaged tissue. In another example, the computing device compares a set of pixel data from the second image 204b to a baseline value of the pixel data. The baseline value of the pixel can be set to indicate that the pixel is occluded. In one example, the computing device determines that a portion of the second image 204b is occluded if a subset of the set of pixel data is below the baseline value. In one example, instead of evaluating individual pixels, the computing device compares the pixel data within a group of pixels. The computing device can determine that there is an occluded portion 206b when it exceeds a specific region. For example, the computing device determines that no additional information is being collected when it exceeds a specific row of data. In one example, the computing device detects an immediate change in the post-occlusion quality that causes the occluded portion 206b (for example, the collected ultrasonic data can indicate that there is little ultrasonic data reflected back to the transducer beyond a specific depth where large ultrasonic reflections occur). The occluded image portion can also be identified by identifying the instruments within the image. For example, since a metal needle efficiently reflects ultrasonic waves, a biopsy needle typically generates a line of bright white pixels in an ultrasonic image. In one example, the computing device examines the second image 204b within the segment based on the imaging ray and identifies an occluded portion 206b if data is expected in the second image 204b but not collected (for example, if the second image 204b generates zeros beyond a specific point). In one example, the computing device detects an occluded portion 206b by detecting air bubbles within the field of view of the second image 204b.The computing device detects the bubbles using the techniques described above. In one example, the computing device detects the bubbles by applying an image processing algorithm trained to detect imaging characteristics of the bubbles. The image processing algorithm can be utilized to identify these known artifacts and then interpolate the occluded regions. In a particular example, the image processing algorithm can look for changes in contrast. For example, regions with little or no change in contrast are more likely to be occluded regions of the image.

[0036] In 208a, the computing device extracts the non-occluded portion of the field of view from the first image 202b. The non-occluded portion from the first image 202b substantially corresponds to the occluded portion 206b of the second image 204b. In some examples, each of the first image 202b and the second image 204b includes position and orientation data (e.g., tracking position) associated with the image. In one example, the computing device can utilize information regarding the field of view and the tracking position to understand the relationship between the first image 202b and the second image 204b. As another example, the computing device can utilize a plurality of first images to determine the non-occluded portion of the first view extracted from the first image 202b having a tracking position substantially similar to that of the second image 204b. In a particular example, the non-occluded portion can be identified within the first image 202b that was not taken at exactly the same tracking position as the second image 204b. In such an example, the position and orientation data associated with the second image 204b can be used to accurately identify where in 3D space the patient's tissue is occluded, and the position and orientation data associated with the first image 202b can be used to identify the non-occluded image data corresponding to this 3D space for use in the composition to the second image 204b.

[0037] In 210a, the computing device generates a composite image 210b. In one example, the computing device generates a composite image by replacing the occluded portion 206b of the second image 204b with the non-occluded portion of the first image 202b. For example, the occluded portion 206b can be extracted from the second image 204b and replaced with the non-occluded portion of the first image 202b. In another example, the computing device generates a composite image by overlaying the non-occluded portion of the first image 202b on the occluded portion 206b of the second image 204b. The composite image can be generated substantially in real time. The system determines a scale and pose adjustment factor between a real-time ultrasound stream (e.g., the second image 204b) and a previously acquired ultrasound image (e.g., the first image 202b), and based on the scale and pose adjustment, can transform a portion of the previously acquired ultrasound image (e.g., the first image 202b). The non-occluded portion of the first image 202b can be deformed to align with the occluded portion 206b of the second image 204b. In a particular example, the non-occluded portion can be identified within the first image 202b that was not taken at exactly the same navigation position and orientation as the second image 204b. In these examples, the computing device can transpose the non-occluded portion of the first image based on a comparison of the navigation data between the first image 202b and the second image 204b, or interpolate between the non-occluded portion of the first image 202b and the occluded portion 206b of the second image 204b based on the navigation data of each image. When transposing the data, the non-occluded portion is from the first image 202b where the position and orientation are close enough, and it may only be necessary to shift the non-occluded portion to the position of the occluded portion 206b of the second image 204b. When the position and orientation between the first image 202b and the second image 204b are more important (but within the range where representative adjustment of the non-occluded portion is possible), interpolation may be necessary.

[0038] In 212a, the computing device displays a synthetic image 210b. Even though the real-time second image has an occlusion portion 206b, the user is presented with an image representation of the tissue within the entire field of view as if the occlusion portion 206b has no effect of occluding the second image 204b. If the computing device does not receive an image with an occlusion portion 206b, the first image 202b is displayed without being changed by the processes described herein. In one example, the computing device enables the user to switch between multiple image display modes. For example, the computing device may include a first mode that generates a synthetic image when an occlusion portion is detected, and a second mode that displays the unmodified real-time image. In some embodiments, the system may be configured to display a graphical user interface (GUI) element that indicates the contour between a portion of the image stream corresponding to the unmodified real-time image data and a portion corresponding to the synthetic image data that complements the occlusion portion. Examples of such a GUI include, but are not limited to, a colored boundary line that outlines the synthetic image portion, or displaying the synthetic image portion in a different color or shade compared to the unmodified image portion, or both.

[0039] Method 200a operates on a computing device in real-time and continuously throughout the procedure. For example, while the instrument is extending within the field of view, the second image 204b continues to be collected along with the occlusion portion 206b, and method 200a operates in real-time to generate and display a composite image 210b using a portion of the first image 202b that was extracted to replace the occlusion portion 206b of the second image 204b. As another example, when the imaging device is moving through a passage and there are air bubbles within the passage, the second image 204b continues to be collected along with the occlusion portion 206b, and method 200a operates in real-time to generate a display composite image 210b using a portion of the first image 202b that was extracted to replace the occlusion portion 206b of the second image 204b. As another example, the first image 202b continues to be collected until a second image 204b having an occlusion portion 206b is identified. The first image 202b and the second image 204b can be continuously received, and new composite images can be displayed throughout the procedure. When method 200a operates, the first image 202b can be selected to be the image received immediately prior to the second image 204b that includes the occlusion portion 206b. Among other advantages, the techniques described herein enable a user to view the entire boundary of a target tissue (e.g., SPN) even while a sampling needle or ablation device is partially inserted into the target tissue to obtain a biopsy or perform an ablation procedure. In this way, valuable information that is lacking in the capabilities provided by conventional medical imaging systems (e.g., EBUS systems) is provided to the user within the displayed image.

[0040] Figure 3A shows an example of a method 300a for generating a synthetic image using a plurality of image modalities. Figure 3B shows an exemplary set of images 300b that are processed into a synthetic image using the method of Figure 3A. In one example, method 300a can be executed by a computer device (e.g., the machine of FIG. 6) communicatively coupled to an EBUS. The computer device includes processing circuitry for performing the various image processing tasks described in connection with method 300a. Further, the computer device can include, among other things, an output display such as a monitor for displaying an ultrasound image and various additional user interface elements.

[0041] When collecting, capturing, or receiving an image or image data, the image may be registered. For example, the image may be registered from an image space in relation to a patient space. Registration can include the orientation, position, location, and / or rotation of the image with respect to the patient. For example, various points within the image or the patient may be identified to register the image. In one example, the points are easily identifiable anatomical landmarks. The points can be used to establish point-based registration between images such that the image data includes information regarding the orientation, position, location, and / or rotation of the image with respect to the patient or other images. When the image sensor is moved, the movement of the points can be used to determine updated information regarding the orientation, position, location, and / or rotation of the image with respect to the patient or other images. In one example, when collecting, capturing, or receiving an image or image data, the image can be registered using 4-dimensional (4D) data. During 4D registration, image data is collected to render a three-dimensional (3D) image (e.g., a volume image) of a target region. The 3D image further includes movement resulting from movement of the patient's anatomical structures such as caused by the patient's respiratory cycle and / or the patient's heartbeat, thus generating 4D data and a 4D image.

[0042] At 302a, the computing device receives a first image 302b of the field of view from a first imaging modality. In one example, the computing device receives the first image 302b from a computed tomography (CT) device. In one example, the first image 302b is of a target region of a patient. For example, the first image 302b is of a target region of a patient. For example, the first image 202b may include in the field of view a target tissue, such as a solitary pulmonary nodule (SPN) located adjacent to just outside the airway, for which a biopsy sample and / or ablation treatment is desired. The first image 302b may have no occlusion. In one example, multiple first images 302b may be acquired at multiple tracking positions. For example, the first image 302b is a CT scan collected at 1 mm slices. In one example, the computing device interpolates between slices to create a new first image 302b. In one example, the first image 302b includes anatomical landmarks and / or indicates the current anatomical state. For example, the anatomical state includes the respiratory state. The first image 302b can be collected during total lung capacity and / or tidal volume. In one example, the computing device receives multiple first images 302b.

[0043] Before receiving the second image 304b, such as at the start of a procedure, the imaging device (e.g., the internal medical imaging system 104) is positioned in the desired field of view. In one example, once the imaging device is positioned at a given location, it can be rotated to collect multiple images at multiple tracking positions. In one example, the imaging device 104, such as when placed within an electromagnetic field generated during a procedure, is an EBUS device with navigation capabilities, such as an embedded sensor coil that provides signals to a navigation system that enables accurate tracking of the position and orientation within the patient. The navigation system of the imaging device can utilize electromagnetic (EM) tracking techniques that enable tracking of the imaging device within six degrees of freedom. In one example, the imaging device is rotated about the longitudinal axis of the device. The imaging device can be rotated between 15 degrees and 360 degrees. Rotating the imaging device longitudinally can generate a fan-shaped series of images. In one example, the computing device interpolates between images collected at different degrees to create additional images. In one example, the tracking positions include the position of the imaging device and the orientation of the imaging device.

[0044] In one example, the imaging system, or another connected system, monitors and collects respiratory information associated with the images. For example, a portion of the respiratory cycle is captured. In one example, the respiratory information is received via a respiratory gating tracking device. For example, the respiratory gating tracking device includes multiple markers that can move, change direction, and change shape during patient movement caused by different stages of the respiratory cycle. In one example, the computing device can use the distance between sets of multiple markers at a given time to determine the stage of the respiratory cycle. In one example, the markers are displayed within the image. For example, the computing device uses the image of the markers to determine the distance between the markers and determine the current stage of the respiratory cycle. Thus, in one example, the respiratory cycle is incorporated into the navigation data to further enhance the direction and position information used for imaging within a particular portion of the anatomical structure.

[0045] In 304a, the computing device receives a second image 304b of the field of view from a second imaging modality. The second image 304b is received after the first image 302b. The second image 302b can be of the target region of the patient. The second image 304b includes an occlusion region 306b. In one example, the second image 304b is received in real time. The second imaging modality is the image sensor 106 of the internal medical imaging system 104. In one example, the computing device receives a plurality of second images 304b. The second images 304b are received following the first image 302b. For example, the first image 302b can be collected before the procedure in which the second image 304b is received. The EBUS device is a real-time imaging device that provides a stream of images. Thus, in these examples, the description of the "second image" is generally used to refer to a second imaging state (e.g., a second imaging state that includes an occlusion). Thus, when method 300a is discussed as "receiving a second image", this can be interpreted as an abbreviation for a stream of images in a second imaging state. In a particular scenario, the computing device can operate on a single image or a stream of images while updating a display device in real time or near real time. For convenience, most of this method is described with respect to individual images, but the operations are performed in real time on a stream of images. In one example, the second image 304b includes a tracking position. In one example, the second image 304b includes an anatomical landmark and / or indicates the current anatomical state. For example, the anatomical state can include a respiratory state.

[0046] In 306a, the computing device identifies one or more occluded portions 306b of the second image 304b. In one example, the computing device identifies one or more occluded portions 306b by evaluating the pixels of the second image 304b. In one example, the computing device determines that all of the pixels within the occluded portion 306b are black or contain data that is not clearly from the imaged tissue. In another example, the computing device compares a set of pixel data from the second image 304b to a baseline value of the pixel data. The baseline value of the pixel can be set to indicate that the pixel is occluded. In one example, the computing device determines that a portion of the second image 304b is occluded if a subset of the set of pixel data is below the baseline value. In one example, instead of evaluating individual pixels, the computing device compares the pixel data within a group of pixels. The computing device may determine that there is an occluded portion 306b when a particular region is exceeded. For example, the computing device can determine that no additional information is being collected when a particular row of data is exceeded. In one example, the computing device detects an immediate change in the post-occlusion quality that causes the occluded portion 306b (e.g., the collected ultrasound data may indicate that there is little ultrasound data reflected back to the transducer beyond a particular depth where a large ultrasound reflection occurs). The occluded image portion can also be identified by identifying the instruments within the image. For example, since a metal needle efficiently reflects ultrasound, a biopsy needle typically generates a line of bright white pixels within an ultrasound image. In one example, the computing device examines the second image 304b within the segment based on the imaging ray and identifies an occluded portion 306b when data is expected in the second image 304b but is not collected (e.g., when the second image 304b generates zeros beyond a particular point). In one example, the computing device detects an occluded portion 306b by detecting air bubbles within the field of view of the second image 304b.The computing device can detect bubbles using the techniques described above. The computing device can detect bubbles by applying an image processing algorithm trained to detect imaging features of the bubbles. The image processing algorithm can be utilized to identify these known artifacts and then interpolate the occluded regions. In a particular example, the image processing algorithm can look for changes in contrast. For example, regions where there is little or no change in contrast are more likely to be occluded regions of the image.

[0047] In 308a, the computing device extracts the unoccluded portion of the field of view from the first image 302b. The unoccluded portion from the first image 302b substantially corresponds to the occluded portion 306b of the second image 304b. The unoccluded portion of the first image 302b is acquired prior to the second image 304b. Each of the first image 302b and the second image 304b includes position and orientation data (e.g., tracking position) associated with the image. In one example, the tracking position includes respiratory state information. In one example, the computing device utilizes information regarding the field of view and the tracking position to understand the relationship between the first image 302b and the second image 304b. As another example, the computing device utilizes a plurality of first images to determine the unoccluded portion of the first view extracted from the first image 302b having a tracking position substantially the same as that of the second image 304b. In a particular example, the unoccluded portion can be identified within the first image 302b that was not taken at exactly the same tracking position as the second image 304b. In such an example, the position and orientation data associated with the second image 304b can be used to accurately identify where in 3D space the patient's tissue is occluded, and the position and orientation data associated with the first image 302b can be used to identify the unoccluded image data corresponding to this 3D space for use in the composition into the second image 304b.

[0048] Before extracting the non-occluded portion from the first image 302b, the first image 302b is deformed to correspond to the second image 304b. In this example, since the first image 302b is obtained from a different imaging modality compared to the second (real-time) image 304b, the first image 302b must be adjusted (deformed) to appropriately correspond to the second image 304b. Alternatively or additionally, the non-occluded portion may be deformed while the composite image 310b is being generated at 310a after the non-occluded portion has been extracted. In one example, the first image 302b is deformed to conform to the current anatomical state of the second image 304b. Such deformation can include identifying anatomical landmarks within the first image 302b and the second image 304b. After identifying the anatomical landmarks, the first image 302b can be deformed such that the anatomical landmarks within the first image 302b are aligned with the anatomical landmarks of the second image 304b. In one example, the anatomical landmarks include blood vessels.

[0049] In one example, when a second image 304b is collected in a specified respiratory state, the computing device proceeds to method 300a. In one example, the specified respiratory state is the total lung capacity and / or the tidal volume. As another example, the first image 302b can be collected in a specified respiratory state (e.g., total lung capacity and tidal volume) and deformed using a deformation matrix to interpolate data and imaging between the specified respiratory states (e.g., total lung capacity and tidal volume). For example, the deformation matrix can be used when the first image 302b and the second image 304b are captured during different respiratory states. Additionally or alternatively, both the first image 302b and the second image 304b can be collected in a predefined respiratory state (e.g., total lung capacity and / or tidal volume) such that all deformations occur in images collected in the same respiratory state. For example, image capture can be triggered based on the respiratory state. As another example, images are continuously collected, but only the images collected in the specified respiratory state are used to generate the composite image. As another example, the first image 302b can be selected from a plurality of first images such that the selected first image 302b is captured during the same respiratory state as the second image 304b.

[0050] In one example, a three-dimensional (3D) model of a target region is generated using a plurality of two-dimensional (2D) images acquired by an image sensor. Using the 3D model, newly captured or received images can be compared to the 3D model to determine the location or position of the images. Location or position information can be used to extract a portion (e.g., a non-occluded portion) of a first image 302b for alignment with an occluded portion 306b of a second image 304b. The extracted portion can overlay the occluded portion 306b, can highlight it, can replace the occluded portion 306b, or can be a combination thereof. As another example, a computing device simulates image data to replace an occluded portion 306b with information that would be provided if that portion were not occluded. As another example, a computing device digitally reconstructs CT image data to virtually fill or replace an occluded portion 306b of a second image 304b when creating a synthetic image 310b. In one example, when collecting, capturing, or receiving an image or image data, the image is registered using four-dimensional (4D) data. For example, image data is collected to render a three-dimensional (3D) image (e.g., a volume image) of a target region. The 3D image can further include motion resulting from a respiratory cycle, and thus 4D data and a 4D image are generated.

[0051] In one example, a CT image is converted to an ultrasound image. In some cases, converting a CT image to an ultrasound image may result in a sharper image or a smoother synthetic image being provided because both images are ultrasound images. For example, previously acquired CT data representing an occluded portion (e.g., a non-occluded portion of CT image data) can be processed to resemble ultrasound data and then synthesized into an occluded portion of a real-time ultrasound image.

[0052] In 310a, the computing device generates a composite image 310b. In one example, the computing device generates a composite image by replacing the occluded portion 306b of the second image 304b with the non-occluded portion of the first image 302b. For example, the computing device extracts the occluded portion 306b from the second image 304b and replaces the occluded portion 306b with the non-occluded portion of the first image 302b. In another example, the computing device generates a composite image by overlaying the non-occluded portion of the first image 302b on the occluded portion 306b of the second image 304b. The composite image 310b can be generated substantially in real time. The system determines the scale and pose adjustment factors between a real-time ultrasound stream (e.g., the second image 304b) and a previously acquired CT image (e.g., the first image 302b), and based on the scale and pose adjustment, can transform a part of the previously acquired CT image (e.g., the first image 302b). The non-occluded portion of the first image 302b can be deformed to align with the occluded portion 306b of the second image 304b. In a particular example, the non-occluded portion can be identified within the first image 302b that was not taken at exactly the same navigation position and orientation as the second image 304b. In these examples, the computing device can transpose the non-occluded portion of the first image based on a comparison of the navigation data between the first image 302b and the second image 304b, or can interpolate between the non-occluded portion of the first image 302b and the occluded portion 306b of the second image 304b based on the navigation data of each image. When transposing the data, the non-occluded portion is from the first image 302b where the position and orientation are close enough, and it may only be necessary to shift the non-occluded portion to the position of the occluded portion 306b of the second image 304b. When the position and orientation between the first image 302b and the second image 304b are more important (but within the range where typical adjustment of the non-occluded portion is possible), interpolation may be necessary.

[0053] In 312a, the computing device displays the composite image 310b. Despite the real-time second image having the occlusion portion 306b, the user is presented with an image representation of the tissue within the entire field of view as if the occlusion portion 306b has no effect of occluding the second image 304b. If the computing device does not receive an image having the occlusion portion 306b, the first image 302b is displayed without being changed by the processes described herein. In one example, the computing device enables the user to switch between multiple image display modes. For instance, the computing device may include a first mode that generates a composite image when an occlusion portion is detected, and a second mode that displays the unaltered real-time image. In some embodiments, the system may be configured to display a graphical user interface (GUI) element that indicates the contour between a portion of the image stream corresponding to the unaltered real-time image data and a portion corresponding to the composite image data that complements the occlusion portion. Examples of such a GUI include, but are not limited to, a colored border that outlines the composite image portion, or displaying the composite image portion in a different color or shade compared to the unaltered image portion, or both.

[0054] Method 300a operates on a computing device in real - time and continuously throughout the procedure. For example, as the instrument extends into the field of view, the second image 304b continues to be collected along with the occlusion portion 306b, and method 300a operates in real - time to generate and display a composite image 310b using a portion of the first image 302b extracted to replace the occlusion portion 306b of the second image 304b. As another example, when the imaging device moves through a passage and there are air bubbles in the passage, the second image 304b continues to be collected along with the occlusion portion 306b, and method 300a operates in real - time to generate a display composite image 310b using a portion of the first image 302b extracted to replace the occlusion portion 306b of the second image 304b. The first image 302b and the second image 304b can be continuously received, and new composite images can be displayed throughout the procedure. As another example, all of the available first image 302b from the first imaging modality is received at once. As another example, the computing device receives the first image 302b of the target region, but additional images are available and can be received a second time if needed. When method 300a operates, the first image 302b can be selected to be the image received immediately prior to the second image 304b that includes the occlusion portion 306b. Among other advantages, the techniques described herein enable the user to view the entire boundary of the target tissue (e.g., SPN) even while a sampling needle or ablation device is partially inserted into the target tissue to obtain a biopsy or perform an ablation procedure. In this way, valuable information that is lacking in the capabilities provided by conventional medical imaging systems (e.g., EBUS systems) is provided to the user within the displayed image.

[0055] FIG. 4A shows an example of a method 400a for generating a composite image with graphic display. FIG. 4B shows an exemplary set of images 400b to be processed into a composite image using the method of FIG. 4A. In one example, the method 400a can be executed by a computer device (e.g., the machine of FIG. 6) communicatively coupled to an EBUS. The computer device includes processing circuitry for performing various image processing tasks described in connection with the method 400a. Additionally, the computer device can include, among other things, an output display such as a monitor for displaying, in particular, ultrasonic images and various additional user interface elements.

[0056] At the start of the method 400a, such as at the start of a procedure, the imaging device (e.g., the internal medical imaging system 104) is positioned so as to be able to image a desired field of view. In one example, the imaging device 104 is an EBUS device with navigation capabilities, such as an embedded sensor coil that provides signals to a navigation system that enables accurate tracking of the position and orientation within the patient when placed, for example, within an electromagnetic field generated during the procedure. The navigation system of the imaging device can utilize electromagnetic (EM) tracking techniques that enable tracking of the imaging device within six degrees of freedom. In one example, the device rotates between 15 degrees and 360 degrees. In one example, interpolation is used to create images between images collected at different degrees. In one example, the tracked position includes the position of the imaging device and the orientation of the imaging device.

[0057] In one example, an imaging system, or another connected system, monitors and collects respiratory information associated with an image. For example, a portion of the respiratory cycle is captured. In one example, the respiratory information is received via a respiratory gating tracking device. For example, the respiratory gating tracking device includes a plurality of markers that can move, change direction, and change shape during patient movement caused by different stages of the respiratory cycle. In one example, a computing device can use the distances between sets of the plurality of markers at a given time to determine the stage of the respiratory cycle. In one example, the markers are displayed within the image. For example, the computing device uses an image of the markers to determine the distances between the markers and to determine the current stage of the respiratory cycle. Thus, in one example, the respiratory cycle is incorporated into the navigation data to further enhance the direction and position information used for imaging within a particular portion of the anatomical structure.

[0058] In 402a, the computing device receives a first image 402b of the field of view. The computing device receives the first image 402b captured from the image sensor 106 of the internal medical imaging system 104. In one example, the first image 402b is of a target region of a patient. For example, the first image 402b is of a target region of a patient. For example, the first image 202b may include in the field of view a target tissue, such as a solitary pulmonary nodule (SPN) located adjacent to just outside the airway, for which a biopsy sample and / or ablation treatment is desired. The first image 402b may be unobstructed. In one example, the first image 202b includes a tracking position. In one example, the computing device receives a plurality of first images 402b. The EBUS device is a real-time imaging device that provides a stream of images. Thus, in these examples, the description of the "first image" is generally used to refer to a first imaging state (e.g., a first imaging state without obstruction). Thus, when method 400a is discussed as "receiving a first image", this can be interpreted as an abbreviation for a stream of images in the first imaging state. In a particular scenario, the computing device may operate on a single image or a stream of images while updating a display device in real-time or near real-time. For convenience, most of the method is described with respect to individual images, but the operations are performed in real-time on a stream of images.

[0059] In 404a, the computing device receives a second image 404b of the field of view. The second image 404b is received after the first image 402b. The second image 404b includes an occlusion region 406b. In one example, the computing device receives the second image 404b in real time. In one example, the computing device may receive multiple second images 404b. In one example, the computing device receives the second image 404b immediately after the first image 402b. In one example, the second image 404b includes a tracking position. The second image 404b that includes the occlusion region 406b may be a consecutive image from the same image stream as the first image 402b was received from. For example, each of the first image 402b and the second image 404b may be provided from an EBUS sampling device to an ultrasonic image processor within the same image stream during a single procedure.

[0060] In 406a, the computing device detects the instrument 406b within the field of view of the second image 404b. The instrument 406b can be the instrument 120 of FIGS. 1A and 1B. If the instrument 406b is a needle, the needle may cause a bright line in the second image 404b. For example, since a metal needle efficiently reflects ultrasound, a biopsy needle typically generates a line of bright white pixels in an ultrasound image. The bright line is used to detect the instrument 406b. In one example, the internal medical imaging system 104 includes a sensor near the exit port 114 so that the sensor can detect when the instrument 120 exits the internal medical imaging system 104. In one example, the sensor is the imaging sensor 106 of the internal medical imaging system 104. For example, the computing device can use control information to detect the instrument 406b. The computing device can detect that a portion of the instrument 406b extends within the field of view of the second image 404b. In one example, the computing device uses artificial intelligence, machine learning, or other imaging techniques to determine that the instrument 406b extends from the internal medical imaging system 104 and thus occludes a portion of the field of view. For example, the computing device detects the instrument 406b by identifying known instrument features within the field of view of the second image 404b. In addition to determining that the instrument 406b has been deployed or extended within the field of view, the computing device can determine the length by which the instrument 406b extends from the internal medical imaging system 104. As another example, the computing device applies a machine learning model trained to detect known image features generated by the instrument 406b.

[0061] In 408a, the computing device identifies one or more occluded portions 408b of the second image 404b. In one example, the computing device identifies one or more occluded portions 408b by evaluating the pixels of the second image 404b. In one example, the computing device determines that all of the pixels within the occluded portion 408b are black or contain data that is clearly not from the imaged tissue. In another example, the computing device compares a set of pixel data from the second image 404b to a baseline value of the pixel data. The baseline value of the pixel can be set to indicate that the pixel is occluded. In one example, the computing device determines that a portion of the second image 404b is occluded if a subset of the set of pixel data is below the baseline value. In one example, instead of evaluating individual pixels, the computing device compares the pixel data within a group of pixels. The computing device can determine that the area behind the instrument 406b is the occluded portion 408b. For example, the computing device determines that no additional information is being collected beyond a particular row of data. For example, the computing device determines that no additional information is being collected beyond the instrument 406b. In one example, the computing device detects an immediate change in the post-occlusion quality that causes the occluded portion 408b (e.g., the collected ultrasonic data can indicate that there is little ultrasonic data reflected back to the transducer beyond a particular depth where a large ultrasonic reflection occurs). The occluded image portion can also be identified by identifying the instrument 406b within the image. For example, since a metal needle efficiently reflects ultrasound, a biopsy needle typically produces a line of bright white pixels in an ultrasound image. In one example, the computing device examines the second image 404b within the segment based on the imaging ray and identifies the occluded portion 408b if data is expected in the second image 404b but not collected (e.g., if the second image 404b produces zeros beyond the instrument 406b).In one example, the instrument 406b (e.g., a sheath) includes an echogenic feature to enhance the detectability of the imaged instrument 406b. For example, the echogenic feature has a high reflectivity. As another example, the echogenic feature is an annular ring structure. The echogenic feature may degrade the image quality and increase the manufacturing cost of the instrument. In one example, when the instrument 406b does not include the echogenic feature, the image features of the sheath can be used to detect the instrument 406b in the second image 404b. In one example, the computing device detects the occluded portion 408b by detecting the instrument 406b within the field of view of the second image 404b. The computing device detects the instrument 406b using the techniques described above. In one example, the computing device detects the instrument 406b by applying an image processing algorithm trained to detect the imaging features of the instrument. The image processing algorithm can be utilized to identify these known artifacts and then interpolate the occluded region 408b. In a particular example, the image processing algorithm can look for changes in contrast. For example, regions with little or no change in contrast are likely to be occluded regions of the image.

[0062] In 410a, the computing device extracts the unobstructed portion of the field of view from the first image 402b. Since the imaging sensor (e.g., transducer) may be small, the instrument 406b may cause occluded portions 408b of various sizes depending on how much the instrument 406b is deployed. However, a complete two-dimensional (2D) imaging slice may be desired to assist in performing a procedure (e.g., image collection, tissue sampling). By extracting the unobstructed portion of the first image 402b corresponding to the occluded portion 408b of the second image 404b, the occluded portion 408b may be filled in to show how the image would look if the occlusion (e.g., instrument 406b, bubble) did not interfere with imaging. The unobstructed portion from the first image 402b substantially corresponds to the occluded portion 408b of the second image 404b. Each of the first image 402b and the second image 404b includes position and orientation data (e.g., tracking position) associated with the image. In one example, the computing device utilizes information regarding the field of view and the tracking position to understand the relationship between the first image 402b and the second image 404b. As another example, the computing device utilizes a plurality of first images to determine the unobstructed portion of the first view extracted from a first image 402b having a tracking position substantially similar to the second image 404b. In a particular example, the unobstructed portion may be identified within the first image 402b that was not taken at exactly the same tracking position as the second image 404b. In such an example, the position and orientation data associated with the second image 404b may be used to accurately identify where in 3D space the patient's tissue is occluded, and the position and orientation data associated with the first image 402b may be used to identify the unobstructed image data corresponding to this 3D space for use in compositing onto the second image 404b.

[0063] In 412a, the computing device generates a composite image 412b. In one example, the computing device generates a composite image by replacing the occluded portion 408b of the second image 404b with the non-occluded portion of the first image 402b. For example, the computing device extracts the non-occluded portion of the first image 402b and generates the composite image 412b by replacing the occluded portion 408b of the second image 404b. In another example, the computing device generates the composite image 412b by overlaying the non-occluded portion of the first image 402b on the occluded portion 408b of the second image 404b. The computing device may generate the composite image 412b substantially in real time. The computing device determines the scale and pose adjustment coefficients between a real-time ultrasound stream (e.g., the second image 404b) and a previously acquired ultrasound image (e.g., the first image 402b), and may transform a portion of the previously acquired ultrasound image (e.g., the first image 402b) based on the scale and pose adjustment. The non-occluded portion of the first image 402b can be deformed to align with the occluded portion 406b of the second image 404b. In a particular example, the non-occluded portion can be identified within the first image 402b that was not taken at exactly the same navigation position and orientation as the second image 404b. In these examples, the computing device can transpose the non-occluded portion of the first image 402b based on a comparison of the navigation data between the first image 402b and the second image 404b, or interpolate between the non-occluded portion of the first image 402b and the occluded portion 408b of the second image 404b based on the navigation data of each image. When transposing the data, the non-occluded portion is from the first image 402b where the position and orientation are close enough, and it may only be necessary to shift the non-occluded portion to the position of the occluded portion 408b of the second image 404b. When the position and orientation between the first image 402b and the second image 404b are more important (but within the range where typical adjustment of the non-occluded portion is possible), interpolation may be necessary.

[0064] In 414a, the computing device generates a graphic representation 414b of the instrument 406b on the composite image 412b. In one example, as the instrument 406b continues to extend, the graphic representation 414b changes dynamically. For example, when the computing device receives a new second image 404b, the composite image 412 can be updated using the new second image 404b, the updated unoccluded portion from the first image 402b, and the updated graphic representation 414b. In one example, if it is identified that the second image 404b has not changed substantially and the same first image 402b is used, the computing device predicts the occluded portion 408b based on the instrument 406b and creates a composite image 412b that replaces the entire predicted occluded portion. Thus, the composite image 412b can remain the same, but the graphic representation 414b of the instrument is continuously updated as the instrument 406b extends further and / or contracts. In one example, the graphic representation 414b looks substantially the same as the instrument 406b. For example, the graphic representation 414b is shaped like a needle. As another example, the graphic representation 414b is shaped like a sheath. In another example, the graphic representation 414b shows both the needle and the sheath. In one example, the graphic representation 414b provides an indication of how far the instrument 406b has extended from the internal medical imaging system 104. In one example, the graphic representation 414b provides an indication that the instrument 406b has been extended by a predefined amount from the internal medical imaging system. Such a graphic representation helps to confirm that the sheath has been extended by a predefined amount before the needle is extended. In one example, the graphic representation 414b includes a numerical display of the extension length of the instrument 406b. In one example, the graphic representation 414b includes a color-coding corresponding to a predefined extension length of the instrument 406b.

[0065] In 416a, the computing device displays the composite image 412b. Despite the real-time second image 404b having an occlusion portion 408b, the user is presented with an image representation of the tissue within the entire field of view as if the occlusion portion 408b had no effect on occluding the second image 404b. If the computing device does not receive an image with an occlusion portion 408b, the first image 402b is displayed without being modified by the processes described herein. In one example, the computing device allows the user to switch between multiple image display modes. For instance, the computing device may include a first mode that generates a composite image when an occlusion portion is detected and a second mode that displays the unmodified real-time image. In some embodiments, the system may be configured to display a graphical user interface (GUI) element that indicates the contour between a portion of the image stream corresponding to the unmodified real-time image data and a portion corresponding to the composite image data that complements the occlusion portion. Examples of such a GUI may include, but are not limited to, a colored border that outlines the composite image portion, or displaying the composite image portion in a different color or shade compared to the unmodified image portion, or both.

[0066] Method 400a operates on a computing device in real-time and continuously throughout the procedure. For example, while instrument 406b extends into the field of view, second image 404b continues to be collected along with occlusion portion 406b, and method 400a operates in real-time to generate and display composite image 412b using a portion of first image 402b extracted to replace occlusion portion 406b of second image 404b. As another example, first image 402b continues to be collected until a second image 404b having occlusion portion 408b is identified. First image 402b and second image 404b can be continuously received, and new composite image 412b can be displayed throughout the procedure. When method 400a operates, first image 402b can be selected to be the image received immediately prior to second image 404b that includes occlusion portion 406b. Among other advantages, the techniques described herein enable a user to view the entire boundary of a target tissue (e.g., SPN) even while a sampling needle or ablation device is partially inserted into the target tissue to obtain a biopsy or perform an ablation procedure. In this way, valuable information that is lacking in the capabilities provided by conventional medical imaging systems (e.g., EBUS systems) is provided to the user within the displayed image.

[0067] The steps or operations of methods 200a, 300a, and 400a are shown in a particular order for convenience and clarity, and many of the operations described can be performed in a different order or in parallel without substantially affecting the other operations. The methods 200a, 300a, and 400a described include operations performed by multiple different actors, devices, and / or systems. It is understood that a subset of the operations described in methods 200a, 300a, and 400a can be attributable to a single actor, device, or system and can be considered a separate stand-alone process or method.

[0068] FIG. 5A is a schematic diagram 500a of the non-occluded field of view 502 from the image sensor 506 of the internal medical imaging system 504. In one example, the internal medical imaging system 504 is inserted into the target region of a patient. The medical imaging system 504 is equivalent to the medical imaging systems described above.

[0069] In one example, the internal medical imaging system 504 is an endobronchial ultrasound (EBUS) tissue sampling device. In one example, the internal medical imaging system 504 is used to navigate towards the target nodule 510 and image it before tissue sampling. In one example, the target nodule 510 is a pre-identified tissue where a biopsy sample is desired. In one example, ultrasound (US) imaging is used in the respiratory region to reliably identify the target region for taking a biopsy sample because US imaging displays both the tissue and the instrument (e.g., a needle or a sheath) in real time.

[0070] In one example, the image sensor 506 is disposed at the distal end of the internal medical imaging system 504. In one example, the image sensor 506 extends along a portion of the distal end. In one example, the image sensor 506 is substantially flat. In one example, the image sensor 506 is an ultrasonic transducer. In one example, the image sensor 506 provides the field of view 502 to the imaging display.

[0071] The field of view 502 of the image sensor extends from the proximal boundary 502a to the distal boundary 502b. In the example shown, the entire field of view 502 is the non-occluded field of view 508.

[0072] FIG. 5B shows a schematic diagram 500b of the needle 520 that causes the occluded portion 522 of the field of view 502 of the internal medical imaging system 504. In one example, the internal medical imaging system 504 is inserted into the target region of a patient.

[0073] In one example, the needle 520 advances through the internal medical imaging system 504 via the working channel. In one example, the needle 520 extends from the exit port of the internal medical imaging system 504. In one example, the exit port includes an inclined path such that the needle 520 extends at an angle from the internal medical imaging system 504. In one example, the needle 520 is a sampling needle for capturing biopsy material of a target tissue (e.g., nodule 510). For example, the target tissue can be pre-identified via a pre-operative image source such as a CT scan. In one example, the internal medical imaging system 504 includes a sheath that extends from the exit port. In one example, the sheath is a flexible plastic sleeve. As an example, the sheath covers the needle 520. As another example, the sheath extends partially outside the internal medical imaging system 504 before the needle 520 extends from the exit port such that the sheath protects the exit port.

[0074] During imaging, when the needle 520 extends from the internal medical imaging system 504, in addition to the non-occluded portion 508, an occluded portion 522 occurs within the field of view 502. The occluded portion 522 extends from a proximal boundary 522a to a distal boundary 522b. The occluded portion 522 is caused by the interruption of the transmission of ultrasonic energy to the tissue beyond the needle 520 in relation to the image sensor 506 (e.g., a US transducer). For example, ultrasonic waves transmitted from the image sensor 506 (e.g., an ultrasonic transducer) can propagate through the tissue until they reach the boundary between the tissue and the needle 520, where a significant amount of ultrasonic energy is reflected towards the image sensor 506 and a small amount continues to propagate away from the image sensor 506 and into the needle 520. Seeing the needle 520 within the field of view 502 may be advantageous for the user, but the occluded portion 522 caused by the needle 520 may lack information indicating the characteristics of the tissue beyond the needle 520. In this example, the field of view 502 and the occluded portion 522 share a common proximal boundary (proximal boundary 502a and proximal boundary 522a), while the distal boundary 522b of the occluded portion 522 will advance distally as the needle 520 extends into the field of view 502. Such a lack of information may affect the ability of the physician to direct the internal medical imaging system 504 towards the target region of the patient, and / or may affect the ability of the physician to direct the needle 520 towards the target nodule 510 within the target region.

[0075] In one example, the internal medical imaging system 504 of FIGS. 5A and 5B includes or is connected to a display device for displaying an image of the field of view 502 acquired by the image sensor 506. In one example, the internal medical imaging system 504 is connected to a computing device. The computing device is capable of image processing (e.g., the machine of FIG. 6).

[0076] FIG. 5C shows a schematic view 550 of the field of view 552 of FIG. 5A where the corresponding unobstructed portion 572 is identified by a computing device. When the computing device receives an image from an internal medical imaging system, the computing device determines when a portion of the image is occluded. In one example, the computing device identifies one or more occluded portions of the received image in real time by evaluating the pixels of the field of view, identifying a needle within the field of view, identifying the area behind the needle within the field of view, and determining a change in quality in a portion of the field of view of the real-time image. Such methods have been described above.

[0077] When an occluded portion is detected, the computing device can search for an image received prior to the image including the occluded portion. For example, the field of view 552 of a previously acquired image can extend from a proximal boundary 552a to a distal boundary 552b and can include the target nodule 560. The computing device identifies an unobstructed portion 572 of the previously acquired image that corresponds to the occluded portion of the real-time image. For example, the corresponding unobstructed portion 572 is aligned with the occluded portion 522 of FIG. 5B. The corresponding unobstructed portion 572 extends from a proximal boundary 572a to a distal boundary 572b. In the example shown, the corresponding unobstructed proximal boundary 572a and the field of view proximal boundary 558a are the same. The previously acquired image also includes an unobstructed portion 558 that corresponds to the unobstructed portion at real-time. Since the corresponding unobstructed portion 572 is located behind the needle, the corresponding unobstructed portion 572 is not also aligned with the lower boundary 552c of the field of view 552, but instead includes a lower boundary 572c that follows the trajectory of the needle.

[0078] After the computing device identifies the corresponding unobstructed portion 572 of the previously acquired image, the computing device extracts the corresponding unobstructed portion 572 of the previously acquired image. By extracting the corresponding unobstructed portion 572 of the previously acquired image, the occluded portion may be filled in to show what the image would look like if the occlusion (e.g., the needle) did not interfere with imaging.

[0079] FIG. 5D shows a schematic diagram 580 of a composite image 582 created from the fields of view of FIGS. 5A and 5B. The computing device displays the composite image 582. Thus, the composite image 582 is the image that is displayed to the user.

[0080] When the computing device receives an image from the internal medical imaging system, the computing device determines when a portion of the real-time image will be occluded. In one example, the computing device performs evaluating the pixels of the field of view, identifying the needle within the field of view, identifying the area behind the needle within the field of view, and determining a change in quality in a portion of the field of view of the real-time image to identify one or more occluded portions of the received image in real time. Such methods have been described above.

[0081] The composite image 582 extends from a proximal boundary 582a to a distal boundary 582b and may include the target nodule 560. The composite image 582 includes an unoccluded portion 588 and an occluded portion 572. The computing device uses the unoccluded portion 588 of the real-time image. Since a portion of the real-time image is occluded, the computing device uses the corresponding unoccluded portion 572 (e.g., the corresponding unoccluded portion 572 as shown in FIG. 5C) from a previous image to create the composite image 582. The unoccluded (occluded) portion 572 extends from a proximal boundary 572a to a distal boundary 572b. In the example shown, the unoccluded proximal boundary 572a and the field of view proximal boundary 582a are the same. Since the unoccluded portion 572 is located behind the needle, the unoccluded portion 572 also does not align with the lower boundary 582c of the field of view 582 but instead includes a lower boundary 572c that follows the trajectory of the needle.

[0082] In one example, the computing device generates the composite image 582 by replacing the occluded portion of the real-time image with the corresponding non-occluded portion 572 of a previously acquired image. For example, the computing device extracts the corresponding non-occluded portion 572 of the previously acquired image and generates the composite image 582 by replacing the occluded portion of the real-time image. In another example, the computing device generates the composite image 582 by overlaying the corresponding non-occluded portion 572 of the previously acquired image on the occluded portion of the real-time image. The computing device can generate the composite image 582 substantially in real time.

[0083] In the illustrated example, the composite image 582 includes a graphical representation 570 of a needle. The computing device generates the graphical representation 570. In one example, as the needle continues to extend, the graphical representation 570 changes dynamically. For example, when the computing device receives a real-time image, the composite image 582 can be updated using the non-occluded portion 588 of the real-time image, the updated corresponding non-occluded portion 572 from the previously acquired image, and the updated graphical representation 570. In one example, the graphical representation 570 looks substantially like the needle. For example, the graphical representation 570 has a shape that looks like a needle. In one example, the graphical representation 570, the composite image 582, or the graphical user interface displaying the composite image 582 provides an indication of how far the needle extends from the internal medical imaging system. In one example, the graphical representation 570, the composite image 582, or the graphical user interface displaying the composite image 582 provides an indication that the needle has extended a predetermined amount from the internal medical imaging system. In one example, the graphical representation 570 includes a numerical display of the extension length of the needle. In one example, the graphical representation 570 includes a color-coding corresponding to a predetermined extension length of the needle.

[0084] FIG. 6 shows a block diagram of an exemplary machine 600 upon which any one or more of the techniques (processes) discussed herein may be executed, according to some embodiments. In alternative embodiments, machine 600 may operate as a stand-alone device and / or may be connected (e.g., network-connected) to other machines. In a networked deployment, machine 600 may operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In one example, machine 600 may function as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 600 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is shown, the term "machine" shall be construed to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one of the methodologies described herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, and the like.

[0085] A machine (e.g., a computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 604, and a static memory 606, some or all of which may communicate with each other via an interlink (e.g., a bus) 608. The machine 600 may further include a display unit 610, an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In one example, the display unit 610, the input device 612, and the UI navigation device 614 may be a touch screen display. The machine 600 may further include a storage device (e.g., a drive unit) 616, a signal generation device 618 (e.g., a speaker), a network interface device 620, and one or more sensors 621 such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 600 may include an output controller 628 such as a serial (e.g., universal serial bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with and / or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0086] The storage device 616 may include a machine-readable medium 622 that stores one or more sets of data structures or instructions 624 (e.g., software) that are implemented or utilized by any one or more of the techniques or functions described herein. The instructions 624 may reside, completely or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during execution by the machine 600. In one example, one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the storage device 616 may constitute a machine-readable medium.

[0087] Machine-readable medium 622 is shown as a single medium, but the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated cache and server) configured to store one or more instructions 624. The term "machine-readable medium" can store, encode, or carry instructions for execution by machine 600, cause machine 600 to execute any one or more of the techniques of the present disclosure, or store, encode, or carry a data structure used by or associated with such instructions. Non-limiting examples of machine-readable media can include solid state memory, optical media, and magnetic media.

[0088] Command 624 can be further transmitted or received via communication network 626 using a transmission medium via network interface device 620 using any one of many transfer protocols (e.g., frame relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communication networks can include, among other things, local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile telephone networks (e.g., cellular networks), plain old telephone service (POTS) networks, wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard family known as Wi-Fi (registered trademark), the IEEE 802.16 standard family known as WiMAX (registered trademark)), the IEEE 802.15.4 standard family, peer-to-peer (P2P) networks, etc. In one example, network interface device 620 can include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to communication network 626. In one example, network interface device 620 can include multiple antennas for wireless communication using at least one of single input multiple output (SIMO), multiple input multiple output (MIMO), or multiple input single output (MISO) techniques. The term "transmission medium" is construed to include any non-transitory medium that can store, encode, or carry instructions for execution by machine 600 and includes digital or analog communication signals or other non-transitory media for facilitating communication of such software.

[0089] The foregoing detailed description includes references to the accompanying drawings that form a part of the detailed description. The drawings are shown by way of example of specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as "examples." Such examples can include elements in addition to those illustrated or described. However, the inventors also contemplate examples in which only the illustrated or described elements are provided. Further, the inventors contemplate examples in which any combination or substitution of the illustrated or described elements (or one or more aspects thereof) is used with respect to a particular example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) illustrated or described herein.

[0090] Where usage is not consistent between this specification and any document incorporated by reference, the usage in this specification prevails. As used herein, the terms "including" and "in which" are used as the plain English equivalents of the respective terms "comprising" and "wherein." Also, in the appended claims, the terms "including" and "comprising" are open-ended, i.e., a system, device, article, composition, formulation, or process that includes elements in addition to those recited after such terms in the claims is still considered to be within the scope of that claim.

[0091] In this specification, the terms "a" or "an" are used to include one or more, regardless of any other instance or use of "at least one" or "one or more", as is common in patent documents. In this specification, the term "or" is used to refer to a non-exclusive "or" such that, unless otherwise specified, "A or B" includes "A but not B", "B but not A", as well as "A and B". In this specification, the terms "comprising" and "therein" are used as the plain English equivalents of the respective terms "including" and "therein". Also, in the appended claims, the terms "comprising" and "including" are unlimited, i.e., they mean a system, device, article, composition, formulation, or process that includes elements in addition to those recited after such terms in the claims, and is still considered to be within the scope of that claim. Further, in the appended claims, terms such as "first", "second", and "third" are used merely as labels and are not intended to impose numerical requirements for their purposes.

[0092] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can also be used by those skilled in the art upon reviewing the above description. The abstract is provided to comply with 37 C.F.R. § 1.72(b) of the United States Patent Law Rules to enable the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it is not to be used to interpret or limit the scope or meaning of the claims. Also, in the forms for carrying out the above invention, various features may be grouped to rationalize the disclosure. This should not be construed as intending that any disclosed features not claimed are essential to any of the claims. Rather, the subject matter of the invention may not be encompassed by all the features of the specific embodiments disclosed. Accordingly, the appended claims are incorporated into the forms for carrying out the invention as examples or embodiments, each claim standing on its own as a separate embodiment, and such embodiments are intended to be combinable with each other in various combinations or permutations. The scope of the present invention should be determined with reference to the appended claims and the full scope of equivalents to which such claims are entitled.

[0093] (Example) The following examples provide an overview of important aspects and embodiments of a system and method for synthesizing pre-acquired image data into occluded regions of a real-time image stream. The examples are written in plain language to illustrate important features of the present invention. The examples are not intended to show the full scope of the invention as defined in the claims, but are intended to highlight certain inventive concepts, components, steps, and advantages in a simplified and non-limiting manner. The examples cover real-time synthesis techniques for filling occlusions in ultrasonic images, multimodality image synthesis methods, graphical occlusion indicators, and related concepts disclosed herein. These examples are intended to supplement the technical details provided elsewhere in this specification.

[0094] (Example 1) A method for real-time replacement of an occluded portion of an ultrasonic image stream. The method can include receiving a first image of a field of view, receiving, subsequent to the first image, a second image of the field of view that includes an occluded region, identifying the occluded portion, extracting a non-occluded portion of the first image corresponding to the occlusion, generating a composite image by replacing the occluded portion with the non-occluded portion, and displaying the composite image in real time.

[0095] (Example 2) Including the subject matter of Example 1, and having an additional function of identifying an occluded portion by detecting an instrument within the field of view.

[0096] (Example 3) Including the subject matter of Example 2, and having an additional function of detecting an instrument by identifying characteristics of a known instrument.

[0097] (Example 4) Including the subject matter of Example 3, and having an additional function of detecting an instrument by applying a machine learning model trained based on characteristics of the instrument.

[0098] (Example 5) Including the subject matter of any one of Examples 2 to 4, and having an additional function of detecting an instrument based on sensor or control information indicating that a part of the instrument extends into the field of view.

[0099] (Example 6) Including the subject matter of any one of Examples 2 to 5, and having an additional function of generating a graphic representation of the instrument on the composite image, and the representation dynamically changes to indicate the extended length of the instrument.

[0100] (Example 7) Including the subject matter of Example 6, and having an additional function of providing a display indicating that the instrument has extended by a predetermined amount.

[0101] (Example 8) It includes the subject matter of Example 7 and has the function that the display is a numerical value of the elongation length.

[0102] (Example 9) It includes the subject matter of Example 7 and has the function that the display uses color coding based on the elongation length.

[0103] (Example 10) It includes the subject matter of any one of Examples 1 to 9 and has an additional function of identifying the occluded part by detecting bubbles using an image processing algorithm.

[0104] (Example 11) It includes the subject matter of any one of Examples 1 to 10 and has an additional function of identifying the occluded part by comparing pixel data with a baseline level and determining the occluded area based on pixels below the baseline level.

[0105] (Example 12) It includes the subject matter of any one of Examples 1 to 11 and has an additional function of tracking the position and orientation of the first image and the second image.

[0106] (Example 13) It includes the subject matter of Example 12 and has an additional function of deforming the non-occluded part of the first image when generating a composite image.

[0107] (Example 14) It includes the subject matter of Example 13 and has an additional function of deforming the non-occluded part by interpolation considering the differences in position and orientation between the first image and the second image.

[0108] (Example 15) An internal medical imaging system including an image sensor, a display device, and a computing device. The computing device can receive a first image and a second image, identify an occluded portion of the second image, extract a non-occluded portion of the first image, generate a composite image, and display the composite image in real time.

[0109] (Example 16) Including the subject matter of Example 15, and having an additional function of a computing device to detect an instrument in the field of view to identify an occluded portion.

[0110] (Example 17) Including the subject matter of Example 16, and having an additional function of detecting an instrument by identifying characteristics of a known instrument.

[0111] (Example 18) Including the subject matter of Example 17, and having an additional function of detecting an instrument by applying an instrument feature detection model.

[0112] (Example 19) Including the subject matter of any one of Examples 16 to 18, and having an additional function of detecting an instrument based on a sensor or control data indicating a part of an instrument extended in the field of view.

[0113] (Example 20) Including the subject matter of any one of Examples 16 to 19, and having an additional function of displaying a graphic representation of an instrument on a composite image.

[0114] (Example 21) Including the subject matter of Example 20, and having an additional function of performing a display indicating that the instrument has extended beyond a predetermined amount.

[0115] (Example 22) Including the subject matter of Example 21, and having a function that the display is a numerical value of the extended length.

[0116] (Example 23) It includes the subject matter of Example 21 and has the feature of using color separation based on the elongation length for display.

[0117] (Example 24) It includes the subject matter of any one of Examples 15 to 23 and has an additional function of identifying occluded parts by detecting bubbles using an image processing algorithm.

[0118] (Example 25) It includes the subject matter of any one of Examples 15 to 24 and has an additional function of identifying occluded parts by comparing pixel data with a baseline level and determining an occluded region based on pixels below the baseline level.

[0119] (Example 26) It includes the subject matter of any one of Examples 15 to 25 and has an additional function of a computing device for tracking the position and orientation of a first image and a second image.

[0120] (Example 27) It includes the subject matter of Example 26 and has an additional function of a computing device for deforming non-occluded parts of a first image when generating a composite image.

[0121] (Example 28) It includes the subject matter of Example 27 and has an additional function of deforming non-occluded parts by interpolation considering the differences in position and orientation between the first image and the second image.

[0122] (Example 29) A method for replacing occlusions in ultrasonic images in real time using multimodality images. This method can include the steps of receiving a previously acquired CT image, receiving a live ultrasonic image with occlusions, extracting non-occluded CT parts, and updating the live ultrasonic image by overlaying the non-occluded CT parts to generate an extended live image.

[0123] (Example 30) It includes the subject matter of Example 29 and has an additional function of deforming the non-occlusive CT portion so as to match the anatomical state of the live ultrasound image.

[0124] (Example 31) It includes the subject matter of Example 30 and has an additional function of deforming the CT portion by identifying anatomical landmarks in the CT and ultrasound images.

[0125] (Example 32) It includes the subject matter of any one of Examples 30 to 31 and has an additional function of deforming the CT portion based on tracking of the respiratory state.

[0126] (Example 33) It includes the subject matter of Example 32 and has an additional function of taking into account the difference in respiratory state between the CT image and the live ultrasound image when deforming the CT portion.

[0127] (Example 34) It includes the subject matter of Example 32 and has an additional function of selecting a CT image having a respiratory state that matches the respiratory state of the live ultrasound image.

[0128] (Example 35) It includes the subject matter of any one of Examples 30 to 34 and has an additional function of deforming the CT portion by interpolation considering the differences in position and orientation between the CT image and the ultrasound image.

[0129] (Example 36) It includes the subject matter of any one of Examples 29 to 35 and has an additional function that the previously acquired image is a CT image.

[0130] (Example 37) It includes the subject matter of any one of Examples 29 to 36 and has an additional function of generating an extended live ultrasound image by overlaying the non-occlusive CT portion.

Explanation of Signs

[0131] 100a Schematic diagram 100b Schematic diagram 102 Target area 104 Internal medical imaging system 106 Image sensor 108 Field of view 108a Proximal boundary 108b Distal boundary 110 Occlusion 112 Occluded portion 112a Proximal boundary 112b Distal boundary 114 Outlet port 120 Instrument 122 Occluded portion 200a Method 200b Image 202b First image 204b Second image 206b Occluded area 206b Occluded portion 210b Composite image 300a Method 300b Image 302b First image 304b Second image 306b Occluded area 306b Occluded portion 310b Composite image 400a Method 400b Image 402b First image 404b Second image 406b Instrument 408b Occluded portion 412b Composite image 414b Graphic representation 500a Schematic diagram 500b Schematic diagram 502 Field of view 502a Proximal boundary 502b Distal boundary 504 Internal medical imaging system 506 Image sensor 508 Non-occluded field of view 508 Non-occluded portion 510 Target nodule 520 Needle 522 Occluded part 522a Proximal boundary 522b Distal boundary 550 Schematic diagram 552 Field of view 552c Lower boundary 558 Non-occluded part 558a Proximal boundary of the field of view 560 Target nodule 570 Graphic representation 572 Non-occluded part 572a Proximal non-occluded boundary 572c Lower boundary 580 Schematic diagram 582 Composite image 582a Proximal boundary 582b Distal boundary 582c Boundary 588 Non-occluded part 600 Machine 602 Hardware processor 604 Main memory 606 Static memory 608 Interlink 610 Display unit[[ID=5³]] 612 Alphanumeric input device 614 User interface (UI) navigation device 616 Storage device 618 Signal generation device 620 Network interface device 621 Sensor 622 Machine-readable medium 624 Instruction [[ID=^{70}]]626 Communication network 628 Output controller It should be noted that in the original text, there is a possible error in the tag , which is likely a misspelling. It is translated as as it is in the original. If this is a real error, it may need to be corrected in the source material for a more accurate translation.

Claims

1. A method for real-time replacement of occluded portions of an ultrasonic image stream, comprising: receiving at least one first image of a patient's field of view via the ultrasonic image stream; receiving a plurality of second images of the field of view via the ultrasonic image stream, wherein the plurality of second images are received subsequent to the at least one first image; identifying occluded portions of the plurality of second images within the field of view, the step of identifying including detecting an instrument within the field of view; extracting at least one non-occluded portion of the at least one first image corresponding to the occluded portion; generating a plurality of composite images by replacing the occluded portions of the plurality of second images with the at least one non-occluded portion from the at least one first image; displaying the plurality of composite images on a display device substantially in real-time during the step of receiving the plurality of second images A method comprising the above steps.

2. The method according to claim 1, wherein the step of detecting the instrument within the field of view comprises identifying features of a known instrument within the field of view.

3. The method according to claim 2, wherein the step of detecting the instrument within the field of view comprises applying a machine learning model trained to detect known image features generated by the instrument.

4. The method according to claim 1, wherein the step of detecting the instrument within the field of view comprises receiving from the instrument sensor or control information indicating that a part of the instrument extends into the field of view.

5. The method according to claim 1, wherein the step of generating the plurality of composite images comprises generating a graphic representation of the instrument causing the occluded portion, the graphic representation dynamically changing to indicate the extended length of the instrument within the field of view.

6. The method according to claim 5, wherein the step of displaying the plurality of composite images comprises providing a display indicating that the instrument has been extended by a predetermined amount.

7. The method according to claim 6, wherein the display is a numerical display of the extended length of the instrument.

8. The method according to claim 6, wherein the display is a graphic display including color coding corresponding to a predetermined extended length of the instrument.

9. A method for real-time replacement of occluded portions of an ultrasonic image stream, comprising: receiving at least one first image of a patient's field of view via the ultrasonic image stream; receiving a plurality of second images of the field of view via the ultrasonic image stream, the plurality of second images being received subsequent to the at least one first image; identifying occluded portions of the plurality of second images, the step comprising detecting bubbles within the field of view; extracting at least one non-occluded portion of the at least one first image corresponding to the occluded portion; generating a plurality of composite images by replacing the occluded portions of the plurality of second images with the at least one non-occluded portion from the at least one first image; displaying the plurality of composite images on a display device in substantially real-time during the step of receiving the plurality of second images. A method comprising the above steps.

10. The method according to claim 9, wherein the step of detecting the bubbles within the field of view comprises applying an image processing algorithm trained to detect ultrasonic characteristics of the bubbles.

11. A method for real-time replacement of occluded portions of an ultrasonic image stream, comprising: receiving at least one first image of a patient's field of view via the ultrasonic image stream; receiving a plurality of second images of the field of view via the ultrasonic image stream, the plurality of second images being received subsequent to the at least one first image; identifying occluded portions of the plurality of second images, the step comprising comparing a set of pixel data of at least one second image from the plurality of second images with a baseline value and determining the occluded portion when a subset of the set of pixel data is below the baseline value; extracting at least one non-occluded portion of the at least one first image corresponding to the occluded portion; Generating a plurality of composite images by replacing the occluded portions of the plurality of second images with the at least one non-occluded portion from the at least one first image; Displaying the plurality of composite images on a display device in substantially real-time during the step of receiving the plurality of second images; A method comprising the steps of: **Claim 12** A method for real-time replacement of occluded portions of an ultrasonic image stream, the method comprising: Receiving at least one first image of a patient's field of view via the ultrasonic image stream; Receiving, via the ultrasonic image stream, a plurality of second images of the field of view, the plurality of second images being received subsequent to the at least one first image; Tracking the position and orientation of the at least one first image and the plurality of second images; Identifying occluded portions of the plurality of second images within the field of view; Extracting at least one non-occluded portion of the at least one first image corresponding to the occluded portion; Generating a plurality of composite images by replacing the occluded portions of the plurality of second images with the at least one non-occluded portion from the at least one first image; Displaying the plurality of composite images on a display device in substantially real-time during the step of receiving the plurality of second images; A method comprising the steps of: **Claim 13** The method of claim 12, wherein the step of generating the plurality of composite images comprises deforming a non-occluded portion of the at least one first image. **Claim 14** The method of claim 13, wherein the step of deforming the non-occluded portion of the at least one first image comprises interpolating in consideration of a difference in position and orientation between the plurality of second images and the at least one first image. **Claim 15** An image sensor configured to acquire an image of a patient's field of view and provide an ultrasonic image stream; A display device; A computing device including a processor and a memory device, the memory device including instructions that, when executed by the processor, cause the computing device to perform the method of any one of claims 1 to 14. An internal medical imaging system comprising

16. A method for real-time replacement of occluded portions of an ultrasonic image stream, comprising: Receiving at least one previously acquired image of a patient's field of view from a CT scan; Receiving a live image including at least a portion of the field of view via the ultrasonic image stream, the live image being received subsequent to the at least one previously acquired image; Identifying an occluded portion of the live image within the field of view; Extracting at least one non-occluded portion of the at least one previously acquired image corresponding to the occluded portion; Updating the live image by overlaying the occluded portion of the live image with the non-occluded portion from the at least one previously acquired image to generate an extended live image; Displaying the extended live image A method comprising

Images