Collateral detection for angiographic images

Abstract

The present disclosure provides for generating a 3D visualization of a blood vessel from intravascular ultrasound (IVUS) images. In particular, the present disclosure provides for reducing jitter between frames of IVUS recordings to provide a smoother appearance of a longitudinal view of a blood vessel from the IVUS image frames and constructing a 3D visualization of the blood vessel from the jitter-compensated IVUS image frames.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 588,546, filed on October 6, 2023, the disclosure of which is incorporated herein by reference. Technical Field

[0002] This disclosure relates to computed tomography (CT) coronary angiography and the co-registration of CT coronary angiography images with intravascular imaging modalities. Background Technology

[0003] CT coronary angiography (CTA or CCTA) is a procedure that uses CT angiography to assess the coronary arteries of the heart. Typically, the patient receives an intravenous injection of contrast agent, followed by a high-speed CT scan of the heart. CTA is often used in conjunction with other imaging modalities, such as intravascular ultrasound (IVUS) or intravascular optical CT (OCT). Physicians will use CTA along with IVUS or intravascular OCT images to assess the degree of coronary artery occlusion, commonly used to diagnose coronary artery disease.

[0004] To assist physicians in interpreting these images, they can be co-registered. For example, each image in a series of IVUS images can be mapped or co-located to the location of blood vessels represented in a CTA image. Therefore, it is necessary to identify reference points in each type of image and map these reference points to each other. Summary of the Invention

[0005] This disclosure provides methods for identifying collateral locations from angiographic images and extracting important information about the collaterals. For example, this disclosure can be implemented to identify collaterals, as well as their size (e.g., diameter) and orientation, from angiographic images.

[0006] It should be understood that collateral vessels serve as key reference points (or landmarks) in CT coronary angiography and are commonly used during co-registration between CT coronary angiography images and intravascular images (e.g., IVUS, etc.). This disclosure can be implemented as part of co-registration techniques to improve alignment between angiography images and a series of intravascular images.

[0007] In a further embodiment, this disclosure can be implemented to identify axial orientation and / or myocardial location in intravascular images (e.g., IVUS images, etc.), thereby enabling physicians to interpret intravascular images more accurately. In other embodiments, the identified collateral vessels and their characteristics (e.g., size, orientation, etc.) can be used to generate a three-dimensional (3D) model of the vessel, thereby facilitating comprehensive visualization and analysis of the vessel.

[0008] In some embodiments, this disclosure is implemented as a method for a cross-modal collateral matching system. The method may include: receiving image frames associated with a patient's blood vessels at a computing device; identifying the location and characteristics of one or more collaterals from the image frames by the computing device, in part based on one or more models of a plurality of machine learning (ML) models; and matching the one or more collaterals with one or more collaterals identified from a series of images by the computing device, wherein the image frames and the series of images are captured in different image modalities.

[0009] In a further embodiment of the method, the characteristic is the direction of the one or more side branches, the diameter of the one or more side branches, or both the direction and width of the one or more side branches.

[0010] In a further embodiment of the method, the orientation and diameter characteristics of the one or more collaterals are inputs to a cross-modal collateral matching process between an extravascular imaging modality and an intravascular imaging modality, wherein the extravascular imaging modality is X-ray angiography or computed tomography angiography, and wherein the intravascular imaging modality is intravascular ultrasound or intravascular optical coherence tomography.

[0011] In a further embodiment of the method, the location of the one or more collateral branches is the input to a cross-modal collateral matching process between an extravascular imaging modality and an intravascular imaging modality, wherein the extravascular imaging modality is X-ray angiography or computed tomography angiography, and wherein the intravascular imaging modality is intravascular ultrasound or intravascular optical coherence tomography.

[0012] In a further embodiment of the method, identifying the location and characteristics of the one or more collateral branches further includes: inferring a segmented version of the image frame using a first ML model among the plurality of ML models, wherein the segmented version of the image frame includes an indication of the blood vessel; inferring a straightened blood vessel from the blood vessel indicated in the segmented version of the image frame using a second ML model among the plurality of ML models; and identifying the one or more collateral branches from the straightened blood vessel.

[0013] In a further embodiment of the method, identifying the one or more collateral branches from the straightened blood vessel further includes: splitting the straightened blood vessel into a left component and a right component; generating a first drawing of connected pixels for the left component and a second drawing of connected pixels for the right component; and determining the location of the one or more collateral branches based on the first drawing and the second drawing.

[0014] In a further embodiment of the method, identifying the one or more collateral branches from the straightened blood vessel further includes determining the width of the one or more collateral branches based on the first drawing and the second drawing.

[0015] In a further embodiment of the method, identifying the one or more collateral branches from the straightened blood vessel further includes determining the orientation of the one or more collateral branches based on the first drawing and the second drawing.

[0016] In a further embodiment of the method, the direction is either the left-hand direction or the right-hand direction.

[0017] In a further embodiment of the method, identifying the one or more collateral branches from the straightened blood vessel further includes: tracing the skeleton of the straightened blood vessel by the computing device; extracting the centerline of the straightened blood vessel from the skeleton by the computing device; tracing the one or more collateral branches of the blood vessel by the computing device based on the skeleton and the centerline; and determining the location of the one or more collateral branches of the blood vessel based on the tracing of the one or more collateral branches.

[0018] In a further embodiment of the method, identifying the one or more collaterals from the straightened blood vessel further includes determining the width of the one or more collaterals based on tracking of the one or more collaterals.

[0019] In a further embodiment of the method, identifying the one or more collaterals from the straightened blood vessel further includes determining the orientation of the one or more collaterals based on tracking of the one or more collaterals.

[0020] In a further embodiment of the method, the direction is either the left-hand direction or the right-hand direction.

[0021] In some embodiments, this disclosure is implemented as a computer-readable storage device. The computer-readable storage device may include instructions executable by a processor of a computing device coupled to an intravascular imaging device and a fluorescein endoscope, wherein, when executed, the instructions cause the computing device to perform any of the methods disclosed herein.

[0022] In some embodiments, this disclosure is implemented as an apparatus. The apparatus may include a processor arranged to be coupled to an intravascular imaging device and a fluorescence endoscope, and the apparatus further includes a memory containing instructions, the processor being arranged to execute the instructions to implement any of the methods disclosed herein.

[0023] In some embodiments, this disclosure is implemented as an apparatus. The apparatus may include a processor and a memory storage device coupled to the processor, the memory storage device including instructions executable by the processor, the instructions, when executed, causing the apparatus to perform the following operations: receiving image frames associated with a patient's blood vessels; identifying the location and characteristics of one or more collateral branches from the image frames, in part based on one or more models from a plurality of machine learning (ML) models; and matching the one or more collateral branches with one or more collateral branches identified from a series of images, wherein the image frames and the series of images are captured in different image modalities.

[0024] In a further embodiment of the device, the characteristic is the direction of the one or more side branches, the diameter of the one or more side branches, or both the direction and width of the one or more side branches.

[0025] In a further embodiment of the device, the orientation and diameter characteristics of the one or more collateral branches are inputs to a cross-modal collateral matching process between an extravascular imaging modality and an intravascular imaging modality, wherein the extravascular imaging modality is X-ray angiography or computed tomography angiography, and wherein the intravascular imaging modality is intravascular ultrasound or intravascular optical coherence tomography.

[0026] In a further embodiment of the device, the location of the one or more collateral branches is an input to a cross-modal collateral matching process between an extravascular imaging modality and an intravascular imaging modality, wherein the extravascular imaging modality is X-ray angiography or computed tomography angiography, and wherein the intravascular imaging modality is intravascular ultrasound or intravascular optical coherence tomography.

[0027] In a further embodiment of the device, when the instructions are executed to identify the location and characteristics of the one or more collateral branches, the device further causes to perform the following operations: infer a segmented version of the image frame using a first ML model among the plurality of ML models, wherein the segmented version of the image frame includes an indication of the blood vessel; infer a straightened blood vessel from the blood vessel indicated in the segmented version of the image frame using a second ML model among the plurality of ML models; and identify the one or more collateral branches from the straightened blood vessel.

[0028] In a further embodiment of the device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the device further causes the device to perform the following operations: splitting the straightened blood vessel into a left component and a right component; generating a first drawing of connected pixels for the left component and a second drawing of connected pixels for the right component; and determining the location of the one or more collateral branches based on the first drawing and the second drawing.

[0029] In a further embodiment of the device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the device further causes the device to determine the width of the one or more collateral branches based on the first drawing and the second drawing.

[0030] In a further embodiment of the device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the device further causes the device to determine the orientation of the one or more collateral branches based on the first drawing and the second drawing.

[0031] In a further embodiment of the device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the device further causes the following operations: the computing device tracks the skeleton of the straightened blood vessel; the computing device extracts the centerline of the straightened blood vessel from the skeleton; the computing device tracks the one or more collateral branches of the blood vessel based on the skeleton and the centerline; and the computing device determines the location of the one or more collateral branches of the blood vessel based on the tracking of the one or more collateral branches.

[0032] In a further embodiment of the device, when the instruction is executed to identify the one or more collateral branches from the straightened blood vessel, the device further enables the device to determine the width of the one or more collateral branches based on the tracking of the one or more collateral branches.

[0033] In a further embodiment of the device, when the instruction is executed to identify the one or more collateral branches from the straightened blood vessel, the device further enables the device to determine the orientation of the one or more collateral branches based on the tracking of the one or more collateral branches.

[0034] In a further embodiment of the device, the direction is either the left or the right.

[0035] In some embodiments, this disclosure can be implemented as a computer-readable storage device. The computer-readable storage device may include instructions executable by a processor of a computing device coupled to an intravascular imaging device and a fluorescein endoscope, wherein, when executed, the instructions cause the computing device to perform the following operations: receive image frames associated with a patient's blood vessels; identify the location and characteristics of one or more collateral branches from the image frames, in part based on one or more models of a plurality of machine learning (ML) models; and match the one or more collateral branches with one or more collateral branches identified from a series of images, wherein the image frames and the series of images are captured in different image modalities.

[0036] In a further embodiment of the computer-readable storage device, the characteristic is the direction of the one or more side branches, the diameter of the one or more side branches, or both the direction and width of the one or more side branches.

[0037] In a further embodiment of the computer-readable storage device, when the instructions are executed to identify the location and characteristics of the one or more collateral branches, the computing device further causes to perform the following operations: infer a segmented version of the image frame using a first ML model of the plurality of ML models, wherein the segmented version of the image frame includes indications of the blood vessels; infer straightened blood vessels from the blood vessels indicated in the segmented version of the image frame using a second ML model of the plurality of ML models; and identify the one or more collateral branches from the straightened blood vessels.

[0038] In a further embodiment of the computer-readable storage device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the computing device further causes to perform the following operations: split the straightened blood vessel into a left component and a right component; generate a first drawing of connected pixels for the left component and a second drawing of connected pixels for the right component; and determine the location of the one or more collateral branches based on the first drawing and the second drawing.

[0039] In a further embodiment of the computer-readable storage device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the computing device further causes the computing device to determine the width of the one or more collateral branches based on the first drawing and the second drawing.

[0040] In a further embodiment of the computer-readable storage device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the computing device further causes the computing device to determine the orientation of the one or more collateral branches based on the first plot and the second plot.

[0041] In a further embodiment of the computer-readable storage device, when the instructions are executed to identify the one or more collateral branches from the straightened blood vessel, the computing device further causes the computing device to perform the following operations: tracing the skeleton of the straightened blood vessel by the computing device; extracting the centerline of the straightened blood vessel from the skeleton by the computing device; tracing the one or more collateral branches of the blood vessel based on the skeleton and the centerline by the computing device; and determining the location of the one or more collateral branches of the blood vessel based on the tracing of the one or more collateral branches. Attached Figure Description

[0042] To facilitate identification of any discussion of an element or action, one or more of the highest-order digits in the figure references refer to the figure number in which the element is first introduced.

[0043] Figure 1 A lateral branch detection system according to at least one embodiment is shown.

[0044] Figure 2 A routine for straightening blood vessels represented in an image frame, according to at least one embodiment, is shown.

[0045] Figure 3 Example image frames and inferred segmented image frames according to at least one embodiment are shown.

[0046] Figure 4 An example segmentation of an image frame and an inferred straightening blood vessel are shown according to at least one embodiment.

[0047] Figure 5 A routine for extracting information about collateral vessels of a blood vessel, according to at least one embodiment, is shown.

[0048] Figure 6A and Figure 6B Example images of blood vessels and collaterals according to at least one embodiment are shown.

[0049] Figure 7 A routine for extracting information about collateral vessels of a blood vessel, according to at least one embodiment, is shown.

[0050] Figure 8A , Figure 8B , Figure 8C , Figure 8D and Figure 8E Example images of blood vessels and collaterals according to at least one embodiment are shown.

[0051] Figure 9A and Figure 9B An exemplary artificial intelligence / machine learning (AI / ML) system suitable for use with at least one embodiment is shown.

[0052] Figure 10 A computer-readable storage medium according to at least one embodiment is shown.

[0053] Figure 11 An example imaging system according to at least one embodiment is shown.

[0054] Figure 12 A graphical representation of a machine in the form of a computer system is shown, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0055] As described above, this disclosure can be implemented to detect collaterals and their characteristics from angiographic images. In part, this disclosure provides for collateral detection using machine learning (ML) models. These ML models can be trained using angiographic images from labeled datasets, where the angiographic images are labeled at the pixel level. Furthermore, the angiographic images can be straightened along a region of interest (e.g., the vessel centerline, etc.) to enhance the representation of the vascular structure. Subsequently, post-processing techniques can be used to extract information (e.g., location index, size details, collateral direction, etc.).

[0056] This offers significant advantages over conventional co-registration workflows. For example, in current co-registration workflows, users need to manually adjust the position of collaterals on angiographic images to align them with collaterals detected in intravascular images. This disclosure can be implemented to automatically identify collaterals in angiographic images, enabling automatic adjustment, which further enhances ease of use and user experience. Furthermore, this disclosure can be implemented to achieve real-time co-registration, which is not possible in conventional co-registration workflows.

[0057] Figure 1 A collateral branch detection system 100 according to an embodiment of this disclosure is illustrated. Generally, the collateral branch detection system 100 is a system configured to identify collateral branches from extravascular (e.g., angiography, etc.) images of a blood vessel and to identify information about the identified collateral branches. Specifically, the collateral branch detection system 100 is configured to receive a CT angiography image 122 and identify a collateral branch 124 represented in the CT angiography image 122 and collateral features 126 of the collateral branch 124. Further, the collateral branch detection system 100 may be configured to identify the collateral branch 124 from a single angiography image (e.g., one of the CT angiography images 122) or a series of angiography images (e.g., movie playback, etc.). Additionally, as will be described herein, the collateral branch detection system 100 is configured to generate a straightened CT angiography image 128 from the CT angiography image 122.

[0058] Therefore, the collateral detection system 100 includes or can be coupled to the extravascular imaging system 102. The extravascular imaging system 102 can be any of various angiography imagers, examples of which are provided below. Figure 11 The combined internal and external imaging system 1100 depicted in the figure is described.

[0059] Further, the collateral detection system 100 includes a computing device 104. The computing device 104 can be any of a variety of computing devices. In some embodiments, the computing device 104 may be integrated into and / or implemented by the console of the extravascular imaging system 102. For some embodiments, the computing device 104 may be a tablet, laptop, workstation, or server communicatively connected to the extravascular imaging system 102. For other embodiments, the computing device 104 may be provided by a cloud-based computing device, such as a compute-as-a-service (CaaS) system accessible via a network (e.g., the Internet, intranet, wide area network, etc.). The computing device 104 may include a processor 110, memory 112, input and / or output (I / O) devices 114, and a network interface 118.

[0060] Processor 110 may include a circuit system or processor logic, such as any of a variety of commercial processors. In some examples, processor 110 may include multiple processors, a multi-threaded processor, a multi-core processor (whether the multiple cores coexist on the same die or on separate dies), and / or some other kind of multiprocessor architecture (in which multiple physically separate processors are linked in some way). Additionally, in some examples, processor 110 may include a graphics processing section and may include dedicated memory, multi-threaded processing, and / or some other parallel processing capability. In some examples, processor 110 may be an application-specific integrated circuit (ASIC) or a field-programmable integrated circuit (FPGA).

[0061] Memory 112 may include logic forming non-volatile memory, or a combination of non-volatile memory and volatile memory, for persistently storing data, a portion of which includes an array of integrated circuits. It should be understood that memory 112 may be based on any of a variety of technologies. In particular, the array of integrated circuits included in memory 112 may be arranged to form one or more types of memory, such as dynamic random access memory (DRAM), NAND memory, NOR memory, etc.

[0062] I / O device 114 can be any of a variety of devices for receiving input and / or providing output. For example, I / O device 114 may include a keyboard, mouse, joystick, foot pedal, haptic feedback device, LED, etc. Display 116 can be a conventional display or a touch-enabled display. Furthermore, display 116 can utilize various display technologies, such as liquid crystal display (LCD), light-emitting diode (LED), or organic light-emitting diode (OLED).

[0063] Network interface 118 may include logical and / or features for supporting communication interfaces. For example, network interface 118 may include one or more interfaces that operate according to various communication protocols or standards to communicate via direct communication links or network communication links. Direct communication may be performed via communication protocols or standards described in one or more industry standards, including their derivatives and variants. For example, network interface 118 may facilitate communication via buses such as PCIe, NVMe, USB, SMBus, SAS (e.g., Serial Attached Small Computer System Interface (SCSI)), SATA, etc. Additionally, network interface 118 may include logical and / or features for enabling communication via various wired or wireless network standards such as the 802.11 communication standard. For example, network interface 118 may be configured to support wired communication protocols or standards such as Ethernet. As another example, network interface 118 can be configured to support wireless communication protocols or standards, such as Wi-Fi, Bluetooth, ZigBee, LTE, 5G, etc.

[0064] Memory 112 may include instructions 120, CT angiography images 122, collateral branches 124, collateral features 126, straightened CT angiography images 128, machine learning (ML) models 130, segmented CT angiography images 136, and vascular centerlines 140. During operation, processor 110 may execute instructions 120 to cause computing device 104 to receive CT angiography images 122 from extravascular imaging system 102. Typically, CT angiography images 122 are CT images of the patient's heart or a portion of the patient's heart captured after contrast agent is injected into the patient's vascular system.

[0065] The processor 110 may further execute instructions 120 to cause the computing device 104 to generate a straightened CT angiography image 128 from the ML model 130 and the CT angiography image 122. In other words, the processor 110 may execute instructions 120 to infer the straightened CT angiography image 128 from the CT angiography image 122 using the ML model 130. As noted, the CT angiography image 122 may be a single image or multiple images in a series of images (e.g., movie playback, etc.). Thus, the straightened CT angiography image 128 will correspondingly be a single image or a series of images.

[0066] ML model 130 may include segmentation model 132 and straightening model 134. Segmentation model 132 may be configured to distinguish components of CT angiography image 122 corresponding to the main vessel from components of CT angiography image 122 corresponding to surrounding tissue and background. For some examples, segmentation model 132 may be configured to infer segmented CT angiography image 136 including indications of main vessel components.

[0067] The straightening model 134 can be configured to straighten the main vessel components represented in the segmented CT angiography image 136. In other words, the straightening model 134 can be configured to infer the straightened CT angiography image 128 based on the segmented CT angiography image 136 and the indication of the vessel centerline (e.g., vessel centerline 140). It should be noted that the identification of the vessel centerline 140 can be determined using various algorithms. However, the specific details of such algorithms are not the subject of this disclosure.

[0068] Processor 110 can further execute instructions 120 to identify collateral branches 124 and collateral features 126 from the straightened CT angiography image 128. This is described in more detail below. However, as a general overview, processor 110 can execute instructions 120 to split the straightened CT angiography image 128 into a left and right portion, represented as a split CT angiography image 138. Based on the split CT angiography image 138, motion-focusing analysis can be performed independently on each side of the vessel. That is, processor 110 can execute instructions 120 to extract the collateral branch location index, size, and orientation.

[0069] Figure 2 , Figure 5 and Figure 7Routines 200, 500, and 700, according to some embodiments of this disclosure, are illustrated. Routines 200, 500, and 700 can be implemented by collateral detection system 100 or another computing device, as outlined in the text, to identify collaterals of a vessel represented in an angiographic image (or a series of images) and information about the identified collaterals. Routine 200 can be implemented to generate a straightened vessel from several CT angiographic images of the vessel. Routines 500 and 700 can be implemented to extract the location and key information (e.g., width, orientation, etc.) of the collaterals from the straightened vessel. It should be noted that routines 200, 500, and 700 are described with reference to a single angiographic image. However, routines 200, 500, and 700 can be iteratively repeated on multiple angiographic images. As another example, each block or step in routines 200, 500, and 700 can be performed on multiple angiographic images.

[0070] Routine 200 may begin at block 202, “Receiving image frames associated with a patient’s vessels from an extravascular imaging device at a computing device,” where angiography image frames may be received at a computing device. For example, computing device 104 of collateral detection system 100 may receive image frames of CT angiography image 122. In some embodiments, frames of CT angiography image 122 may be received from extravascular imaging system 102, while in other embodiments, frames of CT angiography image 122 may be pre-captured by extravascular imaging system 102 and stored in memory (e.g., memory 112, a memory location accessible via network interface 118). In this example, computing device 104 may access frames of CT angiography image 122 from the memory location. In some embodiments, processor 110 may execute instructions 120 to receive from a user one or more frames of CT angiography image 122 to be accessed at block 202.

[0071] Continuing to box 204, “Inferring Segmented Versions of Image Frames Using an ML Model by a Computing Device,” an ML model can be used to infer segmented versions of image frames. For example, processor 110 can execute instruction 120 to infer segmented frames of CT angiography image 136 from frames of CT angiography image 122 received at box 202 using segmentation model 132. This is described in more detail below. Figure 3 Examples of frames from CT angiography image 122 and associated segmented frames of CT angiography image 136 are given (which can be generated as outlined herein).

[0072] Continuing to box 206, “Inferring Straightened Vessels from Segmented Versions of Image Frames Using an ML Model by a Computing Device,” an ML model can be used to infer a straightened representation of the vascular system in a segmented frame. For example, processor 110 can execute instruction 120 to infer a frame of straightened CT angiography image 128 from a frame of segmented CT angiography image 136 inferred at box 204 using straightening model 134. In some embodiments, processor 110 can execute instruction 120 to infer a frame of straightened CT angiography image 128 from segmented CT angiography image 136 using straightening model 134 and vessel centerline 140. This is described in more detail below. Figure 4 Examples of frames from a straightened CT angiography image 128 and frames of an associated segmented CT angiography image 136 are given, along with indications of the centerline (e.g., from the vascular centerline 140), which can be generated as outlined herein.

[0073] Figure 3 An example of a CT angiography frame 302 is depicted, which can be inferred using a segmented CT angiography frame 304 via segmentation model 132 as outlined herein. In some embodiments, the CT angiography frame 302 and the segmented CT angiography frame 304 may be frames of CT angiography image 122 and a straightened CT angiography image 128, respectively. As depicted, vascular structures or vascular systems 306 are depicted in both the CT angiography frame 302 and the segmented CT angiography frame 304.

[0074] Figure 4 An example of a straightened CT angiography frame 402 is depicted, which can be inferred via a straightening model 134 using segmented CT angiography frames 304 and a centerline 404, as outlined herein. In some embodiments, the segmented CT angiography frame 304, the straightened CT angiography frame 402, and the centerline 404 can be frames of segmented CT angiography image 136, straightened CT angiography image 128, and vascular centerline 140, respectively. As depicted, the vascular system 306 represented in the segmented CT angiography frame 304 is straightened in the straightened CT angiography frame 402, thereby obtaining a straightened vessel 406.

[0075] As pointed out, Figure 5Routine 500 is shown, which can be implemented to extract the location and key information (e.g., width, orientation, etc.) of collaterals from a straightened blood vessel. Routine 500 may begin at box 502. At box 502, “Splitting a straightened blood vessel into left and right components by a computing device,” the straightened blood vessel can be split into left and right components. For example, processor 110 may execute instruction 120 to generate a split CT angiography image 138, which includes indications of the left and right components of the blood vessel in the straightened CT angiography image 128. As a particular example, processor 110 may execute instruction 120 to divide or cut a straightened blood vessel (e.g., straightened blood vessel 406, etc.) into left and right components. Figure 6A An example of this situation is depicted in the text.

[0076] Figure 6A An example is depicted of a straightened CT angiography frame 402 and a straightened blood vessel 406 split into a left component 602 and a right component 604. In some embodiments, the processor 110 may execute instructions 120 to divide the straightened blood vessel 406 downward along a centerline to form the left component 602 and the right component 604.

[0077] Return to Figure 5 And routine 500, which can continue from block 502 to block 504. At block 504, “Generate a drawing of connected pixels for each of the left and right components,” a drawing of connected pixels in a straightened segmented image can be generated for both the left and right components of the straightened blood vessel. For example, processor 110 can execute instruction 120 to generate a drawing of connected pixels in each of the left component 602 and right component 604 of the straightened blood vessel 406 from the straightened CT angiography frame 402.

[0078] In some examples, processor 110 may execute instruction 120 to calculate the number of connected pixels for each separated component (e.g., left component 602 and right component 604, etc.) from the top of the cutting line down to the bottom. This will produce two distinct drawings, representing the main vessel outlines on the left and right sides of the vessel, respectively.

[0079] Continuing to box 506, “Determining the Location, Width, and Orientation of Collateral Branches Based on the Drawing,” the location, width, and orientation of collateral branches can be determined. For example, processor 110 can execute instruction 120 to identify the location of a collateral branch and its key features (e.g., width, orientation, etc.). Processor 110 can execute instruction 120 to identify an increase in the number of connected pixels, which can indicate the presence of a collateral branch along the vessel at that location. Visually, this increase can be represented as a peak in the drawing. However, processor 110 can execute instruction 120 to identify whether the increase in the number of connected pixels exceeds a baseline number threshold level. Further, processor 110 can execute instruction 120 to identify the location of collateral branches along the vessel based on these peaks.

[0080] Furthermore, the processor 110 can execute instruction 120 to identify the width or diameter of the side branch based on the peak width. Additionally, since the drawing is divided into left and right components, the direction of each branch can be identified.

[0081] Figure 6B Examples of the identified side branches 606a, 606b, 606c, 606d, and 606e are depicted, along with indications of their width and orientation. For example, side branches 606a, 606b, and 606d are indicated to have a left-hand orientation, while side branches 606c and 606e are indicated to have a right-hand orientation. Furthermore, the width of each side branch is indicated.

[0082] Figure 7 Routine 700 is shown, which can be implemented to extract the location and key information (e.g., width, orientation, etc.) of collateral vessels from a straightened blood vessel. Routine 700 may begin at box 702. At box 702, “Tracing the skeleton of the straightened blood vessel by a computing device,” the skeleton of the straightened blood vessel is traced by a computing device. For example, processor 110 may execute instruction 120 to trace the blood vessel represented in the straightened CT angiography image 128. Figure 8A and Figure 8B An example of this situation is depicted in the text.

[0083] Figure 8A Examples of straightened CT angiography frames 402 and 406 are shown, while Figure 8B The skeleton 802 of the straightened CT angiography frame 402 and the straightened blood vessel 406 is shown, which can be tracked by the processor 110 executing instructions 120.

[0084] Return to Figure 7 And routine 700, which can continue from block 702 to block 704. At block 704, “Extracting the centerline from the skeleton by a computing device”, the centerline of the blood vessels can be extracted from the skeleton. For example, processor 110 can execute instruction 120 to extract the centerline from skeleton 802. Figure 8C An example of this situation is depicted in the text.

[0085] Figure 8C An example is shown of a straightened CT angiography frame 402 and a centerline 804 extracted from the skeleton 802. Return to Figure 7 And routine 700, which can continue from block 704 to block 706. At block 706, “Tracing collaterals based on the skeleton and centerline by a computing device,” collaterals of the vessel can be traced based on the skeleton and centerline. For example, processor 110 can execute instruction 120 to trace the collaterals of the straightened vessel 406 based on the skeleton 802 and centerline 804. Figure 8D An example of this situation is depicted in the text.

[0086] Figure 8D This image shows a straightened CT angiography frame 402 and examples of collateral branches of the straightened vessel 406 traced based on the skeleton 802 and the centerline 804. Collateral branches 806a, 806b, 806c, 806d, and 806e are depicted as being traced on the straightened vessel 406. Return to Figure 7 And routine 700, which can continue from block 706 to block 708. At block 708, “The location, width, and orientation of the side branch are determined by the computing device based on the tracked side branch,” the location, width, and orientation of the side branch can be determined based on the tracked side branch. For example, processor 110 can execute instruction 120 to extract information (e.g., location, width, orientation, etc.) about the side branch (e.g., side branch 806a, etc.) from the tracked side branch. Figure 8E An example of this situation is depicted in the text.

[0087] Figure 8E Examples of the identified side branches 806a, 806b, 806c, 806d, and 806e are depicted, along with indications of their width and orientation. For example, side branches 806a, 806b, 806c, and 806d are indicated to have a right-hand orientation, while side branch 806e is indicated to have a left-hand orientation. Furthermore, the width of each side branch is indicated.

[0088] As noted, in some embodiments, ML models can be used to infer segmented images and straightened blood vessels from segmented images. For example, the processor 110 of computing device 104 can execute instructions 120 to infer a segmented CT angiography image 136 from a CT angiography image 122 using an ML model 130 (e.g., segmentation model 132, etc.), and to infer a straightened CT angiography image 128 from the segmented CT angiography image 136 using an ML model 130 (e.g., straightening model 134, etc.). In this example, the ML model (e.g., ML model 130) can be stored in the memory 112 of computing device 104. However, it should be understood that the ML model needs to be trained before deployment. Figure 9A An ML training environment 900a is shown that can be used to train an ML model, which can later be used to generate (or infer) segmented CT angiography images 136 as described herein. The ML training environment 900a may include an ML system 902, such as a computing device that applies ML algorithms to learn relationships. In this example, the ML algorithm can learn the relationship between a set of inputs (e.g., CT angiography image 122) and an output (e.g., segmented CT angiography image 136).

[0089] The ML system 902 can utilize experimental data 904 collected during several prior procedures. The experimental data 904 may include CT angiography images 122 of several patients. The experimental data 904 may be located in the same location as the ML system 902 (e.g., stored in the storage device 912 of the ML system 902), may be remote from the ML system 902 and accessed via a network interface 918, or may be a combination of local and remote data.

[0090] Experimental data 904 can be used to form training data 906, which includes CT angiography images 122 and corresponding pixel-level segments. These pixel-level segments can be formed based on manual annotation and stored as CT angiography images 908 with the expected segmentation.

[0091] As described above, the ML system 902 may include a storage device 912, which may include a hard disk drive, a solid-state storage device, and / or random access memory. The storage device 912 may store training data 906. Typically, the training data 906 may include information elements or data structures, including indications of the CT angiography image 122 and the associated expected segmentation of the CT angiography image 908. The training data 906 can be used to train the ML model 914a. Depending on the application, different types of models may be used to form the basis of the ML model 914a. For example, in this example, an artificial neural network (ANN) may be particularly well-suited for learning the association between the CT angiography image (CT angiography image 122) and a segmented version of the CT angiography image (e.g., a segmented CT angiography image 136). Convolutional neural networks may also be well-suited for this task. Any suitable training algorithm 916 can be used to train the ML model 914a. Nevertheless, Figure 9A The examples depicted may be particularly well-suited for supervised training or reinforcement learning training algorithms. For supervised training, the ML system 902 can apply CT angiography images 122 as model input 920, expecting segmented CT angiography images 908 to be mapped to these model inputs to learn the association between CT angiography images 122 and segmented CT angiography images 136. In a reinforcement learning scenario, the training algorithm 916 can attempt to maximize some or all (or a weighted combination) of the mappings between model inputs 920 and segmented CT angiography images 136, thereby producing an ML model 914a with minimal error. In some embodiments, training data 906 may be split into “training” data and “test” data, wherein a subset of training data 906 may be used to adjust ML model 914a (e.g., internal weights of the model), while another non-overlapping subset of training data 906 may be used to measure the accuracy of ML model 914a inferring (or generalizing) segmented CT angiography images 136 from “unseen” training data 906 (e.g., training data 906 not used to train ML model 914a).

[0092] The ML model 914a can be applied using processor circuitry 910, which may include suitable hardware processing resources for operating on the logic and structure in storage device 912. The development of training algorithm 916 and / or trained ML model 914a may depend at least in part on hyperparameters 922. In an exemplary embodiment, model hyperparameters 922 may be automatically selected based on hyperparameter optimization logic 924, which may include any known hyperparameter optimization techniques suitable for the selected ML model 914a and the training algorithm 916 to be used. In an alternative embodiment, the ML model 914a may be retrained over time to incorporate new knowledge and / or updated experimental data 904.

[0093] Once the ML model 914a is trained, it can be applied (e.g., by processor circuitry 910, processor 110, etc.) to new input data (e.g., CT angiography images 122 captured before, after, etc., PCI intervention). This input to the ML model 914a can be formatted according to a predefined model input 920 to reflect how the training data 906 is provided to the ML model 914a. The trained ML model 914a can generate segmented CT angiography images 136 from the CT angiography images 122. In this example, the ML model 914a can be deployed as a segmentation model 132.

[0094] The above description relates to a specific type of ML system 902 that applies supervised learning techniques given available training data with input / outcome pairs. However, the invention is not limited to use with a particular ML paradigm, and other types of ML techniques can be used. For example, in some embodiments, ML system 902 may apply, for example, evolutionary algorithms or other types of ML algorithms and models to generate segmented CT angiography images 136 from CT angiography images 122.

[0095] The ML system 902 can be further used to train models to infer straightened blood vessels from segmented representations of the vascular system. Figure 9BAn ML training environment 900b, an example of ML training environment 900a, is shown, configured to train an ML model 914b to infer a straightened CT angiography image 128 from a segmented CT angiography image 136. Therefore, training data 906 may include the segmented CT angiography image 136 and the expected straightened blood vessels 926, and the ML model 914b can be "trained" as described above to infer the straightened CT angiography image 128 from the segmented CT angiography image 136. The trained model ML model 914b can generate the straightened CT angiography image 128 from the segmented CT angiography image 136. In this example, the ML model 914b can be deployed as a straightening model 134.

[0096] Figure 10 A computer-readable storage medium 1000 is illustrated. The computer-readable storage medium 1000 may include any non-transitory computer-readable or machine-readable storage medium, such as optical, magnetic, or semiconductor storage media. In various embodiments, the computer-readable storage medium 1000 may include an article of manufacture. In some embodiments, the computer-readable storage medium 1000 may store computer-executable instructions 1002 that a circuit system (e.g., processor 110, etc.) can utilize to perform. For example, the computer-executable instructions 1002 may include instructions for implementing operations described with respect to the side support detection system 100, which may improve the functionality of the side support detection system 100 as detailed herein. For example, the computer-executable instructions 1002 may include instructions that may cause a computing device to implement… Figure 2 Routine 200 Figure 5 Routine 500 Figure 9A and Figure 9B The instructions for the training algorithm 916. As another example, the computer-executable instructions 1002 may include instructions 120, a segmentation model 132, and / or a straightening model 134. Examples of computer-readable storage media 1000 or machine-readable storage media may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer-executable instructions 1002 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc.

[0097] Figure 11A combined internal and external imaging system 1100 is illustrated, comprising both an intravascular imaging system 1102 (e.g., an IVUS imaging system, etc.) and an extravascular imaging system 1104 (e.g., an angiography imaging system). The combined internal and external imaging system 1100 further includes a computing device 1106, which includes a circuit system, a controller and / or processor, and memory and software (as needed). In some embodiments, a collateral detection system 100 may be incorporated into the computing device 1106. In other embodiments, the computing device 1106 may be configured to capture images (e.g., CT angiography images 122, etc.) for use in a collateral detection processor as described herein. It should be understood that the systems and methods described herein do not require intravascular imaging, and the combined imaging system is described for the sake of clarity. For example, the image recognition techniques described herein for identifying vascular collaterals on extravascular images can be used to co-register one or more extravascular images with a series of intravascular or intravascular images. Typically, the intravascular imaging system 1102 can be configured to generate intravascular imaging data (e.g., IVUS images, etc.), while the extravascular imaging system 1104 can be configured to generate extravascular imaging data (e.g., angiographic images, etc.).

[0098] The extravascular imaging system 1104 may include a stage 1108, which may be arranged to provide sufficient space for positioning the C-arm 1110 of the angiography / fluorescence fluoroscopy unit in an operating position relative to the patient 1112 on the drive unit. The C-arm 1110 may be configured to acquire fluorescence fluoroscopy images in the absence of contrast agent in the blood vessels of the patient 1112 and / or to acquire angiography images in the presence of contrast agent in the blood vessels of the patient 1112.

[0099] Raw radiographic image data acquired by the C-arm 1110 can be transmitted via a transmission cable 1116 to an extravascular data input port 1114. The input port 1114 can be a separate component, integrated into or part of a computing device 1106. The input port 1114 may include a processor that converts the received raw radiographic image data into extravascular image data (e.g., angiography / fluorescence imaging data), for example, in the form of real-time video, DICOM, or a series of individual images. The extravascular image data can be initially stored in memory within the input port 1114 or in memory of the computing device 1106. If the input port 1114 is a separate component from the computing device 1106, the extravascular image data can be transmitted via the transmission cable 1116 to the computing device 1106 and to its input port (not shown). In some alternatives, communication between the devices or processors can be performed wirelessly instead of via cables as depicted.

[0100] Intravascular imaging data can be, for example, IVUS data or OCT data acquired by the intravascular imaging system 1102. The intravascular imaging system 1102 may include an intravascular imaging device, such as an imaging catheter 1120. The imaging catheter 1120 is configured to be inserted into the patient 1112 such that its distal end (including diagnostic components or probe 1122 (e.g., an IVUS probe)) is near the desired imaging location of the vessel. Radiopaque material or markings 1124 located on or near the probe 1122 can provide an indication of the current position of the probe 1122 in the radiographic image. In some embodiments, the imaging catheter 1120 and / or probe 1122 may include a guiding catheter (not shown) that has been inserted into the lumen (e.g., a vessel, such as a coronary artery) of the subject via a guidewire (also not shown). However, in some embodiments, the imaging catheter 1120 and / or probe 1122 may be inserted into the vessel of the patient 1112 without a guidewire.

[0101] In some embodiments, the imaging catheter 1120 and / or probe 1122 may include both imaging capabilities and other data acquisition capabilities. For example, FFR and / or iFR data, data related to pressure, flow, temperature, electrical activity, oxygenation, biochemical composition, or any combination thereof. In some embodiments, the imaging catheter 1120 and / or probe 1122 may further include therapeutic devices such as stents, balloons (e.g., angioplasty balloons), grafts, filters, valves, and / or different types of therapeutic endovascular devices.

[0102] Imaging catheter 1120 is coupled to proximal connector 1126 to connect imaging catheter 1120 to image acquisition device 1128. Image acquisition device 1128 can be coupled to computing device 1106 via transmission cable 1116 or wireless connection. Intravascular image data can be initially stored in memory within image acquisition device 1128 or in memory of computing device 1106. If image acquisition device 1128 is a separate component from computing device 1106, intravascular image data can be transmitted to computing device 1106 via, for example, transmission cable 1116.

[0103] The computing device 1106 may also include one or more additional output ports for transmitting data to other devices. For example, the computer may include output ports for transmitting data to a data archive or memory device 1132. The computing device 1106 may also include a user interface (described in more detail below) that includes a combination of circuitry, processing components, and instructions executable by the processing components and / or circuitry to perform the image recognition and vessel orientation or path finding described herein and / or dynamic co-registration of intravascular and extravascular images using identified vascular pathways.

[0104] In some embodiments, the computing device 1106 may include a user interface device, such as a keyboard, mouse, joystick, touch screen device (such as a smartphone or tablet computer), touchpad, trackball, voice command interface, and / or other types of user interfaces known in the art.

[0105] A user interface can be rendered and displayed on a monitor 1134 connected to computing device 1106 via display cable 1136. Although monitor 1134 is depicted as separate from computing device 1106, in some examples, monitor 1134 may be part of computing device 1106. Alternatively, monitor 1134 may be remote from computing device 1106 and wireless. As another example, monitor 1134 may be part of another computing device (such as a tablet computer) different from computing device 1106, which may be connected to computing device 1106 via a wired or wireless connection. For some applications, monitor 1134 includes a head-up display and / or a head-mounted display. For some applications, computing device 1106 generates output on different types of visual, text, graphics, haptic, audio, and / or video output devices (e.g., speakers, headphones, smartphones, or tablet computers). For some applications, the user interface rendered on monitor 1134 acts as both an input and output device.

[0106] Figure 12A schematic representation of a machine 1200 in the form of a computer system is shown, within which an instruction set can be executed to cause the machine to perform any or more of the methods discussed herein. More specifically, Figure 12 A schematic representation of a machine 1200 in the form of an example computer system is shown, within which instructions 1208 (e.g., software, program, application, app, or other executable code) can be executed to cause the machine 1200 to perform any or more of the methods discussed herein. For example, instructions 1208 can cause the machine 1200 to execute instructions 120, Figure 2 Routine 200 Figure 5 Routine 500 Figure 9A or Figure 9B Training algorithms such as 916, etc. More generally, instruction 1208 enables machine 1200 to identify collateral vessels from one or more CT angiography images of a vessel, as described herein.

[0107] Instruction 1208 transforms a general, unprogrammed machine 1200 into a specific machine 1200 programmed to perform the described and demonstrated functions in a particular manner. In alternative embodiments, machine 1200 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, machine 1200 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1200 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing instructions 1208 specifying actions to be taken by machine 1200. Furthermore, although only a single machine 1200 is shown, the term "machine" should also be considered as a collection of machines 200 that individually or jointly execute instructions 1208 to perform any or more of the methods discussed herein.

[0108] Machine 1200 may include processor 1202, memory 1204, and I / O components 1242, which may be configured to communicate with each other, for example, via bus 1244. In example embodiments, processor 1202 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1206 and processor 1210 capable of executing instructions 1208. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 12 Multiple processors 1202 are shown, but machine 1200 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0109] Memory 1204 may include main memory 1212, static memory 1214, and memory cell 1216, all of which are accessible to processor 1202, for example, via bus 1244. Main memory 1204, static memory 1214, and memory cell 1216 store instructions 1208 embodying any one or more of the methods or functions described herein. Instructions 1208 may also reside wholly or partially in main memory 1212, in static memory 1214, in machine-readable medium 1218 in memory cell 1216, in at least one of the processors 1202 (e.g., in the processor's cache memory), or any suitable combination thereof, during execution by machine 1200.

[0110] I / O component 1242 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement values, etc. The specific I / O component 1242 included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, while headless server machines may not include such touch input devices. It should be understood that I / O component 1242 may include... Figure 12Many other components are not shown. I / O components 1242 are grouped only by function to simplify the following discussion, and this grouping is by no means limiting. In various example embodiments, I / O components 1242 may include output components 1228 and input components 1230. Output components 1228 may include visual components (e.g., displays, such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 1230 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0111] In a further example embodiment, I / O component 1242 may include biometric identification component 1232, motion component 1234, environmental component 1236 or positioning component 1238, and a variety of other components. For example, biometric identification component 1232 may include components for detecting representations (e.g., hand representations, facial representations, voice representations, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1234 may include accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1236 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting hazardous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1238 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure to derive altitude), a direction sensor component (e.g., a magnetometer), etc.

[0112] A wide variety of technologies can be used to implement communication. I / O component 1242 may include communication components 1240 operable to connect machine 1200 to network 1220 or device 1222 via connection 1224 and connection 1226, respectively. For example, communication component 1240 may include a network interface component or another suitable device for interface connection with network 1220. In further examples, communication component 1240 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth components, etc. ® Components (e.g., Bluetooth) ® Low power consumption, Wi-Fi ® Components, and other communication components for providing communication via other modes. Device 1222 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device connected via USB).

[0113] Furthermore, the communication component 1240 can detect identifiers or include components operable for detecting identifiers. For example, the communication component 1240 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes (such as Universal Product Code (UPC) barcodes), multi-dimensional barcodes (such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCCRSS-2D barcodes), and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). Additionally, various information can be derived from the communication component 1240, such as location derived via Internet Protocol (IP) geolocation, location derived via Wi-Fi® signal triangulation, or location derived by detecting NFC beacon signals that indicate a specific location.

[0114] Various memories (i.e., memory 1204, main memory 1212, static memory 1214, and / or the memory of processor 1202) and / or storage units 1216 may store one or more sets of instructions and data structures (e.g., software) embodying any one or more methods or functions described herein or utilized by any one or more methods or functions. These instructions (e.g., instruction 1208) cause various operations to perform the disclosed embodiments when executed by processor 1202.

[0115] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” have the same meaning and are used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should be considered to include, but are not limited to, solid-state memory, and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, such as: semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which fall within the scope of the term “signal medium” discussed below.

[0116] In various example embodiments, one or more portions of network 1220 may be an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, the Internet, a portion of the Internet, a portion of the PSTN, a common legacy telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 1220 or a portion of network 1220 may include a wireless or cellular network, and connection 1224 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless connection. In this example, connection 1224 can implement any of the various types of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, GSM evolution enhanced data rate (EDGE) technology, the 3rd generation partnership program (3GPP) including 3G, fourth-generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), global microwave access interoperability (WiMAX), long-term evolution (LTE) standard, other standards defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0117] Instruction 1208 may be transmitted or received via network 1220 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1240) and utilizing any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 1208 may be transmitted or received via a transmission medium via a connection 1226 (e.g., a peer-to-peer connection) to device 1222. The terms “transmission medium” and “signal medium” have the same meaning and are used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be considered to include any intangible medium that can store, encode, or carry instruction 1208 for execution by machine 1200, and include digital or analog communication signals or other intangible media that facilitate communication of such software. Therefore, the terms “transmission medium” and “signal medium” should be considered to include any form of modulated data signal, carrier wave, etc. The term “modulated data signal” means a signal in which one or more characteristics of the signal are set or altered in such a way as to encode information in the signal.

[0118] The terms used in this document shall conform to their ordinary meaning in the relevant field or the meaning indicated by their use in the context, unless a specific definition is provided.

[0119] In this document, references to “one embodiment” or “an embodiment” do not necessarily refer to the same embodiment, but they may refer to the same embodiment. Unless the context explicitly requires otherwise, throughout the specification and claims, the words “comprise,” “comprising,” etc., should be interpreted in an open-ended rather than exclusive or exhaustive sense; that is, in the sense of “including but not limited to.” Use of singular or plural terms also includes both the plural and singular, unless explicitly limited to one or more. Furthermore, when used in this application, the words “this article,” “above,” “below,” and similar terms refer to the entire application and not any part thereof. When a claim uses the word “or” when referring to a list consisting of two or more items, the word covers all of the following interpretations: any item in the list, all items in the list, and any combination of items in the list, unless explicitly limited to one or the other. Any terms not explicitly defined herein shall be used in their conventional meaning as commonly understood by one of ordinary skill in the art.

Claims

1. A method for a cross-modal support matching system, the method comprising: Receive image frames associated with the patient's blood vessels at the computing device; The computing device identifies the location and characteristics of one or more lateral branches from the image frame, partly based on one or more of a plurality of machine learning (ML) models; as well as The computing device matches the one or more side branches with one or more side branches identified from a series of images, wherein the image frames and the series of images are captured in different image modalities.

2. The method as described in claim 1, wherein, The characteristic is the direction of the one or more side branches, the diameter of the one or more side branches, or both the direction and width of the one or more side branches.

3. The method as described in claim 2, wherein, The orientation and diameter characteristics of the one or more collateral branches are the inputs to the cross-modal collateral matching process between the extravascular imaging modality and the intravascular imaging modality. The extravascular imaging modality is X-ray angiography or computed tomography angiography, and The intravascular imaging modality is intravascular ultrasound or intravascular optical coherence tomography.

4. The method of claim 3, wherein, The location of the one or more collateral branches is the input to the cross-modal collateral matching process between the extravascular imaging modality and the intravascular imaging modality. The extravascular imaging modality is X-ray angiography or computed tomography angiography, and The intravascular imaging modality is intravascular ultrasound or intravascular optical coherence tomography.

5. The method of claim 1, wherein, Identifying the location and characteristics of the one or more lateral branches further includes: A segmentation version of the image frame is inferred using a first ML model among the plurality of ML models, wherein the segmentation version of the image frame includes an indication of the blood vessels; The straightened blood vessels are inferred from the segmented version of the image frame using a second ML model among the plurality of ML models; and Identify the one or more collateral branches from the straightened blood vessel.

6. The method of claim 5, wherein, Identifying the one or more collateral branches from the straightened blood vessel further includes: The straightened blood vessel is divided into a left component and a right component; A first drawing of connected pixels is generated for the left component, and a second drawing of connected pixels is generated for the right component; and The location of the one or more lateral branches is determined based on the first and second drawings.

7. The method of claim 6, wherein, Identifying the one or more collateral branches from the straightened blood vessel further includes determining the width of the one or more collateral branches based on the first drawing and the second drawing.

8. The method of claim 6 or claim 7, wherein, Identifying the one or more collateral branches from the straightened blood vessel further includes determining the orientation of the one or more collateral branches based on the first drawing and the second drawing.

9. The method of claim 8, wherein, The direction is either the left or the right.

10. The method of claim 5, wherein, Identifying the one or more collateral branches from the straightened blood vessel further includes: The computing device tracks the skeleton of the straightened blood vessel; The computing device extracts the centerline of the straightened blood vessel from the skeleton; The computing device tracks the one or more collateral branches of the blood vessel based on the skeleton and the centerline; and The location of the one or more collateral branches of the blood vessel is determined by tracing the one or more collateral branches.

11. The method of claim 10, wherein, Identifying the one or more collateral branches from the straightened blood vessel further includes determining the width of the one or more collateral branches based on tracking of the one or more collateral branches.

12. The method of claim 10 or claim 11, wherein, Identifying the one or more collateral branches from the straightened blood vessel further includes determining the orientation of the one or more collateral branches based on tracing them.

13. The method of claim 11, wherein, The direction is either the left or the right.

14. A computer-readable storage device comprising instructions executable by a processor of a computing device coupled to an intravascular imaging apparatus and a fluorescence endoscope, wherein, When executed, the instructions cause the computing device to perform the method as described in any one of claims 1 to 13.

15. An apparatus comprising a processor arranged to be coupled to an intravascular imaging device and a fluorescence endoscope, the apparatus further comprising a memory including instructions, the processor being arranged to execute the instructions to implement the method as claimed in any one of claims 1 to 13.

Images