Live co-registration of extravascular and intravascular imaging

Abstract

This disclosure provides, for example, the real-time co-registration of intravascular images of a blood vessel with one or more extravascular images during the acquisition of extravascular images of a blood vessel. Key points in the first frame of the extravascular image can be identified, and then these key points can be tracked across other frames of the extravascular image. Based on the location of the key points in the frame, the intravascular image can be co-registered with one (or more) frames of the extravascular image.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 588,559, filed on October 6, 2023, the disclosure of which is incorporated herein by reference. Technical Field

[0002] This disclosure relates to angiography and intravascular imaging modalities, and to the co-registration of angiography images and intravascular imaging modalities. Background Technology

[0003] Ultrasound devices that can be inserted into a patient's vascular system have been proven to be diagnostic for a variety of diseases and conditions. For example, intravascular ultrasound (IVUS) imaging systems have been used as an intravascular imaging modality to diagnose blocked blood vessels and provide information to assist practitioners in selecting and placing stents, choosing sites for atherosclerosis resection, etc.

[0004] An IVUS imaging system includes a control module (containing a pulse generator, image acquisition and processing unit, and a monitor), a catheter, and a transducer disposed within the catheter. The catheter, containing the transducer, is positioned within a lumen or cavity of the area to be imaged (such as the vessel wall or patient tissue near the vessel wall), or near the area to be imaged. The pulse generator in the control module generates electrical pulses, which are delivered to the transducer and converted into acoustic pulses, thus propagating through the patient tissue. The patient tissue (or other structure) reflects the acoustic pulses, and the reflected pulses are absorbed by the transducer and converted back into electrical pulses. The converted electrical pulses are delivered to the image acquisition and processing unit and converted into an image that can be displayed on the monitor.

[0005] CT coronary angiography (CTA or CCTA) is a procedure that uses CT angiography to assess the coronary arteries of the heart via extravascular imaging. Typically, the patient receives an intravenous injection of contrast agent, followed by a scan of the heart using a high-speed CT scanner. CTA and IVUS are often used in combination. For example, a physician might use CTA and IVUS to assess the degree of occlusion in one (or more) coronary arteries, commonly used to diagnose coronary artery disease.

[0006] To assist physicians in interpreting these images, they are often co-registered with each other. For example, each image in a series of IVUS images can be mapped or co-located to the location of the blood vessel represented in the CTA image.

[0007] Co-registration of angiographic and IVUS images is a revolutionary technique that enhances the assessment of coronary artery disease by providing a comprehensive understanding of plaque composition, severity, and vascular anatomy. This advancement has significantly improved the accuracy of stent placement, treatment strategy development, and lesion monitoring.

[0008] Therefore, improved co-registration techniques and workflows are needed. Summary of the Invention

[0009] This disclosure provides a system configured to co-register angiographic and IVUS images live or in real-time. Conventionally, co-registration workflows operate "offline," where co-registration is performed after the imaging procedure is complete. Therefore, this disclosure offers advantages over conventional systems because it can be implemented within the system to provide real-time insights during medical procedures, enabling clinicians to observe dynamic changes in vascular anatomy, plaque characteristics, and stent deployment. Furthermore, this disclosure provides immediate feedback to physicians, enabling better on-site decision-making and allowing for real-time adjustments to the procedure for optimized outcomes.

[0010] In some embodiments, this disclosure can be implemented as a method for identifying collateral vessels from images. The method may include: receiving at a computing device a plurality of extravascular image frames associated with a patient's blood vessels; having the computing device identify the location of a key point in a first frame of the plurality of extravascular image frames; having the computing device identify the location of the key point in a second frame of the plurality of extravascular image frames, partially based on the location of the key point in the first frame; and co-registering a plurality of intravascular image frames with the first frame, partially based on the location of the key point in the first frame; or co-registering the plurality of intravascular image frames with the second frame, partially based on the location of the key point in the second frame; or co-registering the plurality of intravascular image frames with the first frame, and partially based on the location of the key point in the second frame.

[0011] In a further embodiment, the method may include: generating, by the computing device, a first graphic indication of a first frame co-registered with the plurality of intravascular image frames and a second graphic indication of a second frame co-registered with the plurality of intravascular image frames; and sending the first graphic indication and the second graphic indication to a display device to display the co-registered plurality of intravascular images in synchronization with cardiac motion associated with the plurality of extravascular image frames.

[0012] In a further embodiment of the method, the plurality of image frames are image frames captured during a film playback process.

[0013] In a further embodiment of the method, the plurality of intravascular image frames are intravascular ultrasound (IVUS) image frames.

[0014] In a further embodiment, the method may include: receiving an additional extravascular image frame at the computing device; identifying the position of the key point in the additional extravascular frame by the computing device in part based on the position of the key point in the second frame; and co-registering the plurality of intravascular image frames with the additional extravascular image frame in part based on the position of the key point in the additional extravascular image frame.

[0015] In a further embodiment of the method, receiving the additional extravascular image frame includes receiving the additional extravascular image frame during an extravascular imaging procedure.

[0016] In a further embodiment of the method, identifying the location of the key point in the first frame includes: identifying the catheter location in the first frame by the computing device; identifying the centerline of the blood vessel in the first frame by the computing device in part based on the catheter location; and identifying the location of the collateral branches of the blood vessel along the centerline by the computing device.

[0017] In a further embodiment of the method, identifying the location of the catheter in the first frame includes: inferring the location of the tip of the imaging catheter in the first frame by the computing device using a tip recognition machine learning (ML) model; or receiving an indication of the location of the tip of the imaging catheter in the first frame from an input device coupled to the computing device at the computing device.

[0018] In a further embodiment of the method, identifying the catheter location in the first frame further includes inferring the location of the inlet point of the guiding catheter in the first frame using an inlet point identification ML model by the computing device.

[0019] In a further embodiment of the method, co-registering the plurality of intravascular image frames with the first frame, in part based on the position of the key point in the first frame, includes: receiving the plurality of intravascular image frames at the computing device; identifying a subset of frames associated with collateral vessels among the plurality of intravascular image frames by the computing device; mapping the plurality of intravascular image frames onto the centerline by the computing device, in part based on the subset of frames associated with collateral vessels among the plurality of intravascular image frames; matching the collateral vessels associated with the first frame with the collateral vessels associated with the subset of frames among the plurality of intravascular image frames by the computing device; and adjusting the position of the collateral vessels associated with the subset of frames by the computing device, in part based on the matching.

[0020] In a further embodiment of the method, identifying the location of the key point in the second frame of the plurality of extravascular image frames, in part based on the location of the key point in the first frame, includes: tracking the catheter position between the first frame and the second frame by the computing device; tracking the location of the collateral branches of the blood vessel between the first frame and the second frame by the computing device; and identifying the centerline of the blood vessel in the second frame in part based on the location of the tip of the guiding catheter, the inlet point of the guiding catheter, and the collateral branches by the computing device.

[0021] In a further embodiment of the method, identifying the location of the key point in the first frame further includes: the computing device using the tip recognition ML model to infer the location of the tip of the guiding catheter in each of the plurality of extravascular image frames, wherein a confidence value is output from the tip recognition ML model each time the location of the tip of the guiding catheter is inferred; and the computing device selecting the extravascular image frame associated with the highest confidence value from the plurality of extravascular image frames as the first frame.

[0022] In a further embodiment of the method, identifying the location of the key point in the first frame further includes: the computing device identifying the contrast of each of the plurality of extravascular image frames; and the computing device selecting an extravascular image frame with feasible image quality from the plurality of extravascular image frames as the first frame.

[0023] In some embodiments, this disclosure can be implemented as a computer-readable storage device. The computer-readable storage device may include instructions executable by a processor of a computing device coupled to an intravascular imaging device and a fluorescence endoscope, wherein, when executed, the instructions cause the computing device to perform any of the methods disclosed herein.

[0024] In some embodiments, this disclosure can be implemented as an apparatus including a processor arranged to be coupled to an intravascular imaging device and a fluorescence endoscopy device. The apparatus may further include a memory containing instructions, the processor being arranged to execute the instructions to implement any of the methods disclosed herein.

[0025] In some embodiments, this disclosure can be implemented as an apparatus for a cross-modal collateral matching system. The apparatus may include a processor and a memory storage device coupled to the processor, the memory storage device including instructions executable by the processor, the instructions, when executed, causing the apparatus to perform the following operations: receiving a plurality of extravascular image frames associated with a patient's blood vessels; identifying the location of a key point in a first frame of the plurality of extravascular image frames; identifying the location of the key point in a second frame of the plurality of extravascular image frames, partially based on the location of the key point in the first frame; and co-registering a plurality of intravascular image frames with the first frame, partially based on the location of the key point in the first frame; or co-registering the plurality of intravascular image frames with the second frame, partially based on the location of the key point in the second frame; or co-registering the plurality of intravascular image frames with the first frame, and partially based on the location of the key point in the second frame.

[0026] In a further embodiment of the device, the instructions, when executed, further cause the device to perform the following operations: generate a first graphic indication of the first frame co-registered with the plurality of intravascular image frames and a second graphic indication of the second frame co-registered with the plurality of intravascular image frames; and send the first graphic indication and the second graphic indication to a display device to display the co-registered plurality of intravascular images in synchronization with cardiac motion associated with the plurality of extravascular image frames.

[0027] In a further embodiment of the device, the plurality of image frames are image frames from a movie playback captured during a fluorescence perspective procedure.

[0028] In a further embodiment of the device, the plurality of intravascular image frames are intravascular ultrasound (IVUS) image frames.

[0029] In a further embodiment of the device, the instructions, when executed, further cause the device to perform the following operations: receive an additional extravascular image frame; identify the position of the key point in the additional extravascular frame in part based on the position of the key point in the second frame; and co-register the plurality of intravascular image frames with the additional extravascular image frame in part based on the position of the key point in the additional extravascular image frame.

[0030] In a further embodiment of the device, the instruction, when executed to receive the additional extravascular image frame, further causes the device to receive the additional extravascular image frame during an extravascular imaging procedure.

[0031] In a further embodiment of the device, when the instructions are executed to identify the location of the key point in the first frame, the device further causes the device to perform the following operations: identify the catheter location in the first frame; identify the centerline of the blood vessel in the first frame in part based on the catheter location; and identify the location of the collateral branches of the blood vessel along the centerline.

[0032] In a further embodiment of the device, when the instructions are executed to identify the location of the catheter in the first frame, the device further causes the device to perform the following operations: using a tip recognition machine learning (ML) model to infer the location of the tip of the imaging catheter in the first frame; or receiving an indication of the location of the tip of the imaging catheter in the first frame from an input device coupled to the computing device.

[0033] In a further embodiment of the device, when the instruction is executed to identify the catheter location in the first frame, the device further causes the device to use an entry point recognition ML model to infer the location of the entry point of the guiding catheter in the first frame.

[0034] In a further embodiment of the device, when the instructions are executed to co-register the plurality of intravascular image frames with the first frame in part based on the location of the key points in the first frame, the device further causes the device to perform the following operations: receiving the plurality of intravascular image frames; identifying a subset of frames associated with collateral vessels in the plurality of intravascular image frames; mapping the plurality of intravascular image frames onto the centerline in part based on the subset of frames associated with the collateral vessels in the plurality of intravascular image frames; matching the collateral vessels associated with the first frame with the collateral vessels associated with the subset of frames in the plurality of intravascular image frames; and adjusting the position of the collateral vessels associated with the subset of frames in part based on the matching.

[0035] In a further embodiment of the device, when the instructions are executed to identify the position of the key point in the second frame of the plurality of extravascular image frames in part based on the position of the key point in the first frame, the device further causes the device to perform the following operations: tracking the catheter position between the first frame and the second frame; tracking the position of the collateral branches of the blood vessel between the first frame and the second frame; and identifying the centerline of the blood vessel in the second frame in part based on the position of the tip of the guiding catheter, the inlet point of the guiding catheter, and the collateral branches.

[0036] In a further embodiment of the device, when the instruction is executed to identify the location of the key point in the first frame, the device further causes the following operations: using the tip recognition ML model to infer the location of the tip of the guiding catheter in each of the plurality of extravascular image frames, wherein a confidence value is output from the tip recognition ML model each time the location of the tip of the guiding catheter is inferred; and selecting the extravascular image frame associated with the highest confidence value from the plurality of extravascular image frames as the first frame.

[0037] In a further embodiment of the device, when the instruction is executed to identify the location of the key point in the first frame, the device further causes the device to perform the following operations: identify the contrast of each of the plurality of extravascular image frames; and select an extravascular image frame with feasible image quality from the plurality of extravascular image frames as the first frame.

[0038] In some embodiments, this disclosure can be implemented as a computer-readable storage device. The computer-readable storage device may include instructions executable by a processor of a cross-modal collateral matching system, wherein, when executed, the instructions cause the processor to perform the following operations: receiving a plurality of extravascular image frames associated with a patient's blood vessels; identifying the location of a key point in a first frame of the plurality of extravascular image frames; identifying the location of the key point in a second frame of the plurality of extravascular image frames, partially based on the location of the key point in the first frame; and co-registering the plurality of intravascular image frames with the first frame, partially based on the location of the key point in the first frame; or co-registering the plurality of intravascular image frames with the second frame, partially based on the location of the key point in the second frame; or co-registering the plurality of intravascular image frames with the first frame, and partially based on the location of the key point in the second frame.

[0039] In a further embodiment of the computer-readable storage device, the instructions, when executed, further cause the processor to perform the following operations: generate a first graphic indication of the first frame co-registered with the plurality of intravascular image frames and a second graphic indication of the second frame co-registered with the plurality of intravascular image frames; and send the first graphic indication and the second graphic indication to a display device to display the co-registered plurality of intravascular images in synchronization with cardiac motion associated with the plurality of extravascular image frames.

[0040] In a further embodiment of the computer-readable storage device, the plurality of image frames are image frames from a movie playback captured during a fluorescence permeation procedure.

[0041] In a further embodiment of the computer-readable storage device, the plurality of intravascular image frames are intravascular ultrasound (IVUS) image frames.

[0042] In a further embodiment of the computer-readable storage device, the instructions, when executed, further cause the processor to perform the following operations: receive an additional extravascular image frame; identify the location of the key point in the additional extravascular frame in part based on the location of the key point in the second frame; and co-register the plurality of intravascular image frames with the additional extravascular image frame in part based on the location of the key point in the additional extravascular image frame. Attached Figure Description

[0043] To facilitate identification of any discussion of an element or action, one or more of the highest-order digits in the figure references refer to the figure number in which the element is first introduced.

[0044] Figure 1A and Figure 1B A live co-registration system according to at least one embodiment is demonstrated.

[0045] Figure 2 A routine for real-time co-registration of IVUS images with angiography images, according to at least one embodiment, is shown.

[0046] Figure 3 A routine for identifying key points in an initial angiography image frame, according to at least one embodiment, is shown.

[0047] Figure 4 A routine for identifying key points in subsequent angiography image frames, according to at least one embodiment, is shown.

[0048] Figure 5 A routine for co-registering IVUS image frames with angiography image frames based on key points, according to at least one embodiment, is shown.

[0049] Figure 6A and Figure 6B Example angiography image frames according to at least one embodiment are shown, along with the identification of key points and the center line of the blood vessels.

[0050] Figure 7A , Figure 7B and Figure 7C A series of example angiography image frames according to at least one embodiment are shown, along with the identification of key points and the center line of the blood vessels.

[0051] Figure 8A and Figure 8B An exemplary artificial intelligence / machine learning (AI / ML) system suitable for use with at least one embodiment is shown.

[0052] Figure 9 A computer-readable storage medium according to at least one embodiment is shown.

[0053] Figure 10 An example vascular imaging system according to at least one embodiment is shown.

[0054] Figure 11 A schematic representation of a machine in the form of a computer system according to an example embodiment is shown, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0055] As described above, this disclosure provides methods and apparatus for real-time or live co-registration of angiographic and IVUS images. It should be noted that while this disclosure refers to angiography and IVUS as examples and techniques for description, any suitable extravascular and intravascular imaging modalities can be utilized. That is, this disclosure relates to live co-registration of extravascular and intravascular imaging modalities, and not to any specific imaging modality itself.

[0056] Figure 1A and Figure 1B A live co-registration system 100 according to an embodiment of this disclosure is illustrated. Typically, the live co-registration system 100 is a system configured to co-register images of blood vessels captured using different imaging modalities in real time (e.g., during image acquisition, etc.). For example, the live co-registration system 100 may be configured to co-register angiography image frame 118 and IVUS image frame 120. For this purpose, the live co-registration system 100 includes or may be coupled to a vascular imaging system 102. The vascular imaging system 102 may be any of various vascular imaging systems configured to capture images from multiple imaging modalities (e.g., angiography, IVUS, intravascular OCT, etc.). Reference Figure 10 The combined internal and external imaging system 1000 described herein is an example of a vascular imager configured to capture both external (angiography) vascular images and internal (IVUS) vascular images.

[0057] The live co-registration system 100 includes a computing device 104. The computing device 104 can be any of a variety of computing devices. In some embodiments, the computing device 104 may be integrated into and / or implemented by the console of the vascular imaging system 102. For some embodiments, the computing device 104 may be a tablet, laptop, workstation, or server communicatively coupled to the vascular imaging system 102. For other embodiments, the computing device 104 may be provided by a cloud-based computing device, such as a compute-as-a-service (CaaS) system accessible via a network (e.g., the Internet, intranet, wide area network, etc.). The computing device 104 may include a processor 106, memory 108, input and / or output (I / O) devices 110, and a network interface 114.

[0058] Processor 106 may include a circuit system or processor logic, such as any of a variety of commercial processors. In some examples, processor 106 may include multiple processors, a multi-threaded processor, a multi-core processor (whether the multiple cores coexist on the same die or on separate dies), and / or some other kind of multiprocessor architecture (in which multiple physically separate processors are linked in some way). Additionally, in some examples, processor 106 may include a graphics processing section and may include dedicated memory, multi-threaded processing, and / or some other parallel processing capability. In some examples, processor 106 may be an application-specific integrated circuit (ASIC) or a field-programmable integrated circuit (FPGA).

[0059] Memory 108 may include logic forming non-volatile memory, or a combination of non-volatile memory and volatile memory, for persistently storing data, a portion of which includes an array of integrated circuits. It should be understood that memory 108 may be based on any of a variety of technologies. In particular, the array of integrated circuits included in memory 108 may be arranged to form one or more types of memory, such as dynamic random access memory (DRAM), NAND memory, NOR memory, etc.

[0060] I / O device 110 can be any of a variety of devices for receiving input and / or providing output. For example, I / O device 110 may include a keyboard, mouse, joystick, foot pedal, haptic feedback device, LED, etc. Display 112 can be a conventional display or a touch-enabled display. Furthermore, display 112 can utilize various display technologies, such as liquid crystal display (LCD), light-emitting diode (LED), or organic light-emitting diode (OLED).

[0061] Network interface 114 may include logical and / or features for supporting communication interfaces. For example, network interface 114 may include one or more interfaces that operate according to various communication protocols or standards to communicate via direct communication links or network communication links. Direct communication may be performed via communication protocols or standards described in one or more industry standards, including their derivatives and variants. For example, network interface 114 may facilitate communication via buses such as PCIe, NVMe, USB, SMBus, SAS (e.g., Serial Attached Small Computer System Interface (SCSI)), SATA, etc. Additionally, network interface 114 may include logical and / or features for enabling communication via various wired or wireless network standards such as the 802.11 communication standard. For example, network interface 114 may be configured to support wired communication protocols or standards such as Ethernet. As another example, network interface 114 can be configured to support wireless communication protocols or standards, such as Wi-Fi, Bluetooth, ZigBee, LTE, 5G, etc.

[0062] The memory 108 may include instructions 116, angiography image frames 118, IVUS image frames 120, initial angiography image frames 122, catheter positions 124a and 124b, vessel centerlines 126a and 126b, angiography image collateral positions 128a and 128b, IVUS image collateral positions 130, mapped collaterals 132a and 132b, matched collaterals 134a and 134b, subsequent angiography image frames 136, and key points 138a and 138b.

[0063] During operation, processor 106 may execute instructions 116 to cause computing device 104 to receive IVUS image frame 120 and initial angiography image frame 122 from vascular imaging system 102. Typically, angiography image frame 118 may be a series of angiography images (also referred to as angiography films, e.g., movie playback, etc.)) that can be CT images of the patient's heart (or a portion of the patient's heart) captured after contrast agent is injected into the patient's vascular system. Similarly, IVUS image frame 120 may be a series of ultrasound images captured from within a blood vessel as an ultrasound probe is pulled back through a portion of the blood vessel passing through the patient's heart. In some embodiments, angiography image frame 118 may be captured simultaneously with IVUS image frame 120. In other embodiments, IVUS image frame 120 may be captured first, followed by angiography image frame 118, and the live co-registration system 100 may be configured to co-register angiography image frame 118 with IVUS image frame 120 in real time while angiography image frame 118 is being captured. In some embodiments, angiography image frame 118 may be captured first, and then IVUS image frame 120 may be captured. For example, processor 106 may execute instruction 116 to cause computing device 104 to use the previously captured angiography image frame as angiography image frame 118.

[0064] It should be noted that this disclosure provides for "live" co-registration of IVUS image frame 120 with one (or each) of the frames in angiographic image frame 118 as angiographic image frame 118 is captured. Thus, there will necessarily be a first co-registered frame, and subsequent co-registered frames may exist. For example, in some embodiments, co-registration may begin after several (e.g., 2, 3, 4, 5, etc.) frames of angiographic image frame 118 have been captured, and may continue to co-register these IVUS image frames with subsequently captured frames as IVUS image frame 120 is captured. For this purpose, processor 106 may execute instruction 116 to cause computing device 104 to select image frames from angiographic image frame 118 to initiate "live" co-registration. In some embodiments, processor 106 may execute instruction 116 to select a frame with feasible image quality from angiographic image frame 118 as the initial angiographic image frame 122. Figure 1A A live co-registration system 100 is described, which co-registers IVUS image frame 120 with an initial frame (e.g., initial angiography image frame 122) in angiography image frame 118. Figure 1B A live co-registration system 100 is depicted that co-registers subsequent frames (e.g., subsequent angiographic image frame 136) in angiographic image frame 118. As used herein, the term "feasible" may mean an image with minimal vascular crossings, optimal contrast, and the clearest guiding catheter inlet and catheter tip, etc.

[0065] Processor 106 may further execute instructions 116 to enable computing device 104 to identify catheter location 124a. Typically, catheter location 124a indicates the location of both: (1) the inlet of the guiding catheter through which the imaging catheter is introduced, through which IVUS image frame 120 is also captured; and (2) the tip of the imaging catheter in the initial angiography image frame 122 (see reference). Figure 6A and Figure 6B In some embodiments, processor 106 may execute instructions 116 to infer catheter position 124a using a machine learning (ML) model trained to identify the position of the guide catheter inlet and / or imaging catheter tip from angiographic image frames. In such an example, some embodiments of this disclosure may specify that processor 106 is configured to infer catheter position 124a from each frame of angiographic image frames 118 and select the frame with the highest confidence in the identification of catheter position 124a (e.g., as determined by an ML model, etc.) as the initial angiographic image frame 122.

[0066] In some embodiments, a first ML model may be trained and used to infer the position of the imaging catheter tip from angiographic image frame 118, while a second ML model may be trained and used to infer the position of the guiding catheter inlet. In alternative embodiments of this disclosure, processor 106 may execute instructions 116 to receive, for example, indication of a component of catheter position 124a from a user, etc. For example, processor 106 may execute instructions 116 to receive indication of the imaging catheter tip via I / O device 110 (or the like).

[0067] Processor 106 may further execute instructions 116 to identify the vessel centerline 126a from the initial angiographic image frame 122 using catheter position 124a. It should be noted that various techniques can be used to identify the vessel centerline from angiographic images. This disclosure can be implemented using any of these various techniques to identify the vessel centerline 126a from the initial angiographic image frame 122 and catheter position 124a.

[0068] Once processor 106 executes 116 to identify the vessel centerline 126a, processor 106 can execute instruction 116 to identify angiographic image collateral locations 128a defined relative to the initial angiographic image frame 122. Further, processor 106 can execute instruction 116 to identify other reference points (e.g., guide catheter entry point, imaging catheter tip location, etc.). Additionally, processor 106 can execute instruction 116 to identify IVUS image collateral locations 130 from IVUS image frame 120. It should be noted that various techniques can be used to identify collaterals in angiographic or IVUS images. This disclosure can be implemented using any of these various techniques to identify angiographic image collateral locations 128a from angiographic image frame 118 and vessel centerline 126a, and to identify IVUS image collateral locations 130 from IVUS image frame 120.

[0069] The processor 106 may further execute instruction 116 to map the collateral locations 128a and 130 of the angiography image onto the vessel centerline 126a of the initial angiography image frame 122, thereby generating mapped collaterals 132a. Further, the processor 106 may execute instruction 116 to match each position in the collateral location 128a of the angiography image with the corresponding position in the collateral location 130 of the IVUS image, thereby generating matched collaterals 134a.

[0070] Once IVUS image frame 120 is co-registered with the initial frame (e.g., initial angiography image frame 122) in angiography image frame 118, processor 106 can execute instruction 116 to co-register IVUS image frame 120 with other frames in angiography image frame 118. For example, Figure 1B A live co-registration system 100 with subsequent angiographic image frames 136 stored in memory 108 is shown. It should be noted that the initial angiographic image frame 122 and the subsequent angiographic image frame 136 may not be stored separately in memory 108 as depicted; instead, the processor 106 may execute the display 112 to label or designate each frame in the angiographic image frame 118 as either the initial angiographic image frame 122 or the subsequent angiographic image frame 136. However, for clarity of presentation and for ease of reference to the individual frames in the angiographic image frame 118 (e.g., frame N, frame N+1, etc.) and the identified collateral branches and key points, these frames are depicted separately from the angiographic image frame 118 in the figure.

[0071] Processor 106 can execute instructions 116 to identify "key points" in subsequent angiographic image frames 136 from the "key points" of the initial angiographic image frame 122. Typically, key points 138b can include any reference point. Figure 1A and Figure 1B In the embodiments depicted, keypoint 138b includes collateral branches as well as the location of the guiding catheter inlet and the imaging catheter tip. For example, keypoint 138a includes catheter location 124a and angiographic image collateral branch location 128a, while keypoint 138b includes catheter location 124b and angiographic image collateral branch location 128b. Typically, keypoint 138a from an initial frame (e.g., frame N) is used to identify keypoint 138b in subsequent frames (e.g., frame N > 1). For example, processor 106 may execute instruction 116 to identify keypoint 138b (e.g., catheter location 124a and angiographic image collateral branch location 128a) of a subsequent angiographic image frame 136 from keypoint 138a (e.g., catheter location 124a and angiographic image collateral branch location 128a).

[0072] It should be understood that due to perspective shortening of three-dimensional (3D) tissue structures (e.g., blood vessels in angiography image frame 118 and IVUS image frame 120) and limitations of the two-dimensional (2D) domain, the sharpest key points in each frame of angiography image frame 118 will be detected. In some embodiments, processor 106 may execute instructions 116 to identify key points 138b from collateral locations 128a in angiography images using point tracking algorithms (e.g., multi-target tracking algorithms, simple online real-time (SORT) tracking algorithms, joint detection and embedding (JDE) algorithms). In some embodiments, an ML model (e.g., a single-shot detector, etc.) may be trained to track key points across angiography image frames and may be used to infer key points 138b from key points 138a.

[0073] Processor 106 may execute instructions 116 to identify the vessel centerline 126b of subsequent angiography image frames 136 from the guide catheter inlet location 124b and the imaging catheter tip, as well as from subsequent angiography image frames 136. In a further embodiment, processor 106 may execute instructions 116 to identify the vessel centerline 126b of subsequent angiography image frames 136 from catheter location 124b (e.g., guide catheter inlet location and imaging catheter tip location) and angiography image collateral location 128b, thereby enhancing the accuracy of the identified centerline. For some embodiments, processor 106 may execute instructions 116 to identify the vessel centerline 126b from only catheter location 124b (e.g., guide catheter inlet location and imaging catheter tip location) or from both catheter location 124b and angiography image collateral location 128b. For example, if one or more of the catheter locations 124b cannot be reliably identified (e.g., with threshold confidence, etc.), the processor 106 may execute instruction 116 to identify the vascular centerline 126b from the collateral locations 128b in the angiography image. Further, the processor 106 may execute instruction 116 to identify the vascular centerline 126b from identifiable catheter locations (e.g., the inlet of the guiding catheter or the tip of the imaging catheter) and the collateral locations 128b in the angiography image.

[0074] The processor 106 may further execute instruction 116 to map the collateral locations 128b of the angiography image and the collateral locations 130 of the IVUS image onto the vessel centerline 126b of the subsequent angiography image frame 136, thereby generating mapped collaterals 132b. Further, the processor 106 may execute instruction 116 to match each position in the collateral locations 128b of the angiography image with the corresponding positions in the collateral locations 130 of the IVUS image, thereby generating matched collaterals 134b.

[0075] Therefore, this disclosure provides a technique that enables real-time co-registration of IVUS and angiographic images based on dynamic tracking of the catheter pullback path, identification of angiographic collaterals during angiographic playback, and precise alignment of the IVUS collaterals with the identified angiographic collaterals. This offers significant advantages and improvements over current co-registration techniques.

[0076] Figure 2 , Figure 3 , Figure 4 and Figure 5Routines 200, 300, 400, and 500, representing some embodiments of this disclosure, are illustrated. Routine 200 can be implemented by the live co-registration system 100 or another computing device, as described herein, to co-register IVUS images and angiographic images in real time. For example, routine 200 can be implemented to co-register IVUS image frame 120 with one frame of angiographic image frame 118 in real time (e.g., while capturing angiographic image frame 118, etc.). Routine 200 may include routines 300, 400, and 500, which are described more fully below.

[0077] Routine 200 may begin at box 202 “Receiving multiple angiographic image frames of a patient’s blood vessels at a computing device”, where angiographic image frames may be received at a computing device. For example, computing device 104 of live co-registration system 100 may receive angiographic image frame 118 (e.g., from vascular imaging system 102, etc.).

[0078] Routine 200 can continue from box 202 to routine 300, where key points in the first (e.g., N = 1) frame of angiographic image frames can be identified. For example, processor 106 can execute instruction 116 to identify key point 138a from initial angiographic image frame 122, as outlined in routine 300.

[0079] Routine 200 can continue from routine 300 to routine 400 or routine 500. For example, routine 200 can continue from routine 300 to routine 400 to prepare subsequent (e.g., N > 1) frames in the angiography image frame for co-registration. In another example, routine 200 can continue from routine 300 to routine 500 to co-register an IVUS image frame with a first (e.g., N = 1) angiography image frame.

[0080] Routine 200 can continue from routine 400 to routine 500 or return to box 202. For example, routine 200 can continue from routine 400 to routine 500 to co-register IVUS image frames with subsequent (e.g., N > 1) angiographic image frames. Similarly, routine 200 can continue from routine 500 to box 202. Routine 200 can return to box 202 from either routine 400 or routine 500 to receive additional angiographic image frames (e.g., capturing more frames in movie playback, etc.).

[0081] Figure 3Routine 300 is shown, which may begin at box 302. At box 302, “Identification of the first (N = 1) frame in the angiography image frames by the computing device,” the initial frame or “first” frame in the angiography image frames is identified. For example, computing device 104 may identify one frame in angiography image frames 118 to use as the initial angiography image frame 122. Processor 106 may execute instructions 116 to determine or select the initial angiography image frame 122 from the angiography image frames 118, for example, based on the image quality of each frame.

[0082] In other embodiments, processor 106 may execute instruction 116 to identify the first frame in conjunction with the identification of the guide catheter tip. For example, routine 300 may continue from block 302 to block 304. At block 304, “Identification of at least one location of the guide catheter in the first frame by the computing device,” the location of the guide catheter (e.g., IVUS guide catheter, etc.) may be identified. In some embodiments, both the tip of the imaging catheter and the entry point of the guide catheter may be identified. For example, computing device 104 may be configured to identify catheter location 124a and imaging catheter tip from the initial angiographic image frame 122.

[0083] In some examples, processor 106 may execute instruction 116 to infer location using an ML model to identify catheter location 124a and imaging catheter tip (see reference). Figure 6A and Figure 6B In some embodiments, processor 106 may execute instructions 116 to infer the location of the guiding catheter inlet for several frames in angiographic image frame 118, and select the frame in which the guiding catheter inlet is identified with the highest confidence as the initial angiographic image frame 122. In a further example, processor 106 may execute instructions 116 to use different ML models to identify the location of the imaging catheter tip and the guiding catheter inlet point (e.g., refer to...). Figure 6A ).

[0084] Continuing to box 306, “Identifying the centerline of a vessel in the first frame, partially based on the catheter position, by a computing device,” the centerline of a vessel can be identified from the first frame and the catheter position. For example, computing device 104 can be configured to identify the vessel centerline 126b from catheter position 124a and initial angiography image frame 122. Processor 106 can execute instruction 116 to identify the vessel centerline 126a from catheter position 124a and initial angiography image frame 122 based on a centerline mapping algorithm.

[0085] Continuing to box 308, “Identification of collateral vessels of a vessel partially based on the centerline by a computing device,” collateral vessels of a vessel can be identified based on the vessel centerline. For example, computing device 104 can be configured to identify the location 128a of a collateral vessel in an angiography image from the vessel centerline 126a and the initial angiography image frame 122.

[0086] Figure 4 Example 400 is shown, which may begin at box 402. At box 402, “Tracking the position of keypoints in subsequent (N > 1) frames of angiography image frames by a computing device based on the position of keypoints in previous (N-1) frames of angiography image frames,” keypoints can be tracked between (or across) multiple angiography image frames. For example, computing device 104 can be configured to track keypoint 138a from initial angiography image frame 122 to subsequent angiography image frame 136, thereby identifying keypoint 138b. Processor 106 can execute instruction 116 to track or identify keypoint 138b from subsequent angiography image frame 136 based on keypoint 138a and initial angiography image frame 122.

[0087] Continuing to box 404, “Identifying the centerline of a blood vessel in a subsequent frame, partially based on the location of key points by a computing device,” the centerline of a blood vessel can be generated from the subsequent frames and the key point locations. For example, computing device 104 can be configured to identify the centerline 126b of a blood vessel from key point 138b and subsequent angiography image frame 136. Processor 106 can execute instruction 116 to identify the centerline 126b of a blood vessel from key point 138b and subsequent angiography image frame 136 based on a centerline mapping algorithm.

[0088] Figure 5Routine 500 is shown, which may begin at block 502. At block 502, “Receiving IVUS Image Frames of Patient’s Vessels at a Computing Device,” IVUS image frames can be received at a computing device. For example, computing device 104 of the live co-registration system 100 may receive IVUS image frame 120. In some embodiments, IVUS image frame 120 may be received from vascular imaging system 102, while in other embodiments, IVUS image frame 120 may be pre-captured by vascular imaging system 102 and stored in memory (e.g., memory 108, a memory location accessible via network interface 114). In other words, for some examples, the live co-registration system 100 can be configured to co-register the angiography image frame 118 and the IVUS image frame 120 in real time while capturing both; in other embodiments, the live co-registration system 100 can be configured to co-register the angiography image frame 118 and the IVUS image frame 120 while capturing the angiography image frame 118, wherein the IVUS image frame 120 has been previously captured. In some embodiments, the live co-registration system 100 can be configured to co-register the angiography image frame 118 and the IVUS image frame 120 while capturing the IVUS image frame 120, wherein the angiography image frame 118 has been previously captured. For still other embodiments, the live co-registration system 100 can be configured to co-register the angiography image frame 118 and the IVUS image frame 120, wherein both the angiography image frame 118 and the IVUS image frame 120 have been previously captured.

[0089] Continuing to box 504, “Identification of Side Branches in IVUS Image Frames by a Computing Device,” side branches in IVUS image frames can be identified. For example, computing device 104 can be configured to identify IVUS image side branch locations 130 from IVUS image frame 120. Processor 106 can execute instructions 116 to identify IVUS image side branch locations 130 from IVUS image frame 120 using any of a variety of side branch identification techniques.

[0090] Continuing to box 506, “Mapping IVUS collaterals onto a frame (N >= 1) of angiographic image frames by a computing device in part based on the vessel centerline,” the location of an IVUS collateral can be mapped onto frames of angiographic images. For example, computing device 104 can be configured to map IVUS image collateral location 130 onto a frame of angiographic image 118 based on the vessel centerline of that frame. As a specific example, processor 106 can execute instruction 116 to map IVUS image collateral location 130 onto an initial angiographic image frame 122 in part based on vessel centerline 126a, thereby producing mapped collateral 132a. As another specific example, processor 106 can execute instruction 116 to map IVUS image collateral location 130 onto a subsequent angiographic image frame 136 in part based on vessel centerline 126b, thereby producing mapped collateral 132b. It should be noted that processor 106 can execute instruction 116 to map all frames from IVUS image frame 120 onto angiography image frame 118, wherein the mapping of IVUS image collateral location 130 acts as a control point to align angiography collaterals and IVUS collaterals and minimize the effect of fluoroscopy shortening.

[0091] Continuing to box 508, “Matching IVUS Collaterals and Angiographic Collaterals to Each Other by a Computing Device,” IVUS collaterals mapped onto an angiographic image and angiographic image collaterals can be matched to each other. For example, computing device 104 can be configured to match angiographic image collateral location 128a and IVUS image collateral location 130 based on mapped collateral 132a. As a specific example, processor 106 can execute instruction 116 to match angiographic image collateral location 128a with IVUS image collateral location 130 based on mapped collateral 132a, thereby producing a matched collateral 134a. As another specific example, processor 106 can execute instruction 116 to match angiographic image collateral location 128b with IVUS image collateral location 130 based on mapped collateral 132b, thereby producing a matched collateral 134b.

[0092] Continuing to box 510, “Adjusting IVUS Collateral Positions Based on Matching by a Computing Device,” the position of an IVUS collateral can be adjusted based on a match between angiographic image collateral and an IVUS image collateral. For example, computing device 104 can be configured to adjust the position of IVUS image collateral 130 based on a match between angiographic image collateral position 128a and IVUS image collateral position 130 at box 508.

[0093] Figure 6A and Figure 6BExamples of identifying the location of the guiding catheter and the centerline of the blood vessel are shown according to some embodiments. These figures are depicted with reference to a live co-registration system 100 and an initial angiographic image frame 122. However, Figure 6A and Figure 6B It can be implemented to identify the location of the guiding catheter from another angiography image frame in the angiography image frame 118 (e.g., subsequent angiography image frame 136, etc.).

[0094] Figure 6A An ML model 602 is shown, which includes an inlet recognition model 604a and an inlet recognition model 604b. Typically, the ML model 602 can be configured to infer the catheter position 124a from an initial angiographic image frame 122. For example, the inlet recognition model 604a can be configured to infer the tip of the imaging catheter from the angiographic image (e.g., the initial angiographic image frame 122), while the inlet recognition model 604b can be configured to infer the inlet point of the guiding catheter from the angiographic image.

[0095] As described above, in some embodiments, the imaging catheter tip may not be identifiable from angiographic images, or may be identified with a confidence level below a threshold. In such an example, the location of the imaging catheter tip in the initial angiographic image frame 122 may be received by the computing device 104 from the user of the live co-registration system 100.

[0096] Figure 6B A subroutine block 606 is shown, which can be a subroutine of instruction 116 of the real-time co-registration system 100 configured to identify centerlines (e.g., a centerline identification algorithm). That is, instruction 116 can include subroutine block 606, which itself can be executed by processor 106 to identify vascular centerlines (e.g., vascular centerline 126a, etc.). For example, Figure 6B A subroutine box 606 is shown that is configured to identify the vessel centerline 126a on the initial angiography image frame 122 from the initial angiography image frame 122 and the catheter position 124a (e.g., entry point position 608 and imaging tip position 610).

[0097] Figure 7A , Figure 7B and Figure 7C Examples of tracking and / or identifying key points and identifying the vessel centerline from key points are illustrated according to some embodiments. These figures are depicted with reference to a live co-registration system 100 and angiographic image frame 118. For example, these figures depict tracking key points in angiographic image frame 118 to identify an angiographic image with key point 702.

[0098] Figure 7AA subroutine block 704 is shown, which can be a subroutine of instruction 116 of the live co-registration system 100 configured to track key points (e.g., a key point tracking algorithm, etc.) in angiographic image frames. That is, instruction 116 can include subroutine block 704, which itself can be executed by processor 106 to track key points across angiographic image frames 118, thereby identifying angiographic images with key points 702. As described above, key points can include the location of the guiding catheter inlet and the imaging catheter tip (and collateral locations). For example, Figure 7A A subroutine box 704 is shown, which is configured to identify and track key points between the initial angiography image frame 122 and the subsequent angiography image frame 136 based on key points 138a, thereby enabling the identification of key points 138b.

[0099] Figure 7B A keypoint detection and object tracking ML model 708 is shown. Typically, the keypoint detection and object tracking ML model 708 can be configured to infer keypoints 138b from matched collateral branches 134a and angiographic image frame 118. The processor 106 can be configured to execute instructions 116 to infer keypoints 138b from keypoints 138a based on the keypoint detection and object tracking ML model 708 and the angiographic image frame 118.

[0100] Figure 7C A subroutine block 706 is shown, which can be a subroutine of instruction 116 of the live co-registration system 100 configured to identify centerlines from key points in angiographic image frames (e.g., a key centerline identification algorithm, etc.). That is, instruction 116 can include subroutine block 706, which itself can be executed by processor 106 to identify vascular centerlines (e.g., vascular centerline 126b, etc.). For example, Figure 7C A subroutine box 706 is shown, which is configured to identify the vessel centerline 126b on the subsequent angiography image frame 136 from the subsequent angiography image frame 136 and key points 138b.

[0101] As noted, in some embodiments, ML models can be used to infer the location of the guiding catheter and / or key points in angiographic images. For example, the processor 106 of computing device 104 can execute instructions 116 to infer the catheter location 124a from angiographic image frame 118 using ML model 602, or to infer key points 138a, 138b, etc., from angiographic image frame 118 using key point detection and target tracking ML model 708. In this example, the ML model can be stored in the memory 108 of computing device 104. However, it should be understood that the ML model must be trained before deployment. Figure 8AAn ML training environment 800a is shown that can be used to train an ML model, which can later be used, as described herein, to generate (or infer) catheter positions 124a, 124b, etc., from angiographic image frame 118. The ML training environment 800a may include an ML system 802, such as a computing device that applies ML algorithms to learn relationships. In this example, the ML algorithm can learn the relationship between a set of inputs (e.g., angiographic image frame 118) and outputs (e.g., catheter positions 124a, 124b, etc.).

[0102] The ML system 802 can utilize experimental data 804 collected during several prior procedures. Experimental data 804 may include angiographic image frames 118 of several patients. Experimental data 804 may be located in the same location as the ML system 802 (e.g., stored in the storage device 812 of the ML system 802), may be remote from the ML system 802 and accessible via a network interface 814, or may be a combination of local and remote data.

[0103] Experimental data 804 can be used to form training data 806, which includes angiography image frames 118 (e.g., initial angiography image frame 122, subsequent angiography image frames 136, etc.).

[0104] As described above, the ML system 802 may include a storage device 812, which may include a hard disk drive, a solid-state storage device, and / or random access memory. The storage device 812 may store training data 806. Typically, the training data 806 may include information elements or data structures, including angiography image frame 118 and an indication of the associated expected catheter position 824. The training data 806 can be used to train the ML model 808a. Depending on the application, different types of models may be used to form the basis of the ML model 808a. For example, in this example, an artificial neural network (ANN) may be particularly well-suited for learning the association between CT angiography and / or IVUS images (e.g., angiography image frame 118, etc.) in angiography image frame 118 and catheter positions 124a, 124b, etc. (e.g., an indication of the position of the imaging catheter tip and / or the guide catheter inlet point). Convolutional neural networks may also be well-suited for this task. Any suitable training algorithm 816 can be used to train the ML model 808a. Nevertheless, Figure 8AThe examples depicted may be particularly well-suited for supervised training or reinforcement learning training algorithms. For supervised training, the ML system 802 can apply angiographic image frames 118 as model input 818, expecting catheter positions 824 to be mapped to these model inputs to learn the associations between angiographic image frames 118 and catheter positions 124a, 124b, etc. In a reinforcement learning scenario, the training algorithm 816 can attempt to maximize some or all (or a weighted combination) of the mappings between model inputs 818 and catheter positions 124a, 124b, etc., thereby producing an ML model 808a with minimum error. In some embodiments, training data 806 may be split into “training” data and “test” data, wherein a subset of training data 806 may be used to adjust ML model 808a (e.g., the model’s internal weights, etc.), while another non-overlapping subset of training data 806 may be used to measure the accuracy of ML model 808a in inferring (or generalizing) the guide duct position from “unseen” training data 806 (e.g., training data 806 not used to train ML model 808a).

[0105] The ML model 808a can be applied using processor circuitry 810, which may include suitable hardware processing resources for operating on the logic and structures in storage device 812. The development of training algorithm 816 and / or trained ML model 808a may depend at least in part on hyperparameters 820. In an exemplary embodiment, model hyperparameters 820 may be automatically selected based on hyperparameter optimization logic 822, which may include any known hyperparameter optimization techniques suitable for the selected ML model 808a and the training algorithm 816 to be used. In an alternative embodiment, the ML model 808a may be retrained over time to incorporate new knowledge and / or updated experimental data 804.

[0106] Once the ML model 808a is trained, it can be applied (e.g., by processor circuitry 810, processor 106, etc.) to new input data (e.g., angiographic image frames 118 captured before, after, etc., PCI intervention). This input to the ML model 808a can be formatted according to a predefined model input 818 to reflect how the training data 806 is provided to the ML model 808a. The trained ML model 808a can generate catheter positions 124a, 124b, etc., from the angiographic image frames 118. As noted, the ML model 602 can include multiple models. Thus, multiple ML models 808a can be trained as described above to identify (multiple) guide catheter positions.

[0107] The ML system 802 can be further used to train a model to infer key points in one frame of the angiography image frame 118 and key points associated with previous frames of the angiography image frame 118. Figure 8B An ML training environment 800b, an example of ML training environment 800a, is shown, configured to train an ML model 808b to infer keypoint 138b from angiographic image frame 118 and keypoint 138a. Thus, training data 806 may include angiographic image frame 118 and keypoints 138a, 138b, and the ML model 808b may be "trained" as described above to infer keypoint 138b from a frame in angiographic image frame 118 and keypoint 138a. The trained ML model 808b can generate keypoints 138a, 138b from a frame in angiographic image frame 118 and keypoints from previous frames.

[0108] The above description relates to a specific type of ML system 802 that applies supervised learning techniques given available training data with input / outcome pairs. However, the invention is not limited to use with a particular ML paradigm, and other types of ML techniques can be used. For example, in some embodiments, the ML system 802 may apply, for example, evolutionary algorithms or other types of ML algorithms and models to generate keypoints as described above.

[0109] Figure 9 A computer-readable storage medium 900 is illustrated. The computer-readable storage medium 900 may include any non-transitory computer-readable or machine-readable storage medium, such as optical, magnetic, or semiconductor storage media. In various embodiments, the computer-readable storage medium 900 may include an article of manufacture. In some embodiments, the computer-readable storage medium 900 may store computer-executable instructions 902 that a circuit system (e.g., processor 106, etc.) may utilize to perform. For example, the computer-executable instructions 902 may include instructions for implementing operations described with respect to the real-time co-registration system 100, which may improve the functionality of the real-time co-registration system 100 as detailed herein. For example, the computer-executable instructions 902 may include instructions that may cause a computing device to implement… Figure 2 Routine 200 Figure 3 Routine 300 Figure 4 Routine 400 Figure 5The instructions of routine 500. As another example, computer-executable instructions 902 may include instructions 116, ML model 808a, and / or ML model 808b. Examples of computer-readable storage medium 900 or machine-readable storage medium may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer-executable instructions 902 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc.

[0110] Figure 10 A combined internal and external imaging system 1000 is illustrated, comprising both an intravascular imaging system 1002 (e.g., an IVUS imaging system, etc.) and an extravascular imaging system 1004 (e.g., an angiography imaging system). The combined internal and external imaging system 1000 further includes a computing device 1006, which includes a circuit system, a controller and / or processor, memory, and software (as needed). In some embodiments, a live co-registration system 100 may be incorporated into the computing device 1006, or the live co-registration system 100 may contain the computing device 1006. Typically, the intravascular imaging system 1002 may be arranged to generate intravascular imaging data (e.g., IVUS images, etc.), while the extravascular imaging system 1004 may be arranged to generate extravascular imaging data (e.g., angiography images, etc.).

[0111] The extravascular imaging system 1004 may include a stage 1008, which may be arranged to provide sufficient space for positioning the C-arm 1010 of the angiography / fluorescence fluoroscopy unit in an operating position relative to the patient 1012 on the drive unit. The C-arm 1010 may be configured to acquire fluorescence fluoroscopy images in the absence of contrast agent in the blood vessels of the patient 1012 and / or to acquire angiography images in the presence of contrast agent in the blood vessels of the patient 1012.

[0112] Raw radiographic image data acquired by the C-arm 1010 can be transmitted via a transmission cable 1016 to an extravascular data input port 1014. The input port 1014 can be a separate component, integrated into or part of a computing device 1006. The input port 1014 may include a processor that converts the received raw radiographic image data into extravascular image data (e.g., angiography / fluorescence imaging data), for example, in the form of real-time video, DICOM, or a series of individual images. The extravascular image data can be initially stored in memory within the input port 1014 or in memory of the computing device 1006. If the input port 1014 is a separate component from the computing device 1006, the extravascular image data can be transmitted via the transmission cable 1016 to the computing device 1006 and to its input port (not shown). In some alternatives, communication between the devices or processors can be performed wirelessly instead of via cables as depicted.

[0113] Intravascular imaging data can be, for example, IVUS data or OCT data acquired by the intravascular imaging system 1002. The intravascular imaging system 1002 may include an intravascular imaging device, such as an imaging catheter 1020. The imaging catheter 1020 is configured to be inserted into the patient 1012 such that its distal end (including a diagnostic component or probe 1022 (e.g., an IVUS probe)) is near the desired imaging location of the vessel. Radiopaque material or markings 1024 located on or near the probe 1022 can provide an indication of the current position of the probe 1022 in the radiographic image. In some embodiments, the imaging catheter 1020 and / or probe 1022 may include a guiding catheter (not shown) that has been inserted into the lumen (e.g., a vessel, such as a coronary artery) of the subject via a guidewire (also not shown). However, in some embodiments, the imaging catheter 1020 and / or probe 1022 may be inserted into the vessel of the patient 1012 without a guidewire.

[0114] In some embodiments, the imaging catheter 1020 and / or probe 1022 may include both imaging capabilities and other data acquisition capabilities. For example, FFR and / or iFR data, data related to pressure, flow, temperature, electrical activity, oxygenation, biochemical composition, or any combination thereof. In some embodiments, the imaging catheter 1020 and / or probe 1022 may further include therapeutic devices such as stents, balloons (e.g., angioplasty balloons), grafts, filters, valves, and / or different types of therapeutic endovascular devices.

[0115] Imaging catheter 1020 is coupled to proximal connector 1026 to couple imaging catheter 1020 to image acquisition device 1028. Image acquisition device 1028 may be coupled to computing device 1006 via transmission cable 1016 or wireless connection. Intravascular image data may be initially stored in memory within image acquisition device 1028 or in memory of computing device 1006. If image acquisition device 1028 is a separate component from computing device 1006, intravascular image data may be transmitted to computing device 1006 via, for example, transmission cable 1016.

[0116] The computing device 1006 may also include one or more additional output ports for transmitting data to other devices. For example, the computer may include output ports for transmitting data to a data archive or memory device 1032. The computing device 1006 may also include a user interface (described in more detail below) that includes a combination of circuitry, processing components, and instructions executable by the processing components and / or circuitry to perform the image recognition and vessel orientation or path finding described herein and / or dynamic co-registration of intravascular and extravascular images using identified vascular pathways.

[0117] In some embodiments, the computing device 1006 may include a user interface device, such as a keyboard, mouse, joystick, touch screen device (such as a smartphone or tablet computer), touchpad, trackball, voice command interface, and / or other types of user interfaces known in the art.

[0118] A user interface can be rendered and displayed on a display 1034 coupled to computing device 1006 via display cable 1036. Although display 1034 is depicted as separate from computing device 1006, in some examples, display 1034 may be part of computing device 1006. Alternatively, display 1034 may be remote from computing device 1006 and wireless. As another example, display 1034 may be part of another computing device (such as a tablet computer) different from computing device 1006, which may be coupled to computing device 1006 via a wired or wireless connection. For some applications, display 1034 includes a head-up display and / or a head-mounted display. For some applications, computing device 1006 generates output on different types of visual, text, graphics, haptic, audio, and / or video output devices (e.g., speakers, headphones, smartphones, or tablet computers). For some applications, the user interface rendered on display 1034 acts as both an input and output device.

[0119] Figure 11A schematic representation of a machine 1100 in the form of a computer system is shown, within which an instruction set can be executed to cause the machine to perform any or more of the methods discussed herein. More specifically, Figure 11 A schematic representation of a machine 1100 in the form of an example computer system is shown, within which instructions 1108 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause the machine 1100 to perform any or more of the methods discussed herein. For example, instructions 1108 can cause the machine 1100 to execute instructions 116, Figure 2 Routine 200 Figure 3 Routine 300 Figure 4 Routine 400 Figure 5 Routine 500 Figure 8A or Figure 8B Training algorithms such as 816, etc. More generally, instruction 1108 enables machine 1100 to co-register IVUS images with angiographic images from movie playback in real time, as described herein.

[0120] Instruction 1108 transforms a general, unprogrammed machine 1100 into a specific machine 1100 programmed to perform the described and demonstrated functions in a particular manner. In alternative embodiments, machine 1100 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, machine 1100 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1100 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing instructions 1108 specifying actions to be taken by machine 1100. Furthermore, although only a single machine 1100 is shown, the term "machine" should also be considered as a collection of machines 200 that individually or jointly execute instructions 1108 to perform any or more of the methods discussed herein.

[0121] Machine 1100 may include processor 1102, memory 1104, and I / O components 1142, which may be configured to communicate with each other, for example, via bus 1144. In example embodiments, processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1106 and processor 1110 capable of executing instructions 1108. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 11 Multiple processors 1102 are shown, but machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0122] Memory 1104 may include main memory 1112, static memory 1114, and memory cell 1116, all of which are accessible to processor 1102, for example, via bus 1144. Main memory 1104, static memory 1114, and memory cell 1116 store instructions 1108 embodying any one or more of the methods or functions described herein. Instructions 1108 may also reside wholly or partially in main memory 1112, in static memory 1114, in machine-readable medium 1118 in memory cell 1116, in at least one of the processors 1102 (e.g., in the processor's cache memory), or any suitable combination thereof, during execution by machine 1100.

[0123] I / O component 1142 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement values, etc. The specific I / O component 1142 included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, while headless server machines may not include such touch input devices. It should be understood that I / O component 1142 may include... Figure 11Many other components are not shown. I / O components 1142 are grouped only by function to simplify the following discussion, and this grouping is by no means limiting. In various example embodiments, I / O components 1142 may include output components 1128 and input components 1130. Output components 1128 may include visual components (e.g., displays, such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 1130 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0124] In a further example embodiment, I / O component 1142 may include biometric identification component 1132, motion component 1134, environmental component 1136 or positioning component 1138, and a variety of other components. For example, biometric identification component 1132 may include components for detecting representations (e.g., hand representations, facial representations, voice representations, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1134 may include accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1136 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting hazardous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1138 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure to derive altitude), a direction sensor component (e.g., a magnetometer), etc.

[0125] A wide variety of technologies can be used to implement communication. I / O component 1142 may include communication components 1140 operable to couple machine 1100 to network 1120 or device 1122 via couplings 1124 and 1126, respectively. For example, communication component 1140 may include a network interface component or another suitable device for interfacing with network 1120. In further examples, communication component 1140 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth components, etc. ® Components (e.g., Bluetooth) ® Low power consumption, Wi-Fi ® Components, and other communication components for providing communication via other modes. Device 1122 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0126] Furthermore, the communication component 1140 can detect identifiers or include components operable for detecting identifiers. For example, the communication component 1140 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes (such as Universal Product Code (UPC) barcodes), multi-dimensional barcodes (such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCCRSS-2D barcodes), and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). Additionally, various information can be derived from the communication component 1140, such as location derived via Internet Protocol (IP) geolocation, location derived via Wi-Fi® signal triangulation, or location derived by detecting NFC beacon signals that indicate a specific location.

[0127] Various memories (i.e., memory 1104, main memory 1112, static memory 1114, and / or the memory of processor 1102) and / or storage units 1116 may store one or more sets of instructions and data structures (e.g., software) embodying any one or more methods or functions described herein or utilized by any one or more methods or functions. These instructions (e.g., instruction 1108) cause various operations to perform the disclosed embodiments when executed by processor 1102.

[0128] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” have the same meaning and are used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should be considered to include, but are not limited to, solid-state memory, and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, such as: semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which fall within the scope of the term “signal medium” discussed below.

[0129] In various example embodiments, one or more portions of network 1120 may be an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, the Internet, a portion of the Internet, a portion of the PSTN, a common legacy telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 1120 or a portion of network 1120 may include a wireless or cellular network, and coupling 1124 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling 1124 can implement any of the various types of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, GSM evolution enhanced data rate (EDGE) technology, the 3rd generation partnership program (3GPP) including 3G, fourth-generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), global microwave access interoperability (WiMAX), long-term evolution (LTE) standard, other standards defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0130] Instruction 1108 may be transmitted or received via network 1120 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1140) and utilizing any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 1108 may be transmitted or received via a transmission medium via coupling 1126 (e.g., peer-to-peer coupling) to device 1122. The terms “transmission medium” and “signal medium” have the same meaning and may be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be considered to include any intangible medium that can store, encode, or carry instruction 1108 for execution by machine 1100, and include digital or analog communication signals or other intangible media that facilitate communication of such software. Therefore, the terms “transmission medium” and “signal medium” should be considered to include any form of modulated data signal, carrier wave, etc. The term “modulated data signal” means a signal in which one or more characteristics of the signal are set or altered in such a way as to encode information in the signal.

[0131] The terms used in this document shall conform to their ordinary meaning in the relevant field or the meaning indicated by their use in the context, unless a specific definition is provided.

[0132] In this document, references to “one embodiment” or “an embodiment” do not necessarily refer to the same embodiment, but they may refer to the same embodiment. Unless the context explicitly requires otherwise, throughout the specification and claims, the words “comprise,” “comprising,” etc., should be interpreted in an open-ended rather than exclusive or exhaustive sense; that is, in the sense of “including but not limited to.” Use of singular or plural terms also includes both the plural and singular, unless explicitly limited to one or more. Furthermore, when used in this application, the words “this article,” “above,” “below,” and similar terms refer to the entire application and not any part thereof. When a claim uses the word “or” when referring to a list consisting of two or more items, the word covers all of the following interpretations: any item in the list, all items in the list, and any combination of items in the list, unless explicitly limited to one or the other. Any terms not explicitly defined herein shall be used in their conventional meaning as commonly understood by one of ordinary skill in the art.

Claims

1. A method for identifying collateral branches from an image, the method comprising: Receive multiple extravascular image frames associated with the patient's blood vessels at the computing device; The computing device identifies the location of key points in the first frame of the plurality of extravascular image frames; The computing device identifies the location of the key point in the second frame of the plurality of extravascular image frames in part based on the location of the key point in the first frame; as well as Multiple intravascular image frames are co-registered with the first frame, partly based on the location of the key points in the first frame; or The plurality of intravascular image frames are co-registered with the second frame in part based on the location of the key points in the second frame; or Multiple intravascular image frames are co-registered with the first frame in part based on the location of the key points in the first frame, and the multiple intravascular image frames are co-registered with the second frame in part based on the location of the key points in the second frame.

2. The method of claim 1, further comprising: The computing device generates a first graphic indication of the first frame co-registered with the plurality of intravascular image frames and a second graphic indication of the second frame co-registered with the plurality of intravascular image frames; as well as The first graphic indication and the second graphic indication are sent to a display device to display the co-registered multiple intravascular images in sync with cardiac motion associated with the multiple extravascular image frames.

3. The method as described in claim 1, wherein, The plurality of image frames are image frames captured during film playback during a fluorescence permeation procedure.

4. The method of claim 1, wherein, The plurality of intravascular image frames are intravascular ultrasound (IVUS) image frames.

5. The method according to any one of claims 1 to 4, further comprising: Additional extravascular image frames are received at the computing device; The location of the key point in the additional extravascular frame is identified by the computing device in part based on the location of the key point in the second frame; and The plurality of intravascular image frames are co-registered with the additional extravascular image frames, partly based on the location of the key points in the additional extravascular image frames.

6. The method of claim 5, wherein, Receiving the additional extravascular image frame includes receiving the additional extravascular image frame during the extravascular imaging procedure.

7. The method of claim 6, wherein, Identifying the location of the key points in the first frame includes: The computing device identifies the location of the conduit in the first frame; The computing device identifies the centerline of the blood vessel in the first frame, in part, based on the catheter position; and The computing device identifies the positions of the collateral branches of the blood vessel along the centerline.

8. The method of claim 6, wherein, Identifying the location of the catheter in the first frame includes: The location of the imaging catheter tip in the first frame is inferred by the computing device using a tip recognition machine learning (ML) model; or The location of the tip of the imaging catheter in the first frame is received from an input device coupled to the computing device at the computing device.

9. The method of claim 8, wherein, Identifying the catheter location in the first frame further includes inferring the location of the guide catheter's inlet point in the first frame using an inlet point identification ML model by the computing device.

10. The method of claim 9, wherein, Co-registering the plurality of intravascular image frames with the first frame, in part based on the location of the key points in the first frame, includes: The plurality of intravascular image frames are received at the computing device; The computing device identifies a subset of frames associated with collateral vessels from the plurality of intravascular image frames; The computing device maps the plurality of intravascular image frames onto the centerline in part based on a subset of the frames associated with the collateral vessels in the plurality of intravascular image frames; The computing device matches the collateral vessels associated with the first frame with the collateral vessels associated with the subset of frames of the plurality of intravascular image frames; and The computing device adjusts the position of the side branch associated with the subset of frames in part based on the matching.

11. The method of claim 9, wherein, Identifying the location of the key point in the second frame of the plurality of extravascular image frames, in part based on the location of the key point in the first frame, includes: The computing device tracks the position of the conduit between the first frame and the second frame; The computing device tracks the position of the collateral branches of the blood vessel between the first frame and the second frame; and The computing device identifies the centerline of the blood vessel in the second frame based in part on the position of the tip of the guiding catheter, the entry point of the guiding catheter, and the collateral branches.

12. The method of claim 8, wherein, Identifying the location of the key point in the first frame further includes: The computing device uses the tip recognition ML model to infer the position of the tip of the guiding catheter in each of the plurality of extravascular image frames, wherein a confidence value is output from the tip recognition ML model each time the position of the tip of the guiding catheter is inferred; The computing device selects the extravascular image frame associated with the highest confidence value from the plurality of extravascular image frames as the first frame.

13. The method of claim 8, wherein, Identifying the location of the key point in the first frame further includes: The computing device identifies the contrast of each extravascular image frame among the plurality of extravascular image frames; and The computing device selects an extravascular image frame with feasible image quality from the plurality of extravascular image frames as the first frame.

14. A computer-readable storage device comprising instructions executable by a processor of a computing device coupled to an intravascular imaging apparatus and a fluorescence microscope, wherein, When executed, the instructions cause the computing device to perform the method as described in any one of claims 1 to 13.

15. An apparatus comprising a processor arranged to be coupled to an intravascular imaging device and a fluorescence endoscope, the apparatus further comprising a memory including instructions, the processor being arranged to execute the instructions to implement the method as claimed in any one of claims 1 to 13.

Images