Alignment of multiple series of intravascular images

Abstract

The present disclosure provides for processing intravascular ultrasound (IVUS) images from different operations through a blood vessel to generate a mapping between frames of each IVUS operation, and generating a graphical user interface (GUI) to graphically present IVUS operations related to each other. In some examples, blood vessel fiducials are identified in frames in which each IVUS runs, and one or both run offset in time, distance, and / or angle to align the frames with the identified blood vessel fiducials. In addition, the present disclosure provides angularly aligning the viewing angles of the intravascular image with the external image of the blood vessel.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 502,859 filed on May 17, 2023 and U.S. Provisional Patent Application Serial No. 63 / 648,483 filed on May 16, 2024, the disclosures of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to intravascular ultrasound (IVUS) imaging systems. In particular, but not exclusively, the present disclosure relates to associating frames in a first series of IVUS images with frames in a second series of IVUS images. Background Art

[0003] Ultrasound devices that can be inserted into a patient's body have been shown to be able to diagnose a variety of diseases and conditions. For example, intravascular ultrasound (IVUS) imaging systems have been used as an imaging modality to diagnose blood vessel blockages and provide information to help medical professionals select and place stents and other devices to restore or increase blood flow.

[0004] The IVUS imaging system includes: a control module (with a pulse generator, image acquisition and processing components, and a monitor), a catheter, and a transducer located in the catheter. The catheter containing the transducer is located in a lumen or cavity that is located in or near the area to be imaged (e.g., a blood vessel wall, or patient tissue near a blood vessel wall). The pulse generator in the control module generates electrical pulses that are transmitted to the transducer and converted into acoustic pulses that are transmitted through the patient's tissue. The patient's tissue (or other structure) reflects the acoustic pulses, and the reflected pulses are absorbed by the transducer and converted into electrical pulses. The converted electrical pulses are transmitted to the image acquisition and processing components and converted into images that can be displayed on the monitor.

[0005] Typically, doctors take a series of IVUS images at different stages of treatment. However, conventional tools and systems do not allow doctors to compare these different series of IVUS images, but only provide a selected set of measurements obtained from the images. Therefore, there is a need to correlate IVUS images of the same vessel taken at different times and provide a graphical interface to display these images in relation to each other.

[0006] Machine learning (ML) is the study of computer algorithms that improve through experience. Typically, ML algorithms build a model based on sample data (called training data). The model can be used to make inferences (e.g., make predictions or decisions without being explicitly programmed). Understandably, the quality of the model's inferences depends on the training data. Therefore, a larger and more complete knowledge base needs to be available to train these ML models. Summary of the invention

[0007] This summary is intended to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0008] In general, the present disclosure is used to process raw IVUS images, automatically detect lumen and vessel boundaries, and identify regions of interest, or more specifically, identify start and end points between which frames of interest in a series of IVUS images are included.

[0009] For some embodiments, the present disclosure may be implemented as a method. The method may include receiving a first series of intravascular ultrasound (IVUS) images of a patient's blood vessel at a processor, the first series of IVUS images including a first plurality of frames; receiving a second series of intravascular ultrasound (IVUS) images of the patient's blood vessel at the processor, the second series of IVUS images including a second plurality of frames; determining, by the processor, a mapping between the first plurality of frames and the second plurality of frames; and generating, by the processor, a graphical user interface (GUI), the GUI including an indication of the first series of IVUS images and the second series of IVUS images based on the mapping between the first plurality of frames and the second plurality of frames.

[0010] In a further embodiment of the method, determining a mapping between the first plurality of frames and the second plurality of frames may include: identifying a frame of the first plurality of frames that includes a specific vascular fiducial; and identifying a frame of the second plurality of frames that includes the specific vascular fiducial.

[0011] In a further embodiment, the method may include determining an offset between the one of the first plurality of frames and the one of the second plurality of frames.

[0012] In a further embodiment of the method, the GUI comprises an indication of the first series of IVUS images and the second series of IVUS images aligned based on the offset.

[0013] In a further embodiment of the method, the GUI comprises on-axis views of the first series of IVUS images and the second series of IVUS images, and wherein the on-axis views are associated with each other based on the offset.

[0014] In a further embodiment of the method, the GUI comprises longitudinal views of the first series of IVUS images and the second series of IVUS images, wherein the longitudinal views are associated with each other based on an offset.

[0015] In a further embodiment, the method may include executing a machine learning (ML) model to infer a frame in a first plurality of frames that contains a specific vascular reference based on the first series of IVUS images; and executing the ML model to infer a frame in a second plurality of frames that contains the specific vascular reference based on the second series of IVUS images.

[0016] In a further embodiment of the method, the specific vessel fiducial is one of: lumen geometry, vessel geometry, side branch location, calcium morphology, plaque distribution, or guide catheter location.

[0017] In a further embodiment, the method may include receiving a second series of IVUS images from an intravascular imaging device; and receiving the first series of IVUS images from a memory storage device.

[0018] In a further embodiment of the method, the first series of IVUS images are captured during a pre-percutaneous coronary intervention (PCI) procedure.

[0019] In a further embodiment of the method, the second series of IVUS images are captured during a PCI procedure or during a post-PCI procedure.

[0020] In a further embodiment of the method, determining a mapping between the first plurality of frames and the second plurality of frames may include executing a machine learning (ML) model to infer the mapping based on the first series of IVUS images and the second series of IVUS images.

[0021] In a further embodiment of the method, the mapping comprises an indication of an offset between a frame of the first plurality of frames and a frame of the second plurality of frames.

[0022] For certain embodiments, the present disclosure may be implemented as a device comprising a processor coupled to a memory, the memory comprising instructions executable by the processor, the processor being configured to be coupled to an intravascular ultrasound (IVUS) imaging system and configured to execute the instructions, which, when executed, causes the processor to implement the method of any embodiment described herein.

[0023] For some embodiments, the present disclosure may be implemented as at least one machine-readable storage device comprising a plurality of instructions that, in response to being executed by a processor of an intravascular ultrasound (IVUS) imaging system, cause the processor to implement the method of any embodiment described herein.

[0024] For some embodiments, the present disclosure may be implemented as an apparatus for an intravascular imaging system. The apparatus may include: a processor; and a memory device coupled to the processor, the memory device including instructions executable by the processor, which, when executed by the processor, causes the intravascular imaging system to receive a first series of intravascular ultrasound (IVUS) images of a patient's blood vessel, the first series of IVUS images including a first plurality of frames; receive a second series of intravascular ultrasound (IVUS) images of the patient's blood vessel, the second series of IVUS images including a second plurality of frames; determine a mapping between the first plurality of frames and the second plurality of frames; and generate a graphical user interface (GUI) including an indication of the first series of IVUS images and the second series of IVUS images based on the mapping between the first plurality of frames and the second plurality of frames.

[0025] In certain embodiments of the device, when the processor executes the instructions, it also causes the intravascular imaging system to: identify a frame in the first plurality of frames that includes a specific vascular reference and identify a frame in the second plurality of frames that includes a specific vascular reference, wherein the specific vascular reference is one of the following: lumen geometry, vascular geometry, side branch location, calcium morphology, plaque distribution, or guide catheter location.

[0026] In certain embodiments of the device, when the processor executes the instructions, the instructions further cause the intravascular imaging system to: determine an offset between a frame in the first plurality of frames and a frame in the second plurality of frames, wherein the GUI includes an indication of the first series of IVUS images and the second series of IVUS images aligned based on the offset, wherein the GUI includes axial views of the first series of IVUS images and the second series of IVUS images, wherein the axial views are associated with each other based on the offset, or wherein the GUI includes longitudinal views of the first series of IVUS images and the second series of IVUS images, wherein the longitudinal views are associated with each other based on the offset.

[0027] In certain embodiments of the device, when the processor executes the instructions, it also causes the intravascular imaging system to: execute a machine learning (ML) model to infer a frame in a first plurality of frames that contains a specific vascular reference based on the first series of IVUS images; and execute the ML model to infer a frame in a second plurality of frames that contains a specific vascular reference based on the second series of IVUS images.

[0028] In some embodiments, the apparatus may include an intravascular imaging device, wherein at least one of the first series of IVUS images or the second series of IVUS images is received from the intravascular imaging device.

[0029] For some embodiments, the present disclosure may be implemented as a machine-readable storage device. The machine-readable storage device may include a plurality of instructions that, in response to being executed by a processor of an intravascular ultrasound (IVUS) imaging system, cause the processor to: receive a first series of intravascular ultrasound (IVUS) images of a patient's blood vessel, the first series of IVUS images including a first plurality of frames; receive a second series of intravascular ultrasound (IVUS) images of the patient's blood vessel, the second series of IVUS images including a second plurality of frames; determine a mapping between the first plurality of frames and the second plurality of frames; and generate a graphical user interface (GUI) including an indication of the first series of IVUS images and the second series of IVUS images based on the mapping between the first plurality of frames and the second plurality of frames.

[0030] In some embodiments of the machine-readable storage device, when the processor executes the instructions, it also causes the IVUS imaging system to: identify a frame in the first plurality of frames that includes a specific vascular reference and identify a frame in the second plurality of frames that includes a specific vascular reference, wherein the specific vascular reference is one of lumen geometry, vascular geometry, side branch location, calcium morphology, plaque distribution, or guide catheter location. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To easily identify the discussion of any element or action, the most significant digit or digits in a reference number refer to the reference numeral that first introduces that element.

[0032] Figure 1 An IVUS imaging system according to an embodiment of the present disclosure is shown.

[0033] Figure 2 An example angiographic image of a blood vessel is shown.

[0034] Figure 3A and Figure 3B An IVUS image of a blood vessel is shown.

[0035] Figure 4 An IVUS image correlation and visualization system in accordance with at least one embodiment of the present disclosure is shown.

[0036] Figure 5A An example frame-by-frame correlation between frames in one set of IVU images and frames in another set of IVUS images is shown in accordance with at least one embodiment of the present disclosure.

[0037] Figure 5B It shows that according to Figure 5A Plot of correlation scores generated by frame-by-frame correlation of the examples in .

[0038] Fig. 6AAnother example frame-by-frame association between frames in one set of IVU images and frames and angle-shifted versions of the frames in another set of IVUS images is shown in accordance with at least one embodiment of the present disclosure.

[0039] Figure 6B It shows that according to Fig. 6A Plot of correlation scores generated by frame-by-frame correlation of the examples in .

[0040] Figure 7 Another IVUS image correlation and visualization system in accordance with at least one embodiment of the present disclosure is shown.

[0041] Fig. 8A and 8B Several series of IVUS images of blood vessels aligned according to at least one embodiment of the present disclosure are shown.

[0042] Fig.9A and Fig. 9B Example segment-by-segment alignment and offset identification in accordance with at least one embodiment of the present disclosure are shown.

[0043] Fig. 10A and Fig. 10B Another example segment-by-segment alignment and offset identification in accordance with at least one embodiment of the present disclosure is shown.

[0044] Fig.11 A graphical user interface (GUI) is shown in accordance with at least one embodiment of the present disclosure.

[0045] Fig.12 A logical flow is shown for determining a mapping between different IVUS runs of a vessel in accordance with at least one embodiment of the present disclosure.

[0046] Fig.13 Temporal warping based on the longitudinal shift of extracted and vectorized features of a series of IVUS images along two series of IVUS images is shown in accordance with at least one embodiment of the present disclosure.

[0047] Fig.14 An exemplary machine learning (ML) system suitable for use with exemplary embodiments of the present disclosure is shown.

[0048] Fig.15 Another IVUS image correlation and visualization system in accordance with at least one embodiment of the present disclosure is shown.

[0049] Fig.16A Example extravascular images and identified fiducials are shown.

[0050] Fig. 16B , Fig. 16C and Fig.16DExamples of frames from an IVUS run are shown, which are rotated to match the Fig.16A The angles of the fiducials observed in the external image are aligned.

[0051] Fig.17 A GUI in accordance with at least one embodiment of the present disclosure is shown.

[0052] Fig.18 A computer-readable storage medium is shown.

[0053] Fig.19 A schematic diagram of the machine is shown. DETAILED DESCRIPTION

[0054] The features and technical advantages of the present disclosure have been broadly summarized above so that the following detailed description of the present disclosure may be better understood. It will be appreciated by those skilled in the art that the disclosed embodiments may be easily used as a basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Considered in conjunction with the accompanying drawings, the novel features of the present disclosure (including its organization and operation) and further objects and advantages will be better understood from the following description. However, it should be expressly understood that each of the accompanying drawings is provided for illustration and description and is not intended to define limitations of the present disclosure.

[0055] As noted above, the present disclosure relates to IVUS images of patients and lumens (eg, blood vessels) and processing IVUS recordings, or in other words, processing a series of IVUS images. Thus, an example IVUS imaging system, a patient blood vessel, and a series of IVUS images are described.

[0056] Suitable IVUS imaging systems include, but are not limited to, one or more transducers disposed at the distal end of a catheter configured and arranged for percutaneous insertion into a patient.

[0057] Figure 1 One embodiment of an IVUS imaging system 100 is schematically illustrated. The IVUS imaging system 100 includes a catheter 102 that can be coupled to a control system 104. The control system 104 can include, for example, a processor 106, a pulse generator 108, and a drive unit 110. In at least some embodiments, the pulse generator 108 forms electrical pulses that can be input to one or more transducers (not shown) disposed in the catheter 102.

[0058] In some embodiments, mechanical energy from the drive unit 110 can be used to drive an imaging core (also not shown) disposed in the catheter 102. In at least some embodiments, electrical signals transmitted from one or more transducers can be input to the processor 106 for processing. In at least some embodiments, the processed electrical signals from the one or more transducers can be used to form a series of images, as described in more detail below. For example, a scan converter can be used to map scan line samples (e.g., radial scan line samples, etc.) to a two-dimensional Cartesian grid, which can be used as the basis for a series of IVUS images that can be displayed to a user.

[0059] In at least some embodiments, the processor 106 may also be used to control the functions of one or more other components of the control system 104. For example, the processor 106 may be used to control at least one of the frequency or duration of the electrical pulses sent from the pulse generator 108, and the rotation rate of the imaging core by the drive unit 110. In addition, when the IVUS imaging system 100 is configured for automatic pullback, the drive unit 110 may control the speed and / or length of the pullback.

[0060] Figure 2 An extravascular image 200 of a patient's blood vessel 202 is shown. As described, an IVUS imaging system (e.g., IVUS imaging system 100, etc.) is used to capture a series of intraluminal images or "records" or blood vessels (such as blood vessel 202). For example, an IVUS catheter (e.g., catheter 102) is inserted into blood vessel 202, and the above-mentioned record or series of IVUS images is captured when catheter 102 is pulled back from distal end 204 to proximal end 206. Catheter 102 can be pulled back manually or automatically (e.g., under the control of drive unit 110, etc.). The series of IVUS images captured between distal end 204 and proximal end 206 are generally referred to as images from an IVUS run.

[0061] Figure 3A and Figure 3B A two-dimensional (2D) representation of an IVUS image of a blood vessel 202 is shown. For example, Figure 3A An IVUS image 300a is shown, depicting an IVUS recorded longitudinal view of the blood vessel 202 between the proximal end 206 and the distal end 204 .

[0062] Figure 3BAn image frame 300b is shown depicting an on-axis (or short-axis) view of the blood vessel 202 at point 302. In other words, the image frame 300b is a single frame or single image from a series of IVUS images that can be captured between the distal end 204 and the proximal end 206 described herein. As described above, physicians typically capture IVUS runs (e.g., a series of IVUS images) at different stages of treatment. For example, IVUS images can be captured before a percutaneous coronary intervention (PCI) treatment and after a PCI treatment (e.g., stent placement, balloon dilatation, rotational ablation, etc.).

[0063] The present disclosure provides for aligning IVUS runs from different time frames on a frame-by-frame basis, and provides a graphical user interface that correlates IVUS runs to each other to allow physicians to view correlated IVUS runs and thereby gain a more direct understanding of their treatment of a vessel, for example, by observing differences in lesion characteristics through side-by-side comparisons.

[0064] Figure 4 An IVUS image correlation and visualization system 400 according to some embodiments of the present disclosure is shown. Generally, the IVUS image correlation and visualization system 400 is a system for processing, correlating, and presenting multiple series of IVUS images of the same blood vessel. The IVUS image correlation and visualization system 400 can be implemented in a commercial IVUS guidance or navigation system, for example, available from Obtained Guidance System. The present disclosure provides advantages over existing or conventional IVUS navigation systems because conventional systems do not provide the ability to correlate IVUS runs performed at different times.

[0065] For some embodiments, the IVUS image correlation and visualization system 400 can be implemented as part of the control system 104. Alternatively, the control system 104 can be implemented as part of the IVUS image correlation and visualization system 400. As shown, the IVUS image correlation and visualization system 400 includes a computing device 402. Optionally, the IVUS image correlation and visualization system 400 includes the IVUS imaging system 100 and a display 404.

[0066] Notably, while the present disclosure often uses IVUS as an exemplary intravascular imaging modality, the present disclosure may be provided for longitudinally and / or angularly aligning frames from different runs captured using various other intravascular imaging modalities, such as optical coherence tomography (OCT).

[0067] The computing device 402 can be any of a variety of computing devices. In some embodiments, the computing device 402 can be incorporated into and / or implemented by a console of the display 404. In some embodiments, the computing device 402 can be a workstation or server communicatively coupled to the IVUS imaging system 100 and / or the display 404. In other embodiments, the computing device 402 can be provided by a cloud-based computing device, for example, by a computing-as-a-service system accessible via a network (e.g., the Internet, an intranet, a wide area network, etc.). The computing device 402 may include a processor 406, a memory 408, an input and / or output (I / O) device 410, a network interface 412, and an IVUS imaging system acquisition circuit 414.

[0068] Processor 406 may include circuitry or processor logic, such as any of a variety of commercially available processors. In some examples, processor 406 may include multiple processors, a multi-threaded processor, a multi-core processor (whether the multiple cores coexist on the same chip or on separate chips), and / or some other type of multi-processor architecture (through which multiple physically separate processors are linked together in some way). In addition, in some examples, processor 406 may include a graphics processing portion, and may include dedicated memory, multi-threaded processing, and / or some other parallel processing capability. In some examples, processor 406 may be an application specific integrated circuit (ASIC) or a field programmable integrated circuit (FPGA).

[0069] The memory 408 may include logic circuits, a portion of which includes an integrated circuit array, forming a non-volatile memory to persistently store data, or a combination of non-volatile memory and volatile memory. It should be understood that the memory 408 may be based on any of a variety of technologies. Specifically, the integrated circuit array included in the memory 120 may be arranged to form one or more types of memory, such as dynamic random access memory (DRAM), NAND memory, NOR memory, etc.

[0070] I / O device 410 may be various devices for receiving input and / or providing output. For example, I / O device 410 may include a keyboard, a mouse, a joystick, a foot pedal, a display, a touch-enabled display, a tactile feedback device, an LED, etc.

[0071] The network interface 412 may include logic circuits and / or features to support a communication interface. For example, the network interface 412 may include one or more interfaces that operate according to various communication protocols or standards to communicate through direct communication or a network communication link. Direct communication may be performed using a communication protocol or standard described in one or more industry standards (including progeny and variants). For example, the network interface 412 may facilitate communication through a bus, such as a peripheral component interconnect express bus (PCIe), a non-volatile memory express bus (NVMe), a universal serial bus (USB), a system management bus (SMBus), a SAS (e.g., a serially connected small computer system interface (SCSI)) interface, a serial AT connection (SATA) interface, etc. In addition, the network interface 412 may include logic circuits and / or features to allow communication through various wired or wireless network standards (e.g., 902.11 communication standards). For example, the network interface 412 may be arranged to support wired communication protocols or standards, such as Ethernet, etc. As another example, the network interface 412 may be arranged to support wireless communication protocols or standards, such as Wi-Fi, Bluetooth, ZigBee, LTE, 5G, etc.

[0072] The IVUS imaging system acquisition circuit 414 may include a circuit comprising a custom manufactured or specially programmed circuit configured to receive or receive and send signals between the IVUS imaging system 100, the signals including an indication of IVUS operation, a series of IVUS images, or one or more frames of IVUS images.

[0073] The memory 408 may include instructions 416. During operation, the processor 406 may execute the instructions 416 to cause the computing device 402 to receive a series of IVUS images from multiple IVUS runs of a vessel (e.g., from the IVUS imaging system 100, etc.) and store the records in the memory 408 as IVUS images 418a, IVUS images 418b, etc. For example, the processor 406 may execute the instructions 416 to receive an information element from the IVUS imaging system 100, the information element including an indication of IVUS images captured when the catheter 102 was pulled back from the distal end 204 to the proximal end 206, the images including an indication of the anatomy and / or structure of the vessel 202 (the vessel 202 including the vessel wall and plaque). In addition, it should be understood that the processor 406 may execute the instructions 416 to receive IVUS images from multiple runs traversing the vessel (e.g., before PCI, after PCI, at different times, etc.). It should be appreciated that the IVUS images 418a and 418b may be stored in a variety of image formats or even non-image formats or data structures that include an indication of the blood vessel 202. Furthermore, the IVUS images 418a and 418b include several "frames" or individual images that, when represented collinearly, may be used to form an image of the blood vessel 202, for example, as represented by the IVUS image 300a.

[0074] The present disclosure provides for associating IVUS images 418a and IVUS images 418b on a frame-by-frame basis and presenting an associated view of the images in a graphical user interface. In some examples, the processor 406 may execute instructions 416 to identify an IVUS run frame mapping 420 from the IVUS images 418a and the IVUS images 418b using a machine learning (ML) model to infer a mapping (e.g., see Figure 4 In some embodiments, the processor 406 may execute instructions 416 to identify an IVUS run frame map 420 (eg, see FIG. 4 ) from the IVUS image 418a and the IVUS image 418b using a frame-by-frame correlation or a segment-by-segment correlation. Figure 4 ). In some embodiments, this can be accomplished using standard image processing techniques and / or ML reasoning. In other examples, memory 408 can execute instructions 416 to determine one or more fiducials (e.g., through machine learning, through image processing algorithms, etc.), and determine IVUS run frame maps 420 from the identified fiducials (e.g., see Figure 7 ). Although each method of determining the IVUS run frame map 420 is slightly different and discussed separately, once the IVUS run frame map 420 is identified, the methods are similar. In addition, in some embodiments, the IVUS run can be mapped and / or aligned with the angiographic image of the blood vessel (e.g., see FIG. 6 ). Notably, Figure 4 , Figure 7 and Fig.15 IVUS image correlation and visualization systems 400, 700, and 1500 are depicted, respectively. This is done to clearly describe the various alignment techniques disclosed herein. However, it should be noted that the alignment techniques described for one system (e.g., 400) can be used with alignment techniques disclosed for another system (e.g., 700 and / or 1500). For example, it is understood that one alignment technique can be used to align frames longitudinally, while another technique can be used to align frames angularly.

[0075] Now go to Figure 4 . Once the IVUS run frame map 420 is generated, a frame-by-frame association between the IVUS image 418a and the IVUS image 418b can be generated based on the IVUS run frame map 420. For example, the memory 408 can execute instructions 416 to associate each frame of the IVUS image 418a with a corresponding frame of the IVUS image 418b. In addition, the memory 408 can execute instructions 416 to generate a graphical user interface (GUI) 424 that depicts an indication of the frames of the IVUS image 418a, wherein the frames of the IVUS image 418a are associated and / or correlated with the corresponding frames of the IVUS image 418b based on the IVUS run frame map 420.

[0076] In an example of using ML to generate an IVUS operating frame map 420, the processor 406 can execute instructions 416 to execute or "run" an ML model 422, with IVUS images 418a and IVUS images 418b as input to generate an IVUS operating frame map 420. The ML model 422 can infer the IVUS operating frame map 420 from the IVUS images 418a and IVUS images 418b. The memory 408 can store a copy of the ML model 422, and the processor 406 can execute the ML model 422 to generate the IVUS operating frame map 420. In general, the ML model 422 can be any of a variety of ML models. Examples of ML models contemplated herein and even examples of training ML models are provided below.

[0077] In some embodiments, the present disclosure may be used to align IVUS runs based on the association between each frame of one IVUS run and all frames of another IVUS run. Processor 406 may execute instructions 416 to determine frame-by-frame associations 426 of the IVUS runs. For example, processor 406 may execute instructions 416 to iterate over each frame of IVUS image 418a and calculate the association between all frames of IVUS image 418b (e.g., using a benchmark, using ML, using background subtraction, using cross-correlation, etc.). Subsequently, processor 406 may execute instructions 416 to identify the most closely associated frame in IVUS image 418b for each frame in IVUS image 418a. For example, Figure 5A Image frame 502 (e.g., from IVUS image 418a, etc.) and image frames 504a, 504b, 504c, etc. (e.g., from IVUS image 418b, etc.) are depicted. Processor 406 may execute instructions 416 to calculate a correlation (e.g., a correlation value, a score, etc.) between image frame 502 and image frames 504a, 504b, 504c, etc. Figure 5B A graph 506 of the calculated correlation is shown. Graph 506 plots the correlation score of a particular frame from one set of IVUS images (e.g., image frame 502) and frames from another set of IVUS images (e.g., frames 504a, 504b, 504c, etc.). As can be seen, the value of the correlation score is plotted on the y-axis 508, while the frame number from the second set of IVUS images is plotted on the x-axis 510.

[0078] For some embodiments, frame-by-frame associations may be determined for each frame at different rotation angles. Processor 406 may execute instructions 416 to identify, for each frame in a set of IVUS images (e.g., IVUS image 418a, etc.), associations with frames from another set of IVUS images (e.g., IVUS image 418b, etc.) at several rotation angles. Fig. 6A An example of this is shown. During operation of the IVUS image correlation and visualization system 400, the processor 406 can execute instructions 416 to calculate an association score between a frame from one set of IVUS images (e.g., image frame 502 from IVUS image 418a, etc.) and a frame from another set of IVUS images (e.g., image frame 504a from IVUS image 418b, etc.) and a rotated version of the image frame (e.g., rotating the image frames 602a and 602b). As with the frame-by-frame association described above, the processor 406 can execute instructions 416 to calculate an association score between each frame from one set of IVUS images (e.g., IVUS image 418a) and each frame and rotated version from another set of IVUS images (e.g., IVUS image 418b, etc.). Figure 6BA graph 604 of the calculated correlation is shown. Graph 604 plots the correlation score of a particular image frame from one set of IVUS images (e.g., image frame 502) and an image frame from another set of IVU images (e.g., image frame 504a) and a rotated version of the image frame (e.g., rotated image frames 602a and 602b). As can be seen, the value of the correlation score is plotted on the y-axis 606, while the rotation angle is plotted on the x-axis 608.

[0079] In some examples, the processor 406 may execute instructions 416 to generate rotated image frames (e.g., rotated image frames 602a, 602b, etc.) for all possible rotation angles. In such an example, 359 rotated image frames will be generated. In other examples, the processor 406 may execute instructions 416 to generate rotated image frames at a subset of all possible rotation angles (e.g., every 2 degrees, every 5 degrees, every 10 degrees, every 15 degrees, every 20 degrees, every 30 degrees, every 45 degrees, etc.).

[0080] Typically, the IVUS run frame map 420 may include an indication of an offset (e.g., time, distance, rotation, etc.) with which one (or each) of the IVUS images 418a and 418b is adjusted so that they are aligned. As used herein, the term alignment refers to aligning the frames of the images longitudinally and / or angularly.

[0081] In some embodiments, the processor 406 may execute instructions 416 to receive a bookmark (or multiple bookmarks) for identifying a frame of one of the IVUS images 418a and / or IVUS images 418b. The IVUS run frame map 420 may be adjusted to align the bookmarks. In some embodiments, the mapping is not linear. For example, a frame in the IVUS image 418a may be linearly adjusted (e.g., a first distance) and / or rotated (e.g., a first angle) based on its association with a frame in the IVUS image 418b, while an adjacent frame in the IVUS image 418a may be linearly adjusted (e.g., a second distance) and / or rotated (e.g., a second angle) based on its association with the same frame or a different frame from the IVUS image 418b.

[0082] Figure 7 An IVUS image correlation and visualization system 700 is shown according to some embodiments of the present disclosure. In general, the IVUS image correlation and visualization system 700 is a system for processing, correlating, and presenting multiple series of IVUS images of the same blood vessel, similar to the IVUS image correlation and visualization system 400. To simplify the discussion, many components of the IVUS image correlation and visualization system 400 are referenced and reused in the IVUS image correlation and visualization system 700.

[0083] As mentioned above about Figure 4As described with the IVUS image association and visualization system 400, the present disclosure provides for generating an IVUS running frame map 420 for the IVUS image 418a and the IVUS image 418b. In some embodiments, the processor 406 may execute instructions 416 to identify a vascular fiducial 702 in the IVUS image 418a and the IVUS image 418b. In some embodiments, the vascular fiducial 702 may be any one or more coronary artery anatomical fiducials (e.g., lumen geometry, vessel geometry, side branch location, calcium morphology, plaque distribution, guide catheter location, etc.). In some embodiments, the processor 406 executes instructions 416 to identify the vascular fiducial 702 from the IVUS image 418a and the IVUS image 418b using an image processing algorithm (e.g., a geometric image recognition algorithm for identifying lumen contours, etc.). In other embodiments, the memory 408 may include one or more ML models 704 configured to infer the vascular fiducial 702 from the IVUS image (e.g., IVUS images 418a, 418b, etc.). For example, the memory 408 may include the ML models 704, wherein the ML models 704 may include one or more ML models trained to infer fiducials, such as side branch locations, calcium morphology, guide catheter locations, etc. Thus, the processor 406 may execute the ML models 704 to identify the vascular fiducials 702 in the IVUS image 418a and IVUS image 418b frames.

[0084] The processor 406 can execute instructions 416 to generate an IVUS running frame map 420 based on the vascular fiducial 702, for example, by pairing a frame from the IVUS image 418a with a frame from the IVUS image 418b in which the same anatomical fiducial is identified. Given the IVUS running frame map 420, the processor 406 can execute instructions 416 to associate each frame of the IVUS image 418a with a corresponding frame of the IVUS image 418b. In addition, the processor 406 can execute instructions 416 to generate a GUI 424 based on the IVUS running frame map 420, which depicts an indication of the frame of the IVUS image 418a associated with and / or related to the corresponding frame of the IVUS image 418b.

[0085] It should be understood that in some embodiments, the processor 406 can execute instructions 416 and identify the reference in a single frame of each IVUS run (e.g., IVUS images 418a and 418b). For example, the vascular reference 702 may include a side branch identified in a frame of IVUS image 418a and the same side branch identified in a frame of IVUS image 418b. In other embodiments, the processor 406 can execute instructions 416 and identify the reference in multiple frames. In such examples, the reference does not have to be the same. For example, as described above, the vascular reference 702 may include: a side branch position in a frame of IVUS image 418a and the same side branch position in a frame of IVUS image 418b, as well as a guide catheter position in another frame of IVUS image 418a and a guide catheter position in another frame of IVUS image 418b. The examples are not limited to this context.

[0086] Fig. 8A Several IVUS runs are shown for a scale 802 setting. The figure depicts an IVUS run or multiple sets of IVUS images 804a, 804b, and 804c. It should be understood that each set of IVUS images (e.g., multiple sets of IVUS images 804a, 804b, and 804c) includes multiple frames. As described above, for some embodiments, a reference is identified in one or more frames of each set of IVUS images. The figure depicts a reference 806 identified in a frame of each set of IVUS images 804a, 804b, and 804c. An IVUS run frame map 420 may be generated based on a frame identified as indicating (representing, corresponding to, depicting, etc.) a reference 806. For example, the processor 406 may execute instructions 416 to identify a frame containing a reference 806 (or a vascular reference) of a vessel from each set of IVUS images (e.g., a frame from image 804a, image 804b, image 804c, etc.). Processor 406 can execute instructions 416 to identify an offset of frames of a group (or multiple groups) of IVUS images, and when the offset is applied, the frames in each group of IVUS images will be aligned on the scale 802. In some embodiments, the offset can be a time offset, a distance offset, an angle offset, or any combination of time, distance and / or angle offsets. In addition, it should be understood that scale 802 can be any scale that represents or graphically presents an IVUS operation. For example, some IVUS operations are graphically represented as corresponding to a pullback scale, where the pullback scale has a distal end point and a proximal end point along the pullback. For example, the pullback scale can be represented in distance units (e.g., millimeters, etc.). In the context of the offset, an offset can be generated so that when the offset is applied, these frames that are identified as indicating the same reference (e.g., vascular reference 806) are moved or adjusted so that these frames are aligned on the scale.

[0087] For example, Figure 8B Shows Fig. 8A802. However, frames from the sets of IVUS images 804a and 804c have been adjusted based on the identified offsets (e.g., from the IVUS run frame map 420, etc.) to align the frames indicating the fiducials on the scale 802. For example, IVUS image 804a is adjusted by offset 808a to move IVUS image 804a relative to the scale 802, and IVUS image 804c is adjusted by offset 808b to move IVUS image 804c relative to the scale 802. Applying offsets 808a and 808b to IVUS images 804a and 804c, respectively, aligns the IVUS images with the scale 802, and in particular aligns the frames indicating the vessel fiducials 806 in IVUS images 804a, 804b, and 804c with the scale 802. For example, as shown in the figure, when IVUS images 804a, 804b, and 804c are adjusted based on offsets 808a and 808b, the fiducials 806 identified in each IVUS run are aligned. Note that offsets 808a and 808b are depicted as longitudinal offsets, or more specifically, the distance along scale 802 that the frames are offset. However, offsets 808a and 808b can be offset angles instead (e.g., the angle of rotating the frames), or can be offset distances and offset angles. In addition, note that while each IVUS run depicts only a single offset (e.g., offset 808a for IVUS image 804a and offset 808b for IVUS image 804c), multiple offsets for each run can be provided herein (e.g., for frames in a segment, for each frame, for only some frames, etc.).

[0088] A variety of techniques and workflows are provided to identify longitudinal and / or angular offsets of frames in one set of IVUS images (e.g., IVUS image 418a, etc.) in order to align the frames with frames in another set of IVUS images (e.g., IVUS image 418b, etc.). Fig. 8A and Figure 8B Only longitudinal alignment is depicted, but the present disclosure may be implemented to align IVUS operations longitudinally, angularly, and / or both longitudinally and angularly.

[0089] In some embodiments, processor 406 may execute instructions 416 to longitudinally align frames from IVUS image 418a with frames from IVUS image 418b on a segment-by-segment basis. For example, in some embodiments, processor 406 may execute instructions 416 to identify segments based on vessel fiducials 702. Fig.9AIVUS image 418a and identified fiducials 902a and 902b are shown. As described above, these fiducials can be side branches, lumen geometry, vessel geometry, calcium morphology, plaque distribution, etc. Processor 406 can execute instructions 416 to group frames from IVUS image 418a into segments based on identified fiducials 902a and 902b. For example, Fig.9A Frames from IVUS image 418a are shown grouped into segments 904a, 904b, and 904c. Thus, offsets of frames in a set of IVUS images (eg, IVUS image 418a, etc.) may be generated for different segments using the identified vascular fiducials 702.

[0090] Fig. 9B Points representing each longitudinal offset of frames corresponding to references 902a and 902b are shown. From these points, a graph 906 representing the longitudinal offset can be generated, with the longitudinal offset being plotted on a y-axis 908, while each frame in IVUS run 418a is plotted on an x-axis 910. In some embodiments, graph 906 can be generated linearly between these points (e.g., as Fig. 9B In other embodiments, the processor 406 may execute instructions 416 to generate the graph 906 based on one or more line fitting algorithms (e.g., raster-based line fitting, etc.). The longitudinal offset of the frames in each segment 904a, 904b, and 904c may be determined based on the graph 906.

[0091] In some embodiments, processor 406 may execute instructions 416 to rotationally align frames in IVUS image 418a with frames in IVUS image 418b based on vessel fiducial 702. For example, in some embodiments, IVUS run frame map 420 may include an offset angle (e.g., an offset angle for rotating frames). Fig. 10A IVUS image 418a and identified fiducials 1002a, 1002b, and 1002c are shown. As described above, these fiducials can be side branches, lumen geometry, vessel geometry, calcium morphology, plaque distribution, etc. As described above, frames in IVUS image 418a corresponding to fiducials 1002a, 1002b, and 1002c can be mapped to specific frames in IVUS image 418b (e.g., based on vessel fiducials 702, etc.), and the offset angle between the frames can be determined. In another embodiment, based on calculating an association with each frame and a rotated version of each frame (e.g., as described above with respect to Fig. 6A and Figure 6B As described above), the offset angle can be determined.

[0092] Fig. 10BPoints representing each offset angle for frames corresponding to references 1002a, 1002b, and 1002c are shown. From these points, a graph 1004 representing the offset angles can be generated, with the offset angles plotted on the y-axis 1006 and each frame in IVUS run 418a plotted on the x-axis 1008. As described above, 1004 can be generated linearly and / or based on one or more line fitting or line smoothing algorithms.

[0093] It should be appreciated that various techniques and workflows for identifying alignment offsets may be combined. As used herein, "alignment offset" is intended to mean an offset distance (e.g., longitudinal alignment frame) or an offset angle (e.g., angular alignment frame), or both. For example, the IVUS run frame map 420 may include either or both of an offset distance and an offset angle. For some examples, the various offset derivation methods outlined herein may be combined on a segment-by-segment basis. For example, a first segment (e.g., Fig.9A The alignment offset of the frame in the segment 904a, etc.) can be determined based on the second selection of the alignment method disclosed herein, and another segment (for example, Fig.9A As a specific example, the frames in segment 904a can be aligned using frame-by-frame association, while the frames in segment 904b can be aligned using reasoning from the ML model. However, the claims are not limited to this example, and may also include any combination of techniques based on segment-by-segment implementation.

[0094] As described above, a GUI may be generated to present a graphical indication of different IVUS runs in relation to one another, for example, with frames aligned as described herein. Fig.11 1100 is shown, which may be generated according to some embodiments of the present disclosure. In some embodiments, GUI 1100 may be Figure 4 , Figure 7 or Fig.15 For example, processor 406 may execute instructions 416 to generate a GUI having Fig.11 1100 and the graphical components and arrangement of GUI 424. In such an example, processor 406 may execute instructions 416 to cause GUI 1100 to be displayed on display 404.

[0095] GUI 1100 may include a graphical indication of IVUS image 418a and IVUS image 418b. As shown in this example, the graphical indication of IVUS image 418a and IVUS image 418b includes an on-axis view (e.g., on-axis view 1102a and on-axis view 1102b) and a longitudinal view (e.g., longitudinal view 1104a and longitudinal view 1104b). As shown, GUI 1100 may arrange on-axis view 1102a and on-axis view 1102b and longitudinal view 1104a and longitudinal view 1104b in a horizontal (e.g., side by side) visualization. For other embodiments, processor 406 may execute instruction 416 to generate GUI 1100, thereby visualizing on-axis view 1102a and on-axis view 1102b in a vertical arrangement.

[0096] In addition, the GUI 1100 may include a dual view slide bar 1106 and a dual view slider 1108. The dual view slider 1108 may be manipulated (e.g., by a touch screen, a mouse, a joystick, etc.) to slide (or move) through the frames of the IVUS image. When the dual view slider 1108 is moved, the processor 406 may execute instructions 416 to regenerate the GUI 1100 to move the positions of the frame indicators 1110a and 1110b and the dual view slider 1108 located on the longitudinal views 1104a and 1104b. In addition, the on-axis views 1102a and 1102b may change to correspond to the frames from each respective IVUS run that match the positions of the frame indicators 1110a and 1110b.

[0097] Thus, as described herein, one or two IVUS runs can be adjusted (e.g., based on an offset distance and / or an offset angle) to align the IVUS runs with each other. Thus, a user (e.g., a physician) can view different IVUS runs (e.g., a pre-PCI run and a post-PCI run, etc.) with the locations of the blood vessels and corresponding fiducials aligned in a visualization, e.g., depicted in GUI 1100.

[0098] In some embodiments, more than two (2) IVUS runs may be displayed in the GUI. For example, Fig. 8A and Figure 8B Three (3) IVUS runs are shown, which are moved so that the IVUS runs are aligned with each other. Accordingly, a GUI 1100 may be generated to display a graphical indication of each of the three (3) IVUS runs.

[0099] Fig.12A logic flow 1200 for aligning different IVUS runs according to some embodiments of the present disclosure is shown. The logic flow 1200 can be implemented by the IVUS image correlation and visualization system described herein (e.g., IVUS image correlation and visualization system 400, 700, etc.). For clarity and not limitation, the logic flow 1200 is described with reference to the IVUS image correlation and visualization system 400.

[0100] Logic flow 1200 may start at block 1202. At block 1202, “RECEIVE A FIRST SERIES OF IVUS IMAGES OF A PATIENT'S VEssel,” a first series of IVUS images captured by an IVUS catheter percutaneously inserted into a patient's vessel may be received. For example, while catheter 102 is percutaneously inserted into vessel 202, an information element including an indication of IVUS image 418a may be received from IVUS imaging system 100. IVUS image 418a may include an image frame representing an image captured when catheter 102 is pulled back from distal end 204 to proximal end 206. Processor 406 may execute instructions 416 to receive an information element including an indication of IVUS image 418a from IVUS imaging system 100, or directly from catheter 102, as the case may be.

[0101] Continuing to block 1204, “RECEIVE A SECOND SERIES OF IVUS IMAGES OF A PATIENT'S VESSEL,” a second series of IVUS images captured by an IVUS catheter percutaneously inserted into a patient's vessel may be received. For example, an information element including an indication of IVUS image 418b may be received from IVUS imaging system 100, where catheter 102 is (or was) percutaneously inserted into vessel 202. As with IVUS image 418a, IVUS image 418b may include image frames representing images captured when catheter 102 is pulled back from distal end 204 to proximal end 206. However, as described above and contemplated herein, the distal end 204 and proximal end 206 of IVUS image 418a may be located at different locations than the distal end 204 and proximal end 206 of IVUS image 418b. Processor 406 may execute instructions 416 to receive an information element including an indication of IVUS image 418b from IVUS imaging system 100, or directly from catheter 102, as the case may be.

[0102] Continuing to block 1206, “Identify a mapping between frames in the first series of IVUS images and frames in the second series of IVUS images,” a mapping between frames in the first series of IVUS images and frames in the second series of IVUS images may be identified. For example, the processor 406 may execute instructions 416 to generate an IVUS running frame map 420 based on the ML model 422. In another embodiment, the processor 406 may execute the ML model 704 to identify the vascular fiducial 702, and then identify the IVUS running frame map 420 based on the vascular fiducial 702. In another example, the processor 406 may execute instructions 416 to generate an IVUS running frame map 420 based on the association as described above (e.g., frame-by-frame association, angle offset frame-by-frame association, etc.). In another example, the processor 406 may execute instructions 416 to generate the IVUS running frame map 420 segment by segment as described above.

[0103] In any of the above embodiments, the IVUS run frame map 420 may include an indication of an offset (e.g., time, distance, angle, etc.) for one or two series of IVUS images that, when applied, will cause the image to be viewed vertically (e.g., as shown in FIG. Figure 8B As described herein, the IVUS run frame map 420 may indicate an offset distance and / or an offset angle. The examples are not limited to this context.

[0104] In some examples, the processor 406 may execute instructions 416 to map frames based on the longitudinal offset described herein. In such examples, the processor 406 may execute instructions 416 to map frames based on partial overlap and time warping. It should be understood that a set of IVUS images (e.g., IVUS images 418a, etc.) may be captured at a first pullback speed, while another set of IVUS images (e.g., IVUS images 418b, etc.) may be captured at a second pullback speed different from the first pullback speed. In another example, a set of IVUS images (e.g., IVUS images 418a, etc.) may be captured along a first pullback path through a blood vessel, while another set of IVUS images (e.g., IVUS images 418b, etc.) may be captured along a slightly different pullback path, or motion artifacts may appear in the captured IVUS images.

[0105] Thus, while many examples discuss aligning (or registering) IVUS images of different runs based on offset distances and / or offset angles, some embodiments provide that runs may also be aligned (or registered) based on motion overlap and / or time warping.

[0106] For example, Fig.13A diagram 1300 is shown showing the alignment of features extracted and vectorized from two IVUS runs through a vessel. Extracted and vectorized feature 1302a can be generated from IVUS image 418a, while extracted and vectorized feature 1302b can be generated from IVUS image 418b. The features can be aligned based on a time warp along a longitudinal offset, as described herein. That is, as shown in the figure, the frames of the IVUS runs can be longitudinally offset by different amounts to account for different pullback speeds and paths through the vessel.

[0107] Continuing to box 1208, "Generate a graphical user interface including an indication of a first series of IVUS images and a second series of IVUS images, wherein at least one of the first series of IVUS images or the second series of IVUS images is offset (e.g., in time, distance, angle, etc.) based on the mapping to align the first series of IVUS images with the second series of IVUS images longitudinally and / or angularly," a GUI can be generated, wherein the GUI includes a graphical indication of the first series of IVUS images and the second series of IVUS images, and any number of frames from the first series and / or the second series of IVUS images are offset (e.g., in time, distance, angle, etc.) to align the first series of IVUS images with the second series of IVUS images longitudinally and / or angularly. For example, the processor 406 can execute instructions 416 to generate the GUI 424 as described above. As a specific example, the processor 406 can execute instructions 416 to generate the GUI 1100 as the GUI 424 and cause the GUI 1100 to be displayed on the display 404.

[0108] As described above, for some embodiments, the processor 406 of the computing device 402 may execute instructions 416 to generate an IVUS running frame map 420 using the ML model, or to generate a vascular fiducial 702 from the ML model and then generate an IVUS running frame map 420 according to the vascular fiducial 702. In such examples, the ML model may be stored in the memory 408 of the computing device 402. It will be appreciated that the ML model needs to be trained before deployment. Fig.14 An ML environment 1400 is shown that can be used to train an ML model that can later be used to generate (or infer) a map or vascular fiducial as described herein. The ML environment 1400 can include an ML system 1402, such as a computing device that applies an ML algorithm to learn a relationship between an input and infer an output. In this example, the ML algorithm can learn a relationship between an input (e.g., an IVUS image) and an output (e.g., a frame map or vascular fiducial according to an embodiment).

[0109] The ML system 1402 may utilize experimental data 1408 collected during several previous procedures. The experimental data 1408 may include IVUS images from multiple IVUS runs of multiple patients. The experimental data 1408 may be co-located with the ML system 1402 (e.g., stored in a memory 1410 of the ML system 1402), may be remote from the ML system 1402 and accessed through the network interface 1404, or may be a combination of local and remote data.

[0110] The experimental data 1408 may be used to form training data 1412. As described above, the ML system 1402 may include a memory 1410, which may include a hard disk, a solid-state memory, and / or a random access memory. The memory 1410 may store the training data 1412. In general, the training data 1412 may include an information element or data structure that includes an indication of multiple series of IVUS images and a corresponding desired output (e.g., a map or vascular reference). It should be understood that if the desired output is an IVUS frame map, the input may be two (or more, as the case may be) series of IVUS images. For reference Figure 4 In a specific example of , where ML model 1424 is to be trained and deployed as ML model 422, the input may be multiple pairs of a first series of IVUS images and a second series of IVUS images (e.g., more than one IVUS run), and the output may be a mapping associated with each pair of the first series of IVUS images and the second series of IVUS images (e.g., a mapping between IVUS runs). In another example, referring to Figure 7 , where ML model 1424 will be trained and deployed as ML model 704, the input may be a single series of IVUS images (e.g., a single IVUS run), and the output may be frames in the IVUS images in which a vascular fiducial (or fiducials) are identified.

[0111] The training data 1412 may be used to train the ML model 1424. Depending on the application, different types of models may be used to form the basis of the ML model 1424. For example, in this example, an artificial neural network (ANN) may be particularly suitable for learning the association between IVUS images (e.g., IVUS image 418a, IVUS image 418b, etc.) and a reference or frame map (e.g., IVUS run frame map 420, vascular reference 702, etc.). A convolutional neural network may also be well suited for this task. In another example, the ML model 1424 may be based on a spatial transformer (e.g., a spatial transformer network, etc.). As another example, the ML model 1424 may be a plurality of networks, such as a Siamese network, etc.

[0112] Any suitable training algorithm 1420 may be used to train the ML model 1424. For example, the examples described herein may be applicable to supervised training algorithms or reinforcement learning training algorithms. For supervised training algorithms, the ML system 1402 may use the IVUS image 1414 as an input 1430, and the ML model 1424 may generate an expected output (e.g., a mapping or benchmark) for the input 1430. In a reinforcement learning scenario, the training algorithm 1420 may attempt to maximize some or all (or a weighted combination) of the model inputs 1430 that map to outputs 1426 to produce an ML model 1424 with minimal error. In some embodiments, the training data 1412 may be divided into "training" data and "testing" data, where a certain subset of the training data 1412 may be used to adjust the ML model 1424 (e.g., the model's internal weights, etc.), while another non-overlapping subset of the training data 1412 may be used to measure the accuracy of the ML model 1424 to infer (or generalize) the output 1426 from the "unseen" input 1430.

[0113] The ML model 1424 may be applied using the processor circuit 1406, which may include suitable hardware processing resources to operate on the logic and structures in the memory 1410. The development of the training algorithm 1420 and / or the trained ML model 1424 may rely, at least in part, on the hyperparameters 1422. In an exemplary embodiment, the model hyperparameters 1422 may be automatically selected based on logic 1428, which may include any known hyperparameter optimization techniques to suit the selected ML model 1424 and the training algorithm 1420 to be used. In an optional embodiment, the ML model 1424 may be retrained over time to adapt to new knowledge and / or updated experimental data 1424.

[0114] Once the ML model 1424 is trained, it can be applied (e.g., by the processor 406, etc.) to new input data (e.g., IVUS image 418a, IVUS image 418b, etc.). This input to the ML model (e.g., ML model 422, ML model 702, etc.) can be formatted according to a predefined model input 1430 that reflects the manner in which the training data 1412 is provided to the ML model 1424. The ML model 1424 can generate an output 1426, which can be, for example, a generalized or IVUS run frame map 420 or a vascular fiducial 702 as described above.

[0115] The above description relates to a specific type of ML system 1402 that applies supervised learning techniques given available training data with input / output pairs. However, the present invention is not limited to use with a specific ML paradigm, and other types of ML techniques may be used. For example, in some embodiments, the ML system 1402 may apply, for example, evolutionary algorithms or other types of ML algorithms and models to the IVUS running frame map 420 (or the vascular fiducial 702, as the case may be) from the IVUS image 418a and / or the IVUS image 418b.

[0116] In some embodiments, ML model 1424 may be a traditional ML model, such as a neural network, a convolutional neural network, an evolutionary artificial neural network, etc. However, in some embodiments, ML model 1424 may not be a ML model in the traditional sense. For example, ML model 1424 may be a dynamic programming algorithm, where training data 1412 is used to adjust the parameters of the dynamic programming algorithm.

[0117] In some embodiments, the present disclosure may be used to angularly align an IVUS run with a vascular view from an external imaging modality. For example, Fig.15 An IVUS image correlation and visualization system 1500 is shown according to some embodiments of the present disclosure. Generally speaking, the IVUS image correlation and visualization system 1500 is a system for processing, correlating, and presenting IVUS images with external images of the same blood vessel. To simplify the discussion, many components of the IVUS image correlation and visualization system 400 are referenced and reused when describing the IVUS image correlation and visualization system 1500.

[0118] As mentioned above about Figure 4 As described with the IVUS image association and visualization system 400, the present disclosure provides for generating an IVUS running frame map 420 for the IVUS image 418a and the IVUS image 418b. In some embodiments, the IVUS running frame map 420 can be generated based on an external image of a blood vessel. It is worth noting that there are a variety of techniques for aligning an intravascular image (e.g., IVUS images 418a and / or 418b) with an external image. The present disclosure does not reproduce such techniques here. However, for the sake of clarity, it is worth noting that a reference can be identified on the external image, just as on the intravascular image, and the references are mapped to each other to align a frame in the intravascular image with a point on the external image (e.g., in x and y coordinates).

[0119] Thus, in some examples, the IVUS image correlation and visualization system 1500 can be coupled to an external imaging system 1506 (e.g., an angiography machine, a computed tomography (CT) machine, a magnetic resonance imaging (MRI) machine, etc.) that is configured to capture external images of a blood vessel, with which the IVUS images 418a and / or 418b are captured. Alternatively, the IVUS image correlation and visualization system 1500 can be coupled to a memory device that stores external images or external image frames.

[0120] The processor 406 may execute instructions 416 to receive the external image 1502 (or multiple images) from the external imaging system 1506 (or storage device). The processor 406 may execute instructions 416 to identify the fiducials in the external image 1502 and the IVUS image 418a (or IVUS image 418b). For example, the processor 406 may execute instructions 416 to identify the vessel fiducial 702 corresponding to the fiducials in the IVUS image 418a and the external image 1502.

[0121] As described above, there are a variety of techniques for identifying fiducials in internal imaging modes and external imaging modes. For example, side branch identification and matching are often used to align internal images with external images. The present disclosure provides that the processor 406 can execute instructions 416 to identify the fiducial and its position and identify the angle of the fiducial and store an indication of the fiducial position and angle in the vascular fiducial 702. For some embodiments, the processor 406 can use image processing techniques and / or ML reasoning to identify the angle of the fiducial. For example, the ML model 702 can be trained as described above to identify the fiducial and its corresponding angle from the external image 1502. Once the angle of the fiducial in the external image 1502 is identified, the processor 406 can execute instructions 416 to identify an offset angle (e.g., IVUS run frame mapping 420, etc.) to rotate the frame of the IVUS image (e.g., IVUS image 418a and / or 418b) using the offset angle to align the viewing angle with the perspective of the external image 1502. In addition, processor 406 may execute instructions 416 to identify the offset of other frames in the IVUS image given the offset angle of the frame corresponding to the reference (e.g., as described above with respect to Fig. 10A and Fig. 10B etc.).

[0122] For example, Fig.16AAn external image 1502 and two identified fiducials (e.g., side branches) 1602a and 1602b are shown. The processor 406 can execute instructions 416 to identify the angles of the fiducials 1602a and 1602b. It is worth noting that the angles of the fiducials are derived based on the baseline, for example, setting zero (0) degrees to the Z direction from the two-dimensional (2D) image toward the observer, etc. The processor 406 can execute instructions 416 to rotate (or derive an angle offset) the frames in the IVUS image 418a that match the fiducials 1602a and 1602b based on the angles of the fiducials 1602a and 1602b.

[0123] For example, Fig. 16B and Fig. 16C Image frames 1604a and 1604b (e.g., frames from IVUS image 418a, etc.) depicting fiducials 1602a and 1602b, respectively, are shown. Processor 406 may execute instructions 416 to rotate image frames 1604a and 1604b based on the angle of the vessel fiducials represented in external image 1502 (e.g., side branch angles, etc.) and the angle of the fiducials in each respective frame 1604a and 1604b, thereby generating rotated image frames 1606a and 1606b. Rotated image frames 1606a and 1606b are respectively shown in Fig. 16B and Fig. 16C Shown in.

[0124] In some examples, the image frame may be rotated based on a fiducial marker. Fig. 16B 4. In some embodiments, processor 406 may execute instructions 416 to identify a fiducial marker and rotate the image frame based on the angle of the fiducial marker. For example, a fiducial marker 1610 (e.g., a side branch) in image frame 1604a is depicted at approximately 9 o'clock or 270 degrees. The frame may be rotated an angle based on the angle of a fiducial marker in another image frame so that the fiducial marker is aligned at a particular angle. For example, rotated image frame 1606a shows the fiducial marker rotated to 180 degrees.

[0125] Thus, as described above, processor 406 may execute instructions 416 to align frames within an IVUS run with the viewing angle of an external image (eg, external image 1502, etc.) such that the angle of viewing fiducial is aligned between the two imaging modes. Fig.16DA set of external image aligned IVUS images 1608 are shown, which may correspond to frames from IVUS image 418a (or similar images) in which the viewing angle (or viewing angle) has been aligned with the viewing angle of the external image frame 1502. Notably, this provides a significant improvement over conventional techniques. It should be understood that intravascular images are typically independent of the viewing angle. For example, IVUS images are captured as the ultrasound transducer rotates within a blood vessel. Therefore, the actual viewing angle between frames may vary. In addition, the viewing angle of the external image may also change (e.g., based on the patient's position relative to the image acquisition system, etc.). Therefore, the viewing angle between the intravascular and extravascular images is typically inconsistent. The present disclosure addresses this problem.

[0126] In addition, as described above, a GUI may be generated to present a graphical indication of an aligned IVUS run. For example, a GUI may be generated to present a visual representation of a frame from an IVUS run that is aligned with a blood vessel seen in an external image. Fig.17 GUI 1700 is shown, which may be generated according to some embodiments of the present disclosure. In some embodiments, GUI 1700 may be Figure 4 , Figure 7 or Fig.15 For example, processor 406 may execute instructions 416 to generate a GUI having Fig.17 1700 and the graphical components and arrangement of GUI 424. In such an example, processor 406 may execute instructions 416 to cause GUI 1700 to be displayed on display 404.

[0127] The GUI 1700 may include a graphical indication of the external image 1502 and the IVUS external image aligned IVUS image 1608. Thus, when a physician (or user) examines a frame of the IVUS image 418a, the external image aligned IVUS image 1608 will be displayed such that the lumen and fiducials seen in the IVUS image frame will match the angles of the vessels and fiducials (e.g., fiducials 1602a and 1602b) seen in the external image frame.

[0128] Fig.18A computer-readable storage medium 1800 is shown. The computer-readable storage medium 1800 may include any non-transient computer-readable storage medium or machine-readable storage medium, such as an optical, magnetic or semiconductor storage medium. In various embodiments, the computer-readable storage medium 1800 may include an article of manufacture. In some embodiments, the computer-readable storage medium 1800 may store 1802, and a circuit (e.g., processor 106, processor 406, processor circuit 1406, etc.) may execute using the instruction. For example, the computer-executable instructions 1802 may include instructions for implementing the operations described with respect to the instructions 416 and / or the logic flow 1200. Examples of computer-readable storage media 1800 or machine-readable storage media may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer-executable instructions 1802 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc.

[0129] Fig.19 A schematic diagram of a machine 1900 in the form of a computer system is shown in which a set of instructions may be executed to cause the machine to perform one or more of the methods discussed herein. More specifically, Fig.19 A schematic diagram of a machine 1900 in the form of an example computer system is shown in which instructions 1908 (e.g., software, programs, applications, applet, application, or other executable code) may be executed to cause the machine 1900 to perform one or more of the methods discussed herein. For example, the instructions 1908 may cause the machine 1900 to perform Fig.12 More generally, the instructions 1908 may cause the machine 1900 to automatically determine a mapping (e.g., time, distance, angle, etc.) between frames of different IVUS runs (e.g., from a pre-PCI IVUS run, a during-PCI IVUS run, and / or a post-PCI IVUS run) through the same blood vessel and / or between IVUS runs and external images.

[0130] Instructions 1908 convert a general, non-programmed machine 1900 into a specific machine 1900 that is programmed to perform the functions described and shown in a specific manner. In alternative embodiments, the machine 1900 operates as a standalone device, or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1900 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer (or distributed) network environment. The machine 1900 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1908 that specify the actions to be taken by the machine 1900. Further, while a single machine 1900 is illustrated, the term "machine" shall also be taken to include any collection of machines 1900 that individually or jointly execute instructions 1908 to perform any one or more of the methodologies discussed herein.

[0131] The machine 1900 may include a processor 1902, a memory 1904, and an I / O component 1942, which may be configured to communicate with each other, for example, via a bus 1944. In an example embodiment, the processor 1902 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1906 that may execute instructions 1908 and a processor 1910. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.19 Multiple processors 1902 are shown, but the machine 1900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0132] The memory 1904 may include a main memory 1912, a static memory 1914, and a storage unit 1916, both of which may be accessed by the processor 1902 (e.g., via the bus 1944). The main memory 1904, the static memory 1914, and the storage unit 1916 store instructions 1908 that embody one or more of the methods or functions described herein. During execution of the instructions 1908 by the machine 1900, the instructions 1908 may also reside, in whole or in part, within the main memory 1912, within the static memory 1914, within a machine-readable medium 1918 within the storage unit 1916, within at least one of the processors 1902 (e.g., within a cache of the processor), or any suitable combination thereof.

[0133] I / O components 1942 may include a variety of components to receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O components 1942 included in a particular machine will depend on the type of machine. For example, a portable machine (such as a mobile phone) may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It is understood that I / O components 1942 may include Fig.19 1942 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1942 may include output components 1928 and input components 1930. The output components 1928 may include visual components, such as displays (such as plasma display panels (PDPs)), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (such as speakers), tactile components (such as vibration motors, resistance mechanisms), other signal generators, and the like. The input components 1930 may include alphanumeric input components (such as keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (such as mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (such as physical buttons, touch screens or other tactile input components that provide the location and / or force of touch or touch gestures), audio input components (such as microphones), and the like.

[0134] In further example embodiments, the I / O component 1942 may include a biometric component 1932, a motion component 1934, an environmental component 1936, or a position component 1938, as well as various other components. For example, the biometric component 1932 may include a component for detecting expressions (e.g., gestures, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or EEG-based identification), etc. The motion component 1934 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1936 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting dangerous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The location component 1938 may include a location sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure and from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0135] Communication can be achieved using a variety of technologies. I / O components 1942 may include a communication component 1940 that is operable to couple machine 1900 to network 1920 or device 1922 via coupler 1924 and coupler 1926, respectively. For example, communication component 1940 may include a network interface component or another suitable device for interfacing with network 1920. In further examples, communication component 1940 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g., low power ), Components and other communication components to provide communication by other means. Device 1922 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0136] In addition, the communication component 1940 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1940 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode (e.g., a universal product code (UPC) barcode), a multi-dimensional barcode (e.g., a Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCCRSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying a tag audio signal). In addition, various information can be obtained through the communication component 1940, such as obtaining a location through Internet Protocol (IP) geolocation, ... Signal triangulation to obtain location, by detecting NFC beacon signals that may indicate a specific location, and so on.

[0137] Various memories (i.e., memory 1904, main memory 1912, static memory 1914, and / or memory of processor 1902) and / or storage unit 1916 may store one or more instruction sets and data structures (e.g., software) that embody or are utilized by one or more methods or functions described herein. When these instructions (e.g., instructions 1908) are executed by processor 1902, they may cause various operations to implement the disclosed embodiments.

[0138] As used herein, the terms "machine storage medium", "device storage medium", and "computer storage medium" have the same meaning and are used interchangeably in this disclosure. These terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should include, but are not limited to, solid-state memory, optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are included under the term "signal media" discussed below.

[0139] In various example embodiments, one or more portions of network 1920 may be an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, the Internet, a portion of the Internet, a portion of the PSTN, a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, network, other types of networks, or a combination of two or more such networks. For example, network 1920 or a portion of network 1920 may include a wireless or cellular network, and coupler 1924 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupler 1924 may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, including 3G third generation partnership project (3GPP), fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), microwave access world interoperability (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0140] Instructions 1908 may be transmitted or received over network 1920 via a network interface device (e.g., a network interface component included in communication component 1940) using a transmission medium and via any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1908 may be transmitted or received to device 1922 via coupler 1926 (e.g., peer-to-peer coupling) using a transmission medium. The terms "transmission medium" and "signal medium" have the same meaning and are used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be understood to include any intangible medium that can store, encode, or carry instructions 1908 for execution by machine 1900, and include digital or analog communication signals or other intangible media to facilitate the communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be understood to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" refers to a signal whose one or more characteristics are set or changed so as to encode information into the signal.

[0141] Terms used herein shall have their ordinary meaning in the relevant art or the meaning indicated by their usage in context, but if an explicit definition is provided, that meaning shall prevail.

[0142] References to "one embodiment" or "an embodiment" herein do not necessarily refer to the same embodiment, although they may. Unless the context clearly requires otherwise, throughout the specification and claims, the words "including", "comprising", and the like are to be interpreted in an inclusive sense rather than an exclusive or exhaustive sense; that is, in the sense of "including but not limited to". Words using the singular or plural also include the plural or singular, respectively, unless expressly limited to one or more. In addition, when the words "herein", "above", "below", and words of similar meaning are used in this application, they refer to the entire application and not to any part of this application. When the claims use the word "or" to refer to a list of two or more items, the word covers all of the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list, unless expressly limited to one or the other. Any term not expressly defined herein has the conventional meaning commonly understood by technicians in the relevant field.

Claims

1. A method for a computing device, comprising: At a processor, receiving a first series of intravascular ultrasound (IVUS) images of a vessel of a patient, the first series of IVUS images comprising a first plurality of frames; receiving, at the processor, a second series of intravascular ultrasound (IVUS) images of a blood vessel of the patient, the second series of IVUS images comprising a second plurality of frames; determining, by the processor, a mapping between the first plurality of frames and the second plurality of frames; as well as A graphical user interface (GUI) is generated by the processor, the GUI including an indication of the first series of IVUS images and the second series of IVUS images based on the mapping between the first plurality of frames and the second plurality of frames.

2. The method of claim 1, determining a mapping between the first plurality of frames and the second plurality of frames comprises: identifying a frame of the first plurality of frames that includes a specific vascular fiducial; as well as A frame of the second plurality of frames that includes the specific blood vessel fiducial is identified.

3. The method of claim 1 or 2, comprising determining an offset between the one frame of the first plurality of frames and the one frame of the second plurality of frames. 4 . The method of claim 3 , wherein the GUI includes an indication of the first series of IVUS images and the second series of IVUS images aligned based on the offset. 5 . The method according to claim 3 , wherein the GUI includes on-axis views of the first series of IVUS images and the second series of IVUS images, and wherein the on-axis views are associated with each other based on the offset.

6. The method of claim 3 or 4, wherein the GUI includes longitudinal views of the first series of IVUS images and the second series of IVUS images, and wherein the longitudinal views are associated with each other based on the offset.

7. The method according to any one of claims 1 to 6, comprising: executing a machine learning (ML) model to infer a frame of the first plurality of frames containing a specific vascular fiducial based on the first series of IVUS images; as well as The ML model is executed to infer a frame of the second plurality of frames that contains the specific vascular fiducial based on the second series of IVUS images.

8. The method of any one of claims 1 to 7, wherein the specific vessel fiducial is one of: lumen geometry, vessel geometry, side branch location, calcium morphology, plaque distribution, or guide catheter location.

9. The method according to any one of claims 1 to 8, comprising: receiving the second series of IVUS images from an intravascular imaging device; as well as The first series of IVUS images is received from a memory storage device.

10. The method of any one of claims 1 to 9, wherein the first series of IVUS images are captured during a pre-percutaneous coronary intervention (PCI) procedure.

11. The method of claim 10, wherein the second series of IVUS images are captured during a PCI procedure or during a post-PCI procedure.

12. The method of any one of claims 1 to 11, determining a mapping between the first plurality of frames and the second plurality of frames comprises executing a machine learning (ML) model to infer the mapping based on the first series of IVUS images and the second series of IVUS images.

13. The method of claim 12, wherein the mapping includes an indication of an offset between a frame in the first plurality of frames and a frame in the second plurality of frames.

14. An apparatus comprising a processor coupled to a memory, the memory comprising instructions executable by the processor, the processor being configured to be coupled to an intravascular ultrasound (IVUS) imaging system and configured to execute the instructions, the instructions when executed causing the processor to implement the method of claims 1 to 13.

15. At least one machine-readable storage device comprising a plurality of instructions that, in response to being executed by a processor of an intravascular ultrasound (IVUS) imaging system, cause the processor to implement the method of claims 1 to 13.

Images