Domain adaptation for enhancing image features from other imaging modalities

Abstract

Devices and methods are provided for processing an intravascular image of a blood vessel of one imaging modality, and generating, extracting, and adapting features according to the other imaging modality to generate a hybrid image including features from two modalities. Devices and methods are provided for training a depth generative model to adapt domain-specific features from one intravascular imaging modality (e.g., OCT, etc.) to another intravascular imaging modality (e.g., IVUS), and integrating the adapted features into the image from the another intravascular imaging modality.

Description

Cross-references to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 502,809, filed May 17, 2023, the disclosure of which is incorporated herein by reference. Technical Field

[0002] This invention generally relates to intravascular ultrasound (IVUS) imaging systems. In particular, but not exclusively, this invention relates to enhancing the features of IVUS images based on domain adaptations from other imaging modalities, such as optical coherence tomography (OCT). Background Technology

[0003] Insertable ultrasound devices have proven their diagnostic capabilities for a wide range of diseases and conditions. For example, intravascular ultrasound (IVUS) imaging systems have been used as imaging modalities to diagnose blocked blood vessels and provide information to help healthcare practitioners select and place stents and other devices to restore or increase blood flow.

[0004] An IVUS imaging system includes a control module (containing a pulse generator, image acquisition and processing components, and a monitor), a catheter, and a transducer disposed within the catheter. The catheter, containing the transducer, is positioned in or near a lumen or cavity within the area to be imaged, such as the vessel wall or patient tissue adjacent to the vessel wall. The pulse generator in the control module generates electrical pulses, which are delivered to the transducer and converted into acoustic pulses that are transmitted through the patient tissue. The patient tissue (or other structure) reflects the acoustic pulses, and the reflected pulses are absorbed by the transducer and converted back into electrical pulses. The converted electrical pulses are delivered to the image acquisition and processing components and converted into an image that can be displayed on the monitor.

[0005] Other intravascular imaging modalities exist, such as, for example, optical coherence tomography (OCT), which uses infrared light instead of ultrasound to capture images within blood vessels. In some procedures, "hybrid" catheters can be used, where a hybrid catheter can capture both IVUS and OCT images in the same procedure.

[0006] These two imaging modalities need to be utilized in a complementary manner to produce better clinical outcomes. Summary of the Invention

[0007] This invention provides a simplified overview of a set of concepts, which will be further described in the detailed description below. This invention is not intended to necessarily identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0008] Typically, this invention provides for adapting features from one imaging modality (e.g., OCT) to enhance features from another imaging modality (e.g., IVUS) to reduce the need for screening multiple imaging modalities. This invention provides for utilizing machine learning (ML), such as deep generative models (generative adversarial networks, generative diffusion models, etc.), to adapt domain-specific features from one intravascular imaging modality (e.g., OCT, etc.) to another intravascular imaging modality (e.g., IVUS), and integrating the adapted features into IVUS images.

[0009] Therefore, this invention offers advantages over conventional technologies, systems, and treatments because improved diagnostic capabilities (e.g., lesion diagnosis) or improved surgical guidance (e.g., percutaneous coronary intervention (PCI) guidance) can lead to better clinical outcomes. Furthermore, as previously stated, this invention reduces the number of imaging modalities required for screening patients and / or analyzing treatment outcomes.

[0010] In some embodiments, the present invention can be implemented as a method for a computing device. The method may include: a processor receiving a first series of intravascular images of a patient's blood vessels, the first series of intravascular images representing a first imaging modality; the processor generating image features of a second imaging modality based on the first series of intravascular images; the processor enhancing the first series of intravascular images with the image features of the second series of imaging modalities; and the processor generating a graphical user interface including instructions for the enhanced first series of intravascular images.

[0011] In another embodiment, the method may include displaying a graphical user interface on a display connected to a computing device.

[0012] In another embodiment of the method, generating image features of the second imaging modality may include generating a second series of intravascular images of the patient's blood vessels via a machine learning (ML) model, the second series of intravascular images being the second imaging modality; and generating image features of the second imaging modality via an ML model based on the first series of intravascular images.

[0013] In another embodiment of the method, generating image features of a second imaging modality based on a first series of intravascular images may include converting the first series of intravascular images into a second imaging modality via an ML model to form a series of converted intravascular images, wherein the features of the series of converted intravascular images resemble the image features of the second imaging modality; and extracting features from the series of converted images via an ML model.

[0014] In another embodiment of the method, enhancing the first series of intravascular images with image features of the second imaging modality may include generating a series of hybrid intravascular images via an ML model, the series of hybrid intravascular images including image features of the first series of intravascular images and the second imaging modality.

[0015] In another embodiment of the method, the ML model includes a medical image generation network and an auxiliary task network, wherein the auxiliary task network is arranged to preserve the geometry of the extracted features.

[0016] In another embodiment of the method, the ML model is trained using multiple series of intravascular images from the first modality, either paired with or unpaired with corresponding series of intravascular images from the second modality.

[0017] In another embodiment of the method, the medical image generation model is trained using a non-adversarial loss from an auxiliary task network.

[0018] In another embodiment of the method, the ML model includes an encoder network based on a convolutional neural network (CNN) and a first decoder network and a second decoder network.

[0019] In another embodiment of the method, a CNN-based encoder network is arranged to convert a series of intravascular images of a first imaging modality into a series of intravascular images of a second imaging modality, and to convert a series of intravascular images of the second imaging modality into a series of intravascular images of the first imaging modality.

[0020] In another embodiment of the method, a first decoder network is arranged to extract features from the series of intravascular images derived from the first imaging modality.

[0021] In another embodiment of the method, a second decoder network is arranged to extract features from the series of intravascular images derived from the second imaging modality.

[0022] In another embodiment of the method, the ML model is trained using multiple series of intravascular images from the first modality, either paired with or unpaired with corresponding series of intravascular images from the second modality.

[0023] In some embodiments, the invention can be implemented as a device including a memory and a processor, the processor being associated with the memory and configured to be associated with an intravascular ultrasound (IVUS) imaging system. The memory may include instructions executable by the processor, which, when executed, cause the processor to perform any of the methods described herein.

[0024] In some embodiments, the present invention may be implemented as a machine-readable storage device comprising a plurality of instructions responsive to execution by a processor of an intravascular ultrasound (IVUS) imaging system, the plurality of instructions causing the processor to perform any of the methods described herein.

[0025] In some embodiments, the present invention can be implemented as a device including a memory and a processor, the processor being coupled to the memory and configured to be coupled to an intravascular ultrasound (IVUS) imaging system. The memory may include instructions executable by the processor, which, when executed, cause the processor to receive a first series of intravascular images of a patient's blood vessels, the first series of intravascular images representing a first imaging modality; generate image features of a second imaging modality based on the first series of intravascular images; enhance the first series of intravascular images with the image features of the second series of imaging modality; and generate a graphical user interface including instructions for the enhanced first series of intravascular images.

[0026] In some embodiments of the device, when executed by the processor, the instructions also cause the processor to display a graphical user interface on a display connected to the computing device.

[0027] In some embodiments of the device, when executed by the processor, the instructions also cause the processor to generate a second series of intravascular images of the patient's blood vessels via a machine learning (ML) model, the second series of intravascular images being a second imaging modality; and to generate image features of the second imaging modality via the ML model based on the first series of intravascular images.

[0028] In some embodiments of the device, when executed by the processor, the instructions also cause the processor to convert a first series of intravascular images into a second imaging modality via an ML model to form a series of converted intravascular images, wherein the features of the series of converted intravascular images resemble the image features of the second imaging modality; and to extract features from the series of converted images via an ML model.

[0029] In some embodiments of the device, when executed by the processor, the instructions also cause the processor to generate a series of mixed intravascular images via an ML model, the series of mixed intravascular images including image features of a first series of intravascular images and a second imaging modality.

[0030] In some embodiments of the device, the ML model includes a medical image generation network and an auxiliary task network, wherein the auxiliary task network is arranged to preserve the geometry of the extracted features.

[0031] In some embodiments of the device, the ML model is trained using multiple series of intravascular images from a first modality that are paired or unpaired with corresponding series of intravascular images from multiple series of second modalities.

[0032] In some embodiments of the device, the medical image generation model is trained using a non-adversarial loss from an auxiliary task network.

[0033] In some embodiments of the device, the ML model includes an encoder network based on a convolutional neural network (CNN) and a first decoder network and a second decoder network.

[0034] In some embodiments of the device, a CNN-based encoder network is configured to convert a series of intravascular images of a first imaging modality into a series of intravascular images of a second imaging modality, and to convert a series of intravascular images of the second imaging modality into a series of intravascular images of the first imaging modality.

[0035] In some embodiments of the device, a first decoder network is arranged to extract features from the series of intravascular images derived from a first imaging modality.

[0036] In some embodiments of the device, a second decoder network is arranged to extract features from the series of intravascular images derived from the second imaging modality.

[0037] In some embodiments of the device, the ML model is trained using multiple series of intravascular images from a first modality that are paired with or unpaired with corresponding series of intravascular images from a second modality.

[0038] In some embodiments, the present invention may be implemented as a machine-readable storage device comprising a plurality of instructions responsive to execution by a processor of an intravascular ultrasound (IVUS) imaging system, the plurality of instructions causing the processor to receive a first series of intravascular images of a patient’s blood vessels, the first series of intravascular images being a first imaging modality; generate image features of a second imaging modality by the processor based on the first series of intravascular images; enhance the first series of intravascular images by the processor with the image features of the second series of imaging modality; and generate a graphical user interface by the processor including instructions for the enhanced first series of intravascular images.

[0039] In some embodiments of the machine-readable storage device, when executed by a processor, the instructions also cause the processor to display a graphical user interface on a display connected to the computing device.

[0040] In some embodiments of the machine-readable storage device, when executed by a processor, the instructions also cause the processor to generate a second series of intravascular images of the patient's blood vessels via a machine learning (ML) model, the second series of intravascular images being a second imaging modality; and to generate image features of the second imaging modality via the ML model based on the first series of intravascular images.

[0041] In some embodiments of the machine-readable storage device, when executed by a processor, the instructions also cause the processor to convert a first series of intravascular images into a second imaging modality via an ML model to form a series of converted intravascular images, wherein the features of the series of converted intravascular images resemble the image features of the second imaging modality; and to extract features from the series of converted images via an ML model. Attached Figure Description

[0042] For ease of identification of any discussion of a component or behavior, the highest digit or more digits in the reference numerals refer to the drawing number in which the component is first introduced.

[0043] Figure 1 An example intravascular ultrasound (IVUS) system according to at least one embodiment of the present invention is shown.

[0044] Figure 2 An image outside the blood vessel is shown.

[0045] Figure 3A and Figure 3B An image of the blood vessel is shown.

[0046] Figure 4 A multi-image modality adaptation system according to at least one embodiment of the present invention is shown.

[0047] Figure 5 The logical flow according to at least one embodiment of the present invention is shown.

[0048] Figure 6 A machine learning (ML) system suitable for use with exemplary embodiments of the present invention is shown.

[0049] Figure 7 An example machine learning model architecture according to at least one embodiment of the present invention is shown.

[0050] Figure 8 Another example machine learning model architecture is shown according to at least one embodiment of the present invention.

[0051] Figure 9 Another example of a machine learning model architecture according to at least one embodiment of the present invention is shown.

[0052] Figure 10 A computer-readable storage medium is shown.

[0053] Figure 11 A schematic diagram of the machine is shown. Detailed Implementation

[0054] The features and technical advantages of the invention have been summarized above, which will facilitate a better understanding of the following detailed description. Those skilled in the art will understand that the disclosed embodiments can be readily used as the basis for modifications or the design of other structures to achieve the same objectives as the invention. The novel features of the invention, as well as further objects and advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. However, it should be clearly understood that each of the drawings is provided for illustrative and descriptive purposes and is not intended to be a definition of limitation of the invention.

[0055] As previously stated, the present invention relates to intravascular imaging modalities (e.g., IVUS, OCT, etc.) for capturing patient vessels and lumens (e.g., blood vessels), and to processing these images and adapting features from one modality to integrate them into another modality. Therefore, exemplary intravascular imaging systems, patient vessels, and series of intravascular images are described below. It should be noted that the exemplary system is described as an IVUS system. However, it should be understood that, without departing from the scope of the invention, the system may be an OCT system (or other intravascular imaging modalities) and / or a combination of non-IVUS / IVUS systems.

[0056] Suitable intravascular imaging systems, such as IVUS imaging systems, include, but are not limited to, one or more transducers disposed on the distal end of a catheter configured and arranged for percutaneous insertion into a patient. Examples of IVUS imaging systems with catheters can be found, for example, in U.S. Patent Nos. 7,246,959; 7,306,561; and 6,945,938, and U.S. Patent Application Publication Nos. 2006 / 0100522; 2006 / 0106320; 2006 / 0173350; 2006 / 0253028; 2007 / 0016054; and 2007 / 0038111; all of which are incorporated herein by reference.

[0057] Figure 1 An embodiment of an intravascular imaging system 100 is schematically illustrated. The intravascular imaging system 100 includes a catheter 102 that can be coupled to a control system 104. The control system 104 may include, for example, a processor 106, a pulse generator 108, and a drive unit 110. In at least some embodiments, the pulse generator 108 generates electrical pulses that can be input to one or more transducers (not shown) disposed in the catheter 102.

[0058] In some embodiments, mechanical energy from the drive unit 110 can be used to drive an imaging core (also not shown) disposed in the conduit 102. In at least some embodiments, electrical signals transmitted from one or more transducers can be input to the processor 106 for processing. In at least some embodiments, the processed electrical signals from one or more transducers can be used to form a series of images, as described in more detail below. For example, a scan converter can be used to map scan line samples (e.g., radial scan line samples, etc.) onto a two-dimensional Cartesian grid, which can be used as the basis for a series of IVUS images displayed to the user.

[0059] In at least some embodiments, processor 106 may also be used to control the operation of one or more other components of control system 104. For example, processor 106 may be used to control at least one of the frequency or duration of electrical pulses transmitted from pulse generator 108, and the rotational rate of imaging core driven by drive unit 110. Additionally, when intravascular imaging system 100 is configured for automatic retraction, drive unit 110 may control the speed and / or length of retraction.

[0060] As previously stated, the present invention relates to adapting features from one imaging modality to another. Thus, catheter 102 may be a hybrid catheter arranged to emit and receive ultrasound and, for example, infrared light to capture both IVUS and OCT type images. Alternatively, in another example, intravascular imaging system 100 may include catheter 102 and different catheters arranged to capture non-IVUS intravascular images (e.g., OCT, etc.).

[0061] Figure 2 An extravascular image 200 of a patient's vessel 202 is shown. As described, an IVUS imaging system (e.g., an intravascular imaging system 100, etc.) is used to capture a series of intraluminal images or "records" of a vessel, such as vessel 202. For example, an IVUS catheter (e.g., catheter 102) is inserted into vessel 202, and a record or series of IVUS images is captured as catheter 102 is pulled back from distal end 204 to proximal end 206. Catheter 102 can be pulled back manually or automatically (e.g., under the control of a drive unit 110, etc.). The series of IVUS images captured between distal end 204 and proximal end 206 are generally referred to as images from an IVUS run.

[0062] Figure 3A and Figure 3B A two-dimensional (2D) representation of an IVUS image of vessel 202 is shown. For example, Figure 3A IVUS image 300a shows a longitudinal view of an IVUS record depicting a vessel 202 between the proximal end 206 and the distal end 204.

[0063] Figure 3B Image frame 300b depicts a coaxial (or short-axis) view of vessel 202 at point 302. In other words, image frame 300b is a single frame or single image from a series of IVUS images that can be captured between distal 204 and proximal 206, as described herein. As introduced above, physicians typically capture IVUS runs (e.g., series of IVUS images) at different stages of treatment. For example, IVUS images can be captured before percutaneous coronary intervention (PCI) and after PCI procedures (e.g., stent placement, balloon dilation, rotational atherectomy, etc.).

[0064] Furthermore, another series of images of vessel 202 can be captured using a non-IVUS imaging modality (e.g., simultaneously captured with mixed catheters, or captured at another time with different catheters). This invention provides for adapting features from a non-IVUS imaging modality to an IVUS imaging modality and incorporating the adapted features into the IVUS images. It should be understood that although the description discusses adapting non-IVUS imaging features to the IVUS imaging domain, the opposite can also be achieved based on this invention. For example, IVUS imaging features can be adapted to the OCT domain, and the adapted features can be used to enhance OCT images.

[0065] Figure 4 A multi-image modality adaptation system 400 according to some embodiments of the present invention is illustrated. Typically, the multi-image modality adaptation system 400 is a system that adapts an image from one intravascular imaging modality using features from an image from another imaging modality. Furthermore, the multi-image modality adaptation system 400 can be configured to display the adapted image, for example, in a GUI. The multi-image modality adaptation system 400 can be implemented in commercial IVUS guidance or navigation systems, such as, for example, the AVVIGO® guidance system available from Boston Scientific®. The present invention provides advantages over existing or conventional intravascular imaging and guidance systems because it provides composite images including indications of features from multiple imaging modalities. For example, the present invention can adapt IVUS images (which have better vessel wall penetration and larger vessel morphology than other modalities) with features from another modality (e.g., OCT) that are better captured by another modality (e.g., OCT). It should be noted that... Figure 4 The example embodiments depict a system arranged to adapt IVUS images using features from an OCT imaging modality. However, the claims are not limited in this respect, and systems similar to the multi-image modality adaptation system 400 can be implemented to adapt OCT images using features from an IVUS imaging modality.

[0066] In some embodiments, the multi-image modality adaptation system 400 may be implemented as part of the control system 104. Alternatively, the control system 104 may be implemented as part of the multi-image modality adaptation system 400. As depicted, the multi-image modality adaptation system 400 includes a computing device 404. Optionally, the multi-image modality adaptation system 400 includes an intravascular imaging system 100 and a display 406.

[0067] The computing device 404 can be any of a variety of computing devices. In some embodiments, the computing device 404 may be incorporated into and / or implemented by the console of the display 406. In some embodiments, the computing device 404 may be a workstation or server communicatively linked to the computing device 404 and / or the display 406. In other embodiments, the computing device 404 may be provided by a cloud-based computing device, such as a compute-as-a-service system accessible via a network (e.g., the Internet, an intranet, a wide area network, etc.). The computing device 404 may include a processor 408, a memory 410, input and / or output (I / O) devices 412, a network interface 414, and intravascular imaging system acquisition circuitry 416.

[0068] The intravascular imaging system 402 may be an intravascular imaging system configured to generate intravascular images of a specific imaging modality (e.g., intravascular imaging system 100), or it may be an intravascular imaging system having a hybrid catheter arranged to generate intravascular images of multiple modalities. In one example embodiment, the intravascular imaging system 402 may include an IVUS imaging system (e.g., intravascular imaging system 100), an OCT imaging system, or a hybrid IVUS / OCT imaging system.

[0069] Processor 408 may include circuitry or processor logic, such as, for example, any of a variety of commercial processors. In some examples, processor 408 may include multiple processors, multi-threaded processors, multi-core processors (whether multiple cores coexist on the same die or on separate dies), and / or some other kind of multiprocessor architecture, where multiple physically separate processors are linked together in some way by the aforementioned multiprocessor architecture. Additionally, in some examples, processor 408 may include a graphics processing section and may include dedicated memory, multi-threaded processing, and / or some other parallel processing capabilities. In some examples, processor 408 may be an application-specific integrated circuit (ASIC) or a field-programmable integrated circuit (FPGA).

[0070] Memory 410 may include logic, a portion of which includes an array of integrated circuits to form non-volatile memory for persistent storage of data or a combination of non-volatile and volatile memory. It should be understood that memory 410 may be based on any of a variety of technologies. In particular, the array of integrated circuits included in memory 120 may be arranged to form one or more types of memory, such as, for example, dynamic random access memory (DRAM), NAND memory, NOR memory, etc.

[0071] I / O device 412 can be any of a variety of devices for receiving input and / or providing output. For example, I / O device 412 may include a keyboard, mouse, joystick, foot pedal, display, touch-enabled display, haptic feedback device, LED, etc.

[0072] Network interface 414 may include logical and / or features for supporting communication interfaces. For example, network interface 414 may include one or more interfaces operating according to various communication protocols or standards for communication via direct or network communication links. Direct communication may occur via communication protocols or standards described in one or more industry standards (including their successors and variations). For example, network interface 414 may facilitate communication via buses such as, for example, Peripheral Component Interconnect High Speed ​​(PCIe), Non-Volatile Memory High Speed ​​(NVMe), Universal Serial Bus (USB), System Management Bus (SMBus), SAS (e.g., Serial Attached Small Computer System Interface (SCSI)) interfaces, Serial AT Attachment (SATA) interfaces, etc. Additionally, network interface 414 may include logical and / or features for enabling communication via various wired or wireless network standards (e.g., the 502.11 communication standard). For example, network interface 414 may be configured to support wired communication protocols or standards such as Ethernet, etc. As another example, network interface 414 can be configured to support wireless communication protocols or standards, such as, for example, Wi-Fi, Bluetooth, ZigBee, LTE, 5G, etc.

[0073] The intravascular imaging system acquisition circuit 416 may include circuitry, including custom-made or specially programmed circuitry, configured to receive or receive and transmit signals between the intravascular imaging systems 402, including IVUS operation, a series of IVUS images, or indications of frames or multiple frames of IVUS images.

[0074] Memory 410 may include instructions 418. During operation, processor 408 may execute instructions 418 to cause computing device 402 to receive (e.g., from an IVUS imaging system such as an intravascular imaging system 404) a series of IVUS images from a single “run” through the blood vessel and store the recordings as IVUS images 420 in memory 410. For example, processor 408 may execute instructions 418 to receive information elements from intravascular imaging system 404, including indications of IVUS images captured by catheter 102 as it is pulled back from distal 204 to proximal 206, including indications of the anatomy and / or structures (including vessel walls and plaques) of blood vessel 202. It should be understood that IVUS images 420 may be stored in various image formats or even non-image formats or data structures that include indications of blood vessel 202. In addition, IVUS image 420 will include images or representations that can be used to form blood vessel 202 when collinear, such as, for example, several “frames” or individual images represented by IVUS image 300a.

[0075] This invention provides for generating and extracting non-IVUS image modality features from IVUS image 420. For example, processor 408 can execute instructions 418 to generate non-IVUS image features 422 from IVUS image 420. Furthermore, this invention provides for adapting non-IVUS image features 422 to an IVUS imaging modality and integrating them into IVUS image 420 to form an IVUS image adapted with non-IVUS image features 424. Processor 408 can execute instructions 418 to adapt non-IVUS image features 422 to an IVUS imaging modality and generate an IVUS image adapted with non-IVUS image features. Typically, a machine learning (ML) model 426 can be used to derive the IVUS image adapted with non-IVUS image features 424 from IVUS image 420. It should be noted that although the non-IVUS image feature 422 is depicted and described as being generated and stored in memory 410, in some embodiments, the ML model 426 may be trained and configured to generate an IVUS image adapted with the non-IVUS image feature 424 based on the IVUS image 420, and the non-IVUS image feature 422 may be derived during this process but not independently generated or stored in memory 410.

[0076] In addition, the processor 408 may execute instructions 418 to generate a GUI 428 including instructions for an IVUS image adapted with features from the non-IVUS image 424, and to display the GUI 428 on the display 406.

[0077] It should be noted that although a single ML model 426 is depicted, some embodiments may provide an ML model 426 with a model architecture and structure that includes multiple discrete ML models. This will be described in more detail below (e.g., Figure 6 (and thereafter), wherein the remainder of the invention turns to various systems and methodologies for training ML models 426, which may include deep generative model training methodologies.

[0078] Figure 5 A logic flow 500 according to some embodiments of the present invention is illustrated, which is used to generate and extract features from another imaging modality based on an image of one imaging modality, adapt them to the imaging modality, and form a composite image based on the image and the adapted features. The logic flow 500 can be implemented by a multi-image modality adaptation system 400, and for clarity of representation, will be described with reference to a multi-image modality adaptation system 400. However, it is worth noting that the logic flow 500 can also be implemented by an intravascular imaging system different from the multi-image modality adaptation system 400. Furthermore, as described above, the example provides generating, extracting, and adapting non-IVUS image features based on an IVUS image, and generating a composite IVUS image including the original IVUS image and the adapted non-IVUS image features. However, this is not limiting; for example, the invention can be implemented to generate, extract, and adapt IVUS image features based on a non-IVUS image, and generate a composite image including the adapted IVUS image features and the non-IVUS image.

[0079] Logic flow 500 may begin at block 502. At block 502, “Processor receives first series of intravascular images of a patient’s vessel, the first series of intravascular images being a first imaging modality,” a first series of intravascular images of the patient’s vessel may be received. For example, logic flow 500 at block 502 may allow the processor to receive images captured via an IVUS catheter percutaneously inserted into the patient’s vessel. Processor 408 may execute instruction 418 to receive information elements including indications of IVUS image 420 from intravascular imaging system 404, which may include an IVUS catheter (e.g., catheter 102, etc.) because the IVUS catheter has been (or was) percutaneously inserted into vessel 202. It should be understood that IVUS image 420 may include image frames representing images captured as catheter 102 is pulled back from distal end 204 to proximal end 206. Processor 408 may execute instruction 418 to receive information elements including indications of IVUS image 420 from intravascular imaging system 404 or directly from catheter 102 (as applicable).

[0080] Continuing to block 504, "Image features of a second intravascular imaging modality are generated by the processor based on a first series of intravascular images of the first imaging modality," image features of the second intravascular imaging modality can be generated based on a series of intravascular images of the first imaging modality. For example, logic flow 500 at block 504 can generate non-IVUS image features (e.g., OCT imaging modality features, etc.) from the series of IVUS images received at block 502. Processor 408 can execute instruction 418 to generate non-IVUS image features 422 based on IVUS image 420.

[0081] Continuing to box 506, “Enhancing a First Series of Intravascular Images with Features from a Second Intravascular Imaging Modus by a Processor,” the first series of intravascular images can be enhanced with features from a second intravascular imaging modality. For example, processor 408 can execute instruction 418 to generate an IVUS image 424 adapted with non-IVUS image features based on IVUS image 420 and non-IVUS image features 422. In some embodiments, processor 408 can execute instruction 418 such that ML model 426 derives the IVUS image 424 adapted with non-IVUS image features from IVUS image 420 and non-IVUS image features 422. It should be noted that in some embodiments, boxes 504 and 506 can be combined; more precisely, ML model 426 can be used to directly derive the IVUS image 424 adapted with non-IVUS image features from IVUS image 420, and the generation of non-IVUS image features 422 may be invisible or transparent to the user or computing system executing ML model 426. For example, in some embodiments, ML model 426 may be configured to generate (e.g., derive, etc.) a non-IVUS image based on the IVUS image, and to transform features from the generated non-IVUS image to the IVUS image domain. Furthermore, ML model 426 may be configured to integrate the transformed features into the IVUS image.

[0082] Continuing to box 508, “Generating a graphical user interface including indications of an enhanced first series of intravascular images,” a GUI can be generated, wherein the GUI includes indications of an enhanced first series of intravascular images. For example, processor 408 can execute instruction 418 to generate GUI 428, as discussed above. As a concrete example, processor 408 can execute instruction 418 to generate GUI 428, which includes graphical indications of IVUS image 424 adapted with features from non-IVUS images.

[0083] As previously described, in some embodiments, the processor 408 of computing device 402 may execute instructions 418 to generate an IVUS image adapted with non-IVUS image features 424 using ML model 426. Generally, the present invention provides the use of deep generative models to generate and extract non-IVUS imaging modality features from IVUS image 420, and to adapt and integrate the extracted features into IVUS image 420. The present invention provides the use of deep generative models to train (e.g., with paired and / or co-registered data, with unpaired data, etc.) a model (e.g., ML model 426). These different methods are covered independently. However, it should be understood that concepts from one training method can be combined with concepts from another training method. Furthermore, although several training methods are described in detail below, it should be understood that ML model 426 can also be developed and trained using methods not described in detail herein.

[0084] to this end, Figure 6 A various embodiment of the ML training environment 600 according to the present invention is illustrated. The ML training environment 600 can be implemented to train ML models to convert intravascular images from one modality to another (e.g., IVUS to OCT, OCT to IVUS, etc.). The ML training environment 600 may include an ML training system 602, such as computing devices that apply ML algorithms to learn the relationship between inputs and derived outputs. The ML training system 602 may utilize experimental data 608 collected during several previous surgeries. The experimental data 608 may include intravascular images from a first (e.g., IVUS) and a second (e.g., OCT) modality from several patients. The experimental data 608 may be deployed locally to the ML training system 602 (e.g., stored in storage device 610 of the ML training system 602), may be deployed remotely outside the ML training system 602 and accessed via a network interface 604, or may be a combination of local and remote data.

[0085] Experimental data 608 can be used to form training data 612. As described above, the ML training system 602 may include a storage device 610, which may include a hard disk drive, a solid-state storage device, and / or random access memory. The storage device 610 may hold the training data 612. Typically, the training data 612 may include information elements or data structures that include indications of intravascular image modality A 614 and intravascular image modality B 616a. In some embodiments, intravascular image modality A 614 may be paired with intravascular image modality B 616a. In other embodiments, intravascular image modality A 614 and intravascular image modality B 616a may not be paired. As used herein, the term "paired" is intended to mean that intravascular image modality A 614 and intravascular image modality B 616a are co-registered. That is, intravascular image modality A 614 and intravascular image modality B 616a may include a series of images (e.g., a set of frames) for multiple patients, wherein each series of images includes multiple frames. Frames in a series of images in intravascular image modality A 614 for each patient can be co-registered (e.g., mapped or paired) with frames in a series of images in intravascular image modality B 616a for each corresponding patient.

[0086] As previously described, this invention envisions deep generative models, such as GANs or deep diffusion models, for training ML models. Typically, a GAN framework consists of two networks: a generative network that transforms noise vectors into realistic samples, and a discriminator network that classifies samples as real or fake. The training process for these two networks is a minimax game, as the generator's goal is to "fool" the discriminator. GANs can be applied to transform images from one domain to another. Therefore, typically, an ML model 622 will include multiple networks. Depending on the application, different types of models can be used to form the basis of the networks in the ML model 622. For example, in this example, an artificial neural network (ANN) or a convolutional neural network (CNN) might be particularly well-suited for learning associations between images from different domains as described in detail herein.

[0087] Any suitable training algorithm 618 can be used to train the model within ML model 622. However, Figure 6The examples depicted may be particularly well-suited for both supervised and unsupervised training algorithms. For supervised training, the ML training system 602 can apply training data 612 as input along with its annotations, allowing the ML model 622 to generate the desired output (e.g., an intravascular image modality A adapted with features from modality B 626). In some embodiments, the training data 612 can be divided into “training” and “test” data, where a subset of the training data 612 can be used to tune the network in the ML model 622 (e.g., the model’s internal weights, etc.), while another non-overlapping subset of the training data 612 can be used to measure the accuracy of the ML model 622 in inferring (or generalizing) the output based on “unseen” inputs.

[0088] The network in the ML model 622 can be applied using processor circuitry 606, which may include suitable hardware processing resources for executing the logic and structure in storage device 610. The development of training algorithm 618 and / or the trained ML model 622 may depend at least in part on hyperparameters 620. In an exemplary embodiment, model hyperparameters 620 may be automatically selected based on logic 624, which may include any known hyperparameter optimization techniques to suit the selected ML model 622 and the training algorithm 618 to be used. In an alternative embodiment, the ML model 622 may be retrained over time to incorporate new knowledge and / or updated experimental data 608.

[0089] Furthermore, as previously mentioned, the ML model 622 can be a GAN. Therefore, a GAN-specific network architecture and training algorithm can be employed. Details of such a network architecture and training algorithm will be discussed below. However, other details can be provided by those skilled in the art. Once the ML model 622 is trained, it can be applied (e.g., via processor 408, etc.) to new input data (e.g., intravascular images of different modalities). The data format input to the ML model 622 can be replicated in the same manner as the training data 612 provided to the ML model 622. The ML model 622 can generate outputs (e.g., intravascular images of one modality augmented with features from another modality, as discussed herein).

[0090] Furthermore, as mentioned earlier, ML model 622 can be a generative diffusion model. Therefore, network architectures and training algorithms specific to generative diffusion models can be employed. The diffusion model is part of a deep generative model, involving a forward diffusion stage and a reverse denoising stage. The forward diffusion stage gradually alters the input image by iteratively adding Gaussian noise. In the reverse denoising stage, the model learns to recover the input image by iteratively reversing the diffusion process. A recent review of generative diffusion models on images describes the application details: "Diffusion Models in Vision: A Review" by Croitoru FA, Hondru V, Ionescu RT, and Shah M., published in the March 2023 issue of the IEEE Transactions on Pattern Analysis and Machine Intelligence, which is incorporated herein by reference in its entirety.

[0091] Figure 7 An ML model architecture 700 according to some embodiments of the present invention is illustrated. In some embodiments, the ML model architecture 700 may be provided as ML model 622 or ML model 426. Typically, in addition to segmenting the network with a hybrid real label mask, the ML model architecture 700 also utilizes a MedGAN architecture for image-to-image translation and trains these networks as an auxiliary task to preserve morphologically important structures in the translated image, thereby generating enhanced image features from the translated image. An example MedGAN network is described in "MedGAN: Medical Image Translation Using GANs," published in the August 2015 issue of the *LaTeX-like Document Journal*, Volume 14, Issue 8, which is incorporated herein by reference in its entirety.

[0092] The ML model architecture 700 may include a generator network 702, which is trained to receive intravascular images of a first modality (e.g., intravascular image modality A 614, IVUS images, etc.) as input, and the output will be images of a second, different modality that resemble the images of the first modality (e.g., intravascular image modality A converted to modality B 616b, OCT images that resemble IVUS images, etc.). Therefore, given the source domain... and target domain The input to the generator network 702 can be limited to And the output is In some examples, the generator network 702 has a generator architecture that is arranged to be progressively refined via encoder-decoder boxes according to... (For example, intravascular image modality A 614) generation (For example, intravascular image modality A converted to modality B 616b).

[0093] The ML model architecture 700 may also include a feature extractor network 704, a discriminator network 706, and a segmenter network 708, which are arranged to compute style and content losses, perceptual and adversarial losses, and segmentation losses, respectively. These five losses form a loss function to update the model weights through backpropagation. The discriminator network 706 is trained to distinguish between intravascular image modality B 616a and intravascular image modality A converted to modality B 616b, and generates adversarial losses, as well as serving as a trainable feature extractor whose hidden layers are used to compute modified perceptual losses.

[0094] Feature extractor network 704 is trained to extract and transfer style and content from intravascular image modality B 616a and intravascular image modality A converted to modality B 616b to intravascular image modality A adapted with features from modality B 710. Feature extractor network 704 is used to extract deep, rich features. To calculate the style transfer loss, so as to output the style, texture and content that match the target.

[0095] The segmenter network 708 is trained to transform image segmentation features from a source modality (e.g., IVUS) and combine them with images from a target modality (e.g., OCT). Therefore, the ML model architecture 700 provides an ML architecture where features from multiple imaging modalities can be combined to preserve modality-specific features (e.g., lumen / vessel boundaries based on IVUS and calcium segmentation masks based on OCT). The segmenter network 708 is used to preserve the geometry of the features as they are transformed, extracted, and combined into an intravascular image modality A adapted with features from modality B 710. Specifically, the input to the segmenter network 708 can be limited to… And the output is .

[0096] It can be envisioned that the ML model architecture 700 will utilize a non-adversarial loss from the auxiliary task (e.g., the segmenter network 708) during the training of the generator network 702. Therefore, the loss function of the ML model architecture 700 can be expressed as: in, , and It's a hyperparameter.

[0097] Figure 8An ML model architecture 800 according to some embodiments of the present invention is illustrated. In some embodiments, the ML model architecture 800 may be provided as ML model 622 or ML model 426. Typically, the ML model architecture 800 utilizes a convolutional neural network (CNN) encoder trained with adversarial loss to construct a domain-invariant embedding space, which can be used to transform information from a source image modality to a target image modality. Typically, the input to the ML model architecture 800 will be an image from a first image modality, and the output will be an enhanced segmentation mask with features adapted from both the first and second image modalities.

[0098] The ML model architecture 800 may include an encoder 802, which is itself a CNN 804, and the CNN 804 is arranged to receive first and second source data. and ,in It is the first image modality (e.g., OCT), and It is the segmentation mask (e.g., calcification mask, etc.) of the image in the first image modality, and It is the second image modality (e.g., IVUS), and It is a segmentation mask of the image of the second image modality (e.g., cavity / vascular boundaries, etc.). Therefore, encoder 802 is configured to learn domain-invariant representations (F) of the source and target images. ML model architecture 800 also includes discriminator F 806, which can be a classifier network configured to classify the domain of the encoded input F(x).

[0099] Additionally, the ML model architecture 800 provides a segmentation network, an auxiliary edge detection network, and a discriminator to train the network based on adversarial and non-adversarial losses. The ML model architecture 800 includes a mask decoder 808, an edge decoder 810, and an edge discriminator 812. The mask decoder 808 is configured to generate a hybrid segmentation mask. The edge decoder 810 is configured to generate a hybrid edge mask. Furthermore, the edge discriminator 812 is configured to classify the domains of the edge map output by the edge decoder 810 (e.g., ).

[0100] In some embodiments, the ML model architecture 800 can use an objective function. Training was conducted, among which yes And it is associated with the discriminator F806. yes ,and yes Both of these are associated with the mask decoder 808; and yes And associated with edge discriminator 812; and where and These are hyperparameters. It should be understood that the ML model architecture 800 is configured with a mask containing the true labels. and The training is performed using paired images of different modalities.

[0101] Figure 9 An ML model architecture 900 according to some embodiments of the present invention is illustrated. In some embodiments, the ML model architecture 900 may be provided as ML model 622 or ML model 426. Typically, the ML model architecture 900 utilizes a recurrent generative adversarial network (Recurrent-GAN). A typical Recurrent-GAN network consists of two generators and two discriminators that perform image transformation between two image modalities without using paired source and target images. This provides bidirectional image transformation between the source and target domains. One generator transforms the source image into the target image, and the discriminator distinguishes between the generated target image and the real target image. The second generator transforms the target domain image into the source domain image, and the second discriminator classifies the generated source image as real or fake.

[0102] In this way, the generator will attempt to generate more realistic images to counter the discriminator. Recurrent consistency loss, adversarial loss, and non-adversarial loss can be used to train the recurrent-GAN network. It should be understood that an ML model architecture can be implemented to train the network to generate mixed images that include indications from multiple image modalities, where paired images cannot be used to train the network.

[0103] The ML model architecture 900 can operate on unpaired source images 902a (e.g., IVUS images, OCT images, etc.) and target images 904a (e.g., OCT images, IVUS images, etc.), where the source images 902a and target images 904a are images from different domains or modalities. The ML model architecture 900 may include a generator 906 and a source discriminator 908, as well as a target-to-source generator 910 and a target discriminator 912.

[0104] The source-to-target generator 906 can be configured to generate a source-paired target image 902b based on the source image 902a, while the target-to-source generator 910 can be configured to generate a target-paired source image 904b based on the target image 904a. The source discriminator 908 and the target-to-source generator 910 can be configured to classify the source-paired target image 902b and the target-paired source image 904b as real or fake, respectively.

[0105] A hybrid label mask can be generated for both the source image 902a and the target image 904a to be used as the ground truth during training. Once the ML model architecture 900 is trained, it can be used to generate images for training a network similar to the ML model architecture 800. Therefore, the objective function of the ML model architecture 900 can be constrained as follows: in, , , , , as well as As defined above regarding the ML model architecture 800.

[0106] Figure 10 A computer-readable storage medium 1000 is illustrated. The computer-readable storage medium 1000 may include any non-transitory computer-readable or machine-readable storage medium, such as optical, magnetic, or semiconductor storage media. In various embodiments, the computer-readable storage medium 1000 may include an article of manufacture. In some embodiments, the computer-readable storage medium 1000 may store computer-executable instructions 1002, which circuitry (e.g., processor 106, processor 408, etc.) can execute. For example, the computer-executable instructions 1002 may include instructions for implementing operations described with respect to instructions 418 and / or logic flow 500. Examples of the computer-readable storage medium 1000 or machine-readable storage medium may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, and so on. Examples of the computer-executable instructions 1002 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc.

[0107] Figure 11 A diagram is shown of a machine 1100 in the form of a computer system, within which a set of instructions can be executed to cause the machine to perform any or more of the methodologies discussed herein. More specifically, Figure 11 A diagram illustrates a machine 1100 in an example form employing a computer system, within which instructions 1108 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1100 to perform any or more of the methodologies discussed herein. For example, instruction 1108 may cause machine 1100 to execute... Figure 5The logical flow 500, etc. More typically, instruction 1108 enables machine 1100 to extract features from the image of the first modality and integrate them into the image of the second modality.

[0108] Instruction 1108 transforms a general, non-programmable machine 1100 into a specific machine 1100 programmed to perform the described and illustrated functions in a particular manner. In alternative embodiments, machine 1100 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1100 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1100 can be, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, network devices, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing instructions 1108 specifying the actions to be taken by machine 1100. Furthermore, although only a single machine 1100 is shown, the term "machine" should also be considered as a collection of machines 1100 that individually or jointly execute instructions 1108 to perform any one or more of the methodologies discussed herein.

[0109] Machine 1100 may include processor 1102, memory 1104, and I / O components 1142, which may be configured to communicate with each other, for example, via bus 1144. In one example embodiment, processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1106 and processor 1110 capable of executing instructions 1108. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Figure 11 Multiple processors 1102 are shown, but machine 1100 may also include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0110] Memory 1104 may include main memory 1112, static memory 1114, and memory cell 1116, all of which may be accessed by processor 1102, for example, via bus 1144. Main memory 1104, static memory 1114, and memory cell 1116 store instructions 1108, which 1908 embodies any one or more of the methods or functions described herein. During execution of instructions 1108 by machine 1100, instructions 1908 may also reside wholly or partially in main memory 1112, static memory 1114, machine-readable medium 1118 within memory cell 1116, at least one of processors 1102 (e.g., within the processor's cache), or any suitable combination thereof.

[0111] I / O component 1142 may include a wide variety of components to receive input, provide output, generate output, transmit information, exchange information, capture measurement results, and so on. The specific I / O component 1142 included in a particular machine will depend on the type of machine. For example, a portable machine, such as a mobile phone, may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 1142 may include... Figure 11 Many other components are not shown. The I / O components 1142 are grouped according to function only for the sake of simplicity in the discussion below, and this grouping is by no means limiting. In various example embodiments, the I / O components 1142 may include output components 1128 and input components 1130. Output components 1128 may include visual components (e.g., displays, such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 1130 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other orientation devices), touch input components (e.g., physical buttons, touchscreens providing position and / or force for touch or touch gestures, or other touch input components), audio input components (e.g., microphones), etc.

[0112] In another example embodiment, I / O component 1142 may include biometric component 1132, motion component 1134, environmental component 1136 or position component 1138, and various other components. For example, biometric component 1132 may include components for detecting facial expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and recognizing a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1134 may include accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1136 may include, for example, an illuminance sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting hazardous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 1138 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0113] Communication can be implemented using various technologies. I / O component 1142 may include a communication component 1140 operable to connect machine 1100 to network 1120 or device 1122 via connectors 1124 and 1126, respectively. For example, communication component 1140 may include a network interface component or another suitable device that interfaces with network 1120. In further examples, communication component 1140 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components that provide communication via other means. Device 1122 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device connected via USB).

[0114] Furthermore, the communication component 1140 may detect identifiers or include components operable to detect identifiers. For example, the communication component 1140 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCCRSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be derived via the communication component 1140, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting NFC beacon signals that can indicate a specific location, and so on.

[0115] Various memories (i.e., memory 1104, main memory 1112, static memory 1114, and / or the memory of processor 1102) and / or storage units 1116 may store one or more sets of instructions and data structures (e.g., software) that embody or are utilized by any one or more of the method frameworks or functions described herein. These instructions (e.g., instruction 1108) cause various operations to implement the disclosed embodiments when executed by processor 1102.

[0116] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and are used interchangeably in this invention. The terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions and / or data. Therefore, the terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, such as semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium” discussed below.

[0117] In various example embodiments, one or more portions of network 1120 may be an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, Internet, a portion of PSTN, a common legacy telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 1120 or a portion thereof may include a wireless or cellular network, and connector 1124 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless connection. In this example, connector 1124 can implement any of a variety of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, 3rd generation partnership program (3GPP) (including 3G), fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), global microwave access interoperability (WiMAX), long-term evolution (LTE) standard, other standards defined by various standards-setting organizations, other remote protocols or other data transmission technologies.

[0118] Instruction 1108 may be transmitted or received via network 1120 using a transmission medium via a network interface device (e.g., a network interface device included in communication component 1140) and utilizing any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 1108 may be transmitted or received via a transmission medium via a connector 1126 to device 1122 (e.g., a peer connector). The terms "transmission medium" and "signal medium" refer to the same thing and are used interchangeably in this invention. The terms "transmission medium" and "signal medium" should be considered to include any intangible medium that can store, encode, or carry instructions 1108 for execution by machine 1100; and include digital or analog communication signals or other intangible media to facilitate communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" refers to a signal whose characteristics are set or altered in a manner to encode information in the signal.

[0119] The terms used in this document shall conform to their ordinary meaning in the relevant field, or the meaning indicated by their use in the context, unless a clear definition is provided.

[0120] In this document, references to “an embodiment” or “one embodiment” do not necessarily refer to the same embodiment, although they may refer to the same embodiment. Unless the context explicitly requires otherwise, throughout the specification and claims, the words “comprising,” “including,” etc., should be interpreted in an inclusive sense, rather than an exclusive or exhaustive sense; that is, meaning “including but not limited to.” Use of singular or plural terms also includes both the singular and plural, unless explicitly limited to one or more. Additionally, the words “this article,” “above,” “below,” and similar terms, when used in this application, refer to the entire application and not any part thereof. When a claim uses the word “or” to refer to a list of two or more items, the word covers all of the following interpretations: any one of the items in the list, all the items in the list, and any combination of the items in the list, unless explicitly limited to one or the other. Any term not explicitly defined herein has its conventional meaning as commonly understood by one of ordinary skill in the art.

Claims

1. A method for a computing device, comprising: The processor receives a first series of intravascular images of the patient's blood vessels, the first series of intravascular images being a first imaging modality; The processor generates image features of a second imaging modality based on the first series of intravascular images; The processor enhances the first series of intravascular images with the image features of the second series of imaging modalities; and The processor generates a graphical user interface that includes instructions for the enhanced first series of intravascular images.

2. The method of claim 1, further comprising displaying the graphical user interface on a display connected to the computing device.

3. The method according to claim 1 or 2, wherein generating the image features of the second imaging modality comprises: A second series of intravascular images of the patient's blood vessels are generated via a machine learning (ML) model, the second series of intravascular images constituting the second imaging modality; and The image features of the second imaging modality are generated from the first series of intravascular images via the ML model.

4. The method according to claim 3, wherein generating the image features of the second imaging modality based on the first series of intravascular images comprises: The first series of intravascular images are converted into the second imaging modality via the ML model to form a series of converted intravascular images, wherein the features of this series of converted intravascular images resemble the image features of the second imaging modality; and Features are extracted from the series of images via the ML model.

5. The method according to claim 3 or 4, wherein enhancing the first series of intravascular images with the image features of the second imaging modality further comprises generating a series of hybrid intravascular images via the ML model, the series of hybrid intravascular images comprising the first series of intravascular images and the image features of the second imaging modality.

6. The method according to any one of claims 3 to 5, wherein the ML model comprises a medical image generation network and an auxiliary task network, wherein the auxiliary task network is arranged to preserve the geometry of the extracted features.

7. The method of claim 6, wherein the ML model is trained using a plurality of series of intravascular images of the first modality, wherein the plurality of series of intravascular images of the first modality are paired or unpaired with corresponding series of intravascular images of the second modality.

8. The method of claim 7, wherein the medical image generation model is trained using a non-adversarial loss from the auxiliary task network.

9. The method according to any one of claims 3 to 5, wherein the ML model comprises an encoder network based on a convolutional neural network (CNN) and a first decoder network and a second decoder network.

10. The method of claim 9, wherein the CNN-based encoder network is configured to convert a series of intravascular images of the first imaging modality into a series of intravascular images of the second imaging modality, and to convert a series of intravascular images of the second imaging modality into a series of intravascular images of the first imaging modality.

11. The method of claim 10, wherein the first decoder network is arranged to extract features from the series of intravascular images derived from the first imaging modality.

12. The method of claim 11, wherein the second decoder network is arranged to extract features from the series of intravascular images derived from the second imaging modality.

13. The method according to any one of claims 9 to 12, wherein the ML model is trained with a plurality of series of intravascular images of the first modality, wherein the plurality of series of intravascular images of the first modality are paired or unpaired with corresponding series of a plurality of series of intravascular images of the second modality labeled with a real-label mask.

14. An apparatus comprising a processor associated with a memory including instructions executable by the processor, the processor being configured to be associated with an intravascular ultrasound (IVUS) imaging system and configured to execute the instructions, which, when executed, cause the processor to perform the method according to claim 1.

15. At least one machine-readable storage device comprising a plurality of instructions responsive to execution by a processor of an intravascular ultrasound (IVUS) imaging system, the plurality of instructions causing the processor to perform the method according to claim 1.

Images