Angiography image generation and display using multiple machine learning models

Abstract

Systems, methods, and computer program products for generating and displaying angiography images from multiple machine learning models that provide different sensitivity and specificity performance in the segmentation of vascular structures in an angiogram. A graphical user interface with control elements provides user control over the mixture of the multiple machine learning models and other settings, including an a pseudo brightness control and a zoom control. The displayed angiogram image is based on the settings of the control elements, and adjusts in response to changes thereto. The pseudo brightness control controls the mixture of the high specificity model versus the high sensitivity model in the displayed image. The zoom control modifies the magnification of the displayed image and increases the proportional mixture of the high sensitivity model in the displayed image. A widget or other type of control element may also be provided for controlling the mixture of the empirical data versus machine learning model output in the displayed image.

Description

Attorney Docket No.: AWI0012WO ANGIOGRAPHY IMAGE GENERATION AND DISPLAY USING MULTIPLE MACHINE LEARNING MODELS CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S Provisional Patent Application No. 63 / 689,308, filed August 30, 2024 and titled DISPLAY OF ANGIOGRAPHY IMAGES FROM PARAMETERIZED DEEP LEARNING MODELS, which is incorporated by reference herein in its entirety. FIELD OF THE INVENTION

[0002] The present invention relates generally to systems, methods, and computer program products for generating and displaying angiographic images using multiple machine learning models.

[0003] The heart sends blood to the body as a sequence of arterial stroke volumes. The pumped blood crosses the capillaries to the veins and returns to the heart. The presence and motion of blood in the blood vessels can be dynamically imaged with a technique called angiography. In fluoroscopic x-ray angiography, a chemical contrast agent that attenuates x-rays greater than blood or tissue is injected into the vascular system in coordination with a sequence of x-ray images. The chemical (typically iodinated) contrast travels through the vasculature, blocks the passage of the x-rays at a given frame rate, and this spatiotemporal x-ray attenuation pattern creates a sequence of images on an x-ray sensor. These angiographic images are typically two dimensional in space and one dimensional in time.

[0004] In standard practice, the image quality of the angiographic images in a study can be improved by increasing the dose of the injected chemical contrast and / or by increasing the dose of the fluoroscopic x-ray radiation. Such dosage increase(s) can increase the risk of harm to the patient or subject due to toxic side effects of the chemical contrast agent and / or x-ray radiation. For example, the chemical contrast agent can be toxic to the kidneys and produce a significant mass load on the vascular system, which can stress the heart and other vascular structures. Higher x-ray doses can directly injure various bio-Attorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOhigh sensitivity and another optimized for high specificity.Attorney Docket No.: AWI0012WO more processors, causes the one or more processors to perform one or more of the methods described herein.

[0015] In another aspect of the invention, a graphical user interface for an angiographic display system is provided that comprises an arrangement of one or more graphical control elements or widgets controllable by a user to combine the outputs of multiple machine learning models, each optimized for different aspects of angiography image analysis (e.g., in the case of a neural network model, by being trained with different loss functions). In certain embodiments, the graphical user interface may also include one or more graphical control elements or widgets controllable by the user to combine the raw angiographic image from an imaging device with a mixture of images from the multiple machine learning models. In certain embodiments, the graphical user interface may also include one or more graphical control elements or widgets controllable by the user to provide a zoom feature that expands a selected portion of a displayed image. In certain embodiments, the graphical control elements or widgets are displayed on a display connected to the computer and controllable by the user via a user input device (e.g., a keyboard, a trackpad, a touchscreen, a mouse, or combinations thereof) connected to the computer. Such techniques can allow users to easily adjust and optimize the display of angiographic image data.

[0016] Other features and advantages of the invention will be apparent from the specification and drawings.

[0017] For a more complete understanding of the invention, reference is made to the following description and accompanying drawings, in which:

[0018] FIG. 1A shows a screen shot of a graphical user interface with user-adjustable control elements according to a first embodiment, wherein the control elements are set to particular settings to display a raw angiogram image;

[0019] FIG. 1B shows a screen shot of a graphical user interface with user-adjustable control elements according to the first embodiment, wherein the control elements are shown with particular settings to display a segmented image a generated by a machine learning model trained for maximum specificity;

[0020] Fig.1C shows a screen shot of a graphical user interface with user-adjustable control elements according to the first embodiment, wherein the control elements areAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO

[0031] The machine learning models can be any type of suitable machine learning models, such as convolutional neural network models, deep neural network models, deep convolutional neural network models, ViT models, or SVM models. In certain embodiments, all of the machine learning models comprise the same type of machine learning model. In other embodiments, at least one of the machine learning models is a first type of machine learning model and at least one of the other machine learning models is a second type of machine learning model that is different than the first type of machine learning model. An example of a deep neural network model suitable for use with the disclosed embodiments is U-Net (see Ronnenberger et al, “U-Net: Convolutional Networks for Biomedical Image Segmentation, Lecture Notes in Computer Science: Medical Image Computing and Computer-Assisted Intervention,” 2015, pp 234-241, incorporated by reference herein).

[0032] In certain embodiments, when training a machine learning model (such as a U- Net neural network model) for the spatiotemporal segmentation of angiography data, a so-called “label” image and a continuous sequence of angiographic image frames are supplied. The label image represents ground truth (e.g., a correctly segmented image) against which a machine learning models’s predictions are compared when learning. The label image may be drawn by hand by a human expert while inspecting the corresponding angiographic images. As described by U.S. Patent Application No.18 / 192,439 to Butler and U.S. Provisional Patent Application No.63 / 586,141 (both of which are incorporated by reference above), the angiographic image frame of interest can be in the middle of a sequence of angiographic image frames. This arrangement allows the machine learning model to learn the spatiotemporal structure of angiographic data. In one embodiment, the angiographic sequence consists of five frames, and the frame of interest (the target label frame) is the middle (e.g., third) frame.

[0033] In one embodiment of the invention, the spatial dimensions are represented by (512 x 512) pixels. In the label image, the background pixels have a value of ‘0’ and the vessel pixels have a value of ‘1’. A typical angiogram has more background pixels than vessel pixels. The ratio is typically about 15:1 background to vessel pixels.Attorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO widgets may be used to allow a user to control the weights of the image from each predicted model separately. In certain embodiments, three or four separately trained models may be utilized, and each may be assigned a corresponding user-controllable widget on a graphical user interface, whereby a user may be able to control the relative weight / strength / mixture of influence of each model. An illustration of the use of these widgets to achieve these results is further discussed below with respect to FIGS.1A-2C.

[0045] The alpha channel, m, controls the relative mixture in the representation of the two trained models, Mp and Ms. On the display screen, the image data can be thought of as being layered from farthest (e.g., distal) to nearest (e.g., proximal) from the human user in the order y, yp, and ys. The rendering scheme (RS) may be represented as Equation (3): 3RS

[0046] By way of example, if m is set to 1 at every pixel, then only is displayed. If m is set to 0 and ifdis set to 1 at every pixel, then only is displayed. If bothmanddare set to 0 at every pixel, then only the empirical data y is displayed. Such scenarios are delineated below in Table- 1: m d Dataset Displayed 0 0 y 0 0<d<1 0 1 0< m <1 0 0< m <1 0< d <1 0<m<1 1 101 0<d<1Attorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO

[0060] In the screenshot 20a shown in FIG. 2A, graphical control elements 26b-d are set to display the raw angiogram image 27a. In the screenshot 20b shown in FIG.2B, the control elements 26b-d are set to display an image 27b that is a combination of the raw image and a model-generated image generated by the high specificity model, with some panning also applied to the image. In the screenshot 20c in FIG.2C, the control elements 26b-d are set to display the same panning position as FIG.2B, but with a higher zoom factor, and coordinated higher contribution of information from the high sensitivity model.

[0061] Referring now to FIG.2A, the frame selector control element 26a is shown as a slider set to a position between left and right ends of the scroll bar, thus selecting an angiogram image from about halfway through the angiographic study. Raw versus neural network selector control element 26b is shown as a slider set all the way to the left on of the scroll bar so as to display only the raw image 27a in the display window 24 of graphical user interface 22. Zoom selector control element 26d is shown as a slider set all the way to the left on the scroll bar in order to not produce any zooming of image 27a within display window 24 of graphical user interface 22. Pan selector control element 26c is shown as a knob or button in a rectangular field representing the image. The knob or button may be translated in x and y directions within the rectangular field to select a section of the image to display. If the control element 26c is centered in the rectangular field, the entire image is displayed. If the control element 26c is translated away from the center of the rectangular field, only the section of the image corresponding to the position of the control element in the rectangular field is displayed. Pan control element 26c in FIG. 2A is centered, so there has been no panning of image 27 by the user in this example. This configuration provides no contribution from either the high specificity model or the high sensitivity model.

[0062] Referring now to FIG.2B, the frame selector control element 26a is set the same as in FIG.2A, but the raw versus neural network selector control element 26b is set to about 2 / 3 of maximum to show a blend of the raw image and information from one or more neural network models, and zoom control element 26d is set to minimum (i.e., at the far left of the scroll bar) to not produce any zooming of image 27b and to select information only from the high specificity model. Pan selector control element 26c has been moved upward relative to its location in FIG. 2A. The effect of these settings inAttorney Docket No.: AWI0012WO graphical user interface 22 is a segmented angiographic image 27b with high specificity that is panned upwardly.

[0063] In FIG.2C, the frame selector control element 26a, the raw versus neural network model control element 26b, and the pan selector control element 26c are set similarly as in FIG. 2B. However, zoom selector control element 26d is shifted to the right along the horizontal scroll bar. This causes a coupled increase in magnification and an increased contribution of the high sensitivity net model, thus generating a segmented angiographic image 27c of higher sensitivity that is also zoomed-in.

[0064] In the example embodiments depicted in FIGS.1A-2C, display of the control elements of the graphical user interfaces above the image display window allows the control elements to be separately enumerated and viewed by the user. However, it will be appreciated that the control elements described herein may appear in other forms, positions, and orientations on screen. Additionally or alternatively, control elements described herein may be based on direct manipulation of physical control elements. For example, instead of a two-dimensional slider widget to control brightness, a middle finger roll wheel and a computer mouse may be utilized. Instead of a two-dimensional slider widget to control panning, in certain embodiments, a user may employ a mouse and the left click button to drag the image. Instead of a widget to control zoom, a user may press the shift key and rotate the middle finger mouse wheel to apply the zoom metaphor. In certain embodiments, the control elements / widgets may be physical knobs, sliders, and / or buttons configured to adjust the degree of specificity, sensitivity, zoom, image number, panning, etc., in the image panel displayed. In other embodiments, a combination of graphical control elements (e.g., widgets) and physical control elements (e.g., knobs, sliders, buttons, and the like) may be utilized.

[0065] Referring to FIGS.3, 4, and 5, exemplary systems or devices that may be employed for carrying out embodiments of the invention are illustrated. It will be understood that such systems and devices are only exemplary of representative systems and devices, and that other hardware and software configurations are suitable for use with embodiments of the invention. Thus, the embodiments are not intended to be limited to the specific systems and devices illustrated herein, and it will be recognized that other suitable systems and devices can be employed without departing from the spirit and scope of the subject matter provided herein.Attorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO

[0070] X-ray source assembly 32 contains an x-ray source which emits a beam of x-rays when energized. The center ray passes through system isocenter 56 and impinges on a two-dimensional flat panel digital detector 58 housed in x-ray detector array assembly 34. The two-dimensional flat panel digital detector 58 may be, for example, a 2048 x 2048 element two-dimensional array of detector elements. Each element produces an electrical signal that represents the intensity of an impinging x-ray, and hence the attenuation of the x-ray as it passes through the patient. During a scan, x-ray source assembly 32 and x-ray detector array assembly 34 are rotated about system isocenter 56 to acquire x-ray attenuation projection data from different angles. In example devices, the detector array may be able to acquire a given number of (e.g., up to fifty) projections, or image frames, per second. The rate of image frames per second may determine how many image frames can be acquired for a prescribed scan path and speed.

[0071] Referring to FIG.4, the rotation of assemblies 32, 34 and the operation of the x- ray source are governed by a control mechanism 60 of the x-ray system. Control mechanism 60 includes an x-ray controller 62 that provides power and timing signals to x-ray source assembly 32. A data acquisition system (DAS) 64 in control mechanism 60 samples data from detector elements and passes the raw image data to an image reconstructor 65. Image reconstructor 65 receives digitized x-ray data from DAS 64 and may perform high speed image reconstruction according to the methods of the present disclosure. The reconstructed image is applied as an input to a computer 66 which stores the image in a mass storage device 69 or processes the image further. Image reconstructor 65 may be a standalone computer or may be integrated with computer 66. Image reconstructor 65 and / or computer 66 may include various modules, including neural network models, storage, and processors which together function to perform the various functionalities and methodologies described herein, including the methodology described below with respect to FIG.7. Additionally or alternatively, image reconstructor 65 and computer 66 may operate in communication and / or cooperation with additional systems, such as information processing device 80, described below with respect to FIG.5, and / or system 600, described below with respect to FIG.6.

[0072] Control mechanism 60 also includes gantry motor controller 67 and a C-axis motor controller 68. In response to motion commands from computer 66, motor controllers 67, 68 provide power to motors in the x-ray system that produce the rotations about horizontal pivot axis 46 and C-axis 48, respectively. Computer 66 also receivesAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO

[0079] User input devices 96 can include hardware (physical) and / or software (graphical) control elements configured to receive input from a user for processing by components of computer system 80. User input devices can include all possible types of devices and mechanisms for inputting information to computer system 84. These may include a keyboard, a keypad, a touch screen, a touch interface incorporated into a display, audio input devices such as microphones and voice recognition systems, and / or other types of input devices. In various embodiments, user input devices 96 may include a computer mouse, a trackball, a track pad, a joystick, a wireless remote, a drawing tablet, a voice command system, an eye tracking system, or the like. In some embodiments, user input devices 96 are configured to allow a user to select or otherwise interact with objects, icons, text, or the like that may appear on monitor 82 via a command, motions, or gestures, such as a click of a button or the like.

[0080] User output devices 94 can include hardware and / or software elements configured to output information to a user from components of computer system 80. User output devices can include all possible types of devices and mechanisms for outputting information from computer system 84. These may include a display (e.g., monitor 82), a printer, a touch or force-feedback device, audio output devices, or the like.

[0081] Communications interface 98 can include hardware and / or software elements configured to provide unidirectional or bidirectional communication with other devices. For example, communications interface 98 may provide an interface between computer system 84 and other communication networks and devices, such as via an internet connection.

[0082] In one example, information processing device 80 may obtain (e.g., via communications interface 98) a series of angiographic image frames (e.g., an angiogram) at a rate faster than cardiac frequency. For instance, information processing device 80 may use wavelet angiography (e.g., by employing complex-valued wavelet transforms) to generate a spatiotemporal reconstruction of cardiac frequency phenomena in an angiogram obtained at faster than cardiac frequency. Spatiotemporal reconstruction may be performed in accordance with techniques described in U.S. Patent No. 10,123,761, filed July 1, 2016, issued November 13, 2018, and titled DEVICE AND METHOD FOR SPATIOTEMPORAL RECONSTRUCTION OF A MOVING VASCULAR PULSE WAVE IN THE BRAIN AND OTHER ORGANS, which is hereby incorporated by reference herein in its entirety.Attorney Docket No.: AWI0012WO

[0083] Referring to FIG.6, an example computing environment 100 is shown for executing the functions and methodologies of the various embodiments of the invention described herein. Computing environment 100 may include a server 135 in communication with, via network 154, one or more of a client computer system 160, an imaging device 158, and a database 156. Server 135 and client computer system 160 may each include computer system / information processing device 80, described above with respect to FIG.5, or may include a different computer system. Imaging device 158 may include an angiogram imaging system in the form of rotational x-ray system 28 described above with respect to FIGS. 3 and 4, one or more components thereof, and / or a different imaging device. Database 156 may be remotely configured relative to server 135 of system 100, and in communication therewith via network 154.

[0084] Network 154 may include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and include wired, wireless, or fiber optic connections. In general, network 154 can be any combination of connections and protocols known in the art that will support communications between server 135 and database 156, client computer system 160, and imaging device 158 via their respective network interfaces.

[0085] It will be understood that the functional division among components of computing system 100 are not to be construed as a limiting example. In certain embodiments, system server 135 of computing system 100 includes a network interface (I / F) 136, a video controller 138, memory 140, one or more databases / storage 155, and at least one processor 137 in communication with network interface (I / F) 136, video controller 138, memory 140, and database 155. System 100 may also include or be in communication with a display 142 in operative communication with video controller 138 for displaying images, and a user input device 144 configured to provide one or more user inputs to processor 137. In certain embodiments, memory 140 comprises a non-transitory computer readable medium that stores instructions executable by the processor 137 to perform the functions and methodologies of the various embodiments of the invention described herein. For example, memory 140 may store a first machine learning model (Mp) 146, a second machine learning model (Ms) 148, an image generator module 150 for processing and adjusting image data in accordance with outputs from first and second machine learning models 146, 148 and inputs from user input device 144, and a graphical user interface module 152 configured to display a graphical user interface via videoAttorney Docket No.: AWI0012WO controller 138 and display 142. Graphical user interface module 152 may be configured to, for example, provide the various graphical user interfaces described above with respect to FIGS.1A-2C.

[0086] As described above, first machine learning model Mp146 may be a deep learning neural network model trained and optimized for high specificity to vessel pixels, and second machine learning model Ms 148 may be a deep learning neural network model trained and optimized for high sensitivity. User input device 144 may include a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, a thin client, or any programmable electronic device capable of executing computer readable program instructions.

[0087] Each of the modules and models stored in memory 140 of server 135 may include one or more sub modules or models to perform various example functions and methodologies of the present invention, be implemented by any combination of any quantity of software and / or hardware modules or units, and reside within memory 140 of server 135 for execution by a processor, such as processor 137. Database 155 and remote database 156 may include any non-volatile storage media known in the art. For example, databases 155, 156 can be implemented with a tape library, optical library, one or more independent hard disk drives, or multiple hard disk drives in a redundant array of independent disks (RAID). Similarly, image data in databases 155, 156 may conform to any suitable storage architecture known in the art, such as a file, a relational database, an object-oriented database, and / or one or more tables. The above described system and / or components thereof may be used to implement various methodologies of the present invention described herein, including method 200, described below with respect to FIG.7.

[0089] Each element in flowcharts shown or methodologies described herein, such as process 200 shown and described below with respect to FIG.7, depicts a step or a group of steps of novel computer-implemented systems and methodologies for displaying an angiographic image generated from multiple deep learning models in concert with one or more pre-set or user-adjustable parameters. Each step of methodologies described herein may contain one or more sub-steps. For purposes of illustration and explanation, these steps, as well as all other steps identified and described, are presented in a certain logical order. However, it will be appreciated that any exemplary embodiments described hereinAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO 100 based on a condition of the raw angiographic data of the particular angiographic frame.

[0097] At Step 218, processor 137 overlays the first machine learning model display data and the second machine learning display data to generate a mixture of display data corresponding to the particular angiographic image. At Steps 220, 222, processor 137 outputs and displays the mixture of display data corresponding to the particular angiographic image on a display in accordance with the various embodiments described above with respect to FIGS.1A-2C. For example, processor may overlay the machine learning models by aligning their respective pixels in the high specificity and high sensitivity model-generated images and adjusting an opacity of the respective pixels as described herein.

[0098] In addition to acquiring angiographic images, additional cardiac signals / data may be contemporaneously acquired to serve as a cross correlation target. For example, the cardiac signals / data may serve as a reference cardiac signal for phase indexing pixels in the angiographic projections.

[0099] The above description is for the purpose of teaching the person of ordinary skill in the art how to practice the subject of the present application, and it is not intended to detail all those modifications and variations of it which will become apparent to the skilled worker upon reading the description. For example, while examples of a graphical user interface are described herein for combining outputs of one or more machine learning models trained as described herein (e.g., for specificity and sensitivity, respectively), it will be appreciated that the outputs may be displayed using other graphical user interfaces. It will also be appreciated that one or more machine learning models trained as described herein may be provided for integration into an imaging device to post-process raw image data from the imaging device, or as a standalone product (e.g., an image post-processing system or a software-as-a-service product) configured to receive raw image data and provide images with adjustable attributes, such as specificity and sensitivity. Furthermore, while techniques are described for combining the outputs of multiple machine learning models trained to optimize or accentuate different attributes (e.g., specificity and sensitivity) of angiographic images, it will be appreciated that such techniques may also be used to combine the outputs of multiple machine learning models trained to optimize or accentuate attributes of other types of images (e.g., X-ray images of other organs, CT-images, or MRI-images). It willAttorney Docket No.: AWI0012WO additionally be appreciated that, instead of or in addition to neural network models, non- neural network image classifiers (such as Vision Transformers (ViT) and Support Vector Machines (SVMs)) may be used to generate images with desired attributes (such as one segmented image with high specificity and another segmented image with high specificity) from raw image data, and that such generated images may be combined and displayed as described herein. It is intende that all such modifications and variations be included within the scope of the present invention.

Claims

Attorney Docket No.: AWI0012WO What is claimed:

1. A method for displaying angiographic images on a display connected to a computer, the method comprising: obtaining, with the computer, a first angiographic image generated by a first machine learning model from angiographic data and a second angiographic image generated by a second machine learning model from the angiographic data, wherein the first machine learning model is configured to have greater sensitivity performance than the second machine learning model and wherein the second machine learning model is configured to have greater specificity performance than the first machine learning model; and displaying, with the computer via a display, a mixture of the first and second angiographic images generated by the first and second machine learning models.

2. The method of claim 1, wherein the first angiographic image comprises a first set of pixels and the second angiographic image comprises a second set of pixels, and wherein displaying the mixture of the first and second angiographic images comprises superposing the first and second sets of pixels.

3. The method of claim 2, further comprising adjusting, via the computer, the mixture of the first and second angiographic images by increasing or decreasing an opacity of one of the first and second sets of pixels relative to an opacity of the other of the first and second sets of pixels.

4. The method of claim 3, further comprising receiving, with the computer, an input from a user, and adjusting the mixture of the first and second angiographic images based on the input from the user.

5. The method of claim 4, wherein: the input from the user is received from a user-adjustable control element connected to the computer; the control element has a range of adjustment; and at one end of the range, an opacity of the first set of pixels is greater than anAttorney Docket No.: AWI0012WO opacity of the second set of pixels, and at another end of the range, the opacity of the second set of pixels is greater than the opacity of the first set of pixels.

6. The method of claim 5, wherein the control element is a mechanical control element or a graphical control element.

7. The method of claim 6, wherein the control element is a graphical control element, and further comprising displaying, with the computer via the display, a graphical user interface comprising the graphical control element.

8. The method of claim 7, wherein the graphical control element is adjustable with a pointing device connected to the computer.

9. The method of claim 1, wherein the first and second machine learning models are neural network models, the first machine learning model is trained with a first loss function, and the second machine learning model is trained with a second loss function that is different than the first loss function in terms of sensitivity and specificity performance.

10. The method of claim 1, wherein displaying further comprises mixing the angiographic data with the first and second angiographic images generated by the first and second machine learning models.

11. The method of claim 10, further comprising adjusting, with the computer, a mixture of the angiographic data and the first and second angiographic images based on an input from a user received by the computer.

12. A method for displaying angiography images from machine learning models, the method comprising: obtaining angiographic image data generated by utilizing two machine learning models with differing sensitivity and specificity performance; providing a graphical user interface comprising a first graphical control element for controlling a mixture of the two machine learning models;Attorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WOAttorney Docket No.: AWI0012WO

Images