Three-dimensional coronary artery tree reconstruction

Abstract

A system (100) for generating a three-dimensional image of a coronary artery tree (449) includes a memory (151) and a processor (152). The processor (152) is configured to: obtain a sequence of two-dimensional angiographic images (410) corresponding to the heart from a single viewpoint of the imaging device; and using the trained machine learning model (430) to generate a three-dimensional representation (211A) of the coronary artery tree (449) based on the sequence (410) of two-dimensional angiography images and the cardiac motion of the moving heart.

Description

Background Art

[0001] Two-dimensional angiographic imaging is commonly used to visualize the blood vessels of the human heart and examine the quality of blood supply to the human heart. Two-dimensional angiographic imaging allows for the acquisition of images with high spatial and temporal resolution. Two-dimensional angiographic imaging also enables real-time guidance during cardiac interventions. In two-dimensional X-ray imaging, the heart and coronary artery tree are projected onto a two-dimensional plane. However, due to occlusion, vessel overlap, foreshortening, cardiac motion, and / or lung motion, the two-dimensional nature of the visualization may limit the characterization of the desired vessels or lesions. Thus, the interpretation of two-dimensional angiographic images shows large inter-user variability.

[0002] Alternative imaging techniques also exist for vascular imaging. One such alternative imaging technique is three-dimensional computed tomography angiography (CTA) imaging, which can reduce the inter-user variability in interpretation. This improvement is due to the three-dimensional nature of the imaging data. Compared to the imaging data from two-dimensional angiographic imaging, the imaging data from three-dimensional computed tomography angiography is easier to interpret, more suitable for calculating hemodynamic measurements, and more accurate for diagnosing coronary artery disease and detecting lesions. However, compared to two-dimensional angiographic imaging, three-dimensional computed tomography angiography imaging generally also involves a longer scan time and an increased radiation exposure to the patient.

[0003] Thus, in order to combine the advantages of three-dimensional image interpretation, a great deal of research has been conducted to generate a three-dimensional reconstruction of the coronary artery tree based on multiple two-dimensional angiographic images. To date, research on three-dimensional reconstruction of the coronary artery tree has always required X-ray views from at least two different positions / orientations of an X-ray device. In other words, two-dimensional coronary angiographic images must be acquired from viewpoints at at least two positions and orientations of an X-ray device (such as a C-arm X-ray device). Using branch point and vessel detection methods and then matching the information from different viewpoints, a three-dimensional reconstruction of the coronary artery tree can be generated for easier interpretation for diagnostic purposes. However, acquiring two-dimensional angiographic images from multiple viewpoints increases the radiation dose and acquisition time. Summary of the Invention

[0004] According to one aspect of the present disclosure, a system for generating a three-dimensional image of a coronary artery tree includes a processor and a memory. The processor is configured to: obtain a sequence of two-dimensional angiographic images (410) corresponding to a moving coronary artery structure from a single viewpoint of an imaging device; and generate a three-dimensional representation of a coronary artery tree (449) of the coronary artery structure based on the sequence of two-dimensional angiographic images (410) and the cardiac motion of the moving coronary artery structure.

[0005] According to another aspect of the present disclosure, a trained machine learning model is used to reconstruct a three-dimensional representation of a coronary artery tree, and the trained machine learning model includes a neural network model.

[0006] According to yet another aspect of the present disclosure, the imaging device includes an X-ray device, and the sequence of two-dimensional angiographic images is captured from a single viewpoint without moving the X-ray device.

[0007] According to yet another aspect of the present disclosure, the trained machine learning model takes as input a sequence of two-dimensional angiographic images from a single viewpoint of the imaging device and outputs a reconstruction of the three-dimensional representation of the coronary artery tree.

[0008] According to another aspect of the present disclosure, the trained machine learning model outputs a depth map that includes a per-pixel depth image that adds depth information to a projected image from a selected frame.

[0009] According to yet another aspect of the present disclosure, the trained machine learning model outputs the three-dimensional representation of the coronary artery tree as a per-voxel volume reconstruction of a selected frame.

[0010] According to yet another aspect of the present disclosure, the trained machine learning model estimates a three-dimensional transformation or a deformation vector field that allows a reference model of at least one of the heart or the coronary artery tree to be deformed to generate a patient-specific three-dimensional representation of the coronary artery tree as the three-dimensional representation of the coronary artery tree.

[0011] According to another aspect of the present disclosure, a parametric model of a three-dimensional representation of a coronary artery tree is used to reconstruct the coronary artery tree relative to constraints of surface features in a heart synthetic model, and the coronary artery tree in the sequence of two-dimensional angiographic images is fitted to the heart synthetic model.

[0012] According to yet another aspect of the present disclosure, the system further includes a display. When run by a processor, the instructions cause the system to display the three-dimensional representation of the coronary artery tree on the display.

[0013] According to yet another aspect of the present disclosure, when executed by a processor, the instructions further cause the system to: segment and label blood vessels in the coronary artery tree in the two-dimensional angiographic images.

[0014] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium stores a computer program including instructions. When run by a processor, the instructions cause the processor to obtain a sequence of two-dimensional angiographic images (410) corresponding to a moving coronary artery structure from a single viewpoint of an imaging device; and generate a three-dimensional representation of a coronary artery tree (449) of the coronary artery structure based on the sequence of two-dimensional angiographic images (410) and the cardiac motion of the moving coronary artery structure.

[0015] According to one aspect of the present disclosure, a method for generating a three-dimensional image of a coronary artery tree includes acquiring a sequence of two-dimensional angiographic images (410) corresponding to a moving coronary artery structure from a single viewpoint of an imaging device; and generating a three-dimensional representation of a coronary artery tree (449) of the coronary artery structure based on the sequence of two-dimensional angiographic images (410) and the cardiac motion of the moving coronary artery structure.

[0016] According to another aspect of the present disclosure, the method further includes: inputting the sequence of two-dimensional angiographic images into a trained machine learning model; and outputting, by the trained machine learning model, a reconstruction of the three-dimensional representation of the coronary artery tree.

[0017] According to yet another aspect of the present disclosure, the method further includes: inputting the sequence of two-dimensional angiographic images into a trained machine learning model; and outputting, by the trained machine learning model, a depth map including a per-pixel depth image that adds depth information to a projected image from a selected frame.

[0018] According to yet another aspect of the present disclosure, the method further includes: inputting the sequence of two-dimensional angiographic images into a trained machine learning model; and outputting, by the trained machine learning model, the three-dimensional representation of the coronary artery tree as a per-voxel volume reconstruction of a selected frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Example embodiments may be best understood when read in conjunction with the accompanying drawings. It should be emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be increased or decreased arbitrarily for the sake of discussion. Where applicable and practical, like reference numerals refer to like elements.

[0020] Figure 1 Illustrates a three-dimensional coronary artery tree reconstruction system according to an exemplary embodiment.

[0021] Figure 2A Illustrates a series of two-dimensional angiographic images of a coronary artery tree from a single viewpoint of an imaging device according to an example embodiment.

[0022] Figure 2B Illustrates the orientation of the coronary artery tree at each position in a sequence of two-dimensional angiographic images according to an exemplary embodiment.

[0023] Figure 3 Illustrates another series of two-dimensional angiographic images of the coronary artery tree from a single viewpoint of an imaging device according to an exemplary embodiment.

[0024] Figure 4 Illustrates the reconstruction of a three-dimensional coronary artery tree based on multiple two-dimensional angiographic images according to an exemplary embodiment.

[0025] Figure 5 Illustrates another reconstruction of a three-dimensional coronary artery tree based on multiple two-dimensional angiographic images according to an exemplary embodiment.

[0026] Figure 6 Illustrates another reconstruction of a three-dimensional coronary artery tree based on multiple two-dimensional angiographic images according to a representative embodiment.

[0027] Figure 7 Illustrates a method for reconstructing a three-dimensional coronary artery tree according to an exemplary embodiment.

[0028] Figure 8 Illustrates a computer system for implementing a method for reconstructing a three-dimensional coronary artery tree according to another exemplary embodiment. Detailed Description

[0029] In the following detailed description, for purposes of explanation and not limitation, exemplary embodiments are set forth that disclose specific details in order to provide a thorough understanding of embodiments according to the teachings of the present invention. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are still within the scope of the claims. Descriptions of known systems, devices, materials, operating methods, and manufacturing methods may be omitted to avoid obscuring the description of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the capabilities of those of ordinary skill in the art are within the scope of this teaching and may be used in accordance with the representative embodiments. It should be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. The definitions and interpretations of the terms herein supplement the technical and scientific meanings of terms commonly understood and accepted in the technical field of the teachings of the present invention.

[0030] It should be understood that although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another. Thus, a first element or component discussed below may also be referred to as a second element or component without departing from the teachings of the inventive concept.

[0031] As used in the specification and claims, the singular forms of the terms "a," "an," and "the" are intended to include both the singular and the plural forms, unless the context clearly dictates otherwise. Additionally, when used in this specification, the terms "comprising," "including," and / or "containing," and / or similar terms specify the presence of the recited features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0032] Unless otherwise stated, when an element or component is said to be "connected to," "coupled to," or "adjacent to" another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or there may be intervening elements or components. That is, these and similar terms include cases where one or more intervening elements or components may be employed to connect two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this includes only the case where the two elements or components are connected to each other without any intervening or intermediate elements or components.

[0033] The present disclosure, by one or more of its aspects, embodiments, and / or specific features or sub-components, is intended to bring one or more of the specifically pointed out advantages as follows.

[0034] The embodiments described herein reconstruct a three-dimensional coronary artery tree of a coronary artery structure from a sequence of two-dimensional angiography images of the coronary artery structure acquired from a single acquisition position (i.e., a single viewpoint) of an imaging device (e.g., a C-arm). By reconstructing the three-dimensional coronary artery tree from images acquired from a single viewpoint of the imaging device, these embodiments can reduce the scan time during a medical procedure and the patient's radiation exposure. These embodiments can use the cardiac motion of the coronary artery structure, e.g., the cardiac torsion during one cardiac cycle from end-systole to end-diastole, to simulate different viewpoints of the coronary artery structure based on the images acquired from a single viewpoint (i.e., allowing the X-ray device to remain stationary at a single acquisition position). These embodiments can use different simulated viewpoints to reconstruct the three-dimensional coronary artery tree of the coronary artery structure. In some embodiments, a machine learning model (such as a neural network) can be trained to receive, as input, two-dimensional angiography images of the coronary artery structure acquired from a single viewpoint of the imaging device, reconstruct the three-dimensional coronary artery tree from different simulated viewpoints based on different viewpoints of the coronary artery structure, and output the reconstructed three-dimensional coronary artery tree.

[0035] Figure 1 FIG. illustrates a three-dimensional coronary artery tree reconstruction system 100 according to an exemplary embodiment.

[0036] Figure 1 The system 100 in includes an imaging system 101, a computer 140, a display 180, and an AI training system 199. The computer 140 includes a controller 150, and the controller 150 includes at least a memory 151 storing instructions and a processor 152 executing the instructions. Figure 8 A computer that can be used to implement the controller 150 is depicted in, but the controller 150 can include more or fewer elements than Figure 1 or Figure 8 depicted in. In some embodiments, Figure 1 multiple different elements of the system 100 in can include a computer (such as the computer 140) and / or a controller (such as the controller 150).

[0037] The imaging system 101 can be an X-ray system including an X-ray device and one or more detectors. The imaging system 101 is configured to capture two-dimensional angiography images of a coronary artery tree corresponding to a moving coronary artery structure (e.g., the heart) from a single viewpoint of the X-ray device. In some embodiments, the X-ray device can be a C-arm X-ray device. The imaging system 101 can include other elements, such as a control system having a memory storing instructions and a processor running the instructions, and interfaces, such as a user interface allowing a user to input instructions and / or a display for displaying interactive instructions and feedback to the user.

[0038] In some embodiments, the imaging system 101 is configured to acquire two-dimensional angiographic images, such as a complete cardiac cycle from end-diastole (ED) to end-systole (ES). In other embodiments, the imaging system 101 is configured to acquire two-dimensional angiographic images from less than one complete cardiac cycle and is used to construct a three-dimensional coronary artery tree in accordance with the teachings herein. In some other embodiments, the imaging system 101 is configured to acquire two-dimensional angiographic images from more than one cardiac cycle to reconstruct a three-dimensional coronary artery tree in accordance with the teachings herein. In some embodiments, the two-dimensional angiographic images show a clear outline of the coronary artery tree by using contrast agent injection during image acquisition. The imaging system 101 is configured to acquire two-dimensional angiographic images from a single viewpoint (position and orientation) of the imaging system 101.

[0039] The computer 140 and / or the controller 150 may include interfaces, such as a first interface, a second interface, a third interface, and a fourth interface. One or more of the interfaces may include ports, disk drives, wireless antennas, or other types of receiver circuits that connect the computer 140 and / or the controller 150 to other electronic components. One or more interfaces may also include a user interface, such as buttons, keys, a mouse, a microphone, a speaker, a display separate from the display 180, or other elements that a user may use to interact with the computer 140 and / or the controller 150 (e.g., input instructions and receive outputs). The computer 140 may be provided with the imaging system 101 or may receive two-dimensional angiographic images from the imaging system 101 via a communication network such as the Internet.

[0040] The controller 150 may directly perform some of the operations described herein and may indirectly implement other operations described herein. For example, the controller 150 may indirectly control operations, such as by generating and transmitting content to be displayed on the display 180. The controller 150 may directly control other operations, such as logical operations performed by the processor 152 running instructions from the memory 151 based on inputs received via the interfaces from electronic components and / or the user. Thus, when the processor 152 executes instructions from the memory 151, the processes implemented by the controller 150 may include steps that the controller 150 does not directly perform.

[0041] The controller 150 can apply a trained machine learning model, such as a neural network, to reconstruct the three-dimensional coronary artery tree. For example, the processor 152 or another processor can run software instructions to implement the functions of the trained machine learning model herein. The machine learning model can receive, as input, a sequence of two-dimensional angiographic images of the coronary artery structure acquired from a single viewpoint of the imaging device and over at least a portion of the cardiac motion cycle from end-diastole (ED) to end-systole (ES). The machine learning model is trained to simulate different viewpoints of the coronary artery structure based on cardiac motion from the two-dimensional angiographic images, reconstruct the three-dimensional coronary artery tree from the simulated different viewpoints, and output the reconstructed three-dimensional coronary artery tree. In some embodiments, the trained machine learning model can use a reference model of the coronary artery structure or coronary artery tree and estimate a three-dimensional transformation or deformation vector field that deforms the reference model based on cardiac motion to generate a three-dimensional representation of the coronary artery tree as a patient-specific three-dimensional representation.

[0042] In some embodiments, the trained machine learning model can output a depth map including a per-pixel depth image for generating a reconstructed three-dimensional image of the coronary artery tree. In some embodiments, the trained machine learning model can output or cause the computer 140 to output the three-dimensional coronary artery tree as a per-voxel volume reconstruction for a selected frame.

[0043] The controller 150 of the computer 140 can store software in the memory 151 for the processor 152 to run. The software may include instructions for implementing the trained machine learning model and can be used to reconstruct the three-dimensional coronary artery tree from a single image acquisition viewpoint.

[0044] The display 180 can be local to the controller 150 or can be remotely connected to the controller 150. The display 180 can be connected to the controller 150 via a local wired interface (such as an Ethernet cable) or via a local wireless interface (such as a Wi-Fi connection). The display 180 can be connected to other user input devices through which a user can input instructions, including a mouse, a keyboard, a trackball, etc. The display 180 can be a monitor, such as a computer monitor, a display on a mobile device, an augmented reality display, a television, an electronic whiteboard, or another screen configured to display electronic images. The display 180 can also include one or more input interfaces (such as the input interfaces that can be connected to other elements or components mentioned above), and an interactive touch screen configured to display prompts to the user and collect touch inputs from the user.

[0045] Use Figure 1In the system 100, the reconstruction of the three-dimensional coronary artery tree avoids the limitations imposed by relying solely on two-dimensional imaging, such as vessel / lesion overlap or foreshortening and the effects due to cardiac and pulmonary motion, and does not subject the patient to radiation exposure from three-dimensional X-ray imaging. As described herein, the coronary artery tree can be constructed from only a single C-arm acquisition viewpoint, and this also brings efficiencies, such as reduced scan time.

[0046] The AI training system 199 includes a trained machine learning model that is consistent with the teachings herein. The AI training system 199 can be provided by the same entity or system that provides the computer 140 and the display 180, or can be a third-party service that trains machine learning models on behalf of multiple entities. The processor can be configured to train the machine learning model by receiving as input a collection of previous two-dimensional angiography image sequences of the coronary artery structure. Each previous collection may correspond to images acquired during the cardiac cycle of the coronary artery structure. Each set corresponds to a known (ground truth) three-dimensional representation of the coronary artery tree. The processor can train the machine learning model based on the relevant features of the previous two-dimensional images and the corresponding ground truth three-dimensional images. The processor can also receive as input the viewpoint (e.g., position and orientation) of the imaging system when acquiring each collection of previous images. For some collections of previous images, the previous images can be acquired from a single viewpoint of the imaging device, and for some collections of previous images, the previous images can be acquired from at least two different viewpoints of the imaging device. In some embodiments, based on the input and the ground truth, the processor can train the machine learning model to determine the correspondence between the change in the coronary artery tree image view due to the rotational motion of the imaging device and the change in the coronary artery tree image view due to the rotation caused by cardiac torsion. In these embodiments, the model can be trained to apply this relationship to the views in the images acquired at a specific point in the cardiac cycle to generate further image views that simulate the rotation of the imaging device to different viewpoints. In some embodiments, the machine learning model is trained to include a reference model of the coronary artery structure or coronary artery tree and to estimate a three-dimensional transformation or deformation vector field that deforms the reference model based on cardiac motion during the cardiac cycle.

[0047] System 100 is configured to perform processing when processor 152 executes instructions from memory 151. For example, system 100 can be configured to obtain a sequence of two-dimensional angiographic images corresponding to a moving coronary artery structure (e.g., the heart) from a single viewpoint of an imaging device. System 100 is also configured to generate (e.g., using a trained machine learning model) a three-dimensional representation of the coronary artery tree of the coronary artery structure based on the sequence of two-dimensional angiographic images and the motion of the moving coronary artery structure. In some embodiments, the system selects a reference frame (e.g., the first frame in the two-dimensional image sequence) from the sequence of two-dimensional angiographic images and reconstructs a three-dimensional representation of the coronary artery tree (the sequence of two-dimensional angiographic images from a single viewpoint and the motion of the moving coronary artery structure) relative to the selected reference frame. In some embodiments, system 100 can generate a three-dimensional representation of the coronary artery tree while the imaging system captures two-dimensional angiographic images.

[0048] Figure 2A A series of two-dimensional angiographic images of a coronary artery tree from a single viewpoint of an imaging device according to a representative embodiment is illustrated.

[0049] In Figure 2A the three-dimensional representation 211A is captured at end-diastole, while the three-dimensional representation 299A is captured at end-systole. Two-dimensional angiographic images captured from end-diastole, end-systole, and any other part of the cardiac cycle are captured from a single viewpoint corresponding to a single acquisition position of the imaging system 101 in Figure 1 . The two-dimensional angiographic images can be obtained by computer 140 from imaging system 101 and used to generate one or both of the three-dimensional representation 211A and the three-dimensional representation 299A. In some embodiments, a single reference frame is used as a basis for generating a three-dimensional representation of the coronary artery tree based on the underlying motion of the heart and the coronary artery tree reflected in the sequence of two-dimensional angiographic images. In some embodiments, the single reference frame can be a frame at end-diastole, and the stack of two-dimensional angiographic images can include frames for the entire cardiac cycle, including frames for end-diastole of the heart.

[0050] In Figure 2AIn this case, a single viewpoint of the imaging device is used to capture images corresponding to the heart motion from end - diastole to end - systole (ES). The resulting set of two - dimensional angiographic images contains information that can be used to effectively simulate the views of the coronary artery tree that would be captured by moving the imaging device to different viewpoints. In some embodiments, the model detects the torsion (approximate rotation about the cardiac axis) of the heart and uses the torsion to derive a complete three - dimensional representation of the coronary artery tree as if the representation were derived from images from multiple different C - arm acquisition positions, although all the views are from a single C - arm acquisition position. This avoids the need to acquire images from more than one viewpoint, thus reducing the scan time and the radiation exposure to the patient.

[0051] In some embodiments consistent with Figure 2A it, the visible blood vessels of the coronary artery tree in the image can be segmented and labeled, for example, by using a segmentation / labeling algorithm. In some embodiments, a heart motion model can be used to model the motion path of the heart from end - diastole to ES. The motion model can use the segmented and labeled blood vessels. The heart motion model can be three - dimensional rigid, three - dimensional affine, or even a more complex heart motion model. The motion model can estimate the torsion of the visible coronary artery tree during the heart motion between end - diastole and end - systole. In other words, the rotation of some or all points on the coronary artery tree during heart motion can be estimated by the motion model. These point - by - point rotation estimates of the coronary artery tree simulate different viewpoints of the C - arm device on a point - by - point basis. The motion model can use the simulated different viewpoints to reconstruct the three - dimensional coronary artery tree from local viewpoint variations (such as a sequence of two - dimensional images acquired from a single viewpoint of the imaging system). The reconstructed three - dimensional coronary artery tree and any or all of the two - dimensional angiographic images can be displayed on the display 180.

[0052] Figure 2B Illustrated is the orientation of the coronary artery tree at each position in a sequence of two - dimensional angiographic images according to a representative embodiment.

[0053] In Figure 2B it, the three - dimensional representation 211B is captured at end - diastole, while the three - dimensional representation 299B is captured at end - systole. In Figure 2B it, the three - dimensional anatomical torsion of the heart during heart motion is shown by the local rotation at the three - dimensional representation 299B. It is obvious that the coronary artery tree follows the anatomical torsion of the heart throughout the cardiac cycle.

[0054] Figure 3 Illustrated is another series of two - dimensional angiographic images of the coronary artery tree from a single viewpoint of the imaging device according to a representative embodiment.

[0055] As explained herein, three-dimensional reconstruction of a particular cardiac phase can be achieved from the viewpoint of a single imaging system (e.g., a C-arm). In Figure 3 two three-dimensional representations of the coronary artery tree are shown, one at end-diastole and one at end-systole. In Figure 3 corresponding two-dimensional angiographic images are provided for each of the three-dimensional representations. The end-diastolic three-dimensional representation 311 is shown on the left, and the end-systolic three-dimensional representation 399 is shown on the right. As shown for the end-diastolic three-dimensional representation 311, the rotation can be counterclockwise, such as about an approximate axis of rotation, and for the end-systolic three-dimensional representation 399, the rotation can be clockwise, also such as about an approximate axis of rotation.

[0056] In Figure 3 two-dimensional angiographic images of a human heart are taken from a single static viewpoint of the C-arm position in the case of cardiac motion. For each two-dimensional angiographic image, a three-dimensional rendering of the heart and a simulated projection of the coronary artery tree onto the two-dimensional detector plane are provided. As can be seen from the direct comparison on the left and right in Figure 3 the torsion of the heart is seen as corresponding to a rotational movement of the heart about an approximate axis of rotation. The torsion of the heart includes an approximate rotation of the left myocardium about the cardiac axis and is shown from end-diastole (left) to end-systole (right). Observing Figure 3 the cardiac motion in, the orientation of the coronary artery tree changes with the cardiac cycle towards the X-ray system. In an embodiment, a machine learning model is trained to correlate a change in the view of the coronary artery structure image due to the rotational movement of the imaging device with a change in the view of the coronary artery structure image due to the rotation caused by cardiac torsion. The model can apply this change in the view of the image due to the rotation caused by torsion to the image input from a single C-arm viewpoint to simulate different image views of the coronary artery structure. The model uses the simulated image views to generate a three-dimensional reconstruction of the coronary artery tree. In other words, for the Figure 3 two-dimensional angiographic images, the C-arm does not change the viewpoint, but rather uses a model of cardiac torsion to simulate the change in the relative viewpoint.

[0057] Figure 4 illustrates the reconstruction of a three-dimensional coronary artery tree from a plurality of two-dimensional angiographic images according to an illustrative embodiment.

[0058] Based on Figure 4In embodiments, a per-pixel depth map can be generated based on a selected reference frame. This allows the three-dimensional model of the coronary artery tree to be reconstructed as a two-dimensional coronary artery tree supplemented with a depth map, but from a single viewpoint. The reconstruction of a two-dimensional angiographic image based on a single viewpoint can reduce the scan time and the patient's radiation exposure. A pixel depth map can be generated based on the distance / depth value from the acquisition device to the anatomical structure (e.g., the coronary artery tree). Using the distance / depth value, a two-dimensional X-ray image can be transformed into a three-dimensional image because each pixel provides a vector for the distance to the acquisition device.

[0059] In Figure 4 , the machine learning model 430 can reconstruct the coronary artery tree 449 via the depth map. Figure 4 The machine learning model 430 in Figure 4 is a trained neural network. Embodiments of Figure 4 can be considered as an option for the machine learning model 430 to implicitly reconstruct the three-dimensional coronary artery tree reconstruction 440. Implicit reconstruction can be considered that the machine learning model 430 does not calculate the explicit transformation from frame to frame, but relies on, for example, a heart model and the coronary artery tree segmentation from three-dimensional computed tomography angiography data. In Figure 4 , a large set of two-dimensional angiographic images 410 from a single viewpoint can be used to allow the estimation of the three-dimensional coronary artery tree reconstruction 440. The estimation can include the prediction of a map of pixel depths for each pixel by the machine learning model 430. In

[0060] , a set of two-dimensional angiographic images 410 is input into a neural network model as the machine learning model 430, and the neural network model outputs a per-pixel depth map, which is combined with the selected reference frame to obtain a three-dimensional coronary artery tree. The per-pixel depth map can include a per-pixel depth image, which adds depth information to the projection image from the selected frame. The selected frame can be the first frame of the cardiac cycle or a subsequent frame. Figure 3 shows an example of two-dimensional to three-dimensional matching data. Another method for generating such pairs is to match a specific acquisition projection of three-dimensional computed tomography angiography data with the acquired X-ray image.

[0061] InFigure 4 In this case, the machine learning model 430 is used to reconstruct a three-dimensional coronary artery tree from multiple two-dimensional X-ray angiography images. The machine learning model 430 takes as input a stack of angiography images (sorted from end-diastole to end-systole). The machine learning model 430 outputs a per-pixel depth map relative to a reference frame. For example, a first reference frame can be used as the basis for constructing the three-dimensional model, but alternatively a subsequent reference frame can be used as the basis. In some embodiments, multiple reference frames can be used as the basis for the three-dimensional model. The combination of the two-dimensional reference frame and the depth map represents the three-dimensional reconstruction of the coronary artery tree relative to the reference frame.

[0062] Given a two-dimensional to three-dimensional pair (image and corresponding two-dimensional / three-dimensional vasculature segmentation), the machine learning model 430 can use artificial intelligence (AI) methods to reconstruct the three-dimensional shape. Given a set of two-dimensional angiography images during a cardiac cycle and a three-dimensional heart model and coronary artery tree shape matched to each cardiac phase, the machine learning model 430 takes as input a series of two-dimensional angiography images and outputs the desired three-dimensional reconstruction of the heart (e.g., the coronary artery tree). In some embodiments, a three-dimensional depth map can be estimated for a given two-dimensional input image, and the output three-dimensional map can be based on per-pixel depth information. In some embodiments, a stack of two or more two-dimensional angiography images, sorted from end-diastole to end-systole, can be input to the machine learning model 430. Given a two-dimensional to three-dimensional pair and thus the ground truth of the depth information for each X-ray image, the machine learning model 430 can be trained to deliver the depth map of a selected reference frame, e.g., the first one. In other words, the machine learning model 430 can output a per-pixel depth image that adds depth information to the projected image and thus converts the reference two-dimensional image into three-dimensional space. This ultimately results in a three-dimensional reconstruction of the desired coronary artery tree as a two-dimensional image supplemented with AI-based depth estimation, as Figure 4 shown.

[0063] Figure 5 Illustrated is another reconstruction of a three-dimensional coronary artery tree from multiple two-dimensional angiography images according to an illustrative embodiment.

[0064] Figure 5 Illustrated is an embodiment in which the machine learning model 530 can directly reconstruct the coronary artery tree in three-dimensional space, e.g., by using a three-dimensional voxel occupancy map. In Figure 5 this case, the machine learning model 530 is a neural network. Figure 5 The embodiment of can be regarded as a second implicit option, where the machine learning model 530 reconstructs the coronary artery tree as a three-dimensional reconstruction 540. As described above with respect to Figure 4As described, given a two-dimensional to three-dimensional image pair and corresponding two-dimensional / three-dimensional vasculature segmentation, an artificial intelligence (AI) method can be used to reconstruct a three-dimensional representation of the coronary artery tree 550 to produce a three-dimensional reconstruction 540. In Figure 5 this, a set 510 of two-dimensional angiographic images is input into a neural network model as a machine learning model 530, and the neural network model directly outputs a three-dimensional coronary artery tree 530 as a direct volume reconstruction of the coronary artery tree. For example, Figure 5 the output of the neural network in this can be a per-pixel output within a predefined field of view. The machine learning model 530 can output the three-dimensional coronary artery tree as a per-voxel volume reconstruction for the selected frame.

[0065] Given a set of two-dimensional angiographic images during a cardiac cycle and a three-dimensional heart model and coronary artery tree shape matched to each cardiac phase, the machine learning model 430 can be trained by taking as input a series of two-dimensional angiographic images and outputting a desired three-dimensional reconstruction 540 of the heart (e.g., the coronary artery tree). In Figure 5 this, another method of reconstructing a three-dimensional coronary artery tree based on artificial intelligence from multiple two-dimensional X-ray angiographic images is shown. The machine learning model takes as input a stack of angiographic images (sorted from end-diastole to end-systole). Figure 5 Another example of two-dimensional to three-dimensional matching data is shown. In Figure 5 this, a direct volume reconstruction of the coronary artery tree is shown, for example, as a per-pixel output within a predefined field of view.

[0066] Figure 6 Another reconstruction of a three-dimensional coronary artery tree from multiple two-dimensional angiographic images according to an illustrative embodiment is illustrated.

[0067] Figure 6 An embodiment is illustrated in which the machine learning model can directly and explicitly reconstruct the coronary artery tree in three-dimensional space. In Figure 6 this embodiment, the neural network 630 estimates a rotation transformation based on the multi-frame angiographic images input at S610. The neural network 630 outputs the rotation transformation. At S650, the transformation output at S610 is applied to a representative frame used as a basis for the three-dimensional reconstruction. The three-dimensional reconstruction can be optimized by selecting the minimum distance for the movement of three-dimensional points.

[0068] In Figure 6 this, as a trained machine learning model, the neural network 630 is configured to estimate a three-dimensional transformation or deformation vector field that allows a reference model to be deformed. The reference model can be a model of the heart or coronary artery tree and can be deformed to generate a three-dimensional representation of the coronary artery tree as a patient-specific three-dimensional representation of the coronary artery tree.

[0069] In some embodiments, a parametric model of a three-dimensional coronary artery tree can be used to reconstruct the coronary artery tree relative to surface features in a cardiac synthetic model, and the coronary artery tree in a sequence of two-dimensional angiographic images is fitted to the cardiac synthetic model. The constraints of the surface features in the cardiac synthetic model can be used in various embodiments described herein, including Figure 4 , Figure 5 and Figure 6 embodiments.

[0070] Figure 7 FIG. illustrates a method for reconstructing a three-dimensional coronary artery tree according to a representative embodiment.

[0071] At S710, the imaging system 101 captures a sequence of two-dimensional images of an anatomical structure (e.g., the heart) from a single viewpoint corresponding to a single acquisition position of the imaging system 101 in Figure 1 . As described herein, the imaging system 101 can be an X-ray device, and in some embodiments, the imaging system 101 can include a C-arm X-ray device with a single detector.

[0072] At S720, a controller (e.g., a processor, such as computer 140) receives the sequence of two-dimensional images from the imaging system 101. In some embodiments, the controller segments and / or labels the blood vessels in the coronary artery tree of the anatomical structure included in the two-dimensional angiographic images.

[0073] At S730, the controller selects a reference frame from the sequence of two-dimensional images to reconstruct a three-dimensional representation of the anatomical structure (e.g., the coronary artery tree of the anatomical structure). For example, the first frame of the cardiac cycle captured in the sequence of two-dimensional images can be selected as the reference frame for reconstructing the three-dimensional representation of the coronary artery tree of the anatomical structure.

[0074] At S740, the controller generates and outputs a three-dimensional representation of the coronary artery tree. In some embodiments, based on Figure 4 , the controller can apply a trained machine learning model to generate a three-dimensional representation of the coronary artery tree from a single viewpoint of the imaging device during the movement of a moving coronary artery structure (e.g., the heart) and when capturing two-dimensional angiographic images. The machine learning model can generate the three-dimensional representation based on the cardiac motion of the anatomical structure applied to the sequence of two-dimensional images and / or the reference frame from the sequence of two-dimensional images. The trained machine learning model can generate a three-dimensional representation of the coronary artery tree as a depth map including a per-pixel depth image. In some embodiments, the trained machine learning model can generate the depth map as a projected image from the selected reference frame and add depth information. In some embodiments, based on Figure 5 , the trained machine learning model can generate a three-dimensional coronary artery tree as a per-voxel volume reconstruction of the reference frame. In some embodiments, based onFigure 6 The trained machine learning model generates a three-dimensional coronary tree by estimating a three-dimensional transformation or deformation vector field that allows a reference model of at least one heart or coronary tree to be deformed to generate a three-dimensional representation of the coronary tree as a patient-specific three-dimensional representation of the coronary tree. In some embodiments, the parameterized model of the three-dimensional coronary tree can be used to reconstruct the coronary tree relative to constraints on surface features of the coronary structure.

[0075] At S750, the controller displays a three-dimensional coronary artery tree. For example, the display 180 may be based on Figure 1 The processor 152 of the system 100 executes instructions to display a three-dimensional coronary artery tree.

[0076] Figure 8 A computer system implementing a three-dimensional coronary artery tree reconstruction method according to another representative embodiment is illustrated.

[0077] refer to Figure 8 , the computer system 800 includes a set of software instructions that can be executed to cause the computer system 800 to perform any method or computer-based function disclosed herein. The computer system 800 can operate as a stand-alone device, or can be connected to other computer systems or peripheral devices, for example using a network 801. In an embodiment, the computer system 800 performs logic processing based on digital signals received through an analog-to-digital converter.

[0078] In a networked deployment, the computer system 800 can operate as a server or client user computer in a server-client user network environment, or as a peer computer system in a peer (or distributed) network environment. The computer system 800 can also be implemented as or incorporated into various devices, such as a workstation, a fixed computer, a mobile computer, a personal computer (PC), a laptop, a tablet computer, or any other machine capable of executing a set (sequential or other) of software instructions including a controller, the set of which specifies the operations to be taken by the machine. The computer system 800 can be incorporated as a device or incorporated into a device, which is then included in an integrated system containing additional devices. In an embodiment, the computer system 800 can be implemented using an electronic device that provides voice, video or data communication. In addition, the computer system 800 is shown as a monomer, but the term "system" should also be considered to include a collection of any system or subsystem that executes one or more sets of software instructions individually or jointly to perform one or more computer functions.

[0079] like Figure 8As shown in [FIGURE], computer system 800 includes a processor 810. Processor 810 can be regarded as a representative example of a processor of a controller and executes instructions to implement some or all aspects of the methods and processes described herein. Processor 810 is tangible and non-transitory. As used herein, the term "non-transitory" should not be construed as an eternal characteristic of a state, but rather as a characteristic of a state that will persist for a period of time. The term "non-transitory" specifically negates fleeting characteristics, such as those of a particular propagating carrier or signal or other forms that exist transiently only anywhere at any time. Processor 810 is an article of manufacture and / or a machine component. Processor 810 is configured to execute software instructions to perform the functions described in various embodiments herein. Processor 810 can be a general-purpose processor or can be part of an application-specific integrated circuit (ASIC). Processor 810 can also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. Processor 810 can also be a logic circuit, including a programmable gate array (PGA) such as a field-programmable gate array (FPGA), or another type of circuit including discrete gates and / or transistor logic. Processor 810 can be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein can include multiple processors, parallel processors, or both. Multiple processors can be included in or coupled to a single device or multiple devices.

[0080] The term "processor" as used herein encompasses electronic components capable of running programs or machine-executable instructions. References to a computing device that includes a "processor" should be construed as including more than one processor or processing core, such as a multi-core processor. A processor can also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be construed as including a collection or network of computing devices, each of which includes one or more processors. A program has software instructions that are executed by one or more processors, which can be within the same computing device or the one or more processors can be distributed across multiple computing devices.

[0081] The computer system 800 also includes a main memory 820 and a static memory 830, where the memories in the computer system 800 communicate with each other and with the processor 810 via a bus 808. Either or both of the main memory 820 and the static memory 830 can be regarded as representative examples of the memory of the controller, and store instructions for implementing some or all aspects of the methods and processes described herein. The memory 307 described herein is a tangible storage medium for storing data and executable software instructions, and is non-transitory during the time when the software instructions are stored. As used herein, the term "non-transitory" should not be construed as an eternal characteristic of a state, but rather as a characteristic of a state that will persist for a period of time. The term "non-transitory" specifically negates fleeting characteristics, such as those of a particular propagating carrier or signal, or other forms that exist transiently only anywhere at any time. The main memory 820 and the static memory 830 are articles of manufacture and / or machine components. The main memory 820 and the static memory 830 are computer-readable media from which a computer (e.g., the processor 810) can read data and executable software instructions. Both the memory 820 and the static memory 830 can be implemented as one or more of a random access memory (RAM), a read-only memory (ROM), a flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), registers, one or more magnetic disks in a hard disk, a removable disk, a magnetic tape, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a floppy disk, a Blu-ray disc, or any other form of storage medium known in the art. The memory can be volatile or non-volatile, secure and / or encrypted, insecure and / or unencrypted.

[0082] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory directly accessible by a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be construed as potentially being multiple memories. The memory can be, for example, multiple memories within the same computer system. The memory can also be multiple memories distributed among multiple computer systems or computing devices.

[0083] As shown, computer system 800 also includes a video display unit 850, for example, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). Additionally, computer system 800 includes an input device 860, such as a keyboard / virtual keyboard or a touch input screen or voice input with voice recognition, and a cursor control device 870, such as a mouse or a touch input screen or a touchpad. Computer system 800 also optionally includes a disk drive unit 880, a signal generation device 890 (such as a speaker or a remote control), and / or a network interface device 840.

[0084] In one embodiment, as Figure 8 shown, disk drive unit 880 includes a computer-readable medium 882, in which one or more sets 884 (software) of software instructions are embedded. The set 884 of software instructions is read from the computer-readable medium 882 and executed by the processor 810. Additionally, when the processor 810 executes the software instructions 884, one or more steps of the methods and processes described herein are performed. In one embodiment, the software instructions 884 reside, in whole or in part, within the main memory 820, the static memory 830, and / or the processor 810 during execution by the computer system 800. Further, the computer-readable medium 882 may include the software instructions 884 or receive and execute the software instructions 884 in response to a propagated signal, such that a device connected to the network 801 transmits voice, video, or data over the network 801. The software instructions 884 may be transmitted or received over the network 801 via the network interface device 840.

[0085] In one embodiment, a dedicated hardware implementation is constructed, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array, and other hardware components, to implement one or more of the methods described herein. One or more of the embodiments described herein may use two or more specific interconnected hardware modules or devices to perform functions, with associated control that can communicate between and through the modules. Thus, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in this application should be construed as implementing or performing solely with software rather than hardware such as tangible non-transitory processors and / or memories.

[0086] According to various embodiments of the present disclosure, the methods described herein can be implemented using a hardware computer system that executes a software program. Additionally, in exemplary non-limiting embodiments, the implementation may include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can be used to implement one or more of the methods or functions described herein, and the processor 205 described herein can be used to support a virtual processing environment.

[0087] Thus, three-dimensional coronary artery tree reconstruction enables the reconstruction of a three-dimensional representation of the coronary artery tree in selected frames from a sequence of two-dimensional angiographic images from a single viewpoint. The ability to use a series of two-dimensional angiographic images from a single viewpoint corresponding to a single acquisition position reduces radiation exposure, facility time, and operator and equipment requirements, as well as other efficiencies achieved through the teachings herein.

[0088] Although three-dimensional coronary artery reconstruction has been described with reference to several exemplary embodiments, it should be understood that the words used are descriptive and illustrative words, not restrictive words. Changes may be made within the scope of the presently stated and modified appended claims without departing from the scope and spirit of three-dimensional coronary artery reconstruction in its aspects. Although three-dimensional coronary artery tree reconstruction has been described with reference to specific means, materials, and embodiments, three-dimensional coronary artery tree reconstruction is not limited to the disclosed details; rather, three-dimensional coronary artery tree reconstruction extends to all functionally equivalent structures, methods, and uses, such as within the scope of the appended claims.

[0089] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. These illustrations are not intended to be a complete description of all elements and features of the present disclosure described herein. After reviewing the present disclosure, many other embodiments will be apparent to those skilled in the art. Other embodiments may be utilized and other embodiments may be derived from the present disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Additionally, these illustrations are merely representative and may not be drawn to scale. Some of the scales in the illustrations may be exaggerated while other scales may be minimized. Accordingly, the present disclosure and the drawings should be considered illustrative rather than restrictive.

[0090] One or more of the embodiments disclosed herein may be independently and / or jointly referred to by the term "invention" solely for convenience, but this does not limit the scope of the present application to any particular invention or inventive concept. Further, although specific embodiments have been illustrated and described herein, it should be understood that any subsequent arrangement designed to achieve the same or similar purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all subsequent modifications or variations of the various embodiments. Combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon review of the specification.

[0091] The abstract of the present disclosure provided complies with 37 C.F.R. § 1.72(b), and it should be understood when submitting the abstract that the abstract is not for interpreting or limiting the scope or meaning of the claims. Additionally, in the foregoing detailed description, for the purpose of simplifying the present disclosure, various features may be combined together or described in a single embodiment. The present disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, inventive indicia may point to less than all of the features of any of the disclosed embodiments. Accordingly, the following claims are incorporated into the detailed description, where each claim independently defines the claimed indicia.

[0092] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. For that reason, the subject matter disclosed above should be considered illustrative and not restrictive, and the claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Accordingly, to the maximum extent permitted by law, the scope of the present disclosure will be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited to or restricted by the foregoing detailed description.

Claims

1. A controller (150) for generating a three-dimensional image of a coronary artery tree (449), the controller comprising: a processor (152) and a memory (151), the processor (152) being configured to: obtain a sequence of two-dimensional angiographic images (410) corresponding to a moving coronary artery structure from a single viewpoint of an imaging device; and generate a three-dimensional representation of the coronary artery tree (449) of the coronary artery structure based on the sequence of the two-dimensional angiographic images (410) and the cardiac motion of the moving coronary artery structure.

2. The controller (150) according to claim 1, wherein, the processor (152) is configured to apply a trained machine learning model (430) to generate the three-dimensional representation of the coronary artery tree (449), wherein the trained machine learning model is configured to: receive the sequence of two-dimensional angiographic images (410) as an input, and generate the three-dimensional representation of the coronary artery tree (449) based on the cardiac motion of the moving coronary artery structure.

3. The controller (150) according to claim 1, wherein, the imaging device comprises an X-ray device, and the sequence of two-dimensional angiographic images (410) is captured from a single viewpoint without moving the X-ray device.

4. The controller (150) according to claim 1, wherein, the controller is further configured to: select a frame from the sequence of two-dimensional angiographic images (410); obtain a projection image of the selected frame; add depth information to the projection image based on information in the sequence of two-dimensional angiographic images and the motion of the moving coronary artery structure to generate the three-dimensional representation of the coronary artery tree as a depth map; and output a depth map including a per-pixel depth image that adds depth information to the projection image of the selected frame.

5. The controller (150) according to claim 1, wherein, the depth map includes a per-pixel depth image of the coronary artery tree.

6. The controller (150) according to claim 1, wherein, the trained machine learning model (430) outputs the three-dimensional representation (211A) of the coronary artery tree (449) as a per-voxel volume reconstruction for the selected frame.

7. The controller (150) according to claim 1, wherein, the trained machine learning model (430) is configured to estimate a three-dimensional transformation or a deformation vector field that deforms a reference model of at least one of the coronary artery structure or the coronary artery tree (449) to generate the three-dimensional representation (211A) of the coronary artery tree (449).

8. The controller (150) according to claim 1, wherein, The processor is further configured to reconstruct the coronary artery tree (449) relative to constraints of surface features in the cardiac synthetic model using a parametric model of the three-dimensional representation (211A) of the coronary artery tree (449), and the coronary artery tree (449) in the sequence of two-dimensional angiographic images (410) is fitted to the cardiac synthetic model.

9. The controller (150) according to claim 1, wherein, the processor is further configured to: display (180) the three-dimensional representation (211A) of the coronary artery tree (449) on a display (180).

10. The controller (150) according to claim 1, wherein, the processor is further configured to: segment and label blood vessels in the two-dimensional angiographic image (410); and generate the three-dimensional representation of the coronary artery tree (449) of the coronary artery structure based on the blood vessels segmented and labeled in the two-dimensional angiographic image.

11. A non-transitory computer-readable storage medium storing a computer program including instructions that, when executed by a processor, cause the processor to perform the following operations: obtain a sequence of two-dimensional angiographic images (410) corresponding to a moving coronary artery structure from a single viewpoint of an imaging device; and generate a three-dimensional representation of the coronary artery tree (449) of the coronary artery structure based on the sequence of the two-dimensional angiographic images (410) and the cardiac motion of the moving coronary artery structure.

12. A method for generating a three-dimensional image of a coronary artery tree (449), comprising: obtaining a sequence of two-dimensional angiographic images (410) corresponding to a moving coronary artery structure from a single viewpoint of an imaging device; and generating a three-dimensional representation of the coronary artery tree (449) of the coronary artery structure based on the sequence of the two-dimensional angiographic images (410) and the cardiac motion of the moving coronary artery structure.

13. The method according to claim 12, further comprising: inputting the sequence of two-dimensional angiographic images (410) into a trained machine learning model (430); and outputting, by the trained machine learning model (430), a reconstruction of the three-dimensional representation (211A) of the coronary artery tree (449).

14. The method according to claim 12, further comprising: inputting the sequence of two-dimensional angiographic images (410) into the trained machine learning model (430); and outputting, by the trained machine learning model (430), a depth map including a per-pixel depth image that adds depth information to a projection image from a selected frame.

15. The method according to claim 12, further comprising: inputting the sequence of two-dimensional angiographic images (410) into the trained machine learning model (430); and The three-dimensional representation (211A) of the coronary artery tree (449) is output by the trained machine learning model (430) as a voxel-by-voxel volume reconstruction for a selected frame.

Images