Time-resolved angiography

A neural network-based method predicts high-quality 3D angiographic images from 2D images, addressing access and cost issues in 3D angiography, and reducing X-ray dose, thereby improving clinical imaging efficiency and understanding of contrast agent flow.

JP7591186B2Active Publication Date: 2024-11-28KONINKLIJKE PHILIPS NV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024526463
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-12-30
Filing Date
2022-11-03
Publication Date
2024-11-28
Estimated Expiration
2042-11-03

AI Technical Summary

Technical Problem

Existing 3D angiography imaging systems face challenges such as limited access to the patient, higher cost, and the inability to produce high-quality 3D blood vessel images with low X-ray dose without increasing acquisition time, making them unsuitable for clinical investigations requiring contrast-enhanced images.

Method used

A computer-implemented method using a neural network to predict a temporal sequence of 3D angiographic images from a temporal sequence of 2D angiographic images, constrained by volumetric image data, allowing for a deeper understanding of contrast agent flow without the need for precise alignment of the imaging system with the anatomical structure.

Benefits of technology

This approach provides high-quality 3D angiographic images with structural information, overcoming the limitations of traditional 3D angiography by reducing the need for precise system alignment and minimizing X-ray exposure, thus enhancing clinical workflow and image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007591186000001
    Figure 0007591186000001
  • Figure 0007591186000002
    Figure 0007591186000002
  • Figure 0007591186000003
    Figure 0007591186000003
Patent Text Reader

Abstract

A computer-implemented method is provided for providing a temporal sequence of 3D angiographic images 110 representative of the flow of contrast agent through a region of interest 120. The method includes the steps of inputting volumetric image data 130a, 130b and a temporal sequence of 2D angiographic images 140 into a neural network NN1 S130, and generating a predicted temporal sequence of 3D angiographic images 110 representative of the flow of contrast agent through the region of interest 120 in response to the input S140.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] SUMMARY The present disclosure relates to providing a temporal sequence of 3D angiographic images representative of flow of contrast agent through a region of interest. A computer-implemented method, computer program product, and system are disclosed. [Background technology]

[0002] Time-resolved angiographic imaging is often used in clinical research. Compared to static angiographic images, the temporal sequence of angiographic images produced by time-resolved angiographic imaging provides information about the flow of contrast agent through the vasculature. The flow information, in turn, allows for improved understanding of physiological processes in the anatomy. However, the benefits of time-resolved angiographic imaging are achieved at the expense of increased x-ray dose to the patient.

[0003] Both time-resolved 2D angiographic imaging and time-resolved 3D angiographic imaging are used in clinical research. Compared with time-resolved 2D angiographic imaging, the structural information provided by time-resolved 3D angiographic imaging allows a deeper understanding of the flow of contrast agent in the region of interest. The use of time-resolved 3D angiographic imaging may also improve workflow. For example, the use of time-resolved 3D angiographic imaging removes the need to precisely align the imaging system with the region of interest to provide a desired view of the region of interest. The time-resolved 3D angiographic image may be displayed in 3D or as a composite 2D projection from any desired viewing direction to provide one or more simultaneous views of the region of interest.

[0004] Historically, time-resolved 3D angiographic images often had reduced image quality compared to time-resolved 2D angiographic images due to increased demands on image processing and the need to limit x-ray dose to the patient. More recently, angiographic imaging techniques such as digital subtraction angiography (DSA) have been introduced that provide time-resolved 3D angiographic images with similar image quality and x-ray dose as time-resolved 2D angiographic images. Summary of the Invention [Problem to be solved by the invention]

[0005] However, challenges remain to using 3D angiographic imaging for some clinical investigations including, for example, the potential for 3D angiographic imaging systems to limit patient access, the higher cost of 3D angiographic imaging systems, and the need to provide high quality 3D angiographic images with low x-ray dose and without increasing acquisition times. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a computer-implemented method for providing a temporal sequence of 3D angiographic images representative of flow of contrast agent through a region of interest is provided, the method comprising: receiving volumetric image data representative of a region of interest; receiving a temporal sequence of 2D angiographic images representative of flow of contrast agent through a region of interest; inputting volumetric image data and a temporal sequence of 2D angiographic images into a neural network; generating, in response to the input, a predicted temporal sequence of 3D angiographic images representative of flow of contrast agent through the region of interest; Including, The neural network is trained to predict a temporal sequence of 3D angiographic images from a temporal sequence of 2D angiographic images and to constrain the predicted temporal sequence of 3D angiographic images by the volumetric image data.

[0007] Since the predicted temporal sequence of 3D angiographic images is provided by the neural network from the temporal sequence of 2D angiographic images, the method offers the advantages of a temporal sequence of 3D angiographic images, including structural information that allows a deeper understanding of the flow of contrast agent in the region of interest, without the disadvantages of the need to precisely align the imaging system with the anatomical structures and of limited access to the region of interest.

[0008] Further aspects, features, and advantages of the present disclosure will become apparent from the following description of examples which proceeds with reference to the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] 1 is a flow chart illustrating an example of a method for providing a temporal sequence of 3D angiographic images representing flow of contrast agent through a region of interest, according to some aspects of the present disclosure. [Diagram 2] 1 is a schematic diagram illustrating an example of a system for providing a temporal sequence 110 of 3D angiographic images representing flow of contrast agent through a region of interest 120, according to some aspects of the present disclosure. [Diagram 3] 1 is a schematic diagram illustrating a first example of a method for providing a temporal sequence 110 of 3D angiographic images representing flow of contrast agent through a region of interest 120, according to some aspects of the present disclosure. [Figure 4] 1 is a schematic diagram illustrating a second example of a method for providing a temporal sequence 110 of 3D angiographic images representing flow of contrast agent through a region of interest 120, according to some aspects of the present disclosure. [Diagram 5] A schematic diagram showing the orientation of the X-ray source 150b and X-ray detector 150a of the X-ray imaging system 150 relative to the patient's region of interest 120, including the rotation angle of the centerline of the projection X-ray imaging system 150 around the patient's longitudinal axis, in accordance with some embodiments of the present disclosure. [Figure 6] A schematic diagram showing the orientation of the X-ray source 150b and X-ray detector 150a of the X-ray imaging system 150 relative to the patient's region of interest 120, including the tilt angle b of the centerline of the projection X-ray imaging system 150 relative to the patient's cranio-caudal axis, in accordance with some embodiments of the present disclosure. [Figure 7] 1 is a flowchart illustrating a first example of a method for training a neural network to provide a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure. [Figure 8]11 is a flowchart illustrating a second example of a method for training a neural network to provide a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure. [Figure 9] 1 is a schematic diagram illustrating a first example of a method for training a neural network to provide a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Examples of the present disclosure are provided with reference to the following description and drawings. In this description, for the purpose of explanation, many specific details of a particular example are described. In this specification, reference to an "example", "embodiment", or similar language means that a feature, structure, or characteristic described in connection with the example is included in at least that one example. It should also be understood that features described in connection with one example may also be used in another example, and that not all features are necessarily repeated in each example for the sake of brevity. For example, features described in connection with a computer-implemented method may be implemented in a computer program product and in a system in a corresponding manner.

[0011] In the following description, a computer-implemented method is described that involves providing a temporal sequence of 3D angiographic images that represent the flow of contrast agent through a region of interest. Reference is made to an example of a region of interest in the form of a cerebral aneurysm. However, it should be understood that this region of interest serves only as an example, and the method disclosed herein may also be used to provide a temporal sequence of 3D angiographic images that represent the flow of contrast agent through other regions of interest in the vasculature. In general, the region of interest may be anywhere in the vasculature.

[0012] Reference is also made herein to an exemplary therapeutic procedure in the form of an aneurysm coiling procedure, in which a coil of wire is inserted into the aneurysm's sac to reduce blood flow and thereby allow blood inside the aneurysm to clot. However, it should be understood that this therapeutic procedure serves only as an example, and that the methods disclosed herein may also be used in other therapeutic procedures. The methods may be used, for example, in therapeutic procedures such as coronary angioplasty, cerebral endovascular thrombectomy, and tumor embolization. It should also be understood that the methods disclosed herein may generally be used in clinical studies.

[0013] It should be noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including computer-readable instructions that, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program product. The computer program product may be provided by dedicated hardware, or by hardware capable of executing software in association with appropriate software. When provided by a processor, the functions of the method features may be provided by a single dedicated processor, or by a single shared processor, or by multiple individual processors, some of which may be shared. One or more functions of the method features may be provided by a processor shared within a networked processing architecture, such as, for example, a client / server architecture, a peer-to-peer architecture, the Internet, or a cloud.

[0014] Explicit use of the terms "processor" or "controller" should not be construed as exclusively referring to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor "DSP" hardware, read only memory "ROM" for storing software, random access memory "RAM", non-volatile storage, and the like. Furthermore, examples of the present disclosure may take the form of a computer usable storage medium, or a computer program product accessible from a computer readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable storage medium or a computer readable storage medium may be any device that can have, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system or device or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory "RAM", a read-only memory "ROM", a rigid magnetic disk, and an optical disk. Current examples of optical disks include Compact Disk-Read Only Memory "CD-ROM", Compact Disk-Read / Write "CD-R / W", Blu-Ray, and DVD.

[0015] As mentioned above, time-resolved angiographic imaging is often used in clinical research. Time-resolved 2D angiographic imaging and time-resolved 3D angiographic imaging are both used in clinical research. Compared to time-resolved 2D angiographic imaging, the structural information provided by time-resolved 3D angiographic imaging allows for a deeper understanding of the flow of contrast agent in the region of interest. In recent years, angiographic imaging techniques such as digital subtraction angiography "DSA" have been introduced to provide time-resolved 3D angiographic images with similar image quality and x-ray dose as time-resolved 2D angiographic images. However, challenges remain to using 3D angiographic imaging for some clinical investigations, including, for example, the possibility that 3D angiographic imaging systems may limit access to patients, the higher cost of 3D angiographic imaging systems, and the need to provide high-quality 3D angiographic images at low x-ray dose and without increasing acquisition time.

[0016] As an example, an aneurysm coil procedure typically involves the insertion of a wire coil into the aneurysm's sac. This reduces blood flow and allows the blood inside the aneurysm to clot. The goal of aneurysm coiling is to fill the aneurysm sac with enough coil to treat the aneurysm while avoiding overfilling. Prior to an aneurysm coil procedure, static 2D or 3D angiographic images of the aneurysm are often acquired to plan the procedure. During an aneurysm coil procedure, static or time-resolved 2D angiographic images of the aneurysm are often generated using a projection x-ray imaging system to monitor the progress of the procedure. To detect coil material flowing into the parent vessel, the radiographer typically attempts to find an orientation of the projection x-ray imaging system that provides a 2D image where there is a minimal amount of overlap between the sac and the parent vessel and minimal shortening of the parent vessel. However, providing an optimal view of the aneurysm can be difficult. The orientation of the projection x-ray imaging system may need to be adjusted multiple times to provide an optimal view of the aneurysm.

[0017] Sometimes, projection x-ray imaging systems with multiple source-detector configurations are used to simultaneously provide 2D angiographic images with different views of the aneurysm to overcome the limitations of a single view. Such imaging systems are often referred to as "biplane" x-ray imaging systems. The orientation of both of these source-detector configurations may need to be adjusted several times to provide the desired optimal view. Furthermore, as the coil procedure progresses, the optimal view of the aneurysm may change. When a coil is inserted into the aneurysm, blood flow changes both within the aneurysm and within the parent vessel. For example, before the aneurysm coil procedure begins, blood flows into the aneurysm from the portion of the parent vessel proximal to the aneurysm, resulting in slow blood flow to the portion of the parent vessel distal to the aneurysm. As the aneurysm coil procedure progresses, less blood flows into the aneurysm and blood flow to the portion of the parent vessel distal to the aneurysm returns to a normal rate. Also, the aneurysm may be obscured by the inserted coil. As a result, the orientation of the projection x-ray imaging system may need to be re-adjusted multiple times during the aneurysm coil procedure, which impedes workflow. Other examples of procedures in which blood flow is altered during the procedure include atherectomy (ie, removal of plaque or calcification from a blood vessel) and balloon angioplasty (ie, dilation of a blood vessel having a stenosis).

[0018] Providing time-resolved intraoperative 3D angiographic images of the aneurysm may seem to address the challenge of precisely aligning and realigning the orientation of a 2D x-ray imaging system to provide and maintain an optimal view of the aneurysm. However, the use of a 3D imaging system also introduces challenges. For example, acquisition of a 3D image interrupts the procedural workflow, as it typically requires procedural staff to leave the operating room and return after acquisition of the 3D image. Furthermore, to generate the DSA image, an initial 3D mask image must be acquired, followed by a contrast-enhanced 3D image to subtract the background anatomical structures contained in the mask image and retain only the vasculature in the time-resolved 3D angiographic image. This significantly increases image acquisition time and presents the possibility of further reducing x-ray dose when generating a temporal sequence of images. Three-dimensional x-ray imaging systems may also limit access to the region of interest. Furthermore, the cost of a 3D x-ray imaging system is typically higher than that of a 2D x-ray imaging system.

[0019] 1 is a flow chart illustrating an example of a method for providing a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure. With reference to FIG. 1, a computer-implemented method is provided for providing a temporal sequence 110 of 3D angiographic images representing the flow of contrast agent through a region of interest 120. The method includes: A step S110 of receiving volumetric image data 130a, 130b representing a region of interest 120; receiving a temporal sequence 140 of 2D angiographic images representing the flow of contrast agent through a region of interest 120; a step S130 of inputting the volumetric image data 130a, 130b and a temporal sequence 140 of 2D angiographic images into a neural network NN1; generating a temporal sequence of predicted 3D angiographic images 110 in response to the input S140, the temporal sequence representing the flow of contrast agent through the region of interest 120; Including, The neural network NN1 is trained to predict a temporal sequence of 3D angiographic images 110 from a temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images 110 by the volumetric image data 130a, 130b.

[0020] Since the predicted temporal sequence of 3D angiographic images is provided by a neural network from the temporal sequence of 2D angiographic images, the method offers the advantages of the temporal sequence of 3D angiographic images, including structural information that allows a deeper understanding of the flow of contrast agent in the region of interest, without the need to precisely align the imaging system with the anatomical structures and without limited access to the region of interest.

[0021] FIG. 2 is a schematic diagram illustrating an example of a system for providing a temporal sequence 110 of 3D angiographic images representing the flow of contrast agent through a region of interest 120, according to some aspects of the disclosure. The operations described in connection with the method illustrated in FIG. 1 may be performed by the system 200 illustrated in FIG. 2. Similarly, the operations described in connection with the system 200 may be performed in the method described with reference to FIG. 1. FIG. 3 is a schematic diagram illustrating a first example of a method for providing a temporal sequence 110 of 3D angiographic images representing the flow of contrast agent through a region of interest 120, according to some aspects of the disclosure. With reference to FIGS. 1 and 3, in operation S110, volumetric image data representing the region of interest 120 is received. The region of interest may be, for example, an aneurysm, although the region of interest may generally be anywhere in the vasculature, as discussed above.

[0022] Volumetric image data may generally be generated by a volumetric imaging system. For example, a volumetric X-ray imaging system may be used. Volumetric X-ray imaging systems typically generate image data while rotating or stepping an X-ray source-detector configuration around an imaging region, and then reconstruct the image data obtained from multiple rotation angles into a 3D or volumetric image. Examples of volumetric X-ray imaging systems include computed tomography "CT" imaging systems, cone beam CT "CBCT" imaging systems, and spectral CT imaging systems. In some examples, volumetric image data may be generated using digital subtraction angiography "DSA" techniques. Alternatively, volumetric image data may be generated by a magnetic resonance imaging "MRI" system. For example, magnetic resonance angiography "MRA" volumetric image data may be generated by injecting a contrast agent into the vasculature and using an MRI imaging system to use an oscillating magnetic field at a particular resonant frequency to generate images of various anatomical structures.

[0023] In some examples, the volumetric image data may represent pre-operative data generated prior to insertion of an interventional device into the region of interest 120. The volumetric image data may be generated, for example, during a planning stage or immediately prior to an interventional procedure.

[0024] The volumetric image data received in operation S110 may be received, for example, from a volumetric imaging system, or from a computer-readable storage medium, or from the Internet or the cloud. The volumetric image data may be received by one or more processors 170 shown in FIG. 2. The volumetric image data may be received via any form of data communication, including wired communication, optical communication, and wireless communication. As some examples, when wired or optical communication is used, the communication may be via signals transmitted over electrical or optical cables, and when wireless communication is used, the communication may be via, for example, RF or optical signals.

[0025] In one example, the volumetric image data includes a 3D angiographic image 130a representing the region of interest 120. This example is shown in FIG. 3. In this example, the 3D angiographic image 130a may be generated after injection of a contrast agent. The contrast agent highlights the vasculature in the 3D angiographic image 130a. In another example, described below with reference to FIG. 4, the volumetric image data includes a temporal sequence 130b of 3D angiographic images representing a baseline flow of contrast agent through the region of interest 120. With reference to the example of FIG. 3, the volumetric image data may include a single 3D angiographic image 130a, i.e., only one 3D angiographic image 130a, or may include multiple 3D angiographic images. The single 3D angiographic image 130a may be generated by rotating a source-detector configuration of the volumetric imaging system through a single full rotation or less around the region of interest.

[0026] In operation S120, a temporal sequence 140 of 2D angiographic images representing the flow of contrast agent through the region of interest 120 is received. The temporal sequence 140 of 2D angiographic images may be generated by a projection X-ray imaging system that generates 2D X-ray images. Projection X-ray imaging systems typically include a support arm, such as a so-called "C-arm" or "O-arm," that supports an X-ray source-detector configuration. Projection X-ray imaging systems may alternatively include a support arm having a different shape than these examples. Projection X-ray imaging systems typically generate the projection X-ray images with a support arm that is held in a stationary position relative to the imaging region during acquisition of image data. In some examples, the temporal sequence 140 of 2D angiographic images may be generated using a digital subtraction angiography "DSA" technique. The X-ray projection images may be generated by, for example, a projection X-ray imaging system 180 shown in FIG. 2. As an example, the angiographic images may be generated by a Philips Azurion 7 X-ray imaging system, commercially available from Philips Healthcare of Best, The Netherlands.

[0027] The temporal sequence of 2D angiographic images 140 received in act S120 may be received by one or more processors 170 shown in Figure 2. The temporal sequence of 2D angiographic images 140 may be received via any form of data communication, as described above for the volumetric image data. The temporal sequence of 2D angiographic images 140 represents the same region of interest as the volumetric image data 130a, 130b received in act S110.

[0028] In general, the temporal sequence 140 of 2D angiographic images received in operation S120 may be generated later in time than the volumetric image data 130a, 130b. The volumetric image data may represent pre-operative data generated, for example, before the insertion of an interventional device into the region of interest 120. The volumetric image data may be generated, for example, during a planning phase or immediately prior to an interventional procedure. The temporal sequence 140 of 2D angiographic images may represent intra-operative data generated, for example, during the insertion of an interventional device into the region of interest 120.

[0029] In an operation S130, the volumetric image data 130a and the temporal sequence of 2D angiographic images 140 are input to the neural network NN1. This is shown in Figure 3. In response to the input operation S130, in an operation S140, a predicted temporal sequence of 3D angiographic images 110 representing the flow of contrast agent through the region of interest 120 is generated, as also shown in Figure 3. The neural network NN1 is trained to predict the temporal sequence of 3D angiographic images 110 from the temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images 110 by the volumetric image data 130a.

[0030] As mentioned above, in another example, the volumetric image data input in operation S130 includes a temporal sequence 130b of 3D angiographic images representing a baseline flow of contrast agent through the region of interest 120. This example is described with reference to FIG. 4, which is a schematic diagram illustrating a second example of a method for providing a temporal sequence 110 of 3D angiographic images representing a flow of contrast agent through the region of interest 120, according to some aspects of the present disclosure. As in the example of FIG. 3, in the example of FIG. 4, the temporal sequence 130b of 3D angiographic images may be generated after the injection of contrast agent. In the example of FIG. 4, the temporal sequence of 3D angiographic images represents the flow of contrast agent. In general, the temporal sequence 140 of 2D angiographic images received in operation S120 may be generated later in time than the volumetric image data 130a, 130b. The volumetric image data may represent pre-operative data generated, for example, before the insertion of an interventional device into the region of interest 120. The volumetric image data may be generated, for example, during a planning stage or immediately prior to an interventional procedure. The temporal sequence of 2D angiographic images 140 may represent intraoperative data generated, for example, during the insertion of an interventional device into the region of interest 120 .

[0031] In act S130, the volumetric image data 130b and the temporal sequence of 2D angiographic images 140 are input to the neural network NN1. This is shown in Figure 4. In response to the input act S130, in act S140, a predicted temporal sequence of 3D angiographic images 110 representing the flow of contrast agent through the region of interest 120 is generated, as also shown in Figure 4. The neural network NN1 is trained to predict the temporal sequence of 3D angiographic images 110 from the temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images 110 by the volumetric image data 130b.

[0032] In the example of FIG. 3 and the example of FIG. 4, the predicted temporal sequence of 3D angiographic images 110 is constrained by the volumetric image data 130a and 130b, respectively. This constraint may be applied by calculating the value of a loss function that represents the difference between the predicted temporal sequence of 3D angiographic images 110 and the volumetric image data 130a, 130b. In the inference, the value of the loss function may be used to fine-tune the prediction of the neural network. This loss function has the effect of ensuring that the images in the predicted temporal sequence of 3D angiographic images 110 accurately depict the 3D structure of the region of interest in the volumetric image data 130a, 130b. Loss functions such as L1 or L2 loss, structural similarity index, Huber loss, log cosh loss, etc. may be used for this purpose. The value of this loss function may be used to fine-tune the prediction of the neural network by comparing the 3D structure of the region of interest in the predicted temporal sequence of 3D angiographic images 110 with the 3D structure of the region of interest in the volumetric image data 130a, 130b. This comparison may be performed, for example, by calculating a synthetic or traced 3D angiographic image from the temporal sequence of predicted 3D angiographic images 110. The synthetic image may represent a maximum or minimum or average value of image intensity values ​​in the temporal sequence of 3D angiographic images 110. The synthetic image captures the complete 3D structure of the vasculature with the contrast agent flow captured in the temporal sequence of 3D angiographic images 110. Thus, the structural constraint on the temporal sequence of predicted 3D angiographic images 110 may be applied by calculating the value of a first loss function representing the difference between the synthetic 3D angiographic image calculated from the temporal sequence of predicted 3D angiographic images 110 and the input single 3D angiographic image 130a or the synthetic 3D angiographic image generated from the input temporal sequence of 3D angiographic images 130b. Alternatively or additionally, the flow constraint may be applied by calculating the value of a second loss function that represents the difference between a synthetic projection image generated by projecting the predicted 3D angiography image 110 onto a virtual detector plane corresponding to the detector plane of the input 2D angiography image 140 and the input 2D angiography image 140.In inference, the value of the second loss function may similarly be used to fine-tune the predictions of the neural network. This second loss function has the effect of ensuring that the flow in the images in the predicted temporal sequence of 3D angiographic images 110 accurately depicts the flow of contrast agent in the input 2D angiographic images 140. When two loss functions are used to enforce this constraint, the values ​​of the first and second loss functions may be weighted and summed to provide a combined loss value. In inference, the combined loss value may be used to fine-tune the predicted temporal sequence of 3D angiographic images 110, as described above.

[0033] The virtual detector plane may be determined by identifying the detector plane of the input 2D angiographic image 140 using position encoder data representing the orientation of the x-ray imaging system generating the 2D angiographic image 140 relative to the patient's region of interest. Alternatively, the virtual detector plane may be determined by registering the input 2D angiographic image 140 to the input volumetric image data 130a, 130b and determining the orientation of the x-ray imaging system at which the input 2D angiographic image 140 was generated based on the registration.

[0034] In some examples, a confidence value of the predicted 3D angiography image temporal sequence 110 may also be generated. Outputting the confidence value allows a user to adjust the confidence in the prediction of the neural network NN1. The confidence value may be calculated using various techniques. In one example, the value of the loss function generated by the neural network NN1 as described above is used to provide the confidence value. In this example, a relatively low value of the loss function results in a relatively high confidence value, and vice versa. In another example, the confidence value may be assigned to the prediction of the neural network NN1 by observing an attention map generated by the network and evaluating whether the attention of the network is within the region of interest, in which case the confidence value may be relatively high, or elsewhere, in which case the confidence value may be relatively low.

[0035] In another example, a confidence value for the temporal sequence of predicted 3D angiogram images 110 is generated based on a count of the total number of 2D angiogram images used to generate each predicted 3D angiogram image. In this example, the confidence value for an image may be relatively low if a single 2D angiogram image is used to generate the predicted 3D angiogram image, and the confidence value may be relatively high if multiple 2D angiogram images are used.

[0036] In another example, the confidence value of the predicted 3D angiographic image temporal sequence 110 is generated by calculating one or more analytical metrics, such as overlap or shortening, for features within the region of interest. Techniques for calculating such analytical metrics are disclosed in Wilson, D. et al., "Determining X-ray projections for coil treatments of intracranial aneurysms," in IEEE Transactions on Medical Imaging, vol. 18, no. 10, pp. 973-980, Oct. 1999, doi: 10.1109 / 42.811309, where an algorithm is presented for calculating the aforementioned coefficients, such as overlap and shortening. The algorithm uses as input i) a separately defined 3D segmentation of the region of interest and the surrounding anatomical structures, and ii) a model of the geometric and imaging parameters of the X-ray imaging system. In this example, values ​​of the analysis metric may be calculated for the input temporal sequence of 2D angiographic images 140 based on the volumetric image data 130a, 130b providing a model of the region of interest, and these values ​​may be used to generate a confidence value for the predicted temporal sequence of 3D angiographic images 110. In this example, a high degree of redundancy in the input 2D angiographic images 140 may indicate a relatively low degree of confidence in the output of the neural network NN1, and vice versa.

[0037] In another example, a dropout technique may be used to provide a confidence value of the predicted 3D angiography image temporal sequence 110. The dropout technique involves repeatedly inputting the same data into a neural network and randomly excluding a percentage of neurons from the neural network at each iteration while determining the output of the neural network. The output of the neural network is then analyzed to provide a mean value and a variance value. The mean value represents the final output, and the magnitude of the variance indicates whether the neural network is consistent in its predictions, in which case the variance is small and the confidence value is relatively high, or whether the neural network is inconsistent in its predictions, in which case the variance is larger and the confidence value is relatively low.

[0038] Therefore, the confidence value for the predicted temporal sequence of 3D angiographic images 110 is determining a value of a first loss function LF1 representative of the difference between the received volumetric image data 130a, 130b and the predicted 3D angiographic image 110; determining a value of a second loss function LF2 representing the difference between a synthetic projection image generated by projecting the predicted 3D angiographic image 110 onto a virtual detector plane corresponding to the detector plane of the input 2D angiographic image 140 and the input 2D angiographic image 140; generating an attention map indicating the location of one or more coefficients on which the predicted 3D angiogram image 110 is based, and determining a difference between the attention map and the location of the region of interest 120; Counting the total number of temporal sequences of 2D angiographic training images used to generate the predicted 3D angiographic image 110; The above formula may be calculated by one or more of the following:

[0039] The neural network NN1 may include one or more architectures such as, for example, a convolutional neural network "CNN", a recurrent neural network "RNN", a temporal convolutional network "TCN", an encoder-decoder, a variational encoder-decoder, a transformer, and a generative adversarial network "GAN". The neural network NN1 may include multiple fully connected layers or convolutional layers, including pooling, batch normalization, dropout, nonlinear operations, and the like, among others. The neural network may generate a predicted temporal sequence 110 of 3D angiographic images having the same spatial dimensions as the input volumetric image data and the same temporal resolution as the input temporal sequence 140 of 2D angiographic images. For example, the neural network NN1 may output an n×m×p×t output of gray levels, where n×m×p are the dimensions of the volumetric image data and t is the number of frames in the input temporal sequence 140 of 2D angiographic images. The grey levels in the output image indicate updated contrast flow patterns based on the flow observed intraoperatively in the temporal sequence 140 of 2D angiographic images.

[0040] Details of the training of the neural network NN1 are provided below.

[0041] Thus, in both the example of FIG. 3 and the example of FIG. 4, the predicted temporal sequence of 3D angiographic images 110 is generated from the input temporal sequence of 2D angiographic images 140. As mentioned above, the temporal sequence of 2D angiographic images 140 may be generated later in time than the volumetric image data 130a, 130b. In doing so, the method allows a clinical investigation to be performed using the temporal sequence of 2D angiographic images 140, providing the advantages of the temporal sequence of 3D angiographic images without the need to simultaneously perform a 3D angiographic imaging operation on the region of interest. The volumetric image data may represent pre-operative data generated, for example, before the insertion of an interventional device into the region of interest 120, and the temporal sequence of 2D angiographic images 140 may represent intra-operative data generated, for example, during the insertion of an interventional device into the region of interest 120. As a result, an interventional procedure may be performed with an interventional device without the need to acquire an intra-operative temporal sequence of 3D angiographic images.

[0042] The predicted temporal sequence of 3D angiographic images 110 may be used for various purposes. For example, the 3D angiographic images 110 may be output to a display, such as the display 190 shown in FIG. 2. A user input device, such as a touch screen, mouse, or the like, may be provided to allow a user to adjust the viewing angle to view the 3D angiographic images 110 from a desired orientation relative to the region of interest. In one example, a difference image may be generated and output. The difference image represents changes in contrast flow through the region of interest 120. The difference image may represent, for example, changes in contrast flow between successive angiographic images, or between angiographic images acquired at the beginning of a procedure and angiographic images acquired during a procedure.

[0043] In one example, the predicted 3D angiogram image is projected to provide a 2D angiogram image. In this example, the method described above with reference to FIG.

[0044] The method may include projecting the temporal sequence of predicted 3D angiographic images 110 onto a virtual plane of an X-ray detector 150a of an X-ray imaging system 150 to provide a temporal sequence of predicted synthetic 2D angiographic images representing the flow of contrast agent through the region of interest.

[0045] In this example, a composite 2D angiographic image may be generated by projecting the predicted 3D angiographic image from any orientation relative to the region of interest. The composite 2D angiographic image may be generated from multiple orientations relative to the region of interest to facilitate a deeper understanding of the contrast flow within the region of interest and to better visualize the region of interest without having to position the X-ray imaging system in multiple configurations and expose the patient and staff to X-ray radiation due to multiple image acquisitions. The composite 2D angiographic image may be acquired using the predicted 3D angiographic image and a model of the geometry of the X-ray imaging system. The model may include parameters such as, for example, the relative positions of the X-ray source, the X-ray detector, and the region of interest, as well as the dimensions of the X-ray detector. The projection may be obtained by adjusting the orientation a, b of the X-ray source 150b and the X-ray detector 150a relative to the region of interest 120 and projecting the 3D angiographic image onto the detector. 5 is a schematic diagram illustrating the orientation of the X-ray source 150b and the X-ray detector 150a of the X-ray imaging system 150 relative to the patient's region of interest 120, including the rotation angle of the centerline of the projection X-ray imaging system 150 about the patient's longitudinal axis, according to some embodiments of the present disclosure. FIG. 6 is a schematic diagram illustrating the orientation of the X-ray source 150b and the X-ray detector 150a of the X-ray imaging system 150 relative to the patient's region of interest 120, including the tilt angle b of the centerline of the projection X-ray imaging system 150 relative to the patient's cranio-caudal axis, according to some embodiments of the present disclosure. The rotation angle a and the tilt angle b can be adjusted relative to the region of interest 120 and used to obtain a desired composite 2D angiographic image.

[0046] In one example, the composite 2D angiographic images thus provided are used to provide different views of the region of interest to the view provided by the temporal sequence of 2D angiographic images 140 input to the neural network NN1. In this example, the projection operation described above may also include projecting the predicted temporal sequence of 3D angiographic images 110 onto a virtual plane of the X-ray detector 150a of the X-ray imaging system 150 at an orientation a, b of the X-ray source 150b and the X-ray detector 150a of the X-ray imaging system 150 relative to the region of interest 120, so that the temporal sequence of predicted composite 2D angiographic images represents different views of the region of interest relative to the received temporal sequence of 2D angiographic images.

[0047] In one example, the neural network NN1 is trained to predict a temporal sequence of 3D angiographic images 110 from two temporal sequences of 2D angiographic images representing different views of the flow of contrast agent through a region of interest 120. The temporal sequence of 2D angiographic images with different views of the flow of contrast agent through a region of interest may be acquired using an angiographic imaging system with multiple X-ray source-detector configurations, such as a biplane X-ray imaging system. In such imaging systems, the orientation of each source-detector configuration with respect to the region of interest is different. The orientations may be, for example, orthogonal to each other or at another angle relative to each other. The different views may be used to provide additional details about the region of interest. In this example, information from multiple views is used to improve the predicted temporal sequence of 3D angiographic images 110 generated by the neural network. Information from multiple views may improve the prediction of the 3D structure of the region of interest. Information from multiple views can also assist the neural network in resolving ambiguities in images where the flow of contrast agent appears to slow down by accounting for slowing down as a result of changes in vasculature diameter or as a result of shortening.

[0048] In this example, the method described with reference to FIG. receiving a second temporal sequence of 2D angiographic images representing a flow of contrast agent through the region of interest 120, the second temporal sequence of 2D angiographic images representing a different view of the region of interest 120 relative to the temporal sequence of 2D angiographic images 140; inputting a second temporal sequence of 2D angiographic images into the neural network NN1; generating, in response to the input, a predicted temporal sequence of 3D angiographic images 110 further based on a second temporal sequence of 2D angiographic images; Also includes The neural network NN1 is trained to predict a temporal sequence of 3D angiographic images 110 from two temporal sequences of 2D angiographic images representing different views of the flow of contrast agent through a region of interest 120, and to constrain the predicted temporal sequence of 3D angiographic images 110 by volumetric image data 130a, 130b.

[0049] In one example, the temporal sequence of predicted 3D angiographic images 110 is generated for a particular region of the input 2D angiographic image. In this example, the method described with reference to FIG. 1 also includes receiving a user input indicating an extent of a region of interest in the volumetric image data 130a, 130b and / or the 2D angiographic image 140; and outputting the temporal sequence of predicted 3D angiographic images 110 corresponding to the indicated extent of the region of interest in the volumetric image data 130a, 130b and / or the 2D angiographic image 140. The user input may be provided by a user, for example, in the form of annotations on the 2D angiographic image 140. The user may, for example, manually draw a bounding box around the region of interest. The region of interest may also be automatically identified, after which the user selects the automatically identified region of interest. Techniques for automatic identification of the region of interest include image intensity thresholding, region growing, template matching, level setting, active contour modeling, segmentation by neural networks (e.g., U-Net), feature detection, etc. The inference time of the neural network NN1 may be reduced by indicating the extent of the region of interest in the volumetric image data and / or the 2D angiographic image that are input to the neural network NN1.

[0050] In another example, the predicted temporal sequence of 3D angiographic images 110 is generated using a user-specified temporal resolution or a user-specified time window. In these examples, the method described with reference to FIG. 1 also includes the steps of receiving a user input indicating a temporal resolution for the predicted temporal sequence of 3D angiographic images 110; receiving a user input indicating a temporal window for a predicted temporal sequence of 3D angiographic images; outputting a temporal sequence 110 of predicted 3D angiographic images, each having an indicated temporal resolution and / or within an indicated temporal window; Includes.

[0051] User input indicating a time resolution or time window may be provided by a user input device such as a mouse, touch screen, etc., in combination with a graphical user interface "GUI." The GUI may include, for example, a time interval slider that allows a user to select a frame within the predicted temporal sequence of 3D angiographic images 110. The predicted temporal sequence of 3D angiographic images 110 may be color coded by using different colors to represent the arrival time of contrast agent at different locations within the vasculature. The GUI may also include a window size slider that allows a user to select a time resolution of the predicted temporal sequence of 3D angiographic images 110. The GUI may allow display of synthetic projections of the temporal sequence of 3D angiographic images 110 from user-defined orientations relative to the region of interest.

[0052] As mentioned above, in some examples, the temporal sequence of 2D angiographic images 140 may be generated later in time than the volumetric image data 130a, 130b. For example, the volumetric image data 130a, 130b may represent pre-operative data generated prior to insertion of an interventional device into the region of interest 120, and the received temporal sequence of 2D angiographic images 140 may represent intra-operative data generated during insertion of an interventional device into the region of interest 120. Thus, in one example, the flow of contrast agent through the region of interest 120 in the volumetric image data 130a, 130b is different than the flow of contrast agent through the region of interest 120 in the temporal sequence of 2D angiographic images 140 received.

[0053] In this example, the neural network NN1 may also be trained to predict the percentage of interventional procedures for the region of interest that are completed. The neural network NN1 may be trained to do this by including in the training data 3D angiographic training images and a corresponding temporal sequence of 2D angiographic training images, the latter representing the flow of contrast agent through the region of interest during the interventional procedure for the region of interest, to train the neural network NN1 to predict a ground truth percentage of interventional procedures for the region of interest that are completed. The ground truth percentage of interventional procedures for the region of interest that are completed may be calculated from past procedures. During training, the ground truth and the predicted percentage of interventional procedures for the region of interest that are completed may be compared using a loss function, and the value of the loss function contributes to the value of the loss function used to train the neural network NN1. In inference, the neural network NN1 predicts the percentage of interventional procedures for the region of interest that are completed based on the flow in the temporal sequence of predicted 3D angiographic images. In this example, the neural network NN1 is trained to predict, for an image in an input temporal sequence 140 of 2D angiographic images, the percentage of interventional procedures for a region of interest that are completed using an interventional device, and the method includes: outputting a percentage of the interventional procedure for the region of interest that is completed for the predicted temporal sequence of 3D angiographic images 110; Includes.

[0054] The training of the neural network NN1 is described later.

[0055] Generally, training a neural network involves inputting a training data set into the neural network and iteratively adjusting the parameters of the neural network until the trained neural network provides an accurate output. Training is often performed using a dedicated neural processor, such as a graphics processing unit "GPU" or neural processing unit "NPU" or tensor processing unit "TPU". Training often employs a centralized approach, where a cloud-based or mainframe-based neural processor is used to train the neural network. Following training with the training data set, the trained neural network may be deployed to a device for analyzing new input data during inference. The processing requirements during inference are significantly lower than those required during training, allowing the neural network to be deployed to a variety of systems, such as laptop computers, tablets, mobile phones, etc. Inference may be performed, for example, by a central processing unit "CPU", GPU, NPU, TPU, on a server, or in the cloud.

[0056] Thus, the process of training the above-mentioned neural network NN1 includes adjusting its parameters. The parameters, more specifically, weights and biases, control the operation of the activation function in the neural network. In supervised learning, the training process automatically adjusts the weights and biases so that when presented with input data, the neural network accurately provides the corresponding expected output data. To do this, a value or error of a loss function is calculated based on the difference between the predicted output data and the expected output data. The value of the loss function may be calculated using functions such as negative log-likelihood loss, mean squared error, or Huber loss, or cross-entropy loss. During training, the value of the loss function is typically minimized, and training is terminated when the value of the loss function meets a stopping criterion. In some cases, training is terminated when the value of the loss function meets one or more of multiple criteria.

[0057] Various methods are known for solving loss minimization problems, such as gradient descent and quasi-Newton methods. Various algorithms have been developed to implement these methods and their variations, including, but not limited to, stochastic gradient descent (SGD), batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg-Marquardt, momentum methods, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax "optimizers". These algorithms use chaining to calculate the derivatives of the loss function with respect to the model parameters. This process is called backpropagation, because the derivatives are calculated starting at the last layer or output layer and moving towards the first layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted to minimize the error function. That is, adjustments to the model parameters are made starting at the output layer and working backwards in the network until the input layer is reached. In the first training iteration, the initial weights and biases are often randomized. The neural network then predicts output data, which is also random. Backpropagation is then used to adjust the weights and biases. The training process is performed iteratively by making adjustments to the weights and biases at each iteration. Training is terminated when the error, or difference between the predicted output data and the expected output data, is within an acceptable range for the training data or any validation data. The neural network can then be deployed, and the trained neural network will make predictions for new input data using the trained values ​​of its parameters. If the training process is successful, the trained neural network will accurately predict the expected output data from the new input data.

[0058] In one example, the neural network NN1 receiving S210 a 3D angiography training image 130'a representing a region of interest 120 for each of a plurality of patients; receiving S220, for each patient, a temporal sequence of 2D angiographic training images 140' corresponding to the 3D angiographic training images 130'a, the 2D angiographic training images 140' representing flow of contrast agent through the region of interest 120; For each of a plurality of 2D angiographic training images 140' in the temporal sequence for the patient, and for each of a plurality of patients, Step S230 of inputting 2D angiographic training images 140' into the neural network NN1; generating a corresponding predicted 3D angiogram image 110' representing the flow of contrast agent through the region of interest 120; i) the value of a first loss function LF1, which represents the difference between the received 3D angiography training images 130′a and the predicted 3D angiography images 110′; and ii) values ​​of a second loss function LF2 representing the difference between the input 2D angiographic training images 140′ and the projections 160 of the predicted 3D angiographic images 110′ onto faces corresponding to faces of the input 2D angiographic training images 140′; Step S250 of adjusting parameters of the neural network NN1 based on a step S260 of repeating the input step S230, the generating step S240, and the adjusting step S250 until a stopping criterion is met; is trained to predict a temporal sequence of 3D angiographic images 110 from a temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images by volumetric image data 130a.

[0059] This training method is illustrated in Figure 7, which is a flow chart illustrating a first example of a method for training a neural network to provide a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure. The training method is also illustrated in Figure 9, which is a schematic diagram illustrating a first example of a method for training a neural network to provide a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure.

[0060] 7 and 9, in operation S210, 3D angiographic training images 130'a representing the region of interest 120 are received for a plurality of patients. The 3D angiographic training images may be generated by a 3D angiographic imaging system, such as, for example, a computed tomography angiography "CTA", 3D rotational angiography "3DRA", or magnetic resonance angiography "MRA" imaging system. The 3D angiographic training images 130'a for 100 or more patients may be received in operation S210. The images may include a range of various abnormalities for the region of interest. The images may represent patients having different ages, genders, body mass indexes, etc. The 3D angiographic training images 130'a may be generated prior to an interventional procedure for the region of interest.

[0061] In operation S220, a temporal sequence of 2D angiographic training images 140' is received for each of a plurality of patients. The 2D angiographic training images 140' represent a flow of contrast agent through the region of interest 120. The 2D angiographic training images 140' may represent, for example, a flow of contrast agent during an interventional procedure relative to the region of interest. The 2D angiographic training images 140' may be generated by a projection X-ray imaging system. Alternatively, the 2D angiographic training images 140' may be synthetic 2D angiographic images generated by projecting a 3D angiographic image generated by a volumetric X-ray imaging system onto a virtual detector plane, as described above.

[0062] In operation S230, a number of 2D angiographic training images 140' from a number of patients' temporal sequences are input to the neural network NN1. In operation S240, the neural network NN1 makes a prediction, and in operation S250, the parameters of the neural network NN1 are adjusted based on the values ​​of two loss functions LF1 and LF2. The temporal sequences may be randomly selected from a number of patients in the training data set before inputting them to the neural network NN1. Changing the order of the inputs in this way may improve the predictions by preventing the neural network from memorizing the order of predictions for a set order of inputs. In operation S260, operations S230, S240 and S250 are repeated multiple times. The calculation of these loss functions is shown diagrammatically in FIG. 9. The value of the first loss function LF1 represents the difference between the received 3D angiographic training images 130'a and the predicted 3D angiographic images 110'. The value of the second loss function LF2 represents the difference between the input 2D angiographic training image 140' and the projection 160 of the predicted 3D angiographic image 110' onto a plane corresponding to the plane of the input 2D angiographic training image 140'. The plane onto which the predicted 3D angiographic image 110' is projected may be determined as described above, i.e., may be known based on positional encoder data representing the orientation of the X-ray imaging system generating the 2D angiographic training image 140' relative to the patient's region of interest. Alternatively, the plane onto which the predicted 3D angiographic image 110' is projected may be determined by registering the 2D angiographic training image 140' to the 3D angiographic training image 130'a. The value of the loss function may be calculated using a loss function such as L1 or L2 loss, structural similarity index, Huber loss, log cosh loss, etc. The values ​​of the loss function may be weighted and summed to provide a composite value. The backpropagation technique described above may be used to tune the parameters of the neural network NN1 using the combined value of the loss functions LF1 and LF2.

[0063] The values ​​of the first and second loss functions, i.e. LF1 and LF2, used during training are calculated in the same manner as the first and second loss functions described in connection with Figures 3 and 4. The loss functions also have similar effects as those described above. The first loss function LF1 has the effect of ensuring that the images in the predicted temporal sequence of 3D angiographic images 110 accurately depict the 3D structure of the region of interest in the 3D angiographic training images 130'a. The second loss function LF2 has the effect of ensuring that the flows in the images in the predicted temporal sequence of 3D angiographic images 110' accurately depict the flow of contrast agent in the input 2D angiographic training images 140'.

[0064] In another example, the neural network NN1 receiving a temporal sequence 130'b of 3D angiographic training images representing a baseline flow of contrast agent through a region of interest 120 for each of a plurality of patients; receiving S320, for each patient, a temporal sequence of 2D angiographic training images 140' corresponding to the temporal sequence of 3D angiographic training images 130b', the temporal sequence of 2D angiographic training images 140' representing flow of contrast agent through the region of interest 120; For each of a plurality of 2D angiographic training images 140' in the temporal sequence for the patient, and for each of a plurality of patients, a step S330 of inputting 2D angiographic training images 140' into the neural network NN1; generating a corresponding predicted 3D angiogram image 110' representing the flow of contrast agent through the region of interest 120; i) a value of a first loss function LF1 representing the difference between a corresponding 3D angiographic training image from the received temporal sequence of 3D angiographic images 140'b and the predicted 3D angiographic image 110'; and ii) the value of a second loss function LF2 representing the difference between the input 2D angiographic training image 140′ and the projection 160 of the predicted 3D angiographic image 110′ onto a plane corresponding to the plane of the input 2D angiographic training image 140′; Step S350 of adjusting parameters of the neural network NN1 based on

[0065] a step S360 of repeating the input step S330, the generating step S340 and the adjusting step S350 until a stopping criterion is met; is trained to predict a temporal sequence of 3D angiographic images 110 from a temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images by volumetric image data 130b.

[0066] This training method is illustrated in FIG. 8, which is a flow chart showing a second example of a method for training a neural network to provide a temporal sequence of 3D angiographic images representing the flow of contrast agent through a region of interest, according to some aspects of the present disclosure. In this second example, the training of the neural network NN1 is performed in the same manner as described above for the first example, with the difference that instead of performing the training with a single 3D angiographic training image 130'a representing the region of interest 120, a temporal sequence of 3D angiographic training images 130'b is used. Also, when the value of the first loss function is calculated, in the second example, the value of the first loss function LF1 represents the difference between the corresponding 3D angiographic training image from the temporal sequence of 3D angiographic images 140'b received and the predicted 3D angiographic image 110'. The images correspond in the sense that they represent the same time point. Thus, in this example, the predicted 3D angiographic image 110' may have the same time interval as the temporal sequence of 3D angiographic training images 130'b.

[0067] In the training method described with reference to Fig. 7 or with reference to Fig. 8, further operations may be performed. For example, a user input may be received indicating the extent of the region of interest 120 in the 3D angiographic training images 130'b and / or the temporal sequence of 2D angiographic training images 140', and values ​​of the first loss function LF1 and / or the second loss function LF2 may be calculated only within the indicated range. The training is then tailored to the region of interest. Alternatively or additionally, values ​​of the first loss function LF1 and / or the second loss function LF2 may be calculated by applying a higher weighting to the respective loss functions within the indicated range compared to outside the indicated range. This also has the effect of tailoring the training to the region of interest.

[0068] In the training method described with reference to FIG. 7 or the training method described with reference to FIG. 8, training of the neural network NN1 may include inputting a 3D angiographic training image 130'a, or a temporal sequence of 3D angiographic training images 130'b, to the neural network NN1. This is indicated in FIG. 9 by a dashed arrow between these items. Inputting the 3D angiographic training images to the neural network NN1 in this manner may simplify the training of the neural network, since it only needs to estimate the 3D shape or flow changes in the region of interest based on the contrast flow in the input 2D angiographic training images 140'. In contrast, if the 3D angiographic training images are not input to the neural network NN1, the neural network NN1 learns to predict the structural shape and contrast flow from the input 2D angiographic training images 140', and the 3D angiographic training images are simply used to calculate the value of the second loss function LF1.

[0069] In another example, in the training method described with reference to FIG. 7 or the training method described with reference to FIG. 8, the 3D angiographic training image 130′a or the temporal sequence of 3D angiographic training images 130′b may represent pre-operative data generated before the insertion of an interventional device into the region of interest. The temporal sequence of 2D angiographic training images 140′ may represent intra-operative data generated during the insertion of an interventional device into the region of interest. Furthermore, the flow of contrast agent through the region of interest 120′ in the 3D angiographic training image 130′a or in the temporal sequence of 3D angiographic training images 130′b may be different than the flow of contrast agent through the region of interest in the temporal sequence of 2D angiographic training images 140′.

[0070] In another example, the training method described with reference to FIG. receiving a temporal sequence of 3D angiographic training images representative of flow of contrast agent through the region of interest 120 during or after insertion of an interventional device into the region of interest; It may include, The steps S250 and S350 for adjusting the parameters of the neural network NN1 further include: iii) a value of a third loss function LF3 representing the difference between the predicted 3D angiographic image 110′ and corresponding 3D angiographic training images representing the flow of contrast agent through the region of interest during or after insertion of an interventional device into the region of interest; Based on.

[0071] In this example, training data in the form of a temporal sequence of intraoperative and / or postoperative 3D angiographic images of the region of interest is additionally available. Values ​​of the third loss function LF3 may be calculated in a similar manner to the first and second loss functions LF1 and LF2 described above. In this example, these temporal sequences are used during training to adjust the parameters of the neural network NN1. Using the third loss function in this way may provide faster training of the neural network NN1, since it provides the neural network NN1 with direct information about the intraoperative and / or postoperative 3D structure of the region of interest.

[0072] In one example, the neural network NN1 to be trained is a GAN. The GAN may further use adversarial and / or cycle consistency losses to ensure that the temporal sequence of predicted 3D angiography images is in the "style" of the 3D angiography training images. In inference, the adversarial loss may ensure that the corresponding frames of the input temporal sequence of 2D angiography images 140 are aligned with the projection of the predicted 3D angiography images 110' onto a plane corresponding to the plane of the input 2D angiography training images 140'.

[0073] In another example, a computer program product is provided that includes instructions that, when executed by one or more processors, cause the one or more processors to perform a method for providing a temporal sequence of 3D angiographic images 110 representative of flow of contrast agent through a region of interest 120, the method comprising: A step S110 of receiving volumetric image data 130a, 130b representing a region of interest 120; receiving a temporal sequence 140 of 2D angiographic images representing the flow of contrast agent through a region of interest 120; a step S130 of inputting the volumetric image data 130a, 130b and a temporal sequence 140 of 2D angiographic images into a neural network NN1; generating a temporal sequence of predicted 3D angiographic images 110 in response to the input S140, the temporal sequence representing the flow of contrast agent through the region of interest 120; having The neural network NN1 is trained to predict a temporal sequence of 3D angiographic images 110 from a temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images 110 by the volumetric image data 130a, 130b.

[0074] As described above, features of the computer-implemented method may be implemented in a system. Thus, in one example, a system 200 is provided for providing a temporal sequence 110 of 3D angiographic images representing flow of contrast agent through a region of interest 120. The system 200 includes: receiving (S110) volumetric image data 130a, 130b representing a region of interest 120; receiving (S120) a temporal sequence 140 of 2D angiographic images representing flow of contrast agent through a region of interest 120; inputting the volumetric image data 130a, 130b and the temporal sequence of 2D angiographic images 140 into a neural network NN1 (S130); In response to the input, generate (S140) a temporal sequence of predicted 3D angiographic images 110 representative of flow of contrast agent through a region of interest 120; one or more processors 170 configured to The neural network NN1 is trained to predict a temporal sequence of 3D angiographic images 110 from a temporal sequence of 2D angiographic images 140 and to constrain the predicted temporal sequence of 3D angiographic images 110 by the volumetric image data 130a, 130b.

[0075] An example of this system 200 is shown in Figure 2. System 200 includes one or more processors 170 that may perform the operations described above with reference to the flowchart of Figure 1. System 200 may also include one or more of a projection X-ray imaging system 180 for generating the temporal sequence of 2D angiographic images 140, a display 190 for displaying the temporal sequence of predicted 3D angiographic images 110, a GUI, and a user input device configured to receive user input (not shown in Figure 1), such as a keyboard, mouse, touch screen, etc.

[0076] The above examples should be understood as illustrative and not limiting of the disclosure. Further examples are also contemplated. For example, an example described in relation to a computer-implemented method may be provided by a computer program product, or by a computer-readable storage medium, or by the system 200 in a corresponding manner. It should be understood that a feature described with respect to any one example may be used alone or in combination with other described features, or in combination with one or more features of another of the examples, or with combinations of other examples. Moreover, equivalents and modifications not described above may also be used without departing from the scope of the invention as defined in the appended claims. In the claims, the word "comprises" does not exclude other elements or operations, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used. Any reference signs in the claims should not be interpreted as limiting their scope.

Claims

1. 1. A computer-implemented method for providing a temporal sequence of 3D angiographic images representative of flow of a contrast agent through a region of interest, comprising: receiving volumetric image data representative of the region of interest; receiving a temporal sequence of 2D angiographic images representative of flow of the contrast agent through the region of interest; inputting the volumetric image data and the temporal sequence of 2D angiographic images into a neural network; generating, in response to the input, a temporal sequence of the predicted 3D angiographic images representative of flow of the contrast agent through the region of interest; having the neural network is trained to predict a temporal sequence of the 3D angiographic images from a temporal sequence of the 2D angiographic images and to constrain the predicted temporal sequence of the 3D angiographic images by the volumetric image data. Computer-implemented method.

2. The volumetric image data is a 3D angiographic image representative of said region of interest; or a temporal sequence of 3D angiographic images representing a baseline flow of the contrast agent through the region of interest; 2. The computer-implemented method of claim 1, comprising:

3. projecting the temporal sequence of predicted 3D angiographic images onto a virtual plane of an X-ray detector of an X-ray imaging system to provide a temporal sequence of predicted composite 2D angiographic images representative of flow of the contrast agent through the region of interest; The computer-implemented method of claim 1 or claim 2, further comprising:

4. 4. The computer-implemented method of claim 3, wherein the projecting step includes projecting the temporal sequence of predicted 3D angiographic images onto the virtual plane of the X-ray detector of the X-ray imaging system at an orientation of an X-ray source and the X-ray detector of the X-ray imaging system relative to the region of interest, and the temporal sequence of predicted synthetic 2D angiographic images represents a different view of the region of interest relative to the temporal sequence of received 2D angiographic images.

5. receiving a second temporal sequence of 2D angiographic images representative of flow of the contrast agent through the region of interest, the second temporal sequence of 2D angiographic images representing a different view of the region of interest relative to the temporal sequence of 2D angiographic images; inputting a second temporal sequence of the 2D angiographic images into the neural network; generating, in response to the input, the predicted temporal sequence of 3D angiographic images further based on a second temporal sequence of the 2D angiographic images; and the neural network is trained to predict a temporal sequence of the 3D angiographic images from two temporal sequences of 2D angiographic images representing different views of the flow of the contrast agent through the region of interest, and to constrain the predicted temporal sequence of the 3D angiographic images by the volumetric image data.

2. The computer-implemented method of claim 1.

6. and calculating a confidence value of the predicted temporal sequence of 3D angiographic images.

2. The computer-implemented method of claim 1.

7. The confidence value is determining a value of a first loss function representative of a difference between the received volumetric image data and the predicted 3D angiographic image; and / or determining a value of a second loss function representative of the difference between the input 2D angiographic image and a synthetic projection image generated by projecting the predicted 3D angiographic image onto a virtual detector plane corresponding to a detector plane of the input 2D angiographic image; and / or generating an attention map indicating the location of one or more coefficients on which the predicted 3D angiogram image is based and determining a difference between the attention map and the location of the region of interest; and / or calculating a total number of the temporal sequences of the 2D angiographic training images used to generate the predicted 3D angiographic image; The computer-implemented method of claim 6 , wherein the calculation is performed by:

8. receiving a user input indicating the extent of the region of interest in the 3D angiographic image; and / or receiving a user input indicating the extent of the region of interest in the 2D angiographic image; and / or receiving a user input indicating a temporal resolution for the predicted temporal sequence of 3D angiographic images; and / or receiving a user input indicating a temporal window for the predicted temporal sequence of 3D angiographic images; and / or - outputting a temporal sequence of said predicted 3D angiographic images corresponding to said indicated extent of said region of interest in said 3D angiographic images and / or corresponding to said indicated extent of said region of interest in said 2D angiographic images and / or having said indicated temporal resolution and / or within said indicated temporal window, respectively. The computer-implemented method of claim 1 , further comprising:

9. 9. The computer-implemented method of claim 1, wherein the volumetric image data represents pre-operative data generated prior to insertion of an interventional device into the region of interest, the received temporal sequence of 2D angiographic images represents intra-operative data generated during insertion of the interventional device into the region of interest, and a flow of contrast agent through the region of interest in the volumetric image data differs from a flow of contrast agent through the region of interest in the temporal sequence of received 2D angiographic images.

10. the neural network is trained to predict, for the images in the input temporal sequence of 2D angiographic images, a percentage of the interventional procedure for the region of interest that is completed using the interventional device; outputting the percentage of the interventional procedure for the region of interest that is completed for the temporal sequence of the predicted 3D angiographic images.

10. The computer-implemented method of claim 9, further comprising:

11. The neural network comprises: receiving, for each of a plurality of patients, 3D angiographic training images representative of the region of interest; receiving, for each patient, a temporal sequence of 2D angiographic training images corresponding to the 3D angiographic training images, the 2D angiographic training images representing flow of contrast agent through the region of interest; For each of a plurality of 2D angiographic training images in a temporal sequence for a patient, and for each of a plurality of patients, inputting the 2D angiographic training images into the neural network; generating a corresponding predicted 3D angiographic image representative of the flow of the contrast agent through the region of interest; i) a value of a first loss function that represents a difference between the received 3D angiography training images and the predicted 3D angiography images; and ii) a value of a second loss function representing the difference between the input 2D angiographic training image and a projection of the predicted 3D angiographic image onto a plane corresponding to a plane of the input 2D angiographic training image; adjusting parameters of the neural network based on the repeating the inputting, generating and adjusting steps until a stopping criterion is met; 2. The computer-implemented method of claim 1, wherein the method is trained to predict a temporal sequence of 3D angiographic images from a temporal sequence of 2D angiographic images and to constrain the predicted temporal sequence of 3D angiographic images by the volumetric image data.

12. The neural network comprises: receiving, for each of a plurality of patients, a temporal sequence of 3D angiographic training images representative of a baseline flow of the contrast agent through the region of interest; receiving, for each patient, a temporal sequence of 2D angiographic training images corresponding to the temporal sequence of 3D angiographic training images, the temporal sequence of 2D angiographic training images representing flow of contrast agent through the region of interest; For each of a plurality of 2D angiographic training images in a temporal sequence for a patient, and for each of a plurality of patients, inputting the 2D angiographic training images into the neural network; generating a corresponding predicted 3D angiographic image representative of the flow of the contrast agent through the region of interest; i) a value of a first loss function representing a difference between a corresponding 3D angiographic training image from the received temporal sequence of 3D angiographic images and the predicted 3D angiographic image; and ii) a value of a second loss function representing the difference between the input 2D angiographic training image and a projection of the predicted 3D angiographic image onto a plane corresponding to a plane of the input 2D angiographic training image; adjusting parameters of the neural network based on the repeating the inputting, generating and adjusting steps until a stopping criterion is met; 2. The computer-implemented method of claim 1, wherein the method is trained to predict a temporal sequence of 3D angiographic images from a temporal sequence of 2D angiographic images and to constrain the predicted temporal sequence of 3D angiographic images by the volumetric image data.

13. receiving user input indicating an extent of the region of interest in the temporal sequence of 3D angiographic training images and / or 2D angiographic training images; and the values ​​of the first loss function and / or the second loss function are calculated only within the indicated ranges, or the values ​​of the first loss function and / or the second loss function are calculated by applying a higher weighting to the respective loss functions within the indicated range compared to outside the indicated range.

13. A computer-implemented method according to claim 11 or claim 12.

14. 13. The computer-implemented method of claim 11 or 12, wherein each of the 3D angiography training images or the temporal sequence of the 3D angiography training images represents pre-operative data generated prior to insertion of an interventional device into the region of interest and the temporal sequence of the 2D angiography training images represents intra-operative data generated during insertion of the interventional device into the region of interest, and a flow of contrast agent through the region of interest in the 3D angiography training image or the temporal sequence of the 3D angiography training images differs from a flow of contrast agent through the region of interest in the temporal sequence of the 2D angiography training images.

15. receiving a temporal sequence of 3D angiographic training images representative of flow of the contrast agent through the region of interest during or after insertion of the interventional device into the region of interest; and Adjusting parameters of the neural network. teeth, iii) a value of a third loss function representing a difference between the predicted 3D angiographic image and a corresponding 3D angiographic training image representative of the flow of the contrast agent through the region of interest during or after insertion of the interventional device into the region of interest; Based further on 15. The computer-implemented method of claim 14.

Citation Information

Patent Citations

  • Angiographic image acquisition system and method with automatic shutter adaptation for yielding a reduced field of view covering a segmented target structure or lesion for decreasing x-radiation dose in minimally invasive x-ray-guided interventions

    US20110182492A1

  • Visualization of at least one characteristic variable

    US20190005643A1

  • Systems and methods for classification of arterial image regions and features thereof

    WO2020146905A1

  • Anatomical and functional assessment of coronary artery disease using machine learning

    WO2021102408A1