IVUS-enabled contrast agent dual energy x-ray imaging

Abstract

A system (SYS) and related method for image processing comprising an input interface (IN) for receiving input data comprising: i) a spectral projection image of a region of interest comprising a conduit (CN) for the passage of a liquid (CA), the spectral projection image can be obtained in an imaging process by means of a spectral X-ray imaging device (IA1) with a contrast agent (CA) present in the liquid, and ii) additional image data which can be acquired by a further imaging device (IA2) of a non-ionizing type and which represents 3D information of the pipe. A predictor component (PC) predicts a contrast agent concentration in the conduit based on the spectral projection image and the additional image data.

Description

Technical Field

[0001] The invention relates to a system for image processing, an associated method, an imaging device, a computer program element and a computer readable medium. Background Art

[0002] Spectral X-ray systems (eg, like C-arm systems with dual-layer detectors) can enable more advanced imaging methods, especially but not limited to, during minimally invasive procedures. Spectral imaging can image specific materials.

[0003] When imaging poorly radiopaque structures such as blood vessels (eg, coronary arteries of the heart), the use of contrast agents may enable better image contrast with respect to these structures.

[0004] Contrast quantification, blood and contrast agent dilution measurements, and dynamic behavior of contrast agents are measures that can help extract quantitative information about blood flow from X-ray images. For example, blood flow volume, velocity, and distribution have broad physiological significance for vascular diseases such as coronary artery disease and peripheral artery disease. Such flow-related diagnostic quantities / measurements can be used to diagnose disease, plan procedures, evaluate outcomes, etc. The use of spectral imaging systems alone (such as dual-energy X-ray) has provided improved options for contrast agent-based X-ray image analysis.

[0005] However, in certain circumstances, greater precision with respect to such diagnostic quantities / measures may be desired. Summary of the invention

[0006] Therefore, there may be a need for improved contrast agent based imaging with improved utility, in particular with enhanced extraction of physiological information. In particular, there may be a need to extract reliable and accurate physiological information.

[0007] The objects of the invention are solved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.It should be noted that the aspects of the invention described below apply equally to the related method, imaging device, computer program element and computer readable medium.

[0008] According to a first aspect of the present invention, there is provided a system for image processing, comprising:

[0009] At least one input interface is configured to receive input data, the input data including: i) a spectral projection image of a region of interest, the region of interest including a pipe for liquid to pass through, the spectral projection image can be obtained by a spectral X-ray imaging device in an imaging procedure using a contrast agent present in the liquid, and ii) additional image data that can be acquired by another imaging device of a non-ionizing type, the additional image data representing 3D information of the pipe.

[0010] A predictor component is configured to predict at least a concentration of contrast agent in the conduit based on the spectral 2D projection image and the additional image data when the system is in use.

[0011] In an embodiment, the 3D information represents a lumen of a conduit.

[0012] In an embodiment, the system further comprises a flow assessment component configured to calculate a quantity of interest describing one or more flow characteristics of the fluid based at least on a predicted concentration of contrast agent in the conduit.

[0013] In an embodiment, said quantity of interest is a medical quantity of interest.

[0014] In an embodiment, the medical quantity of interest comprises any one or more of: volume flow rate, transit time of contrast agent through at least a portion of the conduit, liquid-contrast agent mixing behavior, and transluminal attenuation gradient.The flow may be related to collateral flow.

[0015] The system thus provides a more cost-effective and practical method (particularly but not exclusively for interventions) to estimate these quantities, which were previously unattainable without expensive rotational X-ray based CT scanner tomography equipment, which is not required in the present invention.

[0016] In an embodiment, the input data comprises in situ pressure readings, wherein the quantity of interest that the flow assessment component can calculate comprises a pressure estimate for a distal end of a section of tubing.

[0017] In an embodiment, the predictor component or the flow assessment component is configured to calculate an initial concentration of contrast agent upon entry into said conduit and / or a rate at which said contrast agent enters said conduit.

[0018] A system according to any preceding claim, wherein the conduit is part of the vascular, urinary or lymphatic system of a mammal, such as a human patient.

[0019] In an embodiment, the further imaging device is configured for intravascular imaging.

[0020] A system according to any one of the preceding claims, wherein the other imaging device is any one or more of: i) an ultrasound imaging device or ii) an optical tomographic coherence imaging device.

[0021] In an embodiment, the spectral X-ray imaging device is of a C-arm type having a detector configured to perform energy-differentiated detection of X-ray intensities.

[0022] In an embodiment, the spectral projection image comprises a time series of frames.

[0023] In an embodiment, the spectral projection image comprises a coronary angiogram.

[0024] In an embodiment, the calculated contrast agent concentration and / or amount of interest is provided for processing, wherein the processing comprises one or more of: i) displaying an indication of the contrast agent concentration and / or amount of interest on a display device, ii) storing it in a medical data storage device, and iii) controlling the operation of a medical device, iv) processing by a decision support system.

[0025] In an embodiment, the predictor component is based on a machine learning model.

[0026] In another aspect, an imaging apparatus is provided, comprising the system according to any one of the preceding claims, further comprising a spectral X-ray imaging apparatus.

[0027] In an embodiment, the apparatus may include a second imaging device. The second imaging device needs to be configured for spectral imaging, but may be so configured in some embodiments if desired.

[0028] In another aspect, there is provided an image processing method, comprising:

[0029] receiving input data, the input data comprising: i) a spectral projection image of a region of interest, the region of interest comprising a pipe for liquid to pass through, the spectral projection image being able to be acquired by a spectral X-ray imaging device in an imaging procedure using a contrast agent present in the liquid, and ii) additional image data that can be acquired by another imaging device of a non-ionizing type, the additional image data representing 3D information of the pipe; and

[0030] When the system is in use, at least a concentration of contrast agent in the conduit is predicted based on the spectral projection image and the additional image data.

[0031] In another aspect, a method of training an ML model based on training data is provided.

[0032] In a further aspect, a computer program element is provided which, when executed by at least one processing unit, is adapted to cause the processing unit to perform an image processing method or an ML training method.

[0033] In another aspect, at least one computer-readable medium is provided, on which the program unit is stored, or on which the machine learning model is stored.

[0034] It is proposed herein to combine spectral (such as dual energy) X-ray projection image data with additional sources of 3D lumen information about the vessel, such as by using intravascular ultrasound (IVUS) data, OTC data or other additional non-ionizing imaging modalities, to achieve accurate dilution quantification and bolus spread tracking, etc. The vessel CN ​​may be a blood vessel (such as a coronary artery or vein, or others), and the liquid / fluid may be blood including a contrast agent mixture of any concentration.

[0035] Without, preferably, 3D lumen information from intravascular imaging, intravascular imaging of blood with contrast agent dilution can only be derived under various simplifying assumptions (e.g., uniform background, circular vessel shape, no foreshortening, etc.). While this modeling without additional imaging to obtain 3D lumen information is not necessarily excluded herein, such additional imaging is indeed used in preferred embodiments because it allows for more accurate, patient-tailored estimates of blood flow volume. Uncertainty in the estimated flow velocity from bolus propagation can be reduced herein.

[0036] It has been found that the combination of 3D IVUS vascular data and spectral (such as dual energy) X-ray projection image data (preferably collected at various locations of the vascular structure) gives good results. This combination allows for a particularly accurate assessment of the dilution / concentration of blood and contrast agent. In addition, bolus propagation velocity and volumetric blood flow rate are estimated. This 3D vascular data and spectral (such as dual energy) X-ray projection image data at multiple locations in the vessel / conduit may also be collected for other additional imaging modalities besides IVUS (such as OTC or other modalities). The second imaging modality is preferably interventional, but not necessarily so.

[0037] Therefore, what is proposed herein is to combine spectral X-ray projection data with additional sources of 3D information about the duct system of interest. The proposed method exploits the extremely high contrast agent focusing imaging capability of spectral projection imaging, as well as the additional 3D luminal information that can be acquired by the non-ionizing imaging modalities (such as IVUS or OTC), and is safer and computationally less expensive than using CT X-ray based reconstruction.

[0038] This 3D information from IVUS or OTC is also more accurate than a CT (computed tomography) scan and does not need to be explicitly registered computationally to the 2D spectral image, since both image streams are recorded in the same session. With intravascular imaging, 2D spectral and 3D information can be recorded simultaneously in parallel, providing local registration of the two image streams.

[0039] X-ray projection images may be provided by a dual-layer C-arm system that provides images during spectral procedures. Other spectral setups may also be used, such as dual sources, photon counting, etc.

[0040] As used herein, the conduit is preferably and generally part of the vascular system (artery or vein), but in some applications, the conduit can be part of a contrast agent delivery device, such as part of a catheter or pipeline, through which the contrast agent is released into the vascular system or other body part of interest. At least a portion of the catheter / pipeline, such as its tip portion, is recorded in the spectral projection image, and the system / method can use the principles described herein to calculate / determine the initial contrast agent concentration, in particular as described above. This operation is preferably done using prior knowledge of the shape and / or material composition of the relevant catheter / pipeline. If the 3D shape is (preferably) known, and this is the case in most cases, then the initial concentration can be determined in this way even without using 3D information from a second imaging device. Therefore, if only an initial concentration is required, a second imaging device is not required.

[0041] A "user" refers to a person who operates an imaging device or oversees an imaging procedure, such as a medical staff member or other person. In other words, a user is generally not a patient.

[0042] In general, "machine learning" includes a computerized device (or module) that implements a machine learning ("ML") algorithm. Some such ML algorithms can adjust a machine learning model that is configured to perform ("learn") a task. Other MLs operate directly on training data, not necessarily using a model. This adjustment or update of the training data corpus is called "training". In general, the task performance of an ML module may improve significantly as training experience accumulates. The training experience may include exposure of suitable training data and models to such data. The more the data reflects the task to be learned, the better the improvement in task performance. "If the training data is a good representation of the distribution of examples that measure the performance of the final system, then the training experience helps improve performance." Performance can be measured by objective tests based on the output generated by the module after receiving test data. Performance can be defined with respect to a specific error rate achieved with given test data. For example, see TM Mitchell, "Machine Learning", page 2, section 1.1, page 6 at 1.2.1, McGraw-Hill, 1997. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Exemplary embodiments of the present invention will now be described with reference to the following drawings, which, unless otherwise indicated, are not to scale, wherein:

[0044] Figure 1 A block diagram of a medical imaging apparatus that may be used, among other things, to support contrast agent-based imaging is shown;

[0045] Figure 2A schematic diagram of a conduit, such as a vessel, in which a contrast agent resides is shown;

[0046] Figure 3 A block diagram of a computer system for calculating contrast agent concentration is shown;

[0047] Figure 4 shows a schematic diagram of a graphical display according to one embodiment contemplated herein;

[0048] Figure 5 It shows that Figure 1 A block diagram of a machine learning model used in an embodiment of a system;

[0049] Figure 6 A training system for training a machine learning model is shown;

[0050] Figure 7 A method for calculating contrast agent concentration based on spectral projection images and a flow chart using the calculated contrast agent concentration are shown; and

[0051] Figure 8 A method for training a machine learning model is shown. DETAILED DESCRIPTION

[0052] First reference Figure 1 , which provides a block diagram of a medical imaging device MIA configured to facilitate the calculation of flow-related quantities, which can be used, for example, to assess a patient's vascular lesions. Specifically, and as will be explained in more detail below, a computer-implemented system SYS is configured herein to accurately calculate the concentration of a contrast agent in a liquid. The liquid containing the contrast agent passes through a conduit CN, such as a pipeline or a piping system or the like. Then, the contrast agent concentration can be used as an input to calculate other diagnostic quantities of interest related to the flow characteristics of the liquid with the contrast agent when it flows through the conduit CN. This paper mainly contemplates blood flow through vascular CN (such as arterial or venous coronary arteries in the human heart) or through other blood vessels of other organs. However, other conduit structures, such as the lymphatic system, the urinary system, the gastric system, etc., are not excluded herein, and non-medical applications, such as speleology or hydrological inspections, inaccessible conduit systems, hydraulic systems, or other conduits in machine parts in vehicles, or inspections in any other type of machinery, etc., are not excluded.

[0053] The medical imaging device MIA may preferably comprise an X-ray based imaging device IA (also referred to herein as "imager" for short). The imaging device IA may be of C-arm / U-arm type, so that projection images λ can be collected along different projection directions P(α) around a region of interest, such as around the stenosis in the mentioned coronary vessels. The imaging device IA comprises an X-ray source XS and an X-ray sensitive detector XD. In some embodiments, the X-ray source XS and the X-ray sensitive detector XD opposite thereto are preferably arranged on a rotatable gantry G. The gantry G may have a recess, in which an examination region is defined, in which the patient or in particular the region of interest is located during imaging. Thus, the rotatable gantry may have a "C", "U" or similar shape, such as a C-arm imaging device envisaged herein. However, such or similar rotational or other dynamic arrangements (in which there is at least some relative movement between the source and the detector) are not necessarily required in all embodiments. Rather, in some embodiments, a planar radiographic arrangement is also envisaged. Instead of a biplane imaging device such as a rotatable gantry, a biplane imaging device having two detectors with intersecting imaging axes (e.g. at right angles) may be used. Figure 1 The rotatable device shown.

[0054] Furthermore, while digital source imaging (such as a flat panel detector) is preferred, where the image itself is acquired in the form of digital data, such an arrangement is not necessarily required. More traditional, cassette-operated analog imaging methods (based on X-ray film) are not excluded herein. The analog image can be digitized after acquisition, for example, by using a digital camera. In some embodiments, imaging based on post-digitization of an X-ray image intensifier or the like is also contemplated.

[0055] In general, imaging includes energizing an X-ray source XS so that an X-ray beam XB is emitted from the focus of the source XS, passes through the inspection area (including the area of ​​interest) and interacts with the patient's tissue material. The interaction of the X-ray beam XB with the tissue material causes the beam to change. The modified beam can then be detected at the detector XD. The conversion circuit (not shown) converts the detected intensity into a digital image, in particular into a projection image. The user can use these images in a medical procedure. During imaging, the patient may need to lie on a patient support PS, such as a table. Soft tissues such as those constituting coronary vessels generally provide poor contrast in native X-ray images. In order to enhance contrast, a contrast agent (sometimes referred to as "CA"), such as an iodine solution, gadolinium or other, is delivered to the patient's bloodstream through an entry point AP before or during imaging. The contrast agent CA can be delivered via a CA delivery device DA (such as a motorized pump), but manual delivery is not excluded herein. The delivery device DA can be operable to deliver a defined volume of contrast agent to the patient. The volume of contrast agent so delivered travels with the bloodstream. After a certain time, contrast agent will accumulate in the region of interest (such as at or around a stenosis). Image acquisition is preferably synchronized with the arrival of the contrast agent at the ROI. Preferably, "contrast" images are acquired when contrast agent CA is present at the ROI. Optionally, additional "non-contrast" images are acquired when contrast agent CA is not present in the ROI.

[0056] In particular, contrast images can be acquired in a time series, wherein the concentration of contrast agent accumulated in the region of interest varies. Image acquisition may begin before contrast agent accumulates at the ROI, may continue, or may terminate after accumulation. Generally speaking, the concentration of contrast agent increases over time after delivery, stabilizes at a specific maximum value, and then gradually fades or decreases over time until it disappears completely. Although in some cases a single or specific discrete still image can be acquired, it is preferred to acquire streaming images or frames acquired at a suitable frame rate because this allows real-time dynamic observation. Specifically, the acquired projection image stream can be visualized as a real-time video source on a display device DD by a visualizer VIZ. Alternatively, or in addition, still images can also be so visualized. The operation of the auxiliary system SYS is preferably based on processing such still or streaming images. The images can be stored in an image memory MEM, or can be processed in other ways.

[0057] X-ray imaging is typically based on the attenuation coefficient of (one or more) regions that constitute the region of interest. The attenuation (coefficient) depends on the energy of the X-ray beam. In order to exploit this energy dependency, a spectral imager IA configured for spectral imaging can be used. This may include solutions on the detector side or on the source side. Detector side solutions include photon counting detector hardware or dual energy setups with dual layer hardware. More than two detector layers can be used, but this is less common. Source side solutions include multi-source setups, kVp switching, filtering, etc.

[0058] The imaging setting in spectral imaging allows the acquisition of multi-dimensional image data. This multi-dimensional image data includes projection images of different energies, rather than acquiring energy integral images provided by traditional (non-spectral) X-ray imaging settings, the latter of which is not excluded in this article. For example, this conventional setting can be used for digital angiography subtraction ("DAS"), in which pairs of projection images at higher and lower concentrations of contrast agents are acquired. These image pairs are then spatially registered and subtracted from each other to enhance the contrast of soft tissues including contrast agents. DSA can also be practiced with a spectral imaging setting.

[0059] However, in order to more accurately quantify the contrast agent concentration / volume of the region of interest, a spectral setting is preferably adopted. This spectral imaging setting includes a spectral image processor SP. The spectral image processor SP can implement a spectral image processing algorithm, in particular a material decomposition algorithm. Alternatively, further spectral image processing algorithms can be envisioned herein. The spectral processor SP processes the multi-energy projection image λ acquired by the source-side or detector-side imaging device just described, and calculates therefrom a spectral image that is different from the original input multi-energy projection image. Therefore, the spectral image is computationally derived from the input multi-energy projection image. Such a spectral image may be located in the projection domain, and may include, for example, a monochromatic image of a desired energy level, a virtual non-contrast image, a contrast-only image (e.g., an iodine image), a Compton scattering image, and the like. However, preferably, the spectral image includes a contrast-only image, and the monochromatic image may correspond to the attenuation coefficient of the contrast agent used. For example, the energy value or energy range of the monochromatic image may be selected to be equal to or include the K-edge energy of a specific contrast agent for good contrast. However, selections such as those based on the K-edge are not mandatory herein, and other energy values ​​or ranges of monochromatic images are also envisioned herein. The monochrome images represent a calculated approximation of an image that might be obtained when imaging using a monochrome X-ray source XS of the corresponding energy level. In most practical cases, the actual X-ray source XS is polychromatic. The virtual non-contrast ("VNC") image represents an approximation of an image that might be obtained if no contrast agent was present in the field of view of the imager IA to the exclusion of other materials. The contrast ("VC") image represents an approximation of an image that might be obtained if only contrast agent was present in the field of view of the imager IA to the exclusion of other materials. Therefore, the image contrast in the VC is entirely or primarily focused on the contrast agent. Preferably, the energy spectrum (projection image) calculated by the energy spectrum processor SP is processed by the facilitation system SYS to replace or supplement the original multi-energy input projection image, as will be described in more detail below.

[0060] For simplicity, the label "λ" may be used herein to indicate an input image that is typically processed by the facilitating system SYS. That is, "λ" may be related to the acquired original projection image, or preferably to the energy spectrum projection image calculated by the energy spectrum processor SP based on the acquired original projection image. The energy spectrum processor SP and the auxiliary system SYS may be implemented on different computing systems, which may be geographically separated, such as in a distributed "cloud" architecture. However, this article also contemplates implementation on a single computing system.

[0061] Although the rotating imaging device IA preferably has multi-directional imaging capabilities, in embodiments, it is primarily envisioned that only projection images are processed in the projection domain, and furthermore, the source 3D information is generally computationally cheaper than tomographic reconstruction, as will be discussed in more detail below. No tomographic reconstruction algorithm is required herein, thereby saving computation time. Therefore, the spectrum processor SP can operate in the projection domain.

[0062] Reference now Figure 2 , which shows a schematic diagram of a situation that may be encountered when imaging a conduit CN through which a liquid with contrast agent CA passes, as envisioned in the embodiments herein. In a preferred embodiment, the conduit (system) CN may include the arterial or venous vasculature of the human heart or other organs. Figure 2 As shown in the upper branch of the vascular system, stenosis may cause the lumen L of the vascular system to narrow. This stenosis may be caused by the accumulation of deposits such as plaques. This stenosis may cause health problems. By image-based examination of the blood flow characteristics in the affected vessel CN, this stenosis, its severity or other aspects of it can be correctly identified. When the blood-contrast agent mixture passes through the vessel CN ​​and fills the lumen, the blood mixed with the contrast agent can provide better imaging contrast of the lumen space structure.

[0063] Continue to refer Figure 2 , which provides a cross-sectional view of such vasculature including branches through which contrast agent CA is flowing with blood along the length l of the respective branch. The region of interest ROI" shown as a dotted line can be geometrically defined automatically (e.g. by segmentation) or as specified by the user (e.g. by using a pointer tool). The region of interest can be demarcated by a graphical user interface on the acquired image and / or can be done by collimator actions as required. The contrast agent concentration calculation described in this article may be associated with the region of interest. Therefore, the main focus of this article is on the calculation of the contrast agent CA concentration in the portion of the vasculature CN defined by the ROI. The ROI may be associated with a sub-portion of the vasculature CN (e.g. Figure 2As shown), such as associated with one or more branches (with or without their side branches), or possibly associated with the entire imaged vascular system in the current field of view. Therefore, the contrast agent concentration refers to the amount of contrast agent, the number of particles, the volume or the mass in the reference volume defined by the region of interest in the corresponding vessel CN. In particular, the main focus is on the amount of contrast agent in the lumen L. The lumen L represents the (entire) space inside the vessel CN ​​surrounded by the vessel W. Preferably, what can be calculated herein is the volume concentration or mass concentration of the contrast agent CA in the lumen L portion inside the ROI. For example, the mass concentration can be defined as the ratio of the mass of the contrast agent in the ROI under consideration to the lumen volume L. When only blood is present in the ROI, the contrast agent concentration may be essentially zero, such as the amount previously reached by the administered contrast agent (bolus) as it moves with the blood flow. At the other extreme, the concentration may be such that there is essentially no blood, such as when the contrast agent replaces all the blood in the ROI under consideration. The contrast agent concentration will be understood to vary over time. Thus, the contrast agent concentration may relate to the current concentration at a given time instant / frame, or may relate to an average of contrast agent concentration readings / values ​​over a period of time, or another such composite quantity that combines contrast agent concentration information.

[0064] Although other applications of the principles described herein in medical and non-medical applications supporting image-based inspection of duct systems are also contemplated herein, ducts CN are primarily referred to herein as vascular CNs, and it should be understood that such other applications are not excluded herein. In addition, contrast agent CA is sometimes referred to herein as iodine, but it should be understood that other radiopaque substances (such as gadolinium, mixtures thereof, or other substances) may also be used herein. Figure 2 Also illustrated in the lower left is the projected footprint of the tip portion TP of the tube or conduit through which contrast agent can be administered and which may appear in the FOV of certain frames if desired.

[0065] Reference now Figure 3 A block diagram of a system SYS for facilitating improved contrast agent-based imaging, specifically, the system can facilitate determining a contrast agent concentration in a vessel CA or a portion thereof. The system can also facilitate determining one or more blood flow-related diagnostic quantities based on the contrast agent concentration. The system SYS can be configured for at least dual-channel processing. Specifically, the system can include one or more input ports IN, through which input data of at least two channels are received. The two-channel input data includes a vascular angiography spectral X-ray image of one channel, and spatial 3D information / data of at least one other channel.

[0066] The spatial 3D information describes the 3D volume structure or configuration of the vessel CN ​​(particularly its lumen L) at the ROI, while the spectral X-ray image represents the number or amount of contrast agent residing in the vessel CN ​​of the considered region of interest at a given time. Specifically, the contrast of the spectral X-ray image may be modulated by the quality of the contrast agent present in the ROI. Therefore, the spectral X-ray image may represent the quality of the contrast agent in the vessel cross section CN at the ROI.

[0067] In some embodiments, the input data and system are three-channel. Therefore, the input data includes in situ pressure readings collected by the pressure reading device MD in addition to the spatial structure lumen 3D data and the spectral image.

[0068] The system SYS jointly processes the dual or triple channel input data to calculate the contrast agent concentration in the vessel CN ​​at the ROI. In particular, such multi-channel input data are processed by a predictor component PC of the system SYS to predict such concentration. For example, in units of mass per volume, such as units of milligrams of contrast agent per milliliter or other suitable quantities. The output data provided by the predictor component PC at the output interface OUT represent the concentration of the contrast agent currently located in the considered region of interest.

[0069] Therefore, the system proposed herein combines information from at least two channels - information from the (angiographic) spectral X-ray image stream and from additional 3D information - to make an accurate estimate of the contrast agent concentration.

[0070] The input data, in particular the spectral X-ray image, may include a series of frames ("stream"). In this case, the output data provided by the predictor component PC may accordingly include a time series of such concentration values / readings. Alternatively, multiple frames are processed in this article to calculate a single contrast agent concentration value as an output. The operation of the system SYS is preferably real-time and / or dynamic. In particular, when the input data receives a new frame, the predictor component PC calculates a new / updated concentration reading. The 3D lumen information according to the second channel can be updated accordingly, or it can be used for one or more spectral angiography frames as needed. A single concentration reading or a series of such concentration readings is output at the output interface OUT. For the sake of brevity, we refer to it as a concentration value in this article, and it should be understood that such concentration values ​​are the concentration values ​​of the contrast agent CA considered in this article. In addition, it should be understood that the output data may include a single such concentration reading for a given frame, or may include a time series of concentration values, with one or more concentration values ​​for each frame considered and received at the input port IN.

[0071] The concentration values ​​provided via the output interface OUT may be of interest per se and can therefore be stored, processed or displayed on a display device DD as required.

[0072] The calculated concentration value can be monitored using a monitoring device MT. The monitoring device MT can use a threshold value process. If the concentration value exceeds or, as the case may be, falls below the threshold, a warning signal is issued to warn the user. The warning signal can be visual, audio, tactile or a combination of any of the above.

[0073] However, in general, what is of interest herein is the amount that can be derived from the concentration value. Therefore, the flow assessment component FAC can be used to calculate the relevant flow characteristics of the liquid including blood and contrast agent based on one or more concentration values ​​outputted. This flow characteristic of the blood / contrast agent mixture based on the contrast agent concentration may be beneficially used to guide the diagnosis or treatment steps downstream. The diagnostic amount can be calculated by the assessment component FAC based on one or more concentration values. The flow characteristic or diagnostic amount can be stored or processed. For example, the diagnostic amount can be transferred to the diagnostic system DS to provide diagnostic information, such as a score for the presence of a medical condition. The score and / or the diagnostic amount can be processed by the graphic display generator GDG to generate a graphic display for displaying on the display device DD. The graphic display can include the representation of one or more quantities of the diagnosis. The diagnostic amount related to the flow can be combined with the concentration value and / or the score to be displayed in the graphic display on the display device DD.

[0074] The monitoring device MT (or a different such device) may be used to monitor said quantity derived from the concentration value, instead of or in addition to the concentration value itself.

[0075] The graphical display generated by the generator GDG may comprise at least part of the input image in addition to the output data, such as concentration values ​​and / or flow properties derived therefrom.

[0076] The calculated diagnostic amount and / or concentration value can be provided to the control interface CI to control other types of medical equipment. The control interface CI provides a control signal m-OUT based on the calculated diagnostic amount and / or concentration value to control the equipment. For example, the control signal mm-OUT can be used as a feedback system to control the settings of one or two imagers IA2, IA2 of a measuring device (such as an in-situ pressure measuring device). The marker TG can optionally be used to mark the corresponding frame and / or the optionally associated 3D measurement result d with the corresponding calculated contrast agent concentration C.

[0077] The structural space 3D information describes in particular the configuration of the lumen in the region of interest in the vessel CN ​​under consideration, preferably derived from or including an image provided by the second imaging modality / imager IA2. The second imaging modality IA2 can be configured to provide such a 3D structural image of the lumen. Suitable such 3D imaging modalities IA2 include ultrasound ("US") imaging or optical coherence tomography ("OCT") images. Preferably, the second imaging modality IA2 is of non-ionizing type to reduce health risks to the patient. Similarly, a single such 3D image (such as a US or (intravascular) OCT image) can be provided and processed by the system SYS as a time series / stream, preferably temporally aligned with a still frame or frames of the spectral image. However, the processing of a single captured frame is not excluded herein.

[0078] Preferably, the second imaging modality IA2 for providing 3D lumen structure information is invasive, such as IVUS. In IVUS, a suitably miniaturized US imaging probe is introduced into the patient through an entry point AP and delivered to the ROI through a duct. The ultrasound imaging probe may include an ultrasound transducer for generating in situ (inside the vessel CN) sound waves that interact with the lumen wall structure of the region of interest. Its reflection is recorded by a corresponding ultrasound receiver and processed into an ultrasound image to display the 3D structure of the lumen. The ultrasound transducer and ultrasound receiver are preferably integrated into the probe as transceiver components, but such a transceiver arrangement is not required here. Suitable US techniques may include PMUT or CMUT. Either one can be designed for multi-frequency IVUS applications. In this technology, one or more vibrating elements ("active sources") can be configured as piezoelectric resistors or capacitive membranes to send and receive different carrier frequencies. In many systems, such transducer elements and discrete collections or arrays, each element can be tuned to a specific frequency.

[0079] Similarly, and also invasively, optical imaging can be used in OTC to acquire 3D structural information using a suitable optical probe. Other types of interventional imaging modalities may include endoscopic ultrasound imaging, in which a probe is inserted along the patient's esophagus to a position close to the heart.

[0080] Although such interventional imaging is very useful, external imaging, such as ultrasound imaging applied from the outside, may sometimes be used instead when a suitable probe is introduced into the patient's body (particularly but not necessarily into the vascular CN). Furthermore, the spatial 3D information provided as input data together with the spectral X-ray projection images may not necessarily include images acquired during the intervention, but may instead relate to previous images obtained in (one or more) previous imaging sessions (such as MRI, CT, or other). Previous images for a given patient can be retrieved from a medical database for the given patient and can be processed herein to replace or supplement the current interventional images provided by the second imaging modality IA2.

[0081] It is preferred to use interventional imaging (such as the IVUS system or OTC described) with the angiographic data stream to appropriately combine them as described herein for processing by the predictor component PC to produce an accurate estimate of the contrast agent concentration. In particular, it has been found that the combination of IVUS images and angiographic spectral data has a particular synergistic effect in producing accurate and realistic contrast agent concentration estimates.

[0082] The predictor component PC may be arranged herein as a machine learning model M suitably trained on training data, as will be described in more detail below. However, other modeling than machine learning is also contemplated herein, such as via traditional analysis methods based on thresholding and / or segmentation and an anatomical and physical understanding of the flow characteristics of the conduit CN and / or the liquid of interest. Hybrid approaches are also contemplated, where part of the calculation is based on machine learning, while other calculations use the analysis methods / modeling.

[0083] Reference now Figure 4 , which shows a schematic diagram of a graphic display GD generated by a graphic display generator GDG based on output data. The graphic display GD may be displayed on a display device DD, for example, during an intervention or in a subsequent post-analysis session.

[0084] The graphic display GD according to an embodiment may include a plurality of panels Pj. The multi-panel graphic display GD may include a panel P1 representing interventional image data, in particular 3D structural information of a lumen. Figure 4 Embodiments of IVUS are presented from multiple views, for example, along the lumen length l, and / or in planar views, which can be reconstructed from UV measurements taken within the lumen.

[0085] One or more additional panels P2, P3 (in some embodiments, a single such additional panel is sufficient) include angiographic spectral image frames. Again, only one frame may be displayed in a single panel P2 or P3, displaying 3D image information together and simultaneously with panel P1. Panels P1, P2, P3 may be displayed simultaneously, or the user may cycle through them in sequence as desired.

[0086] Panel P1 may include two or more sub-panels, such as P11, P12 as shown. One such sub-panel P11 may include images taken along the length of the lumen, while one or more other sub-panels P12 include (one or more) planar views thereon.

[0087] Preferably, one or more graphic correlator components AC may be used to support the user in spatially correlating the 3D information in panel P1 with the angiographic projection data information in panels P2 and / or P3. For example, (one or more) image portions in 3D image panel P1 may be highlighted by overlay structures of different colors, for example, which are associated with corresponding positions in the projection images in panels P2, P3 by lines or other associated symbol widgets.

[0088] Optionally, one or more curves C are shown I , C I’ , which represents the concentration values ​​collected over time shown in panels P4, P5, corresponding to the spectral projection data in one or more panels P2, P3. Likewise, a single panel with a single projection image may be sufficient, optionally associated with a second panel (such as P4) to display an associated fill curve for that projection data.

[0089] Fill curve C I , C I’ The diagram shows the typical phases expected as a concentrated bolus passes through a region of interest of interest. There is a ramp-up phase, where the concentration increases, followed by a plateau phase, where the contrast agent reaches saturation and remains relatively constant, and then a decline phase, where the contrast agent is washed out as it leaves the region of interest.

[0090] Reference now Figure 5 , which shows a schematic diagram of a machine learning (“ML”) model M that can be used to implement a predictor component PC.

[0091] In the ML approach, the potential relationship between the observable contrast on the image in the spectral projection image and the 3D information and the corresponding contrast agent concentration can be implicitly learned from the training data, without necessarily requiring explicit analytical ad hoc modeling. Instead, a general machine learning model M architecture can be used instead. Based on previous training data (such as can be found in a medical database), specific parameters of the machine learning model are adjusted based on the training set of the training phase to implicitly model the relationship. Once fully trained (which can be determined by testing), the trained model M can be deployed in the medical practice described above to accurately and robustly estimate the contrast agent concentration.

[0092] Additional learning inputs may be provided to enrich the input data (λ, d) with contextual data κ. The contextual data κ may include some or all of the patient's physiological characteristics, and / or some in situ measurements. The contextual data κ may include preoperative 3D images such as CT, MRI or others so that the model M can take into account the surrounding anatomical structures for better learning. Alternatively or in addition, the contextual data κ may include biometric features such as gender, age, weight, height, etc.

[0093] The initial (e.g. arbitrarily chosen) parameters θ of the machine learning model M k Adjustments may be made in one or more iterations k based on the training data (possibly enriched by contextual data κ) to capture this underlying mapping or relationship between the combined input data (λ, d) and the contrast agent concentration C. However, as mentioned above, this machine learning model approach is not required, and other approaches, particularly analytical modeling approaches, may also be envisioned as they may provide sufficiently approximate results in some cases.

[0094] Continue and refer to the Figure 5 , which shows a block diagram of a machine learning model M that may be used in some embodiments. The machine learning model may be arranged, for example, as an artificial neural network ("NN"), preferably a convolutional type of NN, referred to as a "CNN". CNNs are useful in processing spatially correlated data, such as image data, the processing of which is contemplated herein, as previously described.

[0095] The network M may include a plurality of computational nodes arranged hierarchically in a cascaded manner, with the data flow from left to right and thus from one layer to another. The model may therefore be of feed-forward type. Recurrent networks are not excluded herein, in particular if the input data (λ, d) has a time series characteristic, such as a stream of data collected over a period of time. This is particularly envisaged herein, as such time series type input data has been found to produce trained models with good robustness. However, this article also envisages processing of input data (λ, d) where one or both of the two channels (λ and d) contain a single frame.

[0096] The model network M can be said to have a deep architecture because it has multiple hidden layers. In a feedforward network, the "depth" is the number of hidden layers Lj (1≤j≤N) between the input layer IL and the output layer OL, while in a recurrent network, the depth is the number of hidden layers multiplied by the number of rounds.

[0097] The machine learning model includes specific parameters, such as weights of convolutional filters CV, which are adjusted during the training phase, as will be explained in detail below. Once trained, i.e., after the parameters are appropriately adjusted, the input X, including the input contrast image λ (particularly the contrast projection image λ) and the 3D data d (such as IVUS image data), which may be enriched by the contextual data κ, is input and received at the input layer IL during deployment or testing. Therefore, the input X is in the form of (λ, d) or (λ, d, κ). The input data X in the deployment or test is propagated through the various layers of the model and appears as the desired result M(X) = C as the output layer OL. Therefore, at the output layer OL, the desired result is produced, including, in some embodiments, a concentration C in a suitable unit, such as mg / ml, etc. Therefore, the output may be regressed to a scalar value C. In some embodiments, instead of regression, classification may be sufficient, where, for example, the output only indicates which range / interval the concentration value C falls into, such as C∈[5mg / ml,10mg / ml].

[0098] Therefore, a machine learning model M such as Figure 5 The neural network examples shown in or others can be understood as acting as transformers, where two-dimensional or higher-dimensional data ((λ.d) or (λ,d,κ)) is transformed into a single scalar value C as the input data passes through the IL, Lj (j≥1), OL layers of the network M.

[0099] The output layer OL may be a regression layer, where the input data, including images (λ.d) or (λ, d, κ), is regressed into a single scalar value C. Alternatively, a classifier layer OL may be used, where the input is classified into a range / interval of C values ​​as described above.

[0100] For classification results, the output layer OL can be configured as a softmax-function layer or similar computational node, where the feature maps from the previous (one or more) nodes are combined into normalized counts to represent the classification probability or score for each class. In a regression setting, OL can be a fully connected layer.

[0101] Preferably, some or all of the hidden layers Lj (1≤j≤N) and the optional input layers are convolutional layers, i.e., comprise one or more convolutional filters CV that process the input feature maps from earlier layers into layer outputs, sometimes called logits. An optional bias term may be applied, for example, by addition. The activation layer processes the logits in a nonlinear manner into the next generation of feature maps, which are then output and passed as input to the next layer, and so on. The activation layer may be implemented as a rectified linear unit RELU as shown, or as a soft-max function, a sigmoid function, a tanh function, or any other suitable nonlinear function. Unlike the convolutional layer, the output layer may be a fully connected layer. Hybrid networks comprising fully connected layers and convolutional layers are also contemplated herein.

[0102] Additional optional layers such as dropout layers and pooling layers P can be used in the CNN model M. The pooling layer reduces the dimensionality of the output, while the dropout layer cuts off the connection between nodes in different layers.

[0103] Hybrid models are also contemplated, optionally including a preprocessing network, such as an autoencoder (AE) or a variational AE (VAE). The preprocessor first preprocesses the input to produce an intermediate output, and then classifies or regresses the intermediate output into a contrast agent concentration value C or range, as described above. For example, the preprocessor AE can be trained to segment the input image to obtain major blood vessels, microvessels, and perfused tissue, etc. Therefore, the various parts of the input image can first be classified into segmentation classes. Then, for each class, the remaining network can determine the amount of contrast in each vascular category / concentration. Therefore, in general, no matter which preprocessor (AE or non-AE) is used, it can be used to generate position-dependent C(s) values ​​(s represents image position or neighborhood / subset), rather than a global value for the entire image. The embodiments herein contemplate local or global C value calculations.

[0104] The trained models may be stored in one or more data memories MEM'.

[0105] Preferably, in order to achieve good throughput, the computing device PU comprises one or more processors (CPUs) supporting parallel computing, such as processors of a multi-core design. In one embodiment, GPU(s) (Graphics Processing Unit) are used.

[0106] Figure 6 A schematic diagram of a training system TS for training a machine learning model is shown, such as Figure 5 The machine learning model shown in or other.

[0107] The training system TS can be used to train such a model M based on training data (x, y) that may be stored in the training data storage TD. The training data (x, y) can be collected from existing medical data, such as PACS, HIS or other databases or storage systems that can be found in medical databases. Such training data may include historical image data or other health records of patients who have undergone similar examinations, where hemodynamic indicators have been calculated, or at least contrast images of relevant ROIs have been obtained. For example, the historical images can be reviewed by human experts. The experts can review the contrast images of historical patients and select historical images corresponding to specific contrast agent concentrations C based on their medical knowledge. Human experts can review the health records of patients to collect records about individual patients to draw their conclusions. Therefore, the human experts can assign corresponding scores y (or labels) to different corresponding samples of the training input x = (λ.d), y may be an estimate of the concentration of C. Alternatively, the label y may be related to the concentration range.

[0108] Respective (historical) context data κ, such as historical concentration estimates, etc., can be used to assist the user in assigning his score y. The context κ can be found in the historical patient's health records. In some cases, the health record context c can be used to infer the correct concentration value C. Specifically, medical records related to historical images can be collected by human users or automatic text analysis tools (such as NLP (natural language pipeline) or string matching tools such as grep or others) to extract the corresponding concentration values, because this information may sometimes be mentioned in the corresponding report or record with reference to a specific image frame or series of frames, such as an average concentration value. In this way, the collection process can be fully or partially automated, so that appropriately labeled training data can be collected more quickly.

[0109] Instead of searching and evaluating historical training data, experiments can also be performed on phantoms to acquire suitable pairs of: i) image + 3D information and ii) their associated contrast concentrations, where both i) and ii) can be changed / controlled by the experimenter. Such an experimental setup can also be used to determine the coefficients of the analytical model function.

[0110] Training, managed by the training system TS, is the process of adjusting model parameters based on training data. An explicit model is not necessarily required, as in some examples the training data itself constitutes the model, such as clustering techniques or k-nearest neighbors. In explicit modeling, such as NN-based methods and many other methods, the model may include a system of model functions / computation nodes whose inputs and / or outputs are at least partially interconnected. The model function or node is associated with parameters θ that are adjusted during training. The model function may include weights of convolution operators and / or nonlinear units, such as those described above. Figure 5The RELU associated with the NN model in . In the NN case, the parameters θ may include the weights of the convolution kernel of the operator CV and / or the weights of the nonlinear unit.

[0111] The parameterized model can be formally written as M θ . Parameter tuning can be achieved by a numerical optimization procedure. The optimization may be iterative. The objective function f can be used to guide or control the optimization process. The parameters are adjusted or updated by the updater UP, thereby improving the objective function. Input training data x i is applied to the model. The model responds to the training data and produces output M(x i )=y i The objective function maps from the parameter space to a set of numbers. The objective function f measures the output y of the training data. i With the corresponding target y i The parameters are adjusted iteratively to reduce the combined deviation until the stop condition preset by the user or designer is met. The objective function can use the distance measurement ||.|| to quantify the deviation.

[0112] In some embodiments (but not all), the combined bias can be implemented as i, y i ) i The sum of some or all of the residual values, and the optimization problem can be expressed with respect to the objective function as follows:

[0113] argmin θ f=∑ k || M θ (x k ),y k || (1)

[0114] In setting (1), the optimization is formulated as the minimization of a cost function f, but we are not limited to this as the dual formulation of maximizing a utility function can be used instead. The summation is performed over training data instances i.

[0115] The cost function f may be pixel / voxel based, such as an L1 or L2 norm cost function. For example, in least squares or similar methods, the (squared) Euclidean distance type cost function in (1) can be used for the regression task mentioned above. When the model is configured as a classifier, the sum in (1) is expressed as one of the cross entropy or Kullback-Leiber divergence or similar values. The Huber cost function can be used.

[0116] The specific functional composition of the updater UP depends on the optimization process implemented. For example, a method such as back / forward propagation or other gradient-based methods can be used to adjust the parameters θ of the model M so as to reduce the error for all or part of the training pairs (x k ,y k ). Such subsets are sometimes called batches, and optimization can be performed in batches until all the training data set is exhausted, or until a predefined number of training data instances have been processed.

[0117] Training may be a one-time operation or it may be repeated as new training data becomes available.

[0118] Optionally, one or more batch normalization operators ("BN", not shown) may be used. The batch normalization operator may be integrated into the model M, for example coupled to one or more convolution operators CV in a layer. The BN operator may mitigate the vanishing gradient effect, which is the gradual reduction in the magnitude of the gradients in repeated forward and backward passes of a gradient-based learning algorithm during the learning phase of the model M. The batch normalization operator BN may be used for training, but may also be used for deployment.

[0119] like Figure 6 The training system shown in can be considered for all learning schemes, especially supervised schemes. In alternative embodiments, unsupervised learning schemes are also contemplated herein. The training system TS can be implemented using a GPU.

[0120] Figure 7 A flow chart of a computer-implemented method that can be used to implement the system SYS described above is shown to calculate the contrast agent concentration and optionally the volume flow of blood, such as a diagnostic quantity derived from the concentration value. However, it should be understood that the steps described below are not necessarily related to the system described above, but can be understood as teachings of themselves.

[0121] In step S705, at least 2 channels of input data are obtained. This step may include using spectral processing to obtain spectral X-ray projection data. Step S705 may also include collecting 3D lumen information, such as by OTC or IVUS or other means. In an IVUS embodiment, 3D lumen information can be obtained by moving the US probe through a pullback operation of a narrow ST. Step S705 may also include two image data streams, spatial or temporal registration of spectral projection data λ and 3DIVUS or OTC or other imaging modality data. The spectral projection data λ may include, for example, coronary CN angiography frames (with contrast agent) and optional fluoroscopic frames (without contrast agent). Such a registration step is optional, for example, in IVUS, local registration can be achieved by parallel acquisition in two channels.

[0122] In some embodiments, it can be assumed that the initial concentration of the contrast agent bolus administered when recording the angiographic frame is known or calculated herein, as will be explained in more detail below. The initial (i.e., initially administered) contrast agent concentration and / or the administration rate of the contrast agent can form context data κ, which is provided here as an additional input to the system (particularly for model M) to be processed together with the energy spectrum input image. Such additional inputs can be used regardless of whether ML is used. Using such context data can obtain better results, better performance, better learning, etc.

[0123] IVUS / OTC 3D data and spectral X-ray projection data may be acquired in any order, or preferably simultaneously in parallel.

[0124] In step S710, input data is received. As described, the input data is preferably dual-channel, including energy spectrum X-ray projection data λ and 3D information d of the lumen structure of the vessel or other conduit to be inspected / imaged as a second channel.

[0125] Preferably, in addition to the X-ray spectral imager IA1 for the first channel λ, 3D lumen information is also provided by a second imaging modality IA2. Specifically, IVUS is used as the second imaging modality IA in a preferred embodiment. Alternatively, OTC may also be used alternatively or additionally. Any other non-ionizing imaging method capable of providing 3D information about lumen structure is also contemplated herein. In a less preferred embodiment, lumen modeling may be used based on preoperative data. Although image data is preferred, this may not necessarily be necessary because geometric modeling based on shape primitives (such as mesh models) may replace or additionally use intraoperative or interoperative image data or preoperative prior image data. A general vessel / lumen model may be used, which is derived from average measurements taken from a sample population of patients. In some general models, the lumen may be set to have an assumed, idealized circular cross-section. Alternatively, the general model may be adjusted based on the biological characteristics of a particular patient. However, preferably, the 3D information is image data acquired during the current process. It has been found that the combination of angiographic spectral projection data and ultrasound (especially intravascular ultrasound) can produce good results. The input image λ and 3D information d are preferably obtained when a certain amount of contrast agent is at least partially located in the region of interest (such as a portion) of the vasculature CN of the region of interest. The dual channel input data (λ, d) may be a time series acquired over time as the contrast bolus passes through the region of interest.

[0126] The input image λ is preferably obtained based on projection data acquired by a spectral X-ray imager IA (such as a dual energy or photon counting device). The projection data is processed by a spectral image processing algorithm to obtain the input spectral X-ray projection λ as an image of the contrast agent only, as described above.

[0127] In step S720 , the dual channel input data (λ, d) are combined / processed using a machine learning algorithm or analytical model to calculate an estimate C for the contrast agent concentration in the region of interest.

[0128] The calculated concentration C may be output at step S730 for further processing, such as storage, display, or use in control operation of a medical device currently being examined, or input into a diagnostic system, etc.

[0129] In a preferred embodiment, this further data processing may include step S740, in which a blood flow characteristic of one or more quantities is calculated, in particular based on the contrast agent concentration value provided in step S730. If desired, some quantities may be calculated in this step using the initial contrast agent calculated here.

[0130] The concentration value may be a single value at a given time, or preferably comprises a time series of such values, such as a filling curve, wherein different contrast agent concentration values ​​are calculated at different moments in time. The different moments in time preferably correspond to moments in time of input data, in particular to the acquisition time of spectral projection images and / or the acquisition time of 3D information, such as 3D information provided by IVUS or by any other suitable 3D non-ionizing additional imaging modality.

[0131] In an embodiment, the input data is 3-channel and therefore includes additional in-situ (i.e., intravascular) pressure readings of the third channel, which can be collected at the ROI by a pressure measurement device. These additional data may include vital sign measurements, such as heart rate, body temperature, etc. In addition, further data received through the third channel may also include background information κ describing the patient's biological characteristics. This can be used for analytical modeling, but is more suitable for machine modeling. Using such contextual data can make the machine learning process more efficient and / or make the performance of the training model more robust. As described above, in addition or as an alternative, the context information κ may include information such as one or more of the following: i) the contrast agent substance (iodine, etc.) administered, ii) its initial concentration, iii) the total volume administered, iv) its administration rate, and so on. In general, the context information κ may describe the context around the contrast agent itself.

[0132] In the following, the above steps will be described in more detail with various embodiments and sub-steps, in particular the step S740 of calculating the flow characteristic based on the concentration value and the step S720 of calculating the concentration value.

[0133] First, turning to step S720 in detail, since the specific material of the contrast agent (such as iodine) is known, and thanks to the spectral projection image processed in this article, the accurate extraction of the contrast agent mass can be facilitated. The spectral projection image is obtained by spectral processing using, for example, the material decomposition mentioned above. The spectral projection image is preferably a contrast agent-only (or contrast-only) image, in which the contrast is entirely or primarily attributed to the contrast agent in the field of view of the spectral imager IA1. Due to spectral imaging, the attenuation contribution from intermediate structures, in particular the attenuation contribution from the surrounding blood into which the contrast agent is injected, is largely eliminated. Therefore, compared with conventional X-ray projection images acquired only in an energy-integrated imaging setting, only the spectral contrast imaging image data preferably proposed in this article is used as input, and the total mass of the contrast agent in the ROI can be more accurately identified. In step 1120, the contrast agent mass M I Extraction can be done by threshold-based segmentation or other methods. Extract the total mass (e.g., iodine) M from a given frame. I , can be performed on a selected vessel segment, such as the portion at the beginning of IVUS pullback. In addition, in step S720, the lumen volume such as IVUS image data is acquired based on the 3D information d. The extracted mass M is formed I The obtained lumen volume V V The ratio M I / V V To quantify the contrast agent concentration C I。

[0134] If a machine learning model is used in step S720, such explicit mass extraction and lumen volume determination operations are not required, since the contrast agent concentration C can be obtained end-to-end by the trained model M. The model M processes the input data (λ, d) jointly in step S720, for example by passing the combined data (λ, d) through a neural network to obtain the quantified C at its output layer. I Used to find contrast agent concentration.

[0135] Turning now to step S740 in more detail, this may include calculating the volume flow rate, bolus travel time t based on the contrast agent / iodine concentration. B , side branch flow, and (inhomogeneous) mixing behavior.

[0136] For example, using the original / initial iodine concentration C O Calculate the mixing ratio of blood and contrast agent This will allow the contrast agent injection site to be determined. O =F I ·R BI The volume flow rate, where F I is the contrast agent injection rate.

[0137] Using a second vessel segment along the IVUS pullback path (or from a second IVUS acquisition), another local concentration C is calculated. I '(See above Figure 4 By comparing the time curve C′ I (t) and C I (t), calculate C′ I (t) = C I (t+t B ) Bolus propagation time between two vessel segments t B .

[0138] Collateral flow in specific vessel segments and heterogeneous mixing behavior in aneurysms can be assessed and emphasized by observing changes in stable iodine concentrations. I .

[0139] In the embodiment, in step S740, based on the concentration value C I Determine TAG (intraluminal attenuation gradient). TAG can be determined using the average contrast opacity in a series of regions of interest along the vessel of interest. Concentration values ​​can be used to estimate the opacity. For example, the ROI can be evenly distributed and perpendicular to the centerline of the vessel (such as an artery). The TAG process can be configured to measure the gradient (rate of decline) of the contrast agent opacity along the vessel. For example, TAG can be defined as the average value of the opacity and how the average value varies along the vessel. For example, TAG can be calculated as the regression coefficient of a line that fits a graph of the average such contrast agent opacity and the distance from a reference point (such as the coronary artery ostium). This type of TAG processing is described below: AJ Einstein in Journal of the American College of Cardiology, vol.61, issue 12, 2013, pp 1280-1282.

[0140] This paper envisions Figure 6 Various refinements of the method in will now be explained.

[0141] In step S740, in order to determine the blood-contrast agent mixing ratio R BI , the initial iodine concentration C0 and / or the contrast agent injection rate in the spectral X-ray projection image of the catheter TP during contrast agent administration (for volume flow rate F O For this purpose, a segmentation of the guide catheter and a region of interest around the catheter, such as at the nozzle portion TP (see Figure 2The lower left of the guide catheter) appears in the frame in the FOV. A known guide catheter model or specification (any one or more of size, (3D) shape and material) helps to improve the calculation of the initial iodine concentration. The specification can be provided by the user interface UI. The region of interest around the catheter is used to monitor the injection rate over time. Therefore, the attenuation contribution of the guide catheter can be eliminated, and since the shape / size of the guide catheter at the region of interest is known, an accurate estimate of the initial contrast agent concentration can be obtained based on the bolus through the guide catheter. The initial concentration obtained based on the catheter / pipeline portion within the field of view can be used in combination with the local concentration in different ROIs of the vessel of interest, because some diagnostic quantities may require the initial concentration to be calculated. The specifications of the shape and material of the pipeline / delivery tube can be provided by the input interface UI or accessed by a database, etc. The predictor component PC may not require any machine learning to determine the initial CA contrast agent concentration based on a priori knowledge of the shape / material of the pipeline / delivery tube. Similarly, if time is taken into account, the administration rate can be determined.

[0142] In a further refinement, the extracted flow velocities can be combined with additional vessel segmentation, or with the 3D IVUS model and catheter pressure data to create a fluid dynamics based estimate of distal pressure in said vessel. This allows for indirect quantification of disease for the microvasculature by calculating, for example, the IMR index. This is of particular interest for patients without significant epicardial vascular disease (NOCAD patients).

[0143] For volume flow and / or bolus travel time, by IVUS pullback across a lesion ST (such as a stenosis / calcified plaque / stent / stent with restenosis), the severity or flow relevance of the vasculature CN can be assessed.

[0144] Reference now Figure 8 , which shows a flowchart of a method for training a machine learning model M, such as for Figure 6 method.

[0145] In step S810, training data is received, for example, in the form of pairs (x k ,y k ). Each pair consists of the training input x k and the associated target y k .x k , as defined above.

[0146] In step S820, the training input x k Applied to the initialized machine learning model M to produce training outputs.

[0147] In step S830, from the associated target y k The training output M(xk ) is quantified by a cost function F. In step S840, one or more parameters of the model are adjusted in one or more iterations in the inner loop to improve the cost function. For example, the model parameters are adjusted to reduce the residual measured by the cost function. If a convolutional model M is used, these parameters include in particular the weights of the convolution operator.

[0148] The training method then returns in an outer loop to step S810 where the next pair of training data is fed. In step S840, the parameters of the model are adjusted so that the aggregate residual of all pairs considered is reduced, in particular minimized. The cost function quantifies the aggregate residual. Forward-backward propagation or similar gradient-based techniques may be used in the inner loop. In step S810, the loop does not iterate over a single pair, but over a collection of training data pairs or batches, and these are then fed in at once for processing in steps S820 and S830.

[0149] Examples of gradient-based optimization may include gradient descent, stochastic gradient, conjugate gradient, maximum likelihood, EM maximization, Gauss-Newton, etc. We also contemplate other methods besides gradient-based methods, such as Nelder-Mead, Bayesian optimization, simulated annealing, genetic algorithms, Monte Carlo methods, etc.

[0150] More generally, the parameters of the model M are adjusted to improve an objective function F, which is either a cost function or a utility function. In an embodiment, the cost function is configured to measure aggregate residuals. In an embodiment, the aggregation of residuals is achieved by summing all or part of the residuals of all considered pairs. Specifically, in a preferred embodiment, the outer summation is performed in batches (subsets of training instances), and the residuals of their summation are considered at once when adjusting the parameters in the inner loop. The outer loop then continues with the next batch, and so on, until the desired number of training data instances has been processed.

[0151] In the above ML embodiments, the model M can be trained to regress or directly classify flow-related diagnostic quantities. In such embodiments, the concentration value is implicitly calculated, rather than explicitly calculated for delivery to the flow assessment component as in the above embodiments, where the diagnostic quantity is analytically calculated in a downstream step based on the predicted concentration. However, this article prioritizes the concentration sought by explicit prediction.

[0152] The components of the system SYS may be implemented as one or more software modules, running on one or more general purpose processing units PU, such as a workstation associated with the imager IA, or on a server computer associated with a group of imagers. The workstation or other (one or more) data processing unit PU may be part of the imaging device IA.

[0153] Alternatively, some or all components of the system SYS may be arranged in hardware, such as a suitably programmed microcontroller or microprocessor, for example an FPGA (field programmable gate array) or a hardwired IC chip, an application specific integrated circuit (ASIC), integrated into the imaging system SYS. In yet another embodiment, the system SYS may be implemented partly in software and partly in hardware.

[0154] The different components of the system SYS may be implemented on a single data processing unit PU. Alternatively, some or more components are implemented on different processing units PU, possibly arranged remotely in a distributed architecture and connectable in a suitable communication network, such as in a cloud setup or a client-server setup or the like.

[0155] One or more features described herein may be configured or implemented as circuits encoded in a computer-readable medium or using circuits encoded in a computer-readable medium, and / or combinations thereof. The circuits may include discrete and / or integrated circuits, systems on a chip (SOCs) and combinations thereof, machines, computer systems, processors and memories, computer programs.

[0156] In a further exemplary embodiment of the present invention, a computer program or a computer program element is provided, characterized in that it is adapted to perform the method steps of the method according to one of the preceding embodiments on a suitable system.

[0157] The computer program element can therefore be stored on a computing unit, which can also be part of an embodiment of the present invention. The computing unit can be suitable for executing the steps of the above method or causing the execution of the steps of the above method. In addition, it can also be suitable for operating the components of the device described above. The computing unit can be suitable for automatically operating and / or executing the user's command. The computer program can be loaded into the working memory of a data processor. The data processor can therefore be equipped to implement the method of the present invention.

[0158] This exemplary embodiment of the invention covers both a computer program that right from the beginning uses the invention and a computer program that by means of an update turns an existing program into a program that uses the invention.

[0159] Furthermore, the computer program element may be able to provide all necessary steps to complete the procedure of the exemplary embodiment of the method as described above.

[0160] According to another exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is proposed, wherein the computer readable medium has a computer program element stored thereon, the computer program element being as described in the previous section.

[0161] The computer program may be stored and / or distributed on a suitable medium (particularly, but not necessarily, a non-transitory medium) provided together with other hardware or as part of other hardware, such as an optical storage medium or a solid-state medium, but may also be published in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0162] However, the computer program may also be offered over a network like the World Wide Web and can be downloaded from such a network into a working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, the computer program element being arranged to perform one of the previously described embodiments of the invention.

[0163] It must be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method claims, while other embodiments are described with reference to device claims. However, it can be concluded by a person skilled in the art from the above and following descriptions that, unless otherwise indicated, any combination of features relating to different subject matters, in addition to any combination of features belonging to the same type of subject matter, is also considered to be disclosed by the present application. However, all features can be combined to provide synergistic effects that exceed the simple sum of the features.

[0164] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the dependent claims.

[0165] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may perform the functions of several items recited in the claims. Although specific measures are recited in mutually different dependent claims, this does not indicate that a collection of these measures cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting the scope. Such reference signs may consist of numbers, letters or any alphanumeric combination.

Claims

1. A system (SYS) for image processing, comprising: at least one input interface (IN) configured to receive input data, the input data comprising: i) a spectral projection image of a region of interest, the region of interest comprising a catheter (CN) for passage of a liquid (CA), the spectral projection image being capable of being acquired by a spectral X-ray imaging device (IA1) in an imaging procedure using a contrast agent (CA) present in the liquid, and ii) additional image data capable of being acquired by another imaging device (IA2) of a non-ionizing type, the other imaging device being configured to perform intravascular imaging in parallel with the spectral X-ray imaging, the additional intravascular image data representing 3D information of the catheter; and A predictor component (PC) is configured to predict at least a concentration of a contrast agent in the catheter based on the spectral 2D projection image and the additional intravascular image data when the system is in use.

2. The system according to claim 1, wherein: The 3D information represents the volume of the lumen (L) of the catheter (CN) within a region of interest (ROI).

3. The system according to claim 2, wherein: The concentration is a mass concentration representing the mass of contrast agent within the volume of the lumen (L) within the region of interest (ROI).

4. The system according to any preceding claim further comprises a flow assessment component (FAC) configured to calculate a medical quantity of interest describing one or more flow characteristics of the fluid based at least on a predicted concentration of contrast agent in the conduit.

5. The system according to claim 4, wherein: The medical quantities of interest include any one or more of: volume flow rate, transit time of contrast agent through at least a portion of the conduit, liquid-contrast agent mixing behavior, and transluminal attenuation gradient.

6. A system according to any one of the preceding claims, wherein: The predictor component is configured to calculate an initial concentration of the contrast agent upon entry into the conduit and / or a rate at which the contrast agent enters the conduit.

7. A system according to any one of the preceding claims, wherein: The conduit is part of the vascular system, urinary system or lymphatic system of a mammal, such as a human patient.

8. A system according to any one of the preceding claims, wherein: The another imaging device (IA2) is any one or more of the following: i) an intravascular ultrasound (IVUS) imaging device, or ii) an optical coherence tomography (OCT) imaging device.

9. The system according to any of the preceding claims, further comprising a spectral processor (SP) implementing a material decomposition algorithm to provide the spectral projection image comprising a contrast-only image.

10. The system according to any of the preceding claims, the predictor component (PC) being based on a machine learning model (M).

11. An imaging apparatus (MIA) comprising the system according to any of the preceding claims, further comprising the spectral X-ray imaging apparatus and a second imaging apparatus (IA2).

12. An image processing method, comprising: Receive (S710) input data, the input data comprising: i) a spectral projection image of a region of interest, the region of interest comprising a pipe (CN) for a liquid (CA) to pass through, the spectral projection image being capable of being acquired by a spectral X-ray imaging device (IA1) in an imaging procedure using a contrast agent (CA) present in the liquid, and ii) additional intravascular image data capable of being acquired by another imaging device (IA2) of a non-ionizing type in parallel with the spectral X-ray imaging, the additional intravascular image data representing 3D information of the pipe; and When the system is in use, at least the concentration of the contrast agent in the tube is predicted (S720) based on the spectral projection image and the additional intravascular image data.

13. A method for training the model according to claim 10 based on training data.

14. A computer program element which, when run by at least one processing unit, is adapted to cause the processing unit to perform the method according to claim 12 or 13.

15. At least one computer-readable medium having a program element according to claim 14 stored thereon, or having a machine learning model according to claim 10 stored thereon.

Images