A computer-implemented method, a computer program product, a data processing unit, a computer-readable storage medium, a use of the computer-implemented method, and a computer-implemented method of training a machine-learning model

Abstract

The invention relates to a computer-implemented method for real-time classification of a medical image (3) to analyse whether each of the plurality of medical images (3) was captured while a body duct (10), in particular the aortic root (11), was under the influence of a contrast agent (14), the method comprising the steps: successively receiving a plurality of medical images (3) in real-time, using an input interface, processing the plurality of medical images (3) using a processing unit (4) operatively connected to the input interface, thereby determining a probability that the medical image (3) was captured while under the influence of the contrast agent (14) based on the medical image (3) and a classification model stored on a memory unit, using the processing unit (4) for generating a classification of the medical image (3) whether the medical image was captured under the influence of the contrast agent (14) based on the probability, transmitting the classification to an output interface in real-time.

Description

[0001] A computer-implemented method, a computer program product , a data processing unit , a computer-readable storage medium, a use of the computer-implemented method, and a computer-implemented method of training a machine-learning model

[0002] The invention relates to a computer-implemented method, a data processing unit , a computer-readable storage medium, and a computer-implemented method of training a machine-learning model for real-time classi fication according to the independent claims .

[0003] Medical imaging such as Magnetic Resonance Imaging (MRI ) , Computer Tomography ( CT ) , or fluoroscopy, in particular angiography, in modern healthcare facilities enables accurate diagnosis and monitoring of anatomical features and medical instruments , in particular when carrying out surgical methods such as a transcatheter aortic valve implantation / replacement ( TAVI / TAVR) . The medical instrument may be a surgical instrument .

[0004] Administering a contrast agent , e . g . into the vascular structure of the patient , enhances visibility and delineation of anatomical structures or vasculatures by increasing the contrast between a targeted tissue in contrast to its surroundings , in particular blood, in medical images . However, structures having a hight absorption such as medical / medical instruments or speci fic anatomical features such as calci fications may not be visible in medical images which were acquired under the influence of contrast agent .

[0005] Due to clinical contraindications , such as hypersensitivity, side ef fects , allergic reactions , contrast-induced nephropathy, extravasation inj ury, or reactions to using high-osmolality contrast media, the administration of contrast agents should be minimi zed .

[0006] The discrimination between medical images which merely provide a momentary " snapshot" in real-time acquired under the influence of contrast agent and without contrast agent is often carried out by clinicians relying on their expertise which is subj ect to inter-observer variability . This manual discrimination is hence potentially error-prone , in particular due to disregarding subtle di f ferences in the image , based on a subj ective j udgment of a clinician, labour-intensive , as well as time-consuming . Furthermore , real-time imaging often requires repeated administration of contrast agent to make certain anatomical structures and / or medical instruments visible again at later stage . Especially when the anatomical structure and / or medical instruments are subj ect to physiological motion, e . g . the cardiac or the respiratory cycle , or when manual / motor-drive medical instrument actuation occurs , re-administration of contrast agent may be required .

[0007] EP 3 719 744 Al provides a computer-implemented method for autonomous segmentation of contrast- filled coronary artery vessels by means of a trained convolutional neural network ( CNN) to output a mask denoting the coronary vessels .

[0008] However, the prior art lacks a reliable computer-implemented method, computer program product , a data processing unit , or a computer-readable medium for automated classi fication, in particular real-time classi fication, of a plurality of medical images to determine whether the medical images were acquired under the influence of a contrast agent . Moreover, there is a need in the prior art to computationally integrate such a classi fication and the corresponding data ( e . g . position / orientation of anatomical structures , landmarks , surgical , or medical instruments ) into a medical workf low / system, in particular for reusing the data at a later stage .

[0009] In addition, there is a need for integration of the data e . g . to provide historical data, research data, and, most importantly, interoperability for computer-implemented medical systems via the data .

[0010] The invention shall further provide a cost-ef fective treatment of patients by reducing the required administration ( s ) of contrast agent . The present invention shall provide a robust classi fication of medical images , minimi zing human error while optimi zing diagnostic precision and enhancing a workflow ef ficiency .

[0011] It is therefore the obj ect of the present invention to overcome these and other disadvantages of the prior art and to provide a computer-implemented method, a computer program product , a data processing unit , a computer-readable storage medium, a use of the computer-implemented method, and a computer-implemented method of training a machine-learning model . The invention is used for classi fication, in particular real-time classi fication, of a medical image to predict whether the medical image was captured under the influence of a contrast agent . These obj ects are solved according to the independent claims and further embodiments result from the dependent claims .

[0012] According to a first aspect of the invention, a computer-implemented method for classi fication, in particular real-time classi fication, of a medical image to predict whether the medical image was captured while a body duct , preferably a cardiac component or a vessel component , in particular the aortic root , the aortic, mitral , pulmonary and / or tricuspid valve , was under the influence of a contrast agent is provided .

[0013] The previously described method, the use , and computer-implemented method of training may be carried out for a transapical , a transseptal , a transarterial , a transvenous , a transsubcla- vian / transaxiallary, a transcarotid, a transatrial , a transcaval valve intervention ( s ) , in particular heart valve repair, implantation or replacement .

[0014] The previously described method, the use , and computer-implemented method of training may be carried out for a cardiac intervention, in particular patent foramen ovale ( PFO) closure , atrial septal defect (ASD) closure , ventricular septal defect (VSD) closure , atrial appendage occlusion / closure , in particular left atrial appendage occlusion / closure , balloon valvuloplasty, percutaneous coronary intervention, alcohol septal ablation, coronary artery bypass grafting, endomyocardial biopsy, atherectomy, cardiac catheter ablation intervention ( s ) .

[0015] The previously described method, the use , and computer-implemented method of training may be carried out for non-cardiac interventions , in particular neurovascular or endovascular interventions , such as angiography, in particular cerebral , pulmonary or spinal angiography, angioplasty / stenting, in particular peripheral , renal , or brain tumor angioplasty / stenting, endovascular aneurysm repair, carotid artery or venous sinus stenting, venography, endovascular coiling, stent-assisted coiling, flow diverter stent placement , thrombectomy, in particular mechanical thrombectomy, emboli zation, in particular abnormal artery-vein mal formation, oncological , arteriovenous mal formation, dural arteriovenous fistula, or trans arterial emboli zation, intracranial stenting for stenosis , subdural hematoma evacuation, intervention ( s ) .

[0016] In particular, the medical images or medical training images may have been acquired during any one of the previously mentioned interventions .

[0017] The method comprises successively receiving at least one or a plurality of medical images , in particular in real-time for several medical images , using an input interface . The at least one medical image is processed using a processing unit operatively connected to the input interface . The processing unit is used to determine a probability that the medical image was captured while under the influence of the contrast agent based on the medical image and a classi fication model stored on a memory unit . The processing unit is further used for generating a classi fication of the medical image whether the medical image was captured under the influence of the contrast agent based on the probability . The method includes transmitting the classi fication to an output interface , preferably in real-time .

[0018] Those images which are classi fied as being captured under the influence of the contrast agent are hereinafter referred to as " contrast images" . Those images which are classi fied as being captured non under the influence of the contrast agent are hereinafter referred to as "non-contrast images" .

[0019] The term " acquired under the influence of the contrast agent" refers to exceeding a speci fic threshold of contrast agent concentration in the body duct . The classi fication may include ex- actly two classes or more than two resulting classes , in particular a plurality of classes which all exceed a speci fic threshold of contrast agent concentration to di f ferent degrees .

[0020] The classi fication enables reliable computational locali zing / modelling of an anatomical feature or of a medical and / or a medical instrument . The visibility may depend on whether the medical image or subsection / region of interest within the medical image was / were acquired under the influence of a contrast agent .

[0021] In particular, it is used for binary classi fication . Alternatively, the classi fication may be indicative of a plurality of probability intervals or the probability itsel f .

[0022] The plurality of medical images may be two-dimensional or three- dimensional medical images , in particular fluoroscopic images .

[0023] The computer implemented method may comprise receiving medical images from more than one imaging modality, in particular receiving fluoroscopic images from a first imaging modality, in particular a fluoroscopic imaging modality, and further medical images from at least one second imaging modality . The at least one first or second imaging modality may be selected from at least one of a magnetic resonance imaging modality, a computer tomography imaging modality, an ultrasound imaging modality, positron emission tomography imaging modality, and / or single photon emission computed tomography imaging modality . The method may include classi fication of the medical images of the at least one second imaging modality to determine whether each medical images was captured under the influence of contrast agent or not in the manner previously described . The contrast agent may be di f ferent for the respective first imaging modality and at least one second imaging modality, e . g . gadolinium-based compounds for a magnetic resonance imaging modality and barium compounds for X-ray based imaging modalities .

[0024] The computer-implemented method can be used intraoperatively during fluoroscopy, in particular while fluoroscopic images , preferably angiograms , are acquired in real-time for carrying out surgical methods . The automated classi fication may allow clinicians or a robotic control to make rapid, immediate , and informed decisions and monitor a surgical intervention more effectively . Fluoroscopic images show the soft tissue characteristics , fluid dynamics of the contrast filled in the body duct , e . g . the aorta, in particular the aortic root , and / or dynamic processes / features to be detected and / or visuali zed in realtime .

[0025] Determining the probability of a newly received medical image may be based on the probability of at least one previously received medical image .

[0026] This allows to take the probability / probabilities of at least one temporally prior processed medical image into consideration for determining the probability of a newly received medical image . This may enhance the accuracy and depth in classi fication of a newly received medical image .

[0027] The method may comprise applying a first algorithm, in particular a machine learning algorithm, to take into account the time between the acquisition of the newly received medical image and the previously received medical images to determine the probability of the newly received medical image . This allows to not only consider a plurality of subsequent medical images , but also include a metric indicative of the time between the acquisition of the successive medical images . This may increase the accuracy of the probability and the classi fication . I f the successive medial images are not detected at an equidistant time interval , e . g . to adj ust a radiation dosage during fluoroscopy of a medical procedure changing the frames per second, the method may comprise receiving the individual time points of the medical images instead of the time .

[0028] The first algorithm may be configured to take into account the determined probabilities of a plurality of successive medical images with the time between an acquisition of the successive medical images to determine a threshold-based classi fication value . The method may comprise generating the classi fication of the medical image based on whether the threshold-based classi fication value exceeds a predefined threshold .

[0029] As an alternative to the predefined threshold, a dynamic threshold may be used which may vary depending on the successive medical images , in particular on a contrast , intensity, or quality of the successive medical images or a duration of the time between the acquisition . The dynamic threshold may allow to take the dynamics into consideration, such as considering the temporal change of the probability, when generating the classi fication of the medical images based on the threshold-based classification value .

[0030] The method may comprise applying a second algorithm, in particular a machine learning algorithm, to the non-contrast and / or contrast images , and in particular a time between the acquisition and generate a predicted probability of whether a future image is a contrast or non-contrast image . The method may comprise generating the classi fication of at least one future medical image based on the predicted probability .

[0031] This may allow to increase the accuracy of the classi fication of future medical images since the ef fect of the contrast agent often gradually increases and gradually fades over time . This enhances a reliability of the predicted probability and may prevent potential false positives / false negatives since the visibility of contrast agent in the medical images is temporally highly correlated and normally remains visible / non-visible in a continuous time interval .

[0032] In addition, the method may comprise determining a predicted time interval indicative of until when the contrast agent will be visible based on the predicted probability, in particular based on a plurality of predicted probabilities . The method may comprise generating an algorithm, in particular a function, to predict a plurality of future probabilities with respect to the respective plurality of future time points . The algorithm may be a machine learning algorithm trained on imaging data and / or historic data . Alternatively, the algorithm may be a deterministic algorithm which is fit as a function of time between medical image acquisition using the respective plurality of prior predicted probabilities or prior medical images .

[0033] The method may include transmitting the predicted time interval to a display or a robotic controller, e . g . to indicate how long the contrast agent will remain visible to a clinician or operate the robotic controller accordingly .

[0034] The generated classi fication of the future medical images may be transmitted to the output interface in real-time . The method may comprise determining a spatial orientation and / or a position of at least one anatomical feature preferably the aortic root , in the medical images based on the contrast images , in particular only on the contrast images . Additionally or alternatively, the method may comprise reconstructing a two- or three-dimensional representation of the at least one anatomical feature based only on the contrast images , and in particular based on a three-dimensional model of the aortic root , preferably a pre-operative three-dimensional computer tomography model . The method may comprise visually displaying the non-contrast images including the two- or three-dimensional representation of the at least one anatomical feature in real-time . The anatomical feature may be the body duct .

[0035] An anatomical feature or body duct may be a non-cardiac, neurovascular, non-neurovascular anatomical feature or body duct , in particular a feature associated with any of the above-mentioned cardiac, neurovascular, or endovascular interventions .

[0036] This may e . g . allow for using only the contrast images for determining the spatial orientation and / or the position of the aortic root and in particular the aortic annulus , the left ventricle , the aortic valve , a calci fication, aortic wall , the coronary arteries , and / or the sinuses of Valsalva . These anatomical features are largely indiscernible in medical fluoroscopy imaging without utili zation of the contrast agent . Only taking these medical images thus increases the precision in reconstructing / visuali zing anatomical regions of interest . The reconstructed two- or three-dimensional representation can further reduce the exposure of patient and / or clinicians to radiation and / or further contrast agent inj ections . The two- or three-dimensional model may further be used for further procedural planning such as positioning of an implant, in particular a heart valve prosthesis within the aortic annulus.

[0037] This may further allow for real-time guidance during surgical or interventional procedures by displaying the contrast images and the two-dimensional representation, e.g. for navigating through the aortic root, in particular when the contrast agent effect fades over time.

[0038] The at least one anatomical feature may be selected from at least one of (i) at least the non-coronary cusp, preferably a tip of the non-coronary cusp, and in particular further a tip of the right and / or left coronary cusp, (ii) at least one coronary artery, in particular both coronary arteries, (iii) the aortic root, (iv) a calcification in proximity of the native heart valve, (v) a contour of the left ventricle, (vi) the interventricular septum separating the right and left ventricle, in particular the membranous septum.

[0039] This / these anatomical feature (s) may enhance the procedural efficacy of medical procedures reliant on robustly identifying the anatomical feature (s) of a patient, in particular medical procedures such as heart valve replacement, preferably TAVI / TAVR.

[0040] Alternatively, the anatomical feature may be selected from at least one of one of the following: mitral annulus, aortic annulus, aortic valve, aortic valve leaflets, aortic sinus (sinuses of Valsalva) , aortic root, aortic valve commissures, papillary muscles, chordae tendineae, coronary sinus, bundle of His, Purkinje fibers, coronary ostia, coronary sulcus, crista termi- nalis, trabeculae carneae, Eustachian valve, oblique sinus, transverse sinus, pericardial cavity, left atrial appendage, right atrial appendage, interatrial septum, interventricular septum, left ventricular outflow tract , right ventricular outflow tract , ligamentum arteriosum, commissures (valve ) , annular calci fication, mitral regurgitation zone , mitral clefts , left recurrent laryngeal nerve , papillary f ibroelastomas , fossa ova- lis , septal perforators , left diagonal branches , obtuse marginal artery, ramus intermedius artery, posterior descending artery, right marginal artery, thebesian veins , small cardiac veins , anterior cardiac veins , middle cardiac vein, great cardiac vein, oblique vein of Marshall , the triangle of Koch ( for AV node location) , sinus node artery, conus artery, atrioventricular groove , pulmonary valve cusps , tricuspid valve chordae , sinus of Valsalva, valvular commissures , anterior leaflet of the mitral valve , posterior leaflet of the mitral valve , left ventricular trabeculations , basal septal thickness , atrial septal pouch, papillary muscle tips , fibrous trigone ( cardiac skeleton) , and / or interleaflet triangles of the aortic valve .

[0041] The method may comprise determining a spatial orientation and / or position of at least one medical instrument , in particular a two-or three-dimensional representation of the at least one medical instrument , in the medical images based on non-contrast images . Determining the orientation and / or position may be based on a known predefined shape of the at least one medical instrument . The method may comprise visually displaying the two-dimensional representation of the at least one medical instrument in the contrast images in real-time .

[0042] The term medical instrument in the context of this application may include e . g . a guidewire , a catheter, and an implant , in particular a heart valve prosthesis , preferably an aortic heart valve prosthesis . The at least one medical instrument in the at least one image may comprise or consist of a medical instrument which is arranged in a cardiac region of the patient , in particular in proximity of a native heart valve , preferably in the non-coronary cusp or in the left ventricle . For example , a pigtail catheter may be placed in the non-coronary cusp and / or a coiled guidewire such as a SAFARI guidewire may be placed in the left ventricle . The method may further comprise determining a relative spatial orientation and / or position of the components of the at least one medical instrument with respect to each other .

[0043] This allows to reliably determine the spatial orientation and / or position of the at least one medical instrument , which is normally also rather radiopaque , not under the influence of the contrast agent in a reliable and precise manner in real-time .

[0044] Visually displaying the two-dimensional representation of the at least one medical instrument in the contrast images may allow for a reliable detection of the position / orientation of the at least one medical instrument with respect to the aortic root or components thereof . Otherwise , the administration of the contrast agent , while enhancing the visibility of the aortic root , can obfuscate the position / orientation of the at least one medical instrument because it is imperceptible in the medical images under the influence of the contrast agent . In alternative to the orientation and / or the position of the at least one medical instrument , the orientation and / or the position of certain anatomical features such as a calci fication in the medical images may be determined based on the non-contrast images .

[0045] The method may comprise merging the reconstructed two- or three- dimensional representation of the at least one anatomical feature with the two- or three-dimensional representation of the at least one medical instrument . The merged two- or three-dimensional representation shows a spatial relationship between the at least one anatomical feature and the at least one medical instrument or components thereof .

[0046] The method may comprise visuali zing the merged reconstructed two- or three-dimensional representation of the aortic root and the at least one medical instrument , e . g . on a display .

[0047] This facilitates concurrent optimi zed reconstruction / visualization of the aortic root , in particular at least a component of the aortic root , and the at least one medical instrument , which enables a more precise and expeditious clinical intervention .

[0048] The method may comprise determining a geometric heart valve prosthesis characteristic based on the spatial orientation and / or position of the at least one medical instrument , and in particular at least one user input . The method may include determining a one- , two- or three-dimensional representation of the at least one geometric heart valve prosthesis characteristic . The method may comprise visually displaying the medical images including the one-two- or three-dimensional representation of the at least one heart valve prosthesis characteristic in the medical images , in particular the medial images which were classi fied as captured under the influence of the contrast agent in real-time .

[0049] The geometric heart valve prosthesis characteristic may indicate a longitudinal subsection of the heart valve prosthesis . The method may comprise receiving the at least one user input defining the longitudinal subsection of the heart valve prosthesis . Alternatively, the method may be adapted to autonomously determined the longitudinal subsection of the heart valve prosthesis . The geometric heart valve prosthesis characteristic may comprise or consist of a perpendicular representation with respect to a longitudinal axis of the heart valve prosthesis .

[0050] This enables a clinician to easily identi fy a desired geometric heart valve prosthesis characteristic, in particular the predetermined longitudinal subsection of the heart valve prosthesis . Thereby, a positioning / orienting of the heart valve prosthesis with respect to the anatomy, in particular the non-coronary cusp, may be facilitated . Partially manually or fully automatically determining the geometric heart valve prosthesis characteristic via at least one user input may allow a clinician to compensate for valve shi fts with respect to the anatomy during valve expansion and / or non-uni form valve expansion of the heart valve prosthesis .

[0051] The medical images acquired under the influence of the contrast agent may be used for predicting and / or determining at least one prosthetic or native valvular insuf ficiency . The at least one prosthetic or native valvular insuf ficiency may be selected from at least one of ( i ) paravalvular leakage occurring around edges of the heart valve prosthesis , ( ii ) regurgitation of the native heart valve , ( iii ) valve stenosis , in particular due to tissue overgrowth or due to pannus formation, ( iv) a risk for thrombosis , (v) a risk for a conduction pathway, preferably the His conduction pathway, of the heart being obstructed, in particular based on the spatial orientation and / or position of the at least one anatomical feature including the interventricular septum, preferably further based on the spatial orientation and / or position of the at least one medical instrument . This enables to improve patient outcomes as the prosthetic or native valvular insuf ficiency can be reliably predicted / deter- mined based on medical images , preferably solely on the medical images , prior / in parallel to the surgical procedure .

[0052] The method may comprise applying a first pre-trained machine learning model , in particular a first neural network, to the medical images classi fied as acquired under the influence of the contrast agent . The first pre-trained machine learning model is preferably applied for determining the spatial orientation and / or position of the at least one anatomical feature . The method may comprise applying a second pre-trained machine learning model , in particular a second neural network, to the medical images classi fied as acquired under the influence of the contrast agent , wherein the second pre-trained machine learning model is preferably applied for determining the spatial orientation and / or position of the at least one medical instrument .

[0053] This enables to apply machine learning models optimi zed for the contrast and non-contrast medical images to enhance a predictive capability in contrast to a common machine learning model for determining the spatial orientation / position of the anatomical feature and the medical instrument in both the contrast and noncontrast medical images .

[0054] The first machine learning model may be trained, and preferably validated, using contrast medical images , in particular exclusively using contrast medical images .

[0055] The second machine learning model may be trained, and preferably validated, using non-contrast medical images , in particular exclusively using non-contrast medical images . The machine learning model ( s ) may be a convolutional neural network, in particular a residual neural network comprising a plurality of convolutional layers . The neural network ( s ) may be configured to extract hierarchical feature maps from the medical images , in particular including downsampling of at least one of the medical images and extracting abstract feature maps and up- sampling of the abstract feature maps to an original resolution of the at least one medical image .

[0056] The method may comprise measuring a concentration of the contrast agent within the body duct , in particular the aortic root using an intracorporeally arranged sensor, in particular attached to a medical instrument . The method may comprise using the processing unit for generating the classi fication of the plurality of medical images further taking into account the measured concentration of the contrast agent .

[0057] The concentration may be measured via the sensor based on measuring the volume of contrast agent which is inj ected or measured by detecting a chemical concentration of the contrast agent in the blood .

[0058] Alternatively, an extracorporeal sensor may be adapted for measuring the concentration of contrast agent , e . g . via the contrast in the imaging data, in particular via averaging the contrast in a subsection of the imaging data corresponding to the body duct or a region of interest in the body duct .

[0059] This allows to determine the concentration of contrast agent , e . g . at a target site , in a more precise / reliable manner which may make the classi fication of the medical images more robust . In addition, the method may comprise determining a gradient of the concentration between subsequent medical images which may support the classi fication accuracy further .

[0060] The method may comprise pre-processing the medical images using the processing unit by defining at least one region of interest corresponding to a subregion of the medical images . The at least one region of interest is arranged around a section of the medical images corresponding to a medical instrument and / or corresponding to an anatomical section, in particular the aorta, preferably the aortic root . In addition or alternatively, a plurality of regions of interest may be arranged spaced apart from each other, in particular in a chess-board pattern . The processing unit may be used to determine the probability of each medical image based on the at least one region of interest of the medical image or by taking into account a plurality of regions of interest of the medical image .

[0061] The probability may be exclusively determined based on the region of interest or the plurality of regions of interest .

[0062] Administration of contrast agent is typically mainly visible in the corresponding body ducts / vasculature of the patient . Therefore , at least one region of interest which is associated with the anatomy of the patient may enhance the classi fication Using at least one region of interest or even di f ferent regions of interests during di f ferent stages of the medical intervention may further allow for a more intricate visuali zation / reconstruction at the region / regions of interest , e . g . corresponding to a location of the aortic root or components thereof . The region of interest may be associated with the body duct , preferably the aorta or even a target region of the aortic root . The region of interest may be moved in a synchroni zed manner corresponding to a traj ectory of a medical instrument within the body duct . This may allow for providing more details and precision at the current position / orientation of the medical instrument , in particular of a distal tip of the medical instrument , within the body duct , e . g . during progressive advancement of the medical instrument from the femoral artery across the aortic arch into the aortic annulus .

[0063] Using the plurality of regions of interest of the medical image or di f ferent medical images may further yield better classi fication results by enhancing the precision of determining the temporal contrast agent spatial spread / dispersion within the body duct . The method may further comprise determining the probability and classi fication of individual regions of interest of the medical image separately . This may enhance the determination of the position / the orientation or two-or three-dimensional representation of the anatomical feature / medical instrument corresponding the individual regions of interest .

[0064] The method may comprise the steps of receiving at least one first medical image which includes a first anatomical feature or a medical instrument and receiving a second medical image which includes a di f ferent second anatomical feature or medical instrument . The first and second medical images may be acquired simultaneously or sequentially . The method may comprise processing the first and second medical image using the processing unit , thereby determining the probabilities that the first and the second medical image was acquired under the influence of the contrast agent , respectively, based on the second and first medical image and the classi fication model . The method may comprise using the processing unit for generating a first and second classi fication of the first and second medial image whether the medical images were captured under the influence of the contrast agent based on the probabilities . The method may comprise transmitting the first and second classi fication to the output interface , preferably in real-time .

[0065] The medical instruments may relate to medical , in particular surgical instruments , which were introduced into the body duct , in particular into a cardiac component .

[0066] This may enable to classi fy that the first and a second anatomical f eature / medical instrument which are of interest are suf ficiently visible the first and second images without the processing unit having to preprocess the first and second medical images to generate regions of interest for the first and second anatomical f eatures / medical instruments .

[0067] Alternatively, the method may comprise pre-processing the first and second medical images using the processing unit by defining at least a first region of interest of the first image which includes a first anatomical f eature / medical instrument and a second region of interest of the second image which includes a second anatomical f eature / medical instrument . Alternatively, similarly as described above , only one medical image may be pre-processed to define the first and second region of interest including the first / second anatomical f eature / medical instrument , respectively .

[0068] The method steps for these first and second regions of interest may be executed in a manner consistent with the method detailed for the first and second images described above . During medical procedures di f ferent anatomical f eatures / medical instruments are preferably visible simultaneously or visible sequentially to mitigate potential surgical complications . For example , a first anatomical feature such as an aortic annulus and a second anatomical feature such as a left ventricular outflow tract should preferably be visible simultaneously for performing TAVI / TAVR . This may ensure that both are visible , in particular simultaneously, based on the first and the second classi fication .

[0069] Another aspect of the invention relates to a computer-implemented method, in particular a previously described computer-implemented method . The method comprises successively receiving a plurality of medical images in real-time using an input interface . A processing unit connected to the input interface is used for analysing an initial image set comprising at least one of the medical images taken at an initial time point , thereby generating an initial result . The method comprises predicting a subsequent result for at least one subsequent image set taken at a subsequent time point based on the initial result , thereby generating a predicted subsequent result . The method comprises analysing a subsequent image set comprising at least one medical image received at the subsequent time point , thereby generating an analysed subsequent result . The method comprises generating a refined subsequent result based on the predicted subsequent result and the analysed subsequent result .

[0070] The results may comprise or consist of an orientation and / or a position of an anatomical feature , such as the aortic root , a medical instrument , or components thereof . Alternatively, the results may comprise or consist of a distance or relative orientation between the anatomical feature , the medical instrument , or components thereof . The medical instrument can consist of or comprise a heart valve prosthesis, a guidewire, or a catheter. The anatomical feature may e.g. comprise or consist of the aortic annulus, the aortic arc, a leaflet, or a calcification. It is also possible to consider a plurality of distances or relative orientations between the components of the medical instrument, of the anatomical feature, or the medical instrument and the anatomical feature.

[0071] This method allows to determine a more refined subsequent result by merging the predicted subsequent result and the analysed subsequent result.

[0072] When combining the predicted subsequent result and the analysed subsequent result, a weighting algorithm, in particular a machine learning weighting algorithm, can be used. The weighting algorithm is configured to weight the respective impact of the predicted subsequent result and the analysed subsequent result on the refined subsequent result.

[0073] The method may comprise adjusting the algorithm's weighting according to an initial image set's quality, reflecting the accuracy / certainty of the predicted subsequent result, and / or a subsequent image's quality / visibility of features, reflecting the accuracy / certainty of the analysed subsequent result. The analysed subsequent result may for example indicate no accurate position / orientation of an anatomical feature / medical instrument, e.g. if the subsequent image is a contrast / non-con- trast image.

[0074] A classification of a medical image as previously described (i.e. whether the medical images were acquired while a body duct was under the influence of a contrast agent) may be further used. The method may comprise adjusting the weighting based on the classi fication of the medical image ( s ) of the initial image set and / or the subsequent image set . In particular, when the classi fication of the medical image ( s ) of the initial image set and the subsequent image set are not the same , the weighting of the predicted subsequent result and the analysed subsequent result may be adj usted . The weighting of the predicted subsequent result and the analysed subsequent result may be adj usted based on the respective level of confidence . The respective level of confidence may be influenced by the absence or presence of contrast agent in the target region . This may further enhance the generated refined subsequent result by taking into account the visibility in contrast and non-contrast images . Alternatively, the method may include receiving the previously described probability of the medical image instead of the generated classi fication to further enhance the weighting adj ustment .

[0075] The initial result may be determined based on at least two medical images obtained at separate initial time points , preferably the initial time point , and a time point temporally prior to the initial time point .

[0076] This enables a more exact predicted subsequent result by including more medical images of the initial image set . In addition, this allows to take into account temporal dynamic movement .

[0077] The predicted subsequent result may be based on a velocity, and in particular an acceleration, of the medical instrument . The method may comprise receiving location data indicative of the position and / or the orientation of the medical instrument of the medical images by an actuator which controls the position and / or the orientation of the medical instrument . The method may com- prise determining the velocity, and in particular the acceleration, of the medical instrument based on the location data, in particular by using the processing unit .

[0078] Alternatively or additionally, the predicted subsequent result may be based on a velocity, and in particular an acceleration, of an anatomical feature , in particular based on a periodic recurring movement such as a cardiac or respiratory cycle . The method may comprise detecting the recurring movement and determining the velocity, and in particular the acceleration, based on the recurring movement .

[0079] The method steps of ( I ) generating the predicted subsequent result and / or ( I T ) generating the refined subsequent result may be based on a mathematical prediction algorithm . The mathematical prediction algorithm may be selected from at least one of ( i ) a Kalman filter, ( 11 ) a particle filter, ( ill ) a recurrent neural network, ( iv) a weighted sum, and (v) an optical flow algorithm . Generating the predicted subsequent result ( I ) and generating the refined subsequent result ( I T ) may be based on di f ferent mathematical prediction algorithms .

[0080] The Kalman filter is particularly suitable for compensating for Gaussian noise and predicting linear transitions to provide a real-time prediction . The weighted sum may ef fectively include a reliability or importance of speci fic medical images .

[0081] Another aspect of the invention relates to a computer program product comprising instructions which, when the program is executed on a computer, cause the computer to carry out the steps of the previously described method for classi fication of the plurality of medical images to analyse whether at least one of the plurality of medical images was captured while a body duct was under the influence of the contrast agent . Another aspect of the invention relates to a system, in particular comprising or consisting of a data processing unit , comprising means for carrying out the steps of the previously described method for classi fication .

[0082] Another aspect of the invention relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the previously described computer-implemented method .

[0083] Another aspect of the invention relates to a use of the previously described computer-implemented method for classi fication, in particular based on the classi fication of each of the plurality of medical images . The method may be used for ( I ) planning a medical procedure , in particular a transcatheter aortic, mitral , tricuspid, and / or a pulmonary valve or valve component implantation or replacement . The method may be used for ( I T ) displaying the medical images on a display in real-time including the two- dimensional representation of the aortic root and / or the two-dimensional representation of the medical instrument . The method may be used for ( I T T ) displaying a merged two- or three-dimensional representation of the aortic root with the two- or three- dimensional representation of the medical instrument . The method may be used for ( TV) adj usting a dosage of a fluoroscopy imaging device for the acquisition of the medical images .

[0084] The method may be used for controlling a robotic system for ( i ) translationally and / or rotationally moving at least one medical instrument , in particular a pigtail catheter, a heart valve prosthesis , and / or a heart valve prosthesis delivery system, and preferably be used when crossing the native aortic annulus . The method may be used for controlling a robotic system for ( ii ) positioning and / or deploying a heart valve prosthesis , in particular within the aortic annulus .

[0085] The method may be used for ( iii ) optimi zing an amount of contrast agent to be administered and / or a time interval of administering the contrast agent into the body duct .

[0086] At least one of the previously described uses of the method may be based on the two- or three-dimensional representation of the aortic root and / or the two- or three-dimensional representation of the medical instrument .

[0087] The system comprising means for carrying out the method steps may further be configured for carrying out the previously described uses ( I ) - ( TV) and / or surgical uses ( i ) - ( iii ) of the method .

[0088] In addition to the previously described advantages , these uses , and the system allow to reduce the clinician ' s workload, freeing them to focus on the treatment of the patient and increases the ef ficacy and safety of medical procedures , in particular surgical procedures , preferably TAVI / TAVR .

[0089] Another aspect of the invention relates to a computer-implemented method of training a machine-learning model for classi fication of a plurality of medical images to predict whether the medical image was captured under the influence of a contrast agent , in particular according to the previously described computer-implemented method . The method of training comprises receiving an input training dataset comprising medical training images , and contrast agent classi fication labels which indicate whether the respective medical training image was acquired under the influence of a contrast agent . The input training dataset may comprise a patient classi fication label which indicates di fferent patients of the medical training images . The method of training may comprise pre-processing the medical training images by normali zing a pixel intensity of the received medical training images , preferably based on the respective patient classi fication label . The method of training may comprise defining at least one region of interest , in particular a plurality of regions of interest in the medical training images , in particular around a section of the medical images corresponding to a medical instrument and / or corresponding to a body duct , in particular the aorta, preferably the aortic root . The method of training may comprise partitioning the medical training images , labels , and in particular regions of interest or the plurality of regions of interests into a training subset and a validation subset . The method of training may comprise training a machine learning model on the training subset based on a loss function . The method of training may comprise validating the trained machine learning model on the validation subset . The method of training may comprise tuning the hyperparameters of the machine learning model based on a performance metric . The method of training may comprise outputting the trained, validated, and tuned machine learning model .

[0090] The input dataset may comprise time point labels indicative at which time point the medical training image was acquired . This may provide additional information on the temporal relation between the medical training images which influence the dynamic of anatomical features or medical instruments . This further allows to use a wider array of medical training images acquired using di f ferent frame rates , e . g . between 1 - 60 frames per second, in particular 15 to 30 frames per second . Further embodiments of the invention and improvements of the described embodiments will become apparent with the following description of embodiments .

[0091] The invention is now described with reference to certain embodiments and figures which show :

[0092] Figure 1 : a schematic representation of a plurality of unlabeled medical images which were processed using a computer-implemented method and labeled with a predicted binary classi fication indicative of whether the medical images were captured while a body duct was under the influence of a contrast agent ,

[0093] Figure 2 : a schematic representation of determining a probability for successively received medical images based on the computer-implemented method and a histogram of the respective probabilities over the number of the medical images / an elapsed time ,

[0094] Figure 3A: a schematic three-dimensional representation of the aortic root and a schematic three-dimensional representation of a medical instrument which were merged to a merged three- dimensional representation,

[0095] Figure 3B : a schematic two-dimensional representation of the aortic root merged with a two-dimensional representation of a medical instrument which were included in a medical two- dimensional fluoroscopy image ,

[0096] Figure 3C : a two-dimensional fluoroscopy image with two regions of interest ,

[0097] Figures 3D - 3E : a schematic two-dimensional representation of the heart valve prosthesis and the aortic annulus having a marker which is indicative of a circumferential orientation of the heart valve prosthesis and a marker which is indicative of a circumferential orientation of the aortic annulus , Figure 3F : a schematic three-dimensional representation of the heart valve prosthesis and the aortic root which indicates a tilt of the heart valve prosthesis with respect to the aortic annulus ,

[0098] Figures : 4A to 4D : a first medical image on the left which was acquired under the influence of a contrast agent and a second medical image on the right which was acquired not under the influence of the contrast agent showing a medical instrument and a deployed heart valve prosthesis respectively, Figure 5 : a schematic representation of generating a predicted subsequent result based on an initial result for two medical images , determining an analyzed subsequent result based on a subsequent medical image , and combining the predicted subsequent result and the analyzed subsequent result to generate a refined subsequent result ,

[0099] Figures 6A - 6B : displayed medical images including a geometric heart valve prosthesis characteristic and a first position of a first anatomical feature and a second position and orientation of a second anatomical feature .

[0100] Figure 1 shows a schematic representation of a plurality of nonlabeled medical images 3 ( top of Fig . 1 ) which were processed using a computer-implemented method and labeled with a predicted binary classi fication indicative of whether the respective medical images 3 were captured under the influence of a contrast agent 14 (bottom of Fig . 1 ) . The medical images 3 are fluoroscopy images which were acquired during a TAVI / TAVR procedure . However, the medical images 3 may alternatively be pre-operative or post-operative medical images 3 . The label is schematically illustrated by the terms " contrast" and "non-contrast " on the processed medical images 3 in the bottom of Fig . 1 . The contrast agent 14 can be identi fied in the medical images 3 by increased contrast in the aortic root 11 in the medical images 3 which were acquired under the influence of the contrast agent 14 and may be used for the classi fication .

[0101] The top rows of medical images 3 of figure 1 show a medical instrument of a TAVI / TAVR system which comprises a delivery device 19 , a pigtail catheter 16 , and a heart valve prosthesis 17 which was advanced by femoral catheteri zation to the aortic root 11 . The contrast agent 14 is preferably administered directly from a pigtail catheter 16 which was advanced through the aortic root 11 . The delivery device 19 comprises a sensor 8 for measuring the concentration of the contrast agent directly in the aortic root 11 such that determining the probability may be enhanced by the measured concentration of the contrast agent .

[0102] The bottom rows of figure 1 show a medical instrument of the TAVI / TAVR system which comprises a heart valve prosthesis 17 which was deployed in the aortic annulus 12 .

[0103] The medical images 3 on the left were acquired under the influence of the contrast agent 14 within the aortic root 11 . They enable enhanced visuali zation of the aortic root 11 and in particular the aortic annulus 12 . Anatomic landmarks can thus be visuali zed in the medical images 3 using the contrast agent 14 .

[0104] The medical images 3 on the right were acquired not under the influence of the contrast agent 14 . They enable enhanced visuali zation of the medical instruments such as the pigtail catheter 16 , the heart valve prosthesis 17 , and the delivery device 19 which are more radiopaque . Some anatomical landmarks such as calci fications are also more clearly visible in the medical images not acquired under the influence of the contrast agent 14 (not shown in Fig . 1 ) . The classi fication of " contrast" and "non-contrast" based on the computer-implemented method shown in Fig . 1 can, therefore , provide information on which anatomic landmarks , in particular the aortic root and components thereof or the medical instrument or components thereof , are visible or more clearly visible . This allows to make a decision on which medical images 3 may be used for further processing . Further processing may e . g . relate to a registration of the anatomic landmark ( s ) to a pre-operative computer tomography, creating a two- or three-dimensional representation of the anatomic landmark ( s ) and / or medical instrument ( s ) , visually displaying the anatomic landmark ( s ) and / or the medical instrument ( s ) , and / or robotic control . A concentration measurement of contrast agent 14 by the sensor 8 may be used to render the classi fication of the medical images 3 more accurate .

[0105] Figure 2 shows a schematic representation of determining a probability pt-i , Pt , Pt+i , - for each successively received medical image 3 based on the computer-implemented method and a histogram of the respective probabilities over the number of the medical images 3 / an elapsed time . The probabilities pt-i , Pt , Pt+i , - of each medical image 3 are indicative of whether the medical image 3 was acquired under the influence of a contrast agent . The probabilities are determined independently from each other based on a machine learning classi fication model on a memory unit . The machine learning algorithm was accordingly trained with contrast images and non-contrast images which were labelled accordingly, such that the machine learning algorithm is e . g . susceptible to subtle contrast changes of the images indicating a presence or absence of the contrast agent .

[0106] The histogram is an exemplary algorithm for linking the successive medical images 3 to the time of acquisition of the medical images 3 to calculate a threshold-based classi fication value . Since the increase in the concentration of the contrast agent in the body duct, e.g. aortic root, may be described by a quasisteady, often monotonically increasing function, a temporal analysis, such as using the histogram data, can significantly improve the classification. A time point when a specific threshold of contrast agent concentration or contrast enhancement of the medical images 3 is exceeded can be determined. This may define an outcome of the classification, in particular a binary classification.

[0107] Based on the histogram or another time-based algorithm, a timebased probability of the medical images 3 is determined in realtime. The classification whether the medical image 3 was acquired under the influence of the contrast agent can thus be determined based on the time-based probability of the medical images 3.

[0108] The lower dashed ellipse of Fig. 2 shows a low probability increasing monotonically over time. These medical images 3 in Fig. 2 may exemplary be classified as not acquired under the influence of the contrast agent, since the contrast agent concentration / contrast enhancement in the medial images 3 may not yet be sufficient to identify the anatomical features.

[0109] The upper dashed ellipse of Fig. 2 shows a peak of the probabilities of the medical images 3 which are classified as acquired under the influence of the contrast agent. Outliers, images subject of X-ray dosage adjustment, or images acquired while / after adjusting a position / an orientation of imaging device for the acquisition of the medical images 3, e.g. a C-arm, may still be classified correctly, by considering the time-based probability. Figure 3A shows a schematic three-dimensional representation 104 of the aortic root 11 and a schematic three-dimensional representation 102 of a medical instrument which were merged to a merged three-dimensional representation 106 . The medical instrument in Fig . 3A is formed by a heart valve prosthesis 17 , a pigtail catheter 16 , a balloon catheter 18 , and / or a delivery device 19 . The medical images for reconstructing the merged three- dimensional representation 106 were acquired by a fluoroscopy imaging device 9 formed by a C-arm which is connected to a processing unit 4 for carrying out the computer-implemented method .

[0110] The three-dimensional representation 104 of the aortic root 11 ( only shown in two-dimensions in the image for better visibility) and the aorta 10 was carried out by the processing unit 4 based on the contrast images . The three-dimensional representation 104 of the aortic root 11 may further be refined by carrying out a registration of the medical images and a pre-operative three-dimensional computer tomography model of the aortic root 11 . This enhances image quality by providing three-dimensional clinical data of the patient instead of relying merely on two- dimensional fluoroscopy data for the three-dimensional representation 104 .

[0111] The three-dimensional representation 102 of the medical instrument 17 , 18 , 19 was generated by the processing unit 4 based on the non-contrast images . The three-dimensional representation 102 of the medical instrument may further be refined by registration of the medical images and a virtual model of the medical instrument , including known dimensions of the medical instrument . The clinician or a robotic controller may for example input the information about the medical instrument for the regis- tration which is used into the processing unit such that the dimensions of the medical instrument are known based on the virtual model corresponding to the selected instrument .

[0112] The medical instrument is radiopaque or may be formed by an inflatable balloon catheter 18 which can be inflated by a radiopaque inflation medium such that the medical instrument is clearly visible without any contrast agent .

[0113] Merging the three-dimensional representations 104 , 102 of the aortic root 11 and the medical instrument via the processing unit 4 enables determining a spatial relationship, e . g . the relative position and / or orientation, of the aortic root 11 and the medical instrument or elements thereof . This enhances an ef ficiency to perform surgical procedures and leads to superior clinical outcomes . Moreover, reducing the cognitive load on the clinician leads to better average clinical performance . In addition, this allows to mitigate the respective adverse ef fects on visibility of anatomical feature and medical instruments in medical images acquired under the influence and not under the influence of contrast agent by including the relevant information in the merged model simultaneously .

[0114] Figure 3A further shows that the merged three-dimensional representation 106 may be visually displayed on a display 6 connected to the processing unit 4 such that a clinician can easily assess the ideal placement and or positioning of the medical instrument with respect to an aortic annulus 12 . The merged three-dimensional representation 106 may further be used for fully automated placement of the heart valve prosthesis 17 , e . g . via a robotic controller . Additionally or alternatively, the merged three-dimensional representation 106 may be configured for sup- porting the placement of the heart valve prosthesis 17 by a clinician by providing visual , tactile , or auditive indicators , e . g . a colour coding scheme , displaying an arrow, a vibration scheme , or a sound scheme , which are indicative of the correct placement of the heart valve prosthesis 17 or a delivery device thereof ( see Figs . 3C - 3F) .

[0115] Figure 3B shows a schematic two-dimensional representation 103 of the aortic root 11 merged with a two-dimensional representation 105 of a medical instrument which were included in a medical two-dimensional fluoroscopy image 3 . The two-dimensional representation 103 of the aortic root 11 is shown as a dashed line in a two-cusp view . The two-dimensional representation 103 of the aortic root 11 was determined based on prior medical images 3 which were classi fied as detected under the influence of a contrast agent .

[0116] Figure 3B shows a processing unit 4 which is connected to an imaging device 9 , e . g . a C-arm for an acquisition of the medical fluoroscopy images 3 . The processing unit 4 is further configured for merging the two-dimensional representations of the aortic root 11 and the medical instrument and including the merged representation in the medical image 3 in real-time . The two-dimensional fluoroscopy image 3 was acquired not under the influence of a contrast agent and may be displayed together with a merged two-dimensional representation 107 of the aortic root 11 and the medical instrument on a display 6 ( see Fig . 3A) in realtime .

[0117] The medical instrument in Fig . 3B includes a heart valve prosthesis 17 which is included in the two-dimensional representation 105 of the medical instrument as a dashed line indicative of a position and / or orientation of the heart valve prosthesis . The two-dimensional representation of the aortic root 11 includes a dashed line which is indicative of an ideal target position and / or orientation for placement of the heart valve prosthesis 17 in the aortic annulus 12 which was retrieved based on the medical images 3 acquired under the influence of the contrast agent . This allows a clinician or a fully autonomous robotic system to control the placement / deployment of the heart valve prosthesis 17 at the target position / orientation . This positioning / deployment may be supported by providing visual or auditive indicators as previously described, in particular di fferent visual indicators than dashed lines . In addition, the two-dimensional representation may include speci fic landmarks used by clinicians such as the " 3 mm line" used as a reference of the depth below the aortic annulus for more reliable placement .

[0118] Figure 3C shows a two-dimensional fluoroscopy image 3 with two regions of interest 5 , 51 highlighted by a dashed rectangle respectively . The first region of interest 5 is arranged around a distal tip of a medical instrument formed by a heart valve prosthesis 17 at a position corresponding to a position / an orientation of the aortic root in the medical image 3 . The second region of interest 51 is arranged proximally to the first region of interest 5 and corresponds to a position / an orientation of the aortic arch . The region of interest 5 , 51 may be chosen such that the classi fication by the computer-implemented method is particularly reliable in the region of the distal tip of the medical instrument . This allows a more exact classi fication whether the medical images 3 / regions of interest 5 , 51 were acquired under the influence of a contrast agent in the proximity of the distal tip or a target position / a target orientation . Hence , navigation during TAVI / TAVR, e . g . through the aortic arch, may be facilitated and a more precise position for placement / deployment of the heart valve prosthesis 17 , e . g . within the aortic annulus , can be achieved .

[0119] An intensity value for each region of interest 5 , 51 may be determined and used for determining a probability that a respective medical image was acquired under the influence of the contrast agent . Alternatively, the classi fication method may only be used on the region of interest 5 , 51 to determine the probability and, thereby, optimi ze a resulting classi fication result for the speci fic region of interest 5 , 51 . The method may also be based on a plurality of regions of interest 5 , 51 which are arranged spaced apart from each other as shown in Fig . 3C or regions of interest which are arranged in a chess-board pattern, such that a successive change in contrast / intensity values can be monitored over an elapsed time to determine the probability of each medical image 3 . Using at least one region of interest 5 , 51 may enable the computer-implemented method to disregard regions of the medical images 3 which are not influenced by the contrast agent or are not relevant for a medical / surgical procedure / a step of the medical / surgical procedure , in particular when performing TAVI / TAVR . The region of interest or regions of interest 5 , 51 may be used in the previously described method steps of Figs . 1 - 3B .

[0120] Figures 3D and 3E show a schematic two-dimensional representation of a heart valve prosthesis and the aortic annulus having a marking 171 which is indicative of a circumferential orientation of the heart valve prosthesis 17 and a marking 121 which is indicative of a circumferential orientation of the aortic annulus . The circumferential orientation of the heart valve prosthesis may be determined by a non-rotationally symmetric radiopaque design of the medical instrument , in particular via radiopaque markings or a design of the heart valve prosthesis .

[0121] The circumferential orientation of the aortic annulus may be determined based on the cusps of the native aortic valve annulus .

[0122] The markings 171 indicative of the circumferential orientation of the heart valve prosthesis 17 were determined based on the non-contrast images with the previously described method . The markings 121 indicative of a circumferential orientation of the aortic annulus were determined based on the contrast images with the previously described method .

[0123] The orientation of these markings 171 , 121 with respect to each other may allow for simpli fied circumferential alignment of the heart valve prosthesis with respect to the aortic annulus , in particular in respect to the cusps of the aortic valve , by a clinician or a robotic controller . The method may include the step of calculating these markings 171 , 121 via a processing unit as previously described and displaying these visual indicators in the form of markings 171 , 121 on a display .

[0124] As shown in Fig . 3D, the markers 171 , 121 are not aligned while Fig . 3E shows that the markings 171 , 121 are essentially perfectly aligned which indicates a target orientation / a target position of the heart valve prosthesis . These visual indicators may facilitate a deployment of the heart valve prosthesis in the aortic annulus . Figure 3F shows a schematic three-dimensional representation of the heart valve prosthesis 17 and the aortic root 12 which indicates a tilt of the heart valve prosthesis 17 with respect to the aortic annulus 12 . In addition, to the previously described markings 171 , 121 of the heart valve prosthesis 17 which indicate the circumferential orientation of the heart valve prosthesis 17 with respect to the aortic annulus 12 , the three-dimensional representation includes a colour coding scheme . The colour coding scheme indicates a tilt T of the heart valve prosthesis 17 in a current medical image with respect to a target position in the aortic annulus 12 . The colour coding scheme or arrows can further indicate a longitudinal position of the heart valve prosthesis 17 with respect to the aortic annulus 12 . This allows a reliable placement of the aortic valve prosthesis within the aortic annulus and renders TAVI / TAVR procedures less demanding for clinicians without necessitating an intensive learning process . Proper placement may further improve hemodynamic, prevent paravalvular leakage , valve migration, or embolization, maximi ze a li fespan of the implant , and optimi ze the overall patient outcomes .

[0125] Figures 4A to 4D show a first medical image 3 which was acquired under the influence of a contrast agent and a second medical image 3 which was acquired not under the influence of the contrast agent indicated by labels " contrast" and "non-contrast " showing a medical instrument and a deployed heart valve prosthesis respectively . The medical instrument in Figs . 4A and 4B includes a collapsed heart valve prosthesis 17 , a delivery device 19 , and a pigtail catheter 16 which was advanced into the ventricle of a heart for facilitating deployment of the heart valve prosthesis 17 . The deployed heart valve prosthesis 17 in Figs . 4C and 4D is deployed in the aortic annulus 12 after success fully performing a TAVI / TAVR procedure while the rest of the medical instrument was removed .

[0126] In Figs . 4A and 4C the aortic annulus 12 is visible when the body duct in the medical image 3 was under the influence of a contrast agent ( see dashed arrow) . In figures 4B and 4D the aortic annulus 12 is not visible anymore since the body duct in the medial image 3 was not under the influence of a contrast agent ( see dashed arrow) .

[0127] However, the radiopaque components of the medical instrument 16 , 17 , 19 , in particular the deployed heart valve 17 , are not visible in Figs . 4A or 4C but instead visible in Figs . 4B and 4D .

[0128] The medical instrument 16 , 17 , 19 and / or the anatomical feature , such as the aortic annulus 12 , undergo perpetual motion, due to manipulation by a clinician or a robot of the medical instrument , or due to the cardiac cycle , due to the respiratory cycle , or due to mutual interactions between the medical instrument 16 , 17 , 19 and the anatomical feature ( s ) . A position and / or an orientation of the medical instrument 16 , 17 , 19 and / or the anatomical feature may hence constantly change . Consequently, it would be advantageous to track the dynamic behaviour of the medical instrument 16 , 17 , 19 and / or the anatomical feature , in particular between medical image sets which were at least partially acquired under the influence of the contrast agent and not under the influence of the contrast agent .

[0129] Figure 5 shows a schematic representation of generating a predicted subsequent result Pred+1at a time t + 1 based on an initial result IR which was determined based on an initial image set comprising two medical images 3 denoted by ft-i> ft acquired at times t - 1 , t . Figure 5 further shows determining an analysed subsequent result SR = Det^+1based on a subsequent medical image 31 denoted by ft+, and combining the predicted subsequent result Pred^+1and the analysed subsequent result Det^+1to generate a refined subsequent result p+1= . In this manner, the refined subsequent result may be continuously determined for successive medical images based on the following relationship ptl+i= aDet+1+ pPredtl+1. The results refer to the position and / or the orientation of an anatomical feature and / or of a medical instrument or components thereof . The anatomical feature may e . g . be an aortic annulus , a left ventricle , an aortic valve , a calci fication, an aortic root , an aortic arc, a coronary ostia, or a mitral valve .

[0130] In Fig . 5 , generating a predicted subsequent result Pred^+1is carried out by using a recurrent neural network, and combining the predicted subsequent result and the analysed subsequent result Detf+1is carried out using a weighted sum which is supported by a machine learning algorithm . The parameters a and fl represent weighting parameters which are adj usted based on a machine learning model to account for the visibility of the medical instrument or anatomical features such as the aortic root and in particular the aortic annulus in the medical images 3 , 31 .

[0131] The predicted subsequent result Pred^+1is preferably based on multiple medical images 3 , in particular at least two medical images 3 , as shown in Fig . 5 which allows to take the dynamic behaviour such as a velocity and / or acceleration of the tracked feature into consideration . This allows to generate a more reliable predicted subsequent result of the future position and / or orientation of the tracked feature . The weighting of the parameters a and ft may further be refined based on a received classi fication, in particular binary classification, or probability of the medical images 3 , 31 as previously described . This classi fication or probability is indicative of the visibility of the tracked feature , e . g . medical instrument and / or anatomical feature , in the medical images 3 , 31 due to the contrast agent . The classi fication or probability may be indicative as to whether the predicted subsequent result Pred+1or the analysed subsequent result Det^+1is more reliable for determining the refined subsequent result which indicates the position and / or the orientation of the medical instrument or the anatomical feature . This is particular advantageous i f the position and / or orientation of the medical instrument or the anatomical feature cannot be determined in the analysed subsequent result Detf+1.

[0132] The refined subsequent result p+1may be displayed or transmitted to allow a clinician or robotic controller to receive a more reliable position and / or an orientation of the medical instrument or the anatomical feature . The position and / or the orientation may be determined and displayed / transmitted in real-time . In a preferred embodiment , the position and / or the orientation is / are directly included in the fluoroscopy medical images or in a two-or three-dimensional representation of the medical instrument and / or the anatomical feature when being visually displayed .

[0133] Figures 6A and 6B show medical images 3 including a geometric heart valve prosthesis characteristic 21 and a first position of a first anatomical feature 22 , and a second position and orientation of a second anatomical feature 23 . For illustrative purposes , both medical images 3 were acquired under the influence of contrast agent , such that the patient ' s anatomy is discernible . These medical images 3 are fluoroscopy images which are acquired during a TAVI / TAVR procedure .

[0134] The medical images 3 of figures 6A and 6B further show a catheter 16 having a bent distal tip, e . g . a pigtail catheter, being placed in proximity of the non-coronary cusp and a pre-shaped guidewire 24 , e . g . a SAFARI guidewire placed in the left ventricle of the heart . These medical images 3 including the geometric heart valve prosthesis characteristic 21 and the anatomical features 22 , 23 are displayed to a clinician on a display to facilitate placement of a heart valve prosthesis 17 in the native anatomy .

[0135] The geometric heart valve prosthesis characteristic 21 is selected based on an identi fied position / orientation of the heart valve prosthesis 17 in the medical image 3 and a user input . The method may be configured for determining the total length of the heart valve prosthesis 17 . The user input may speci fy a longitudinal subsection of the heart valve prosthesis 17 , e . g . 20 percent of a total length of the heart valve prosthesis . The geometric heart valve prosthesis characteristic may therefore denote a longitudinal subsection originating from the bottom of the heart valve prosthesis 17 of 20 percent of the total length for example . A center of the heart valve prosthesis 17 may be easily identi fiable by the system / during the method due to a radiopaque marker of the heart valve prosthesis 17 . The radiopaque marker may also denote a speci fic section of the heart valve prosthesis 17 or an orientation of the heart valve prosthesis 17 . The geometric heart valve prosthesis characteristic 21 is displayed in Figs . 6A and 6B as a line which extends perpendicularly with respect to a longitudinal axis of the heart valve prosthesis 17 . The first position of the first anatomic characteristic 22 denotes a tip of the non-coronary cusp which is displayed in the medical images 3 as a black dot . The figures 6A and 6B further show that the position of the first anatomic characteristic 22 is connected via a dashed line with the geometric heart valve prosthesis characteristic 21 which facilitates alignment of the heart valve prosthesis 17 . The second position and orientation of the second anatomical feature 23 is displayed as a circle denoting positions and orientation of the coronary arteries 231 , 232 . Thereby, the orientation of the coronary arteries 231 , 232 can be inferred by the clinician / sof tware for placement of the heart valve prosthesis 17 and / or placement of the pigtail catheter 16 in proximity to the non-coronary cusp .

[0136] The medical images 3 may be processed to include further posi- tions / orientations of anatomical features , in particular a posi- tion / orientation of the interventricular septum (not shown in Figs . 6A and 6B ) such that a prosthetic valvular insuf ficiency in from of a risk for the His conduction pathway of the heart being obstructed can be determined . The risk for the His conduction pathway being obstructed can be determined based on the po- sition / orientation of the heart valve prosthesis 17 in the medical images 3 and the position / orientation of the interventricular septum .

[0137] Moreover, the medical images 3 may be processed to further include an anatomical feature denoting a contour of the left ventricle to facilitate the positioning of a coiled guidewire , such as a SAFARI guidewire , in the left ventricle during the medical procedures , in particular the TAVI / TAVR procedure .

Claims

Claims1. A computer-implemented method for classification of a medical image (3) to analyse whether at least one of a plurality of medical images (3) was captured while a body duct (10) , preferably a cardiac component or a vessel component, in particular the aortic root (11) , the aortic, mitral, pulmonary, and / or tricuspid valve, was under the influence of a contrast agent (14) , the method comprising the steps: successively receiving at least one or a plurality of medical images (3) , preferably in real-time for several medical images (3) , using an input interface, processing the at least one medical image (3) using a processing unit (4) operatively connected to the input interface, thereby determining a probability that the medical image (3) was captured while under the influence of the contrast agent (14) based on the medical image (3) and a classification model stored on a memory unit, using the processing unit (4) for generating a classification of the medical image (3) whether the medical image (3) was captured under the influence of the contrast agent (14) based on the probability, transmitting the classification to an output interface, preferably in real-time.

2. The computer-implemented method according to claim 1, wherein the plurality of medical images (3) are two-dimensional or three-dimensional medical images, in particular fluoroscopic images.

3. The computer-implemented method according to one of the preceding claims, wherein the method comprises the step of:determining the probability of a newly received medical image (3) based on the probability of at least one previously received medical image (3) .

4. The computer-implemented method according to claim 3, wherein the method comprises the step of: applying a first algorithm, in particular a machine learning algorithm, to take into account the time between the acquisition of the newly received medical image (3) and the previously received medical image (3) to determine the probability.

5. The computer-implemented method according to one of the preceding claims, wherein the method comprises the steps of :Applying a second algorithm, in particular a machine learning algorithm, to the non-contrast medical images and / or those medical images (3) which were classified as acquired under the influence of the contrast agent, and in particular a time between the acquisition, and generate a predicted probability of whether a future image is a contrast or non-contrast image, predicting the classification of at least one future medical image (3) based on the predicted probability.

6. The computer-implemented method according to one of the preceding claims, wherein the method comprises the step of: determining a spatial orientation and / or a position of at least one anatomical feature (10) in the medical images (3) based only on the plurality of medical images (3) which were classified as captured under the influence of the contrast agent (14) and / orreconstructing a two- or three-dimensional representation (103, 102) of the at least one anatomical feature (10) based only on image registration of the plurality of medical images (3) which were classified as captured under the influence of the contrast agent (14) , and in particular based on a three-dimensional model of the at least one anatomical feature (10) , preferably a pre-op- erative three-dimensional computer tomography model, and preferably visually displaying the medical images (3) including the two- or three-dimensional representation (103) of the at least one anatomical feature (10) in the medical images (3) which were not classified as captured under the influence of the contrast agent in real-time .

7. The computer-implemented method according to claim 6, wherein the at least one anatomical feature (10) is selected from at least one of: at least the non-coronary cusp, preferably a tip of the non-coronary cusp, and in particular further a tip of the right and / or left coronary cusp, at least one coronary artery, in particular both coronary arteries, the aortic root, a calcification in proximity of the native heart valve, a contour of the left ventricle, the interventricular septum separating the right and left ventricle, in particular the membranous septum.

8. The computer-implemented method according to one of the preceding claims, wherein the method comprises the step of:determining a spatial orientation and / or position of at least one medical instrument (16, 17, 18) , and in particular a two- or three-dimensional representation (105, 104) of the at least one medical instrument (16, 17, 18) , in the medical images (3) based on the plurality of medical images (3) which were classified as captured not under the influence of the contrast agent, and in particular based on a known predefined shape of the at least one medical instrument (16, 17, 18) , and preferably visually displaying medical images (3) the two- or three-dimensional representation (103) of the at least one medical instrument (16, 17, 18) in the medical images (3) which were classified as captured under the influence of the contrast agent in real-time.

9. The computer-implemented method according to one of the claims 6 or 7 and 8, wherein the method comprises the step of :Merging the reconstructed two- or three-dimensional representation (103, 102) of the at least one anatomical feature (10) with the two- or three-dimensional representation (105, 104) of the at least one medical instrument (16, 17, 18) , wherein the merged two- or three-dimensional representation (107, 106) shows a spatial relationship between the at least one anatomical feature (10) and the at least one medical instrument (16, 17, 18) .

10. The computer-implemented method according to claim 8 or 9, wherein the method comprises: determining a geometric heart valve prosthesis characteristic (21) based on the spatial orientation and / or position of the at least one medical instrument (16,17, 18) , preferably the heart valve prosthesis (17) , and in particular at least one user input, determining a one-, two-, or three-dimensional representation of the at least one geometric heart valve prosthesis characteristic (21) , visually displaying the medical images (3) including the one-, two-, or three-dimensional representation (103) of the at least one geometric heart valve prosthesis characteristic (21) in the medical images (3) , in particular the medical images (3) which were classified as captured under the influence of the contrast agent in real-time.

11. The computer-implemented method according to one of the preceding claims, wherein the medical images (3) acquired under the influence of the contrast agent are used for predicting and / or determining at least one prosthetic or native valvular insufficiency, in particular selected from at least one of: paravalvular leakage occurring around edges of a heart valve prosthesis, regurgitation of the native heart valve, valve stenosis, in particular due to tissue overgrowth or due to pannus formation, a risk for thrombosis, a risk for a conduction pathway, preferably the His conduction pathway, of the heart being obstructed, in particular based on the spatial orientation and / or position of the at least one anatomical feature (10) including the interventricular septum, preferably further based on the spatial orientation and / or position of the at least one medical instrument (16, 17, 18) .

12. The computer-implemented method according to one of the preceding claims, wherein the method comprises: applying a first pre-trained machine learning model, in particular a first neural network, to the medical images (3) classified as acquired under the influence of the contrast agent, wherein the first pre-trained machine learning model is preferably applied for determining the spatial orientation and / or position of the at least one anatomical feature (10) and applying a second pre-trained machine learning model, in particular a second neural network, to the medical images (3) classified as acquired under the influence of the contrast agent, wherein the second pre-trained machine learning model is preferably applied for determining the spatial orientation and / or position of the at least one medical instrument (16, 17, 18) .

13. The computer-implemented method according to one of the preceding claims, wherein the method comprises the steps of : measuring a concentration of the contrast agent within the body duct (10) , in particular the aortic root (11) , using an intracorporeally arranged sensor (8) , in particular attached to a medical instrument (16, 17, 18) , using the processing unit (4) for generating the classification of the plurality of medical images (3) supported by the measured concentration of the contrast agent .

14. The computer-implemented method according to one of the preceding claims, wherein the method comprises the step of:pre-processing the medical images (3) using the processing unit (4) by defining at least one region of interest (5, 51) corresponding to a subregion of the medical images (3) , wherein the at least one region of interest (5, 51) is arranged around a section of the medical images (3) corresponding to a medical instrument (16, 17, 18) and / or corresponding to an anatomical section, in particular the aorta, preferably the aortic root (11) , and / or a plurality of regions of interest (5, 51) are arranged spaced apart from each other, in particular in a chessboard pattern, determining the probability of each medical image (3) using the processing unit (4) based on the at least one region of interest (5, 51) of the medical image (3) , in particular by taking into account a plurality of regions of interests (5, 51) of the medical image (3) .

15. The computer-implemented method according to one of the preceding claims, wherein the method comprises the steps of :Receiving at least one first medical image (3) which includes a first anatomical feature or a medical instrument and Receiving at least one second medical image (3) which includes a different second anatomical feature or a medical instrument, wherein the first and second medical images were acquired simultaneously or sequentially, processing the first and second medical image (3) using the processing unit (4) , therebyDetermining the probabilities that the first and the second medical image (3) was acquired while under theinfluence of the contrast agent (14) , respectively, based on the second and first medical image (3) and the classification model,Using the processing unit (4) for generating a first and second classification of the first and second medical image (3) whether the medical images (3) were captured under the influence of the contrast agent (14) based on the probabilities, Transmitting the first and second classification to the output interface, preferably in real-time.

16. A computer-implemented method, in particular according to one of the preceding claims, wherein the method comprises the steps of: successively receiving a plurality of medical images (3) in real-time, using an input interface, analysing an initial image set comprising at least one of the medical images (3) taken at an initial time point using a processing unit (4) connected to the input interface thereby generating an initial result, predicting a subsequent result for at least one subsequent image set taken at a subsequent time point based on the initial result, thereby generating a predicted subsequent result, analysing a subsequent image set comprising at least one medical image received at the subsequent time point, thereby generating an analysed subsequent result, generating a refined subsequent result based on the predicted subsequent result and the analysed subsequent result .

17. The computer-implemented method according to claim 16, wherein the step ofDetermining the initial result at an initial time point is based on the initial image set comprising at least two received medical images (3) obtained at separate time points, preferably the initial time point, and a time point temporally prior to the initial time point.

18. The computer-implemented method according to one of the claims 16 or 17, wherein the step of: generating the predicted subsequent result and / or generating the refined subsequent result is based on a mathematical prediction algorithm, in particular at least one of (i) a Kalman filter, (ii) a particle filter, (iii) a recurrent neural network, (iv) a weighted sum and (v) an optical flow algorithm, preferably on different mathematical prediction algorithms.

19. Computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method of one of the claims 1 - 18.

20. A system, in particular comprising or consisting of a data processing unit, comprising means for carrying out the steps of the method of one of the claims 1 - 18.

21. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method of one of the claims 1 - 18.

22. Use of the computer-implemented method of one of the claims 1 - 18 for at least one of:planning a medical procedure, in particular a transcatheter aortic, mitral, pulmonary tricuspid valve or valve component implantation or replacement, selecting and / or customizing at least one medical instrument (16, 17, 18) , displaying the medical images (3) on a display (6) in real-time including the two-dimensional representation (103) of the aortic root (11) and / or the two-dimensional representation (105) of the medical instrument (16, 17, 18) , displaying the merged two- or three-dimensional representation (103, 102) of the aortic root (11) with the two- or three-dimensional representation (105, 104) of the medical instrument (16, 17, 18) , and adjusting a dosage of a fluoroscopy imaging device for the acquisition of the medical images (3) .

23. A computer-implemented method of training a machine-learning model for classification of a plurality of medical images (3) to predict whether the medical image (3) was captured while a body duct, in particular the aortic root (11) , a mitral, a pulmonary, a tricuspid valve, or a valve component, was under the influence of a contrast agent, in particular according to the computer-implemented method of one of the claims 1 - 18, comprising: receiving an input training dataset comprising medical training images, and contrast agent classification labels which indicate whether the medical training image was acquired under the influence of a contrast agent (14) , and in particular a patient classification label which indicates different patients of the medical training images,optionally pre-processing the medical training images by normalizing a pixel intensity of the received medical training images, preferably based on the patient classification label, preferably defining at least one region of interest (5, 51) , in particular a plurality of regions of interest in the medical training images, in particular around a section of the medical training images corresponding to a medical instrument (16, 17, 18) and / or corresponding to a body duct (10) , in particular the aorta, preferably the aortic root (11) , partitioning the medical training images, labels, and in particular regions of interests or the plurality of regions of interest, into a training subset and a validation subset, training a machine learning model on the training subset based on a loss function, and validating the trained machine learning model on the validation subset, tuning the hyperparameters of the machine learning model based on a performance metric, and outputting the trained, validated, and tuned machine learning model.

Images