Method and system for annotating x-ray fluoroscopy images

Abstract

System for annotating X-ray fluoroscopy images, configured for performing the steps of: Providing a 3D anatomical surface model of at least a part of a person's heart; Determining at least one local anatomic surface property for positions on a surface in said part; Obtaining a 2D X-ray fluoroscopy image from said at least part of said persons heart; Determining corresponding landmarks in the 3D anatomical surface model and the 2D X-ray fluoroscopy image; wherein The step of generating an annotated image comprises: Annotating, such that said corresponding landmarks coincide, the 2D X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property.

Description

[0001] Method and system for annotating X-ray fluoroscopy images

[0002] The present invention relates to a method and system for annotating X-ray images, in particular X-ray fluoroscopy images, and more in particular for annotating a real-time or live generated video stream- and thus moving - fluoroscopy images.

[0003] Publications in which images are annotated are known in the art. US Patent application US20120296196A1 describes a technique for puncturing the atrial septum, for positioning a so called MitraClip on the leaflets of the mitral valve. Although also related to a persons heart, this publication has little to do with positioning an electrode in the ventricular septum; a technique in which puncturing the septum has to be avoided in any case.

[0004] US Patent application US2020138521A1 describes a technique for annotating one 3D roadmap model, obtained by weighted combination of information from multiple 3D models. Registering the 3D roadmap with a stored DICOM image dataset from a PACS or VNA excludes providing a real time overlay and does not allow to be used during invasive treatments.

[0005] X-ray fluoroscopy is a technique that is commonly used during invasive medical treatments since it allows images to be generated of an area of interest of a patient, while allowing medical personnel to have access to the patient. An X-ray fluoroscopy image is a 2 dimensional (2D) image by definition. In order to create a three dimensional (3D) image by means of fluoroscopy, multiple 2D should be made from different angles. Throughout this application, when fluoroscopy is mentioned, imaging from the same angle is meant, and thus the generation of 2D images, unless otherwise stated or implied. As a consequence, the invention relates to mapping a 3D model onto a 2D image. The 2D image may be a live generated image, and given the fact that fluoroscopy may be applied continuously during a time interval, be a live generated image, continuously changing in time.

[0006] One of the invasive treatments it is used for is the placement of pacemakers and in particular during the step of positioning and placing pacemaker electrodes, it may provide useful information with respect to the electrodes actual position relative to the heart. These electrodes, and in fact in general the metal and plastics materials they are composed of, are well visible in the X-ray images.

[0007] According to the prior art, this electrode is often to be positioned in the apex of the right ventricle. The electrode is inserted into the superior vena cava at a distance from the heart and guided through the vein into the heart. For finding this location during invasive treatment it is nowadays common to use a combination of X-ray fluoroscopy and detecting or sensing physical resistance, when putting the electrode, guided by its wire, into position. This is possible since the superior vena cava connects to the heart at the right top side, and the wire can be inserted through the right atrium and the right ventricle to the apex, which is its lowermost point. When this point is reached, an increased resistance is felt, and on X-ray fluoroscopy it can be seen that the wire no longer propagates but tends to curve. Additionally, when the wire is further inserted, it can be seen that it starts to bend, which is also an indication that tissue is reached and further positioning should be done by screwing.

[0008] Alternatively or additionally, it is possible to detecting the influence of a signal fed through the electrode on a patient' s heart pace when it has reached the apex. Such influence may be determined by known technical means, such as the use of an electro-cardiogram.

[0009] A new method for implanting a pacemaker into the heart has recently been proposed, wherein instead of placing a pacemaker electrode in the apex of the right ventricle, the electrode is now suggested to be placed in the tissue of the septum which divides the left and right ventricle of the heart. Per definition, this method thus relates to the ventricular system, and not to the atrial system. Wherever in this application the septum is mentioned, this is therefore the septum of the ventricular system, which divides the left and right ventricle of the heart. This method is also referred to as Conduction System Pacing.

[0010] Currently there are a limited number of clinical studies showing that this new location prevents 'pacing induced heart failure', a deterioration of the cardiac function which could occur after a period of right ventricle apex pacing. Moreover the new location also seems to repair left ventricular dyssynchrony since the native conduction system of the heart can be re-activated after a so-called left bundle branch block.

[0011] The new method is currently performed and studied with all existing equipment, and leads to a lot of different approaches by different doctors. The standard equipment used to implant a pacemaker is X- Ray fluoroscopy. An X-Ray method which allows real-time imaging of the beating heart. On these images the pacemaker wires are clearly visible but the tissue contrast is almost or even totally absent, making it impossible to distinguish the tissue from the blood or visualize tissue characteristics like myocardial infarction.

[0012] During the pacemaker implantation the experienced doctors navigate based on the electrical signals measured by electrocardiograms (ECG) that are altered by the (active) pacemaker wire, while in the heart. Not all doctors are able to do this and therefore every doctor follows its own strategy, creating a lot of diversity and disabling standardization and the definition of guidelines.

[0013] Although in early publications (Conduction System Pacing for Cardiac Resynchronisation, Arrhythm Electrophysiol Rev. 2021 Apr; 10(1): 51-58) it is was assumed that the exact location of the electrode in the septum was not that important, during first practical attempts it is found that in some patients it is impossible to penetrate the tissue, while in others the septum is easily penetrated by a pacemaker screw, which is present at the leading end of the electrode and serves for fixation of the electrode once its correct position is found. This may be caused because the tissue constitution of the septum varies between individuals.

[0014] According to an aspect of the present invention, it is found that providing information about the septum tissue during invasive treatment can take away the practical disadvantages of performing the newly proposed treatment.

[0015] Tissue however, is hardly visible on X-ray fluoroscopy in general, and in case of the septum which is surrounded by blood and other tissue such as the left and right ventricle walls, practically invisible.

[0016] The publication of the international patent application W02017200385A1 by the same inventors, which is herein incorporated by reference, describes a method and device for verifying the position of an interventional device. In this publication, methods and devices are disclosed for annotating X-ray fluoroscopy images with information of amongst others tissue and organ information.

[0017] However, for enabling a practical implementation of Conduction System Pacing more detailed information seems to be required. Also for similar purposes, like the positioning of leadless pace makers or taking a biopsy, the same considerations apply.

[0018] It is therefore a goal of the present invention to provide a method for providing annotated X-ray images that are in particular useful for Conduction System Pacing, that when performed on or with suitable instruments and / or technologies, serves as a medical instrument or system. Wherever the invention describes a method, a system (configured or adapted) for performing the method is also meant as part of the invention.

[0019] In a first aspect, the present invention proposes a method for annotating X-ray fluoroscopy images, comprising the steps of:

[0020] Providing a 3D anatomical surface model of at least a part of a person's heart;

[0021] Determining at least one local anatomic surface property for positions on a surface in said part;

[0022] Obtaining a real-time 2D X-ray fluoroscopy image from said at least part of said persons heart;

[0023] Determining corresponding landmarks in the 3D anatomical surface model and the real-time 2D X-ray fluoroscopy image;

[0024] Wherein the step of generating an annotated image comprises: Annotating, such that said corresponding landmarks coincide, the real-time 2D X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property.

[0025] A 3D anatomical surface model is known in the art as a model that defines the tissue surfaces that form inner or outer boundaries for a part of the human body, for instance the heart. A 3D anatomical surface model of the heart thus inter alia defines the inner walls of the atria the ventricles, as well as the outer walls thereof, in a three dimensional space. The 3D surface model may be a combined epicardial and endocardial model, i.e. the 3D surface model comprises both internal and external surfaces of a region of interest of the heart, which region preferably comprises both the left and the right ventricle and more preferably comprises the complete heart. The landmarks may comprise anatomical landmarks, and preferably wherein the anatomical landmarks comprise at least landmarks on the heart such as the left or right atrium, left or right ventricle, aorta, sinus coronarius, coronary arteries and / or combinations thereof. In particular the landmarks may comprise at least one of the right atrium, the right ventricle or the left ventricle.

[0026] A well known and suitable example of an X-ray image is a so called Rbntgen image, while X-ray fluoroscopy is a study of moving body structures that can be seen as an X-ray movie. A continuous X- ray beam is passed through the body part being examined. The beam is detected by a light-sensitive plate of which the information is transmitted to a TV-like monitor so that the body part and its motion can be seen in detail. Therefore, the X-ray images according to the invention are real-time 2D X-ray images, and more in particular a (continuous) real-time 2D X-ray movie. This technique allows the invention to provide an undelayed combined image, thus at real-time. In a further aspect, an angiogram can be made by the addition of a contrast fluid in the blood, which allows to follow (blood) movements.

[0027] The mentioned landmarks may be anatomic landmarks, such as (parts of) blood vessels, (parts of) valves and (parts of) the hearts chambers as well as (parts of) ribs and the spine which can be seen in the X-ray image, or external landmarks, such as markers positioned outside the persons body, for instance objects recognizable on an X-ray image, on the chest. The markers may also be pacemaker wires (pre)positioned in a persons veins and thus indicate a known location with respect to the heart.

[0028] An annotated image according to the invention may also be referred to in the art as a combined image, or an image provided with an extra visible layer. At least one local anatomic surface property of said part of said persons heart means that for at least one point on the surface in the part of the heart this property is known, but preferably this is the case for multiple surface points, and more preferably for all points constituting the surface of said part of the heart. The at least part of the heart may herein of comprise the entire heart.

[0029] The step of providing a 3D anatomical surface model of at least a part of a person's heart may be done prior to obtaining an X-ray fluoroscopy image, in particular when it is to be based on measurements. As described in the aforementioned W02017200385A1, this can for instance be done by means of a CT scan, an MRL These types of measurement usually require encapsulation of a person in a measurement device and are for that reason alone already not suitable to be done during invasive treatment. The method according to the invention is - when performed on or with suitable devices - intended to be applied as a technical support during invasive treatments, such as the application of a pacemaker or electrode thereof, of the taking of a biopsy. That means, that the annotated image according to the invention is also displayed on suitable display means, such as a monitor screen or augmented or virtually reality means such as a head set. The annotation may be depicted on separately displayed X-ray (fluoroscopy) video provided to a doctor, which may be a snapshot of an X-ray (fluoroscopy) image.

[0030] Alternatively, the step of providing a 3D anatomical surface model may be performed by means of calculations based on the X-ray image or (anatomic) markers therein. This can be done, for instance by computer aided modelling, or by selecting a 3D surface model from a database, based on similarities of the actual X-ray image with saved X-ray images from earlier scans or based on personal data of the person that is scanned, such as size, weight, age and gender. Although heart tissue may not be visible in an X-ray image, a contour of the heart may be determined based on the shape of the lungs. With the contour and its location, a 3D model of the heart may be constructed.

[0031] In both cases, the 3D surface model may as an additional method step be computer-animated, in order to move along with the persons beating and thus moving heart, as visible in the X-ray image. As an alternative, the 3D surface model may be recalculated or redetermined in real time, in order to reflect the movement of the beating heart.

[0032] In general, according to the invention the step of providing a 3D anatomical surface model comprises calculating said model based on data obtained from a CT, MRI or Ultrasound scan of said persons heart or a combination of at least two of these technologies, which has appeared to provide more detailed information about the heart muscle, such as its thickness or providing a model generated or selected from a database based on patient information or information about the persons heart derived from the X-ray fluoroscopy imaging. It may further comprise repeatedly determining corresponding landmarks in the anatomical surface model and the X-ray fluoroscopy image; and repeatedly generating an annotated image wherein at least part of the 3D anatomical surface model is annotated in the X-ray fluoroscopy image, such that said corresponding landmarks coincide in order to obtain a moving annotated image.

[0033] The local anatomic surface property that is annotated onto the X-ray image may be selected in accordance with the medical treatment to be performed on the person. Possible annotations are: a surface material such as regular tissue, scar tissue, muscle tissue or conductive tissue like His, left bundle branch and right bundle branch tissue; a surface material such as a granuloma or a (local) lesion; a direction in which the surface locally extends, or a direction related thereto, such as a direction perpendicular to the direction in which the surface locally extends or a direction in which an interventional device may be positioned; a border of a tissue forming the surface; a thickness of a tissue forming the surface; a cross section of a tissue forming the surface in a plane intersecting the persons heart; an electric activity, blood flow or density of blood vessels of a tissue forming the surface.

[0034] A surface material such as regular tissue, scar tissue or muscle tissue can be determined by postprocessing of medical image data, artificial intelligence, and / or local measurements performed on the tissue either determined through measurement of local electrical activity of the tissue or the result of local stimulation of the tissue of which the effect is measured through another measurement device. The postprocessing of the medical image data includes, but is not limited to segmentation of anatomical structures like cardiac chambers and cardiac tissue, and the assessment of local tissue characteristics in the segmented cardiac tissue, and parameters derived from the segmentation of the cardiac chambers of cardiac tissue. The tissue characteristics can be useful to distinguish healthy from diseased tissue and so target a therapy or a biopsy to respectively healthy and suspect tissue.

[0035] A direction in which the surface locally extends, or a direction related thereto, such as a direction perpendicular to the direction in which the surface locally extends can be determined by creation of a 3D surface model based on the segmentation of anatomical structures like cardiac chambers and cardiac tissue. The direction perpendicular to the surface is based on the local extent of the 3D surface model. Secondary directions that are relevant for the therapy can be determined based on the direction of the surface or the direction perpendicular to the surface and other surface characteristics. The direction property is of interest to allow the alignment of the interventional device and a local direction in order to obtain the highest chance for a successful therapy and reduce the risk of a hazardous situation like an insertion of an interventional device parallel to the tissue, or a tissue perforation.

[0036] This may for instance be the case when a pacemaker with a bipolar lead is to be positioned in a septum of a persons heart. Such lead comprises a top and a ring electrode, with a fixed distance between them, for instance a centimetre. The bipolar electrode should be positioned in such way that the tip and the ring electrodes are in contact with the left bundle branch and right bundle branch respectively. Since these may be at a smaller distance from each other than the distance between the electrodes, the only way to touch both electrodes is to position the pace maker at an angle with the septum tissue surface. Such angle can be calculated based on the ratio of the distance between the electrodes and the thickness of the septum, and then be annotated as a guidance.

[0037] A thickness of a tissue forming the surface can be determined by a local distance between two surfaces constituting the inner and the outer surface of the tissue of interest; Similar to tissue characteristics a local tissue thickness can be used to distinguish healthy from diseased tissue and so target a therapy or a biopsy to respectively healthy and suspect tissue. Moreover the tissue thickness can aid the calculation of the optimal direction of the interventional device.

[0038] A cross section of a tissue forming the surface in a plane intersecting the persons heart can be determined by defining a direction in an organ or tissue, and defining a plane perpendicular to the defined organ or tissue axis such that the intersection between the plane and the organ or tissue segmentation(s) forms a line or an area that is at the location of the planned intervention, in the direction that is relevant for the planned intervention. The cross-section provides insight into the depth of the intervention with respect to the organ or tissue of which the cross section is made

[0039] An electric activity can be measured by local measurements performed on the surface of the tissue either determined through measurement of local electrical activity of the tissue or the result of local stimulation of the surface of the tissue of which the effect is measured through another measurement device measuring the resulting effect on the body surface electrocardiogram. This method allows assessment of the location of the interventional device with respect to the different aspects of the conduction system like the HIS bundle, left bundle branch and right bundle branch or normal myocardium.

[0040] Blood flow or perfusion can be measured by perfusion imaging using contrast enhanced imaging techniques using either a nuclear labeling technique or a tissue perfusion technique. The blood flow or perfusion provides insight into the viability of the tissue and allows to distinguish healthy from diseased tissue and so target a therapy or a biopsy to respectively healthy and suspect tissue. The location, size and density of blood vessels can be measured by an angiographic imaging technique. The annotation of the location, size and density of blood vessels prevents that interventions are done in an area in which the risk for a perforation of the blood vessels, and as a consequence hematoma creation is too high.

[0041] The visible indication that is used as the annotation, may be a color indication, a mark indication such as a dot or a line or a ruler or an arrow, or a readable indication such as a character. Evidently, combinations of the mentioned annotation types are also thinkable. Herein, it is possible to indicate different annotations with a different annotation type, such as indicating cross section with a colored line and directions with arrows. Moreover, according to an embodiment of the invention, also the surfaces from the 3D anatomical surface model themselves are annotated. This provides the opportunity to combine annotations and to provide for instance a 3D surface model with indications of a direction perpendicular to a surface, surfaces with their corresponding thickness or indications where muscles or scar tissue are located. In some invasive treatments, besides exact locations, exact directions or depths are relevant, and a combination of the 3D surface model with annotations provides a solution for this need. The person performing the invasive treatment may then use the 3D surface model and the annotations as a guidance and take decisions based on the annotated X-ray image.

[0042] In a more advanced embodiment, the method according to the invention comprises providing a value, for instance in a lookup table or database, indicating the suitability of anatomic surface properties for one or more specific invasive treatments such as the positioning of a pacemaker electrode or taking a biopsy. Then, based on said value, the suitability of the local anatomic surface for said one or more specific invasive treatments may be determined and the step of generating an annotated image comprises annotating an indication representing the suitability.

[0043] As an example, for an invasive treatment where scar tissue or muscles tissue are to be avoided, these may be indicated in red, while suitable tissue is marked in green. The same may go for a direction in which the surface is approached while performing invasive treatment. A correct direction or position of the interventional device or a direction or position of the interventional device that is at least within a certain threshold from an intended direction or position may be indicated with a different marking or color than an incorrect direction. In this case, the annotation provides a (preferably realtime) feedback on the positioning of the treatment device. It is further thinkable that annotations other than visible ones are used, such as a sound when a suitable location and direction are used or the other way round: when a location and direction are not found suitable. In yet another embodiment, the method according to the invention comprises obtaining further data of the persons heart, such as data from earlier invasive treatments such as their position on the surface or successfulness, measurement data such as an electrocardiogram; and / or measurement data such as an angiography, in particular a coronary angiography; wherein the step of generating an annotated image comprises annotating a visible indication representing said further measurement data. An electrocardiogram may provide information about the conductivity of the tissue of the heart, including the muscle tissue, and provide information about the effects of for instance heart attacks that have taken place and their results. An angiogram may provide information about n about the vascularization and viability of the tissue'.

[0044] In a further embodiment, the method according to the invention comprises providing a value, for instance in a lookup table or database, representing the suitability of anatomic surface properties combined in view of said further measurement data, for one or more specific invasive treatments such as the positioning of a pacemaker electrode or taking a biopsy, and based on said value, determining the suitability of the local anatomic surface in view of said further measurement data for said one or more specific invasive treatments, wherein the step of generating an annotated image comprises providing an annotation representing the suitability.

[0045] In an embodiment, the present invention relates to a method adapted for providing annotated images suitable for Conduction System Pacing. Herein, the 3D anatomical surface model of a person's heart comprises at least the septum between the left and the right ventricle; and the step of providing a 3D anatomical surface model comprises determining a surface property of said septum; and the step of generating a combined image comprises annotating an indication of said surface property of the septum in the X-ray fluoroscopy image.

[0046] Being located centrally in the lower part of the heart, between the left and the right ventricle, and being surrounded by heart muscle tissue, the septum is not visible on X-Ray images of the heart. Without an indication of its location, it is difficult to position and to place an electrode and a lot of trial and error is involved in placing a pacemaker electrode. Moreover, such trial and error, comprising attempts for placing the electrode at the correct position against the septum and subsequently to fixate it by screwing, may leave damaged tissue at those locations where an unsuccessful attempt was made, or worse: may result in a pierced septum.

[0047] For that reason, according to the invention, the surface property which is annotated may be one of the group of: a cross section of a tissue forming the surface of said at least part of the left ventricle including the septum; a thickness of a tissue forming the septum surface; a border of a tissue forming the septum; a direction in which the septum surface locally extends, or a direction related thereto, such as perpendicular to the direction in which the surface locally extends.

[0048] A likelihood estimator for a location (E.G. HIS, LBB or LBB-area, artery, vein)

[0049] This information enables a person performing an invasive treatment for placing a pace maker electrode for Conduction System Pacing in a patients septum, and to directly place it at the correct location and to apply it in the correct direction. The electrode may be guided by guiding tools, which allows to bring the electrode in a certain position and additionally to control its direction prior to screwing. As a result, with the aid of the method according to the present invention, applying Conduction System Pacing technology has become easier and more reliable, with a severely reduced chance on unsuccessful attempts and damaging heart tissue.

[0050] The invention also generally relates to a device or system configured for performing at least a number of the method steps described above.

[0051] More specifically the invention relates to a device or system for providing a 3D anatomical surface model of at least a part of a person's heart comprising a data processor configured for: determining at least one local anatomic surface property for positions on a surface in said part; receiving or obtaining an X-ray fluoroscopy image from said at least part of said persons heart;

[0052] Determining corresponding landmarks in the 3D anatomical surface model and the X-ray fluoroscopy image;

[0053] Annotating, such that said corresponding landmarks coincide, the X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property.

[0054] The invention will now be elucidated into more detail with reference to the following figures. Herein:

[0055] Figure 1 shows a schematical view of a person's heart with a pace maker wire;

[0056] Figure 2 shows a cross section of the left and right ventricle;

[0057] Figure 3 shows a prior art X-ray image of a person's heart with pace maker wires;

[0058] Figure 4 shows an annotated X-ray image according to the invention;

[0059] Figure 5 shows an annotated X-ray image according to the invention;

[0060] Figures 6 A-B show annotated X-ray images according to the invention; Figure 1 shows a perspective cross section 1 of a person's heart 2 with a pace maker electrode 4 lead by a wire 3. The electrode is screwed in the septum 5, which is the tissue between the left ventricle 6 and the right ventricle 7. As visible in detail A, the electrode 4 is positioned at a location where the so called HIS bundle 8 coming from the AV node 9 is split into a left bundle branch 8A and a right bundle branch 8B and it connects to both. This may be an intentional placement of the electrode for Conduction System Pacing, when there's reason to believe that both left and right bundles should be paced.

[0061] Figure 2 shows a second cross section of the heart 2 from figure 1, taken along the line B-B', with various positions of the electrode 4 and its wire 4 in the septum 5. From top to bottom of the figure an attempt V can be seen wherein the electrode is too close to a vein, an attempt W wherein the electrode is not screwed far enough in the septum, an attempt X wherein the electrode extends through the septum, a correct placement Y and an attempt Z wherein the electrode is placed under a wrong angle.

[0062] Figure 3 shows a standard X-Ray image 10 of a person's heart according to the prior art. In the picture, pace maker wires 3 and electrodes 4 can be distinguished, but no tissue of the heart is visible. From this image it becomes clear that the information needed for positioning an electrode correctly as shown in figure 2 cannot be derived from an X-ray image alone.

[0063] Figure 4 shows an annotated X-ray image according to the invention, wherein 3D surfaces of parts of the heart are depicted. A first annotation is formed by the different colors that are used for different parts of the persons heart. Furthermore, additional annotations, indicated with self-explanatory text in the picture, are added to the X-ray images.

[0064] Figure 5 shows an annotated X-ray image of the process of placing a pace maker electrode. In this case, a left ventricle epicardium is depicted, which shows the inner and outer surface of the left ventricle, at a selected cross section of the heart. In other words: the wall of tissue forming the left ventricle is located between these two rings. The septum forms part of this wall and as explained in figure 2, the electrode should be positioned in a specific position and orientation with respect to the septum. The annotated left ventricle epicardium allows shows this position to a cardiologist while putting the electrode in place, and allows to do so first time right.

[0065] Figures 6A and 6B show a set-up 11 for performing the method according to the invention. An X-Ray tube 12 is used to irradiate a person's heart 2 while a projection 13 thereof is analyzed. The projection also comprises an annotation, formed by a projection 14' of a vector 14 indicating a desired orientation of the electrode 4 for screwing it into the septum 5. The alignment in the image plane can be seen in the projection. The difference in projection length between the vector 14' and a projection 4' of the electrode is a measure of the difference in orientation in the third dimension, which is the dimension perpendicular to the plane of the two dimensional figure 6A, 6B. Agreement in projection length between the vector and the electrode as shown in figure 6B is a measure of the corresponding orientation in the third dimension.

[0066] This may for instance be used when a pacemaker with a bipolar lead is to be positioned in a septum of a person's heart. Such lead comprises a top and a ring electrode, with a fixed distance between them, for instance a centimetre. The bipolar electrode should be positioned in such way that the tip and the ring electrodes are in contact with the left bundle branch and right bundle branch respectively. Since these may be at a smaller distance from each other than the distance between the electrodes, the only way to touch both electrodes is to position the pace maker at an angle with the septum tissue surface. Such angle can be calculated based on the ratio of the distance between the electrodes and the thickness of the septum, and then be annotated as a guidance.

[0067] When applying the method, additional information, such as information from an ECG, may be used, to determine that the target tissue issue is actually reached.

[0068] The invention relates to a method according to the following clauses 1-15, as well as a device and / or system for performing such method or for serving as an aid for performing such method, as well as a device according to clause 16.

[0069] Clauses

[0070] 1. Method and system for annotating X-ray fluoroscopy images, comprising the steps of:

[0071] Providing a 3D anatomical surface model of at least a part of a person's heart; Determining at least one local anatomic surface property for positions on a surface in said part;

[0072] Obtaining an X-ray fluoroscopy image from said at least part of said persons heart; Determining corresponding landmarks in the 3D anatomical surface model and the X-ray fluoroscopy image; characterized in that:

[0073] The step of generating an annotated image comprises: o Annotating, such that said corresponding landmarks coincide, the X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property. 2. Method according to clause 1, wherein the local anatomic surface property is selected from the list of: a surface material such as regular tissue, scar tissue, muscle tissue or conductive tissue like His, left bundle branch and right bundle branch tissue; a surface material such as a granuloma or a (local) lesion; a direction in which the surface locally extends, or a direction related thereto, such as a direction perpendicular to the direction in which the surface locally extends or a direction in which an interventional device may be positioned; a border of a tissue forming the surface; a thickness of a tissue forming the surface; a cross section of a tissue forming the surface in a plane intersecting the persons heart; an electric activity, blood flow or density of blood vessels of a tissue forming the surface;

[0074] A likelihood estimator for a location (E.G. HIS, LBB or LBB-area, artery, vein).

[0075] 3. Method according to clause 1 or 2, wherein the visible indication is a color indication, a mark indication such as a dot or a line or a ruler, or an arrow, or a readable indication such as a character.

[0076] 4. Method according to any of the preceding clauses, further comprising annotating the X-ray fluoroscopy image with the surfaces from the 3D anatomical surface model themselves.

[0077] 5. Method according to any of the preceding clauses, comprising the step of: providing a value, for instance in a lookup table or database, indicating the suitability of anatomic surface properties for one or more specific invasive treatments such as the positioning of a pacemaker electrode or taking a biopsy; based on said value, determining the suitability of the local anatomic surface for said one or more specific invasive treatments; wherein the step of generating an annotated image comprises annotating an indication representing the suitability.

[0078] 6. Method according to any of the preceding clauses, comprising: obtaining further data of the persons heart, such as: o data from earlier invasive treatments such as their position on the surface or successfulness; o measurement data such as an electrocardiogram; and / or o measurement data such as an angiography, in particular a coronary angiography; wherein the step of generating an annotated image comprises: o annotating a visible indication representing said further measurement data.

[0079] 7. Method according to clause 5 and 6, comprising: providing a value, for instance in a lookup table or database, representing the suitability of anatomic surface properties combined in view of said further measurement data, for one or more specific invasive treatments such as the positioning of a pace maker electrode or taking a biopsy; based on said value, determining the suitability of the local anatomic surface in view of said further measurement data for said one or more specific invasive treatments, wherein the step of generating an annotated image comprises providing an annotation representing the suitability.

[0080] 8. Method according to any of the preceding clauses, wherein the step of providing a 3D anatomical surface model comprises:

[0081] Calculating said model based on data obtained from a CT, MRI or Ultrasound scan of said persons heart or a combination of at least two of these technologies;

[0082] Providing a model generated or selected from a database based on patient information or information about the persons heart derived from the X-ray fluoroscopy imaging;

[0083] Providing a model generated or selected from a database based on patient information or information about the persons heart derived from the X-ray fluoroscopy imaging, finetuned by information from one or more imaging modalities CT, MRI, Ultrasound and / or other measurements: pressure, electrograms.

[0084] 9. Method according to any of the preceding clauses, comprising: repeatedly determining corresponding landmarks in the anatomical surface model and the X- ray fluoroscopy image; and repeatedly generating an annotated image wherein at least part of the 3D anatomical surface model is annotated in the X-ray fluoroscopy image, such that said corresponding landmarks coincide; repeatedly determining a fiducial in the X-ray fluoroscopy image synchronizing the cardiac cycle in the X-ray fluoroscopy with the deformation of the 3D anatomical surface model synchronizing the 3D anatomical surface model with the breathing induced motion by determining a fiducial marker in the X-ray fluoroscopy image, and subtract the cardiac motion in order to obtain a moving annotated image.

[0085] 10. Method according to any of the preceding clauses, wherein: the 3D anatomical surface model of a persons heart comprises at least the septum between the left and the right ventricle; and:

[0086] The step of providing a 3D anatomical surface model comprises determining a surface property of said septum; and

[0087] The step of generating a combined image comprises annotating an indication of said surface property of the septum in the X-ray fluoroscopy image.

[0088] 11. Method according to clause 10, wherein the surface property is at least one of the group of: a cross section of a tissue forming the surface of said at least part of the left ventricle including the septum; a thickness of a tissue forming the septum surface; a border of a tissue forming the septum; a direction in which the septum surface locally extends, or a direction related thereto, such as perpendicular to the direction in which the surface locally extends.

[0089] 12. Method according to any of the preceding clauses, wherein the 3D surface model is a combined epicardial and endocardial model, i.e. the 3D surface model comprises both internal and external surfaces of a region of interest of the heart, which region preferably comprises both the left and the right ventricle and more preferably comprises the complete heart.

[0090] 13. Method according to clause 12, wherein the landmarks comprise anatomical landmarks, and preferably wherein the anatomical landmarks comprise at least landmarks on or around the heart such as the left or right atrium, left or right ventricle, aorta, sinus coronarius, coronary arteries and / or combinations thereof. In particular the landmarks may comprise at least one of the right atrium, the right ventricle or the left ventricle.

[0091] 14. Method according to any of the preceding clauses, wherein the annotation comprises an angle with the septum tissue surface for positioning a a bipolar lead comprising a top and a ring electrode, wherein the angle is calculated such that the tip and the ring electrodes of said lead end up in contact with the left bundle branch and right bundle branch respectively.

[0092] 15. Method according to any of the preceding clauses, comprising annotating a correct direction or position of the interventional device or a direction or position of the interventional device that is at least within a certain threshold from an intended direction or position and in particular indicating the correct direction or position with a different marking or color than an incorrect direction or position.

[0093] 16. Device for providing a 3D anatomical surface model of at least a part of a person's heart comprising a data processor configured for: - determining at least one local anatomic surface property for positions on a surface in said part; receiving or obtaining an X-ray fluoroscopy image from said at least part of said persons heart;

[0094] Determining corresponding landmarks in the 3D anatomical surface model and the X-ray fluoroscopy image;

[0095] Characterized by being configured for:

[0096] Annotating, such that said corresponding landmarks coincide, the X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property.

Claims

Claims1. System for annotating X-ray fluoroscopy images, configured for performing the steps of:Providing a 3D anatomical surface model of at least a part of a person's heart;Determining at least one local anatomic surface property for positions on a surface in said part;Obtaining a real-time 2D X-ray fluoroscopy image from said at least part of said persons heart;Determining corresponding landmarks in the 3D anatomical surface model and the 2D X-ray fluoroscopy image; characterized by: generating an annotated image, comprising: o Annotating, such that said corresponding landmarks coincide, the real-time 2D X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property.

2. System according to claim 1, wherein the local anatomic surface property is selected from the list of: a surface material such as regular tissue, scar tissue, muscle tissue or conductive tissue like His, left bundle branch and right bundle branch tissue; a surface material such as a granuloma or a (local) lesion; a direction in which the surface locally extends, or a direction related thereto, such as a direction perpendicular to the direction in which the surface locally extends or a direction in which an interventional device may be positioned; a border of a tissue forming the surface; a thickness of a tissue forming the surface; a cross section of a tissue forming the surface in a plane intersecting the persons heart; an electric activity, blood flow or density of blood vessels of a tissue forming the surface;A likelihood estimator for a location (E.G. HIS, LBB or LBB-area, artery, vein).

3. System according to claim 1 or 2, wherein the visible indication is a color indication, a mark indication such as a dot or a line or a ruler, or an arrow, or a readable indication such as a character.

4. System according to any of the preceding claims, further comprising annotating the X-ray fluoroscopy image with the surfaces from the 3D anatomical surface model themselves.

5. System according to any of the preceding claims, comprising the step of: providing a value, for instance in a lookup table or database, indicating the suitability of anatomic surface properties for one or more specific invasive treatments such as the positioning of a pacemaker electrode or taking a biopsy; based on said value, determining the suitability of the local anatomic surface for said one or more specific invasive treatments; wherein the step of generating an annotated image comprises annotating an indication representing the suitability.

6. System according to any of the preceding claims, comprising: obtaining further data of the persons heart, such as: o data from earlier invasive treatments such as their position on the surface or successfulness; o measurement data such as an electrocardiogram; and / or o measurement data such as an angiography, in particular a coronary angiography; wherein the step of generating an annotated image comprises: o annotating a visible indication representing said further measurement data.

7. System according to claim 5 and 6, comprising: providing a value, for instance in a lookup table or database, representing the suitability of anatomic surface properties combined in view of said further measurement data, for one or more specific invasive treatments such as the positioning of a pace maker electrode or taking a biopsy; based on said value, determining the suitability of the local anatomic surface in view of said further measurement data for said one or more specific invasive treatments, wherein the step of generating an annotated image comprises providing an annotation representing the suitability.

8. System according to any of the preceding claims, wherein the step of providing a 3D anatomical surface model comprises:Calculating said model based on data obtained from a CT, MRI or Ultrasound scan of said persons heart or a combination of at least two of these technologies;Providing a model generated or selected from a database based on patient information or information about the persons heart derived from the X-ray fluoroscopy imaging;Providing a model generated or selected from a database based on patient information or information about the persons heart derived from the X-ray fluoroscopy imaging, finetuned by information from one or more imaging modalities CT, MRI, Ultrasound and / or other measurements: pressure, electrograms.

9. System according to any of the preceding claims, comprising: repeatedly determining corresponding landmarks in the anatomical surface model and the X- ray fluoroscopy image; and repeatedly generating an annotated image wherein at least part of the 3D anatomical surface model is annotated in the X-ray fluoroscopy image, such that said corresponding landmarks coincide; repeatedly determining a fiducial in the X-ray fluoroscopy image; synchronizing the cardiac cycle in the X-ray fluoroscopy with the deformation of the 3D anatomical surface model; synchronizing the 3D anatomical surface model with the breathing induced motion by determining a fiducial marker in the X-ray fluoroscopy image, and subtract the cardiac motion; in order to obtain a moving annotated image.

10. System according to any of the preceding claims, wherein: the 3D anatomical surface model of a persons heart comprises at least the septum between the left and the right ventricle; and:The step of providing a 3D anatomical surface model comprises determining a surface property of said septum; andThe step of generating a combined image comprises annotating an indication of said surface property of the septum in the X-ray fluoroscopy image.

11. System according to claim 10, wherein the surface property is at least one of the group of: a cross section of a tissue forming the surface of said at least part of the left ventricle including the septum; a thickness of a tissue forming the septum surface; a border of a tissue forming the septum; a direction in which the septum surface locally extends, or a direction related thereto, such as perpendicular to the direction in which the surface locally extends.

12. System according to any of the preceding claims, wherein the 3D surface model is a combined epicardial and endocardial model, i.e. the 3D surface model comprises both internal and external surfaces of a region of interest of the heart, which region preferably comprises both the left and the right ventricle and more preferably comprises the complete heart.

13. System according to claim 12, wherein the landmarks comprise anatomical landmarks, and preferably wherein the anatomical landmarks comprise at least landmarks on or around the heart such as the left or right atrium, left or right ventricle, aorta, sinus coronarius, coronary arteries and / or combinations thereof. In particular the landmarks may comprise at least one of the right atrium, the right ventricle or the left ventricle.

14. System according to any of the preceding claims, wherein the annotation comprises an angle with the septum tissue surface for positioning a a bipolar lead comprising a top and a ring electrode, wherein the angle is calculated such that the tip and the ring electrodes of said lead end up in contact with the left bundle branch and right bundle branch respectively.

15. System according to any of the preceding claims, comprising annotating a correct direction or position of the interventional device or a direction or position of the interventional device that is at least within a certain threshold from an intended direction or position and in particular indicating the correct direction or position with a different marking or color than an incorrect direction or position.

16. System according to any of the preceding claims for comprising a data processor configured for: determining at least one local anatomic surface property for positions on a surface in said part; receiving or obtaining an X-ray fluoroscopy image from said at least part of said persons heart;Determining corresponding landmarks in the 3D anatomical surface model and the X-ray fluoroscopy image;Characterized by being configured for: o Annotating, such that said corresponding landmarks coincide, the X-ray fluoroscopy image with a visible indication representing the determined at least one local anatomic surface property.

Images