Method and system for annotating X-ray fluoroscopic images

Abstract

A system for annotating X-ray fluoroscopic images, configured to perform the steps of: providing a 3D anatomical surface model of at least a portion of a human heart; determining at least one local anatomical surface feature for a location on the surface of the portion; obtaining a 2D X-ray fluoroscopic image from the at least portion of the human heart; and determining corresponding landmarks in the 3D anatomical surface model and the 2D X-ray fluoroscopic image, wherein the step of generating an annotated image includes annotating the 2D X-ray fluoroscopic image with visible indications representing the determined at least one local anatomical surface feature such that the corresponding landmarks match.

Description

Technical Field

[0001] The present invention relates to a method and system for annotating X-ray images, specifically X-ray fluoroscopic images, and more specifically for annotating real-time or live-generated, and thus moving, video stream fluoroscopic images.

Background Art

[0002] Documents indicating that an image can be annotated are known in the art. US Patent Application Publication No. 20120296196A1 describes a technique for puncturing the atrial septum to position a so-called MitraClip on the leaflet of the mitral valve. Although related to the human heart, this document has little to do with the positioning of electrodes within the ventricular septum, which is a technique where septum puncture must be avoided in any case.

[0003] US Patent Application Publication No. 2020138521A1 describes a technique for annotating a single 3D roadmap model obtained by weighting and combining information from multiple 3D models. The method of registering this 3D roadmap with a stored DICOM image dataset from a PACS or VNA precludes the provision of real-time overlays and cannot be used during invasive procedures.

[0004] Fluoroscopy is a commonly used technique during invasive medical procedures because it allows healthcare professionals to generate images of a patient's region of interest while accessing the patient. By definition, fluoroscopic images are two-dimensional (2D) images. To create a three-dimensional (3D) image by fluoroscopy, it is necessary to create multiple 2D images from different angles. Throughout this application, when fluoroscopy is referred to, unless otherwise stated or suggested, it means imaging from the same angle, i.e., generating 2D images. Therefore, the present invention relates to mapping a 3D model onto a 2D image. The 2D image may be a live-generated image, or, given the fact that fluoroscopy can be applied continuously over a certain time interval, it may be a live-generated image that changes continuously over time.

[0005] One of the invasive procedures used is pacemaker placement, and particularly during the positioning and placement steps of the pacemaker electrodes, useful information can be provided regarding the actual position of the electrodes relative to the heart. These electrodes, and in fact the metal and plastic materials that make them up, are generally well visible in X-ray images.

[0006] According to conventional techniques, this electrode is often positioned at the apex of the right ventricle. The electrode is inserted into the superior vena cava at a certain distance from the heart and guided into the heart through the vein. To locate this position during the invasive procedure, it is now common to combine fluoroscopy with detecting or sensing the physical resistance when positioning the wire-guided electrode in place. This is possible because the superior vena cava connects to the upper right side of the heart, allowing the wire to be inserted through the right atrium and right ventricle to its lowest point, the apex. Upon reaching this point, an increase in resistance is felt, and on fluoroscopy, the wire may appear to be bending but no longer advancing. As the wire is inserted further, it may begin to bend, which is also an indication that it has reached the tissue, and further positioning should be done by screwing it in.

[0007] Alternatively or additionally, it is possible to detect the effect of the signals supplied through the electrodes on the patient's cardiac pace once they reach the apex. Such effects can be determined by known technical means, such as the use of an electrocardiogram.

[0008] Instead of placing pacemaker electrodes at the apex of the right ventricle, a new method for implanting pacemakers within the heart has recently been proposed, which now suggests placing electrodes in the diaphragm tissue that separates the left and right ventricles of the heart. Therefore, by definition, this method relates to the ventricular system and not to the atrial system. Where the diaphragm is referred to in this application, it refers to the diaphragm of the ventricular system that separates the left and right ventricles of the heart. This method is also called conduction system pacing.

[0009] Currently, there are limited clinical studies demonstrating that this new location prevents "pacing-induced heart failure," which is a deterioration of cardiac function that can occur after the period of right ventricular apical pacing. Furthermore, this new location appears to repair left ventricular dyssynchrony, because the heart's native conduction system can be reactivated after so-called left bundle branch block.

[0010] This new method is currently being implemented and researched using all existing equipment, leading to many different approaches by different physicians. The standard equipment used to implant pacemakers is fluoroscopy. The X-ray method allows for real-time imaging of the beating heart. While pacemaker wires are clearly visible on these images, tissue contrast is almost or completely absent, making it impossible to distinguish between tissue and blood or visualize tissue characteristics such as myocardial infarction.

[0011] During pacemaker implantation, an experienced physician navigates based on electrical signals measured by an electrocardiogram (ECG) that are altered by the (active) pacemaker wire within the heart. Not all physicians are capable of doing this, and therefore each physician follows their own strategy, resulting in a great deal of variation and making it impossible to standardize and define guidelines.

[0012] Previous literature (Conduction System Pacing for Cardiac Resynchronisation, Arrhythm Electrophysiol Rev. April 2021, 10(1):51-58) suggested that the precise positioning of electrodes in the septum was not important. However, initial practical attempts have shown that in some patients, tissue penetration is impossible, while in others, the septum, located at the electrode tip, is easily penetrated by the pacemaker screw that functions to secure the electrode once its correct position is found. This may be due to individual differences in the tissue composition of the septum.

[0013] According to one aspect of the present invention, it is found that providing information about septal tissue during an invasive procedure can eliminate the practical disadvantages of performing a newly proposed procedure.

[0014] However, tissue is generally barely visible on X-ray fluoroscopy, and in the case of diaphragms surrounded by blood and other tissues (such as the left and right ventricular walls), they are practically invisible.

[0015] The international patent application WO2017200385A1, incorporated by reference herein, by the same inventors, describes a method and apparatus for verifying the location of an intervention device. This document discloses a method and apparatus for annotating fluoroscopic images with information such as information about other tissues and organs.

[0016] However, more detailed information appears to be needed to enable the practical implementation of conduction system pacing. The same considerations apply to similar purposes, such as the positioning of leadless pacemakers or the collection of biopsies. [Overview of the project]

[0017] The objective of the present invention is therefore to provide a method for providing annotated radiographic images, which, when performed using or on appropriate instruments and / or techniques, function as a medical device or system, particularly useful for conduction system pacing. Whenever the present invention describes a method, a system for carrying out that method (a system configured or adapted for carrying out that method) is also intended to be part of the present invention.

[0018] In a first embodiment, the present invention proposes a method for annotating X-ray fluoroscopic images, the method being: - A step of providing a 3D anatomical surface model of at least a portion of the human heart, - A step of determining at least one local anatomical surface characteristic of the location on the surface in the aforementioned part, - A step of obtaining a real-time 2D X-ray fluoroscopic image from at least a portion of the person's heart, - A step of determining corresponding landmarks between a 3D anatomical surface model and real-time 2D X-ray fluoroscopy images. Includes, The step of generating an annotated image is: - The process includes annotating a real-time 2D fluoroscopic image with a visible indication representing at least one local anatomical surface feature determined to coincide with the corresponding landmark.

[0019] 3D anatomical surface models are known in the art as models that define tissue surfaces forming the internal or external boundaries for parts of the human body, such as the heart. Therefore, a 3D anatomical surface model of the heart defines, in particular, the internal walls of the atria and ventricles in three-dimensional space, as well as their external walls. The 3D surface model may also be a combined epicardial and endocardial model, i.e., the 3D surface model includes both the internal and external surfaces of the region of interest of the heart, the region preferably includes both the left and right ventricles, and more preferably the entire heart. Landmarks may include anatomical landmarks, preferably including at least landmarks on the heart such as the left or right atrium, left or right ventricle, aorta, coronary sinus, coronary arteries, and / or a combination thereof. In particular, landmarks may include at least one of the right atrium, right ventricle, or left ventricle.

[0020] A well-known and appropriate example of an X-ray image is the so-called radiograph, while X-ray fluoroscopy is the study of moving body structures, which can be viewed as an X-ray movie. A continuous X-ray beam passes through the body part being examined. The beam is detected by a photosensitive plate, and the information is transmitted to a monitor such as a television, allowing for a detailed view of the body part and its movement. Therefore, the X-ray image according to the present invention is a real-time 2D X-ray image, or more specifically, a (continuous) real-time 2D X-ray movie. This technology enables the present invention to provide composite images without delay, and therefore in real time. In a further embodiment, it is possible to create an angiogram by adding a contrast agent to blood, thereby allowing the movement of (blood) to be tracked.

[0021] The landmarks mentioned may be anatomical landmarks identifiable on X-ray images, such as blood vessels (or parts thereof), valves (or parts thereof), heart chambers (or parts thereof), and ribs and spine (or parts thereof), or they may be external landmarks, such as markers placed outside the human body (e.g., objects recognizable on a chest X-ray). The marker may also be a pacemaker wire (pre-placed) in a person's vein, thereby indicating a known location relative to the heart.

[0022] An annotated image according to the present invention may also be referred to in the art as a composite image or an image with an additional visible layer. The at least one local anatomical surface property of the portion of the human heart means that this property is known for at least one point on the surface of the portion of the heart, preferably for multiple surface points, and more preferably for all points constituting the surface of the portion of the heart. In this specification, at least a portion of the heart may include the entire heart.

[0023] The step of providing a 3D anatomical surface model of at least a portion of the human heart may be performed prior to obtaining a fluoroscopic image, especially if this is based on measurements. As described in the aforementioned WO2017200385A1, this can be done, for example, by a CT scan or MRI. These types of measurements typically require the placement of a person inside the measuring device and for that reason alone are already unsuitable to be performed during invasive procedures. The method according to the present invention is intended to be applied as technical support during invasive procedures such as the application of a pacemaker or its electrodes, or the taking of a biopsy, when performed on or using a suitable device. This means that the annotated image according to the present invention may also be displayed on a suitable display means such as augmented reality or virtual reality means such as a monitor screen or headset. The annotation may be drawn on an individually displayed X-ray (fluoroscopic) video provided to the physician, which may be a snapshot of the X-ray (fluoroscopic) image.

[0024] Instead, the step of providing a 3D anatomical surface model can be performed by calculations based on X-ray images or markers (anatomical) therein. This can be done, for example, by computer-aided modeling or by selection of a 3D surface model from a database based on similarities between the actual X-ray image and a stored X-ray image from a previous scan or on personal data of the person being scanned such as size, weight, age, gender. The heart tissue may not be visible within the X-ray image, but the outer shape of the heart can be determined based on the shape of the lungs. Using the outer shape and its location, a 3D model of the heart can be constructed.

[0025] In either case, the 3D surface model can be computer-animated as an additional method step to move in accordance with the person's beating and thus moving heart so as to be visible within the X-ray image. Alternatively, the 3D surface model can be recalculated or re-determined in real time to reflect the movement of the beating heart.

[0026] Generally, according to the present invention, the step of providing a 3D anatomical surface model includes the step of calculating the model based on data obtained from a CT, MRI or ultrasound scan of the person's heart or a combination of at least two of these techniques, this step providing more detailed information about the myocardium such as myocardial thickness or providing a model generated or selected from a database based on information about the person's heart obtained from fluoroscopy or patient information. This may further include the step of repeatedly determining corresponding landmarks between the anatomical surface model and the fluoroscopy image and the step of repeatedly generating an annotated image (annotating at least a part of the 3D anatomical surface model with the fluoroscopy image so as to obtain a moving annotated image where the corresponding landmarks coincide).

[0027] The local anatomical surface characteristics to be annotated on the X-ray image can be selected according to the medical treatment performed on the person. Possible annotations are - Surface materials such as normal tissue, scar tissue, muscle tissue or histiocytes, conductive tissues such as left and right bundle branch tissues, - Surface materials such as granulomas or (localized) lesions, - The direction in which the surface spreads locally, or a direction related to this direction, such as a direction perpendicular to the direction in which the surface spreads locally, or a direction in which an intervention device can be positioned, - The boundary of the tissue forming the surface, - The thickness of the tissue forming the surface, - The cross-section of the tissue forming the surface in a plane intersecting the human heart, - The electrical activity, blood flow, or vascular density of the tissue forming the surface are.

[0028] Surface materials such as normal tissue, scar tissue or muscle tissue are carried out on tissues determined through either measurement of local electrical activity of the tissue or as a result of local stimulation of the tissue whose effect is measured through another measuring device, and can be determined by medical image data, artificial intelligence, and / or post-processing of local measurements. Post-processing of medical image data includes, but is not limited to, segmentation of anatomical structures such as ventricles and heart tissues, and evaluation of local tissue characteristics in the segmented heart tissues and parameters derived from the segmentation of the ventricles of the heart tissues. Tissue characteristics are useful for distinguishing healthy tissue from diseased tissue, thereby enabling treatment or biopsy to be targeted to healthy and suspicious tissues respectively.

[0029] The direction in which the surface locally spreads, or directions related to that direction (e.g., directions perpendicular to the direction in which the surface locally spreads), can be determined by creating a 3D surface model based on segmentation of anatomical structures such as ventricles and cardiac tissue. Directions perpendicular to the surface are based on the local spread of the 3D surface model. Secondary directions related to treatment can be determined based on the direction of the surface or directions perpendicular to the surface and other surface properties. Directional properties are of interest in the alignment of the intervention device with the local direction in order to achieve the highest probability of treatment success and reduce the risk of dangerous situations such as insertion of the intervention device parallel to the tissue or tissue perforation.

[0030] This could apply, for example, to a pacemaker with bipolar leads positioned within the diaphragm of a human heart. Such leads comprise a tip electrode and a ring electrode, with a fixed distance between them, for example, 1 centimeter. The bipolar electrodes should be positioned so that the tip electrode and the ring electrode contact the left and right bundle branches of the heart, respectively. Since these may be at a shorter distance from each other than the distance between the electrodes, the only way to ensure contact between both electrodes is to position the pacemaker at a certain angle to the tissue surface of the diaphragm. Such an angle can be calculated based on the ratio of the distance between the electrodes to the thickness of the diaphragm and then annotated as a guide.

[0031] The thickness of the tissue forming the surface can be determined by the local distance between the two surfaces that make up the inner and outer surfaces of the tissue of interest. Similar to tissue properties, local tissue thickness can be used to distinguish between healthy and diseased tissue, thereby allowing for targeted treatment or biopsy of healthy and suspected tissue, respectively. Furthermore, tissue thickness can assist in calculating the optimal orientation of intervention devices.

[0032] The cross-section of tissue forming the surface in a plane intersecting the human heart can be determined by defining the orientation in the organ or tissue and defining a plane perpendicular to the defined organ and tissue axes, thereby forming a line or region where the intersection between the plane and the organ or tissue segmentation is located in the orientation related to the planned intervention. The cross-section provides insight into the depth of the intervention in the organ or tissue from which the cross-section is created.

[0033] Electrical activity can be measured by local measurements performed on the tissue surface, which are determined either through the measurement of local electrical activity in the tissue or through the results of local stimulation of the tissue surface measured via another measuring device that measures the effect that the effect has on the electrocardiogram of the body surface. This method allows for the evaluation of the location of intervention devices on different aspects of the conduction system, such as the bundle of His, left bundle branch, right bundle branch, or the normal myocardial layer.

[0034] Blood flow or perfusion can be measured by perfusion imaging using contrast-enhanced imaging techniques, either nuclear labeling techniques or tissue perfusion techniques. Blood flow or perfusion provides insight into tissue viability, distinguishing between healthy and diseased tissue, thereby allowing treatment or biopsy to be targeted to healthy and suspected tissues, respectively.

[0035] The location, size, and density of blood vessels can be measured by angiography imaging techniques. Annotations regarding the location, size, and density of blood vessels help prevent interventions from being performed in areas where the risk of vascular perforation and subsequent hematoma formation is very high.

[0036] Visible indications used as annotations may be color indications, mark indications such as dots or lines or rulers or arrows, or legible indications such as letters. Obviously, combinations of the aforementioned types of annotations are also possible. In this specification, it is possible to show different types of annotations, such as indicating a cross section with direction using colored lines and arrows. Furthermore, according to embodiments of the present invention, the surface of the 3D anatomical surface model itself can also be annotated. This provides an opportunity to combine annotations and to provide a 3D surface model having, for example, indications in a direction perpendicular to the surface, a surface with its corresponding thickness, or an indication of the location of muscle tissue or scar tissue. In some invasive procedures, precise direction and depth are relevant in addition to precise location, and the combination of a 3D surface model and annotations provides a solution for this need. A person performing an invasive procedure can use the 3D surface model and annotations as a guide and make decisions based on the annotated X-ray images.

[0037] In a more advanced embodiment, the method according to the present invention includes the step of providing values, for example, values ​​in a lookup table or database, that indicate the suitability of anatomical surface characteristics for one or more specific invasive procedures, such as positioning pacemaker electrodes or taking biopsies. The suitability of the local anatomical surface for the one or more specific invasive procedures may then be determined based on the values, and the step of generating an annotated image includes the step of annotating indications that represent suitability.

[0038] For example, invasive procedures that should avoid scar tissue or muscle tissue may be indicated in red, while appropriate tissues may be marked in green. The same may apply to the direction of the surface being approached during an invasive procedure. The correct direction or position of the intervention device, or the direction or position of the intervention device within at least a certain threshold from the intended direction or position, may be indicated by a different marking or color than the incorrect direction. In this case, the annotation provides (preferably in real time) feedback on the positioning of the treatment device. Further consideration may be given to using non-visible annotations, such as sound, when the appropriate location and direction are used, or conversely, when the location and direction are not considered appropriate.

[0039] In yet another embodiment, the method according to the present invention includes the step of obtaining further data of a human heart, such as data from a previous invasive procedure, such as the position or success of a surface procedure, measurement data such as an electrocardiogram, and / or measurement data such as angiography, particularly coronary angiography, and the step of generating an annotated image includes the step of annotating visible indications representing the further measurement data. The electrocardiogram can provide information about the conductivity of cardiac tissues, including muscle tissue, and can provide information about a heart attack that occurred and its consequences, for example. The angiography can provide information about the vascularization and viability of the tissues.

[0040] In a further embodiment, the method according to the present invention includes the steps of: providing a value (e.g., a value in a lookup table or database) representing the suitability of anatomical surface characteristics combined in consideration of the further measurement data for one or more specific invasive procedures, such as positioning pacemaker electrodes or taking biopsies; and determining the suitability of a local anatomical surface based on the value, in consideration of the further measurement data for the one or more specific invasive procedures, wherein the step of generating an annotated image includes providing annotations representing the suitability.

[0041] In one embodiment, the present invention relates to a method adapted to provide an annotated image suitable for conduction system pacing. In this specification, a 3D anatomical surface model of a human heart comprises at least a diaphragm between the left and right ventricles, the step of providing the 3D anatomical surface model includes the step of determining the surface properties of the diaphragm, and the step of generating a composite image includes the step of annotating an indication of the surface properties of the diaphragm to the fluoroscopic image.

[0042] Located in the lower central part of the heart between the left and right ventricles and surrounded by cardiac muscle tissue, the diaphragm is not visible on cardiac X-ray images. Without indication of its location, positioning and placing electrodes is difficult, and pacemaker electrode placement involves much trial and error. Furthermore, trial and error, including attempts to position the electrodes correctly relative to the diaphragm and then secure them by screwing, can leave damaged tissue at the site of unsuccessful attempts, and in worse cases, can result in perforated diaphragm.

[0043] For that reason, according to the present invention, the surface properties that can be annotated are, - Cross-section of the tissue forming the surface of at least a portion of the left ventricle, including the diaphragm. - Thickness of the tissue forming the septum surface, - Boundary of tissues that form a septum, - The direction in which the diaphragm surface spreads locally, or a direction related to that direction (for example, a direction perpendicular to the direction in which the surface spreads locally), - Likelihood estimates for location (e.g., hiss, LBB or LBB area, artery, vein) It could be one of the groups.

[0044] This information allows the person performing the invasive procedure to position the pacemaker electrode for conduction pacing within the patient's diaphragm, to position it directly in the correct location, and to apply it in the correct direction. The electrode may also be guided by a guide tool, which brings the electrode into a certain position and, additionally, allows for control of its orientation before screwing it in. As a result, with the assistance of the method according to the present invention, the application of conduction pacing techniques becomes easier and more reliable, and the likelihood of unsuccessful attempts and damage to cardiac tissue is significantly reduced.

[0045] The present invention is also generally related to apparatus or systems configured to carry out at least some of the method steps described above.

[0046] More specifically, the present invention relates to an apparatus or system for providing a 3D anatomical surface model of at least a portion of a human heart, comprising a data processor, wherein the data processor - To determine at least one local anatomical surface characteristic for a location on the surface in the aforementioned part, - Receiving or acquiring an X-ray fluoroscopic image from at least a portion of the person's heart, - Determining corresponding landmarks between a 3D anatomical surface model and X-ray fluoroscopic images. - Annotating the fluoroscopic image with a visible indication representing at least one local anatomical surface feature determined to coincide with the corresponding landmark. It is configured for this purpose.

[0047] The present invention will be revealed in more detail with reference to the following drawings. [Brief explanation of the drawing]

[0048] [Figure 1] This is a schematic diagram of the heart of a person with a pacemaker wire. [Figure 2] This is a cross-sectional view of the left and right ventricles. [Figure 3] This is a conventional X-ray image of the heart of a person with a pacemaker wire. [Figure 4] This is an annotated X-ray image according to the present invention. [Figure 5] This is an annotated X-ray image according to the present invention. [Figure 6A-6B] This is an annotated X-ray image according to the present invention. [Modes for carrying out the invention]

[0049] Figure 1 shows a perspective cross-sectional view 1 of a human heart 2 with a pacemaker electrode 4 guided by a wire 3. The electrode is screwed into the diaphragm 5, which is the tissue between the left ventricle 6 and the right ventricle 7. As shown in detail A, the electrode 4 is positioned where the so-called HIS bundle 8, coming from the AV node 9, branches into the left bundle branch 8A and the right bundle branch 8B, and is connected to both. This may be a deliberate placement of the electrode for conduction system pacing, in case there is reason to think that both the left and right bundle branches should be paced.

[0050] Figure 2 shows a second cross-sectional view of the heart 2 from Figure 1, taken along line BB', at various positions of the electrode 4 and its wire 4 in the diaphragm 5. From top to bottom of the figure, we can see attempt V where the electrode is too close to the vein, attempt W where the electrode is not screwed deep enough into the diaphragm, attempt X where the electrode extends through the diaphragm, the correct placement Y, and attempt Z where the electrode is positioned at an incorrect angle.

[0051] Figure 3 shows a standard X-ray image of a human heart using conventional technology. In this image, the pacemaker wire 3 and electrode 4 are identifiable, but the cardiac tissue is not visible. This image clearly shows that the information necessary to correctly position the electrodes, as shown in Figure 2, cannot be obtained from the X-ray image alone.

[0052] Figure 4 shows an annotated X-ray image according to the present invention, in which the 3D surface of different parts of the heart is depicted. The first annotation is formed using different colors for different parts of the human heart. In addition, additional annotations, indicated by self-explanatory text within the image, are added to the X-ray image.

[0053] Figure 5 shows an annotated X-ray image of the process of placing pacemaker electrodes. In this case, the left ventricular epicardium, showing the inner and outer surfaces of the left ventricle in a selected cross-section of the heart, is depicted. In other words, the wall of tissue that forms the left ventricle is located between these two rings. The diaphragm forms part of this wall, and as explained in Figure 2, the electrodes should be positioned in a specific position and orientation relative to the diaphragm. The annotated left ventricular epicardium allows the cardiologist to see this position when placing the electrodes in place, enabling them to perform the procedure correctly from the first time.

[0054] Figures 6A and 6B show a setup 11 for carrying out the method according to the present invention. A human heart 2 is irradiated using an X-ray tube 12, and its projection image 13 is analyzed. The projection image also includes annotations formed by a projection image 14' of a vector 14 indicating the desired orientation of the electrode 4 for screwing the electrode 4 into the diaphragm 5. Alignment in the image plane can be confirmed in the projection image. The difference in projection length between the vector 14' and the projection image 4' of the electrode is a measure of the difference in orientation in three dimensions (a dimension perpendicular to the two-dimensional plane in Figures 6A and 6B). The agreement in projection length between the vector and the electrode shown in Figure 6B is a measure of the corresponding three-dimensional orientation.

[0055] This is used, for example, when a pacemaker with bipolar leads is positioned within the diaphragm of a human heart. Such leads comprise a tip electrode and a ring electrode, with a fixed distance between them, for example, 1 centimeter. The bipolar electrodes should be positioned so that the tip electrode and the ring electrode contact the left and right bundle branches of the heart, respectively. Since these may be at a shorter distance from each other than the distance between the electrodes, the only way to ensure contact between both electrodes is to position the pacemaker at a certain angle to the tissue surface of the diaphragm. Such an angle can be calculated based on the ratio of the distance between the electrodes to the thickness of the diaphragm and then annotated as a guide.

[0056] When applying this method, additional information, such as data from an ECG, may be used to determine whether the problem in the target organization has actually been reached.

[0057] The present invention relates to the methods described in the following clauses 1 to 15, apparatus and / or systems that function to carry out or assist in carrying out such methods, and apparatus according to clause 16.

[0058] Clause 1. A method and system for annotating X-ray fluoroscopic images, - A step of providing a 3D anatomical surface model of at least a portion of the human heart, - A step of determining at least one local anatomical surface characteristic for a location on the surface in the aforementioned part, - The step of obtaining an X-ray fluoroscopic image from at least a portion of the person's heart, - A step of determining the corresponding landmarks in the 3D anatomical surface model and the fluoroscopic image. Methods and systems including, - The step of generating an annotated image is A method and system comprising the step of annotating a fluoroscopic image with a visible indication representing at least one local anatomical surface feature determined to coincide with the corresponding landmark. 2. Local anatomical surface characteristics are as follows: - Surface materials such as normal tissue, scar tissue, muscle tissue or hist, and conductive tissues such as left and right fascicle tissues. - Surface material such as granuloma or (localized) lesions, - The direction in which the surface spreads locally, or a direction related to this direction (for example, a direction perpendicular to the direction in which the surface spreads locally), or the direction in which the intervention device can be positioned. - Boundaries of tissues that form the surface, - Thickness of the tissue forming the surface, - Cross-section of tissue forming the surface in a plane intersecting with the human heart, - Electrical activity, blood flow, or vascular density of the tissues forming the surface, - Likelihood estimators for location (e.g., hiss, LBB or LBB area, artery, vein), The method described in Clause 1, selected from the options provided. 3. The method according to Clause 1 or 2, wherein the visible indication is a color indication, a mark indication such as a dot or line or a ruler or arrow, or a legible indication such as letters. 4. The method according to any one of clauses 1 to 3, further comprising the step of annotating an X-ray fluoroscopic image using a surface from the 3D anatomical surface model itself. 5. The method is - A step of providing values ​​in, for example, a lookup table or database that indicate the suitability of anatomical surface characteristics for one or more specific invasive procedures, such as positioning pacemaker electrodes or taking biopsies, - A step of determining the suitability of the local anatomical surface for one or more specific invasive procedures based on the aforementioned values. Includes, - The method described in any one of Clauses 1 to 4, wherein the step of generating an annotated image includes the step of annotating indications that represent appropriateness. 6. - Further data on the human heart, • Data from previous invasive procedures, such as their location or success rate on the surface. • Measurement data such as electrocardiograms, and / or • Measurement data from angiography, especially coronary angiography. This includes steps to obtain further data on the human heart, - The step of generating an annotated image is • The step of annotating the visible indications representing the aforementioned further measurement data. The method described in any one of the clauses 1 to 5, including the method described in any one of the clauses 1 to 5. 7. - A step of providing, for example, a value in a lookup table or database, wherein the value represents the suitability of anatomical surface characteristics combined with the further measurement data for one or more specific invasive procedures, such as positioning pacemaker electrodes or taking biopsies. - A step of determining the suitability of the local anatomical surface based on the aforementioned values, taking into account the further measurement data for one or more specific invasive procedures. Includes, - The step of generating an annotated image includes the step of providing annotations that indicate appropriateness, The method described in Article 5 or 6. 8.3 The step of providing a 3D anatomical surface model is - A step of calculating the model based on data obtained from a CT, MRI, or ultrasound scan of the person’s heart, or data obtained from a combination of at least two of these techniques. - A step of providing a model that is generated or selected from a database based on patient information or information about a person's heart derived from X-ray fluoroscopy images, - A step of providing a model that is generated or selected from a database based on patient information or information about a person's heart obtained from X-ray fluoroscopy imaging, wherein the model is fine-tuned by information from one or more imaging modalities, i.e., CT, MRI, ultrasound, and / or other measurements, i.e., pressure, electrography. The method described in any one of the clauses 1 to 7, including the method described in any one of the clauses 1 to 7. 9. - A step of repeatedly determining the corresponding landmarks in the anatomical surface model and fluoroscopic images, - A step of repeatedly generating annotated images, wherein at least a portion of a 3D anatomical surface model is annotated in a fluoroscopic image, thereby matching the corresponding landmarks. - A step of repeatedly determining the criteria within the X-ray fluoroscopy image, - A step of synchronizing the cardiac cycle in X-ray fluoroscopy with the deformation of a 3D anatomical surface model, - To obtain annotated images with movement, the 3D anatomical surface model is synchronized with respiration-induced motion by determining reference markers within the fluoroscopic image and subtracting the movement of the heart. The method described in any one of the clauses 1 to 8, including the method described in any one of the clauses 1 to 8. 10. - A 3D anatomical surface model of the human heart includes at least a diaphragm between the left and right ventricles. - The step of providing a 3D anatomical surface model includes the step of determining the surface properties of the diaphragm, The method according to any one of the claims 1 to 9, wherein the step of generating a composite image includes the step of annotating the indication of the surface properties of the diaphragm in the X-ray fluoroscopic image. 11. Surface properties are, - Cross-section of the tissue forming the surface of at least a portion of the left ventricle, including the diaphragm. - Thickness of the tissue forming the septum surface, - Boundary of tissues that form a septum, - Directions related to that direction, such as the direction in which the diaphragm surface expands locally, or the direction perpendicular to the direction in which the surface expands locally. The method according to clause 10, wherein the method is at least one of the group. 12. The method according to any one of the claims 1 to 11, wherein the 3D surface model is a combined epicardial and endocardial model, i.e., the 3D surface model comprises both the internal and external surfaces of a region of interest of the heart, the region preferably comprising both the left ventricle and the right ventricle, and more preferably comprising a complete heart. 13. The method according to Clause 12, wherein the landmarks include anatomical landmarks, preferably including at least landmarks on or around the heart, such as the left or right atrium, left or right ventricle, aorta, coronary sinus, coronary arteries, and / or a combination thereof, and in particular the landmarks may include at least one of the right atrium, right ventricle, or left ventricle. 14. The method according to any one of the clauses 1 to 13, wherein the annotation comprises an angle with respect to the tissue surface of the diaphragm for positioning a bipolar lead comprising an upper electrode and a ring electrode, the angle being calculated such that the tip electrode and ring electrode of the lead ultimately contact the left and right bundles, respectively. 15. The method according to any one of the clauses 1 to 14, comprising the step of annotating the correct orientation or position of the intervention device, or the orientation or position of the intervention device within at least a certain threshold from the intended orientation or position, the annotating step of particularly indicating the correct orientation or position using a different marking or color from the incorrect orientation or position. 16. Apparatus for providing a 3D anatomical surface model of at least a portion of a human heart, comprising a data processor, the data processor is - To determine at least one local anatomical surface characteristic for the location on the surface in the aforementioned part, - Receiving or acquiring an X-ray fluoroscopic image from at least a portion of the person's heart, - Determining corresponding landmarks in 3D anatomical surface models and fluoroscopic images. In a device configured for this purpose, the data processor is The fluoroscopic image is annotated using a visible indication representing at least one local anatomical surface feature determined to coincide with the corresponding landmark. A device characterized by being configured for a specific purpose.

Claims

1. It is a system for annotating X-ray fluoroscopic images. - A step of providing a 3D anatomical surface model of at least a portion of the human heart. - A step of determining at least one local anatomical surface characteristic of the location on the surface in the aforementioned part, - A step of obtaining a real-time 2D X-ray fluoroscopic image from at least a portion of the person's heart, - A step of determining corresponding landmarks between the 3D anatomical surface model and the real-time 2D X-ray fluoroscopy image. In a system configured to carry out the following: - The step of generating annotated images is The step of annotating the real-time 2D fluoroscopic image using a visible indication representing the determined at least one local anatomical surface characteristic such that the corresponding landmarks match. A system characterized by including

2. The system according to claim 1, wherein the local anatomical surface characteristics are selected from the following list: - Surface materials such as normal tissue, scar tissue, muscle tissue or histocyte, left and right fasciculus tissues, - Surface materials such as granulomas or (localized) lesions, - A direction in which the surface spreads locally, or a direction related to that direction, for example, a direction perpendicular to the direction in which the surface spreads locally, or a direction in which the intervention device can be positioned. - Boundaries of tissues that form the surface, - Thickness of the tissue forming the surface, - Cross-section of the tissue forming the surface in a plane intersecting with the human heart, - Electrical activity, blood flow, or vascular density of the surface-forming tissues, - Likelihood estimators for location (e.g., hiss, LBB or LBB region, artery, vein).

3. The system according to claim 1 or 2, wherein the visible indication is a color indication, a mark indication such as a dot or line or ruler or arrow, or a readable indication such as letters.

4. The system according to any one of claims 1 to 3, further comprising the step of annotating the real-time 2D X-ray fluoroscopic image using the surface of the 3D anatomical surface model itself.

5. - A step of providing values ​​(e.g., a lookup table or database) indicating the suitability of anatomical surface characteristics for one or more specific invasive procedures, such as positioning pacemaker electrodes or taking biopsies. - A step of determining the suitability of the local anatomical surface for one or more specific invasive procedures based on the aforementioned values. Includes, - The step of generating the annotated image includes the step of adding annotations indicating appropriateness. The system according to any one of claims 1 to 4.

6. - Includes the step of obtaining further data on the person’s heart, the further data on the person’s heart is • Data from previous invasive procedures, such as surface location or success rate. - Measurement data such as electrocardiograms, and / or • Measurement data from angiography, especially coronary angiography. This includes, - The step of generating the annotated image is - The step of annotating the aforementioned further measurement data with a visible indication, The system according to any one of claims 1 to 5.

7. - A step of providing a value (e.g., a lookup table or database) that indicates the suitability of anatomical surface characteristics combined in consideration of the further measurement data for one or more specific invasive procedures, such as positioning pacemaker electrodes or taking biopsies. - A step of determining the suitability of the local anatomical surface based on the above value, taking into account the further measurement data for one or more specific invasive procedures. Includes, - The step of generating the annotated image includes the step of adding annotations indicating appropriateness. The system according to claim 5 or 6.

8. The step of providing the 3D anatomical surface model is - A step of calculating the 3D anatomical surface model based on data obtained from a CT, MRI, or ultrasound scan of the person's heart, or from a combination of at least two of these techniques. - A step of providing a model generated or selected from a database based on information about the person's heart or patient information obtained from the real-time 2D X-ray fluoroscopy imaging, - A step of providing a model generated or selected from a database based on information about the person's heart or patient information obtained from the real-time 2D X-ray fluoroscopy imaging, wherein the model is fine-tuned by information from one or more imaging modalities, i.e., CT, MRI, ultrasound and / or other measurements, i.e., pressure, electrography. The system according to any one of claims 1 to 7, including the system described in any one of claims 1 to 7.

9. To obtain an animated annotated image, - A step of repeatedly determining corresponding landmarks between the 3D anatomical surface model and the real-time 2D X-ray fluoroscopy image, - A step of repeatedly generating annotated images, wherein the real-time 2D X-ray fluoroscopy image is annotated with at least a portion of the 3D anatomical surface model such that the corresponding landmarks match. - A step of repeatedly determining a reference (fiducial) in the real-time 2D X-ray fluoroscopy image, - A step of synchronizing the cardiac cycle in the real-time 2D X-ray fluoroscopy with the deformation of the 3D anatomical surface model, - A step of synchronizing the 3D anatomical surface model with breathing-induced motion by determining a fiducial marker in the real-time 2D X-ray fluoroscopy image and subtracting the movement of the heart. The system according to any one of claims 1 to 8, including

10. - A 3D anatomical surface model of the human heart includes at least a diaphragm between the left and right ventricles. - The step of providing the 3D anatomical surface model includes the step of determining the surface properties of the diaphragm, - The step of generating a composite image includes the step of annotating the real-time 2D X-ray fluoroscopy image with indication of the surface properties of the diaphragm. The system according to any one of claims 1 to 9.

11. The aforementioned surface characteristics are - Cross-section of the tissue forming the surface of at least a portion of the left ventricle, including the diaphragm. - Thickness of the tissue forming the septum surface, - Boundary of tissues that form a septum, - The direction in which the diaphragm surface expands locally, or a direction related to that direction, for example, a direction perpendicular to the direction in which the surface expands locally. The system according to claim 10, wherein the system is at least one of the group.

12. The system according to any one of claims 1 to 11, wherein the 3D surface model is combined with an epicardial and endocardial model, that is, the 3D surface model comprises both the internal and external surfaces of a region of interest of the heart, and this region of interest preferably comprises both the left ventricle and the right ventricle, and more preferably the entire heart.

13. The system according to claim 12, wherein the landmarks include anatomical landmarks, preferably the anatomical landmarks include at least supercardiac or pericardiac landmarks such as the left atrium or right atrium, left ventricle or right ventricle, aorta, coronary sinus, coronary arteries and / or a combination thereof, and in particular the landmarks may include at least one of the right atrium, right ventricle or left ventricle.

14. The system according to any one of claims 1 to 13, wherein, in order to position a bipolar lead comprising a tip electrode and a ring electrode, the annotation has an angle with respect to the surface of the diaphragm tissue, the angle is calculated such that the tip electrode and the ring electrode of the lead ultimately contact the left and right bundle branches, respectively.

15. The system according to any one of claims 1 to 14, comprising the step of annotating the direction or position of an intervention device to a direction or position of an intervention device within at least a certain threshold from the correct direction or position or intended direction or position, the annotating step of indicating the correct direction or position using a different marking or color from the incorrect direction or position.

16. A system according to any one of claims 1 to 15, comprising a data processor, wherein the data processor is - To determine at least one local anatomical surface characteristic for a location on the surface in the aforementioned part, - Receiving or acquiring real-time 2D X-ray fluoroscopic images from at least a portion of the person's heart, - To determine the corresponding landmarks between the 3D anatomical surface model and the real-time 2D X-ray fluoroscopy image. It is configured for, The aforementioned data processor, - Annotating the real-time 2D fluoroscopic image with a visible indication representing at least one local anatomical surface characteristic determined to coincide with the corresponding landmark. The system according to any one of claims 1 to 15, characterized by being configured for the purpose of

Images