Respiration-based method for tracking cardiac anatomy, computer-implemented method for training a neural network to semantically segment real-time MRI recordings, data processing system, computer program product, and computer-readable storage medium
The method addresses the limitations of existing cardiac MRI techniques by using real-time MRI and neural network segmentation to track cardiac anatomy under free breathing, simplifying the process and improving accuracy in cardiac function assessment.
Patent Information
- Application Number
- PCT/EP2024/064524
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-11-14
- Filing Date
- 2024-05-27
- Publication Date
- 2024-12-05
AI Technical Summary
Current cardiac MRI methods, such as cine-MRI, are susceptible to motion artifacts due to long recording times and require controlled breathing, limiting their ability to accurately assess cardiac function and structure, especially in pathological cases, and existing software solutions cannot efficiently evaluate real-time MRI images under free breathing conditions.
A computer-implemented method for respiration-dependent cardiac anatomy tracking using real-time MRI images, which segments voxels into myocardium, ventricles, and lung sections using a trained neural network, allowing for automatic recognition and synchronization of cardiac and respiratory cycles without external biosensors, enabling three-dimensional representation of heart anatomy at defined physiological times.
This method simplifies the MRI imaging process, reduces patient stress, and allows for accurate evaluation of cardiac function and structure under free breathing conditions, providing robust and error-free interpretation of cardiac and respiratory physiology, and can evaluate both healthy and pathological hearts.
Smart Images

Figure EP2024064524_05122024_PF_FP_ABST
Abstract
Description
[0001] Respiration-dependent method for cardiac anatomy tracking and computer-implemented method for training a neural network for semantic segmentation of real-time MRI images, data processing system, computer program product and computer-readable storage medium
[0002] Cardiac magnetic resonance imaging (CMR), also known as cardiovascular MRI, is a state-of-the-art technique. This is a magnetic resonance imaging (MRI) technique used for the noninvasive assessment of the function and structure of the cardiovascular system.
[0003] Especially for the functional examination of the cardiovascular system, static images are insufficient. For this reason, various methods are used to capture temporal sequences of cardiac movement. The most widely used method is by far the so-called cine MRI.
[0004] In cine MRI, an electrocardiogram (ECG) or finger pulse measurement is recorded on the patient during the MRI scan to trigger the acquisition of MRI images at specific times during the cardiac cycle. This is usually done by taking a single image per cardiac cycle. Using the ECG to trigger image acquisition ensures that the images are acquired at consistent times during the cardiac cycle, allowing for an accurate assessment of cardiac function. This is particularly important for the diagnosis and treatment of cardiovascular diseases, where accurate measurements of cardiac function are essential as a basis for treatment decisions. Due to the fact that the resulting image quality of cine MRI is highly susceptible to motion artifacts, the patient's breathing must be controlled and deliberately stopped during the acquisition of individual images.From the images recorded in successive cardiac cycles, a film-like sequence can then be calculated or compiled, giving the impression of a real-time recording.
[0005] Real-time MRI is a new method for functional cardiac imaging, developed by Prof. Frahm at the Max Planck Institute for Multidisciplinary Sciences. Real-time MRI technology enables video recording at frame rates of over 30 frames per second thanks to significantly shorter acquisition times. This is made possible, among other things, by newly developed mathematical methods for image reconstruction. This enables truly real-time recordings for the first time, making movements and processes inside the body, such as the beating of the heart, visible for the first time. In contrast to cine MRI technology, this allows arrhythmias and cardiac function to be examined under free breathing. Due to the novelty of this technology, there are currently only a few processing techniques that allow the efficient evaluation of the resulting data.Although it is already possible to fully automatically evaluate cine MRI images using neural networks using commercial software, the processes and methods used have the disadvantage that they are not applicable to cardiac MRI under free breathing.
[0006] Cardiac MRI is recommended for the diagnosis of various conditions in relevant guidelines of the European Society of Cardiology (ESC) and the American Heart Association (AHA), including the ESC guideline "Adults with Congenital Heart Defects" (ACHD). It is used as a reference standard for assessing cardiac structure and function and is valuable for surgical planning in complex congenital heart diseases. In Germany, there are over 50 facilities certified as cardiac MRI centers by the German Society of Cardiology (DGK), and 281 cardiologists hold the DGK's additional qualification in cardiac MRI. In particular, the current state-of-the-art procedures have the following disadvantages:
[0007] CINE MRI: In CINE MRI, the cardiac phase is gated based on the R-wave or pulse wave. This means that part of the image is captured during multiple heartbeats, creating an artificial cardiac cycle. While this technique allows for three-dimensional imaging of the heart during a specific respiratory phase, the resulting data does not provide information about the temporal progression of cardiac movement immediately before and after the synchronized time. Furthermore, this imaging technique is highly susceptible to motion artifacts due to the long acquisition times. External biosignals, such as chest straps, ECGs, and navigator data, are typically also used, which significantly increase the complexity and effort of the acquisition.
[0008] External biosignals: In state-of-the-art methods, the respiratory phase and the cardiac phase are usually recorded and determined using external measurement methods.
[0009] There are currently software solutions on the market, both in the scientific context (CaFuR, Fraunhofer) and in the clinical field (cvi42, CircleCVI), that allow for the automatic segmentation of healthy hearts (ventricles, myocardium) in MRI images. CaFuR enables this specifically for real-time MRI, while cvi42 only provides it for cine MRI. Both solutions have in common that the respiratory phase cannot be taken into account, making respiration-dependent stroke volume determination impossible. Furthermore, the analysis is limited to healthy hearts. Pathological cardiac anatomies cannot be evaluated. Current status in clinical / scientific practice: The currently used / proprietary methods for examining / imaging hearts during free breathing involve a high level of additional effort, which affects the recording, processing, and interpretation of multiple signals.
[0010] Based on the aforementioned disadvantages of the prior art, the object of the present invention is to provide an improved computer-implemented method for respiration-dependent cardiac anatomy tracking.
[0011] According to a first aspect, the present invention relates to a computer-implemented method for respiration-dependent cardiac anatomy tracking, the method comprising the steps: a) acquiring a sequence of real-time MRI images of at least one slice of the short axis of the heart or the 4-chamber view of a living being, comprising at least a partial section of the lung, with an acquisition time per slice of at least one respiratory cycle; b) segmenting the voxels of the images from step a) into at least the classes: myocardium, left ventricle, right ventricle, and the partial section of the lung, using a trained computer-implemented neural network for the semantic segmentation of MRI images of the heart, preferably obtained via a method according to claim 15; c) summing the number of all segmented voxels per class and per image from step b);d) Determining the minimum and maximum of the summed number of voxels of each class from step c) of the respective slice across the sequence of images; e) Assigning the minima and maxima from step d) of the class of the partial section of the lung to a physiological state of the respiratory cycle and / or assigning the minima and maxima from step d) of the class of the myocardium and / or the left and / or right ventricle to a physiological state of the cardiac cycle of the living being; f) Selecting one MRI image from step a) per slice and per defined combination of the physiological state of the respiratory cycle and the cardiac cycle by searching for a combination of selected extreme points of the respiratory cycle and the cardiac cycle and defining it as a defined physiological time point for each combination;and g) composing all selected images from step f) per defined physiological time point of all slices from step a) into a three-dimensional representation of the cardiac anatomy at each defined physiological time point.;
[0012] The method according to the invention has the advantage that the phases of the cardiac cycle and the corresponding physiological time points are automatically recognized and assigned as part of the computer-implemented processing, based exclusively on the generated MRI images. This significantly reduces the effort and the necessary knowledge of the personnel performing the process and also reduces the stress of the patients during the generation of the MRI images. According to the invention, the use of external biosensors, for example, for measuring the respiratory cycle, can be dispensed with. The method according to the invention considerably simplifies the process of acquiring MRI images while the patient is breathing freely, since no consideration needs to be given to arrhythmias, breathing behavior, or the recording of external biosignals during the MRI acquisition.The entire evaluation and interpretation of the MRI images, as well as the resulting synchronization and display, can be performed as post-processing steps using the method according to the invention. This is very robust, since no measurement errors (other than the actual MRI measurement) need to be taken into account. In particular, the method also allows only steps b) to g) to be carried out separately and independently of method step a), for example, if corresponding MRI images of a patient are already available.
[0013] The thickness of the acquired slices corresponds to a voxel, and the acquired slices are directly adjacent to one another, with the thickness or size of the voxels corresponding to or defining the distance between the acquired slices. The third dimension of the voxels outside the plane of the heart thus provides virtually no additional information, since the two-dimensional image or the two-dimensional MRI image of the slice is simply constantly extruded. In this regard, it is possible to improve the method by smoothing the discrete information, for example, by interpolating the values of two immediately adjacent voxels, such that the values between the voxels of two adjacent slices are linearly interpolated.
[0014] In the context of this application, the sequence of real-time MRI images is understood to mean a temporally continuous cardiac MRI image, which is a two-dimensional image generated using MRI technology. For the purposes of the present invention, living beings are understood to include both humans and animals that have lungs and a corresponding heart, as well as a recurring respiratory cycle.
[0015] In the context of the present invention, the respiratory cycle is understood to be the continuously repeating process of lung respiration or, more precisely, the external respiration of the patient.
[0016] In this context, the following terms should also be further defined: Inspiration - expansion of the thorax and lungs by respiratory muscles, decreasing pleural pressure, expansion of the lungs, inflowing air into the lungs of the living being,
[0017] Expiration - contraction of the thorax and lungs due to restoring forces and gravity in interaction with accessory respiratory muscles, air flowing out of the lungs of the organism.
[0018] The method steps e) and f) of the invention can also be understood as synchronization if the generated MRI images in the method step f) are assigned to a physiological time point of the respiratory cycle and the cardiac cycle, the images are synchronized in time with the respiratory and cardiac cycle.
[0019] According to the invention, the voxels of the recording from step a) can be segmented into at least the left and / or right ventricle class. However, the segmentation of both left and right ventricles is not absolutely necessary for the synchronization provided according to the invention in step g), since the left and right ventricles contract at the same frequency. Consequently, according to the invention, only the right or left ventricle can be used for synchronization, for example. A combination of the recording and evaluation of the right and left ventricles can be used to improve the accuracy of the synchronization result.
[0020] Preferably, the method according to the invention can additionally comprise the method step after step g): h) attaching the images before and / or after the physiological time point and creating a time-resolved three-dimensional representation of the physiological time point. According to the invention, it can be provided that in step a), slices of the entire cardiac anatomy are acquired successively along the longitudinal axis of the heart.
[0021] The slices can be recorded at a distance of an assumed voxel relative to each other along the longitudinal axis.
[0022] However, larger distances between the layers to be recorded can also be selected, with interpolation between the layers then being carried out.
[0023] According to the invention, it can be provided that in step f) the selection of an MRI image from step a) per slice and per defined combination of the physiological state of the respiratory cycle and the cardiac cycle is carried out by searching for combinations of the smallest distance between the selected extreme points of the respiratory cycle and the cardiac cycle.
[0024] Furthermore, it can preferably be provided that in step a), the sequence of images is recorded at a recording frequency that corresponds to at least twice the heartbeat rate. For example, 30 images can be recorded per second.
[0025] Preferably, it can be provided that in step e) the class of the part of the lung is assigned at a physiological time point of the respiratory cycle and the classes of the myocardium and / or the left and / or right ventricle are assigned at a physiological time point of the cardiac cycle.
[0026] It can be provided that the minimum number of summed voxels of the class of the left or right ventricle of a slice is assigned to the physiological time of end-systole and / or that the maximum number of summed voxels of the class of the left or right ventricle of a slice is assigned to the physiological time of end-diastole.
[0027] Furthermore, it can be provided that the maximum number of summed voxels of the class of the partial section of the lung is assigned to the physiological time of end inspiration and / or the minimum number of summed voxels of the class of the partial section of the lung is assigned to the physiological time of end expiration. By considering the number of voxels of the class of the partial section, it is possible to infer or calculate the states of the respiratory cycle based solely on the image information of the MRI images. Consequently, the invention eliminates the need for external biosignal sensors.
[0028] According to the invention, it can optionally be provided that in method step a) an acquisition time per slice of the real-time MRI of several respiratory cycles is selected.
[0029] It can further preferably be provided that the physiological state of the respiratory cycle and / or the cardiac cycle is selected from at least one element from the group of: end inspiration, end expiration, end systole, end diastole.
[0030] To improve the synchronization in step e), it can be provided that in method step a) at the same time as the recording of the real-time MRI, at least one signal of a further biosignal is additionally recorded via at least one biosignal sensor, and wherein in method step e) the physiological phases are determined with additional use of the at least one further recorded biosignal.
[0031] It can be provided that an electrocardiogram and / or a signal for a respiratory phase is used as an additional biosignal. It can be provided that, in step g), the geometry of the cardiac anatomy is used to calculate the heart's stroke volume. Of course, the three-dimensional image of the cardiac anatomy created in process step g) can also be used to determine other cardiac parameters.
[0032] According to a further aspect, the present invention relates to a computer-implemented method for training a neural network for the semantic segmentation of real-time MRI images of a heart, wherein the neural network has a U-Net architecture and the combination of a binary cross-entropy loss and a Sorensen-Dice coefficient is used as the loss function for training the neural network.
[0033] The computer-implemented method can be implemented in pyTorch.
[0034] Seven convolutions can be used on the encoder and decoder sides. Fixed input dimensions of 512 x 512 pixels can be used, which may require resizing the training data. A batch size of 14,1000 epochs, linear learning rate, momentum 0,000, sgd (stochastic gradient intercept) optimizer, training data (42 subjects), 942 training images, validation images (8 subjects), and 153 validation images can be used to train the computer-implemented method.
[0035] According to a third aspect, the present invention relates to a data processing system comprising a processor configured to execute the steps of the method according to the first aspect of the present invention. Furthermore, a fourth aspect of the present invention relates to a computer program product comprising instructions that, when executed by a computer, cause the computer to execute the steps of the method according to the first aspect of the present invention.
[0036] In addition, a computer-readable storage medium may be provided comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to the first aspect of the invention.
[0037] In other words, the present invention can be represented or described as follows:
[0038] Unlike many other methods, the invention utilizes real-time MRI images. Apart from measuring the short axis of the heart, in which any part of at least one lung is cut in each slice, no further measurements are necessary. However, it is possible to incorporate additional biosignals such as ECG or respiratory phase determination using a chest strap into the calculations. The entire imaging process consists of
[0039] Real-time MRI imaging with an acquisition time of at least one respiratory cycle per slice, ideally more semantic segmentation into the classes: myocardium, at least one ventricle, and the part of the lung depicted in the respective image, of all acquired MRI images using a neural network (e.g. UNet architecture)
[0040] Summing up all segmented pixels per class per image layer by layer Determination of all extreme points of all classes of the respective layer
[0041] Interpretation of the extreme points as physiological phases (e.g., end inspiration as the highest point of the summed pixels of the lungs of a slice, end systole as the lowest point of the summed pixels of the left ventricle of a slice)
[0042] Finding an "optimal" image per slice per combination as physiological phase of respiration and cardiac cycle (e.g. end inspiration + end systole) by searching for the combination of the minimum distance between both detected selections at extreme points
[0043] Compilation of all selected images per combination of physiological phases into a 3D representation of the heart at this defined physiological time point. Optionally appending the images before and after the physiological time point to create a 4D representation of the physiological time point.
[0044] This process differs essentially from all previous solutions by reducing the measurement methods to a single and sole measurement using real-time MRI and subsequent evaluation and interpretation based solely on the images thus generated.
[0045] Advantages are: simple, less error-prone measurement of the entire cardiac and respiratory physiology by only one measuring instrument (real-time MRI) Evaluation and interpretation of the measured images only after complete recording and thus no influence of the measurement by evaluation
[0046] Recording and analysis of pathological hearts is also possible. No adjustment of the analysis to the respiratory or cardiac cycle is necessary. The image created is not just reduced to a predetermined physiological time point, instead the condition can be depicted at any time point. Subsequent adjustment of the analysis is possible. -> The physiological time points can be freely adjusted retrospectively through user interpretation: Selected time points can be changed in each layer for each class. The effects of breathing on the heart can be represented more precisely, either indirectly by adjusting the segmentation or directly by selecting the images for the respective synchronization. By identifying a specific, physiologically clearly defined time point (not just inspiration in general, but specifically the end of inspiration, etc.).
[0047] The following refers to the attached figures to explain the present invention in more detail. Figures 1-4 show individual steps of the method according to the invention, specifically using the example of an MRI scan and the evaluation and display of the heart of a patient with a univentricular heart. Figure 5 schematically illustrates the entire process chain provided by the invention. The following figures show an exemplary embodiment of the method according to the invention for respiration-dependent cardiac anatomy tracking.
[0048] In this example, a Siemens mMR Biograph MRI was used to acquire and create real-time MRI images. The real-time reconstruction computers were assembled by the Max Planck Institute for Multidisciplinary Sciences and include eight graphics cards.
[0049] For semantic segmentation, a PyTorch implementation of a deep neural network in a so-called UNet architecture was used. A combination of binary cross-entropy loss and the Sorensen-Dice coefficient was used as the loss function for optimization. The model was trained with 942 manually segmented images from MRI scans of 42 patients, each as different as possible, over 1000 epochs with a batch size of 14. It should be noted that any other models and methods for semantic segmentation could be used. Furthermore, a completely different method for recognizing the respective anatomies is also conceivable, provided that these (e.g., geometric descriptions) can be used to derive a mapping of pixels to the respective anatomy.
[0050] To select physiological combinations of respiratory phase and cardiac cycle, the extreme points of the summed lung pixels per slice were first determined and classified into high and low points. Subsequently, from all physiological time points of the cardiac phase, those that were closest in time to the respective extreme points of the respiratory phase were filtered out. This was achieved by comparing the temporal intervals between all physiological time points of the cardiac phase and those of the respiratory phase.
[0051] By assigning the combinations found in this way, four 4D representations of the heart (each with two respiratory points, two points in the cardiac cycle) could be created, showing the physiology at the time of the combinations, as well as the entire cardiac cycle connected to it.
[0052] Figure 1 shows a schematic representation of a patient examination using a real-time MRI. As shown in Figure 2, real-time MRI videos or a sequence of individual images of the heart are acquired along the so-called short axis. The videos are generated, for example, at a rate of 30 frames per second in multiple slices of the heart. Figure 2 shows a correspondingly generated MRI cross-sectional view of the short axis of the heart on the right side of the figure.
[0053] As can be seen from Fig. 3, all generated MRI images are segmented fully automatically using a numerical network. The left image in Fig. 3 shows the already classified voxel assignments of an MRI image. This means that each voxel in each MRI image is assigned a class. In a subsequent process step, the voxels segmented in this way are added together for each class in each generated MRI image. This results in a curve for the summed number of voxels per class for each slice across the generated MRI images (see right image in Fig. 3), from which the respiratory phase and cardiac phase can be read. Naturally, the sequence of MRI images for each slice begins with a random combination of respiratory and cardiac phase. By determining the extreme values of the cardiac and respiratory phase in each slice, specific combinations of respiratory and cardiac phase can be detected.
[0054] The information obtained from the combination of respiratory and cardiac phases can be compiled to determine the corresponding cardiac volumes, allowing various vital parameters for diagnosis and therapy to be automatically generated on this basis (see Fig. 4). Fig. 5 shows the schematic overview of the automatic analysis process, from the acquisition of the MRI data to the creation of a three-dimensional representation of the cardiac anatomy and an optional subsequent evaluation.
Claims
PATENT CLAIMS 1. A computer-implemented method for respiration-dependent cardiac anatomy tracking, comprising the steps of: a) acquiring a sequence of real-time MRI images of at least one slice of the short axis of the heart or the 4-chamber view of a living being, comprising at least a partial section of the lung, with an acquisition time per slice of at least one respiratory cycle; b) segmenting the voxels of the images from step a) into at least the classes: myocardium, left ventricle, right ventricle, and the partial section of the lung, using a trained computer-implemented neural network for the semantic segmentation of MRI images of the heart, preferably obtained via a method according to claim 15; c) summing the number of all segmented voxels per class and per image from step b);d) Determining the minimum and maximum of the summed number of voxels of each class from step c) of the respective slice over the sequence of recordings; e) Assigning the minima and maxima from step d) of the class of the partial section of the lung to a physiological state of the respiratory cycle and / or; Assigning the minima and maxima from step d) of the class of the myocardium and / or the left and / or right ventricle to a physiological state of the cardiac cycle of the living being; f) selecting one MRI image from step a) per slice and per defined combination of the physiological state of the respiratory cycle and the cardiac cycle by searching for a combination of selected extreme points of the respiratory cycle and the cardiac cycle and defining each combination as a defined physiological time point; and g) compiling all selected images from step f) per defined physiological time point of all slices from step a) to form a three-dimensional representation of the cardiac anatomy at each defined physiological time point.
2. The method according to claim 1, wherein the method after step g) additionally comprises the step of: h) appending the images before and / or after the physiological time point and creating a time-resolved three-dimensional representation of the physiological time point.
3. Method according to one of the preceding claims, wherein in step a) layers of the entire cardiac anatomy are recorded one after the other along the longitudinal axis of the heart.
4. The method according to claim 3, wherein the layers are directly adjacent to one another or are recorded at a defined distance from one another in the direction of the longitudinal axis of the heart.
5. Method according to one of the preceding claims, wherein in step f) the selection of an MRI image from step a) per slice and per defined combination of the physiological state of the respiratory cycle and the cardiac cycle is carried out by searching for the combination of the smallest distance between the selected extreme points of the respiratory cycle and the cardiac cycle.
6. Method according to one of the preceding claims, wherein in step a) the recording of the sequence of recordings takes place at a recording frequency which corresponds to at least twice the heartbeat frequency.
7. Method according to one of the preceding claims, wherein in step e) the class of the part of the lung is assigned to a physiological time point of the respiratory cycle and the classes of the myocardium and / or the left and / or right ventricle are assigned to a physiological time point of the cardiac cycle.
8. Method according to one of the preceding claims, wherein the minimum of the number of summed voxels of the class of the left and / or right ventricle of a slice is assigned to the physiological time of end-systole and / or wherein the maximum of the number of summed voxels of the class of the left and / or right ventricle of a slice is assigned to the physiological time of end-diastole.
9. Method according to one of the preceding claims, wherein the maximum of the number of summed voxels of the class of the partial section of the lung is assigned to the physiological time of the end inspiration and / or wherein the minimum of the number of summed voxels of the class of the partial section of the lung is assigned to the physiological time of the end expiration.
10. Method according to one of the preceding claims, wherein in method step a) an acquisition time per slice of the real-time MRI of several respiratory cycles is selected.
11. Method according to one of the preceding claims, wherein the physiological state of the respiratory cycle and / or the cardiac cycle is selected from at least one element from the group of: end-inspiration, end-expiration, end-systole, end-diastole.
12. Method according to one of the preceding claims, wherein in method step a) at least one further biosignal is additionally recorded via at least one biosignal sensor at the same time as the real-time MRI is recorded, and wherein in method step e) the physiological phases are determined with additional use of the at least one further biosignal.
13. The method according to claim 12, wherein an electrocardiogram and / or a signal for a respiratory phase is used as a further biosignal.
14. Method according to one of the preceding claims, wherein in step g) the geometry of the cardiac anatomy is used to calculate the stroke volume of the heart.
15. Computer-implemented method for training a neural network for the semantic segmentation of real-time MRI images of a heart, wherein the neural network has a U-Net architecture and the combination of a binary cross-entropy loss and a Sorensen-Dice coefficient is used as the loss function for training the neural network.
16. A data processing system comprising a processor configured to perform the steps of the method according to any one of claims 1 to 14.
17. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 14.
18. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 14.
