A multi-order time difference method for heart motion feature learning

Abstract

The application discloses a multi-order time series difference method for heart motion feature learning, and aims at the problem that current heart motion tracking methods mainly require the displacement field of myocardial motion, lack the representation of the velocity field of myocardial motion, the acceleration field after force and other inherent physiological characteristics of myocardial, and proposes a multi-order time series difference method for heart motion feature learning, compared with the existing heart motion feature learning method, the application uses the CineMorph framework combining the UNet network architecture of frame perception and the time continuous Transformer block, captures the periodic motion of the heart, not only effectively extracts the motion field of each time point, but also obtains the high-precision time-continuous Lagrangian motion field of the heart from the original cine magnetic resonance imaging. The advantage of this way is that the dynamic change of the heart in the periodic motion can be accurately captured, so that the motion state of the heart at different time points can be better reflected.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically a multi-order temporal difference method for learning cardiac motion features. Background Technology

[0002] Cine magnetic resonance imaging plays a crucial role in cardiac motion tracking due to its non-invasive nature and superior imaging capabilities. This technology allows for detailed visualization of the heart's anatomy and function throughout the entire cardiac cycle, capturing high-resolution images at multiple stages. By tracking myocardial motion and deformation, clinicians can accurately assess cardiac function, identify cardiac motion abnormalities, and evaluate conditions such as coronary artery disease, myocarditis, and hypertrophic cardiomyopathy.

[0003] Current cardiac motion tracking methods are primarily derived from Lagrange fields. However, these methods cannot clearly identify abnormal cardiac motion, which limits their applicability in clinical applications.

[0004] In recent years, unsupervised methods based on deep learning have become an efficient design solution for cardiac motion tracking. These methods typically decompose the motion tracking problem into a pairwise registration process, directly or indirectly generating Lagrange motion fields.

[0005] Using classic pairwise registration networks, such as VoxelMorph Balakrishnan et al. (2019), motion fields can be learned between two consecutive or arbitrary images. When applied to consecutive images, the obtained motion fields need to be transformed into Lagrangian motion fields to achieve motion tracking between any two images. A classic work is the SequenceMorph method (2023) proposed by Ye et al., which proposes a bidirectional generative differential isomorphic registration network to estimate the inter-frame motion fields between any two consecutive coordinate systems, and then reassembles them into Lagrangian motion fields between the reference frame and any other coordinate system through a differentiable synthesis layer. Considering the temporal continuity between consecutive frames, SequenceMorph demonstrates superior tracking performance and the feasibility of the motion decomposition and reconstruction principle.

[0006] Unlike SequenceMorph, Lu et al. introduced temporal relationships through a bidirectional recurrent neural network, automatically learning the spatiotemporal information of multiple images to directly estimate the Lagrange motion field between the reference image and other images. However, these methods rely on scaling and square integral schemes (Hernandez et al. (2007) and Arsigny et al. (2006) to reconstruct the deformation field. This dependence limits their ability to capture temporal continuity, especially for large deformation motions.

[0007] Building upon this, Meng He et al. introduced an unsupervised deep learning method called CineMorph to learn temporally continuous motion fields that can be recombined into Lagrange motion fields. The method comprises a frame-aware UNet and multiple temporally continuous Transformer blocks, reducing computational costs and increasing the overall framework's flexibility, thereby improving the accuracy and efficiency of cardiac motion estimation. Simultaneously, by implementing semigroup regularization to guarantee the model's diffeomorphism property, it eliminates dependencies on scaling and square integrals. Extensive experiments on the public ACDC dataset demonstrate that CineMorph outperforms previous state-of-the-art models. These methods provide a reference for studying abnormal cardiac motion.

[0008] The main drawbacks of existing technologies include:

[0009] (1) The shortcomings of existing cardiac motion feature learning methods: Existing cardiac motion feature learning methods lack effective unsupervised learning models to handle the complexity of cardiac motion and cannot clearly identify abnormal cardiac motion; when processing continuous time series data, they cannot effectively capture the dynamic changes of cardiac motion, resulting in insufficient smoothness and continuity in the reconstruction of the motion field.

[0010] (2) Based on the shortcomings of existing motion fields: Existing methods focus on the trajectory of material points and their state over time, resulting in a Lagrange motion field that only describes myocardial motion from a kinematic perspective, lacking consideration of myocardial mechanics, which limits its applicability in clinical applications.

[0011] (3) Based on the shortcomings of existing motion field analysis methods: Existing methods do not perform detailed analysis of the motion field, ignore the fusion processing of multimodal information, and have insufficient feature extraction. They cannot fully learn cardiac motion characteristics, which may lead to incorrect motion feature analysis and affect the accuracy of diagnosis.

[0012] (4) Based on the shortcomings of existing cardiac motion feature processing methods: The data characteristics and information content of each modality are different. Existing cardiac motion feature processing methods often have difficulty effectively integrating this information, which can lead to information loss or insufficient representation and fail to capture the complex motion patterns of the heart. Summary of the Invention

[0013] This invention addresses the shortcomings of existing technologies by focusing on the displacement field of myocardial motion in current cardiac motion tracking methods, which lack characterization of the velocity field and acceleration field after force application, among other inherent physiological characteristics of the myocardium. It proposes a multi-order temporal difference method for learning cardiac motion characteristics. Based on deep learning, it uses cinematic magnetic resonance images to learn cardiac motion characteristics, employing first-order and second-order differences to obtain the corresponding velocity and acceleration fields, and then identifies myocardial motion analysis methods for lesion regions based on cinematic images.

[0014] First, cinematic magnetic resonance images are input into the CineMorph framework to learn time-continuous motion fields, which are then reconstructed into time-continuous Lagrange motion fields (displacement fields). First-order and second-order differences are performed on the Lagrange motion fields to obtain the corresponding velocity and acceleration fields, respectively. Next, the displacement, velocity, and acceleration fields are input into the Vision Transformer model to learn cardiac motion features. Then, a multimodal fusion method is used, and the results are input into a residual perceptron and a myocardial infarction classifier based on a fully connected network. The myocardial infarction prediction results are then used for rapid screening, accurate diagnosis, and prognostic assessment of patient disease.

[0015] Specifically, the present invention is achieved through the following technical solutions:

[0016] This invention discloses a multi-order temporal difference method for learning cardiac motion features, including:

[0017] 1) Input the original cinematic magnetic resonance image into the CineMorph framework to extract the Lagrange motion field (displacement field);

[0018] 2) Perform first-order and second-order finite differences on the Lagrange field to obtain the corresponding velocity field and acceleration field, respectively;

[0019] 3) Construct a Vision Transformer model to extract cardiac motion features;

[0020] 4) Input the Lagrange motion field, velocity field and acceleration field into the Vision Transformer model to learn cardiac motion features, and then use the splicing method to perform multimodal fusion to obtain the multimodal cardiac motion feature fusion result;

[0021] 5) Input the fusion result of multimodal cardiac motion features into the residual perceptron, and use the residual network to obtain high-precision cardiac motion features;

[0022] 6) Input the high-precision cardiac motion features into a myocardial infarction classifier based on a fully connected network to obtain the myocardial infarction prediction results;

[0023] 7) Utilize myocardial infarction prediction results to obtain rapid screening, accurate diagnosis, and prognostic assessment of whether a patient has the disease.

[0024] As a further improvement, the present invention describes the process of inputting the original film magnetic resonance image into the CineMorph framework to extract the Lagrange motion field (displacement field), specifically as follows:

[0025] Using the CineMorph framework, which combines a frame-aware UNet network architecture with temporally continuous Transformer blocks, the periodic motion of the heart is captured, and the displacement and deformation fields at each time point are extracted. These continuous frames are modeled, the motion field is temporally analyzed, and semigroup regularization is implemented to finally obtain a high-precision temporally continuous Lagrange motion field.

[0026] As a further improvement, the method described in this invention for obtaining the corresponding velocity field and acceleration field by performing first-order and second-order differences on the Lagrange motion field is as follows:

[0027] By performing a first-order difference on the Lagrange motion field, we obtain the velocity field of the entire system. If the position of particle i at time n is x... i (n), the first-order difference formula for the velocity field is:

[0028]

[0029] By performing a second-order difference on the Lagrange motion field, the acceleration field of the entire system is obtained. Acceleration is the rate of change of velocity relative to time. The acceleration field is calculated by performing a difference on the velocity field. If the velocities of particle i at times n and n+1 are v... i (n) and v i (n+1), the acceleration of particle i at time n is a i (n), the second-order difference formula for the acceleration field is:

[0030]

[0031] As a further improvement, the present invention specifically involves inputting the Lagrange motion field, velocity field, and acceleration field into the Vision Transformer model to learn cardiac motion features, as follows:

[0032] The Lagrange motion field, velocity field, and acceleration field are converted into tensor data suitable for input to the VisionTransformer. Self-attention operation is performed on each frame of the image to identify motion changes in different parts of the heart in the spatial dimension, and to identify the rhythm and dynamic evolution of the heart's motion in the temporal dimension, thereby obtaining the global motion characteristics of the heart.

[0033] As a further improvement, the present invention inputs the fusion result of multimodal cardiac motion features into a residual perceptron and uses a residual network to obtain high-precision cardiac motion features, specifically as follows:

[0034] The fusion result of multimodal cardiac motion features is input into a residual perceptron, and skip connections are introduced to enable effective information transfer between different layers, retain more detailed information, and obtain high-precision cardiac motion features.

[0035] The beneficial effects of this invention are as follows:

[0036] (1) Compared with existing cardiac motion feature learning methods, this invention uses a CineMorph framework that combines a frame-aware UNet network architecture with temporally continuous Transformer blocks to capture the periodic motion of the heart. This not only effectively extracts the motion field at each time point, but also obtains a high-precision temporally continuous Lagrange motion field of the heart from the original cinematic magnetic resonance imaging. The advantage of this approach is that it can accurately capture the dynamic changes of the heart in periodic motion, thereby better reflecting the motion state of the heart at different time points.

[0037] (2) Compared with existing sports fields, this invention performs first-order and second-order difference calculations on the Lagrange sports field, thereby obtaining the corresponding velocity field and acceleration field. This innovative method can not only accurately characterize the velocity changes of the heart's motion, but also further characterize the acceleration field describing the mechanical properties of the heart, thus providing more detailed and comprehensive characteristics of the heart's motion.

[0038] (3) Compared with existing motion field analysis methods, this invention comprehensively considers the Lagrange motion field, velocity field, and acceleration field, respectively inputting the Lagrange motion field, velocity field, and acceleration field into the Vision Transformer model. This allows for the identification of motion changes in different parts of the heart in the spatial dimension and the identification of the rhythm and dynamic evolution of cardiac motion in the temporal dimension. This innovative approach provides important technical support for the comprehensive identification of cardiac motion.

[0039] (4) Compared with existing methods for processing cardiac motion features, this invention employs a splicing method to perform multimodal fusion of cardiac motion features, comprehensively combining information from the Lagrange motion field, velocity field, and acceleration field. This multimodal information fusion method greatly enhances the understanding of dynamic changes in the heart and improves the expressive power of motion features. To further improve the accuracy of analysis, this invention inputs the fused features into a residual perceptron, enhancing the model's learning ability for cardiac motion features. This enables more accurate extraction of high-quality, detailed motion features, thereby improving the ability to identify abnormal cardiac motion.

[0040] (5) This invention has strong universality and relatively low computational requirements, enabling it to run efficiently with limited computing resources. This makes the method highly promising for clinical applications, especially in the early identification, dynamic monitoring, and prediction of abnormal cardiac motion, with broad application prospects. Furthermore, the method of this invention can achieve high-accuracy cardiac motion analysis, providing more precise motion feature extraction and abnormality detection than traditional methods, thus providing clinicians with more reliable diagnostic evidence. Attached Figure Description

[0041] Figure 1 This is a flowchart of the algorithm of the present invention. Detailed Implementation

[0042] This invention discloses a multi-order temporal difference method for learning cardiac motion features. The identification and analysis of abnormal myocardial motion using cine magnetic resonance imaging (MRI) has significant clinical implications in the diagnosis and treatment of cardiovascular diseases. Various cardiovascular diseases, such as coronary artery disease, myocarditis, and hypertrophic cardiomyopathy, can lead to abnormal myocardial motion, exhibiting similar clinical symptoms. Therefore, accurately identifying abnormal myocardial motion is crucial for diagnosing potential diseases, assessing disease severity, and determining appropriate treatment strategies. The technical solution of this invention is the construction process of a method for identifying abnormal myocardial motion, mainly summarized as follows:

[0043] First, cinematic magnetic resonance images are input into the CineMorph framework to learn time-continuous motion fields, which are then reconstructed into time-continuous Lagrange motion fields (displacement fields). First-order and second-order differences are performed on the Lagrange motion fields to obtain the corresponding velocity and acceleration fields, respectively. Next, the displacement, velocity, and acceleration fields are input into the Vision Transformer model to learn cardiac motion features. Then, a multimodal fusion method is used, and the results are input into a residual perceptron and a myocardial infarction classifier based on a fully connected network. The myocardial infarction prediction results are then used for rapid screening, accurate diagnosis, and prognostic assessment of patient disease.

[0044] Technical solutions such as Figure 1 As shown.

[0045] 5.1 Extracting the Initial Lagrange Sports Field using the CineMorph Framework

[0046] 5.1.1 First, the movie magnetic resonance images are input into the CineMorph framework. The CineMorph framework combines the frame-aware UNet network architecture with the time-continuous Transformer block, which can capture the periodic motion of the heart, extract the displacement field and deformation field at each time point, model these continuous frames, perform temporal analysis on the motion field, and implement semigroup regularization to finally obtain a high-precision time-continuous Lagrange motion field.

[0047] UNet, mentioned in section 5.1.1, is a convolutional neural network architecture commonly used for image segmentation tasks. It was first proposed by Olaf Ronneberger et al. in 2015, initially designed for medical image segmentation. Its key feature is a symmetrical encoder-decoder structure. The encoder extracts image features through layer-by-layer convolution and pooling, while the decoder gradually restores the image's spatial resolution through deconvolution. Between the encoder and decoder, the network directly passes the feature maps from the encoding stage to the decoding stage via skip connections, helping to preserve image detail. This design enables UNet to achieve good results in segmentation tasks.

[0048] The Transformer block mentioned in section 5.1.1 is the basic building block of the Transformer model, mainly composed of a self-attention mechanism and a feedforward neural network. In the self-attention mechanism, each element of the input sequence dynamically adjusts its representation by calculating its relationship with other elements, thereby capturing long-range dependencies; while the feedforward neural network performs further non-linear transformations on the representation of each element. Each Transformer block also includes residual connections and layer normalization to improve the stability of the training process. Multiple Transformer blocks are stacked together to form deep network structures, which are widely used in natural language processing and other tasks.

[0049] 5.2 Extracting the velocity field and acceleration field

[0050] Displacement, velocity, and acceleration fields are three important physical quantities describing the motion of points of matter at different time scales during cardiac motion. The displacement field reflects the positional changes of cardiac tissue at various time points, the velocity field reflects the motion of different points within the heart, and the acceleration field describes the mechanical properties of the myocardium. In cardiac motion modeling, considering these three fields comprehensively helps to fully describe the characteristics of cardiac motion, thereby improving the model's ability to learn from the heart. Our method obtains the velocity and acceleration fields by differencing the Lagrange motion field. The specific differencing process is as follows:

[0051] 5.2.1 Perform a first-order difference on the Lagrange motion field to obtain the velocity field of the entire system. If the position of particle i at time n is x... i(n), the first-order difference formula for the velocity field is:

[0052]

[0053] 5.2.2 By performing a second-order difference on the Lagrange motion field, the acceleration field of the entire system is obtained. Acceleration is the rate of change of velocity relative to time. The acceleration field is calculated by performing a difference on the velocity field. If the velocities of particle i at times n and n+1 are v... i (n) and v i (n+1), the acceleration of particle i at time n is a i (n), the second-order difference formula for the acceleration field is:

[0054]

[0055] 5.3 Processing displacement, velocity, and acceleration fields

[0056] 5.3.1 The displacement field, velocity field, and acceleration field are input as image sequences into the Vision Transformer neural network to learn the dynamic changes of cardiac motion in both space and time. First, these motion fields are converted into tensor data suitable for input to ViT. Through self-attention operation on each frame of image, the motion changes of different parts of the heart are identified in the spatial dimension, while the rhythm and dynamic evolution of cardiac motion are identified in the temporal dimension, thereby obtaining the global motion features of the heart.

[0057] 5.3.2 The cardiac motion features learned by the Vision Transformer network are fused into multimodal features using a concatenation method. Then, the feature is input into a residual perceptron. By introducing skip connections, information can be effectively transferred between different layers, retaining more detailed information and ensuring high-precision feature transfer.

[0058] 5.3.3 The cardiac motion features obtained through the residual perceptron are input into a myocardial infarction classifier based on a fully connected network. The myocardial infarction classifier further processes the input features and outputs the probability of whether the patient has myocardial infarction. Finally, it provides rapid screening, accurate diagnosis and prognostic assessment of whether the patient has the disease.

[0059] The Vision Transformer, mentioned in section 5.3.1, is a deep learning model for computer vision tasks. Unlike traditional convolutional neural networks, the Vision Transformer segments the input image into fixed-size image patches, then flattens these patches and converts them into one-dimensional vectors, which serve as the model's input. Through a self-attention mechanism, the Vision Transformer can capture the dependencies between distant pixels in an image without relying on convolutional operations. The Vision Transformer performs exceptionally well in both natural language processing and image data domains.

[0060] In section 5.3.1, the Vision Transformer neural network is used to learn cardiac motion features, with displacement, velocity, and acceleration fields input as image sequences into the neural network. The Vision Transformer neural network can be replaced by feature extraction neural networks such as CNN+LSTM / GRU, U-Net, and GAN, all of which can learn cardiac motion features.

[0061] In section 5.3.2, the cardiac motion features learned by the Vision Transformer network are fused into multimodal features using a concatenation method. This concatenation method can be replaced by more advanced multimodal feature fusion methods, all of which can perform multimodal fusion of cardiac motion features.

[0062] Section 5.3.3 will introduce cardiac motion features from a residual perceptron into a fully connected network-based myocardial infarction classifier. This classifier can be used in place of other cardiac disease classifiers, all of which can output the probability of a patient having the disease, thus enabling rapid screening, accurate diagnosis, and prognostic assessment.

[0063] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-order temporal difference method for learning cardiac motion features, characterized in that, include: 1) Input the original cinematic magnetic resonance image into the CineMorph framework to extract the Lagrange motion field; 2) Perform first-order and second-order finite differences on the Lagrange field to obtain the corresponding velocity field and acceleration field, respectively; 3) Construct a VisionTransformer model to extract cardiac motion features; 4) Input the Lagrange motion field, velocity field and acceleration field into the VisionTransformer model to learn cardiac motion features, and then use the splicing method to perform multimodal fusion to obtain the multimodal cardiac motion feature fusion result; 5) Input the fusion result of multimodal cardiac motion features into the residual perceptron, and use the residual network to obtain high-precision cardiac motion features; 6) Input the high-precision cardiac motion features into a myocardial infarction classifier based on a fully connected network to obtain the myocardial infarction prediction results; 7) Utilizing myocardial infarction prediction results, rapid screening, accurate diagnosis, and prognostic assessment of patient disease are achieved; specifically, the process of inputting the original cinema magnetic resonance imaging into the CineMorph framework to extract the Lagrange motion field involves: Using the CineMorph framework, which combines a frame-aware UNet network architecture with temporally continuous Transformer blocks, the periodic motion of the heart is captured, and the motion field and deformation field at each time point are extracted. These continuous frames are modeled, the motion field is temporally analyzed, and semigroup regularization is implemented to finally obtain a high-precision temporally continuous Lagrange motion field.

2. The multi-order temporal difference method for learning cardiac motion features according to claim 1, characterized in that, The specific steps for obtaining the corresponding velocity field and acceleration field by performing first-order and second-order differences on the Lagrange kinematic field are as follows: By performing a first-order difference on the Lagrange motion field, we obtain the velocity field of the entire system. If the position of particle i at time n is x... i (n), the first-order difference formula for the velocity field is: By performing a second-order difference on the Lagrange motion field, the acceleration field of the entire system is obtained. Acceleration is the rate of change of velocity relative to time. The acceleration field is calculated by performing a difference on the velocity field. If the velocities of particle i at times n and n+1 are v... i (n) and v i (n+1), the acceleration of particle i at time n is a i (n), the second-order difference formula for the acceleration field is:

3. The multi-order temporal difference method for learning cardiac motion features according to claim 2, characterized in that, The method of inputting the Lagrange motion field, velocity field, and acceleration field into the Vision Transformer model to learn cardiac motion features specifically involves: The Lagrange motion field, velocity field, and acceleration field are converted into tensor data suitable for input into the Vision Transformer. Self-attention operation is performed on each frame of the image to identify motion changes in different parts of the heart in the spatial dimension, and to identify the rhythm and dynamic evolution of the heart's motion in the temporal dimension, thereby obtaining the global motion characteristics of the heart.

4. The multi-order temporal difference method for learning cardiac motion features according to claim 1, 2, or 3, characterized in that, The process of inputting the fused results of multimodal cardiac motion features into a residual perceptron and using a residual network to obtain high-precision cardiac motion features specifically involves: The fusion result of multimodal cardiac motion features is input into a residual perceptron, and skip connections are introduced to enable effective information transfer between different layers, retain more detailed information, and obtain high-precision cardiac motion features.

Images