Heart motion feature learning-oriented multi-order time sequence difference method

Abstract

The invention discloses a multi-order time sequence difference method for heart motion feature learning, and aims to solve the problem that a current heart motion tracking method mainly requires to take a displacement field of myocardial motion and lacks characterization of inherent physiological features of myocardium such as a speed field of the myocardial motion and an acceleration field after stress. Compared with an existing heart motion feature learning method, the heart motion feature learning-oriented multi-order time sequence difference method has the advantages that a UNet network architecture combining frame perception and a CineMorph framework of a time-continuous Transform block are used for capturing the periodic motion of the heart, the motion field of each time point is effectively extracted, and the time sequence difference is more accurate. And a high-precision time-continuous Lagrange motion field of the heart can be obtained from original movie magnetic resonance imaging. The method has the advantage that the dynamic change of the heart in periodic motion can be accurately captured, so that the motion states of the heart at different time points can be better reflected.

Description

Technical Field

[0001] The patent of this invention relates to the field of medical image processing technology, specifically, it is a multi-order time difference method for learning cardiac motion features. Background Art

[0002] Cine MRI plays a vital role in cardiac motion tracking due to its non-invasiveness and superior imaging capabilities. The technology allows for detailed visualization of the anatomy and function of the heart throughout the cardiac cycle, capturing high-resolution images at multiple stages. By tracking myocardial motion and deformation, clinicians can accurately assess cardiac function, identify cardiac motion abnormalities, and evaluate conditions such as coronary artery disease, myocarditis, and hypertrophic cardiomyopathy.

[0003] Current cardiac motion tracking methods are mainly derived from Lagrangian motion fields. However, these methods cannot clearly identify abnormal cardiac motion, which limits their applicability in clinical applications.

[0004] In recent years, unsupervised methods based on deep learning have become an effective design for cardiac motion tracking. These methods usually decompose the motion tracking problem into a pairwise registration process, directly or indirectly generating a Lagrangian motion field.

[0005] Using classic pairwise registration networks such as VoxelMorph Balakrishnan et al. (2019), motion fields can be learned between two consecutive or arbitrary two images. When applied to consecutive images, the obtained motion field needs to be converted into a Lagrangian motion field to achieve motion tracking between any two images. The classic work is the SequenceMorph method proposed by Ye et al. (2023), which proposes a bidirectional generative differential isomorphism registration network to estimate the inter-frame motion field between any two consecutive coordinate systems, and then recombines them into a Lagrangian motion field between the reference system and any other coordinate system through a differentiable synthesis layer. Considering the temporal continuity between consecutive frames, SequenceMorph shows superior tracking performance and the feasibility of the motion decomposition and recombination principle.

[0006] Different from SequenceMorph, Lu et al. introduced temporal relations through a bidirectional recurrent neural network, automatically learned the spatiotemporal-temporal information of multiple images, and directly estimated the Lagrangian motion field between the reference image and other images. However, these methods rely on scaling and square integration schemes Hernandez et al. (2007); Arsigny et al. (2006) to reconstruct the deformation field. This reliance limits their ability to capture temporal continuity, especially for large deformation motions.

[0007] On this basis, Meng he et al. introduced an unsupervised deep learning method called CineMorph to learn time-continuous motion fields that can be reassembled into Lagrangian motion fields. The method includes a frame-aware UNet and multiple time-continuous Transformer blocks, which can reduce computational costs and improve the flexibility of the overall framework, thereby improving the accuracy and efficiency of cardiac motion estimation. At the same time, the differential homeomorphism properties of the model are guaranteed by implementing semigroup regularization, eliminating the dependence on scaling and square integrals. Extensive experiments were conducted on the public ACDC dataset, and the results showed that CineMorph outperformed previous state-of-the-art models. These methods provide a reference for studying abnormal cardiac motion.

[0008] The main disadvantages faced by existing technologies include:

[0009] (1) Disadvantages of existing cardiac motion feature learning methods: Existing cardiac motion feature learning methods lack an effective unsupervised learning model to handle the complexity of cardiac motion and cannot clearly identify abnormal cardiac motion; when processing continuous time series data, they cannot effectively capture the dynamic changes of cardiac motion, resulting in the reconstruction of the motion field not being smooth and continuous enough.

[0010] (2) Disadvantages of the existing motion field: The existing method focuses on the motion trajectory of material points and their time-varying state, and obtains a Lagrangian motion field that describes myocardial motion only from a kinematic level. It lacks consideration of the myocardial mechanics level, which limits its applicability in clinical applications.

[0011] (3) Disadvantages of existing motion field analysis methods: Existing methods do not perform detailed analysis of the motion field, ignore the fusion processing of multimodal information, and insufficiently extract features. They are unable to fully learn cardiac motion characteristics, which may lead to erroneous motion feature analysis and affect the accuracy of diagnosis.

[0012] (4) Disadvantages of existing cardiac motion feature processing methods: The data characteristics and information content of each modality are different. Existing cardiac motion feature processing methods often find it difficult to effectively integrate this information, resulting in information loss or insufficient representation, and are unable to capture the complex motion patterns of the heart. Summary of the invention

[0013] The present invention makes improvements on the deficiencies of the prior art. In view of the fact that the current cardiac motion tracking method mainly requires the displacement field of the myocardial motion, but lacks the characterization of the inherent physiological characteristics of the myocardium, such as the velocity field of the myocardial motion and the acceleration field after being stressed, a multi-order time difference method for learning cardiac motion features is proposed. Based on deep learning, the cardiac motion features are learned using movie magnetic resonance images, and the first-order difference and second-order difference are used to obtain the corresponding velocity field and acceleration field, so as to identify the myocardial cardiac analysis method of the diseased area based on the movie image.

[0014] First, the movie magnetic resonance images are input into the CineMorph framework to learn the time-continuous motion fields, and these motion fields are reconstructed into time-continuous Lagrangian motion fields (displacement fields). The corresponding velocity field and acceleration field are obtained by taking the first-order difference and the second-order difference of the Lagrangian motion field. Then, the displacement field, velocity field and acceleration field are respectively input into the Vision Transformer model to learn the cardiac motion characteristics, and then the splicing method is used for multimodal fusion, and the results are input into a residual perceptron and a myocardial infarction classifier based on a fully connected network. The myocardial infarction prediction results are used to obtain rapid screening, accurate diagnosis and prognosis evaluation of whether the patient is ill.

[0015] Specifically, the present invention is achieved through the following technical solutions:

[0016] The present invention discloses a multi-order temporal difference method for learning cardiac motion features, comprising:

[0017] 1) Input the original movie magnetic resonance image into the CineMorph framework to extract the Lagrangian motion field (displacement field);

[0018] 2) Perform first-order and second-order differences on the Lagrangian motion field to obtain the corresponding velocity field and acceleration field respectively;

[0019] 3) Construct a Vision Transformer model to extract cardiac motion features;

[0020] 4) The Lagrangian motion field, velocity field and acceleration field are respectively input into the Vision Transformer model to learn the cardiac motion features, and then the splicing method is used for multi-modal fusion to obtain the multi-modal cardiac motion feature fusion result;

[0021] 5) Input the fusion results of multimodal cardiac motion features into the residual sensor, and use the residual network to obtain high-precision cardiac motion features;

[0022] 6) Inputting the high-precision cardiac motion features into a myocardial infarction classifier based on a fully connected network to obtain the myocardial infarction prediction result;

[0023] 7) Utilize the myocardial infarction prediction results to obtain rapid screening, accurate diagnosis and prognosis assessment of whether the patient is ill.

[0024] As a further improvement, the present invention provides the method of inputting the original movie magnetic resonance image into the CineMorph framework to extract the Lagrangian motion field (displacement field), specifically:

[0025] The CineMorph framework, which combines the frame-aware UNet network architecture and the time-continuous Transformer block, captures the periodic motion of the heart, extracts the displacement field and deformation field at each time point, models these continuous frames, performs temporal analysis on the motion field, and implements semigroup regularization. Finally, a high-precision time-continuous Lagrangian motion field is obtained.

[0026] As a further improvement, the first-order difference and the second-order difference of the Lagrangian motion field described in the present invention are respectively obtained as follows:

[0027] Perform the first-order difference on the Lagrangian motion field to obtain the velocity field of the entire system. If the position of particle i at time n is x i (n), the first-order difference formula of velocity field is:

[0028]

[0029] Taking the second-order difference of the Lagrangian motion field, we get the acceleration field of the whole system. Acceleration is the rate of change of velocity with respect to time. By taking the difference of the velocity field, we can calculate the acceleration field. If the velocity of particle i at time n and n+1 is v respectively, i (n) and v i (n+1), the acceleration of particle i at time n is a i (n), the second-order difference formula of the acceleration field is:

[0030]

[0031] As a further improvement, the present invention provides the method of inputting the Lagrangian motion field, velocity field and acceleration field into the Vision Transformer model to learn the cardiac motion characteristics, specifically:

[0032] The Lagrangian motion field, velocity field and acceleration field are converted into tensor data suitable for input into VisionTransformer. The self-attention operation is performed on each frame of the image to identify the motion changes of different parts of the heart in the spatial dimension. At the same time, the rhythm and dynamic evolution of the heart movement are identified in the temporal dimension to obtain the global motion characteristics of the heart.

[0033] As a further improvement, the present invention inputs the multimodal cardiac motion feature fusion result into the residual sensor and uses the residual network to obtain high-precision cardiac motion features, specifically:

[0034] The fusion results of multimodal cardiac motion features are input into a residual perceptron, and jump connections are introduced to enable effective transmission of information between different layers, retaining more detailed information and obtaining high-precision cardiac motion features.

[0035] The beneficial effects of the present invention are:

[0036] (1) Compared with the existing cardiac motion feature learning methods, the present invention uses the CineMorph framework that combines the frame-aware UNet network architecture and the time-continuous Transformer block to capture the periodic motion of the heart, which not only effectively extracts the motion field at each time point, but also can obtain the high-precision time-continuous Lagrangian motion field of the heart from the original movie magnetic resonance imaging. The advantage of this method is that it can accurately capture the dynamic changes of the heart in periodic motion, thereby better reflecting the motion state of the heart at different time points.

[0037] (2) Compared with the existing motion field, the present invention performs first-order difference and second-order difference calculations on the Lagrangian motion field, thereby obtaining the corresponding velocity field and acceleration field. This innovative method can not only accurately depict the velocity changes of cardiac motion, but also further depict the acceleration field that describes the mechanical properties of the heart, thereby providing a more detailed and comprehensive cardiac motion feature.

[0038] (3) Compared with the existing motion field analysis method, the present invention comprehensively considers the Lagrangian motion field, velocity field and acceleration field, and inputs the Lagrangian motion field, velocity field and acceleration field into the Vision Transformer model respectively, so as to identify the motion changes of different parts of the heart in the spatial dimension and the rhythm and dynamic evolution of the heart motion in the temporal dimension. This innovative processing method provides important technical support for the comprehensive identification of heart motion.

[0039] (4) Compared with the existing cardiac motion feature processing method, the present invention uses a splicing method to fuse cardiac motion features in a multi-modal manner, and comprehensively combines the information of the motion features of the Lagrangian motion field, velocity field, and acceleration field. This multi-modal information fusion method greatly enhances the understanding of the dynamic changes of the heart and improves the expression ability of motion features. In order to further improve the analysis accuracy, the present invention inputs the fused features into a residual sensor, which enhances the model's learning ability for cardiac motion features, and can more accurately extract high-quality and delicate motion features, thereby improving the ability to identify abnormal cardiac motion.

[0040] (5) The present invention has strong universality and relatively low computational requirements, and can run efficiently under limited computing resources. This makes the method have great potential in clinical applications, especially in the early identification, dynamic monitoring and prediction of abnormal cardiac motion, and has broad application prospects. In addition, the method of the present invention can achieve high-accuracy cardiac analysis, provide more accurate motion feature extraction and abnormality detection than traditional methods, and provide more reliable diagnostic basis for clinicians. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a flow chart of the algorithm of the present invention. DETAILED DESCRIPTION

[0042] The present invention discloses a multi-order time difference method for learning cardiac motion features. The recognition and analysis of abnormal myocardial motion by movie magnetic resonance images has important clinical significance in the diagnosis and treatment of cardiovascular diseases. Various cardiovascular diseases, such as coronary artery disease, myocarditis and hypertrophic cardiomyopathy, can cause abnormal myocardial motion and show similar clinical symptoms. Therefore, accurate identification of abnormal myocardial motion is crucial for diagnosing potential diseases, assessing the severity of the disease and determining appropriate treatment strategies. The technical solution of the present invention is the construction process of the method for identifying abnormal myocardial motion, which can be summarized as follows:

[0043] First, the movie magnetic resonance images are input into the CineMorph framework to learn the time-continuous motion fields, and these motion fields are reconstructed into time-continuous Lagrangian motion fields (displacement fields). The corresponding velocity field and acceleration field are obtained by taking the first-order difference and the second-order difference of the Lagrangian motion field. Then, the displacement field, velocity field and acceleration field are respectively input into the Vision Transformer model to learn the cardiac motion characteristics, and then the splicing method is used for multimodal fusion, and the results are input into a residual perceptron and a myocardial infarction classifier based on a fully connected network. The myocardial infarction prediction results are used to obtain rapid screening, accurate diagnosis and prognosis evaluation of whether the patient is ill.

[0044] Technical solutions such as Figure 1 shown.

[0045] 5.1 Extracting the Initial Lagrangian Motion Field Using CineMorph Framework

[0046] 5.1.1 First, the movie MRI images are input into the CineMorph framework. The CineMorph framework combines the frame-aware UNet network architecture and the time-continuous Transformer block to capture the periodic motion of the heart, extract the displacement field and deformation field at each time point, model these continuous frames, perform temporal analysis on the motion field, and implement semigroup regularization. Finally, a high-precision time-continuous Lagrangian motion field is obtained.

[0047] The UNet mentioned in 5.1.1 is a convolutional neural network architecture commonly used in image segmentation tasks. It was first proposed by Olaf Ronneberger et al. in 2015 and was originally designed for medical image segmentation. It is characterized by a symmetrical encoding-decoding structure. The encoder part extracts image features through layer-by-layer convolution and pooling, and the decoder part gradually restores the spatial resolution of the image through deconvolution. Between the encoder and the decoder, the network directly transfers the feature map of the encoding stage to the decoding stage through jump connections to help retain the detailed information of the image. This design enables UNet to achieve good results in segmentation tasks.

[0048] The Transformer block mentioned in 5.1.1 is the basic building block of the Transformer model, which mainly consists of two parts: the self-attention mechanism and the feedforward neural network. In the self-attention mechanism, each element of the input sequence dynamically adjusts its representation by calculating the relationship with other elements to capture long-distance dependencies; while the feedforward neural network performs further nonlinear transformations on the representation of each element. Each Transformer block also includes residual connections and layer normalization to improve the stability of the training process. Multiple Transformer blocks are stacked together to form a deep network structure, which is widely used in natural language processing and other tasks.

[0049] 5.2 Extracting velocity and acceleration fields

[0050] Displacement field, velocity field and acceleration field are three important physical quantities that describe the motion of material points at different time scales during cardiac motion. The displacement field reflects the position change of cardiac tissue at each time point, the velocity field reflects the motion of each point in the heart, and the acceleration field describes the mechanical properties of the myocardium. In cardiac motion modeling, comprehensive consideration of these three fields helps to fully describe the characteristics of cardiac motion, thereby improving the model's ability to learn about the heart. Our method obtains the velocity field and acceleration field by differentiating the Lagrangian motion field. The specific difference process is as follows:

[0051] 5.2.1 Take the first-order difference of the Lagrangian motion field to obtain the velocity field of the entire system. If the position of particle i at time n is x i(n), the first-order difference formula of velocity field is:

[0052]

[0053] 5.2.2 Perform a second-order difference on the Lagrangian motion field to obtain the acceleration field of the entire system. Acceleration is the rate of change of velocity with respect to time. The acceleration field is calculated by taking a difference on the velocity field. If the velocity of particle i at time n and n+1 is v respectively, i (n) and v i (n+1), the acceleration of particle i at time n is a i (n), the second-order difference formula of the acceleration field is:

[0054]

[0055] 5.3 Processing displacement, velocity and acceleration fields

[0056] 5.3.1 Input the displacement field, velocity field and acceleration field as image sequences into the Vision Transformer neural network to jointly learn the dynamic changes of heart motion in space and time. First, convert these motion fields into tensor data suitable for input into ViT. Through the self-attention operation on each frame of the image, the motion changes of different parts of the heart are identified in the spatial dimension, and the rhythm and dynamic evolution of heart motion are identified in the temporal dimension, thereby obtaining the global motion characteristics of the heart.

[0057] 5.3.2 The cardiac motion features learned by the Vision Transformer network are concatenated to perform multimodal feature fusion, and then input into a residual perceptron. By introducing jump connections, information can be effectively transmitted between different layers, retaining more detailed information and ensuring high-precision feature transmission.

[0058] 5.3.3 The cardiac motion features obtained through the residual perceptron are input into a myocardial infarction classifier based on a fully connected network. The myocardial infarction classifier further processes the input features and outputs the probability of whether the patient has myocardial infarction. Finally, rapid screening, accurate diagnosis and prognosis assessment of whether the patient is ill are obtained.

[0059] The Vision Transformer mentioned in 5.3.1 is a deep learning model for computer vision tasks. Unlike traditional convolutional neural networks, the Vision Transformer divides the input image into fixed-size image blocks, and then flattens and converts these image blocks into one-dimensional vectors as the input of the model. Through the self-attention mechanism, the Vision Transformer can capture the dependencies between distant pixels in the image without relying on convolution operations. Vision Transformer performs well in both natural language processing and image data.

[0060] In section 5.3.1, the Vision Transformer neural network is used to learn the cardiac motion features, and the displacement field, velocity field, and acceleration field are input into the neural network as image sequences. The Vision Transformer neural network can be replaced by feature extraction neural networks such as CNN+LSTM / GRU, U-Net, and GAN, which can all learn cardiac motion features.

[0061] In section 5.3.2, the cardiac motion features learned by the Vision Transformer network are fused by splicing method for multimodal features. The splicing method can be replaced by a more advanced multimodal feature fusion method, which can also perform multimodal fusion of cardiac motion features.

[0062] In section 5.3.3, the cardiac motion features of the residual perceptron are input into a myocardial infarction classifier based on a fully connected network. The myocardial infarction classifier can be replaced by other heart disease classifiers, which can output the probability of whether the patient is sick, thereby achieving rapid screening, accurate diagnosis and prognosis assessment of patients.

[0063] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-order temporal difference method for learning cardiac motion features, characterized in that: include: 1) Input the original movie magnetic resonance image into the CineMorph framework to extract the Lagrangian motion field (displacement field); 2) Perform first-order and second-order differences on the Lagrangian motion field to obtain the corresponding velocity field and acceleration field respectively; 3) Construct a Vision Transformer model to extract cardiac motion features; 4) The Lagrangian motion field, velocity field and acceleration field are respectively input into the Vision Transformer model to learn the cardiac motion features, and then the splicing method is used for multi-modal fusion to obtain the multi-modal cardiac motion feature fusion result; 5) Input the fusion results of multimodal cardiac motion features into the residual sensor, and use the residual network to obtain high-precision cardiac motion features; 6) Inputting the high-precision cardiac motion features into a myocardial infarction classifier based on a fully connected network to obtain the myocardial infarction prediction result; 7) Utilize the myocardial infarction prediction results to obtain rapid screening, accurate diagnosis and prognosis assessment of whether the patient is ill.

2. The multi-order temporal difference method for cardiac motion feature learning according to claim 1, characterized in that: The method of inputting the original movie magnetic resonance image into the CineMorph framework to extract the Lagrangian motion field (displacement field) is specifically as follows: The CineMorph framework, which combines the frame-aware UNet network architecture and the time-continuous Transformer block, captures the periodic motion of the heart, extracts the displacement field and deformation field at each time point, models these continuous frames, performs temporal analysis on the motion field, and implements semigroup regularization. Finally, a high-precision time-continuous Lagrangian motion field is obtained.

3. The multi-order temporal difference method for cardiac motion feature learning according to claim 1 or 2, characterized in that: The first-order difference and second-order difference of the Lagrangian motion field are respectively obtained to obtain the corresponding velocity field and acceleration field as follows: Perform the first-order difference on the Lagrangian motion field to obtain the velocity field of the entire system. If the position of particle i at time n is x i (n), the first-order difference formula of velocity field is: The second-order difference of the Lagrangian motion field is used to obtain the acceleration field of the entire system. Acceleration is the rate of change of velocity with respect to time. The acceleration field is calculated by differentiating the velocity field. If the velocity of particle i at time n and n+1 is v respectively i (n) and v i (n+1), the acceleration of particle i at time n is a i (n), the second-order difference formula of the acceleration field is:

4. The multi-order temporal difference method for cardiac motion feature learning according to claim 3, characterized in that: The Lagrangian motion field, velocity field and acceleration field are respectively input into the Vision Transformer model to learn the cardiac motion features, specifically: The Lagrangian motion field, velocity field, and acceleration field are converted into tensor data suitable for input into the Vision Transformer. The self-attention operation is performed on each frame of the image to identify the motion changes of different parts of the heart in the spatial dimension. At the same time, the rhythm and dynamic evolution of the heart motion are identified in the temporal dimension to obtain the global motion characteristics of the heart.

5. The multi-order temporal difference method for cardiac motion feature learning according to claim 1, 2 or 4, characterized in that: The multi-modal cardiac motion feature fusion result is input into the residual sensor, and the high-precision cardiac motion feature is obtained by using the residual network, specifically: The fusion results of multimodal cardiac motion features are input into a residual perceptron, and jump connections are introduced to enable effective transmission of information between different layers, retaining more detailed information and obtaining high-precision cardiac motion features.

Images