A method and device for directly delineating myocardial infarction area from cardiac magnetic resonance image sequence based on deep learning

Abstract

The present invention discloses a method and device for directly delineating myocardial infarction areas from cardiac magnetic resonance image sequences based on deep learning, combining deep learning, i.e., performing myocardial segmentation through a U-Net structure, fusing motion features and pixel features, and finally learning features through a ConvLSTM structure to obtain the range of myocardial infarction. This method has strong universality and can work on different myocardial sections of the same patient. It has high accuracy and segmentation accuracy, low calculation requirements, high time efficiency, and does not require contrast agents. It can be widely used in clinical practice. It is a high-accuracy contrast-free myocardial infarction area segmentation technology that can avoid the damage of gadolinium contrast agents to human organs. It is particularly suitable for patients with kidney disease, does not require complex modeling and mathematical calculations, saves time, and has better universality and practicality.

Description

Technical Field

[0001] The patent of this invention relates to the field of medical image analysis technology. Specifically, it is a method and device for directly delineating myocardial infarction areas from cardiac magnetic resonance image sequences based on deep learning. Background Art

[0002] Myocardial infarction generally refers to myocardial necrosis caused by persistent ischemia and hypoxia of the coronary arteries. In the acute phase, clinical manifestations are often severe and persistent pain behind the sternum, which cannot be completely relieved by rest and nitrates, accompanied by increased serum myocardial enzyme activity and progressive electrocardiogram changes, which may be complicated by arrhythmia, shock or heart failure, and are often life-threatening. Since 2005, the mortality rate of acute myocardial infarction has been increasing rapidly, with ischemic cardiomyopathy increasing by 90%. It is estimated that the number of deaths from 2016 to 2030 will be nearly 10 million (China Cardiovascular Disease Report 2018). Cardiac magnetic resonance imaging can intuitively evaluate myocardial contractile function, myocardial deformation and other characteristics, and is of great significance for the prevention and diagnosis of myocardial infarction.

[0003] In clinical work, delayed gadolinium enhancement (LGE)-cardiac magnetic resonance imaging (CMR) is currently considered the gold standard for noninvasive detection of infarct areas in vivo. However, this approach has certain risks because gadolinium contrast agents are contraindicated in patients with chronic end-stage renal disease. According to the US Renal Data System, more than 40% of patients with chronic kidney disease have cardiovascular disease, and approximately 20% of patients with myocardial infarction have chronic kidney disease. In addition, recent studies have shown that gadolinium may be deposited in the skin, dentate nucleus, and globus pallidus in patients with normal renal function. Therefore, it would be a huge clinical advancement to directly identify the myocardial infarction area without contrast agents, which can not only be used to identify the presence, location, and transmural extent of myocardial infarction, but also to predict the recovery of dysfunctional myocardial segments in ischemic heart disease.

[0004] Before the advent of deep learning technology, there were three main methods for the segmentation of myocardial infarction areas without contrast agents. The first method, such as the statistical image sequence alignment method proposed by Noble et al. in Ultrasound in Medicine and Biology in 2002, established the correspondence between MRI image frames and quickly extracted motion features from the image sequence by calculating the displacement of the myocardium. The second method was published by Shi and Liu in the Journal of Medical Image Analysis in 2003, which introduced different regularization strategies to estimate cardiac motion from medical image sequences. The third method is the statistical shape modeling method proposed by Hufnagel et al. in the International Journal of Computer-Assisted Radiosurgery in 2008, which attempts to introduce prior knowledge by modeling possible motion and statistical representation of shape to overcome the inherent complexity of the data.

[0005] Although the above methods have been successful to some extent, there are still some problems in their application in clinical settings.

[0006] Due to the rapid development of deep learning technology, the application of deep learning in cardiac magnetic resonance imaging (CMRI) processing has increased significantly in recent years, especially in the segmentation of cardiac structures. Some research groups have also tried to use deep learning for the segmentation of myocardial infarction without contrast agents, using deep learning to detect morphological and kinematic abnormalities of the heart, thereby identifying and locating myocardial infarction.

[0007] In 2017, at the 8th International Symposium on Cardiac Statistical Atlas and Computational Models, Khened et al. proposed a fully automatic deep learning network. The training data was processed by Fourier analysis for region of interest extraction, and the parameters of the network were optimized by training using a dual cost function (i.e., weighted cross entropy and Dice coefficient), thereby realizing the segmentation of the left ventricle, right ventricle, and myocardium from cardiac magnetic resonance (MR) images using a dense convolutional neural network (DenseNet). The 11 extracted myocardial feature parameters were used in the automatic diagnosis of heart disease using a random forest classifier, which could preliminarily diagnose whether the patient had dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy (HCM), myocardial infarction (MNF), and other diseases.

[0008] In 2018, Baessler et al. published an article in the journal Radiology. Based on reproducibility, machine learning and correlation analysis, they used stepwise dimensionality reduction and texture feature selection to select features, and used delayed enhancement (DE) images as the reference standard to diagnose myocardial infarction in contrast-free CMRI images. The experiment showed that texture analysis of contrast-free CMRI images can distinguish between ischemic scars caused by myocardial infarction and normal myocardium, thereby diagnosing subacute and chronic myocardial infarction with high accuracy. It laid the foundation for the subsequent diagnosis of myocardial infarction by learning the pixel features of the infarct area.

[0009] In 2018, Xu et al. published a new joint motion feature learning architecture in the journal Medical Image Analysis. The architecture consists of three seamlessly connected functional layers: the cardiac localization layer uses Faster R-CNN to automatically crop the region of interest sequence involving the left ventricle from the cardiac imaging sequence; the motion feature extraction layer uses a long short-term memory recurrent neural network (LSTM) to construct patch-based motion features through local intensity changes between fixed-size pixel block sequences (cropped from image sequences), and uses optical flow technology to construct image-based features through global intensity changes between adjacent images to describe the motion of each pixel; the fully connected discrimination layer can combine the two motion features in each pixel, and then establish the correspondence between the motion feature and the tissue identity (i.e., whether there is infarction) in each pixel, thereby achieving pixel-level segmentation of myocardial infarction area.

[0010] In 2019, Chen et al. proposed a deep learning constraint framework in the journal IEEE Access, integrating optical flow features for myocardial infarction in medical image sequences. The framework consists of two stages. In the first stage, a fixed-size static pixel block is obtained by sliding a fixed-size window in the left ventricular myocardial area, and the motion information at the same position is extracted using the optical flow method. The basic autoencoder (SDAE) is trained by minimizing the objective function and the square error. The intensity and motion features of the pixel block are extracted using the trained stacked denoising autoencoder, and Gaussian noise is added for repeated training optimization. Finally, an SDAE containing two hidden layers is obtained. In the second stage, fixed-length features are extracted from the second SDAE hidden layer, and a label of whether the pixel block is an infarction is generated by delaying the enhanced image, and the support vector machine (SVM) is used to classify it into infarction and non-infarction.

[0011] In 2021, Manchio et al. published a machine learning study on hypertrophic cardiomyopathy (HCM) in the European Heart Journal-Journal of Cardiovascular Imaging. Because the scar tissue of HCM is the same as the tissue repaired after myocardial infarction, it is similar to our research. The study used an XGBoost model with 10-fold cross-validation to extract the 7 most important features from a total of 2,613 features. In the end, the study found that the presence of replacement fibrosis due to HCM can be identified by its higher signal myocardial intensity, texture heterogeneity and roughness, lower myocardial volume, and reduced myocardial wall thickening. This provides a reference idea for the feature extraction part of our study of myocardial infarction (MI).

[0012] (1) Disadvantages of segmenting the myocardial infarction area using gadolinium contrast agents: This method has certain risks because gadolinium contrast agents are lethal to patients with chronic end-stage renal disease, and about 20% of patients with acute myocardial infarction have chronic kidney disease.

[0013] (2) Disadvantages of contrast-free myocardial infarction area segmentation methods without deep learning: First, segmentation establishment-correspondence calculation of strain is a complex process, which may be a limiting factor. Second, these methods are time-consuming and cumbersome, requiring the use of careful modeling techniques. In addition, the complex spatiotemporal motion of the heart may lead to implementation difficulties.

[0014] (3) Pixel-level myocardial infarction area segmentation method combining optical flow method and LSTM Disadvantages: The whole process is complicated, and the error in each step will lead to subsequent continuous impact. In addition, the model not only requires a large amount of label calibration, but also contains a large number of parameters, requiring more than 138M parameters. And it is only effective for the middle layer of the myocardium, which will waste images of other layers.

[0015] (4) Disadvantages of the myocardial infarction area segmentation method using SDAE and SVM: It can only classify the infarction and non-infarction of 10×10 pixel blocks on the myocardium, and finally only about 30 pixel blocks can be classified, so it can only determine the approximate range of the infarction, which is not precise enough. Moreover, it is only effective for the middle layer of the myocardium, which will waste images of other layers.

[0016] In summary, how to accurately, efficiently and time-savingly segment the myocardial infarction area without contrast agents has become an important research topic at present. Summary of the invention

[0017] The present invention has made improvements to the deficiencies of the prior art, and provides a method and device for directly delineating the myocardial infarction area from a cardiac magnetic resonance image sequence based on deep learning, segmenting the myocardium and the epicardium through the U-Net model, extracting the motion characteristics and grayscale characteristics of the myocardium, building a neural network with ConvLSTM as the core, training the neural network and identifying the presentation technology of the myocardial infarction area based on movie images, and realizing a prediction method for directly delineating the myocardial infarction (MI) area from a cardiac magnetic resonance image sequence without contrast agent. The basic contents of the technical solution for segmenting the myocardial infarction area of ​​the present invention mainly include:

[0018] The present invention discloses a method for directly delineating myocardial infarction areas from cardiac magnetic resonance image sequences based on deep learning, comprising:

[0019] Obtain cardiac magnetic resonance movie sequences, and segment the myocardium and epicardium of each frame in the cardiac magnetic resonance imaging (CMRI) movie sequence using the U-Net model;

[0020] Binarize the segmented myocardium and epicardium of each frame to obtain the myocardial boundary, extract the motion features and pixel features of each frame, splice and fuse them, and store them in a two-dimensional feature matrix;

[0021] The infarcted area of ​​the corresponding sequence of LGE images with gadolinium contrast agent in the two-dimensional feature matrix is ​​marked and converted into a 1×120 vector as the myocardial infarction area label;

[0022] Based on the obtained two-dimensional feature matrix and the corresponding myocardial infarction area label, a neural network model with ConvLSTM as the core is constructed, wherein the neural network model with ConvLSTM as the core includes a ConvLSTM layer, a convolution layer, an expansion layer, and a fully connected layer;

[0023] When training the neural network model with ConvLSTM as the core, the 1×120 vector output by the model is compared with the real myocardial infarction area label, and the binary cross entropy "binary_crossentropy" is used as the loss function; (because whether there is myocardial infarction corresponding to each angle is a binary classification problem)

[0024] Use the "Adam" optimizer to iteratively train the network and optimize the network parameters. When the training is completed, test the accuracy on the test set and save the model with the highest accuracy.

[0025] The input of the model is the feature matrix extracted previously, and the output of the model is a 1×120 vector, which can be used to mark the myocardial infarction area. The myocardial infarction area label is used for training to minimize the loss in the test set, and the myocardial infarction area segmentation model with the highest accuracy is obtained. Using the obtained myocardial infarction area segmentation model, any given new patient's CMRI film sequence is segmented by U-Net and then the motion and pixel features are extracted. After that, the features are fused and input into the myocardial infarction area segmentation model to obtain the output 1×120 vector, thereby marking the myocardial infarction area.

[0026] As a further improvement, the method of extracting motion features of each frame according to the present invention is specifically as follows:

[0027] The midpoint of the myocardial epicardium boundary is extracted as the center point of the entire image. Rays are drawn from the center point outward at an angle interval of 3 degrees, and the intersection points with the myocardial endomyocytium and myocardial epicytium boundaries are taken to obtain the coordinates of the intersection points of the myocardium and the myocardium in 120 directions. The extracted motion features are:

[0028] 1) The coordinate difference x between the inner membrane intersection point and the x-axis of the center point of the graph 1 , y-axis coordinate difference y 1 ,distance

[0029] 2) The x-axis coordinate difference between the outer membrane intersection point and the center point of the graph 2 , y-axis coordinate difference y 2 ,distance

[0030] 3) The x-axis coordinate difference between the inner and outer membranes 3 , y-axis coordinate difference y 3 ,distance

[0031] 4) The x-axis coordinate difference x between the inner membrane intersection point and its counterclockwise adjacent point 4 , y-axis coordinate difference y 4 ,distance

[0032] 5) The x-axis coordinate difference between the outer membrane intersection point and its counterclockwise adjacent point x 5 , y-axis coordinate difference y 5 ,distance The U-Net model is used to segment the myocardium and epicardium, extract the motion characteristics and grayscale feature information of the myocardium, build a neural network with ConvLSTM as the core, and train the neural network and present the myocardial infarction area.

[0033] As a further improvement, the method of extracting pixel features of each frame described in the present invention is as follows: take points at corresponding angles between the inner and outer membranes and the outer membrane, take point O, and take out 25 pixel matrices around point O. This represents the morphological features of the myocardium.

[0034] As a further improvement, the splicing and fusion described in the present invention is to expand the myocardial pixel features into a one-dimensional vector and splice it under the myocardial motion features.

[0035] As a further improvement, the infarction area of ​​the LGE image with gadolinium contrast agent in the corresponding sequence in the annotated two-dimensional feature matrix of the present invention is converted into a 1×120 vector as a myocardial infarction area label. Specifically, the pixel-level myocardial infarction range is first marked on the LGE image under the guidance of a doctor, and then rays are made outward at an angle interval of 3 degrees with the center point of the outer membrane as the center, and the myocardium is divided into 120 parts at equal angles. The pixel-level myocardial infarction range corresponds to whether there is myocardial infarction in each of the 120 equally divided myocardiums, and no infarction is marked as 0, and infarction is marked as 1, thereby converting it into a 1×120 vector as the myocardial infarction area label.

[0036] As a further improvement, the input of the neural network model with ConvLSTM as the core described in the present invention is the extracted feature matrix, and the output of the model is a 1×120 vector. (The myocardial infarction area can be marked using this vector.)

[0037] As a further improvement, the present invention outputs a 120-dimensional vector from the neural network with ConvLSTM as the core, and the value of each dimension is between 0 and 1, 0 represents no infarction, and 1 represents infarction.

[0038] As a further improvement, the present invention preprocesses the cardiac magnetic resonance imaging movie sequence and then uses the U-Net model to segment the myocardium and epicycium. The preprocessing is to obtain the coordinates of the heart area by training YOLO-v4, then capture the image according to the coordinates, and then rotate and straighten the image.

[0039] As a further improvement, the preprocessing described in the present invention is specifically as follows: first process the size of the image, cut out the area of ​​interest, record the coordinate values ​​of its four vertices as a training set, and input it into the YOLO-v4 model. After the training is completed, the characteristics of the heart area are learned. After that, for each original image, it can give the coordinates of the four vertices of the area of ​​interest, cut out the image according to this coordinate, and then rotate and straighten the image so that the left ventricle and the right ventricle are on the same horizontal line in the image, and the preprocessing part is completed.

[0040] The present invention also discloses a device for directly delineating a myocardial infarction area from a cardiac magnetic resonance image sequence based on deep learning, comprising:

[0041] Segmentation unit: obtain cardiac magnetic resonance film sequences, and segment the myocardium and epicardium of each frame in the cardiac magnetic resonance imaging (CMRI) film sequence through the U-Net model;

[0042] Feature extraction unit: Binarize the segmented myocardium and epicardium of each frame to obtain the myocardial boundary, extract the motion features and pixel features of each frame, splice and fuse them, and store them in a two-dimensional feature matrix;

[0043] Labeling unit: label the infarct area of ​​the corresponding sequence of LGE images with gadolinium contrast agent in the two-dimensional feature matrix and convert it into a 1×120 vector as the myocardial infarction area label;

[0044] Constructing a neural network model unit: based on the obtained two-dimensional feature matrix and the corresponding myocardial infarction area label, constructing a neural network model with ConvLSTM as the core, wherein the neural network model with ConvLSTM as the core includes a ConvLSTM layer, a convolution layer, an expansion layer, and a fully connected layer;

[0045] Training unit: When training in a neural network model with ConvLSTM as the core, the 1×120 vector output by the model is compared with the actual myocardial infarction area label, and binary cross entropy "binary_crossentropy" is used as the loss function;

[0046] Optimization unit: Use the "Adam" optimizer to iteratively train the network and optimize the network parameters. When the training is completed, the accuracy is tested on the test set and the model with the highest accuracy is saved.

[0047] The beneficial effects of the present invention are:

[0048] The present invention combines deep learning and proposes a method for directly delineating the myocardial infarction area from a cardiac magnetic resonance image sequence based on deep learning, that is, myocardial segmentation is performed through a U-Net structure, motion features and pixel features are integrated, and finally the features are learned through a ConvLSTM structure to obtain the range of myocardial infarction. This method has strong universality and can work on different myocardial sections of the same patient. It has high accuracy and segmentation precision, low computational requirements, high time efficiency, and does not require contrast agents. It can be widely used in clinical practice and is a high-accuracy contrast-free myocardial infarction area segmentation technology. Specifically:

[0049] 1) Use the U-Net model to segment the myocardium and epicardium, train YOLO-v4 to obtain the coordinates of the heart area, then capture the image according to the coordinates, and then rotate and straighten the image.

[0050] 2) With the center of the epithelium as the origin, rays are drawn at 3-degree intervals to obtain the intersections with the myocardium and the epithelium. Then, the motion features of the inner and outer membranes of the heart and the pixel features of 25 pixels around the center of the inner and outer membranes are extracted, and the motion features and pixel features are expanded and spliced ​​into a feature matrix.

[0051] 3) A deep learning model with ConvLSTM as the core, which inputs a feature matrix of a CMRI sequence, can output an infarct area vector, and the infarct area can be intuitively marked based on the output vector.

[0052] 4) Compared with the myocardial infarction area segmentation method using gadolinium contrast agent, this method is a contrast agent-free segmentation method, which can avoid the damage of gadolinium contrast agent to human organs and is particularly suitable for patients with kidney disease.

[0053] 5) Compared with the contrast-free myocardial infarction area segmentation method without deep learning, this method is a method combined with deep learning, which does not require complex modeling and mathematical calculations, saves time, and has better universality and practicality.

[0054] 6) Compared with the pixel-level myocardial infarction area segmentation method combining optical flow and LSTM, this method has a simple model structure, stronger robustness, and requires smaller labels and storage space. Training and testing only require home computers, with fast calculation speed and real-time requirements. And because the powerful U-Net network is used to segment the myocardium, more cross-sectional myocardial shapes can be effectively segmented to avoid data waste.

[0055] 7) Compared with the myocardial infarction area segmentation method using SDAE and SVM, the segmentation of this method is more detailed. The entire myocardium is divided into 120 feature vectors for output, and finally an accuracy of 3 degrees can be achieved, which is conducive to obtaining more detailed myocardial infarction features and can effectively segment more cross-sectional myocardial shapes to avoid data waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a deep learning-based segmentation model for the area of ​​myocardial infarction without contrast agent;

[0057] Figure 2 This is a flowchart of myocardial segmentation;

[0058] Figure 3 This is the result diagram of preprocessing and myocardial segmentation;

[0059] Figure 3 a is a frame in the cardiac magnetic resonance imaging (CMRI) movie sequence, i.e., the original image; Figure 3 b is the region of interest obtained after preprocessing; Figure 3 c is the segmented epicardium; Figure 3 d is the segmented myocardium;

[0060] Figure 4 It is the feature matrix diagram;

[0061] Figure 5 This is a diagram of the convolutional neural network structure with ConvLSTM as the core;

[0062] Figure 6 is the segmentation result image; Figure 6 e is a cardiac MRI image of delayed gadolinium enhancement (LGE); Figure 6 The white area in f is the myocardial infarction area marked on the LGE image under the guidance of an experienced physician; Figure 6 The white edge range in g is the myocardial infarction area segmented from the corresponding sequence without contrast agent, which is consistent with that with contrast agent. DETAILED DESCRIPTION

[0063] The present invention discloses a method and device for directly delineating myocardial infarction regions from cardiac magnetic resonance image sequences based on deep learning, which is a prediction method for directly delineating myocardial infarction (MI) regions from cardiac magnetic resonance image sequences without contrast agents based on deep learning. The basic contents of the technical solution for segmenting myocardial infarction regions of the present invention mainly include:

[0064] The U-Net model is used to segment the myocardium and epicardium, extract the motion characteristics and grayscale feature information of the myocardium, build a neural network with ConvLSTM as the core, and train the neural network and present the myocardial infarction area.

[0065] Figure 1 It is a deep learning-based segmentation model for the area of ​​myocardial infarction without contrast agent;

[0066] Obtain cardiac magnetic resonance movie sequences, and segment the myocardium and epicardium of each frame in the cardiac magnetic resonance imaging (CMRI) movie sequence using the U-Net model;

[0067] Binarize the segmented myocardium and epicardium of each frame to obtain the myocardial boundary, extract the motion features and pixel features of each frame, splice and fuse them, and store them in a two-dimensional feature matrix;

[0068] The infarcted area of ​​the corresponding sequence of LGE images with gadolinium contrast agent in the two-dimensional feature matrix is ​​marked and converted into a 1×120 vector as the myocardial infarction area label;

[0069] Based on the obtained two-dimensional feature matrix and the corresponding myocardial infarction area label, a neural network model with ConvLSTM as the core is constructed, wherein the neural network model with ConvLSTM as the core includes a ConvLSTM layer, a convolution layer, an expansion layer, and a fully connected layer;

[0070] When training the neural network model with ConvLSTM as the core, the 1×120 vector output by the model is compared with the real myocardial infarction area label, and the binary cross entropy "binary_crossentropy" is used as the loss function; (because whether there is myocardial infarction corresponding to each angle is a binary classification problem)

[0071] Use the "Adam" optimizer to iteratively train the network and optimize the network parameters. When the training is completed, test the accuracy on the test set and save the model with the highest accuracy.

[0072] The input of the model is the feature matrix extracted previously, and the output of the model is a 1×120 vector, which can be used to mark the myocardial infarction area. The myocardial infarction area label is used for training to minimize the loss in the test set, and the myocardial infarction area segmentation model with the highest accuracy is obtained.

[0073] Using the obtained myocardial infarction area segmentation model, any given new patient's CMRI film sequence is segmented by U-Net and then the motion and pixel features are extracted. After that, the features are fused and input into the myocardial infarction area segmentation model to obtain the output 1×120 vector, thereby marking the myocardial infarction area.

[0074] Segmentation of the endomyocardium and epicardium

[0075] Figure 2 This is a flowchart of myocardial segmentation;

[0076] Preprocessing: After the start, in order to improve the accuracy of myocardial segmentation and reduce the running time, the CMRI movie sequence is preprocessed first. During preprocessing, the size of the image must be processed first, the area of ​​interest is cut out, the coordinate values ​​of its four vertices are recorded as a training set, and input into the YOLO-v4 model. After the training is completed, it can learn the characteristics of the heart area, and then for each original image, it can give the coordinates of the four vertices of the area of ​​interest. According to this coordinate, the image is intercepted, and then the image is rotated and straightened so that the left ventricle and the right ventricle are on the same horizontal line in the image, and the preprocessing part is completed.

[0077] The pre-sorted myocardial endothelium labels and the corresponding original images are put into the U-Net network as training sets, and the training model 1 can extract the myocardial endothelium from the pre-processed images; the myocardial epicardium labels and the corresponding original images are put into the U-Net network as training sets, and the training model 2 can extract the myocardial epicardium from the pre-processed images. The part between the endothelium and the epicardium boundary (the epicardium minus the endothelium) is the complete myocardium to be studied.

[0078] Figure 3 This is the result diagram of preprocessing and myocardial segmentation; Figure 3 (a) A frame in a cardiac magnetic resonance imaging (CMRI) movie sequence, i.e., the original image; Figure 3 (b) is the region of interest obtained after preprocessing; Figure 3 (c) is the segmented epicardium; Figure 3 (d) is the segmented myocardium.

[0079] Extracting myocardial feature matrix

[0080] The characteristics of the myocardium include motion characteristics and pixel characteristics, which are described as follows.

[0081] 1. Myocardial motion features: For each frame of the image, the inner and outer membranes that have been distinguished are binarized to extract the myocardial boundary. The midpoint of the outer membrane boundary is extracted as the center point of the entire image. Rays are drawn from the center point outward at an angle interval of 3 degrees, and the intersection points with the inner and outer membrane boundaries are taken to obtain the coordinates of the inner and outer membrane intersection points in 120 directions. The extracted motion features are: 1) The coordinate difference x between the inner membrane intersection point and the x-axis of the center point of the image 1 , y-axis coordinate difference y 1 ,distance 2) The x-axis coordinate difference between the outer membrane intersection point and the center point of the graph 2 , y-axis coordinate difference y 2 ,distance 3) The x-axis coordinate difference between the inner and outer membranes 3 , y-axis coordinate difference y 3 ,distance 4) The x-axis coordinate difference x between the inner membrane intersection point and its counterclockwise adjacent point 4 , y-axis coordinate difference y 4 ,distance 5) The x-axis coordinate difference between the outer membrane intersection point and its counterclockwise adjacent point x 5 , y-axis coordinate difference y 5 ,distance

[0082] 2. Myocardial pixel features: Take the points at the corresponding angles between the inner and outer membranes and the outer membrane, take the point O in the middle, and take out the 25 pixel matrices around point O, which represent the morphological characteristics of the myocardium.

[0083] 3. Expand the myocardial pixel features into a one-dimensional vector and splice it under the motion information. The feature information extracted from 120 angles is processed in this way, normalized after splicing, and forms a feature matrix. The number of rows in the feature matrix is ​​15+25=40 rows, and the number of columns is 120 (120 angles), so the size of the feature matrix is ​​40×120. Figure 4 It is the feature matrix diagram;

[0084] The method for extracting pixel features is to extract a 5×5 pixel block around the middle point of the inner and outer membranes, and then expand it and splice it after the motion feature. The alternative method can be to extract a pixel block of any size around the middle point of the inner and outer membranes, extract a pixel block of any size around the intersection of the myocardium and the endomyocytium, and extract a pixel block of any size around the intersection of the myocardium and the epicardium, which can all characterize the pixel features of the myocardial infarction area.

[0085] Building a deep learning model with ConvLSTM as the core

[0086] Figure 5 This is a diagram of the convolutional neural network structure with ConvLSTM as the core;

[0087] The established deep learning network structure is shown in the figure, which is mainly composed of ConvLSTM2D layer, Conv2D layer, Flatten layer, and Dense layer, as shown in the figure. For each CMRI movie sequence, the input feature matrix size is 20 (or 25) × 40 × 120. 20 (or 25) represents that each sequence has 20 (or 25) frames of pictures, and 40 × 120 is the size of the feature matrix extracted from a picture. The number of convolution kernels in the ConvLSTM2D layer is 20, and its size is (3, 3), so its output vector is still 20 (or 25) × 40 × 120. It should be noted that although the size of the first dimension of the input and output is 20, their meanings are different. The first dimension of the input is 20 (or 25), which means that the picture has 20 (or 25) frames, and the first dimension of the output is 20, which means that there are 20 convolution kernels to extract feature information. Each convolution kernel extracts information from the feature matrix sequence of 20 (or 25) frames through long short-term memory, so it can extract changes in position features, that is, motion features.

[0088] The output result is then batch normalized, which can accelerate convergence, control overfitting, and reduce the network's insensitivity to initialization weights. Since the training set data is small, a Dropout layer is added to further avoid overfitting. It is then input into the convolutional layer (Conv2D), whose number of convolution kernels is 1, size is (3, 3), and activation function is "relu". After further feature extraction, a 40×120 two-dimensional feature matrix is ​​obtained. Batch normalization is then performed again, and a Dropout is added. It is then expanded into one dimension through the flattening layer (Flatten), and its length is 4800. In order to meet the final output size, a fully connected layer (Dense) is added, and finally a 120-dimensional feature vector is output. The value of each dimension is between 0 and 1, 0 represents no infarction, and 1 represents infarction. Because it is a multi-label classification problem with multiple true values, "sigmoid" is used as the activation function so that the values ​​of the output 120 vectors are between 0 and 1.

[0089] The labels are classified according to the delayed enhancement (LGE) images, also with the center point of the outer membrane as the center. Under the guidance of the doctor, the corresponding 120 angles are marked to see if there is myocardial infarction. No infarction is marked as 0, and infarction is marked as 1. The size is 1×120.

[0090] During training, use the "Adam" optimizer and the binary cross entropy "binary_crossentropy" loss function, because each angle is a separate binary classification problem. The output vector between 0 and 1 is rounded to 0 or 1 to calculate the loss, and the data is iteratively trained to optimize the network parameters. When the training is completed, the accuracy is tested on the test set and the model is saved.

[0091] After the ConvLSTM2D layer, any number of convolutional layers (Conv2D) and pooling layers (MaxPooling2D) can be added to further extract deep features.

[0092] The present invention also discloses a device for directly delineating a myocardial infarction area from a cardiac magnetic resonance image sequence based on deep learning, comprising:

[0093] Segmentation unit: obtain cardiac magnetic resonance film sequences, and segment the myocardium and epicardium of each frame in the cardiac magnetic resonance imaging (CMRI) film sequence through the U-Net model;

[0094] Feature extraction unit: Binarize the segmented myocardium and epicardium of each frame to obtain the myocardial boundary, extract the motion features and pixel features of each frame, splice and fuse them, and store them in a two-dimensional feature matrix;

[0095] Labeling unit: label the infarct area of ​​the corresponding sequence of LGE images with gadolinium contrast agent in the two-dimensional feature matrix and convert it into a 1×120 vector as the myocardial infarction area label;

[0096] Constructing a neural network model unit: based on the obtained two-dimensional feature matrix and the corresponding myocardial infarction area label, constructing a neural network model with ConvLSTM as the core, wherein the neural network model with ConvLSTM as the core includes a ConvLSTM layer, a convolution layer, an expansion layer, and a fully connected layer;

[0097] Training unit: When training in a neural network model with ConvLSTM as the core, the 1×120 vector output by the model is compared with the actual myocardial infarction area label, and binary cross entropy "binary_crossentropy" is used as the loss function;

[0098] Optimization unit: Use the "Adam" optimizer to iteratively train the network and optimize the network parameters. When the training is completed, the accuracy is tested on the test set and the model with the highest accuracy is saved.

[0099] Prediction of infarct area in new patients

[0100] After the above-mentioned deep learning network training is completed, that is, the weight value has been determined, the CMRI film sequence of the new patient is preprocessed, and then the inner and outer myocardium can be segmented through the U-Net network, and the feature matrix of the sequence can be extracted. The feature matrix is ​​used as the input of the deep learning network. The calculated feature vector represents the area of ​​infarction. To visualize this area, the 1×120 vector composed of the output 0 and 1 corresponds to the color of the myocardial contour of this part. If the vector value corresponding to the angle is 1, the myocardial contour is displayed in white, indicating that there is infarction in the area; if the vector value corresponding to the angle is 0, the myocardial contour is not displayed, indicating that there is no infarction in the area. Doctors can use the displayed image to assist in judging the scope of myocardial infarction. Figure 6 e is a cardiac MRI image of delayed gadolinium enhancement (LGE); Figure 6 The white area in f is the myocardial infarction area marked on the LGE image under the guidance of an experienced physician; Figure 6 The white edge range in g is the myocardial infarction area segmented from the corresponding sequence without contrast agent, which is consistent with that with contrast agent.

[0101] The present invention has been tested experimentally.

[0102] At present, there are 233 CMRI film sequences of patients with confirmed myocardial infarction. Under the guidance of doctors from Zhejiang University Run Run Shaw Hospital, we have accurately labeled the myocardial infarction area. 40 sequences were randomly selected as test sets, 39 sequences as validation sets, and 154 sequences as training sets. Using the Keras+TensorFlow architecture, 40 rounds of training on a Quadro RTX 8000 GPU took about 120 seconds, and the test set accuracy can reach about 80%. Using the model to predict new sequences, it takes about 6ms for one picture, which is very fast and has good real-time performance.

[0103] Judging from the test accuracy, this method has certain practicality and can be used as an auxiliary means for doctors to diagnose myocardial infarction.

[0104] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for directly delineating myocardial infarction areas from cardiac magnetic resonance image sequences based on deep learning, characterized in that: include: Obtain cardiac magnetic resonance movie sequences, and segment the myocardium and epicardium of each frame in the cardiac magnetic resonance imaging (CMRI) movie sequence using the U-Net model; Binarize the segmented myocardium and epicardium of each frame to obtain the myocardial boundary, extract the motion features and pixel features of each frame, splice and fuse them, and store them in a two-dimensional feature matrix; The infarcted area of ​​the corresponding sequence of LGE images with gadolinium contrast agent in the two-dimensional feature matrix is ​​marked and converted into a 1×120 vector as the myocardial infarction area label; Based on the obtained two-dimensional feature matrix and the corresponding myocardial infarction area label, a neural network model with ConvLSTM as the core is constructed, wherein the neural network model with ConvLSTM as the core includes a ConvLSTM layer, a convolution layer, an expansion layer, and a fully connected layer; When training the ConvLSTM-based neural network model, the 1×120 vector output by the model is compared with the true myocardial infarction region label, and binary cross entropy "binary_crossentropy" is used as the loss function; Use the "Adam" optimizer to iteratively train the network and optimize the network parameters. When the training is finished, test the accuracy on the test set and save the model with the highest accuracy. The extraction of motion features of each frame is specifically as follows: The midpoint of the myocardial epicardium boundary is extracted as the center point of the entire image. Rays are drawn from the center point outward at an angle interval of 3 degrees, and the intersection points with the myocardial endomyocytium and myocardial epicytium boundaries are taken to obtain the coordinates of the intersection points of the myocardium and the myocardium in 120 directions. The extracted motion features are: 1) The x-axis coordinate difference x1 and y-axis coordinate difference y1 between the inner membrane intersection point and the center point of the image, and the distance 2) The x-axis coordinate difference x2 and y-axis coordinate difference y2 between the intersection of the outer membrane and the center of the image, and the distance 3) The x-axis coordinate difference x3 and y-axis coordinate difference y3 of the intersection point between the inner and outer membranes, and the distance 4) The x-axis coordinate difference x4 and the y-axis coordinate difference y4 between the inner membrane intersection and its counterclockwise adjacent point, and the distance 5) The x-axis coordinate difference x5 and the y-axis coordinate difference y5 of the outer membrane intersection and its counterclockwise adjacent point, the distance The U-Net model is used to segment the myocardium and epicardium, extract the motion and grayscale feature information of the myocardium, build a neural network with ConvLSTM as the core, and train the neural network and present the myocardial infarction area. The extraction of pixel features of each frame is specifically as follows: taking points at corresponding angles between the inner and outer membranes, taking point O, and taking out 25 pixel matrices around point O; The splicing and fusion is to expand the myocardial pixel features into a one-dimensional vector and splice it under the myocardial motion features; the infarction area of ​​the LGE image with gadolinium contrast agent in the corresponding sequence in the annotated two-dimensional feature matrix is ​​converted into a 1×120 vector as a myocardial infarction area label. Specifically, under the guidance of a doctor, the pixel-level myocardial infarction range is first marked on the LGE image, and then the myocardium is divided into 120 parts at equal angles with the center point of the outer membrane as the center and rays are made outward at an angle interval of 3 degrees. The pixel-level myocardial infarction range corresponds to whether there is myocardial infarction in each of the 120 equally divided myocardiums. No infarction is marked as 0, and infarction is marked as 1, thereby converting it into a 1×120 vector as a myocardial infarction area label.

2. The method for directly delineating myocardial infarction areas from cardiac magnetic resonance image sequences based on deep learning according to claim 1, characterized in that: The input of the neural network model with ConvLSTM as the core is the extracted feature matrix, and the output of the model is a 1×120 vector.

3. The method for directly delineating myocardial infarction area from cardiac magnetic resonance image sequences based on deep learning according to claim 2, characterized in that: A 120-dimensional vector is output from the neural network with ConvLSTM as the core. The value of each dimension is between 0 and 1, where 0 represents no infarction and 1 represents infarction.

4. The method for directly delineating myocardial infarction area from cardiac magnetic resonance image sequences based on deep learning according to claim 1, 2 or 3, characterized in that: The U-Net model is used to segment the endomyocardium and epicycium after preprocessing the cardiac magnetic resonance imaging movie sequence. The preprocessing is to obtain the coordinates of the heart area by training YOLO-v4, then capture the image according to the coordinates, and then rotate and straighten the image.

5. The method for directly delineating myocardial infarction area from cardiac magnetic resonance image sequences based on deep learning according to claim 4, characterized in that: The preprocessing is specifically as follows: first process the size of the image, cut out the area of ​​interest, record the coordinate values ​​of its four vertices as a training set, input it into the YOLO-v4 model, and after the training is completed, learn the characteristics of the heart area. After that, for each original image, it can give the coordinates of the four vertices of the area of ​​interest, cut out the image according to this coordinate, and then rotate and straighten the image so that the left ventricle and the right ventricle are on the same horizontal line in the image, and the preprocessing part is completed.

6. A device for directly delineating myocardial infarction areas from cardiac magnetic resonance image sequences based on deep learning, characterized in that: include: Segmentation unit: obtain cardiac magnetic resonance film sequences, and segment the myocardium and epicardium of each frame in the cardiac magnetic resonance imaging (CMRI) film sequence through the U-Net model; Feature extraction unit: Binarize the segmented myocardium and epicardium of each frame to obtain the myocardial boundary, extract the motion features and pixel features of each frame, splice and fuse them, and store them in a two-dimensional feature matrix; Labeling unit: label the infarct area of ​​the corresponding sequence of LGE images with gadolinium contrast agent in the two-dimensional feature matrix and convert it into a 1×120 vector as the myocardial infarction area label; Constructing a neural network model unit: based on the obtained two-dimensional feature matrix and the corresponding myocardial infarction area label, constructing a neural network model with ConvLSTM as the core, wherein the neural network model with ConvLSTM as the core includes a ConvLSTM layer, a convolution layer, an expansion layer, and a fully connected layer; Training unit: When training in a neural network model with ConvLSTM as the core, the 1×120 vector output by the model is compared with the real myocardial infarction area label, and binary cross entropy "binary_crossentropy" is used as the loss function; Optimization unit: Use the "Adam" optimizer to iteratively train the network and optimize the network parameters. When the training is completed, the accuracy is tested on the test set and the model with the highest accuracy is saved. The extraction of motion features of each frame is specifically as follows: The midpoint of the myocardial epicardium boundary is extracted as the center point of the entire image. Rays are drawn from the center point outward at an angle interval of 3 degrees, and the intersection points with the myocardial endomyocytium and myocardial epicytium boundaries are taken to obtain the coordinates of the inner and outer membrane intersection points in 120 directions. The extracted motion features are: 1) The x-axis coordinate difference x1 and y-axis coordinate difference y1 between the inner membrane intersection point and the center point of the image, and the distance 2) The x-axis coordinate difference x2 and y-axis coordinate difference y2 between the intersection of the outer membrane and the center of the image, and the distance 3) The x-axis coordinate difference x3 and y-axis coordinate difference y3 of the intersection point between the inner and outer membranes, and the distance 4) The x-axis coordinate difference x4 and the y-axis coordinate difference y4 between the inner membrane intersection and its counterclockwise adjacent point, and the distance 5) The x-axis coordinate difference x5 and the y-axis coordinate difference y5 of the outer membrane intersection and its counterclockwise adjacent point, the distance The U-Net model is used to segment the myocardium and epicardium, extract the motion and grayscale feature information of the myocardium, build a neural network with ConvLSTM as the core, and train the neural network and present the myocardial infarction area. The extraction of pixel features of each frame is specifically as follows: taking points at corresponding angles between the inner and outer membranes, taking point O, and taking out 25 pixel matrices around point O; The splicing and fusion is to expand the myocardial pixel features into a one-dimensional vector and splice it under the myocardial motion features; The infarction area of ​​the LGE image with gadolinium contrast agent in the corresponding sequence in the annotated two-dimensional feature matrix is ​​converted into a 1×120 vector as a myocardial infarction area label. Specifically, the pixel-level myocardial infarction range is first marked on the LGE image under the guidance of a doctor, and then the myocardium is divided into 120 equal angles by making rays at an angle interval of 3 degrees with the center point of the outer membrane as the center. The pixel-level myocardial infarction range corresponds to whether there is myocardial infarction in each of the 120 equally divided myocardiums, and no infarction is marked as 0, and infarction is marked as 1, thereby converting it into a 1×120 vector as a myocardial infarction area label.

Images