Fully automatic left atrial myocardial strain analysis method based on deep learning

Abstract

The application discloses a kind of full-automatic left atrial myocardial strain analysis methods based on deep learning, belong to medical image processing field, comprising: constructing joint group registration and segmentation model, the joint group registration and segmentation model includes group registration encoder, segmentation encoder, inter-task attention module, group registration decoder, segmentation decoder, image deformation module and mask deformation module;Define loss function, the loss function includes registration loss, smoothing loss, segmentation loss and inter-task loss;Constructing cardiac cine image dataset, and based on the cardiac cine image dataset training and testing the joint group registration and segmentation model;For newly collected cardiac cine image sequence, the trained model is used to predict motion field sequence, inverse motion field sequence and mask sequence, and left atrial myocardial strain map, strain curve and strain rate curve are calculated.The application improves the accuracy, efficiency and repeatability of strain analysis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a full-automatic left atrial myocardial strain analysis method based on deep learning. BACKGROUND

[0002] Left atrial myocardial strain can be divided into total strain, passive strain and active strain, which respectively reflect the storage function, drainage function and pressure boosting function of the left atrium. Cardiac magnetic resonance cine imaging technology is widely used in myocardial strain measurement due to its high spatial resolution, good soft tissue contrast and no ionizing radiation. The existing cardiac magnetic resonance cine imaging technology is mainly based on balanced steady-state free precession sequence and spoiled gradient echo sequence. The images collected are usually two-dimensional image sequences, including short-axis plane, 2-chamber long-axis plane, 4-chamber long-axis plane and 3-chamber long-axis plane. Cardiovascular magnetic resonance feature tracking (CMR-FT) is a method for measuring myocardial strain from conventional cardiac cine images. It uses feature tracking algorithm to estimate the motion of left atrial myocardium from long-axis cine images, and combines with segmentation mask to calculate strain indicators. Compared with the traditional ultrasound speckle tracking method, CMR-FT has higher spatial resolution, larger imaging field of view and better repeatability. The existing CMR-FT method mainly uses optical flow or pairwise registration algorithm to estimate the motion field between adjacent frames, and obtains the motion field from the end-diastolic frame to each frame by combining these motion fields, and then calculates the strain.

[0003] However, the existing CMR-FT method has some limitations. The frame-by-frame tracking method will cause the tracking error to accumulate throughout the cardiac cycle, especially in the late diastolic phase, which is called drift effect, thereby reducing the accuracy of strain measurement. Another type of CMR-FT method chooses to directly estimate the large amplitude motion between the end-diastolic frame and the end-systolic frame, which may cause more serious tracking error. In addition, the existing CMR-FT method generally lacks anatomical structure level constraints, resulting in the estimated motion field often being inconsistent with the actual anatomy of the left atrial myocardium. Finally, the existing CMR-FT method cannot simultaneously automatically segment the left atrial myocardium, and the strain calculation still depends on time-consuming and subjective manual segmentation. SUMMARY

[0004] To solve the above technical problems, the application provides a full-automatic left atrial myocardial strain analysis method based on deep learning to solve the problems existing in the prior art.

[0005] To achieve the above purpose, in a first aspect, the application provides a full-automatic left atrial myocardial strain analysis method based on deep learning, comprising:

[0006] construct a joint group registration and segmentation model, the joint group registration and segmentation model comprising a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module, and a mask deformation module;

[0007] define a loss function, the loss function comprising a registration loss, a smoothing loss, a segmentation loss, and an inter-task loss;

[0008] construct a cardiac cine image dataset, and train and test the joint group registration and segmentation model based on the cardiac cine image dataset;

[0009] for a newly acquired cardiac cine image sequence, use the trained model to predict a motion field sequence, an inverse motion field sequence, and a mask sequence, and calculate a left atrial myocardial strain map, a strain curve, and a strain rate curve based on the motion field sequence, the inverse motion field sequence, and the mask sequence.

[0010] Preferably, the group registration encoder takes a cardiac cine image sequence as input, and extracts encoding features for the group registration task through convolution, attention, and pooling operations.

[0011] Preferably, the segmentation encoder takes a cardiac cine image sequence as input, and extracts encoding features for the segmentation task through convolution, attention, and pooling operations.

[0012] Preferably, the inter-task attention module is used to:

[0013] for the group registration task, taking the encoding features for the group registration task as target input and the encoding features for the segmentation task as auxiliary input, the inter-attention is used to calculate fusion features for the group registration task;

[0014] for the segmentation task, taking the encoding features for the segmentation task as target input and the encoding features for the group registration task as auxiliary input, the inter-attention is used to calculate fusion features for the segmentation task.

[0015] Preferably, the group registration decoder takes the fusion features for the group registration task as input, and calculates a velocity field sequence through convolution and upsampling operations, and performs scaling and squaring operations on the velocity field sequence to obtain a motion field sequence and an inverse motion field sequence.

[0016] Preferably, the segmentation decoder takes the fusion features for the segmentation task as input, and calculates a mask sequence through convolution and upsampling operations.

[0017] Preferably, the image deformation module takes the cardiac cine image sequence and the motion field sequence as input, and calculates a deformation image sequence through a deformation operator.

[0018] Preferably, the mask deformation module calculates a sequence of deformation masks by a deformation operator with the sequence of masks and the sequence of motion fields as inputs.

[0019] Preferably, the registration loss optimizes the group registration network by minimizing the difference between the sequence of deformation images.

[0020] The smoothness loss optimizes the group registration network by minimizing the spatial and temporal regularization terms of the sequence of motion fields.

[0021] The segmentation loss optimizes the segmentation network by minimizing the difference between the sequence of masks and the sequence of manually annotated labels.

[0022] The inter-task loss optimizes the group registration and segmentation networks by minimizing the difference between the sequence of deformation masks.

[0023] Preferably, the calculation of the left atrial myocardial strain comprises:

[0024] Mapping all pixels to the average coordinate system by the inverse motion field of the end diastolic frame, and then mapping the pixels in the average coordinate system to each frame by the motion field of each frame to obtain the sequence of motion fields from the end diastolic frame to each frame.

[0025] Calculating the longitudinal unit vector of each pixel in the left atrial myocardial region according to the end diastolic mask of the sequence of masks to form a direction field.

[0026] Combining the sequence of motion fields and the direction field, calculating the myocardial strain map, the strain curve and the strain rate curve.

[0027] Compared with the prior art, the present application has the following advantages and technical effects:

[0028] The present application provides a full-automatic left atrial myocardial strain analysis method based on deep learning, comprising: first, constructing a joint group registration and segmentation model, the joint group registration and segmentation model comprising a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module and a mask deformation module; second, defining a loss function, the loss function comprising a registration loss, a smoothness loss, a segmentation loss and an inter-task loss; third, constructing a cardiac cine image dataset, and training and testing the joint group registration and segmentation model based on the cardiac cine image dataset; and finally, for a newly acquired cardiac cine image sequence, using the trained model to predict a sequence of motion fields, a sequence of inverse motion fields and a sequence of masks, and calculating a left atrial myocardial strain map, a strain curve and a strain rate curve according to the sequence of motion fields, the sequence of inverse motion fields and the sequence of masks.

[0029] The application estimates the motion of left atrial myocardium by using group registration network, eliminates the drift effect caused by frame-by-frame tracking, and avoids large amplitude motion between end diastolic frame and end systolic frame, thereby overcoming the limitations of existing pair tracking algorithm.

[0030] The application adopts a multi-task framework of joint group registration and segmentation, can cooperatively optimize group registration and segmentation tasks in the training stage, and can realize automatic analysis of left atrial myocardial strain in the test stage, thereby significantly improving the accuracy, efficiency and repeatability of strain analysis. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their

[0032] Figure 1 Flow chart of the deep learning-based full-automatic left atrial myocardial strain analysis method of the embodiment of the application;

[0033] Figure 2 Algorithm principle diagram of the embodiment of the application;

[0034] Figure 3 Result schematic diagram of the embodiment of the application. DETAILED DESCRIPTION

[0035] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flow chart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0037] Embodiment one

[0038] As shown in the figure, the embodiment provides a deep learning-based full-automatic left atrial myocardial strain analysis method, which comprises: Figures 1-2

[0039] Step S1, a joint group registration and segmentation model is constructed, and the joint group registration and segmentation model comprises a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module and a mask deformation module;

[0040] ​1) Group registration encoder: takes the sequence of cardiac movie images as input variable, extracts the encoding features of the group registration task as output variable through a series of 3D convolution, max-pooling operations.

[0041] 2) Segmentation encoder: takes the sequence of cardiac movie images as input variable, extracts the encoding features of the segmentation task as output variable through a series of 3D convolution, max-pooling operations.

[0042] 3) Inter-task attention module: for the group registration task, the inter-task attention module takes the encoding features of the group registration task as target input variable, takes the encoding features of the segmentation task as auxiliary input variable, obtains the fusion features of the group registration task as output variable by calculating the mutual attention between the two input variables. For the segmentation task, the inter-task attention module takes the encoding features of the segmentation task as target input variable, takes the encoding features of the group registration task as auxiliary input variable, obtains the fusion features of the segmentation task as output variable by calculating the mutual attention between the two input variables.

[0043] Specifically, in each inter-task attention module, the feature vector at each element of the target input variable is mapped through two different linear mappings to form key and value respectively, and the feature vector at each element of the auxiliary input variable is mapped through a linear mapping to form query. The inner product between query and key is calculated and normalized as the attention weight, and the weight is used to weight the sum of value to obtain the output variable.

[0044] 4) Group registration decoder: takes the fusion features of the group registration task as input variable, calculates the sequence of velocity fields through a series of 3D convolution, upsampling operations, and then performs scaling and squaring operations on the sequence of velocity fields to obtain the sequence of motion fields and the sequence of inverse motion fields as output variables. In order to register all images to an average coordinate system, the mean value of the velocity field sequence in the time dimension is constrained to be zero.

[0045] 5) Segmentation decoder: takes the fusion features of the segmentation task as input variable, calculates the sequence of masks as output variable through a series of 3D convolution, upsampling operations.

[0046] 6) Image warping module: takes the sequence of cardiac movie images as the first input variable, takes the sequence of motion fields output by the group registration decoder as the second input variable, calculates the sequence of warped images as output variable through the warping operator.

[0047] 7) Mask warping module: takes the sequence of masks output by the segmentation decoder as the first input variable, takes the sequence of motion fields output by the group registration decoder as the second input variable, calculates the sequence of warped masks as output variable through the warping operator.

[0048] Step S2, defining a loss function including a registration loss, a smoothing loss, a segmentation loss and an inter-task loss;

[0049] 1) Registration loss: taking the warped image sequence output by the image warping module as the input variable, the group registration network is optimized by minimizing the difference between each frame of warped image. The difference function adopted can be pixel-wise signal variance, global low-rank difference measure, local low-rank difference measure, etc.

[0050] The calculation method is to flatten each frame of warped image into a column vector, and arrange it into a Casorati matrix in time sequence, and then take the nuclear norm of the Casorati matrix.

[0051] 2) Smoothing loss: taking the motion field sequence output by the group registration decoder as the input variable, the motion field sequence is prompted to change as smoothly as possible in the spatial and temporal dimensions by minimizing the spatial and temporal regular terms of the motion field sequence. The regular term adopted can be a smoothing regular term based on first or second derivative.

[0052] Specifically, the spatial regular term adopted is the sum of squares of the second-order spatial derivative of the motion field sequence, and the temporal regular term adopted is the sum of squares of the second-order temporal derivative of the motion field.

[0053] 3) Segmentation loss: taking the mask sequence output by the segmentation decoder as the first input variable and the manually labeled label sequence as the second input variable, the segmentation network is optimized by minimizing the difference between the mask and the label. The difference function adopted can be cross-entropy, Dice loss, Focal loss, etc., which is used to measure the degree of overlap of left atrial myocardium in a pair of masks.

[0054] 4) Inter-task loss: taking the warped mask sequence output by the mask warping module as the input variable, the group registration and segmentation network are constrained by minimizing the difference between each frame of mask. For the group registration network, the structural consistency constraint of the left atrial myocardium can improve the anatomical rationality of the motion field; for the segmentation network, the temporal smoothness constraint of the motion field sequence can enhance the dynamic continuity of the mask sequence. The difference function adopted is Groupwise Dice loss, which is used to measure the degree of overlap of left atrial myocardium in multiple masks.

[0055] Step S3, constructing a cardiac cine image dataset, and training and testing the joint group registration and segmentation model based on the cardiac cine image dataset;

[0056] Specifically, the long-axis cardiac cine image sequence of the subject is collected, and the following processing flow is performed:

[0057] 1) The spatial resolution is unified to 1.0x1.0 mm by resampling 2 ;

[0058] 2) Crop the image to 160x160 pixels;

[0059] 3) Normalize the image gray value to 0 to 1;

[0060] 4) Manually annotate the left atrial myocardium to generate the segmentation label. The processed data is randomly divided into training set, validation set and test set in the ratio of 7:1:2.

[0061] In the training stage, the parameters of the joint group registration and segmentation model are learned by minimizing the loss function of the training set through the Adam algorithm until the loss function of the validation set no longer decreases. In the test stage, the performance of the model is evaluated on the test set to test its generalization performance.

[0062] Step S4: For the newly collected cardiac cine image sequence, the motion field sequence, the inverse motion field sequence and the segmentation mask sequence are predicted by the trained model, and the left atrial myocardial strain map, the strain curve and the strain rate curve are calculated according to the motion field sequence, the inverse motion field sequence and the mask sequence.

[0063] The calculation process of the left atrial myocardial strain is as follows:

[0064] 1) Map all pixels to the average coordinate system through the inverse motion field of the end diastolic frame, and then map the pixels in the average coordinate system to each frame through the motion field of each frame to obtain the motion field sequence from the end diastolic frame to each frame;

[0065] 2) Calculate the longitudinal unit vector of each pixel in the left atrial myocardial region according to the end diastolic mask of the mask sequence to form a direction field;

[0066] 3) Combine the above motion field sequence and direction field, calculate the strain value at each myocardial pixel according to the expression of myocardial strain, and construct a myocardial strain map. Take the average of the strain values in the whole left atrial myocardial region to obtain the curve of the global myocardial strain changing with time. Derive the global myocardial strain along the time dimension to obtain the curve of the myocardial strain rate changing with time.

[0067] Figure 3The results of this embodiment are shown in the following table. The first and second columns are the results of left atrial myocardial strain analysis by two comparative methods; the third column is the results of left atrial myocardial strain analysis by the method of the present application. The first row is the strain map at the end-systole; the second row is the strain map at the late diastole (the last frame); the third row is the strain curve; and the fourth row is the strain rate curve. As indicated by the arrows, the end-systole strain maps generated by the two comparative methods both exhibit poor spatial smoothness, which does not conform to the physiological movement of the myocardium. Since the cine images of the heart are periodic, the strain value at the last frame should theoretically tend to zero. However, as indicated by the arrows, there is a significant non-zero strain region in the late diastole strain map generated by comparative method 2, which indicates that the accuracy of the diastolic strain determined by comparative method 2 is poor. In contrast, the strain map obtained by the method of the present application exhibits good spatial smoothness, and the strain value at the late diastole is close to zero. The strain and strain rate curves estimated by the three methods exhibit similar trends. The strain and strain rate values determined by the method of the present application and comparative method 1 are relatively close, while the values determined by comparative method 2 are significantly lower.

[0068] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions easily conceived by those skilled in the art within the technical scope disclosed by the present application should be encompassed within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A fully automated method for left atrial myocardial strain analysis based on deep learning, characterized in that, Includes the following steps: A joint group registration and segmentation model is constructed, which includes a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module, and a mask deformation module. The group registration encoder takes a sequence of cardiac cinema images as input, and the segmentation encoder takes a sequence of cardiac cinema images as input. The inter-task attention module is used for: For the group registration task, the encoded features of the group registration task are used as the target input, and the encoded features of the segmentation task are used as the auxiliary input. The fusion features of the group registration task are obtained through mutual attention calculation. For the segmentation task, the encoded features of the segmentation task are used as the target input, and the encoded features of the group registration task are used as the auxiliary input. The fusion features of the segmentation task are obtained by mutual attention calculation. The mutual attention calculation is as follows: the feature vector of each element of the target input variable is transformed into key and value through two different linear mappings, and the feature vector of each element of the auxiliary input variable is transformed into query through a linear mapping. The inner product between query and key is calculated and normalized as attention weight. Then, the value is weighted and summed to obtain the fused feature. Define a loss function, which includes registration loss, smoothing loss, segmentation loss, and inter-task loss; The inter-task loss minimizes the differences between deformable mask sequences while constraining the group registration and segmentation networks, and the difference function adopted is the Groupwise Dice loss. Construct a cardiac film image dataset, and train and test the joint group registration and segmentation model based on the cardiac film image dataset; For newly acquired cardiac cine image sequences, a trained model is used to predict motion field sequences, inverse motion field sequences, and mask sequences. Based on the motion field sequences, inverse motion field sequences, and mask sequences, the strain map, strain curve, and strain rate curve of the left atrial myocardium are calculated.

2. The method according to claim 1, characterized in that, The group registration encoder extracts the coding features of the group registration task through convolution, attention, and pooling operations.

3. The method according to claim 1, characterized in that, The segmentation encoder extracts the encoded features of the segmentation task through convolution, attention, and pooling operations.

4. The method according to claim 1, characterized in that, The group registration decoder takes the fused features of the group registration task as input, calculates the velocity field sequence through convolution and upsampling operations, and performs scaling and squaring operations on the velocity field sequence to obtain the motion field sequence and the inverse motion field sequence.

5. The method according to claim 4, characterized in that, The segmentation decoder takes the fused features of the segmentation task as input and calculates the mask sequence through convolution and upsampling operations.

6. The method according to claim 5, characterized in that, The image deformation module takes the cardiac film image sequence and the motion field sequence as input and calculates the deformed image sequence through deformation operators.

7. The method according to claim 6, characterized in that, The mask deformation module takes the mask sequence and the motion field sequence as input and calculates the deformation mask sequence through the deformation operator.

8. The method according to claim 7, characterized in that, The registration loss is optimized by minimizing the differences between deformed image sequences to improve the group registration network; The smoothing loss optimizes the group registration network by minimizing the spatial and temporal regularization terms of the motion field sequence; The segmentation loss optimizes the segmentation network by minimizing the difference between the mask sequence and the manually labeled sequence.

9. The method according to claim 8, characterized in that, The calculation of left atrial myocardial strain includes: By mapping all pixels to the average coordinate system through the inverse motion field of the end-diastolic frame, and then mapping the pixels in the average coordinate system to each frame through the motion field of each frame, a motion field sequence from the end-diastolic frame to each frame is obtained. The longitudinal unit vector of each pixel in the left atrial myocardial region is calculated based on the end-diastolic frame mask in the mask sequence to form a direction field; Based on the motion field sequence and direction field, myocardial strain diagram, strain curve and strain rate curve are calculated.

Images