Full-automatic left atrial myocardial strain analysis method based on deep learning

Abstract

The invention discloses a full-automatic left atrial myocardial strain analysis method based on deep learning, and belongs to the field of medical image processing. The combined group registration and segmentation model comprises a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module and a mask deformation module; defining a loss function, wherein the loss function comprises registration loss, smoothing loss, segmentation loss and inter-task loss; constructing a heart movie image data set, and training and testing the joint group registration and segmentation model based on the heart movie image data set; and for a newly collected heart film image sequence, predicting a motion field sequence, an inverse motion field sequence and a mask sequence by using the trained model, and calculating a left atrial myocardial strain graph, a strain curve and a strain rate curve. According to the invention, the accuracy, efficiency and repeatability of strain analysis are improved.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a fully automated method for left atrial myocardial strain analysis based on deep learning. Background Technology

[0002] Left atrial myocardial strain can be categorized into total strain, passive strain, and active strain, reflecting the left atrium's storage, drainage, and pressurization functions, respectively. Cardiac magnetic resonance cine imaging (MRI) is widely used for myocardial strain measurement due to its high spatial resolution, good soft tissue contrast, and lack of ionizing radiation. Existing cardiac MRI techniques are primarily based on balanced steady-state free precession sequences and disruptive gradient echo sequences. The acquired images are typically two-dimensional image sequences, including short-axis, two-chamber, four-chamber, and three-chamber long-axis planes. Cardiovascular magnetic resonance feature tracking (CMR-FT) is a method for determining myocardial strain from conventional cardiac cine images. It utilizes feature tracking algorithms to estimate left atrial myocardial motion from long-axis cine images and combines this with segmentation masks to calculate strain parameters. Compared to traditional ultrasound speckle tracking methods, CMR-FT offers higher spatial resolution, a larger imaging field of view, and better repeatability. Existing CMR-FT methods mainly use optical flow or pairwise registration algorithms to estimate the motion field between adjacent frames. By combining these motion fields, the motion field from the end-diastolic frame to each frame is obtained, and then the strain is calculated.

[0003] However, existing CMR-FT methods have some limitations. Frame-by-frame tracking leads to the accumulation of tracking errors throughout the cardiac cycle, especially in late diastole; this phenomenon, known as the drift effect, reduces the accuracy of strain measurements. Another class of CMR-FT methods chooses to directly estimate the large-amplitude motion between the end-diastolic and end-systolic frames, which can result in even more severe tracking errors. Furthermore, existing CMR-FT methods generally lack anatomical constraints, causing the estimated motion field to frequently deviate from the actual anatomy of the left atrial myocardium. Finally, existing CMR-FT methods cannot simultaneously and automatically segment the left atrial myocardium; strain calculation still relies on time-consuming and subjective manual segmentation. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a fully automated left atrial myocardial strain analysis method based on deep learning, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a fully automated method for left atrial myocardial strain analysis based on deep learning, comprising:

[0006] A joint group registration and segmentation model is constructed, which includes a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module, and a mask deformation module.

[0007] Define a loss function, which includes registration loss, smoothing loss, segmentation loss, and inter-task loss;

[0008] Construct a cardiac film image dataset, and train and test the joint group registration and segmentation model based on the cardiac film image dataset;

[0009] For newly acquired cardiac cine image sequences, a trained model is used to predict motion field sequences, inverse motion field sequences, and mask sequences. Based on the motion field sequences, inverse motion field sequences, and mask sequences, the strain map, strain curve, and strain rate curve of the left atrial myocardium are calculated.

[0010] Preferably, the group registration encoder takes a sequence of cardiac film images as input and extracts the coding features for the group registration task through convolution, attention, and pooling operations.

[0011] Preferably, the segmentation encoder takes a sequence of heart movie images as input and extracts the encoded features for the segmentation task through convolution, attention, and pooling operations.

[0012] Preferably, the inter-task attention module is used for:

[0013] For the group registration task, the encoded features of the group registration task are used as the target input, and the encoded features of the segmentation task are used as the auxiliary input. The fusion features of the group registration task are obtained through mutual attention calculation.

[0014] For the segmentation task, the encoded features of the segmentation task are used as the target input, and the encoded features of the group registration task are used as the auxiliary input. The fused features of the segmentation task are obtained by mutual attention calculation.

[0015] Preferably, the group registration decoder takes the fusion features of the group registration task as input, calculates the velocity field sequence through convolution and upsampling operations, and performs scaling and squaring operations on the velocity field sequence to obtain the motion field sequence and the inverse motion field sequence.

[0016] Preferably, the segmentation decoder takes the fused features of the segmentation task as input and calculates the mask sequence through convolution and upsampling operations.

[0017] Preferably, the image deformation module takes the cardiac film image sequence and the motion field sequence as input and calculates the deformed image sequence through a deformation operator.

[0018] Preferably, the mask deformation module takes the mask sequence and the motion field sequence as input and calculates the deformation mask sequence through the deformation operator.

[0019] Preferably, the registration loss is optimized by minimizing the differences between deformed image sequences to optimize the group registration network;

[0020] The smoothing loss optimizes the group registration network by minimizing the spatial and temporal regularization terms of the motion field sequence;

[0021] The segmentation loss optimizes the segmentation network by minimizing the difference between the mask sequence and the manually labeled sequence;

[0022] The inter-task loss is achieved by minimizing the differences between deformable mask sequences, while simultaneously constraining the group registration and segmentation networks.

[0023] Preferably, the calculation of the left atrial myocardial strain includes:

[0024] By mapping all pixels to the average coordinate system through the inverse motion field of the end-diastolic frame, and then mapping the pixels in the average coordinate system to each frame through the motion field of each frame, a motion field sequence from the end-diastolic frame to each frame is obtained.

[0025] The longitudinal unit vector of each pixel in the left atrial myocardial region is calculated based on the end-diastolic frame mask of the mask sequence to form a direction field;

[0026] Based on the motion field sequence and direction field, myocardial strain diagram, strain curve and strain rate curve are calculated.

[0027] Compared with the prior art, the present invention has the following advantages and technical effects:

[0028] This invention provides a fully automated left atrial myocardial strain analysis method based on deep learning, comprising: first, constructing a joint group registration and segmentation model, wherein the joint group registration and segmentation model includes a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module, and a mask deformation module; second, defining a loss function, wherein the loss function includes registration loss, smoothing loss, segmentation loss, and inter-task loss; then, constructing a cardiac cine image dataset, and training and testing the joint group registration and segmentation model based on the cardiac cine image dataset; finally, for newly acquired cardiac cine image sequences, using the trained model to predict motion field sequences, inverse motion field sequences, and mask sequences, and calculating left atrial myocardial strain maps, strain curves, and strain rate curves based on the motion field sequences, inverse motion field sequences, and mask sequences.

[0029] This invention utilizes a group registration network to estimate the motion of the left atrial myocardium, which not only eliminates the drift effect caused by frame-by-frame tracking, but also avoids directly estimating the large-amplitude motion between the end-diastolic and end-systolic frames, thus overcoming the limitations of existing pairwise tracking algorithms.

[0030] This invention employs a multi-task framework that combines group registration and segmentation, which not only enables collaborative optimization of group registration and segmentation tasks during the training phase, but also allows for automatic analysis of left atrial myocardial strain during the testing phase, thereby significantly improving the accuracy, efficiency, and repeatability of strain analysis. Attached Figure Description

[0031] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 This is a flowchart of a fully automated left atrial myocardial strain analysis method based on deep learning, according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the algorithm principle in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the results of an embodiment of the present invention. Detailed Implementation

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0037] Example 1

[0038] like Figures 1-2 As shown, this embodiment provides a fully automated left atrial myocardial strain analysis method based on deep learning, including:

[0039] Step S1: Construct a joint group registration and segmentation model, which includes a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module, and a mask deformation module.

[0040] 1) Group registration encoder: Using cardiac movie image sequences as input variables, it extracts the coding features of the group registration task as output variables through a series of 3D convolution and max pooling operations.

[0041] 2) Segmentation encoder: Using a sequence of heart movie images as input variables, it extracts the encoding features of the segmentation task as output variables through a series of 3D convolution and max pooling operations.

[0042] 3) Inter-task Attention Module: For the group registration task, the inter-task attention module uses the encoded features of the group registration task as the target input variable and the encoded features of the segmentation task as the auxiliary input variable. It calculates the mutual attention between the two input variables to obtain the fusion features of the group registration task as the output variable. For the segmentation task, the inter-task attention module uses the encoded features of the segmentation task as the target input variable and the encoded features of the group registration task as the auxiliary input variable. It calculates the mutual attention between the two input variables to obtain the fusion features of the segmentation task as the output variable.

[0043] Specifically, in each inter-task attention module, the feature vector at each element of the target input variable is transformed into the key and value through two different linear mappings, respectively. The feature vector at each element of the auxiliary input variable is transformed into the query through a single linear mapping. The inner product between the query and the key is calculated and normalized to serve as the attention weight. Then, the value is weighted and summed using this weight to obtain the output variable.

[0044] 4) Group Registration Decoder: Using the fused features of the group registration task as input variables, it calculates the velocity field sequence through a series of 3D convolutions and upsampling operations. Then, it performs scaling and squaring operations on the velocity field sequence to obtain the motion field sequence and inverse motion field sequence as output variables. In order to register all images to a single average coordinate system, the mean of the velocity field sequence is constrained to zero in the time dimension.

[0045] 5) Segment Decoder: It takes the fused features of the segmentation task as input variables and calculates the mask sequence as output variables through a series of 3D convolution and upsampling operations.

[0046] 6) Image Deformation Module: The first input variable is the cardiac film image sequence, the second input variable is the motion field sequence output by the group registration decoder, and the output variable is the deformed image sequence calculated by the deformation operator.

[0047] 7) Mask Deformation Module: The mask sequence output by the segment decoder is used as the first input variable, and the motion field sequence output by the group registration decoder is used as the second input variable. The deformed mask sequence is calculated by the deformation operator and used as the output variable.

[0048] Step S2: Define the loss function, which includes registration loss, smoothing loss, segmentation loss, and inter-task loss;

[0049] 1) Registration Loss: Using the deformed image sequence output by the image deformation module as input variable, the group registration network is optimized by minimizing the differences between each frame of deformed images. The difference function adopted can be pixel-wise signal variance, global low-rank difference measure, local low-rank difference measure, etc.

[0050] The calculation method is as follows: flatten each frame of deformed image into a column vector, arrange them into a Cascoati matrix in chronological order, and then take the nuclear norm of the Cascoati matrix.

[0051] 2) Smoothing Loss: Using the motion field sequence output by the group registration decoder as the input variable, the spatial and temporal regularization terms of the motion field sequence are minimized to make the changes in the spatial and temporal dimensions of the motion field sequence as smooth as possible. The regularization terms adopted can be smoothing regularization terms based on the first or second derivative.

[0052] Specifically, the spatial regularization term is the sum of squares of the second spatial derivatives of the motion field sequence, and the temporal regularization term is the sum of squares of the second temporal derivatives of the motion field.

[0053] 3) Segmentation Loss: Using the mask sequence output by the segmentation decoder as the first input variable and the manually labeled sequence as the second input variable, the segmentation network is optimized by minimizing the difference between the mask and the label. The difference function used can be cross-entropy, Dice loss, Focal loss, etc., which is used to measure the degree of overlap of the left atrial myocardium in a pair of masks.

[0054] 4) Inter-task loss: Using the deformed mask sequence output by the mask deformation module as input variables, the group registration and segmentation networks are constrained by minimizing the differences between masks in each frame. For the group registration network, the structural consistency constraint of the left atrial myocardium can improve the anatomical rationality of the motion field; for the segmentation network, the temporal smoothness constraint of the motion field sequence can enhance the dynamic continuity of the mask sequence. The difference function adopted is the Groupwise Dice loss, which is used to measure the degree of overlap of the left atrial myocardium in multiple masks.

[0055] Step S3: Construct a cardiac film image dataset, and train and test the joint group registration and segmentation model based on the cardiac film image dataset;

[0056] Specifically, long-axis cardiac cine image sequences were acquired from the subjects, and the following processing procedure was performed:

[0057] 1) Unify spatial resolution to 1.0 × 1.0 mm through resampling. 2 ;

[0058] 2) Crop the image to 160×160 pixels;

[0059] 3) Normalize the image grayscale values ​​to between 0 and 1;

[0060] 4) Manually label the left atrial myocardium to generate segmentation labels. Randomly divide the processed data into training, validation, and test sets in a 7:1:2 ratio.

[0061] During the training phase, the parameters of the joint group registration and segmentation model are learned by minimizing the training set loss function using the Adam algorithm until the validation set loss function no longer decreases. During the testing phase, the model's performance is evaluated on the test set to verify its generalization ability.

[0062] Step S4: For the newly acquired cardiac cine image sequence, predict the motion field sequence, inverse motion field sequence, and segmentation mask sequence using the trained model. Based on the motion field sequence, inverse motion field sequence, and mask sequence, calculate the left atrial myocardial strain map, strain curve, and strain rate curve.

[0063] The calculation process for left atrial myocardial strain is as follows:

[0064] 1) Map all pixels to the average coordinate system through the inverse motion field of the end-diastolic frame, and then map the pixels in the average coordinate system to each frame through the motion field of each frame to obtain the motion field sequence from the end-diastolic frame to each frame.

[0065] 2) Calculate the vertical unit vector of each pixel in the left atrial myocardial region based on the end-diastolic frame mask of the mask sequence to form a direction field;

[0066] 3) Combining the above motion field sequence and direction field, calculate the strain value at each myocardial pixel according to the expression for myocardial strain to construct a myocardial strain map. Average the strain values ​​across the entire left atrial myocardial region to obtain a curve showing the global myocardial strain over time. Differentiate the global myocardial strain along the time dimension to obtain a curve showing the myocardial strain rate over time.

[0067] Figure 3This is a schematic diagram of the results of this embodiment. The first and second columns show the left atrial myocardial strain analysis results of the two comparative methods; the third column shows the left atrial myocardial strain analysis results of the method of this invention. The first row is the strain map at end-systole; the second row is the strain map at late diastole (the last frame); the third row is the strain curve; and the fourth row is the strain rate curve. As indicated by the arrows, the end-systolic strain maps generated by both comparative methods exhibit poor spatial smoothness, which does not conform to the physiological motion of the myocardium. Since cardiac cine images are periodic, the strain value of the last frame should theoretically approach zero. However, as indicated by the arrows, the late diastolic strain map generated by comparative method 2 has a significant non-zero strain region, indicating that its measured diastolic strain accuracy is poor. In contrast, the strain map obtained by the method of this invention exhibits good spatial smoothness, and the strain value at late diastole is close to zero. The strain and strain rate curves estimated by the three methods show similar trends. The strain and strain rate values ​​measured by the method of this invention are closer to those of comparative method 1, while the values ​​measured by comparative method 2 are significantly lower in comparison.

[0068] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fully automated method for left atrial myocardial strain analysis based on deep learning, characterized in that, Includes the following steps: A joint group registration and segmentation model is constructed, which includes a group registration encoder, a segmentation encoder, an inter-task attention module, a group registration decoder, a segmentation decoder, an image deformation module, and a mask deformation module. Define a loss function, which includes registration loss, smoothing loss, segmentation loss, and inter-task loss; Construct a cardiac film image dataset, and train and test the joint group registration and segmentation model based on the cardiac film image dataset; For newly acquired cardiac cine image sequences, a trained model is used to predict motion field sequences, inverse motion field sequences, and mask sequences. Based on the motion field sequences, inverse motion field sequences, and mask sequences, the strain map, strain curve, and strain rate curve of the left atrial myocardium are calculated.

2. The method according to claim 1, characterized in that, The group registration encoder takes a sequence of cardiac film images as input and extracts the coding features for the group registration task through convolution, attention, and pooling operations.

3. The method according to claim 1, characterized in that, The segmentation encoder takes a sequence of heart-shaped movie images as input and extracts the encoded features for the segmentation task through convolution, attention, and pooling operations.

4. The method according to claim 1, characterized in that, The inter-task attention module is used for: For the group registration task, the encoded features of the group registration task are used as the target input, and the encoded features of the segmentation task are used as the auxiliary input. The fusion features of the group registration task are obtained through mutual attention calculation. For the segmentation task, the encoded features of the segmentation task are used as the target input, and the encoded features of the group registration task are used as the auxiliary input. The fused features of the segmentation task are obtained by mutual attention calculation.

5. The method according to claim 1, characterized in that, The group registration decoder takes the fused features of the group registration task as input, calculates the velocity field sequence through convolution and upsampling operations, and performs scaling and squaring operations on the velocity field sequence to obtain the motion field sequence and the inverse motion field sequence.

6. The method according to claim 5, characterized in that, The segmentation decoder takes the fused features of the segmentation task as input and calculates the mask sequence through convolution and upsampling operations.

7. The method according to claim 6, characterized in that, The image deformation module takes the cardiac film image sequence and the motion field sequence as input and calculates the deformed image sequence through deformation operators.

8. The method according to claim 7, characterized in that, The mask deformation module takes the mask sequence and the motion field sequence as input and calculates the deformation mask sequence through the deformation operator.

9. The method according to claim 8, characterized in that, The registration loss is optimized by minimizing the differences between deformed image sequences to improve the group registration network; The smoothing loss optimizes the group registration network by minimizing the spatial and temporal regularization terms of the motion field sequence; The segmentation loss optimizes the segmentation network by minimizing the difference between the mask sequence and the manually labeled sequence; The inter-task loss is achieved by minimizing the differences between deformable mask sequences, while simultaneously constraining the group registration and segmentation networks.

10. The method according to claim 9, characterized in that, The calculation of left atrial myocardial strain includes: By mapping all pixels to the average coordinate system through the inverse motion field of the end-diastolic frame, and then mapping the pixels in the average coordinate system to each frame through the motion field of each frame, a motion field sequence from the end-diastolic frame to each frame is obtained. The longitudinal unit vector of each pixel in the left atrial myocardial region is calculated based on the end-diastolic frame mask in the mask sequence to form a direction field; Based on the motion field sequence and direction field, myocardial strain diagram, strain curve and strain rate curve are calculated.

Images