A myocardial image segmentation and classification method based on multi-task learning

Abstract

The present invention relates to a myocardial image segmentation and classification method based on multi-task learning, in which a preprocessed cardiac MRI image is sent to a cross-task feature subnet, which uses cascaded deep separation convolution to extract multi-scale features and train shared parameters. Then, the obtained feature maps are sent to the segmentation subnet and the classification subnet for separate training. The segmentation subnet uses multiple long jump connections to send feature maps of different levels and scales in the cross-task feature subnet to a joint pyramid upsampling module for multi-scale feature fusion and upsampling, and obtains a myocardial segmentation mask with the same size as the original image; the classification subnet uses a residual attention mechanism to extract the deep features required for classification, effectively classify the cardiac MRI images, and obtain the diagnosis result of dilated cardiomyopathy.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a myocardial image segmentation and classification method based on multi-task learning. Background Art

[0002] Dilated cardiomyopathy is a common chronic heart disease and the most common type of cardiomyopathy in the world, accounting for 90% of all cardiomyopathy cases. It can lead to heart failure, thromboembolism and sudden death. Cardiac MRI is currently the only imaging examination method that can non-invasively evaluate necrotic myocardium, myocardial fibrosis and epileptic scar components in clinical practice. It can accurately determine the stage of disease development and has gradually become the preferred method for diagnosing cardiomyopathy.

[0003] In the diagnosis of dilated cardiomyopathy, left ventricular volume and left ventricular ejection fraction are the main indicators for evaluating the severity of the disease in patients with dilated cardiomyopathy, and ventricular and myocardial segmentation are important prerequisites for calculating these indicators. In clinical practice, the conventional analysis methods for dilated cardiomyopathy include single-point sampling method and regional averaging method. The single-point sampling method requires the calculation of the average Hounsfield unit value, area and other related indicators based on the anchor points in the cardiac MRI image. The regional averaging method requires clinical workers to manually outline the inner and outer membranes of the left ventricular myocardium, and then analyze and calculate the signals of the entire myocardial area to calculate the average value of the region of interest. The above process is time-consuming and laborious, and due to subjective factors, the annotation results are not repeatable and the accuracy varies greatly.

[0004] There are still some areas that need to be improved in the existing image processing network models based on deep learning. For example, most of them are single-task learning models that only learn one task at a time and have only one objective function, and cannot output image segmentation and classification results at the same time. For example, many existing image segmentation networks achieve the purpose of expanding the receptive field and improving segmentation accuracy by repeatedly stacking dilated convolutions. However, doing so will result in high computational complexity and memory usage. Single-task learning network models often ignore the rich correlation information between different tasks, resulting in poor model generalization performance and unsatisfactory training results.

[0005] Therefore, designing an end-to-end multi-task-based automatic segmentation and disease-assisted diagnosis method for cardiac MRI images has important clinical value and research significance. Summary of the invention

[0006] In view of the shortcomings of the existing technology, a myocardial image segmentation and classification method based on multi-task learning is proposed. A segmentation and classification multi-task model MTN-DCM for auxiliary diagnosis of dilated cardiomyopathy is proposed. The multi-task model MTN-DCM for auxiliary diagnosis of dilated cardiomyopathy includes three sub-networks, namely: a cross-task feature sub-network for feature extraction, a classification sub-network for diagnosis of dilated cardiomyopathy, and a segmentation sub-network for myocardial region segmentation. The segmentation and classification methods specifically include:

[0007] Step 1: The two-dimensional T1mapping sequence magnetic resonance images and LGE images of the short axis of the heart obtained by the cardiology department are used as the MRI data set. The MRI data set includes the segmentation results of the left ventricular myocardium manually annotated by experts as the labels of the segmentation algorithm. The labels of the classification algorithm are dilated cardiomyopathy and healthy.

[0008] Step 2: The preprocessed cardiac MRI dataset is fed into the cross-task feature subnet, which uses 6 cascaded deep separable residual modules (SRMs) to extract multi-scale features and train shared parameters, specifically:

[0009] Step 21: Perform shallow feature extraction on the cardiac MRI image through two cascaded 3×3 ordinary convolutions;

[0010] Step 22: The extracted shallow features are then passed through the sequentially cascaded deep separable residual modules SRM for multi-scale feature extraction. The deep separable residual module SRM adopts residual connection. In each deep separation residual module, the separated convolution layers with cascaded hole rates of 3, 5, and 7 and a convolution kernel size of 3×3 are used to extract multi-scale features, and 1×1 ordinary convolution is used instead of identity mapping for residual connection to obtain cross-task features.

[0011] Step 23: The extracted cross-task features are respectively passed to the segmentation subnet and the classification subnet for segmentation and classification, and further feature extraction and fusion;

[0012] Step 3: Execute the segmentation task, and send the feature maps of 1 / 4, 1 / 8 and 1 / 16 of the original image size extracted in different stages of step 2 to the segmentation subnet for multi-scale feature fusion. The segmentation subnet uses multiple long jump connections to send the feature maps of different levels and scales in the cross-task feature subnet to the joint pyramid upsampling JPU module in the segmentation subnet for multi-scale feature fusion and upsampling to obtain the myocardial segmentation mask of the same size as the original image. Specifically:

[0013] Step 31: In the segmentation sub-network of multi-scale feature fusion, firstly, a joint pyramid upsampling module is used to perform multi-scale feature fusion on the feature maps of three different sizes transmitted in the cross-task feature sub-network, and then the joint pyramid upsampling module is used to combine the multi-scale information of deep semantics and fine structure to generate a first feature map that is conducive to segmentation;

[0014] Step 32: Cascade two depth separation convolutions after the joint pyramid upsampling JPU module to further extract features, and then cascade two sub-pixel convolutions to upsample the first feature map, and finally output a segmentation result with the same size as the input image;

[0015] Step 4: Perform the classification task. Send the feature map of 1 / 32 original image size obtained in step 2 to the classification subnet for high-level semantic feature extraction. The classification subnet uses the residual attention mechanism to extract the deep features required for classification, effectively classify the cardiac MRI images, and obtain the diagnosis results of dilated cardiomyopathy. Specifically:

[0016] Step 41: In the classification sub-network based on the residual attention mechanism, a residual attention module RAM is firstly used to further extract effective features that have an important guiding role in the classification task and suppress redundant features from the feature map transmitted across the task feature sub-network;

[0017] Step 42: The residual attention module RAM includes a trunk branch and a mask branch. The trunk branch is composed of multiple cascaded residual convolution modules, and performs feature processing on the input feature map to obtain the trunk feature map. The mask branch adopts an encoding-decoding structure and adopts a combination of bottom-up and top-down attention to learn an attention feature mask with the same size as the trunk output.

[0018] Step 43: Using the residual learning method, the attention feature mask is multiplied and superimposed pixel by pixel with the backbone feature map, and output to the depth separation convolution layer and global average pooling layer cascaded after the residual attention module for further feature extraction. The global average pooling layer retains spatial information, locates classification targets, and regularizes the network structure to prevent network overfitting.

[0019] Step 44: Input the output result of step 43 to the fully connected layer to output the classification result to determine whether the patient suffers from dilated cardiomyopathy.

[0020] According to a preferred embodiment, the process of obtaining the attention feature mask by the mask branch in step 42 is as follows:

[0021] First, a fast bottom-up feedforward scan is performed in the encoder part, using three residual units and two maximum pooling layers for feature extraction;

[0022] Secondly, a top-down global information expansion is performed in the decoder part to guide the input features at each position, and symmetrical upsampling is performed using two layers of linear interpolation with the same number of maximum pooling layers in the encoder to ensure that the input and output features have the same size;

[0023] Finally, skip connections between the encoder-decoder structure are added to capture information at different scales.

[0024] According to a preferred implementation, the multi-scale fusion process in step 31 specifically includes:

[0025] The feature maps of three different sizes in the cross-task feature subnet are sent to the joint pyramid upsampling JPU module, upsampled to the same size and then spliced. Then, they are passed through a depth-separable dilated pyramid and convolved with dilation rates of 1, 2, 4, and 8 respectively, and then spliced. Finally, they are convolved again to obtain the fused feature map.

[0026] The beneficial effects of the present invention are:

[0027] 1. The present invention makes full use of the strong correlation between the dilated cardiomyopathy discrimination task and the left ventricular myocardium segmentation task for multi-task learning, utilizes the noise added between multiple tasks, and introduces inductive bias for regularization, which helps to alleviate overfitting and enhance the generalization ability of the model; at the same time, by sharing some parameters and structures across task feature subnets, multi-task parallel learning is performed, the dynamic characteristics of network parameter updating are changed, and the learning rate is improved.

[0028] 2. In performing segmentation tasks, the present invention proposes a joint pyramid upsampling (JPU) module to replace the stacked dilated convolution operation, which can not only improve the computational efficiency, but also make full use of the information of feature maps at different levels to extract multi-scale information.

[0029] 3. In performing classification tasks, the present invention proposes a residual attention module RAM to specifically extract features that have an important guiding role in the classification task, enhance the role of effective features and suppress redundant information, and at the same time use the residual network structure to reduce the difficulty of network training and effectively solve the gradient disappearance phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a network structure diagram of the multi-task learning network model MTN-DCM of the present invention;

[0031] Figure 2 It is a structural diagram of the deep separable residual module SRM;

[0032] Figure 3It is a schematic diagram of the structure of the joint pyramid upsampling JPU module;

[0033] Figure 4 It is a schematic diagram of the structure of the residual attention module RAM;

[0034] Figure 5 It is a comparison of segmentation results of different methods on T1mapping data;

[0035] Figure 6 It is the comparison of segmentation results of different methods on LGE data;

[0036] Figure 7 It is the class activation map of the last convolutional layer of the classification subnet;

[0037] Figure 8 This is the result of the ablation experiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0039] The following is a detailed description with reference to the accompanying drawings.

[0040] In view of the shortcomings of the prior art, the present invention proposes a segmentation and classification multi-task model MTN-DCM for auxiliary diagnosis of dilated cardiomyopathy. Figure 1 As shown in the figure, the multi-task model includes three sub-networks, namely: a cross-task feature sub-network for feature extraction, a classification sub-network for dilated cardiomyopathy diagnosis, and a segmentation sub-network for myocardial region segmentation. The segmentation and classification methods specifically include:

[0041] Step 1: The two-dimensional T1mapping sequence magnetic resonance images and LGE images of the short axis of the heart obtained by the cardiology department are used as the MRI dataset. The MRI dataset includes the segmentation results of the left ventricular myocardium manually annotated by experts as the label of the segmentation algorithm, and the labels of the classification algorithm are dilated cardiomyopathy and healthy.

[0042] Step 2: The preprocessed cardiac MRI dataset is fed into the cross-task feature subnet, which uses 6 cascaded deep separable residual modules (SRMs) to extract multi-scale features and train shared parameters, specifically:

[0043] Step 21: Perform shallow feature extraction on the cardiac MRI image by cascading two 3×3 ordinary convolutions.

[0044] Step 22: The extracted shallow features are then passed through a series of cascaded deep separable residual modules (SRMs) to perform multi-scale feature extraction to extract high-level semantic features of the image. Figure 2 The figure is a schematic diagram of the structure of the deep separable residual module SRM. The deep separable residual module SRM adopts residual connection. In each deep separable residual module, the separable convolutional layers with cascaded hole rates of 3, 5, and 7 and convolution kernel size of 3×3 are used to extract multi-scale features. The separable convolutional layer completely separates the learning tasks of spatial correlation and inter-channel correlation, reducing the number of model parameters, and uses 1×1 ordinary convolution instead of identity mapping for residual connection to enhance the network's expression ability and obtain cross-task features.

[0045] Step 23: The extracted cross-task features are passed to the segmentation subnet and classification subnet for segmentation and classification respectively, and then further feature extraction and fusion are performed.

[0046] The cross-task feature subnetwork is a parameter-sharing subnetwork of the multi-task model of the present invention, which extracts the common features and information of the two tasks of segmentation and classification for hard parameter sharing, thereby reducing the risk of model overfitting.

[0047] Step 3: Execute the segmentation task, which is also the multi-scale feature fusion stage. The feature maps of 1 / 4, 1 / 8 and 1 / 16 of the original image size extracted in different stages of step 2 are sent to the segmentation subnet for multi-scale feature fusion. The segmentation subnet uses multiple long jump connections to send the feature maps of different levels and scales in the cross-task feature subnet to the joint pyramid upsampling JPU module in the segmentation subnet for multi-scale feature fusion and upsampling, and obtain the myocardial segmentation mask of the same size as the original image. The joint pyramid upsampling JPU module is shown in Figure 3. Specifically:

[0048] Step 31: In the segmentation sub-network of multi-scale feature fusion, the joint pyramid upsampling JPU module is first used to perform multi-scale feature fusion on the three different sizes of feature maps transmitted in the cross-task feature sub-network. Then, the joint pyramid upsampling JPU module is used to combine the multi-scale information of deep semantics and fine structure to generate a feature map that is beneficial to segmentation.

[0049] The multi-scale fusion process specifically includes: sending the feature maps of three different sizes in the cross-task feature subnet into the joint pyramid upsampling JPU module, upsampling to the same size and then splicing, and then passing through a depth-separable hollow pyramid, performing convolutions with expansion rates of 1, 2, 4, and 8 respectively and then splicing. Finally, a fused feature map is obtained after another convolution.

[0050] Step 32: After the joint pyramid upsampling JPU module, two depth separation convolutions are cascaded to further extract features, and then two sub-pixel convolutions are cascaded to upsample the feature map, and finally the segmentation result with the same size as the input image is output.

[0051] By replacing the dilated convolution operation with a joint pyramid upsampling module, we can not only improve the computational efficiency, but also make full use of the information of feature maps at different levels to extract multi-scale information.

[0052] Step 4: Perform the classification task. Send the feature map of 1 / 32 original image size obtained in step 2 to the classification subnet for high-level semantic feature extraction. The classification subnet uses the residual attention mechanism to extract the deep features required for classification, effectively classify the cardiac MRI images, and obtain the diagnosis results of dilated cardiomyopathy. Specifically:

[0053] Step 41: In the classification subnetwork based on the residual attention mechanism, a residual attention module RAM is first used to further extract effective features that have an important guiding role in the classification task from the feature map transmitted across the task feature subnetwork and suppress redundant features at the same time; the structure of the residual attention module RAM is as follows: Figure 4 shown.

[0054] Step 42: The residual attention module RAM includes a trunk branch and a mask branch. The trunk branch is composed of multiple cascaded residual convolution modules. Perform feature processing and output the backbone feature map ,in and Represent the space and channel positions respectively.

[0055] The mask branch adopts an encoding-decoding structure and uses a combination of bottom-up and top-down attention to learn an attention feature mask with the same size as the trunk output. .

[0056] The process of obtaining the attention feature mask by the mask branch is as follows: first, a fast bottom-up feedforward scan is performed in the encoder part, and three residual units and two maximum pooling layers are used for feature extraction and rapid increase of the receptive field; secondly, a top-down global information expansion is performed in the decoder part to guide the input features of each position, and two layers of linear interpolation with the same number of maximum pooling layers in the encoder are used for symmetrical upsampling to ensure that the input and output features are of the same size. Finally, jump connections are added between the encoder and decoder structures to capture information of different scales.

[0057] The residual attention module effectively enhances the feature maps transmitted across the task feature subnets and suppresses noise. Multiple residual units are repeatedly used to alleviate the gradient vanishing phenomenon caused by the network being too deep. At the same time, the feature tensors before and after the mask are used as the input of the next layer using the attention mechanism to obtain richer high-level semantic features.

[0058] Step 43: Using the residual learning method, the attention feature mask is multiplied and superimposed pixel by pixel with the backbone feature map, and output to the deep separation convolution layer and global average pooling layer cascaded after the residual attention module for further feature extraction. The global average pooling layer retains spatial information, locates classification targets, and regularizes the network structure to prevent network overfitting.

[0059] Step 44: Input the output result of step 43 to the fully connected layer to output the classification result to determine whether the patient suffers from dilated cardiomyopathy.

[0060] In the image segmentation task, the widely used Dice Similarity Coefficient (DSC), Accuracy, Intersection-Over-Union (IoU) and Hausdorff Distance (HD) are used as evaluation indicators. DSC is a function for evaluating similarity. It is mainly used in image segmentation evaluation to calculate the similarity or overlap between the predicted segmentation result X and the label Y, and the value range is between [0,1]. Accuracy is the proportion of correctly classified samples to all samples. Generally speaking, the higher the Accuracy, the better the model performance. IoU, also known as the Jaccard index, is very similar to DSC and is also used to evaluate the similarity between the predicted result and the label, and the value range is between [0,1]. HD is also a commonly used evaluation indicator in image segmentation, which describes the degree of similarity between two sets of points. The boundaries of image segmentation are more sensitive. Simply put, HD is the maximum value of the distance between the closest points of two trajectories. Therefore, in the segmentation task, the smaller the HD, the better for predicting the segmentation result X and label Y.

[0061] In the image classification task, four evaluation indicators are widely used in image classification experiments: accuracy, sensitivity (also known as recall), specificity, and area under the curve (AUC). Accuracy is the most common evaluation indicator in classification tasks, which refers to the proportion of correctly predicted samples to all samples. Generally speaking, the higher the accuracy, the better the classifier effect. Sensitivity refers to the proportion of all positive samples that are classified correctly, which measures the classifier's ability to recognize positive samples. The higher the sensitivity, the stronger the model's ability to recognize positive samples. Specificity refers to the proportion of all negative samples that are classified correctly, which measures the classifier's ability to recognize negative samples. The higher the sensitivity, the stronger the model's ability to recognize negative samples. AUC refers to the area under the ROC (Receiver Operating Characteristic) curve. The horizontal and vertical coordinates of the ROC curve are FPR (false negative rate) and TPR (true positive rate). The smaller the FPR and the higher the TPR, the better the model. Therefore, the larger the area under the ROC (AUC), or the closer the curve is to the upper left corner, the better the model classification effect.

[0062] In order to verify the effectiveness of the segmentation of the multi-task network, it is compared with U-Net, U-Net++, SegNet and Deeplab v3+, four classic segmentation methods in this technical field. The first column of Table 1 shows the quantitative comparison results of the MTN-DCM-based method proposed in the present invention and the four classic medical image segmentation methods in segmentation and classification on the T1 mapping dataset. As can be seen from Table 1, the segmentation results obtained by MTN-DCM on the T1 mapping data reached 82.4%, 95.9%, 70.6% and 0.266mm in the four quantitative indicators of DSC, Accuracy, IoU and HD, respectively, which are better than the other methods. Among the other methods, SegNet and Deeplabv3+ with the highest DSC are also 2.5% lower than MTN-DCM.

[0063] The first column of Table 2 shows the quantitative comparison results of the MTN-DCM-based method proposed in the present invention and four classic medical image segmentation methods in segmentation and classification on LGE data. A similar situation can also be seen from Table 2. The segmentation results obtained by MTN-DCM on LGE data reached 82.4%, 95.9%, 70.6% and 0.266mm in the four quantitative indicators of DSC, Accuracy, IoU and HD, respectively. DSC and IoU are 3.4% and 5% higher than the highest Deeplabv3+ among the other methods. The above fully illustrates that MTN-DCM can not only capture the characteristics of the left ventricular myocardium of cardiac MRI images more discriminatively and achieve the optimal segmentation performance, but also performs well on different cardiac MRI data, and the model has strong generalization.

[0064] Table 1 Quantitative comparison results on T1mapping data

[0065]

[0066] Table 2 Quantitative comparison results on LGE data

[0067]

[0068] Figure 5 and Figure 6 They are the left ventricular myocardial segmentation results on T1 mapping data and LGE data, respectively. From left to right, they are the original image, manual standard segmentation result, SegNet network segmentation result, U-Net network segmentation result, U-Net++ network segmentation result, Deeplab v3+ network segmentation result and the segmentation result of the MTN-DCM network proposed in this invention.

[0069] from Figure 5 and Figure 6 It can be seen that the method of the present invention has achieved good results in left ventricular myocardial segmentation on both data sets. Compared with several other methods, the segmentation results of the method of the present invention are closer to the manual standard segmentation results. It can also be observed from the subjective visual effect that the method proposed by the present invention has a strong discrimination ability for irregular and discontinuous structures (as shown by the white arrows in the figure), and is not prone to over-segmentation, while the SegNet method is prone to under-segmentation, and the U-Net, U-Net++ and Deeplab v3+ methods are prone to over-segmentation. In addition, it can be found that the method of the present invention is prone to over-segmentation and under-segmentation when the segmentation target is relatively small (as shown by the black arrows in the figure).

[0070] In order to verify the classification effectiveness of the method proposed in the present invention, it is compared with four currently popular classification methods: VGG16, ResNet50, Inceptionv3 and Xception. The second column of Table 1 and Table 2 shows the classification quantitative evaluation results of several methods on T1 mapping and LGE data, respectively. Figure 7 Shows the class activation maps for positive and negative samples of the classification subnet.

[0071] From the first column of Table 1, it can be seen that the accuracy of the method of the present invention reaches 95.7%, which is 7.5%, 6.9%, 7.5% and 1.1% higher than VGG16, ResNet50, Inceptionv3 and Xception respectively, indicating that the method of the present invention has the best classification effect on T1 mapping data. Although the specificity of VGG16 and Inceptionv3 is 100%, the sensitivity is 0, indicating that the two methods have poor recognition ability for positive samples, i.e., diseased samples, and all samples are judged as healthy categories, while the sensitivity and specificity of the method of the present invention are 77.9% and 98.1% respectively, indicating that it can handle the problem of class imbalance well and avoid falling into overfitting. In addition, the AUC of the method of the present invention is also the highest, reaching 97.9%, which is 53.4%, 42.6%, 11.4% and 0.2% higher than VGG16, ResNet50, Inceptionv3 and Xception respectively, which once again confirms its good classification performance.

[0072] From the second column of Table 2, we can see that the accuracy, sensitivity and AUC of the method of the present invention are the best, which are 96.1%, 64.9% and 91.8% respectively, indicating that it can also achieve the best classification effect on LGE data. Although the specificity of VGG16, ResNet50, Inceptionv3 and Xception is as high as 99.7% and 99% respectively, their corresponding sensitivities are very low, among which Inceptionv3 is even only 2.7%, indicating that these methods cannot handle the problem of data category imbalance well and fall into overfitting. The good classification performance of the method of the present invention on LGE further shows that it has good classification ability on different cardiac MRI data sets.

[0073] from Figure 7 It can be seen that the class activation map of the positive sample is more concentrated, while the class activation map of the negative sample is more divergent, indicating that the network can indeed learn the differences between samples of different classes, further proving the classification effectiveness of this method from the perspective of model interpretability.

[0074] In order to verify the advantages of the multi-task learning method in the present invention, the present invention removes the classification subnetwork and segmentation subnetwork of MTN-DCM respectively, and conducts experimental tests and evaluations on the remaining segmentation single-task learning and classification single-task learning. The penultimate row in Table 1 and Table 2 shows the experimental results of segmentation single-task learning and classification single-task learning on T1 mapping and LGE data respectively.

[0075] As can be seen from Table 1, on the T1 mapping data, the evaluation indicators of both segmentation single-task learning and classification single-task learning are almost worse than those of the multi-task learning model: in the segmentation task, the gains of multi-task learning in DSC, Accuracy, IoU and HD are 0.5%, 4.2%, 0.7% and 0.019mm respectively; in the classification task, the gains of multi-task learning in Accuracy, Sensitivity, Specificity and AUC are 1.2%, -1.3%, 1.6% and 0.5% respectively.

[0076] A similar situation can be seen from Table 2. On the LGE data, the evaluation indicators of single-task learning are almost worse than those of the multi-task learning model: in the segmentation task, the gains of multi-task learning in DSC, Accuracy, IoU and HD are 0.5%, 5.2%, 0.9% and 0.039mm respectively; in the classification task, the gains of multi-task learning in Accuracy, Sensitivity, Specificity and AUC are 2.1%, 64.9%, -1.9% and 7.7% respectively, among which only the Specificity indicator is reduced by 1.9%. This is because the Specificity and Sensitivity indicators of single-task classification learning are 100% and 0 respectively, that is, single-task classification learning directly judges all samples as negative classes with a large number of samples, resulting in overfitting. This shows that multi-task learning is more advantageous than single-task learning in dealing with the problem of imbalanced categories.

[0077] In order to verify the influence of each module in the MTN-DCM model proposed in the present invention, the present invention also conducted an ablation experiment on T1 mapping data. The experimental results are as follows: Figure 8 shown.

[0078] from Figure 8It can be seen that after removing the deep separation residual module, residual attention module and JPU module respectively, the performance of the segmentation task and classification task has declined. Among them, the classification task is more obviously affected by the deep separation residual module, and the accuracy, AUC and specificity have decreased by about 7%, 17% and 25% respectively. The above experiments show that the three modules are all beneficial modules of the present invention, which can improve the segmentation and classification performance at the same time.

[0079] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention description and its drawings are illustrative and do not constitute a limitation of the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. A myocardial image segmentation and classification method based on multi-task learning, characterized in that: A segmentation and classification multi-task model MTN-DCM for auxiliary diagnosis of dilated cardiomyopathy is proposed. The multi-task model includes three sub-networks, namely: a cross-task feature sub-network for feature extraction, a classification sub-network for diagnosis of dilated cardiomyopathy, and a segmentation sub-network for myocardial region segmentation. The segmentation and classification method specifically includes: Step 1: The two-dimensional T1mapping sequence magnetic resonance images and LGE images of the short axis of the heart obtained by the cardiology department are used as MRI data sets. The MRI data sets include the segmentation results of the left ventricular myocardium manually annotated by experts as the labels of the segmentation algorithm, and the labels of the classification algorithm are dilated cardiomyopathy and healthy; Step 2: The preprocessed cardiac MRI dataset is fed into the cross-task feature subnet, which uses 6 cascaded deep separable residual modules (SRMs) to extract multi-scale features and train shared parameters, specifically: Step 21: First, shallow feature extraction of cardiac MRI images is performed through two cascaded 3×3 ordinary convolutions; Step 22: The extracted shallow features are then subjected to multi-scale feature extraction by sequentially cascading deep separable residual modules SRM. The deep separable residual module SRM adopts residual connection. In each deep separation residual module, a separation convolution layer with a hole rate of 3, 5, and 7 and a convolution kernel size of 3×3 is sequentially cascaded to extract multi-scale features, and a 1×1 ordinary convolution is used instead of the identity mapping for residual connection to obtain cross-task features. Step 23: The extracted cross-task features are respectively transferred to the segmentation subnet and the classification subnet for segmentation and classification, and further feature extraction and fusion; Step 3: Execute the segmentation task, and send the feature maps of 1 / 4, 1 / 8 and 1 / 16 of the original image size extracted at different stages of step 2 to the segmentation subnet for multi-scale feature fusion. The segmentation subnet uses multiple long jump connections to send the feature maps of different levels and scales in the cross-task feature subnet to the joint pyramid upsampling JPU module in the segmentation subnet for multi-scale feature fusion and upsampling to obtain a myocardial segmentation mask with the same size as the original image. Specifically: Step 31: In the segmentation subnetwork of multi-scale feature fusion, firstly, a joint pyramid upsampling module JPU is used to perform multi-scale feature fusion on the feature maps of three different sizes transmitted in the cross-task feature subnetwork, and then the joint pyramid upsampling module is used to combine the multi-scale information of deep semantics and fine structure to generate a first feature map that is conducive to segmentation; Step 32: Cascade two depth separation convolutions after the joint pyramid upsampling JPU module to further extract features, and then cascade two sub-pixel convolutions to upsample the first feature map, and finally output a segmentation result with the same size as the input image; Step 4: Perform the classification task, and send the feature map of 1 / 32 original image size obtained in step 2 to the classification subnet for high-level semantic feature extraction. The classification subnet uses the residual attention mechanism to extract the deep features required for classification, classify the cardiac MRI images, and obtain the diagnosis results of dilated cardiomyopathy. Specifically: Step 41: In the classification sub-network based on the residual attention mechanism, a residual attention module RAM is firstly used to further extract effective features that have an important guiding role in the classification task and suppress redundant features from the feature map transmitted across the task feature sub-network; Step 42: The residual attention module RAM includes a trunk branch and a mask branch. The trunk branch is composed of multiple cascaded residual convolution modules, and the input feature map is processed to obtain a trunk feature map. The mask branch adopts an encoding-decoding structure and adopts a combination of bottom-up and top-down attention to learn an attention feature mask with the same size as the trunk output. Step 43: Using residual learning, the attention feature mask is multiplied and superimposed pixel by pixel with the backbone feature map, and output to the depth separation convolution layer and global average pooling layer cascaded after the residual attention module for further feature extraction. The global average pooling layer retains spatial information, locates classification targets, and regularizes the network structure to prevent network overfitting. Step 44: Input the output result of step 43 to the fully connected layer to output the classification result to determine whether the patient suffers from dilated cardiomyopathy.

2. The myocardial image segmentation and classification method according to claim 1, characterized in that: The process of obtaining the attention feature mask by the mask branch in step 42 is as follows: First, a fast bottom-up feedforward scan is performed in the encoder part, using three residual units and two maximum pooling layers for feature extraction; Secondly, a top-down global information expansion is performed in the decoder part to guide the input features at each position, and symmetrical upsampling is performed using two layers of linear interpolation with the same number of maximum pooling layers in the encoder to ensure that the input and output features have the same size; Finally, skip connections between the encoder-decoder structure are added to capture information at different scales.

3. The myocardial image segmentation and classification method according to claim 2, characterized in that: The multi-scale fusion process in step 31 specifically includes: The feature maps of three different sizes in the cross-task feature subnet are sent to the joint pyramid upsampling JPU module, upsampled to the same size and then spliced. Then, they are passed through a depth-separable dilated pyramid and convolved with dilation rates of 1, 2, 4, and 8 respectively, and then spliced. Finally, they are convolved again to obtain the fused feature map.

Images