A method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion

Abstract

The present invention relates to the cross-technical field of deep learning and medical image recognition and diagnosis, and specifically discloses a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion. The diagnostic model construction steps are as follows: S01: Image collection and annotation: Acquire cardiac MR images as sample data, and obtain the cardiac label corresponding to the MR images after manual annotation; S02: Image preprocessing: Including normalization processing, data enhancement and data cropping; S03: Model construction: Including image restoration network construction and image segmentation network construction, to perform image restoration tasks; Image segmentation network construction is used to segment cardiac images; S04: Image pre-training: Train the image restoration network, the goal is to allow the encoder module of the image restoration network to fully learn the characteristics of cardiac fibrosis images; S05: Model training. The purpose of the present invention is to improve the segmentation accuracy and diagnostic accuracy of the network model for cardiac fibrosis images.

Description

Technical Field

[0001] The present application relates to the interdisciplinary technical field of deep learning and medical image recognition and diagnosis, and specifically discloses a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion. Background Art

[0002] Cardiac fibrosis is an important pathological change that has a significant impact on the function of the heart. Accurate diagnosis of cardiac fibrosis is extremely important for the early detection and treatment of cardiovascular diseases. According to the World Health Report, heart disease is one of the leading causes of death worldwide, of which cardiovascular disease (CVD) causes about 17.3 million deaths each year, and this number is expected to increase to more than 23.6 million by 2030. Cardiac fibrosis, as a major histological feature of myocardial damage, is closely related to a variety of heart diseases, such as arterial hypertension, valvular heart disease, diabetic cardiomyopathy, hypertrophic cardiomyopathy, dilated cardiomyopathy, and heart aging. Cardiac magnetic resonance imaging (CMR) is an important means of diagnosing cardiac fibrosis, especially through late gadolinium enhancement (LGE) technology, which can effectively identify cardiac fibrosis. CMR plays an important role in the detection and tracking of congenital or acquired heart disease by using a strong magnetic field and a computer to create detailed images of the heart and its surroundings. However, accurate diagnosis of cardiac fibrosis relies on high spatial resolution imaging technology, and traditional diagnostic methods may require manual feature extraction, which is time-consuming, labor-intensive and prone to errors. Therefore, the development of automated diagnostic methods is of great value in improving diagnostic efficiency and reducing errors.

[0003] At present, there have been many works on the automatic diagnosis of cardiac fibrosis, and the methods can be roughly divided into two categories: one is the classic machine learning algorithm, and the other is the automatic diagnosis using deep learning. The work of machine learning algorithms includes: Pu et al. collected images of 273 patients with cardiomyopathy for radiomics research, used logistic regression analysis to find predictive influencing features, and constructed a CMR model. Radiomics features were extracted from the maximum wall thickness (MWT) layer and the entire left ventricular (LV) myocardium. The radiomics model was established using extreme gradient lifting. By fusing image features and radiomics models to establish a comprehensive model, the final diagnostic accuracy can reach 89.02%, the sensitivity can reach 92.54%, and the AUC value can reach 0.898. Campese et al. combined support vector machines (SVM) and convolutional neural networks (CNN) to complete the binary classification task of determining whether there is scar in cardiac tissue. The final model achieved an accuracy of 71% and a sensitivity of 72%.

[0004] Work using deep learning algorithms includes Popescu et al., who developed a multi-stage network based on deep learning for automatic segmentation of myocardial and scar fibrosis and extraction of clinical features in contrast-enhanced cardiac magnetic resonance imaging (LGE-CMR). Specifically, a three-stage neural network was constructed to first identify the left ventricular (LV) region of interest (ROI), then segment the ROI into active myocardium and enhanced areas, and finally adjust the prediction results to meet anatomical constraints through a post-processing stage. A total of 155 two-dimensional LGE-CMR patient scans and 246 synthetic LGE-like scans were used for training and testing. The results showed that the predicted left ventricle and scar segmentation achieved balanced accuracy of 96% and 75%, respectively. Marco et al. built a convolutional neural network model to solve the problem of detecting myocardial fibrosis in early contrast-enhanced cardiac computed tomography (CE-CCT) images. CE-CMR and (early and late) CE-CCT examinations were performed on 50 patients with known left ventricular dysfunction (LVD). Based on the pattern of CE-CMR, patients were classified as ischemic or non-ischemic LVD. The researchers extracted myocardial segments based on the American Heart Association (AHA) 16-segment model on early CE-CCT images and marked them as scarred or non-scarred based on manual tracing on late CE-CCT. A deep learning model was developed to classify each segment. By analyzing 44,187 left ventricular segments, the model achieved an accuracy of 71% and an AUC of 76%. In addition, by comparing CE-CMR with the corresponding early CE-CCT results, the consistency between the model and reality reached 89%. This shows that deep learning detects left ventricular segments affected by myocardial fibrosis in early CE-CCT acquisitions without the need for additional contrast injection or radiation dose;

[0005] Cardiac fibrosis is a pathological condition characterized by abnormal deposition of non-contractile extracellular matrix in the cardiac interstitium, which leads to changes in cardiac structure and function. The characteristic features of cardiac fibrosis include hardening and scarring of cardiac tissue, which may lead to impairment of cardiac systolic and diastolic function. Currently, there are the following problems in the diagnosis of cardiac fibrosis:

[0006] 1. Limitations of machine learning algorithms: The use of machine learning algorithms for the diagnosis of cardiac fibrosis has limitations. Machine learning is highly dependent on manual feature engineering and requires medical experts to extract features from a large number of data sources such as medical images, blood test indicators, and biomarker levels. Different experts may have different views on which features are most important, which also leads to consistency issues during model training, affecting the stability and accuracy of model diagnosis.

[0007] 2. Cardiac fibrosis data is heterogeneous: The clinical manifestations and imaging features of patients with cardiac fibrosis may vary due to individual differences, disease stages, and comorbidities, resulting in high data heterogeneity. When processing such data, machine learning algorithms and simple neural network algorithms may have difficulty capturing the core pathological features of cardiac fibrosis, affecting the accuracy of the final diagnosis of the disease.

[0008] 3. Small amount of cardiac fibrosis data: The current research work has a relatively small number of samples of cardiac fibrosis patients, usually only a few hundred cases. For machine learning or deep learning, it is impossible to extract generalizable features, and the model's extraction ability is insufficient, which will ultimately reduce the diagnostic accuracy of cardiac fibrosis diseases.

[0009] 4. ROI labeling is required: For most current deep learning algorithms, the heart part in the MR image needs to be marked first, and then the neural network learns the corresponding heart-labeled image to complete the diagnosis of cardiac fibrosis. This approach is not an end-to-end extraction and requires manual labeling of the heart. Each labeling of the heart takes extra time.

[0010] 5. Unclear boundaries of the lesion area: Cardiac fibrosis can be focal or diffuse, and its morphological characteristics may include strip-like, spotted or sheet-like distribution. Different types of fibrosis have different morphologies, so the boundaries between different morphological areas and normal myocardium will also be different.

[0011] In view of the above problems, the present invention proposes a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion.

[0012] In response to Problem 1 and Problem 2, the present invention proposes an improved Mask R-CNN algorithm, which introduces an innovative dual-level feature pyramid network structure, including pixel-Feature pyramid network (p-FPN) and region-Feature pyramid network (r-FPN), to achieve more refined feature fusion and gradually improve the mask resolution. In addition, the present invention also designs an efficient Feature Aggregation Module (FAM), which enhances the model's learning of key features of cardiac fibrosis images by dynamically adjusting the weights of regional-level features and pixel-level features. At the same time, the self-attention feature fusion mechanism improves the segmentation accuracy and diagnostic accuracy of the network model for cardiac fibrosis tasks.

[0013] In response to question 3, the present invention adopts self-supervised learning technology to enhance the model's ability to extract features from cardiac MR images. Specifically, the present invention designs an encoder-decoder architecture, which uses the ResNet module as the basis of the encoder, and then randomly masks the original image. The goal of image recovery is to reconstruct the masked part of the image. At the same time, the randomly masked MR image is randomly scaled and randomly rotated as data enhancement. Then the model restores the original cardiac MR image through two stages of feature extraction and image reconstruction. Through this process, the encoder can deeply explore the intrinsic semantic information of cardiac fibrosis data, reduce the dependence on large-scale annotated data sets, and alleviate the problem of small number of cardiac fibrosis patient samples.

[0014] In response to question 4, the present invention proposes an improved Mask R-CNN algorithm, which segments and classifies cardiac MR images through the model. Multi-task learning can increase the generalization ability of the model. At the same time, this is an end-to-end network structure, which can simplify the traditional manual labeling process. Through the advanced functions of the network model, the algorithm of the present invention can automatically identify and segment the heart area and diagnose cardiac fibrosis.

[0015] In response to question 5, the present invention adopts a combination strategy of multiple loss functions in the neural network training process. These loss functions include classification loss, bounding box loss, Dice loss, and edge loss designed specifically for the problem of unclear boundaries. The edge loss is calculated as follows: first, the Laplacian operator is applied to the real mask image to enhance the edge features in the image. Then, the enhanced soft edge map is converted into a clear binary edge map through thresholding. Similarly, the mask image predicted by the model also undergoes edge detection and binarization steps to generate the corresponding predicted edge map. Finally, the edge loss is obtained by calculating the difference between the predicted edge map and the real edge map. The edge features of cardiac fibrosis are emphasized by the Laplacian operator to help the model capture the high-frequency detail information of the cardiac fibrosis morphology. Summary of the invention

[0016] The purpose of the present invention is to improve the segmentation accuracy and diagnostic accuracy of the network model for cardiac fibrosis images.

[0017] In order to achieve the above object, the present invention provides the following basic scheme:

[0018] 1. A method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion. The steps of constructing the diagnosis model are as follows:

[0019] S01: Image collection and annotation: Obtain cardiac MR images as sample data, and obtain the cardiac labels corresponding to the MR images after manual annotation;

[0020] S02: Image preprocessing: including normalization, data enhancement and data cropping. The image is preprocessed using normalization, data enhancement and data cropping.

[0021] S03: Model construction: including image restoration network construction and image segmentation network construction. The image restoration network uses self-supervised learning to perform image restoration tasks; the image segmentation network is used to segment cardiac images and diagnose cardiac fibrosis;

[0022] The details are as follows:

[0023] The image restoration network is constructed as follows: it includes constructing an encoder and a decoder. The encoder is part A, which includes convolutional layers, Res-Block and pooling layers.

[0024] Convolutional layer: used to extract cardiac fibrosis image features;

[0025] Res-Block: It consists of two convolutional blocks, making it easier for the network to learn the identity mapping;

[0026] Pooling layer: highlights the significant features in cardiac fibrosis images while reducing the amount of calculation and the number of parameters;

[0027] The decoder is part B, which includes upsampling layers, convolution blocks, and short-circuit structures;

[0028] Upsampling layer: Use the ConvTranspose layer to upsample the low-resolution feature map to a higher resolution;

[0029] Convolutional block: The specific components include: convolutional layer, LeakyReLU activation function and BN layer;

[0030] Short-circuit structure: directly connect the low-level features in the encoder to the corresponding layers of the decoder;

[0031] The image segmentation network is constructed as follows: improve the Mask R-CNN, build the network BackBone, build the feature pyramid network FPN, and build the feature aggregation module FAM;

[0032] S04: Image pre-training: Train the image restoration network to allow the encoder of the image restoration network to fully learn the characteristics of cardiac fibrosis images;

[0033] S05: Model training: After the model pre-training is completed, the obtained image restoration network encoder is used as the BackBone in the improved Mask R-CNN. At the same time, the encoder parameters trained in the first stage are migrated to Mask R-CNN, and ReduceLROnPlateau learning rate automatic attenuation is used for model training.

[0034] The principles and effects of this basic solution are:

[0035] 1. Compared with the prior art, the network uses an improved Mask R-CNN algorithm. The present invention introduces an innovative two-level feature pyramid network structure, including pixel-FPN and region-FPN, and designs an efficient feature aggregation module FAM.

[0036] 2. Construct an encoder-decoder architecture for self-supervised learning. The encoder part uses the ResNet module as the basis. The input of the model is the original image processed by random mask. The goal of the model is to reconstruct the part of the image that is blocked by the mask. At the same time, we implement a variety of data enhancement techniques on the MR images after random masking. The trained encoder model is used as the backbone and integrated into the improved Mask R-CNN algorithm.

[0037] 3. The present invention uses an improved Mask R-CNN algorithm to segment and classify cardiac MR images through a model. This is a multi-task learning for the model, which can increase the generalization ability of the model, while solving the problem of requiring manual labeling and realizing end-to-end network structure design.

[0038] 4. Use multiple loss functions, classification loss, bounding box loss, Dice loss, and the proposed margin loss.

[0039] 5. Compared with the prior art, in the prior art, most of the machine learning used for cardiac fibrosis diagnosis tasks relies on manual feature engineering, or uses simple convolutional neural networks for identification and diagnosis, which is time-consuming and has low accuracy. The present invention proposes an improved Mask R-CNN algorithm, which uses a dual-level feature pyramid network (Dual-Level FPN) structure, including pixel-level FPN and region-level FPN, which can capture the global context and local details of cardiac MR images and achieve more refined feature fusion. The designed feature aggregation module FAM enhances the model's learning of key features of cardiac fibrosis images by dynamically adjusting the region-level feature weights and pixel-level feature weights. At the same time, the self-attention feature fusion mechanism improves the segmentation accuracy and diagnostic accuracy of the network model for cardiac fibrosis images.

[0040] 6. In the prior art, the number of samples of patients with cardiac fibrosis is usually small, only a few hundred cases, which limits the generalization ability of the model. In order to solve this problem, the present invention adopts self-supervised learning technology to enhance the model's ability to extract features of cardiac MR images. We designed a model based on an encoder-decoder architecture, in which the encoder part uses a ResNet module as the basis. The input of the model is the original image processed by random masking, and the goal of the model is to reconstruct the part of the image that is blocked by the mask. In order to further enhance the robustness of the model, we implemented a variety of data enhancement techniques on the MR images after random masking. Through the two stages of feature extraction and image reconstruction, the model can restore the original cardiac MR images. This process prompts the encoder to deeply explore the intrinsic semantic information of cardiac fibrosis data, thereby reducing the dependence on large-scale annotated data sets and effectively alleviating the problem of insufficient sample number. We use the encoder model as Backbone and integrate it into the improved Mask R-CNN algorithm. This integration enables the model to more accurately locate and identify lesion areas when processing cardiac fibrosis MR images, thereby improving the accuracy and reliability of diagnosis.

[0041] 7. In the prior art, it is necessary to first mark the heart part in the MR image, and then let the neural network learn the corresponding heart-labeled image to complete the diagnosis of cardiac fibrosis. The present invention uses the improved Mask R-CNN algorithm through its multi-task learning framework, which not only enhances the model's recognition ability of the heart area, but also improves the generalization performance of the model. This multi-task learning strategy enables the network to simultaneously learn the features of segmentation and classification in the cardiac fibrosis task, thereby sharing knowledge between different tasks and improving the overall diagnostic accuracy. This algorithm adopts an end-to-end network structure design, which means that from the original MR image input to the final heart segmentation and classification output, the entire process is automated and no manual intervention is required. This end-to-end design significantly reduces the reliance on manual labeling, saves valuable time resources, and also reduces the errors introduced by manual operations.

[0042] 8. In the prior art, the model cannot accurately distinguish the edges of cardiac fibrosis. The present invention adopts a combination strategy of multiple loss functions in the neural network training process. These loss functions include classification loss, bounding box loss, Dice loss, and edge loss specially designed for the problem of unclear boundaries. Edge loss highlights the edge features of the cardiac fibrosis area by using the Laplace operator, thereby helping the model to more accurately capture the high-frequency detail information of the cardiac fibrosis morphology. The design of this edge loss enables the model to better understand and distinguish the boundaries between different types of cardiac fibrosis areas and normal myocardium. By combining the training method of multiple loss functions, the model of the present invention can more accurately identify and locate the cardiac fibrosis lesion area, thereby improving the accuracy and reliability of the diagnosis of cardiac fibrosis.

[0043] Further, in step S01, the heart label includes two different labels, 0 and 1. Specifically, the heart labels are: label 0 is the background label, label 1 is the heart label, and at the same time, a judgment is made on whether the heart is fibrotic. Labels 0 and 1 are also printed. 0 represents normal and 1 represents fibrosis. The labeled sample data is divided into a training set and a test set.

[0044] Further, in step S02, normalization processing is performed: (x-μ) / σ, where x is the HU value of the pixel in the cardiac MR image, u is the mean HU value of all pixels, and σ is the standard deviation of all pixels. Data enhancement: including enhancement using random Gaussian noise, random contrast, random mirror flipping, random horizontal flipping and random rotation.

[0045] Further, in step S03, the cardiac fibrosis image features include texture and edge information, and the specific composition of the convolution block is as follows: the convolution layer further extracts the high-level features of the cardiac MR image, the LeakyReLU activation function introduces nonlinear changes, so that the model can learn and simulate more complex function mappings, and the BN layer is used to speed up the training.

[0046] Furthermore, in step S03, the image segmentation network is used to further refine and upsample the feature map extracted by the encoder to restore the resolution of the original input image. The ConvTranspose layer utilizes the structural information in the training data to reduce image artifacts and blur.

[0047] Furthermore, in step S03, the overall structure of the network is improved based on Mask R-CNN. The first improvement is the BackBone of the network, which is replaced by the encoding layer of the image restoration network in step S03, and the parameters of the encoding layer are migrated. The image restoration network learns the cardiac image restoration task through self-supervision, and the encoding layer learns the semantic information and typical features of the cardiac image. The network parameters are migrated to speed up the convergence speed of the improved Mask R-CNN model. The second improvement is the feature pyramid network FPN, which introduces a two-level feature pyramid network structure. The two-level feature pyramid network structure includes pixel-level FPN and region-level FPN, which capture the global context and local details of cardiac MR images. The feature aggregation module FAM dynamically adjusts the region-level feature weights and pixel-level feature weights.

[0048] Further, in step S04, during the training of the model, the Adam optimizer is used for model training, and the specific formula is as follows:

[0049]

[0050] N is the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.

[0051] Furthermore, in step S05, the patience parameter is set to 50 times. When the loss does not decrease in 50 consecutive epochs, the learning rate is automatically reduced to 1 / 10 of the original value. A total of 5000 rounds of training are performed. The loss function uses a mixture of multiple losses, including the dice loss for calculating the segmentation loss, the cross entropy loss for calculating the classification loss, and the edge loss proposed for the unclear boundary problem of cardiac fibrosis lesion area. The formula of dice loss is as follows:

[0052]

[0053] Where T is the true annotation, P is the predicted area, and the Dice loss function maximizes the intersection between the model output and the true label.

[0054] Further, in step S05, the formula of Cross entropy is as follows:

[0055]

[0056] Where y is the true label, is the predicted probability value, which is between 0 and 1.

[0057] Furthermore, to address the unclear boundary problem of cardiac fibrosis, an edge loss was designed.

[0058] The calculation method is as follows:

[0059] Step 1: Generate edge map Use the Laplacian operator to operate the ground-truth mask to obtain a soft edge map that reflects edge information. The Laplacian operator is a second-order derivative operator that can highlight the edges and details in the image. The specific steps are as follows:

[0060] 1. Define the Laplace kernel. The kernel used is as follows:

[0061]

[0062] 2. Convolution operation: Use the above Laplacian kernel and the true mask to perform convolution operation. For each pixel in the true mask, multiply the pixel value in its neighborhood with the corresponding value in the Laplacian kernel, and then sum the results to get the new value of the pixel.

[0063] 3. Result conversion: superimpose the image processed by the Laplace filter kernel with the original image to enhance the edge while retaining the content of the image edge;

[0064] The second step is to perform thresholding on the soft edge map obtained by the Laplacian operator and convert it into a binary edge map. This binarization process divides the pixels in the edge map into two categories: edge and non-edge.

[0065] Step 3: Predict the edge map. Repeat the above steps using the mask predicted by the model to obtain the predicted edge map.

[0066] Step 4: Calculate the edge loss and use the above dice loss formula to calculate the edge loss L edge , so far we get our training loss function L total For: L total =L Dice +L ce +L edge . BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1A schematic diagram of an image restoration network in a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion proposed in an embodiment of the present application is shown;

[0069] Figure 2 A specific composition diagram of a convolution block in a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion proposed in an embodiment of the present application is shown;

[0070] Figure 3 A schematic diagram of an image segmentation and classification network in a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion proposed in an embodiment of the present application is shown;

[0071] Figure 4 A process diagram of pixel-level FPN in a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion proposed in an embodiment of the present application is shown;

[0072] Figure 5 A schematic diagram of the structural design of a regional-level FPN in a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion proposed in an embodiment of the present application is shown;

[0073] Figure 6 A schematic diagram of a feature aggregation module (FAM) in a method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0074] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0075] Implementation example Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 As shown:

[0076] A method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion.

[0077] The steps for building the diagnostic model are as follows:

[0078] S01: Image collection and annotation: Obtain cardiac MR images as sample data, and obtain the cardiac labels corresponding to the MR images after manual annotation;

[0079] Specifically: obtain cardiac MR images as sample data, and obtain the heart labels corresponding to the MR images after manual labeling. The labels include 2 different labels, 0 and 1. Specifically, label 0 is the background label, and label 1 is the heart label. At the same time, make a judgment on whether the heart is fibrotic, and also give labels 0 and 1. 0 represents normal and 1 represents fibrosis. The labeled sample data is divided into a training set and a test set. Both the training set and the test set have two labels. Label 0 is the background label, and label 1 is the heart label.

[0080] S02: Image preprocessing: including normalization, data enhancement and data cropping. The image is preprocessed using normalization, data enhancement and data cropping.

[0081] Specifically: in step S02, normalization processing is performed: (x-μ) / σ, where x is the HU value of the pixel in the cardiac MR image, u is the mean HU value of all pixels, and σ is the standard deviation of all pixels. Data enhancement: including the use of random Gaussian noise, random contrast enhancement, random mirror flipping, random horizontal flipping and random rotation. Data cropping: cropping the data to a size of [128,128,128].

[0082] S03: Model construction: including image restoration network construction and image segmentation network construction. The image restoration network uses self-supervised learning to perform image restoration tasks; the image segmentation network is used to segment cardiac images and diagnose cardiac fibrosis;

[0083] The details are as follows:

[0084] Image restoration network construction: including encoder and decoder. The encoder is part A, which includes convolutional layer, Res-Block and pooling layer.

[0085] Convolutional layer: used to extract cardiac fibrosis image features;

[0086] Res-Block: It consists of two convolutional blocks, making it easier for the network to learn the identity mapping;

[0087] Pooling layer: highlights the significant features in cardiac fibrosis images while reducing the amount of calculation and the number of parameters;

[0088] Specific:

[0089] The network consists of an encoder and a decoder structure. The encoder is part A, which includes convolutional layers, Res-Block and pooling layers. Let's talk about the encoder part first. The convolutional layer uses 3x 3x 3 convolutions. Its function is to extract cardiac fibrosis image features, such as texture and edge information. The role of the pooling layer is to highlight the significant features in cardiac fibrosis images while reducing the amount of calculation and the number of parameters. Finally, Res-BlockK consists of two convolutional blocks. The specific composition of the convolutional block is as follows: Figure 2 As shown in the figure, the convolution layer further extracts the advanced features of cardiac MR images, the LeakyReLU activation function introduces nonlinear changes, allowing the model to learn and simulate more complex function mappings, and the BN layer is used to speed up training and improve model generalization. By adjusting and scaling the activation output, the input distribution of each layer is more stable, reducing the problem of internal covariate shift. The short-circuit structure makes it easier for the network to learn the identity mapping, thereby solving the degradation problem in deep networks.

[0090] The decoder is part B, which includes upsampling layers, convolution blocks, and short-circuit structures;

[0091] Upsampling layer: Use the ConvTranspose layer to upsample the low-resolution feature map to a higher resolution;

[0092] Convolutional block: The same composition as the convolutional layer in the image restoration network construction;

[0093] Short-circuit structure: directly connect the low-level features in the encoder to the corresponding layers of the decoder;

[0094] The decoder is part B, which includes upsampling layers, convolution blocks and short-circuit structures. Its function is to further refine and upsample the feature maps extracted by the encoder to restore the resolution of the original input image. The composition of the convolution block is the same as above. The upsampling uses the ConvTranspose layer, which is a transposed convolution operation that upsamples the low-resolution feature map to a higher resolution by learning. Compared with traditional upsampling methods (such as nearest neighbor interpolation), the ConvTranspose layer can more effectively learn the upsampling process of features because it utilizes the structural information in the training data. For image restoration tasks, it can reduce image artifacts and blur. The short-circuit structure connects the low-level features in the encoder directly to the corresponding layers of the decoder, making it easier for the network to learn the identity mapping, solving the gradient vanishing problem in deep networks, and allowing the network to retain the fine features of the low layer during the upsampling process, which is very effective for accurately restoring cardiac fibrosis images. The short-circuit structure also enhances the representation ability of the model, allowing the model to learn global and local features at the same time, further increasing the restoration effect.

[0095] Image segmentation network construction: Improve based on Mask R-CNN, build the network BackBone, build feature pyramid network FPN and build feature aggregation module FAM;

[0096] Specific: such as Figure 3 As shown in the figure above, the overall structure of the network is improved based on Mask R-CNN. The first improvement is the BackBone of the network. The encoding layer of the image restoration network in the first stage is used to replace it, and the parameters of the encoding layer are migrated. The image restoration network completes the task of restoring cardiac images through self-supervised learning. The encoding layer learns the semantic information and typical features of cardiac images. Migrating network parameters can accelerate the convergence speed of the improved Mask R-CNN model and improve the model's feature extraction ability for cardiac fibrosis images. The second improvement is the feature pyramid network FPN. The present invention introduces a two-level feature pyramid network structure, including pixel-level FPN and regional-level FPN, which can capture the global context and local details of cardiac MR images and achieve more refined feature fusion. The third improvement is the designed feature aggregation module FAM. By dynamically adjusting the regional-level feature weights and pixel-level feature weights, the model's learning of key features of cardiac fibrosis images is enhanced. At the same time, the feature fusion mechanism of self-attention improves the segmentation accuracy and diagnostic accuracy of the network model for cardiac fibrosis images.

[0097] like Figure 4 As shown in the figure above, the pixel-FPN process is a top-down network structure that captures contextual information in images through feature maps of multiple scales. These feature maps come from different levels of the encoding layer in the image restoration network, and each level corresponds to a different resolution, thus forming a multi-scale feature representation. By integrating semantic information from different levels of the network, the model can understand the image content at different scales. The top-down structural design enables high-level semantic information to be effectively transmitted to low levels, enhancing the model's understanding of the global context. At the same time, the constructed feature pyramid reuses the same features at multiple resolutions, improving computational efficiency.

[0098] like Figure 5 As shown,

[0099] The above figure shows the structural design of the region-FPN. Region-FPN is a new structure proposed in this invention. It starts with the regional features of RoI alignment and generates a top-down regional feature hierarchy by gradually fusing the complementary information from pixel-FPN. The spatial resolution of these features gradually increases with the level. Region-FPN focuses on enhancing regional features, enabling the model to capture more fine-grained visual information of cardiac fibrosis images, especially on the edges and details of the target. The feature aggregation module (FAM) uses the self-attention fusion mechanism to increase the effective fusion of regional and pixel-level features. RPN has the same role as in the Mask R-CNN network, quickly generating high-quality candidate regions on the feature map, so that the network can effectively detect targets of various sizes. The role of the ROI Align network is to map the candidate region ROI generated by RPN to a fixed-size feature map for accurate target classification and mask generation. It is worth noting that we use the 3D version of Mask R-CNN, so compared with the traditional one, here we choose trilinear interpolation to interpolate features in three dimensions in ROI Align, but its function is consistent with ROI Align in the classic Mask R-CNN. The candidate region generated by RPN is accurately mapped to a fixed-size feature map through the ROI Align network for subsequent segmentation and classification. For classification, the feature map is subsequently expanded and connected to two fully connected layers FC for judgment.

[0100] like Figure 6 As shown,

[0101] As shown in the figure above, the feature aggregation module (FAM) consists of an upsampling layer, a convolution layer, a GAP module, and a softmax module. Zi is the feature after ROI Align alignment, that is, the pixel-FPN feature, and Pi is the region-FPN feature. FAM first upsamples the Zi feature to keep it consistent with the Pi feature size, then adds the two features for fusion, performs GAP on the added feature map, that is, global average pooling, generates a channel feature vector F1, uses one-dimensional convolution on the channel feature vector, performs information interaction between channels to obtain the feature vector F2, and then uses softmax to weight the feature vector channel, calculates the weight vectors α and β corresponding to the Zi feature and the Pi feature, multiplies the weight vector with the feature map, and obtains the attention enhancement feature map of Zi and the attention enhancement feature map of Pi. Finally, the two enhanced feature maps are added to obtain the mixed enhanced feature map. The FAM module enhances the model's learning of the key features of cardiac images. At the same time, the feature fusion mechanism based on self-attention helps the model better learn the mixed features of cardiac images and improves the segmentation accuracy of the network model for cardiac fibrosis tasks.

[0102] S04: Image pre-training: Train the image restoration network to allow the encoder of the image restoration network to fully learn the characteristics of cardiac fibrosis images;

[0103] Specifically: In step S04, the image restoration network is trained, with the goal of allowing the network encoder to fully learn the characteristics of cardiac fibrosis images. In the model training, the Adam optimizer is used for model training, with momentum parameters of 0.93 and 0.99 set, the initial learning rate set to 0.0001, and a total of 5000 rounds of training. The goal is to repair the randomly occluded parts of the input image, and the loss function uses the MSE loss function, which measures the mean of the square of the difference between the model prediction value and the actual value. The specific formula is as follows:

[0104]

[0105] N is the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.

[0106] S05: Model training: After the model pre-training is completed, the obtained image restoration network encoder is used as the BackBone in Mask R-CNN. At the same time, the encoder parameters trained in the first stage are migrated to Mask R-CNN, and ReduceLROnPlateau learning rate automatic attenuation is used for model training.

[0107] Specific:

[0108] After the model pre-training is completed, the obtained image restoration network encoder is used as the BackBone in Mask R-CNN. At the same time, the encoder parameters trained in the first stage are migrated to Mask R-CNN. The initial learning rate is set to 0.0001, and the ReduceLROnPlateau learning rate is automatically decayed. The patience parameter is set to 50 times. When the loss does not decrease in 50 consecutive Epochs, the learning rate is automatically reduced to 1 / 10 of the original. A total of 5000 rounds of training are performed, and the loss function uses a mixture of multiple losses, including dice loss for calculating segmentation loss, cross entropy loss for calculating classification loss, and edge loss proposed for the unclear boundary of cardiac fibrosis lesion area.

[0109] The formula of dice loss is as follows:

[0110]

[0111] Where T is the true annotation, P is the predicted area, and the Dice loss function maximizes the intersection between the model output and the true label.

[0112] In step S05, the formula of Cross entropy is as follows:

[0113]

[0114] Where y is the true label, is the predicted probability value, which is between 0 and 1.

[0115] In view of the unclear boundary of cardiac fibrosis, the edge loss calculation method is designed as follows:

[0116] Step 1: Generate edge map Use the Laplacian operator to operate the ground-truth mask to obtain a soft edge map that reflects edge information. The Laplacian operator is a second-order derivative operator that can highlight the edges and details in the image. The specific steps are as follows:

[0117] 1. Define the Laplace kernel. The kernel used is as follows:

[0118]

[0119] 2. Convolution operation: Use the above Laplacian kernel and the true mask to perform convolution operation. For each pixel in the true mask, multiply the pixel value in its neighborhood with the corresponding value in the Laplacian kernel, and then sum the results to get the new value of the pixel.

[0120] 3. Result conversion: superimpose the image processed by the Laplace filter kernel with the original image to enhance the edge while retaining the content of the image edge;

[0121] The second step is to perform thresholding on the soft edge map obtained by the Laplacian operator and convert it into a binary edge map. This binarization process divides the pixels in the edge map into two categories: edge and non-edge.

[0122] Step 3: Predict the edge map. Repeat the above steps using the mask predicted by the model to obtain the predicted edge map.

[0123] Step 4: Calculate the edge loss and use the above dice loss formula to calculate the edge loss L edge , so far we get our training loss function L total For: L total =L Dice +L ce +L edge .

[0124] The key points of the present invention are:

[0125] 1. The network uses an improved Mask R-CNN algorithm. The present invention introduces an innovative two-level feature pyramid network structure, including pixel-FPN and region-FPN, and also designs an efficient feature aggregation module FAM.

[0126] 2. Construct an encoder-decoder architecture for self-supervised learning. The encoder part uses the ResNet module as the basis. The input of the model is the original image processed by random mask. The goal of the model is to reconstruct the part of the image that is blocked by the mask. At the same time, we implement a variety of data enhancement techniques on the MR images after random masking. The trained encoder model is used as the backbone and integrated into the improved Mask R-CNN algorithm.

[0127] 3. The present invention uses an improved Mask R-CNN algorithm to segment and classify cardiac MR images through a model. This is a multi-task learning for the model, which can increase the generalization ability of the model, while solving the problem of requiring manual labeling and realizing end-to-end network structure design.

[0128] 4. Use multiple loss functions, classification loss, bounding box loss, Dice loss, and the proposed margin loss.

[0129] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion, characterized in that: The steps for building the diagnostic model are as follows: S01: Image collection and annotation: Obtain cardiac MR images as sample data, and obtain the cardiac labels corresponding to the MR images after manual annotation; S02: Image preprocessing: including normalization, data enhancement and data cropping. The image is preprocessed using normalization, data enhancement and data cropping. S03: Model construction: including image restoration network construction and image segmentation network construction. The image restoration network uses self-supervised learning to perform image restoration tasks; the image segmentation network is used to segment cardiac images and diagnose cardiac fibrosis; The details are as follows: The image restoration network is constructed as follows: it includes constructing an encoder and a decoder. The encoder is part A, which includes convolutional layers, Res-Block and pooling layers. Convolutional layer: used to extract cardiac fibrosis image features; Res-Block: It consists of two convolutional blocks, making it easier for the network to learn the identity mapping; Pooling layer: highlights the significant features in cardiac fibrosis images while reducing the amount of calculation and the number of parameters; The decoder is part B, which includes upsampling layers, convolution blocks, and short-circuit structures; Upsampling layer: Use the ConvTranspose layer to upsample the low-resolution feature map to a higher resolution; Convolutional block: The specific components include: convolutional layer, LeakyReLU activation function and BN layer; Short-circuit structure: directly connect the low-level features in the encoder to the corresponding layers of the decoder; The image segmentation network is constructed as follows: improve the Mask R-CNN, build the network BackBone, build the feature pyramid network FPN, and build the feature aggregation module FAM; S04: Image pre-training: Train the image restoration network to allow the encoder of the image restoration network to fully learn the characteristics of cardiac fibrosis images; S05: Model training: After the model pre-training is completed, the obtained image restoration network encoder is used as the BackBone in the improved Mask R-CNN. At the same time, the encoder parameters trained in the first stage are migrated to Mask R-CNN, and ReduceLROnPlateau learning rate automatic attenuation is used for model training.

2. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 1, characterized in that: In step S01, the heart label includes two different labels, 0 and 1. Specifically, the heart labels are: label 0 is the background label, label 1 is the heart label, and at the same time, a judgment is made on whether the heart is fibrotic. Labels 0 and 1 are also printed. Label 0 represents normal and label 1 represents fibrosis. The labeled sample data is divided into a training set and a test set.

3. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 2, characterized in that: In step S02, normalization processing is performed: (x-μ) / σ, where x is the HU value of the pixel in the cardiac MR image, u is the mean HU value of all pixels, and σ is the standard deviation of all pixels. Data enhancement: includes using random Gaussian noise, random contrast, random mirror flipping, random horizontal flipping and random rotation for enhancement.

4. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 1, characterized in that: In step S03, the cardiac fibrosis image features include texture and edge information. The convolution layer further extracts high-level features of the cardiac MR image. The LeakyReLU activation function introduces nonlinear changes, enabling the model to learn and simulate more complex function mappings. The BN layer is used to speed up training.

5. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 4, characterized in that: In step S03, the image segmentation network further refines and upsamples the feature map extracted by the encoder to restore the resolution of the original input image. The ConvTranspose layer uses the structural information in the training data to reduce image artifacts and blur.

6. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 5, characterized in that: In step S03, the overall structure of the network is improved based on Mask R-CNN. The first improvement is the BackBone of the network, which is replaced by the encoding layer of the image restoration network in step S03, and the parameters of the encoding layer are migrated. The image restoration network learns the cardiac image restoration task through self-supervision. The encoding layer learns the semantic information and typical features of the cardiac image. The network parameters are migrated to speed up the convergence speed of the improved Mask R-CNN model. The second improvement is the feature pyramid network FPN, which introduces a two-level feature pyramid network structure. The two-level feature pyramid network structure includes pixel-level FPN and region-level FPN, which captures the global context and local details of cardiac MR images. The feature aggregation module FAM dynamically adjusts the region-level feature weights and pixel-level feature weights.

7. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 6, characterized in that: In step S04, during the model training, the Adam optimizer is used for model training. The specific formula is as follows: N is the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.

8. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 7, characterized in that: In step S05, the patience parameter is set to 5 times. When the loss does not decrease in 50 consecutive epochs, the learning rate is automatically reduced to 1 / 10 of the original value. A total of 5000 rounds of training are performed. The loss function uses a mixture of multiple losses, including diceloss for calculating segmentation loss, cross entropy for calculating classification loss, and edge loss for the unclear boundary problem of cardiac fibrosis lesion area. The formula of dice loss is as follows: Where T is the true annotation, P is the predicted area, and the Dice loss function maximizes the intersection between the model output and the true label.

9. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 8, characterized in that: In step S05, the formula of Cross entropy is as follows: Where y is the true label, is the predicted probability value, which is between 0 and 1.

10. The method for constructing a cardiac fibrosis diagnosis model based on multi-task attention feature fusion according to claim 9, characterized in that: In view of the unclear boundary of cardiac fibrosis, the edge loss calculation method is designed as follows: Step 1: Generate edge map Use the Laplacian operator to operate the ground-truth mask to obtain a soft edge map that reflects edge information. The Laplacian operator is a second-order derivative operator that can highlight the edges and details in the image. The specific steps are as follows:

1. Define the Laplace kernel. The kernel used is as follows:

2. Convolution operation: Use the above Laplacian kernel and the true mask to perform convolution operation. For each pixel in the true mask, multiply the pixel value in its neighborhood with the corresponding value in the Laplacian kernel, and then sum the results to get the new value of the pixel.

3. Result conversion: superimpose the image processed by the Laplace filter kernel with the original image to enhance the edge while retaining the content of the image edge; The second step is to perform thresholding on the soft edge map obtained by the Laplacian operator and convert it into a binary edge map. This binarization process divides the pixels in the edge map into two categories: edge and non-edge. Step 3: Predict the edge map. Repeat the above steps using the mask predicted by the model to obtain the predicted edge map. Step 4: Calculate the edge loss and use the above diceloss formula to calculate the edge loss to obtain L edge , so far we get our training loss function L total For: L total =L Dice +L ce +L edge .

Images