A left ventricular quantification method based on deep multi-task regression network

Abstract

The present application relates to a kind of left ventricular quantification method based on deep multitask regression network, it utilizes model to carry out left ventricular quantification to the heart MR image to be processed, model includes feature extraction module and segmentation quantification module.Feature extraction module includes feature representation network, index estimation network.Feature representation network includes deep convolutional encoder-decoder module.Index estimation network includes MS-CNN module and bidirectional LSTM module.MS-CNN module utilizes multiple independent CNN model, adopts different kernel size and pooling size, from the high-level feature map of encoder-decoder module, extract more effective low-level feature.Bidirectional LSTM module is used to capture the dynamic change of heart structure.Segmentation quantification module includes the encoder-decoder structure of processing heart MR image.Encoder-decoder structure includes two sub-encoders connected by short-range jump connection.The present application solves the technical problem that existing technology lacks mutual assistance of segmentation and quantification task.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation, and in particular to a method for left ventricular quantification based on a deep multi-task regression network. Background Technology

[0002] Cardiac magnetic resonance (MR) images not only reflect the anatomical structure of the heart but also serve to assess and detect cardiovascular diseases, making the analysis and study of cardiac MR images of great significance. However, in clinical applications, clinicians often find it difficult to manually analyze the large volume of medical images. Therefore, utilizing computer-aided technology to assist clinicians in accurately and efficiently processing image data has become an indispensable part of clinical work.

[0003] Currently, computer-aided techniques for cardiac MR images include image segmentation and quantization. Image segmentation provides clinicians with visual assistance and qualitative analysis, while image quantization offers accurate and detailed quantitative analysis. Existing advanced methods can be categorized into three types: the first performs only contour segmentation on cardiac MR images; the second performs only functional index quantification analysis on cardiac MR images; and the third combines segmentation and quantization. These three types of methods will be briefly introduced below.

[0004] The first category, for left ventricular (LV) segmentation, primarily utilizes convolutional neural network (CNN) frameworks based on deep learning. For example, Khened et al. proposed a fully automated method that segments the LV, RV, and myocardium from cardiac MR images using a densely connected fully convolutional neural network. Zotti et al. proposed a novel CNN architecture that embeds the heart shape before segmenting the left and right ventricles. Avendi et al. combined deep learning-based algorithms with deformable models to achieve fully automated left ventricular segmentation. Besides these methods, clinical research approaches in segmentation are incredibly diverse, with some of the most representative works today including:

[0005] Yang et al. proposed a lightweight fully convolutional network model called LFCN for cardiac MRI segmentation tasks. LFCN is an improvement on the U-Net model, introducing separable convolutions and residual connections to reduce model size. By replacing traditional convolution operations with separable convolutions, LFCN reduces the number of parameters and increases network depth, while maintaining the stability of deep networks through residual connections. LFCN employs an encoder-decoder structure, consisting of two parts. In the downsampling path (encoder), the input cardiac MR image is processed through separable blocks (SB) and downsampling blocks (TB) to obtain high-level semantic features. Then, in the upsampling path (decoder), the spatial resolution of the original image is restored using transposed convolutions, separable blocks (SB), and skip connections from the downsampling path. The output feature map of the last separable block (SB) is passed through a 1×1 convolution and the final segmentation result is generated using a softmax activation function. Through these improvements, LFCN significantly reduces model size while maintaining accuracy.

[0006] Le et al. proposed a heart segmentation model called DR-Unet++, which is an improvement on DR-Unet. In this model, they retained the design of double ResPath blocks and PASPP blocks, while introducing DC blocks to replace the remaining blocks for more efficient extraction of spatial information, and adding SE blocks to enhance model attention to important features. Through these improvements, the DR-Unet++ model showed better performance and accuracy on the heart segmentation task. However, DR-Unet++ requires more parameters than DR-Unet, thus significantly increasing training time and computational cost.

[0007] The second category concerns left ventricular (LV) quantification. Deep learning-based methods can generally be divided into two types: two-stage estimation methods and end-to-end estimation methods. Two-stage estimation methods, as the name suggests, consist of two stages. One stage is used for cardiac MR image representation, and the other stage is used for estimating relevant clinical indicators. In the cardiac MR image representation process, the image is typically represented using features extracted by manual methods or learning-based methods. For example, unsupervised cardiac image representation is learned by a multi-scale convolutional deep belief network. In the final indicator estimation stage, relevant clinical indicators are typically obtained through a deep regression model. The main idea of ​​end-to-end estimation methods is to build a neural network architecture and use cardiac MR images as input to the network to directly obtain the desired output (such as clinical indicators). For example, by selecting end-diastolic and end-systolic images as network input, the cardiac volume at end-diastolic and end-systolic is directly obtained through an end-diastolic and deep CNN network. In addition, there are other architectures, such as deep convolutional autoencoders (DCAEs) for cardiac image representation and multi-output convolutional neural networks for multi-type exponential regression. As can be seen, there are many research methods in the field of cardiac quantitative analysis. Below are some advanced clinical research methods for left ventricular (LV) quantification in recent years.

[0008] Xue et al. proposed a deep multi-task learning network called FullLVNet for fully quantifying left ventricular images with intra- and inter-task correlations. The network consists of a deep convolutional neural network (CNN) for feature embedding, two parallel recurrent neural network (RNN) modules for temporal dynamics modeling, and four linear models for final estimation. In the final estimation stage, FullLVNet improves generalization through two methods: grouping regularization of intra- and inter-task correlations and phase-guided constraints. By fully utilizing the expressive features extracted by deep CNNs and the advantages of RNNs in effectively modeling dynamic time, along with a two-step strategy, FullLVNet achieves state-of-the-art accuracy for quantization tasks.

[0009] Xue et al. also proposed a model called Residual Recurrent Neural Network (ResRNN). ResRNN consists of two networks: a CNN network and an RNN network. The CNN network extracts low-dimensional representations from cardiac MR images and makes a preliminary estimate of the relative wall thickness (RWT) for each frame. The RNN network then utilizes the differences between adjacent frames to perform a more accurate RWT estimate. In the RNN network, the temporal RNN models the features of temporally adjacent frames to capture long-term temporal dependencies, while the spatial RNN models the interdependencies between wall thickness predictions. By using CNN-embedded images, both temporal and spatial RNNs can effectively capture the dynamic deformation of the myocardium and the diversity of heart shape, thereby achieving accurate quantitative estimation of RWT.

[0010] The third category involves methods that combine left ventricular (LV) segmentation and quantification. Current clinical research methods combine these two tasks in parallel, leveraging their complementary relationship to improve the accuracy of the research. These methods primarily include:

[0011] The Indices-JSQ method proposed by Du et al. uses a DCED (Digital Convolutional Edge Model) for image representation during the segmentation stage. This DCED is a fully convolutional neural network structure consisting of an encoder module and a corresponding decoder module. It uses the convolutional layers as an encoder for effective feature representation, and the decoder network maps low-resolution encoder features to full-input resolution features. The decoder uses the pooling index computed in the max-pooling step of the corresponding encoder for non-linear upsampling. In the cardiac MR image quantization stage, a multi-task regression model is designed to estimate left ventricular clinical parameters. By learning shared representations across multiple tasks, the number of data sources and the overall model parameter size can be reduced, improving the model's generalization ability and estimation efficiency.

[0012] The CSRNet method proposed by Wang et al. employs the DenseNet method of densely connected convolutional networks for segmentation during the segmentation stage, leveraging the dense connections between layers. This architecture reduces the number of parameters that need to be learned and encourages feature reuse throughout the network. After an image has undergone segmentation, the softly segmented image is fed into a single-scale CNN convolutional neural network for regression prediction.

[0013] Sulaiman Vesal et al. proposed a 2D+t spatiotemporal convolutional multi-task learning framework. In the segmentation stage, referencing the DR-UNet network architecture, this method introduces the ideas of dilated convolutions and residual connections. They create stacked dilated convolutional blocks whose outputs are summed together. In this way, each subsequent layer has full access to previously learned features at different dilation rates, allowing the network to capture global and local contextual information by augmenting its own fields. In the quantization stage, the network employs spatiotemporal convolution, considering not only spatial but also temporal information. 3D CNNs are applied to 2D+t image frames, preserving the corresponding temporal information and propagating it to subsequent network processing layers. The use of these modules enables it to represent more robust structural features and temporal information.

[0014] In summary, existing technologies have the following problems:

[0015] (1) Lack of mutual support between task segmentation and quantification

[0016] Simultaneous segmentation and quantization of cardiac MR images is of great significance for the auxiliary diagnosis and differential diagnosis of cardiovascular diseases. An increasing number of cardiac image-assisted techniques aim to simultaneously perform segmentation and quantization tasks. However, previous techniques have some shortcomings, failing to fully integrate segmentation and quantization tasks to achieve mutual complementarity. Currently, there are two main methods for simultaneous segmentation and quantization of cardiac images: The first method involves first segmenting the image, and then quantizing the heart based on the results obtained from the segmentation network. The quantization result of this method is greatly affected by the segmentation result, requiring efforts to reduce segmentation errors. The second method involves inputting the image into two parallel channels, performing segmentation and quantization tasks independently, and ultimately generating segmentation and quantization results simultaneously. However, previous methods essentially still treat segmentation and quantization as two separate tasks, failing to effectively unify them and leaving a certain disconnect.

[0017] (2) Insufficient utilization of contextual information in long sequence processing

[0018] Common methods for capturing sequence data include traditional RNNs (Recurrent Neural Networks), unidirectional LSTMs, GRUs (Gate-based Recurrent Neural Networks), Transformers (Neural Networks based on self-attention mechanisms), and ConvLSTMs (Network Architectures combining Convolutional Neural Networks and LSTMs). However, these common sequence data capture networks generally suffer from insufficient utilization of contextual information in long sequence processing. For example, traditional RNNs suffer from gradient vanishing and gradient exploding when handling long-term dependencies, making it difficult to effectively model long-term memory. Unidirectional LSTMs and GRUs can only consider past contextual features and cannot simultaneously utilize future information. This may lead to the loss of certain key information in the sequence or the inability to accurately predict subsequent events. Compared to LSTMs and GRUs, Transformers typically require more parameters to model sequence data. This can lead to higher computational costs and greater storage requirements, especially when processing large-scale datasets.

[0019] (3) Insufficient acquisition of detailed information at different scales

[0020] Common feature extraction networks include those based on convolutional neural networks (CNNs), autoencoders, pre-trained models, and attention mechanisms. However, these classic feature extraction networks are insufficient in extracting detailed information. For example, traditional CNNs are limited in processing multi-scale information; they typically use fixed convolutional kernel and pooling layer sizes, making them unable to flexibly adapt to inputs of different scales. Autoencoders tend to learn low-dimensional representations of input data but may ignore some high-level semantic information. Summary of the Invention

[0021] To address the technical problem of the lack of mutual assistance between segmentation and quantization tasks in existing technologies, this invention provides a left ventricular quantification method based on a deep multi-task regression network.

[0022] This invention is achieved using the following technical solution:

[0023] A method for left ventricular quantification based on a deep multi-task regression network includes the following steps:

[0024] Data processing is performed on several preset cardiac MR images;

[0025] A model is built based on several pre-set cardiac MR images after data processing; the model includes a feature extraction module and a segmentation and quantization module.

[0026] The model was used to quantize the left ventricle in the cardiac MR images to be processed.

[0027] The feature extraction module includes a feature representation network and an exponential estimation network. The feature representation network includes a deep convolutional encoder-decoder module for extracting cardiac structural features relevant to segmentation and quantization tasks. The exponential estimation network includes an MS-CNN module and a bidirectional LSTM module. The MS-CNN module utilizes multiple independent CNN models with different kernel sizes and pooling sizes to extract more effective low-level features from the high-level feature maps of the encoder-decoder module for cardiac image processing. The bidirectional LSTM module is used to capture dynamic changes in cardiac structures.

[0028] The segmentation and quantization module includes an encoder-decoder structure for processing cardiac MR images. The encoder-decoder structure comprises two sub-encoders connected by short-range jumps. Each sub-encoder contains four blocks, each with a similar structure, including convolutional layers, activation functions, max-pooling layers, and dropout layers from high to low. These convolutional layers progressively extract feature information from the input image from high to low. In the second convolutional layer of each block, dilated convolutions are used to increase the receptive field. The decoder module is similar to the encoder module, also consisting of four blocks. Upsampling layers are used to restore the low-level feature maps to the resolution of the input image. The other layers in the decoder module have the same parameter settings as the encoder module, and feature integration is performed on the remaining network blocks connected by long-range and short-range jumps. Through layer-by-layer upsampling and feature fusion, the decoder module generates an output feature map with the same size and semantic information as the input image, achieving accurate image segmentation.

[0029] As a further improvement to the above scheme, the data processing method includes data preprocessing and data labeling.

[0030] Furthermore, the data preprocessing includes: feature point marking, image rotation, ROI cropping, and resizing. First, the heart region is located and key points are marked on each preset cardiac MR image. Then, the image is rotated to a predetermined angle based on the key points. Next, a suitable ROI region is selected and cropped on the rotated image. Finally, all cropped images are resized to a predetermined pixel size. The suitable ROI region refers to an ROI region that can completely contain the entire outline of the heart.

[0031] Furthermore, the data annotation includes: normalizing the actual clinical physical values ​​of the left ventricular chamber diameter and myocardial ventricular wall thickness according to the size of the predetermined pixel, and simultaneously normalizing the area of ​​the myocardium and the heart chamber according to the image area of ​​the predetermined pixel.

[0032] As a further improvement to the above scheme, after the image passes through the feature representation network, it is input into the MS-CNN module of the exponential estimation network to obtain multi-level local features. MS-CNN includes three parallel CNN models, which perform multi-level feature extraction through a multi-scale strategy, and the outputs of the three CNN models are fused as the final output of MS-CNN.

[0033] Furthermore, a bidirectional LSTM module is set up at the bottom of the exponential estimation network to capture the dynamic deformation of the image, with a stride of 20, and the image features extracted by MS-CNN are used as its input.

[0034] As a further improvement to the above scheme, the model is trained before it is used.

[0035] Furthermore, the model training employs a five-fold cross-training method to rapidly train the entire model.

[0036] Furthermore, after training, the model needs to be tested and evaluated.

[0037] Preferably, the prediction results are denormalized to convert them into actual physical values ​​in order to obtain metrics for evaluating the performance of the network framework.

[0038] In summary, the present invention has the following beneficial effects.

[0039] 1. Simultaneously achieve segmentation and quantization of cardiac MR images:

[0040] The model can accurately segment cardiac MR images and, based on this, perform quantitative analysis of the heart. Simultaneously, the segmentation and quantification of cardiac MR images help doctors quickly obtain relevant information about the heart's structure and function, which is beneficial for the diagnosis and treatment of cardiovascular diseases such as coronary heart disease.

[0041] 2. Achieve multi-layer feature integration:

[0042] Unlike previous single codec modules, the model of this invention employs an integrated codec, replacing the previous single codec with a codec using short-range skip connections. A residual network block containing long-range and short-range skip connections is used for multi-level feature integration. For example, the information flow of a function arriving at the decoder module is named Dec1_d (d = 1, 2, 3), and long-range and short-range skip connections are fused from different function blocks, including Enc1_d, Dec1_d+1, and Dec2_d+1. In this way, the present invention ultimately achieves multi-level feature integration, resulting in more accurate segmentation performance.

[0043] 3. Make full use of contextual features:

[0044] In cardiac MR image quantization tasks, unlike RNN networks which suffer from the fatal weakness of vanishing gradients and traditional LSTMs which can only access information from previous contexts at each time step, the model in this invention uses a Bi-LSTM network to capture information from the sequence dataset and maintain contextual features from both the past and future. Therefore, this technique effectively solves the gradient vanishing / exploding problem faced by traditional RNN networks when processing long sequence data, while better utilizing past and future contextual information, thus improving the modeling and prediction capabilities for datasets with strong time dependencies.

[0045] The innovative design of this invention lies in:

[0046] 1. Design of an integrated deep convolutional encoding / decoding module

[0047] Unlike previous single codec modules, the model of this invention employs an integrated codec, replacing the previous single codec with a codec using short-range skip connections. A residual network block containing long-range and short-range skip connections is used for multi-level feature integration. For example, the information flow of a function arriving at the decoder module is named Dec1_d (d = 1, 2, 3), and long-range and short-range skip connections are fused from different function blocks, including Enc1_d, Dec1_d+1, and Dec2_d+1. In this way, the present invention ultimately achieves multi-level feature integration, resulting in more accurate segmentation performance.

[0048] 2. A novel multi-scale CNN design

[0049] In the index estimation stage, this invention employs a novel multi-scale CNN. After passing through the feature representation network, the extracted features are sent to the multi-scale CNN module in the index estimation stage, achieving the goal of obtaining multi-level local features. The multi-scale CNN module consists of three parallel CNN models. The first CNN model contains four convolutional layers with 3x3 kernels, 2x2 pooling windows, and a stride of 2. It is then connected to two fully connected layers, with a dropout layer added after the first fully connected layer. The second CNN model contains three convolutional layers with 3x3 kernels, followed by a max-pooling layer, a dropout layer, and two fully connected layers. Unlike the first model, the max-pooling window in the second CNN model is set to 5x5 with a stride of 2. The third CNN model has an almost identical structure to the second CNN model, except that the kernel size is changed to 5x5. After MS-CNN employs a multi-scale strategy for multi-level feature extraction, this invention fuses the outputs of the three independent CNN models, using the local image features as the final output of MS-CNN.

[0050] 3. Achieving more full utilization of contextual features by introducing a Bi-LSTM network.

[0051] In cardiac MR image quantization tasks, unlike RNN networks which suffer from the fatal weakness of vanishing gradients and traditional LSTMs which can only access information from previous contexts at each time step, this invention introduces a Bi-LSTM network to capture information from the sequence dataset and maintain contextual features from both the past and future. Therefore, this technique effectively solves the gradient vanishing / exploding problem faced by traditional RNN networks when processing long sequence data, while better utilizing past and future contextual information, thus improving the modeling and prediction capabilities for datasets with strong time dependencies. Attached Figure Description

[0052] Figure 1 This is a graph showing the relationship between left ventricular segmentation contour plots and quantified clinical indicators.

[0053] Figure 2 This is a simplified diagram illustrating the problem that this invention can solve compared to traditional methods.

[0054] Figure 3 This is a flowchart of the left ventricular quantification method based on a deep multi-task regression network according to the present invention.

[0055] Figure 4 for Figure 3 An enlarged view of the left-hand area.

[0056] Figure 5 for Figure 3 An enlarged view of the right-hand area.

[0057] Figure 6 This is a schematic diagram of the five-fold cross-training method used in this invention.

[0058] Figure 7 This is a diagram of the model architecture used in this invention.

[0059] Figure 8 for Figure 7 An enlarged view of the upper left area.

[0060] Figure 9 for Figure 7 An enlarged view of the upper right area.

[0061] Figure 10 for Figure 7 An enlarged view of the lower left area.

[0062] Figure 11 for Figure 7 An enlarged view of the lower right area.

[0063] Figure 12 This is the detailed architecture of the MS-CNN module used in this invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] It should be noted that when a component is said to be "installed on" another component, it can be directly on the other component or it may be in a component that is centered on it. When a component is said to be "set on" another component, it can be directly set on the other component or it may also be in a component that is centered on it. When a component is said to be "fixed to" another component, it can be directly fixed to the other component or it may also be in a component that is centered on it.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0067] This invention, through comparison with excellent methods in recent years, reveals that methods combining segmentation and quantization are often more effective than methods performing segmentation or quantization separately. Research shows that in the field of left ventricular (LV) segmentation and quantization, segmentation is closely related to quantization; the effectiveness of segmentation often determines the accuracy of quantization, while the quantization indicators often reflect the effectiveness of segmentation. Figure 1 It can be clearly seen that the area of ​​the segmented myocardial outline is the area of ​​the myocardium, and the area of ​​the cardiac chamber is its internal region. Figure 1 This is a graph showing the relationship between the left ventricular segmentation contour map and quantitative clinical indicators. Region (a) represents the segmentation contour obtained from the original image, region (b) represents the area of ​​the center of the segmentation map, region (c) represents the diameter of the left ventricular chamber in three directions, and region (d) represents the different thicknesses of the myocardial wall in different directions in the segmentation map.

[0068] In clinical observation, the diameter of the cardiac chambers and the thickness of the myocardial wall in various directions are crucial, and segmentation maps can effectively reflect these clinical indicators. Therefore, considering only segmentation or quantization while neglecting the relationship between the two often fails to yield the desired results. Typically, segmentation is achieved by assigning pixels or regions in an image to specific categories or labels, while quantization processes these categories or labels numerically. Quantization maps each pixel or region in the segmentation result to a specific value or label, enabling quantitative description and statistical analysis of the segmentation results. Therefore, after quantization, the segmentation results can be better understood and characterized, supporting quantitative evaluation and comparison of the performance of different methods, and providing a foundation for further analysis and applications. Quantization provides more detailed, accurate, and comparable information, thereby helping this invention better understand and utilize the segmentation results.

[0069] To better process cardiac MR images, this invention attempts to combine segmentation and quantization tasks. In the field of medical imaging, existing research has shown that simultaneous segmentation and quantization are necessary. This allows for the effective learning of the interrelationship between segmentation and quantization, and the two tasks can share information and mutually reinforce each other during the learning process. A review of current advanced clinical methods fully demonstrates that using a multi-task learning framework for simultaneous segmentation and quantization of the left ventricle is a reasonable and effective approach.

[0070] The method of this invention aims to improve the accuracy and efficiency of cardiac image segmentation and quantization, and overcomes the limitations of traditional methods. By introducing a new network structure and optimization algorithm, the method of this invention can better capture the details and features of cardiac structures, achieving more accurate segmentation and quantization results.

[0071] The method of this invention aims to improve the accuracy and efficiency of cardiac image segmentation and quantization, and overcomes the limitations of traditional methods. By introducing a new network structure and optimization algorithm, the method of this invention can better capture the details and features of cardiac structures, achieving more accurate segmentation and quantization results. The main technical problems solved by this invention are as follows: Figure 2 The diagram shown illustrates the problems that this invention can solve compared to traditional methods: (1) lack of mutual assistance between segmentation and quantization tasks; (2) insufficient utilization of contextual information in long sequence processing; and (3) insufficient acquisition of detailed information at different scales. These three technical problems have been described in the background section and will not be repeated here.

[0072] This invention belongs to the field of medical image analysis and proposes a deep learning-based model for cardiac segmentation and quantization, comprising four modules: data processing, model building, model training, and model testing and evaluation. This invention can help clinicians analyze cardiac images during diagnosis and treatment, avoiding the deterioration of heart disease due to inaccurate diagnosis, and has significant clinical application value and market prospects.

[0073] The deep learning-based heart segmentation and quantization model of this invention includes the following four modules (such as...). Figure 3 As shown in the image, the model building section includes: a feature extraction module and a segmentation and quantization module; the model testing and evaluation section includes evaluation metrics, quantitative analysis, qualitative analysis, and comparative analysis. To better illustrate... Figure 3 The content is for Figure 3 To zoom in on a specific area, such as... Figure 4 and Figure 5 As shown, Figure 4 for Figure 3 An enlarged view of the left-hand area; Figure 5 for Figure 3 An enlarged view of the right-hand area.

[0074] Module 1: Data Processing

[0075] Step 1 (S1): Data preprocessing, including feature point marking, rotation, ROI cropping, and resizing operations, ultimately yielding an 80×80 pixel image;

[0076] Step 2 (S2): Data annotation. After obtaining uniformly sized cardiac MR images through image preprocessing, this invention requires data annotation.

[0077] Module 2: Model Building

[0078] Step 3 (S3): Build the feature extraction module;

[0079] Step 4 (S4): Build the segmentation and quantization module;

[0080] 3. Module Three: Model Training

[0081] Step 5 (S5): Train the model by using the five-fold cross-training method to quickly train the entire model;

[0082] 4. Module Four: Model Testing and Evaluation

[0083] Step 6 (S6): Denormalize the prediction results to convert them into actual physical values ​​in order to obtain metrics for evaluating the performance of the network framework.

[0084] Step 7 (S7): Evaluate the model results through qualitative analysis, quantitative analysis, and comparative analysis.

[0085] Of course, these are the tasks that need to be done before the model is officially run. Once the model is trained and used, the left ventricular quantization method based on deep multi-task regression network mainly includes the following steps: data processing of several preset cardiac MR images; model building based on several preset cardiac MR images after data processing; the model includes a feature extraction module and a segmentation and quantization module; and left ventricular quantization of the cardiac MR images to be processed using the model.

[0086] Module 1, the data processing section, includes two parts: data preprocessing and data labeling.

[0087] (S1) Data preprocessing includes feature point labeling, image rotation, ROI cropping, and resizing. First, the heart region is located on the cardiac MR image and key points are labeled. Then, the image is rotated based on the key points to obtain an appropriate angle. Next, suitable ROI regions are selected on the rotated image and cropped. Finally, all cropped images are resized to 80×80 pixels.

[0088] (S2) In the data annotation stage, the actual clinical physical values ​​of the left ventricular chamber diameter and myocardial wall thickness were normalized to an 80-pixel size, and the area of ​​the myocardium and heart chambers was also normalized to an 80×80-pixel image area. This ensures the consistency of scale between different images and provides an accurate data basis for subsequent analysis and comparison.

[0089] Therefore, the relevant data processing details include: data preprocessing and data labeling.

[0090] Before using this dataset of cardiac images, this invention requires several preprocessing steps, including feature point labeling, rotation, ROI cropping, and resizing. First, feature point labeling is performed. This invention first locates the cardiac region on the cardiac MR image and identifies and labels two key points within that region. Next, image rotation is performed. Using the labeled key points as references, this invention rotates the cardiac MR image to an appropriate angle. Then, ROI cropping is performed. On the rotated image, this invention selects a suitable ROI region that completely encompasses the entire outline of the heart and crops it. Finally, resizing is performed. Since the cropped images may have different sizes, this invention uniformly resizes all cropped images to 80×80 pixels. After these preprocessing steps, the images obtained by this invention basically meet the requirements of the experiment.

[0091] After image preprocessing, this invention yields cardiac MR images of uniform size. Now, this invention requires data annotation of these images. This invention normalizes the true clinical physical values ​​of the left ventricular chamber diameter and myocardial wall thickness to an image size of 80 pixels. Simultaneously, the areas of the myocardium and heart chambers are also normalized to an image area of ​​80×80 pixels. This ensures scale consistency between different images and provides an accurate data basis for subsequent analysis and comparison.

[0092] Module 2 model building includes two parts: feature extraction module and segmentation and quantization module.

[0093] (S3) Feature Extraction Module: This module consists of a Feature Representation Network (FRN), a deep convolutional encoder-decoder module, used to extract cardiac structural features relevant to segmentation and quantization tasks. The Index Estimation Network (IEN) comprises an MS-CNN module and a bidirectional LSTM module. The MS-CNN utilizes multiple independent CNN models with different kernel and pooling sizes to extract more effective low-level features from the high-level feature maps of the encoder-decoder module for cardiac image processing. The bidirectional LSTM module is used to capture dynamic changes in cardiac structures. This design enables the entire model to estimate cardiac parameters more accurately and provide more comprehensive structural information.

[0094] (S4) Constructing the Segmentation and Quantization Module: In cardiac image segmentation, this invention employs an encoder-decoder structure to process cardiac MR images. The encoder module consists of two related sub-encoders (Encoder 1 and Encoder 2), connected by short-range skip connections. Each sub-encoder contains four blocks, each with a similar structure, including convolutional layers, activation functions, max-pooling layers, and dropout layers. These convolutional layers progressively extract feature information from the input image from high to low layers. In the second convolutional layer of each block, this invention uses dilated convolutions to increase the receptive field. Then, the outputs of the two sub-encoders are merged through short-range skip connections to better represent the image features. The decoder module is similar to the encoder module, also consisting of four blocks, where upsampling layers are used to restore the low-level feature maps to the resolution of the input image. The other layers in the decoder module have the same parameter settings as the encoder module and feature integration is performed on the remaining network blocks through long-range and short-range skip connections. Through layer-by-layer upsampling and feature fusion, the decoder module produces an output feature map with the same size and semantic information as the input image, achieving accurate image segmentation. This design can accurately acquire detailed information from heart images and capture the overall features of the image, thereby improving segmentation performance.

[0095] After passing through the Feature Representation Network (FRN), the image is input into the MS-CNN module of the Exponential Estimation Network (IEN) to obtain multi-level local features. MS-CNN consists of three parallel CNN models that perform multi-level feature extraction using a multi-scale strategy, fusing the outputs of the three CNN models as the final output of MS-CNN. A bidirectional LSTM module, located at the bottom of the IEN with a stride of 20, is used to capture dynamic deformations of the image and takes the image features extracted by MS-CNN as its input.

[0096] Therefore, this invention uses an encoder-decoder structure for cardiac image segmentation, and employs a bidirectional LSTM module and an MS-CNN module to achieve segmentation quantization, thereby obtaining accurate cardiac image segmentation quantization results.

[0097] Therefore, Module Two, Model Building, outlines the various modules for building the model, including: feature extraction, segmentation, and quantization. The specific architecture diagram of the model is shown below. Figure 7 As shown. For better demonstration Figure 7 The content in the middle, for Figure 7 To zoom in on a specific area, such as... Figure 8 , Figure 9 , Figure 10 , Figure 11 As shown, Figure 8 for Figure 7 An enlarged view of the upper left area; Figure 9 for Figure 7 An enlarged view of the upper right area; Figure 10 for Figure 7 An enlarged view of the lower left area; Figure 11 for Figure 7 An enlarged view of the lower right area.

[0098] For the feature extraction module, the Feature Representation Network (FRN) consists of a deep convolutional encoder-decoder module, which extracts features relevant to segmentation and quantization tasks to reflect cardiac structural information. Based on the features of the FRN, pixel-level classification of the feature maps allows the invention to directly generate contour images of the left ventricle. The Index Estimation Network (IEN) comprises an MS-CNN module and a bidirectional LSTM module. The MS-CNN module extracts more efficient low-level features from the high-level feature maps generated by the encoder-decoder module using multiple independent CNN models with different kernel and pooling sizes for cardiac image processing. The bidirectional LSTM module captures dynamic changes in cardiac structure. Therefore, after deploying the MS-CNN and bidirectional LSTM modules, the final index can be directly estimated by establishing a multi-task regression model that considers the relationships between multiple indices. This design enables the entire model to estimate cardiac indices more accurately and provide more comprehensive structural information.

[0099] The segmentation and quantization modules can be synthesized and introduced as a single segmentation and quantization module. In cardiac image segmentation, this invention employs an encoder-decoder structure to represent cardiac MR images. In the Feature Representation Network (FRN), this invention uses only convolutional layers without fully connected layers. This fully convolutional neural network structure ensures that the invention ultimately obtains a binary ventricular contour image with the same resolution as the input image, while simultaneously representing the global feature information of the input image. This design guarantees that the invention can accurately acquire detailed information of cardiac images and capture the overall features of the image. In the encoder module of the Feature Representation Network (FRN), this invention introduces a short-range skip connection mechanism, which consists of two related sub-encoders (encoder 1 and encoder 2). Each sub-encoder contains four blocks (Enc1 / 2_x, where x = 1, ..., 4), and the corresponding blocks of the two sub-encoders are connected by short-range skip connections (SR-SCx, where x = 1, ..., 4). For Encoder 1, it consists of four blocks (Enc1_x, where x = 1, ..., 4). Each block has a similar structure, including convolutional layers with a 1×1 kernel and ReLU activation, convolutional layers with a 3×3 kernel and ReLU activation, and max-pooling layers with a 2×2 kernel and a 25% dropout layer. These convolutional layers are used to extract feature maps from the input image, progressively extracting feature information from higher to lower layers, from the first to the last convolutional layer of the encoder module. However, this process generates a large number of training parameters. Therefore, this invention adds max-pooling and dropout layers after each convolutional layer to filter out unnecessary redundant information in the feature maps and reduce the number of training parameters in the network. In the second convolutional layer (3×3 kernel size) of each block, this invention uses a layer with a dilated convolution factor of 2 or 3 to obtain a larger receptive field. The advantage of applying dilated convolution is that it increases the receptive field of the convolutional kernel while keeping the number of parameters constant, and at the same time ensures that the output size of the feature map remains unchanged. Then, this invention merges the outputs of the two sub-encoder blocks through short-range skip connections. Compared to a single encoder module, fusing the outputs of two encoders with different parameters via short-range skip connections can more effectively represent image features.

[0100] The FRN decoder module, similar to the encoder module, also consists of four blocks (Dec1 / 2_x, where x = 4, ..., 1), corresponding to each block of the encoder module. The difference lies in the use of 2×2 upsampling layers in the decoder module to restore the low-level feature maps generated by the encoder module to the same resolution as the input image. Therefore, the other layers in the decoder module have the same parameter settings as the encoder module, including the remaining network blocks with long-range and short-range skip connections for multi-level feature integration. For example, features from different blocks Enc1_d, Dec1_d+1, and Dec2_d+1 are fused through long-range and short-range skip connections before the feature information stream is passed to the decoder module's block named Dec1_d (where d = 1, 2, 3). In this way, the invention ultimately achieves multi-level feature integration, and more accurate segmentation performance can be obtained through these features. Therefore, by upsampling and fusing the feature maps extracted by the encoder module layer by layer, the decoder module ensures that the output feature map has the same size and semantic information as the input image, thereby achieving accurate image segmentation.

[0101] After the image is represented by the Feature Representation Network (FRN), it is fed into the MS-CNN module of the Exponential Estimation Network (IEN) to obtain multi-level local features. Figure 12 The detailed architecture of the MS-CNN module is shown, which consists of three parallel CNN models.

[0102] The first CNN model contains four convolutional layers with 3×3 kernels. Each convolutional layer is followed by a 2×2 max-pooling layer with a stride of 2. It is then connected to two fully connected layers, with a dropout layer added after the first fully connected layer. The second CNN model contains three convolutional layers with the same kernel size as the first. Similarly, each convolutional layer is followed by a corresponding max-pooling layer, a dropout layer, and two fully connected layers. Unlike the first model, the max-pooling layer in the second CNN model is set to 5×5 with a stride of 2. The third CNN model has the same structure as the second model, except that the kernel size is changed from 3×3 to 5×5.

[0103] In the MS-CNN module, this invention employs a multi-scale strategy for multi-level feature extraction. Subsequently, this invention fuses the outputs of three independent CNN models and uses this fusion as the final output of MS-CNN to obtain a representation of local image features.

[0104] To represent the dynamic deformation of images, this invention uses a bidirectional LSTM module. This bidirectional LSTM module is located at the bottom of the Exponential Estimation Network (IEN). During network training, since the input image is 20 frames (one cardiac cycle), this invention sets the stride of the bidirectional LSTM module to 20. This invention uses the image features extracted by the MS-CNN module of the IEN as the input to the bidirectional LSTM module, thereby capturing the dynamic deformation of the heart structure.

[0105] This invention uses an MS-CNN module and a bidirectional LSTM module to quantize the segmentation results and capture the dynamic deformation of the image, thus better showcasing the characteristics of the heart structure.

[0106] In the model training process of Module 3, (S5) this invention implemented the framework on the PyTorch platform. Then, this invention performed five-fold cross-validation on the training set (as follows). Figure 6 (As shown). The training set was divided into five equal parts. One part was selected as the validation set, and the remaining parts were used as the training set to train the model. During testing, leave-one-out cross-validation was used to ensure that every participant participated in the cross-validation, thereby improving model performance and generalization ability.

[0107] For model training, this invention implements its framework on the PyTorch platform. For segmentation tasks, this invention uses binary cross-entropy as the loss function; for quantization tasks, it uses mean squared error (MSE) as the loss function. This invention selects the Adam optimizer with a learning rate of 0.001. During training, this invention sets the batch_size parameter to 8 and the epochs parameter to 160.

[0108] The framework of this invention is implemented using a P100 GPU and the Linux operating system as the underlying platform. To evaluate the performance of the framework, this invention employs leave-one-out cross-validation. During training, this invention selects one object (containing 20 images) as the test set each time, and uses the remaining objects (a total of 2880 images) as the training set. Next, this invention performs five-fold cross-validation on the training set, evenly dividing the training set into five parts. One part is selected as the validation set, while the remaining part is used to train the model. This process is repeated 145 times.

[0109] In Module Four, Model Testing and Evaluation, (S6) before conducting the model testing phase, this invention first requires post-processing of the network's prediction results. This includes converting the prediction results from a normalized form back to actual physical values. Subsequently, this invention can evaluate the performance of the network framework by comparing the differences between the actual physical values ​​of the prediction results and the physical values ​​of real clinical indicators. This invention then uses DM (Dice Metric) to quantify the overlapping area between the predicted contour and the manually drawn contour, i.e., the intersection of the predicted contour and the standard contour. Simultaneously, this invention uses MAE (Mean Absolute Error) to evaluate the quantization results of the cardiac images and measure the performance of the invention's framework.

[0110] First, let's look at the testing. In this section, the invention uses qualitative analysis to test and evaluate the segmentation results of the Feature Representation Network (FRN), quantitative analysis to test and evaluate the quantification results of the Index Estimation Network (IEN), and a performance comparison between this method and state-of-the-art methods. Before conducting the model testing phase, the invention first needs to post-process the network's prediction results. This includes converting the prediction results from a normalized form back to actual physical values. Subsequently, the invention can evaluate the performance of the network framework by comparing the differences between the actual physical values ​​of the prediction results and the physical values ​​of real clinical indicators. Such post-processing ensures the accuracy of the evaluation, guarantees the comparability of the prediction results with real data, and provides a basis for further analysis and improvement. The specific denormalization formula is shown below:

[0111] For left ventricular chamber diameters (dim1, dim2, dim3) and myocardial wall thicknesses (RWTs: A, AS, IS, I, IL, AL):

[0112] physical measurements=resultgt×(rateresize×pixspacing)

[0113] Regarding the area of ​​the myocardium and cardiac chambers (A-myo and A-cav):

[0114] physical measurements=resultgt×(rateresize×pixspacing) 2

[0115] Where, resultgt represents the normalized clinical indicator value, rateresize represents the rate of reversal of the resizing process, and pixelspacing represents the pixel spacing of the MR image for each subject.

[0116] To evaluate the model's performance metrics, this invention performed the following analyses (S7): qualitative analysis of the segmentation results of the Feature Representation Network (FRN); quantitative analysis of the quantization results of the Index Estimation Network (IEN); and performance comparison with state-of-the-art methods. In summary, this invention's deep learning-based heart segmentation model evaluates the performance of segmentation and quantization tasks using DM and MAE metrics, and assesses its advancement by comparing its performance with state-of-the-art methods.

[0117] For cardiac image segmentation, this invention uses the Dice coefficient to measure the overlap area between the predicted contour and the hand-drawn contour (i.e., ground reality). The overlap area refers to the intersection of the predicted contour lines and the ground reality contour. The Dice coefficient (DM) can be easily calculated using the following formula:

[0118]

[0119] Where C gt and C p C represents the regions of the manually drawn contour and the predicted contour, respectively. inter This represents their intersection. Using this formula, the present invention can assess the similarity between the predicted results and the actual situation, thereby quantitatively evaluating the segmentation performance. The value of DM ranges from 0 to 1 (1 represents the best result). Therefore, the higher the DM value, the better the segmentation performance of the framework of the present invention.

[0120] For the quantitative evaluation of cardiac images, this invention uses Mean Absolute Error (MAE) to assess the framework performance. Before calculating the MAE, this invention denormalizes the physical measurements of relevant clinical indicators to recover their true values. Then, this invention calculates the physical measurement error between the predicted results and the actual situation, obtains the error values ​​of all samples, and takes the average of these errors as the MAE. This metric helps this invention measure the average deviation between the predicted results and the actual situation, thereby evaluating the quantitative performance of the framework.

[0121] The formula for calculating MAE is as follows:

[0122]

[0123] Where y pt Represents the actual value, y p This represents the predicted value.

[0124] First, a qualitative analysis is performed to evaluate the segmentation results of the Feature Representation Network (FRN). The specific experimental procedure is as follows: In this invention's experiment, 145 patients were tested, and the Dice coefficient distribution of each subject was plotted. Based on the results, the segmentation results were observed and compared with actual conditions. If the Dice coefficient of most subjects fluctuates around 0.9 (the Dice coefficient ranges from 0 to 1, with 1 representing the optimal result), then the segmentation result of this invention is considered ideal.

[0125] To more intuitively demonstrate the segmentation effect of the framework of this invention, in the later stage of the experiment, this invention selected 10 consecutive images from 7 patients (a total of 20 images per cardiac cycle). These images were displayed at intervals of every other image, with each row representing the image segmentation result of one object. This invention tested and evaluated the segmentation results of the Feature Representation Network (FRN) based on the plotted images.

[0126] Secondly, quantitative analysis is performed. This invention employs a quantitative analysis method to test and evaluate the quantification results of the Exponential Estimation Network (IEN). In this process, this invention presents real-world data from 2,900 images and plots the values ​​directly estimated by the method of this invention on the same image. By observing the red markers mainly distributed around the y=x line, this invention can clearly determine whether the exponential estimation method of this invention is effective.

[0127] Finally, a comparative analysis was conducted, comparing the performance of this invention with state-of-the-art methods. This invention compared its method with existing advanced fully automated (e.g., FCN, U-net, Seg-Net, Conv-Deconv, Indices-JSQ) and semi-automated (e.g., MSVR, dual DBN) methods for left ventricular myocardial segmentation. It is important to note that all segmentation methods compared in this invention focused solely on the segmentation task, while this invention's method handles both segmentation and quantization tasks simultaneously within a multi-task framework. Based on the experimental results and parameters, this invention can determine whether its combined segmentation and quantization method demonstrates superior performance in left ventricular segmentation compared to methods performing segmentation alone.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for left ventricular quantification based on a deep multi-task regression network, comprising the following steps: Data processing is performed on several preset cardiac MR images; A model was built based on several pre-set cardiac MR images after data processing. The model was used to quantize the left ventricle in the cardiac MR images to be processed. The model is characterized by comprising an encoder-decoder module, an MS-CNN module, and a bidirectional LSTM module. The encoder-decoder module is used to extract cardiac structural features related to segmentation and quantization tasks. The MS-CNN module utilizes multiple independent CNN models with different kernel sizes and pooling sizes to extract more effective low-level features from the high-level feature maps of the encoder-decoder module for cardiac image processing. The MS-CNN includes three parallel CNN models that perform multi-scale feature extraction using a multi-scale strategy, and the outputs of the three CNN models are fused as the final output of the MS-CNN. The bidirectional LSTM module is used to capture dynamic changes in cardiac structures. After deploying the MS-CNN module and the bidirectional LSTM module, an exponent is estimated based on a multi-task regression model. The encoder-decoder module consists of two sub-encoders connected by short-range jumps. Each sub-encoder contains four blocks, each with a similar structure, including convolutional layers with a 1×1 kernel and ReLU activation, convolutional layers with a 3×3 kernel and ReLU activation, max-pooling layers, and dropout layers. These convolutional layers progressively extract feature information from the input image from high to low layers. In the second convolutional layer of each block, dilated convolutions are used to increase the receptive field. The decoder module corresponds to the encoder module and also consists of four blocks. Upsampling layers are used to restore the low-level feature maps to the resolution of the input image. The other layers in the decoder module have the same parameter settings as the encoder module, and feature integration is performed on the remaining network blocks connected by long-range and short-range jumps. Through layer-by-layer upsampling and feature fusion, the decoder module produces an output feature map with the same size and semantic information as the input image, achieving accurate image segmentation.

2. The left ventricular quantification method based on deep multi-task regression network according to claim 1, characterized in that, The data processing includes data preprocessing and data labeling.

3. The left ventricular quantification method based on deep multi-task regression network according to claim 2, characterized in that, The data preprocessing includes: feature point marking, image rotation, ROI cropping, and resizing. First, the heart region is located and key points are marked on each preset cardiac MR image. Then, the image is rotated to a predetermined angle based on the key points. Next, a suitable ROI region is selected and cropped on the rotated image. Finally, all cropped images are resized to a predetermined pixel size. The suitable ROI region refers to an ROI region that can completely contain the entire outline of the heart.

4. The left ventricular quantification method based on deep multi-task regression network according to claim 3, characterized in that, The data annotation includes: normalizing the actual clinical physical values ​​of the left ventricular cavity diameter and myocardial ventricular wall thickness according to the size of the predetermined pixels, and simultaneously normalizing the area of ​​the myocardium and the cavity according to the image area of ​​the predetermined pixels.

5. The left ventricular quantification method based on a deep multi-task regression network according to claim 1, characterized in that, By setting a bidirectional LSTM module to capture dynamic deformations of the image with a stride of 20, the image features extracted by MS-CNN are used as its input.

6. The left ventricular quantification method based on deep multi-task regression network according to claim 1, characterized in that, The model is also trained before it is used.

7. The left ventricular quantification method based on a deep multi-task regression network according to claim 6, characterized in that, The model training uses a five-fold cross-training method to quickly train the entire model.

8. The left ventricular quantification method based on a deep multi-task regression network according to claim 6, characterized in that, After training, the model needs to be tested and evaluated.

9. The left ventricular quantification method based on a deep multi-task regression network according to claim 8, characterized in that, The prediction results are denormalized to become actual physical values, thereby obtaining metrics for evaluating the performance of the network framework.

Images