A shape self-learning based cardiac MRI image segmentation method

Abstract

The application discloses a kind of based on shape self-learning cardiac MRI image segmentation method, by establishing the segmentation structure based on three-dimensional deep learning network structure, under the mutual self-learning of two multi-task head network, using end-to-end learning mode carries out full supervision segmentation learning;Input image is extracted features by main network, and simultaneously input two subnetwork structures, respectively for the classification information of pixel point by pixel and its nearest distance with contour are predicted;Simultaneously based on the constraint of focus loss function and the self-learning constraint of directional distance conversion will make the model for the shape and position information of target have this better control.The method of the application can alleviate the inaccurate prediction of the segmentation network for the contour surface and the inaccurate prediction of the small target to a certain extent, so as to better assist intelligent medical analysis and diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of cardiac MRI medical image segmentation technology, specifically relating to a cardiac MRI image segmentation method based on shape self-learning. Background Technology

[0002] Heart disease, as one of the top five most dangerous diseases threatening human health in recent years, makes the study of its anatomical structure of great significance. Medical image segmentation, a task that isolates pixel-based target regions from background information, plays a crucial role in the diagnosis, understanding, and research of this disease. However, in the real world, unlike natural image structures, the heart is a 3D structure, and it exhibits individual specificity, meaning the details of the heart structure vary greatly from patient to patient. Furthermore, the heart's outline also possesses unique characteristics, such as... Figure 1 As shown, the morphology of the heart varies greatly among different patients, and its surface structure is intricate. Therefore, how to better enable the model to learn the contour information of the heart and obtain more accurate segmentation results has become a key research focus. Furthermore, due to the special nature of medical images, the target region for segmentation differs significantly from the background region in terms of voxel count. Typically, the former accounts for only about 0.2% to 0.3% of the latter. This huge difference causes the gradient of the model during training to be overwhelmed by the predominantly background information, leading the model to a suboptimal solution.

[0003] Directed distance graphs, as a common traditional algorithm in the field of robotics, have a natural constraint on the contour by evaluating the distance between each voxel and the nearest contour point. At the same time, the focus loss function continuously adjusts the learning focus of the model by reducing the weight of the gradient of easy-to-learn samples and increasing the weight of difficult samples, thereby achieving the goal of balancing the learning focus.

[0004] In the traditional field of machine learning, certain prior information is usually used to better segment images. This prior information includes two types: weak prior and strong prior. Weak prior mainly includes anatomical structure information, while strong prior is mostly shape information, such as clustering and dynamic contour methods. However, these methods have high requirements for the accuracy of prior information and have difficulty transferring to different structural organizations.

[0005] With the gradual development of deep learning, more and more end-to-end learning methods have emerged. Among them, Fully Convolutional Neural Networks (FCN) and its variant UNet have injected new vitality into model segmentation, classification, and object detection. The literature [Cui H, Yuwen C, Jiang L, et al. Multiscale attention guided U-Net architecture for cardiac segmentation in short-axis MRI images[J]. Computer Methods and Programs in Biomedicine,2021,206:106142] uses a combination of UNet structure and multi-level attention pyramid structure to automatically learn the structure of various sizes and shapes of the target of interest. The paper [Khened M, Kollerathu VA, Krishnamurthi G. Fully convolutional multi-scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers[J]. Medical image analysis,2019,51:21-45] uses a combination of deep fully convolutional neural networks and Inception architecture to train on cardiac images. They also use cross-entropy loss and Descein loss functions to constrain the model's parameter updates. While these methods have achieved good results, these models only address feature extraction for cardiac structures, which exhibit significant individual variability, through network design, loss function design, and attention mechanisms, without considering the direct impact of the organ structure itself on segmentation. Furthermore, since segmentation errors are mainly concentrated on the predicted contour edges, how to better constrain these voxel points located at the edges of the target region has become a key research focus in academia. Summary of the Invention

[0006] In view of the above, the present invention provides a cardiac MRI image segmentation method based on shape self-learning. The method generates pixel-by-pixel prediction results and directed distance map prediction results through a network with a multi-task structure. At the same time, it uses a loss function based on a focusing function and a directed distance map transformation to effectively constrain the segmentation and contour learning.

[0007] A shape-based self-learning method for cardiac MRI image segmentation includes the following steps:

[0008] (1) Collect 3D cardiac MRI images from different individuals with region category labels (including background, left ventricle, right ventricle, myocardium, etc.), scale these images to the same size and normalize them before dividing them into training set and test set;

[0009] (2) Construct a deep learning network model for cardiac MRI image segmentation, which includes a backbone network and a pixel-level segmentation head network f seg And the directed distance graph head network f dis The backbone network is used to extract features from the image and input the extracted feature maps into f. seg and f dis In the prediction, f seg f is used to predict the probability that a voxel in an image belongs to various regions (including various heart regions and background regions). dis Used to predict the nearest distance from voxels to the contours of various cardiac regions in an image;

[0010] (3) Input the training set images one by one into the above deep learning network model for training, and design the corresponding loss function for iterative updating of the network model;

[0011] (4) Input the test set images into the trained network model, and extract the pixel-level segmentation head network f. seg The predicted output is used as the segmentation result of the cardiac MRI image.

[0012] Furthermore, the backbone network adopts a VNet structure as a feature extractor.

[0013] Furthermore, the specific process of training the deep learning network model in step (3) is as follows:

[0014] 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer;

[0015] 3.2 Input the training set images one by one into the network model, and the network model outputs the pixel-by-pixel classification prediction result p through forward propagation. seg And the directed distance prediction result p dis Calculate the prediction result p seg and p dis The loss function L between the label and the label all ;

[0016] 3.3 Based on the loss function L all The optimizer iteratively updates the model parameters using gradient descent until the loss function L is reached. all Convergence complete, training finished.

[0017] Furthermore, the loss function L all The expression is as follows:

[0018] L all =L dis +λL slb +L focal

[0019] Where: L dis For the prediction result p dis The L2 norm constraint term between the label and the SDF graph, where the SDF graph is obtained by directed distance transformation of the labels, L slb For the prediction result p seg The self-learning cross-entropy function between the SLB graph and the prediction result p. dis L is obtained through reverse directed distance transformation. focal For the prediction result p seg The pixel-wise reweighting constraint term between the label and the label, where λ is the scaling factor.

[0020] Furthermore, the L2 norm constraint term L dis The expression is as follows:

[0021]

[0022] Where: p dis (i,j,k,c) represents the predicted nearest distance from the voxel in the i-th row, j-th column, k-th depth direction to the contour of the c-th type of heart region in the image. SDF(i,j,k,c) represents the directed distance from the voxel in the i-th row, j-th column, k-th depth direction to the contour of the c-th type of heart region in the image. W, H, and D represent the width, height, and depth of the image, respectively, and C represents the total number of heart region categories.

[0023] Furthermore, the expression for the directed distance SDF(i,j,k,c) is as follows:

[0024]

[0025] Where: x(i,j,k) represents the voxel in the i-th row, j-th column, and k-th depth direction of the image, ω represents the c-th type of heart region in the image, ω in ω represents the set of voxels within the cardiac region ω. out This represents the set of voxels outside the cardiac region ω. The set of voxels representing the contour ω of the heart region, Γ(i,j,k) represents the orthogonal distance from voxel x(i,j,k) to the contour ω of the heart region. out Represents set ω out The orthogonal distance from any voxel to the ω-contour of the cardiac region, Γ in Represents set ω inThe orthogonal distance from any voxel to the ω-profile of the heart region.

[0026] Furthermore, the expression for the orthogonal distance Γ(i,j,k) is as follows:

[0027]

[0028] in: This represents the shortest distance from voxel x(i,j,k) to the contour ω of the heart region.

[0029] Furthermore, the self-learning cross-entropy function L slb The expression is as follows:

[0030]

[0031] Where: p seg (i,j,k,c) represents the predicted probability that the voxel in the i-th row, j-th column, and k-th depth direction belongs to the c-th type of heart region, and SLB(i,j,k,c) represents p dis (i,j,k,c) are the label values ​​obtained through inverse directed distance transformation, where W, H, and D are the width, height, and depth of the image, respectively, and C is the total number of categories in the heart region.

[0032] Furthermore, the expression for the label value SLB(i,j,k,c) is as follows:

[0033]

[0034] Where: p dis (i,j,k,c) represents the predicted closest distance from the voxel in the i-th row, j-th column, and k-th depth direction in the image to the contour of the c-th type of heart region.

[0035] Furthermore, the pixel-by-pixel reweighting constraint term L focal The expression is as follows:

[0036]

[0037]

[0038] Where: p seg (i,j,k,b) represents the predicted probability that the voxel in the i-th row, j-th column, and k-th depth direction belongs to the b-th region, and gt(i,j,k,b) represents the label value of the voxel in the i-th row, j-th column, and k-th depth direction belonging to the b-th region. f(i,j,k,b) represents the distance constraint coefficient of the voxel in the i-th row, j-th column, and k-th depth direction in the image. W, H, and D are the width, height, and depth of the image, respectively. B is the total number of image regions (including various heart regions and background regions). γ is a hyperparameter, and α is the constraint coefficient.

[0039] This invention establishes a segmentation structure based on a 3D deep learning network. Through mutual self-learning between two multi-task head networks, it employs an end-to-end learning approach for fully supervised segmentation learning. The input image extracts features through the backbone network and is simultaneously input into two sub-network structures, which predict the classification information of each pixel and its nearest distance to the contour. Furthermore, constraints based on the focusing loss function and the self-learning constraint of directed distance transformation enable the model to better control the shape and position information of the target. This invention can alleviate, to some extent, the inaccuracy of segmentation networks in predicting contour surfaces and small targets, thereby better assisting intelligent medical analysis and diagnosis. Attached Figure Description

[0040] Figure 1 3D schematic diagrams of the heart structure for different patients.

[0041] Figure 2 This is a schematic diagram of the deep learning network model of the present invention.

[0042] Figure 3 This is a schematic diagram of the steps of the image segmentation method of the present invention.

[0043] Figure 4 The visualization results of cardiac segmentation for patients with relatively uniform cardiac occupancy are shown in the following images: (a) is the visualization result of the training set images (axial slice), (b) is the visualization result of the pixel-by-pixel annotation of the training set (axial slice), (c) is the segmentation result generated by the model (axial slice), and (d) is the 3D view of the segmentation result generated by the model.

[0044] Figure 5 The visualization results of cardiac segmentation for patients with relatively small overall cardiac lesions are shown in the following figures: (a) is the visualization result of the training set images (axial slices), (b) is the visualization result of the pixel-by-pixel annotation of the training set (axial slices), (c) is the segmentation result generated by the model (axial slices), and (d) is the 3D view of the segmentation result generated by the model. Detailed Implementation

[0045] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 3 As shown, the present invention provides a cardiac MRI image segmentation method based on shape self-learning, comprising the following steps:

[0047] (1) Scale the 3D cardiac MRI images to the same size and normalize these images;

[0048] (2) Input the image into a 3D deep learning backbone network structure to extract features;

[0049] (3) The feature maps after the backbone network are fed into the pixel-level segmentation head network and the directed distance map head network respectively to obtain the output predicted value p. seg With p dis ;

[0050] (4) Convert the manually labeled gt into a directed distance graph, the result of which is SDF;

[0051] (5) The output directed distance prediction value p dis Apply L2 norm constraints to the SDF graph generated in step (4). dis ;

[0052] (6) The generated distance prediction map p dis Perform a reverse directed distance transformation to generate an SLB, and then... seg Instead of self-learning the cross-entropy function L slb ;

[0053] (7) For the network to output the prediction result p pixel by pixel seg Pixel-wise reweighting constraint L with the real label focal ;

[0054] (8) L dis L slb L focal The total loss L after summing the three loss functions all Perform gradient backpropagation and iteratively update network parameters.

[0055] The deep learning network model of this invention is as follows: Figure 2 As shown, for an input image x, it extracts feature information from a segmentation structure with VNet as the backbone network, and obtains pixel-by-pixel classification prediction p through two task branches. seg And directed distance graph prediction p dis Among them, pixel-by-pixel classification prediction p seg The L value will be calculated pixel-by-pixel reweighted based on the focus loss function and the label. focal Simultaneously, the directed distance graph predicts p. dis The labels transformed through the directed distance graph will be subjected to a L2 norm L. dis Constraints, loss function L focal L2 norm loss function dis The following are respectively:

[0056]

[0057]

[0058]

[0059] Where: P seg (i,j,k,b) represents the predicted probability that voxel x(i,j,k) in the i-th row, j-th column, and k-th depth direction belongs to the b-th class region. W, H, and D represent the width, height, and depth of the image, respectively. gt(i,j,k,b) is the label value of voxel x(i,j,k,b) in the i-th row, j-th column, and k-th depth direction belonging to the b-th class region. α is the constraint coefficient, and w f (i,j,k,b) are the distance constraint coefficients of voxel x(i,j,k), P dis (i,j,k,c) represents the predicted distance from voxel x(i,j,k) in the i-th row, j-th column, k-th depth direction to the contour of the c-th type of heart region, and SDF(i,j,k,c) represents the directed distance from voxel x(i,j,k) in the i-th row, j-th column, k-th depth direction to the contour of the c-th type of heart region.

[0060] The specific definition of the directed distance graph above is as follows: Let ω represent the c-th class region in the input image x. The directed distance function Γ is the orthogonal distance from a given point x(i,j,k) in the metric space to the boundary of region ω, and its sign is determined by whether x(i,j,k) is inside region ω. This function is defined to have a positive value Γ at points x(i,j,k) outside region ω, and this value decreases as x(i,j,k) approaches the boundary of region ω; and to have a negative value Γ at points x(i,j,k) inside region ω, and this value increases as x(i,j,k) approaches the boundary of region ω.

[0061]

[0062] Where: ω in The set of voxels representing the region ω. The set of voxels representing the contour of region ω, ω out The set of voxels outside region ω is represented by d, which is a distance function. Therefore, the entire image can be formed as a directed graph by describing the distance relationship between each point and the contour.

[0063] Therefore, the Directed Distance Graph Transformation Expression (SDF) is as follows:

[0064]

[0065] Wherein: Γ out Γ is the distance from any voxel point located within region ω to the contour. inLet SDF(i,j,k,c) be the distance from any voxel point located outside region ω to the contour. SDF(i,j,k,c) is the directed distance between voxel x(i,j,k) in the i-th row, j-th column, and k-th depth direction and the contour of region c. This SDF represents a directed distance map normalized to -1 to 1. Its absolute value represents the distance between each voxel point and the nearest contour, and its sign represents its location.

[0066] This transformation process is also reversible, for the directed distance prediction graph p of the model. dis (i,j,k,c), as defined, the parts predicted as negative or zero represent the interior of the set, while the rest represent the background. Therefore, by setting each class of values ​​less than zero in the directed distance prediction result to 1 and the remaining positions to 0, p can be... dis (i,j,k,c) is converted into self-learning labels, and the resulting SLB expression is:

[0067]

[0068] The above self-learning constraint L slb for:

[0069]

[0070] Where: P seg (i,j,k,c) is the probability prediction value of voxel x(i,j,k) in the i-th row, j-th column, and k-th depth direction in the image belonging to the c-th type of heart region, and SLB(i,j,k,c) is the label value of voxel x(i,j,k,c) in the i-th row, j-th column, and k-th depth direction in the image after inverse directed distance transformation belonging to the c-th type of heart region.

[0071] The overall loss function expression is:

[0072] L all =λ*L slb +L dis +L focal

[0073] Where: λ is the proportionality coefficient.

[0074] Pixel-by-pixel segmentation head network f seg And the directed distance prediction head network f dis These are two similar lightweight parallel structures connected after the backbone network. In this embodiment, the pixel-by-pixel segmentation head network f seg The directed distance prediction head network f consists of a 3D convolutional layer with a kernel size of 3×3×3 followed by a 3D convolutional layer with a kernel size of 1×1×1. disIt consists of a 3D convolutional layer with a kernel size of 3×3×3, a 3D convolutional layer with a kernel size of 1×1×1, and a tanh activation layer connected together.

[0075] For the segmentation network, the objective function is solved iteratively using the stochastic gradient descent algorithm, resulting in the following iterative steps:

[0076]

[0077]

[0078] Where: lr is the learning rate, B is the total number of categories, and w is the network parameters. For the gradient operator, the stochastic gradient descent algorithm is used to accelerate the convergence speed.

[0079] We used left atrial MRI images in our experiments to verify the effectiveness of this invention. All images were processed and normalized before the experiments were conducted. Our experiments used an NVIDIA 1080Ti GPU for accelerated computation, with a batch size of 4. The images were resized to 112×112×80 pixels, the initial learning rate of the segmentation network was 0.01, and the entire training process required 60 iterations.

[0080] Figure 4 and Figure 5 These are the heart segmentation visualization results for two patients with significantly different heart shapes. Figure 4 The visualization results show that the overall occupancy of the heart is relatively even. (a) is the original image, (b) is the axial direction annotation result, (c) is the axial direction prediction result, and (d) is the 3D visualization result. It can be seen from (b) and (c) that the segmentation result of the method of the present invention is very similar to the annotation shape, and the contour line can be seen from the 3D view (d) to be smooth and uniform. Figure 5 The visualization results for a small overall cardiac lesion are shown in (a) the original image, (b) the axial direction annotation result, (c) the axial direction prediction result, and (d) the 3D visualization result. As can be seen from the results, the algorithm of this invention still has good accuracy in the case of relatively small targets, and it is very close to the annotation result.

[0081] The above results show that the method of the present invention improves the overall accuracy of the predicted surface contour by taking into account the relationship between each voxel point and the nearest contour, and by combining traditional algorithms with deep learning algorithms. This provides strong support for the diagnosis and research of diseases in smart healthcare.

[0082] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. It will be apparent to those skilled in the art that various modifications can be made to the above embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A method for segmenting cardiac MRI images based on shape self-learning, comprising the following steps: (1) Collect 3D cardiac MRI images from different individuals with region category labels, scale these images to the same size and normalize them before dividing them into training and test sets; (2) Construct a deep learning network model for cardiac MRI image segmentation, which includes a backbone network and a pixel-level segmentation head network. and the directed distance graph head network The backbone network is used to extract features from the image and input the extracted feature maps into the respective networks. and In making predictions, Used to predict the probability that voxels in an image belong to different regions. Used to predict the nearest distance from voxels to the contours of various cardiac regions in an image; (3) Input the training set images one by one into the above deep learning network model for training, and design the following loss function for iterative updating of the network model; in: For loss function, For the prediction results The L2 norm constraint term between the label and the SDF graph, where the SDF graph is obtained by directed distance transformation of the labels. For the prediction results The self-learning cross-entropy function between the prediction result and the SLB graph, where the SLB graph is the prediction result. Obtained through reverse directed distance transformation For the prediction results The pixel-by-pixel reweighting constraint between the label and the label λ This is the proportionality coefficient. γ For hyperparameters, α These are constraint coefficients; For the first in the image i Line number j Liede k voxels in the depth direction to the first c The nearest distance prediction value of the heart-like region contour. For the first in the image i Line number j Liede k voxels in the depth direction to the first c The directed distance of the heart-like region outline. W , H, D These are the width, height, and depth of the image, respectively. C The total number of categories for the heart region. B The total number of categories in the image region; Indicates the first in the image i Line number j Liede k Voxels in the depth direction belong to the first c Probability prediction values ​​for heart-like regions express Label values ​​obtained through reverse directed distance transformation; Indicates the first in the image i Line number j Liede k Voxels in the depth direction belong to the first b The predicted probability value of the class region, Indicates the first in the image i Line number j Liede k Voxels in the depth direction belong to the first b The label value of the class region, Indicates the first in the image i Line number j Liede k Distance constraint coefficient of voxels in the depth direction; (4) Input the test set images into the trained network model, and take the pixel-level segmentation head network. The predicted output is used as the segmentation result of the cardiac MRI image.

2. The cardiac MRI image segmentation method according to claim 1, characterized in that: The backbone network uses a VNet structure as a feature extractor.

3. The cardiac MRI image segmentation method according to claim 1, characterized in that: The specific process of training the deep learning network model in step (3) is as follows: 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 3.2 Input the training set images one by one into the network model, and the network model outputs the pixel-by-pixel classification prediction results through forward propagation. and directed distance prediction results Calculate the prediction results and Loss function between labels ; 3.3 Based on the loss function The optimizer iteratively updates the model parameters using gradient descent until the loss function is reached. Convergence complete, training finished.

4. The cardiac MRI image segmentation method according to claim 1, characterized in that: The directed distance The expression is as follows: in: Indicates the first in the image i Line number j Liede k Voxels in the depth direction, Indicates the first in the image c Heart-like region Indicates the heart region Internal voxel set, Indicates the heart region External voxel collection, Indicates the heart region The set of voxels of the outline. Voxel representation To the heart area Orthogonal distance of the contours Represents a set Any voxel to the heart region Orthogonal distance of the contours Represents a set Any voxel to the heart region Orthogonal distance of the contour.

5. The cardiac MRI image segmentation method according to claim 4, characterized in that: The orthogonal distance The expression is as follows: in: Voxel representation To the heart area The closest distance to the outline.

6. The cardiac MRI image segmentation method according to claim 1, characterized in that: The tag value The expression is as follows: in: For the first in the image i Line number j Liede k voxels in the depth direction to the first c The nearest distance prediction of the heart-like region contour.

Images