Multi-task consistency segmentation network method for cartilage segmentation based on deep shape prior model

Abstract

The invention provides a multi-task consistency segmentation network method for cartilage segmentation based on a deep shape prior model, belongs to the field of cartilage joint segmentation network training in multi-modal MRI, and solves the hard segmentation problem under the conditions of complex image background, small training data volume and the like. According to the method, the concept of shape feature description is defined and comprises the inner side surface, the mask segmentation and the contour shape, the cartilage segmentation network based on inner side surface modeling, the mask segmentation and the contour shape is established, the cooperative improvement problem among multiple tasks is solved, and the consistency of the multiple tasks is achieved; the concept of shape prior description represents the shape distribution of the articular cartilage; a neural network model based on shape prior description is established; a shape prior network model is introduced into a multi-task consistency segmentation network. Through the guidance of shape feature description and the constraint of shape prior description, a very different deep learning network framework is established, and the method has very high accuracy and strong generalization ability.

Description

Technical Field

[0001] The present invention relates to a multi-task consistent segmentation network, belonging to the field of training cartilage joint segmentation networks in multi-modal MRI, and specifically designs a multi-task consistent segmentation network method for cartilage segmentation based on a deep shape prior model. Background Art

[0002] Due to their powerful learning ability and end-to-end learning process, deep neural networks have significantly improved the segmentation efficiency and accuracy, playing a leading role in medical image segmentation tasks. The current state-of-the-art cartilage segmentation methods mainly focus on two-dimensional UNet methods and three-dimensional U-Net methods, or training two-dimensional and three-dimensional U-Nets in a multi-stage framework.

[0003] However, the above methods may hinder the correct segmentation of cartilage from other adjacent structures, severely diseased parts, and regions with low contrast and blurred boundaries between contacting cartilages. Therefore, over-segmentation or under-segmentation often occurs, and incomplete segmentation of the joint cartilage of interest is also produced. Specifically, the main challenges in segmenting joint cartilage mainly include: large deformations of cartilage shape caused by severe osteoarthritis (OA); similar or low-contrast textures of adjacent tissues, severely diseased parts, and contacting cartilages; small amount of training data; and poor generalization ability for test data due to the above limitations.

[0004] Therefore, there is an urgent need for a method that can accurately segment cartilage. Summary of the Invention

[0005] Aiming at the defects of over-segmentation or under-segmentation in the prior art, the present invention proposes a multi-task consistent segmentation network method for cartilage segmentation based on a deep shape prior model, which has a robust segmentation method, significantly improved accuracy, and strong generalization ability.

[0006] The present invention is implemented by the following technical solutions: A multi-task consistent segmentation network method for cartilage segmentation based on a deep shape prior model, comprising the following steps:

[0007] Step A: Establish a multi-task consistent segmentation network based on shape feature description; divide the segmentation task into three parts, which are respectively responsible for the medial surface extraction task, the cartilage segmentation task, and the contour detection task;

[0008] Step B: Establish a deep shape prior network model to achieve the reconstruction of a given image; the shape prior description represents the shape distribution of joint cartilage;

[0009] Step C: Integrate the two network models established in Steps A and B to establish a multi-task consistency segmentation network constrained by a shape prior model, guided by the establishment of a cartilage segmentation network model and constrained by deep prior shape modeling, to improve the robustness and accuracy of segmentation;

[0010] Step D: Based on recursive learning for testing, adopt a recursive strategy for iterative optimization in the testing phase. This deep neural network imposes prior constraints by minimizing the network training loss function on the output mask of the segmentation network; during the optimization process, through recursive learning, continuously optimize the cartilage shape prior and the cartilage segmentation prediction mask during the iteration process, making the shape of the cartilage segmentation prediction mask closer to the actual shape, so as to complete cartilage segmentation with high precision.

[0011] Preferably, in Step A, the cartilage can be locally described as a thin disk, corresponding to finding the position using the eigenvalues of these Hessian matrices, and obtaining the ground truth of the inner surface and the boundary of the cartilage from the manually labeled cartilage region using the gradient vector and the Hessian matrix; it includes the following steps:

[0012] (1). For the cartilage contour detection task, let I(x) be the three-dimensional image of voxel x, be the Hessian matrix of the image blurred by the isotropic Gaussian function, with a standard deviation of σ; similarly, let be λ1, λ2, λ3 respectively, and their corresponding eigenvectors be e1, e2, e3 (|e1| = |e2| = |e3|); is used for cartilage enhancement to obtain the initial cartilage region; the above values are used as the initial values for accurate boundary localization; on this basis, let the eigenvalue e1 represent the cartilage surface normal, and further accurately locate the cartilage boundary using the first directional derivative along the normal direction.

[0013] (2). For the inner surface extraction task, the cartilage region should include all potential inner surfaces; the eigenvalue e3 is expected to give the orthogonal direction of the articular cartilage, and only |λ3| is larger on its inner surface; in addition, use local second-order approximation to extract the inner surface, which is composed of and the gradient vector; obtain the orthogonal direction of the articular cartilage using the eigenvector e3 corresponding to the Hessian matrix blurred by the isotropic Gaussian function, then use local second-order approximation to extract the inner surface, and quantitatively evaluate the accuracy of the extracted inner surface on 30 test data sets using the Dice similarity coefficient.

[0014] (3). Reconstruction of the inner surface

[0015] The backpropagation of the neural network requires that the embedding process of the network be differentiable. A smooth estimation algorithm for reconstruction operations is proposed to achieve cartilage shape reconstruction based on the medial surface, making the neural network differentiable during the training process. Behind the backbone network, two branches are designed: the distance transformation prediction branch and the medial surface prediction branch. Among them, the output of the distance transformation prediction branch is denoted as dis v , which is negative when the voxel v is within the cartilage region and positive when it is in the background. The probability feature map of the medial surface prediction branch is denoted as MS v ;

[0016] To make the voxel values belonging to articular cartilage in the reconstructed shape probability map as close to 1 as possible and the voxel values in the background as close to 0 as possible, an approximate reconstruction algorithm based on the sigmoid function is proposed, and the value r of the reconstructed shape probability map at voxel V is defined v . By applying the differentiable chain rule, the derivatives of the function r v with respect to dis v and |MS v | can be obtained respectively;

[0017] Define the supervised loss function L between recon and the ground truth mask Z;

[0018] Applying this method to all points in the 3D image, the final probability map of the reconstructed shape can be obtained.

[0019] Preferably, in step A, a multi-task consistent segmentation network based on shape feature description uses visualtransformer as the backbone, which utilizes the multi-head attention mechanism to extract different features of the input image, meeting the requirements of multi-task consistency. Among them, the segmentation prediction branch, the medial surface prediction branch, and the contour prediction branch are supervised by the segmentation mask, the medial surface mask, and the contour mask respectively;

[0020] Among them, the supervised loss functions L seg , L con , and L med are defined respectively;

[0021] Next, define the consistency loss recon ∈ Y recon between the shape reconstruction shape mapping y seg ∈ Y seg based on the differentiable medial surface and the predicted segmentation mapping y

[0022] Then, use the first directional derivative to obtain the contour of the shape reconstruction shape y recon ∈ Y recon based on the medial surface and the predicted segmentation map y seg ∈ Yseg The contour can define a consistency loss composed of the two and the predicted contour y con ∈ Y con and the contour of the predicted segmentation map to form a consistency loss

[0023] Preferably, in step B, a shape prior description is defined, which represents the shape distribution of multiple objects; and a deep shape prior network model composed of Autoencoder1 and Autoencoder2 is established; wherein, Autoencoder1 is composed of a decoder and an encoder to output a reconstructed image; in order to optimize the parameters, Autoencoder1 is trained to minimize the reconstruction error L r ; in Autoencoder2, the encoded output of Z is represented as f e (Z; θ) = I e , where θ is the learnable parameter of the encoder layer f e (·, θ), and the optimal parameter θ is obtained after training Autoencoder2 * and remains unchanged in subsequent training; in addition, a shared layer of Autoencoder1 and Autoencoder2 is designed, and the KL divergence is used to define the shared layer loss L s and minimize it to make the predicted cartilage shape distribution converge to the true cartilage shape distribution. For this stage, the total loss L is known;

[0024] Preferably, in step C, the training of the multi-task consistency segmentation network based on the shape prior model constraint is divided into two steps: pre-training and fine-tuning; during pre-training, the network is trained on the 3D MRI volume I and the ground truth Z; the segmentation network is incorporated into the deep shape prior network described in step B, and the segmentation network model for downstream tasks is trained by simply fine-tuning all pre-trained parameters. After re-training the segmentation network, a shape prior constraint is given to the re-trained segmentation network when training the network; for this purpose, the segmentation network is placed after the autoencoder learning of the shape prior network model, and the segmentation network only needs to be pre-trained once and then fine-tuned for downstream tasks.

[0025] Preferably, two publicly available knee image datasets, OAI Imorphics Knee and OAI ZIBKnee, and a non-public clinical hip image dataset, Hip dataset, are used; four metrics are used to evaluate the segmentation performance: Dice similarity coefficient (DSC), average surface distance (ASD), volume difference (VD), and volume overlap error (VOE).

[0026] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0027] The multi-task consistent segmentation network method based on the deep shape prior model for cartilage segmentation of the present invention defines: (1) the concept of shape feature description, including the medial surface, mask segmentation, and contour shape, which describe the geometric characteristics of the articular cartilage shape, and establishes a cartilage segmentation network based on medial surface modeling, mask segmentation, and contour shape, solving the problem of collaborative improvement between multi-tasks and achieving multi-task consistency; (2) the concept of shape prior description, representing the shape distribution of articular cartilage; and establishes a neural network model based on shape prior description. Further, the shape prior network model is introduced into the multi-task consistent segmentation network. Through the guidance of shape feature description and the constraint of shape prior description, a very different deep learning network framework is established, with high accuracy and strong generalization ability.

[0028] The main advantage of the network method for cartilage segmentation of the present invention is that it can handle low contrast and blurred boundaries between cartilage and other adjacent structures, highly deformed parts, and difficult cases of contacting cartilage.

[0029] The method was tested on two publicly available knee image datasets and one clinical hip image dataset, and its results were compared with the recently published methods. The obtained results show that the method of the present invention significantly improves the accuracy of overall cartilage segmentation, especially for knees or hips with severe arthritis. Brief Description of the Drawings

[0030] Figure 1 is the network framework of the present invention;

[0031] Figure 2 is the shape prior modeling scheme based on shape prior description of the present invention;

[0032] Figure 3 is the multi-task consistent network framework diagram based on deep shape prior of the present invention;

[0033] Figure 4 is the schematic diagram of the first iteration of the shape prior and the predicted segmentation mask of the present invention;

[0034] Figure 5 is the schematic diagram of the iterative update process of the shape prior and the predicted segmentation mask of the present invention;

[0035] Figure 6 is the flow chart of the model of the present invention;

[0036] Figure 7 is the comparison diagram of the segmentation results between the method of the present invention and 3D U-Net. Detailed implementation manners

[0037] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described below with reference to the accompanying drawings and embodiments. Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention may be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific implementations disclosed below.

[0038] The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation of the present invention first defines two concepts:

[0039] (1) Shape feature description, such as the medial surface, mask segmentation and contour shape, which describe the geometric characteristics of the joint cartilage shape; and a cartilage segmentation network based on medial surface modeling, mask segmentation and contour shape is established, which solves the problem of collaborative improvement between multiple tasks and realizes multi-task consistency.

[0040] (2) Shape prior description, which represents the shape distribution of joint cartilage; and is guided by the establishment of a cartilage segmentation network model and constrained by depth prior shape modeling. In addition, a prior shape constraint network model is introduced into the multi-task consistency segmentation network. The combination of this multi-task consistency segmentation network and the prior shape guidance network model improves the segmentation ability of the deep learning network under the above complex conditions.

[0041] As Figure 1 shown, the multi-task consistency segmentation network framework method based on a depth shape prior model of the present invention mainly includes the following steps:

[0042] Step A: Establish a multi-task consistency segmentation network based on shape feature description; define the cartilage shape feature description as the medial surface, segmentation and contour of joint cartilage, which characterize the geometric characteristics of joint cartilage; divide the segmentation task into three parts, which are respectively responsible for the medial surface extraction task, the cartilage segmentation task and the contour detection task;

[0043] Step B: Guided by the cartilage segmentation network model, establish a depth shape prior network model to realize the reconstruction of a given image;

[0044] Step C: Integrate the two network models established in Step A and Step B, that is, after introducing the shape prior network model into the multi-task consistency segmentation network based on shape feature description, establish a multi-task consistency segmentation network constrained by a shape prior model, which is guided by the establishment of a cartilage segmentation network model and constrained by depth prior shape modeling to improve the robustness and accuracy of segmentation;

[0045] Step D: Test based on recursive learning.

[0046] The multi - task consistency segmentation network method based on the deep shape prior model proposed in the embodiments of the present invention for deep learning can effectively improve the segmentation performance of the network.

[0047] Specifically, the method of the present invention will be described in detail below:

[0048] Step A: The cartilage shape feature description is defined as the medial surface, segmentation, and contour of articular cartilage, which characterize the geometric properties of articular cartilage. The proposed multi - task consistency segmentation network based on shape feature description is divided into three parts, including medial surface extraction task, cartilage segmentation task, and contour detection task.

[0049] For the segmentation network, a single shape feature limits the segmentation result. Especially when the shape feature is not significant, the improvement effect is significantly weakened. Multi - task learning can solve multiple related tasks together and achieve the sharing of internal features. These tasks are trained to optimize the output. However, the main drawback of existing multi - task learning networks is that it is difficult to perform collaborative refinement among multiple tasks. To overcome this problem, the present invention proposes a very different multi - task learning network that consistently models the medial surface and contour shape of articular cartilage, not only achieving the multi - task coordination, compatibility, and consistency of the prediction results, but also improving the discriminative ability of the network for large shape changes under different OA states and the blurred or even disappeared boundaries of severely OA articular cartilage.

[0050] Let D and R be the ground - truth data set and the prediction result set respectively. The prediction result is Y, and the ground - truth mask is Z. Let Y seg ∈R, Z seg ∈D be the predicted segmentation map and the ground - truth segmentation map, Y con ∈R, Z con ∈D be the predicted contour and the ground - truth contour, Y med ∈R, Z med ∈D be the predicted medial surface and the ground - truth medial surface. Define the supervised loss function:

[0051]

[0052] Next, define the consistency loss between the shape reconstruction shape mapping y recon ∈Y recon based on the differentiable medial surface and the predicted segmentation mapping y seg ∈Y seg

[0053]

[0054] Then, use the first - order directional derivative to obtain the medial - surface - based shape reconstruction shape y recon ∈Y​recon The contour and the predicted segmentation map y seg ∈ Y seg For the contour, the consistency loss composed of the two can be defined and the consistency loss composed of the predicted contour y con ∈ Y con and the predicted segmentation map The consistency loss is composed of

[0055]

[0056] The loss of the segmentation network proposed is defined as:

[0057]

[0058] Use the gradient vector and the Hessian matrix to obtain the ground truth of the medial surface of the cartilage and the cartilage boundary from the manually labeled cartilage region.

[0059] 1. The contour detection task of the cartilage

[0060] Let I(x) be the three-dimensional image of the voxel x, be the Hessian matrix of the image blurred by the isotropic Gaussian function, and its standard deviation is σ. Similarly, let be λ1, λ2, λ3 respectively, and their corresponding eigenvectors are e1, e2, e3 (|e1| = |e2| = |e3|). is used for cartilage enhancement to obtain the initial cartilage region. These values are used as the initial values for accurate boundary localization. Using the well-known theory that the first-order derivative has an extreme value at the boundary point, let the eigenvalue e1 represent the normal direction of the cartilage surface. Based on the initial point, the first directional derivative along the normal direction is further used for accurate localization of the cartilage boundary.

[0061] 2. For the medial surface extraction task

[0062] The cartilage region should include all potential medial surfaces. The eigenvalue e3 is expected to give the orthogonal direction of the articular cartilage, and only |λ3| is large on its medial surface. In addition, the local second-order approximation is used to extract the medial surface, and this approximation is composed of and the gradient vector. Similarly, the orthogonal direction of the articular cartilage is obtained using the eigenvector corresponding to the Hessian matrix blurred by the isotropic Gaussian function, and then the local second-order approximation is used to extract the medial surface.

[0063] To quantitatively evaluate the accuracy of the detected cartilage boundary, the Dice similarity coefficient (DSC) metric was used for measurement. The larger the value, the higher the accuracy. Finally, tests were conducted on 30 test data sets, and the average DSC obtained was 98.4%. These results indicate that the detected cartilage boundary is accurate and can provide correct information for cartilage volume prediction. Similarly, the quantitative accuracy of the extracted medial surface was evaluated.

[0064] 3. Reconstruction of the Medial Surface

[0065] Since the backpropagation of the neural network requires the embedding process of the network to be differentiable, a smooth estimation algorithm for reconstruction operations was proposed to achieve the reconstruction of the cartilage shape based on the medial surface, making the neural network differentiable during the training process; behind the backbone network, two branches were designed: the distance transformation prediction branch and the medial surface prediction branch

[0066] Let the output of the distance transformation prediction branch be denoted as dis v (ψ, ψ dis ), denoted as dis v :

[0067]

[0068] Let MS v (Ψ, Ψ med ) be the probability feature map of the medial surface prediction branch, simply denoted as MS v , representing the probability that the voxel V on the predicted articular cartilage is the medial surface.

[0069] To make the voxel values belonging to the articular cartilage in the reconstructed shape probability map as close to 1 as possible and the voxel values in the background as close to 0 as possible, an approximate reconstruction algorithm based on the sigmoid function was proposed. The value of the reconstructed shape probability map at voxel V is defined as:

[0070]

[0071] The derivatives of r v with respect to dis v and |MS v | can be obtained by applying the differentiable chain rule. Let z = -dis v ·|MS v | / K, then the differentiability proof is as follows:

[0072]

[0073] As can be seen from Equation (8), it is negative when the voxel is within the cartilage region and positive at the background. Therefore, the distance transformation map was reconstructed into a predicted segmentation image using a sigmoid-based function, and the medial surface points MS vThe probability and parameter K are used to modify and adjust the shape of the reconstructed cartilage. The ground truth segmentation mask Z is used to supervise and train the reconstructed shape probability map. A supervised loss function between and Z is defined as:

[0074]

[0075] Applying this method to all points in the 3D image, the final probability map of the reconstructed shape can be obtained.

[0076] Step B: A shape prior description is defined, which represents the shape distribution of multiple objects; and a deep shape prior network model composed of Autoencoder1 and Autoencoder2 is established, as Figure 2 shown. Among them, Autoencoder1 consists of a decoder and an encoder to achieve the output of the reconstructed image.

[0077] Suppose there is a set of images I, features F, and the reconstructed cartilage shape I s . Then, the encoder and decoder are respectively represented as I→F, F→I s , for a given image I, its reconstructed image is represented as I r = g d (f e (I; Θ; ζ). To minimize the reconstruction error, Autoencoder1 is trained, and this reconstruction error is expressed using the mean squared error as:

[0078]

[0079] In Autoencoder2, the encoded output of Z is represented as f e (Z; θ) = I e , after training, the optimal parameter θ * can be obtained and remains unchanged in subsequent training. In addition, a shared layer of Autoencoder1 and Autoencoder2 is designed, and the loss is defined as:

[0080] L s = KL(f e (i; Θ) || f e (Z; θ * )) (14)

[0081] The purpose of the loss is to minimize the KL divergence, making the predicted cartilage shape distribution converge to the true cartilage shape distribution. For this stage, the total loss L is known.

[0082] Step C: For the hard segmentation cases where the adjacent tissue textures, severe lesion sites, and contact cartilage are similar or have low contrast, a new deep learning network framework is established. Guided by shape features and combined with shape prior constraints, the segmentation becomes more robust and accurate.

[0083] The training of the multi-task consistency segmentation network is divided into two steps: pre-training and fine-tuning. During pre-training, the network is trained on the 3D MRI volume I and the ground truth Z. For fine-tuning, the segmentation network is incorporated into the deep shape prior network described in Step B. By simply fine-tuning all the pre-trained parameters, the segmentation network is trained for the downstream tasks, and the shape prior constraint is imposed on the re-trained segmentation network during network training. For this purpose, the segmentation network is placed after the autoencoder learning of the shape prior network model. The segmentation network only needs to be pre-trained once, and the subsequent fine-tuning for downstream tasks incurs very low cost.

[0084] As Figure 3 shown, the pre-trained segmentation network is placed behind the autoencoder learning. Correspondingly, the shape prior network is incorporated into the pre-trained segmentation network, and the final loss function for fine-tuning the segmentation network is as follows:

[0085] L total (I, Z; Θ, ζ, η) = L prior-net (I, Z; Θ, ζ) + λL tseg-net (I, Z; Θ, ζ, η) (15)

[0086] By minimizing L total (I, Z; Θ, ζ, η), the optimal parameters can be obtained:

[0087] (Θ * , ζ * , η * ) = argmin Θ,ζ,η (I, Z; Θ, ζ, η) (16)

[0088] Step D: Different from the recently proposed deep neural network with prior constraints, a recursive strategy is adopted for iterative optimization in the test stage. This network imposes prior constraints by minimizing the network training loss function on the output mask of the segmentation network. During the optimization process, through recursive learning, the cartilage shape prior and the cartilage segmentation prediction mask are continuously iteratively optimized, making the shape of the cartilage segmentation prediction mask closer to the actual cartilage shape, thus completing cartilage segmentation with high accuracy.

[0089] During the test process, an iterative strategy is described. As Figure 4 shown, in the first iteration, the test image I is used as the input of Autoencoder1, and the output is denoted as Moreover, through the segmentation network, In the subsequent iterative process, such as Figure 5 as shown, take as the input of Autoencoder2 to obtain the output In addition, use as the shape prior constraint of the segmentation network and continue the iteration until the predefined maximum number of iterations is reached.

[0090] The method of the multi-task consistent segmentation network based on the depth shape prior model for cartilage segmentation trained in the embodiments of the present invention is used to perform segmentation on two publicly available knee joint image datasets and one clinical hip joint image dataset, and the segmentation results are compared with those of other state-of-the-art methods. The specific analysis is as follows:

[0091] The segmentation network method proposed in the present invention is evaluated on two different publicly available MRI datasets (OAI Imorphics and OAI ZIB) and one non-public MRI dataset (clinical hip joint image dataset). The following are the two publicly available datasets:

[0092] (1.1) The OAI ZIB dataset consists of 507 MRI images, which have been thoroughly manually segmented by experienced users at the Zuse Institute in Berlin. The data covers the full range of OA grades and has a strong trend of severe cases.

[0093] (1.2) The OAI Imorphics dataset consists of 88 MRIs, and the manual segmentation is provided by Imorphics. This dataset only contains moderate and severe OA cases. The non-public database consists of 28 MRIs, and the corresponding expert segmentation is manually annotated by the clinical cooperation unit.

[0094] (1) MR images of 9 hips are obtained using a fat-suppressed 3D fast spoiled gradient echo (SPGR) sequence on a 1.5T MR system (Horizon, General Electric). The imaging parameters are as follows: repetition time (TR) / echo time (TE) 24.4 / 5.7 ms; flip angle, 200 degrees; section thickness, 1.5 mm; in-plane resolution, 0.625 mm; imaging matrix, 256×256; imaging direction, coronal.

[0095] (2) MR images of 19 hip joints were obtained on a 3.0T magnetic scanner (GE Healthcare, Waukesha, WI, USA). The imaging parameters were as follows: repetition time (TR) / echo time (TE), 28 / 4 ms; flip angle, 250 degrees; section thickness, 1.5 mm; in-plane resolution, 0.625 mm; imaging matrix, 256×256; imaging orientation, sagittal plane.

[0096] To facilitate comparison with previous work, the widely used Dice similarity coefficient (DSC), average surface distance (ASD), volume difference (VD), and volume overlap extent (VOE) were adopted as evaluation metrics.

[0097] To objectively and truly evaluate the accuracy and performance of the segmentation network method proposed in the present invention, it was applied to the OAIZIB, OAI Imorphics, and Hip datasets. The number allocation of training samples and test samples was as follows: The segmentation accuracy of OAI ZIB and OAIImorphics was subjected to 2-fold cross-validation. For the OAI ZIB dataset (a total of 507 cases), the 507 datasets were randomly divided into two groups (253 groups and 254 groups), with one group for training and the other for testing. For the OAI Imorphics dataset (a total of 88 cases), the BL and (12m) FU images of 44 cases were used for training.

[0098] Before the input network, the multimodal volume is non-zero extracted, and the intensity is normalized to zero mean and unit variance of a single volume for standardization and homogenization. After obtaining the network prediction, post-processing is performed. If the total number of predicted enhancing tumors is less than the threshold, the enhancing tumor voxels are first replaced with necrosis, and then connected component analysis is performed to remove small components. The threshold for each dataset is independently determined by optimizing the average DSC. The remaining 44 cases are used for testing. Four-fold cross-validation is performed to evaluate the segmentation accuracy of the hip joint dataset (a total of 28 hip joints). Four-fold cross-validation is performed on the hip joint dataset (28 hip joints) to evaluate the segmentation accuracy. It is randomly divided into 4 groups, each group has 7 of the same size, and 3 of them are used for training, and the remaining 1 group is used for testing. In the experiment, in order to reduce the influence of discretization, an interpolation method is used to interpolate the 3D MR images so that the signal sampling is isotropic in all three directions, and the image matrix size is doubled. In this study, Sinc interpolation is used to create isotropic voxel sizes. After Sinc interpolation, the knee dataset is cropped to 126×126×110 voxels, and the hip dataset sub-volume is cropped to 120×120×120 voxels as the network input. The framework is trained by the AdamW optimizer for 12,000 iterations with an initial learning rate of 0.001. The difference in segmentation accuracy between the cartilage segmentation method proposed by the present invention and other methods is evaluated using a paired t-test, and all statistical tests are significant at P<0.05.

[0099] The segmentation network method of the present invention is compared with the current state-of-the-art cartilage segmentation methods. The following are the comparison results with other state-of-the-art methods on three different datasets:

[0100] 1. Results of the OAI ZIB dataset

[0101] Table 1 summarizes the comparison results of the segmentation network method proposed by the present invention and the state-of-the-art methods in the OAI ZIB database. The average DSCs of FC and TC measured by this method are 96.72±0.04% and 92.77±0.05% respectively, which are the highest compared with the existing methods. In addition, most of the reported results are better than other methods. Although the ASD (0.11±0.07mm) and VD (0.98±8.14) of FC obtained by the best method previously proposed by Ebrahimkhani et al. are lower than those of the segmentation network method proposed by the present invention, the results are also comparable to this result in terms of ASD and VD. The errors of the segmentation network method proposed by the present invention for ASD and VD are 0.12±0.05mm and -1.2±7.22% respectively.

[0102] Table 1 Comparison results of the segmentation network method proposed by the present invention and the state-of-the-art methods in the OAI ZIB database

[0103]

[0104] 2. OAI Imorphics Dataset Results

[0105] The comparison of the segmentation network method proposed by the present invention with other state-of-the-art methods on the OAI validation dataset of baseline (BL) and baseline (FU) is listed in Table 2. In baseline BL, the segmentation network method proposed by the present invention is the best method for all metrics. Using this method, the DSCs of FC and TC are measured to be 98.70±0.02% and 98.46±0.03% respectively. These results are better than the previous best methods, which reported excellent performance. The ASD and VOE errors of this method are: 0.15±0.12 mm and 2.46±3.05% for FC, and 0.03±0.04 mm and 3.11±2.84% for TC, both of which are better than other methods and lower than the previous best methods. The VD errors of FC and TC in the previous best scheme are -1.32±2.3% and 0.51±4.67% respectively, while the VD errors of this method are -1.52±1.65% and 0.62±2.16% respectively.

[0106] Table 2 Comparison Results of the Segmentation Network Method Proposed by the Present Invention with Other State-of-the-art Methods on the OAI Validation Dataset of Baseline (BL) and Baseline (FU)

[0107]

[0108] 3. Hip Dataset Results

[0109] The segmentation network method proposed by the present invention is compared with the U-Net and nnU-Net methods. Table 3 gives the quantitative comparison results of hip cartilage segmentation by three different methods. The DSCs of this method for acetabular cartilage and femoral cartilage are significantly higher than those of U-Net and nnU-Net. The average ASD, VOE and VD of acetabular cartilage are 0.29 mm, 5.47±8.36% and 16.5±7.82% respectively, and the average ASD, VOE and VD of femoral cartilage are 0.24 mm, -4.28±10.62% and 14.42±6.6% respectively. These values are the lowest compared with the previous best method proposed by nnU-Net. The results show that the proposed method achieves the best performance in all metrics, and the variance obtained by the segmentation network method of the present invention is the smallest, indicating the robustness to different test data.

[0110] Figure 7 The visual segmentation results obtained by 3D nnU-Net and the proposed method in the hip joint image dataset with severe cartilage loss are shown. Compared with the segmentation network method proposed by the present invention, the existing 3D nnU-Net gives incorrect connections in areas with lower severe cartilage loss.

[0111] Table 3 Quantitative comparison results of hip cartilage segmentation using three different methods based on the hip joint dataset

[0112]

[0113] The segmentation network method of the present invention was tested on two publicly available knee image datasets and one clinical hip image dataset, and its results were compared with recently published methods. The results show that the method for cartilage segmentation of the present invention can improve the performance of regions containing low contrast, blurred boundaries between cartilage and adjacent structures, highly deformed parts, and regions in contact with cartilage, significantly improving the accuracy of overall cartilage segmentation, especially for severely arthritic knees or hips.

[0114] In summary, the multi-task consistent segmentation network based on the deep shape prior model for cartilage segmentation of the present invention establishes a very different deep learning network framework through the guidance of shape feature description and the constraint of shape prior description, with high accuracy and strong generalization ability, significantly improving the accuracy and generalization ability of knee and hip cartilage segmentation, and having a variability level similar to those obtained from human raters. In particular, it can handle difficult cases with low contrast and blurred boundaries between cartilage and other adjacent structures, highly deformed parts, and regions in contact with cartilage.

Claims

1. A multi-task consistent segmentation network method based on a depth shape prior model for cartilage segmentation, characterized in that Including the following steps: Step A: Establish a multi-task consistent segmentation network based on shape feature description; define the cartilage shape feature description as the medial surface, segmentation, and contour of articular cartilage to characterize the geometric properties of articular cartilage; divide the segmentation task into three parts, which are responsible for medial surface extraction, cartilage segmentation, and contour detection tasks respectively; Step B: Establish a deep shape prior network model to achieve the reconstruction of a given image; the shape prior description represents the shape distribution of articular cartilage; Step C: Introduce the network model established in Step B into the network model established in Step A, and jointly establish a multi-task consistent segmentation network constrained by the shape prior model, guided by establishing a cartilage segmentation network model and constrained by deep prior shape modeling to improve the robustness and accuracy of segmentation; Step D: Based on the test of recursive learning, adopt a recursive strategy for iterative optimization in the test stage, and this network imposes a prior constraint by minimizing the network training loss function on the output mask of the segmentation network; During the optimization process, through recursive learning, continuously optimize the cartilage shape prior and the cartilage segmentation prediction mask during the iterative process, making the shape of the cartilage segmentation prediction mask closer to the actual cartilage shape, so as to complete cartilage segmentation with high precision.

2. The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 1, characterized in that: In Step A, the cartilage can be locally described as a thin disk, which corresponds to finding the position using the eigenvalues of these Hessian matrices, and using the gradient vector and Hessian matrix to obtain the groundtruth of the medial surface and cartilage boundary of the cartilage from the manually labeled cartilage area; including the following steps: (1) For the task of cartilage contour detection, let \(I(x)\) be the 3D image of voxel \(x\), be the Hessian matrix of the image blurred by the isotropic Gaussian function, with a standard deviation of \(\sigma\); similarly, let be \(\lambda_1\), \(\lambda_2\), \(\lambda_3\) respectively, and their corresponding eigenvectors be \(e_1\), \(e_2\), \(e_3\) (\(\vert e_1\vert=\vert e_2\vert=\vert e_3\vert\)); is used for cartilage enhancement to obtain the initial cartilage region; The above values are used as the initial values for accurate boundary localization; then, using the well-known theory that the first derivative has an extreme value at the boundary point, let the eigenvalue e1 represent the cartilage surface normal, and on the basis of the initial point, further accurately locate the cartilage boundary using the first directional derivative along the normal direction.

3. The multi-task consistency segmentation network based on a depth shape prior model for cartilage segmentation according to claim 2, characterized in that: It also includes the following (2) For the medial surface extraction task, the cartilage region should include all potential medial surfaces; the eigenvalue e3 is expected to give the orthogonal direction of articular cartilage, and only |λ3| is large on its medial surface; in addition, the medial surface is extracted using a local second-order approximation, which consists of and the gradient vector; the orthogonal direction of articular cartilage is obtained using the eigenvector e3 corresponding to the Hessian matrix blurred by an isotropic Gaussian function, then the medial surface is extracted using a local second-order approximation, and the Dice similarity coefficient is used to quantitatively evaluate the accuracy of the extracted medial surface on 30 test data sets.

4. The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 3, characterized in that: It also specifically includes the following (3) Reconstruction of the medial surface The backpropagation of the neural network requires that the embedding process of the network is differentiable. A smooth estimation algorithm for reconstruction operations is proposed to achieve the reconstruction of the cartilage shape based on the medial surface, making the neural network differentiable during the training process; behind the backbone network, design two branches: the distance transformation prediction branch and the medial surface prediction branch; Let the output of the distance transformation prediction branch be denoted as dis v (Ψ, ψ dis ), denoted as dis v , where Ψ is the network parameter and ψ dis is the parameter of the distance transformation prediction branch; according to the definition of the distance transformation graph, dis v is expressed as: Wherein, C is the articular cartilage in the 3DMR image, and u and v are two different voxels in the 3D MRI volume. is the contour of C, C in and C out are the inner region and the outer region of the articular cartilage, respectively. Let MS v (Ψ, Ψ med ) be the probability feature map of the inner surface prediction branch, abbreviated as MS v , representing the probability that the voxel V on the predicted articular cartilage is the inner surface; Ψ med is the parameter of the inner surface prediction branch; In order to make the voxel values belonging to articular cartilage in the reconstructed shape probability map as close to 1 as possible and the voxel values in the background as close to 0 as possible, an approximate reconstruction algorithm based on the sigmoid function is proposed. The value of the reconstructed shape probability map at voxel V is defined as: Apply the differentiable chain rule to find r separately v for dis v and the derivative of |MS v |; Select a larger parameter K to control the steepness of the curve and make it as close as possible to the step function in Equation (9); r v Implement it as a sigmoid function, and use the differentiable chain rule to find the function r respectively v For dis v and |MS v |, let z = -dis v ·|MS v | / K, then the differentiability can be proved as follows: As can be seen from Equation (8), it is negative when the voxel is within the cartilage region and positive at the background. Therefore, a sigmoid-based function is used to reconstruct the distance transform map into a predicted segmentation image, and the probability of the medial surface points MS v and a large parameter K are used to modify and adjust the shape of the reconstructed cartilage; the ground truth segmentation mask Z is used to supervise and train the reconstructed shape probability map, and a supervised loss function between and Z is defined: Applying this method to all points in the three-dimensional image can obtain the final probability map of the reconstructed shape.

5. The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 4, characterized in that: In Step B, define the shape prior description, which represents the shape distribution of multiple objects; and establish a deep shape prior network model composed of Autoencoder1 and Autoencoder2; Among them, Autoencoder1 consists of a decoder and an encoder to achieve the output of the reconstructed image; in order to minimize the reconstruction error, Autoencoder1 is trained; Autoencoder2 is trained to obtain the optimal parameters of its encoder, which remain unchanged in subsequent training; In addition, a shared layer of Autoencoder1 and Autoencoder2 is designed, and the loss of the shared layer is defined using the KL divergence and minimized to make the predicted shape distribution converge to the true cartilage shape distribution.

6. The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 5, characterized in that: In step C, the training of the multi-task consistency segmentation network based on the shape prior model constraint is divided into two steps: pre-training and fine-tuning; during pre-training, the network is trained on 3D MRI volume I and ground truth Z; in order to perform fine-tuning, the segmentation network is incorporated into the deep shape prior network described in step B, and the segmentation network model for the downstream task is trained by simply fine-tuning all pre-trained parameters. After re-training the segmentation network, a shape prior constraint is imposed on the re-trained segmentation network when training the network; for this purpose, the segmentation network is placed after the autoencoder learning of the shape prior network model, and the segmentation network only needs to be pre-trained once, and subsequent fine-tuning for downstream tasks.

7. The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 1, characterized in that: In step A, for the multi-task consistency segmentation network based on shape feature description, visual transformer is used as the backbone. Visual transformer uses the multi-head attention mechanism to extract different features of the input image, meeting the requirements of multi-task consistency; among them, three partial tasks are guided by three different supervision methods: the segmentation prediction branch, the medial surface prediction branch, and the contour prediction branch are supervised by the segmentation mask, the medial surface mask, and the contour mask respectively; Let D and R be the true value data set and the prediction result set respectively. Denote the prediction result set as Y and the true value mask as Z. Let Y seg ∈R, Z seg ∈D be the predicted segmentation map and the ground truth segmentation map, Y con ∈R, Z con ∈D be the predicted contour and the ground truth contour, Y med ∈R, Z med ∈D be the predicted inner surface and the ground truth inner surface. Define the supervised loss function: wherein represents voxel-level summation in a three-dimensional image, represents image-level summation in a dataset; Next, define the shape mapping y for shape reconstruction based on the differentiable medial surface recon ∈Y recon and the consistency loss between the predicted segmentation mapping y seg ∈Y seg ​ Among them, represents the voxel-level summation in the 3D image, represents the image-level summation in the dataset; Then, the shape y of the shape reconstruction based on the medial surface is obtained using the first directional derivative recon ∈Y recon of the contour; similarly, the predicted segmentation map y seg ∈Y seg of the contour is obtained using the first-order directional derivative; assume is the contour of the reconstructed cartilage shape, is the contour of the predicted segmentation map; Define the consistency loss composed of and as well as the consistency loss composed of the predicted contour y and con ∈Y con and ​ Among them, represents the voxel-level summation in the 3D image, represents the image-level summation in the dataset; Finally, define the consistency loss composed of the predicted contour y con ∈ Y con and ​ Among them, represents the voxel-level summation in the 3D image, represents the image-level summation in the dataset; The loss of the segmentation network is defined as:

8. The multi-task consistency segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 7, characterized in that: In step B, a shape prior description is defined, which represents the shape distribution of multiple objects; and a deep shape prior network model consisting of Autoencoder1 and Autoencoder2 is established. Among them, Autoencoder1 consists of a decoder and an encoder to achieve the output of the reconstructed image. In order to optimize these parameters, Autoencoder1 is trained to minimize the reconstruction error; Autoencoder2 is trained to obtain the optimal parameters of its encoder, which remain unchanged in subsequent training; Suppose there is a set of images I, features F, and reconstructed cartilage shapes I s , then, the encoder and decoder are respectively represented as where Θ and ζ are the learnable parameters of the encoder and decoder respectively. For a given image I, its reconstructed image is represented as I r = g d (f e (I; Θ; ζ); In order to optimize these parameters, the autoencoder is trained to minimize the reconstruction error, which is also called the loss function. This loss function is expressed using the mean square error as: In Autoencoder2, the encoded output of Z is denoted as f e (Z; θ) = I e , where θ are the learnable parameters of the encoder layer f e (·, θ), and the optimal parameters θ are obtained after training Autoencoder2 * and remain unchanged; in addition, a shared layer of Autoencoder1 and Autoencoder2 is designed, and the loss is defined by the Kullback-Leibler (KL) divergence: L s = KL(f e (i; Θ)||f e (Z; θ * )) (14) The purpose of the loss is to minimize the KL divergence and make the predicted cartilage shape distribution converge to the true cartilage shape distribution. For this stage, the total loss L is known; In step C, in order to perform fine-tuning, the segmentation network is incorporated into the deep shape prior network described in step B, and the final loss function for fine-tuning the segmentation network is as follows: L total (I, Z; Θ, ζ, η) = L prior-net (I, Z; Θ, ζ) + λL tseg-net (I, Z; Θ, ζ, η) (15) where λ > 0 controls the contribution of the segmentation network loss, usually set as λ = 0.1, and the optimal parameters are obtained by minimizing L total (I, Z; Θ, ζ, η). (Θ * ,ζ * ,η * ) = argmin Θ,ζ,η (I, Z; Θ, ζ, η) (16).

9. The multi-task consistent segmentation network method based on a depth shape prior model for cartilage segmentation according to claim 8, wherein: Two publicly available knee image datasets, OAIImorphicsKnee and OAIZIBKnee, and one non - publicly available clinical hip image dataset, Hip dataset, were used; the Dice similarity coefficient (DSC), average surface distance (ASD), volume difference (VD), and volume overlap error (VOE) were used to evaluate the segmentation performance.

Images