Prostate MRI segmentation method based on multi-scale context modeling

Through a multi-scale context modeling module and a first-in-first-out dynamic adjustment mechanism, combined with the Leaky ReLU layer and jump connection, the shortcomings of capturing global context information in prostate MRI segmentation are solved, and segmentation accuracy is improved, especially at the apex and bottom of the prostate. The segmentation effect of the end area.

CN119027432APending Publication Date: 2024-11-26AFFILIATED HUSN HOSPITAL OF FUDAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411064690.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing prostate MRI segmentation methods perform poorly in capturing remote relationships and global context information, resulting in inaccurate segmentation results, especially in the apical and bottom areas of the prostate, which is difficult to extract boundary information.

Method used

Using a multi-scale context modeling module and a first-in-first-out dynamic adjustment mechanism, the boundary pixel representation is enhanced through the self-attention mechanism and feature storage module, the feature vector selection is optimized, and the Leaky ReLU layer and jump connection are combined to build a deep learning network for prostate MRI. segmentation.

Benefits of technology

The accuracy of prostate MRI segmentation is improved, especially the segmentation effect at the apical and bottom areas, which enhances the attention of the backbone network to the boundary area and improves the accuracy of segmentation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027432A_ABST
    Figure CN119027432A_ABST
Patent Text Reader

Abstract

The invention discloses a prostate MRI segmentation method based on multi-scale context modeling, and the method is characterized in that the method comprises the following steps: constructing a prostate MRI segmentation model based on a deep learning network, training the prostate MRI segmentation model through employing a training data set, and deploying the prostate MRI segmentation model; and inputting MRI image data obtained in real time into the prostate MRI segmentation model, and performing real-time reasoning by the prostate MRI segmentation model to obtain a final segmentation result. The present invention provides a multi-scale context modeling module that enhances the representation of boundary pixels by minimizing the effects of irrelevant features, thereby improving the segmentation results. In addition, a first-in first-out dynamic adjustment mechanism is further introduced, and selection of feature vectors is optimized, especially in the top end area and the bottom end area of the prostate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a prostate MRI segmentation method based on multi-scale context modeling, belonging to the field of medical image processing technology. Background Technology

[0002] Prostate cancer is the second leading cause of cancer death among men worldwide and a leading cause of cancer death in adult men. Early-stage prostate lesions are difficult to detect on imaging, and more than half of prostate cancer patients are diagnosed at an advanced stage. The most common treatment for prostate cancer is external beam radiation therapy (EBRT). To achieve precise radiation therapy and minimize damage to surrounding healthy tissues from the radiation, accurate segmentation of the prostate gland using magnetic resonance imaging (MRI) is a crucial step in the treatment of prostate cancer.

[0003] Because the boundaries between the prostate and surrounding tissues and organs are not clear in MRI images, manually delineating the prostate is very time-consuming and can lead to differences in segmentation results depending on the operator. In recent years, with the gradual development of Artificial Intelligence (AI) and AI-based deep learning image processing technologies, deep learning algorithms have been widely used in the research and application of medical image-assisted diagnostic systems. The emergence of the U-Net model has revolutionized the field of medical image segmentation. This model uses an encoder-decoder convolutional neural network (CNN) architecture with multi-scale skip connections between the encoder and decoder, which can maximize the preservation of high-resolution image information. Many subsequent segmentation models have been proposed based on U-Net. Among them, nnU-Net uses instance normalization instead of batch normalization in U-Net and Leaky ReLU instead of ReLU. Compared to U-Net, U-Net++ uses more nesting and denser skip connections to better capture fine-grained details of foreground objects. Fully convolutional networks improve segmentation performance by fusing feature maps from different depths to integrate global and local information. 3D U-Net, VNet, and DenseVoxelNet are U-Net architectures that use 3D convolutions instead of 2D convolutions. They have been proven effective for image data with similar cross-pixel distances in all three dimensions. Although U-Net-based models have greatly contributed to advancements in medical image segmentation, they often perform poorly in capturing long-range relationships and global contextual information due to limitations in the receptive field of the convolutional kernels. Therefore, researchers have naturally turned their attention to self-attention (SA). TransUNet was the first to apply ViT to medical image segmentation tasks. It combines the features of both Transformer and U-Net architectures, introducing the self-attention mechanism of the Transformer architecture to capture global contextual information and improve feature representation capabilities. Swin-UNet is the first U-shaped network purely based on the more efficient Swin transformer, outperforming models based on FCN methods. UNETR and SiwnUNETR are Transformer architectures extended for 3D input. While ViT-based networks offer improved performance compared to U-Net, these methods utilize standard or moving-window-based SA, which is fine-grained local SA and may overlook the interaction between local and global information. Therefore, prostate segmentation requires enhancing the pixel representation of boundaries by incorporating specific feature extraction modules and optimizing feature vector selection. Summary of the Invention

[0004] The objective of the present invention is to improve the accuracy of prostate MRI segmentation.

[0005] To achieve the above objective, the technical solution of the present invention provides a prostate MRI segmentation method based on multi-scale context modeling, which is characterized by including the following steps:

[0006] First step: Obtain MRI image data with prostate organ range annotation, and construct an MRI image dataset.

[0007] Second step: After preprocessing all the MRI image data in the MRI image dataset, construct a training dataset and a test dataset based on the MRI image dataset.

[0008] Third step: Construct a prostate MRI segmentation model based on a deep learning network, and use the training dataset to train the prostate MRI segmentation model. Among them, in the prostate MRI segmentation model:

[0009] The input MRI image data undergoes a non-linear transformation through a Leaky ReLU layer, and the output of the Leaky ReLU layer is input into N layers of feature encoders. Among them, the input of the first layer of feature encoder is the output of the Leaky ReLU layer, and the input of the nth layer of feature encoder is the input of the (n - 1)th layer of feature encoder, 1 < n ≤ N. At the same time, the encoded feature maps extracted by each layer of feature encoder are saved as the skip link of the current layer. The first skip link to the (N - 1)th skip link are respectively sent into (N - 1) layers of feature fusion modules, and (N - 2) layers of context modeling modules are respectively added between every two of the first skip link to the (N - 1)th skip link; Except for the (N - 1)th layer of feature fusion module, the nth layer of feature fusion module fuses the output of the (n + 1)th layer of context modeling module and the nth layer of skip link for feature fusion and channel compression, while the (N - 1)th layer of feature fusion module fuses the upsampled Nth layer encoded result and the (N - 1)th layer of skip link for feature fusion and channel compression. Among them, the Nth layer encoded result is to fuse the Nth layer of skip link with multiple dataset-level features stored in the feature storage module, and these dataset-level features have the highest similarity with the current Nth layer of skip link; Each layer of context modeling module has two inputs, one of which is the result after upsampling the output of the current layer of feature fusion module, and the other input is the result after calculating the class probability distribution of the current layer of skip link; The output of each layer of feature fusion module passes through a segmentation head to obtain the segmentation result of the current layer, and the segmentation result of the first layer is the segmentation result finally output by the prostate MRI segmentation model;

[0010] The fourth step is to evaluate and optimize the trained prostate MRI segmentation model using the test dataset, and then deploy the prostate MRI segmentation model.

[0011] The fifth step involves inputting the real-time acquired MRI image data into the prostate MRI segmentation model, which then performs real-time inference to obtain the final segmentation result.

[0012] Preferably, in the second step, the preprocessing of the MRI image data includes the following steps:

[0013] Step 201: Crop the non-zero regions of the MRI image data;

[0014] Step 202: Select the median voxel spacing of the entire MRI image dataset, and interpolate each MRI image data in the current MRI image dataset to the isotropic voxel spacing.

[0015] Step 203: Normalize each MRI image data using Z-score.

[0016] Preferably, the feature encoder's encoding process for the input data includes one convolution operation and one max pooling operation.

[0017] Preferably, during the training of the prostate MRI segmentation model, updating the feature storage module includes the following steps:

[0018] Initialize the feature storage module;

[0019] Obtain multiple dataset-level features that have the highest similarity to the current Nth layer skip link from the feature storage module;

[0020] The new dataset-level features are obtained by fusing the multiple dataset-level features obtained in the previous step with the Nth layer skip link described above.

[0021] While pushing the new dataset-level features obtained in the previous step into the feature storage module, the bottom-most dataset-level features stored in the feature storage module are popped out, completing an update of the feature storage module.

[0022] Preferably, the two inputs of each context modeling module are defined as features f. sf and feature f seg , where feature f seg The feature f is the result of interpolation calculation for the skip links in the current layer. sf Given the upsampled output of the current layer's feature fusion module, each layer's context modeling module is based on feature f. sf and feature f segThe following steps are used to obtain the output R. sf :

[0023] Step 301: Calculate feature f using a self-attention mechanism. sf and feature f seg Similarity between W sf :

[0024]

[0025] In the formula, C represents the number of channels. Represents matrix multiplication;

[0026] Step 302: Calculate the similarity aggregation semantic features using the following formula to obtain the image-level representation R. sf As the output of each layer of context modeling module:

[0027] Preferably, in the third step, the current layer skip link uses the following formula to implement the category probability distribution, and the obtained result is expressed as f. seg Then we have:

[0028]

[0029] In the formula: l represents the image foreground and background category classification obtained through deep supervision, 0 represents the image background category, and 1 represents the image foreground category; Nor represents the normalization equation; D l R represents the predicted probability that all pixels in the image corresponding to the current layer skip link belong to category l; l This indicates the true value of the deep supervised segmentation result of the image corresponding to the skip link in the current layer, belonging to category l.

[0030] This invention provides a multi-scale context modeling module that enhances the representation of boundary pixels by minimizing the influence of irrelevant features, thereby improving segmentation results. Furthermore, this invention introduces a first-in-first-out dynamic adjustment mechanism to optimize feature vector selection, particularly in the apical and basal regions of the prostate.

[0031] Compared with existing technical solutions, the beneficial effects of the present invention are as follows:

[0032] This invention enhances the pixel representation of boundary regions by increasing the backbone network's attention to these regions. A key challenge in prostate segmentation is that the top and bottom regions are typically very small, making it difficult to extract boundary information. To address this, this invention proposes a first-in, first-out (FIFO) strategy to dynamically adjust feature vectors and select the optimal feature vector to improve the segmentation performance of the top and bottom slices of the prostate. Attached Figure Description

[0033] Figure 1This is a network structure diagram of the prostate MRI segmentation model based on multi-scale context modeling of the present invention;

[0034] Figure 2 This is a flowchart of the multi-scale context modeling module of the present invention;

[0035] Figure 3 This is a flowchart of the feature storage module of the present invention. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0037] The prostate MRI segmentation method based on multi-scale context modeling disclosed in this invention includes the following steps:

[0038] Step 1: Obtain MRI image data containing annotations of the prostate organ area and construct an MRI image dataset. The MRI image dataset may include T2-weighted imaging (T2WI) scan sequences, etc., and the annotation of the prostate region in the MRI image data is done manually by a professional doctor.

[0039] In this embodiment of the invention, the collected MRI image data is recorded in DICOM format. Precise annotation of the prostate region is manually performed by at least two experienced radiologists. To evaluate the performance of the prostate MRI segmentation network based on multi-scale context modeling, each MRI image in the test dataset obtained from the MRI image dataset includes manually annotated prostate regions. The segmentation output of the prostate MRI segmentation network is compared with the results of manual annotation to assess the segmentation performance of the network. This method can effectively reduce the errors that may be introduced by relying solely on manual annotation or using only the prostate MRI segmentation network for segmentation.

[0040] Step 2: Preprocess all MRI image data in the MRI image dataset. The preprocessing process includes cropping, resampling, and normalization.

[0041] In this embodiment of the invention, the non-zero regions of the MRI image dataset are first cropped to reduce computational cost; then, the median voxel spacing of the entire MRI image dataset is selected, and the MRI image data in this MRI image dataset is interpolated to an isotropic voxel spacing of [0.66×0.66×5] mm3, and Z-score is used to normalize each data example.

[0042] Step 3: Construct a deep learning network for the context modeling module. After initializing the parameters in this deep learning network, train the network using the MRI image dataset obtained in Step 2, continuously optimizing and updating the network's weights and parameters to obtain an efficient and accurate prostate MRI segmentation model. Figure 1 As shown, the main structure of the constructed prostate MRI segmentation model includes a feature encoder, a context modeling module, a feature storage module, a feature fusion module, and a feature decoder.

[0043] In this invention, the parameters of the prostate MRI segmentation model mainly include network depth, initial weights, stride, bias, number of convolutional kernels, kernel size, number of context modeling modules, and number of Leaky ReLU layers. The weights and biases are continuously updated and iterated as the network training progresses, while the number of convolutional kernels, kernel size, number of context modeling modules, and number of Leaky ReLU layers are determined experimentally and are parameters that guarantee the optimal segmentation structure; these parameters remain unchanged during network training.

[0044] In this embodiment of the invention, step 3 specifically includes the following steps:

[0045] Step 3-1, in this embodiment of the invention, combined with Figure 1 The prostate MRI segmentation model includes N feature encoders, N-1 feature decoders (each feature decoder is used to upsample the input), 1 Leaky ReLU layer, N-2 context modeling modules, N-1 feature fusion modules and a feature storage module, where N is the optimal positive integer determined through multiple comparative experiments.

[0046] In this invention, considering that the number of network layers N has a significant impact on feature extraction and segmentation results, a deep learning network with different layers is built for training by pre-setting the value of N. The network uses Leaky ReLU as the activation function to perform a non-linear transformation on the input data, allowing positive inputs to pass while scaling negative inputs. This non-linear characteristic helps the deep learning network learn more complex data patterns. Compared to ReLU, it can reduce the occurrence of neuron death and prevent the gradient of ReLU from being 0 when the input is less than 0, which would cause the weights not to be updated during backpropagation, thereby improving the training efficiency and performance of the deep learning network. Furthermore, the performance of the deep learning network is evaluated and the segmentation effect of the prostate tissue region is analyzed using multiple indicators such as the Dice Similarity Coefficient (DSC) and the 95% Hausdorff Distance (HD) to obtain the most suitable value of network layer N. In this invention, the prostate MRI segmentation model achieves the best segmentation effect on the prostate region when N=5.

[0047] Step 3-2: In the prostate MRI segmentation model, the preprocessed prostate image is subjected to N feature encoding operations. Each encoding process includes one convolution operation and one max pooling operation to reduce the spatial resolution of the feature map while increasing the number of channels in the feature map.

[0048] Step 3-3: At each stage of the encoding path, the extracted encoded feature maps are saved as skip connections. Skip connections allow the deep learning network to reuse the encoded feature maps from the encoding stage in the decoding stage and the context modeling module, retaining more image feature information.

[0049] like Figure 1 As shown, the feature map obtained in each layer of the encoding process has two functions: 1. It is fused with the decoded features obtained from the previous layer's feature decoder or the output of the context modeling module, and used as an input to the current layer's context modeling module; 2. It is directly calculated by performing probability distribution to obtain the current layer's deep supervised features, which are used as another input to the current layer's context modeling module.

[0050] Steps 3-4: Except for the last level of jump links, add a context modeling module between each level of jump link, defining the two inputs of the context modeling module as f. sf and f seg .

[0051] f seg The deep supervised feature obtained from the class probability distribution D, as shown in Equation (1), has two channels:

[0052]

[0053] In the formula: f seg The size is H×W×C, where H×W represents the pixel size of the current layer image, and C is the number of channels; l represents the image foreground and background category classification obtained through deep supervision, with 0 representing the image background category and 1 representing the image foreground category; Nor represents the normalization equation; D l R represents the predicted probability that all pixels belong to category l; l This indicates the true value of the deep supervised segmentation result belonging to category l.

[0054] f sf This refers to semantic features at the image level, representing the semantic features of the currently trained image. Definition For each layer of the feature decoder, the semantic features obtained are j∈[1,N-1]. When j=N-1, This is the result of upsampling the output of the last feature encoder layer. When j = N-2, It is The result of upsampling is obtained after feature fusion and channel compression via the feature fusion module, which is connected to the penultimate skip link. When 1 <j≤N-3, It is the result of upsampling after feature merging and channel compression of the output of the j-th layer skip link and the j+1-th layer context modeling module.

[0055] Steps 3-5: To reduce the influence of irrelevant features, in each layer of context modeling module, the following formula is used to evaluate the two semantic features f. sf and f seg Aggregation is performed, and f is computed using a self-attention mechanism. sf and f seg Similarity between W sf .

[0056]

[0057] In the formula, C is the number of channels, and H and W represent the height and width of the image, respectively. This represents matrix multiplication.

[0058] Steps 3-6: Calculate similarity and aggregate semantic features using the following formula to obtain the image-level representation R. sf As the output of each layer of context modeling module.

[0059]

[0060] Steps 3-7: In each skip link of step 3-3, repeat the operations of steps 3-5 to 3-6 to obtain more detailed image-level features.

[0061] Step 4: To overcome the insufficient robustness of image-level semantic features, this embodiment of the invention also introduces a feature storage container in the prostate MRI segmentation model. This feature storage container uses a first-in-first-out mechanism to dynamically store image-level features and obtain dataset-level features, specifically including the following steps:

[0062] Define the dataset-level features as f dl It refers to the feature region representation obtained from the training data of the entire dataset. During the training phase of the prostate MRI segmentation model, the dataset-level features f stored in the feature storage container are used. dl During real-time inference after training the prostate MRI segmentation model, the dataset-level features f are not updated. dl .

[0063] Data set-level features f stored in the feature storage container dl Update using the following method:

[0064] First, initialize a feature storage container of size n×H×W×C to dynamically store f. dl Where n is the storage for f dl The number of features, H×W×C, is the dataset-level feature f. dl The size. The feature f of the last skip link is combined with the existing dataset-level features f in the feature storage container. dl Similarity calculation is performed, and the three dataset-level features f with the highest similarity to feature f in the feature storage container are selected. dl —respectively represented as Three dataset-level features f dl The features are fused with feature f to obtain the updated dataset-level features, denoted as f. d ′ l As shown in the following formula:

[0065]

[0066] In the formula, δ represents the transformation function that reduces the number of input matrix tensor channels.

[0067] Update the dataset-level features f d ′ l While pushing the feature storage container, the dataset-level features at the bottom of the feature storage container are popped out, completing one update of the feature storage container.

[0068] When using the prostate MRI segmentation model for real-time inference, the feature f of the last skip link is fused with the three dataset-level features with the highest similarity in the feature storage container, and this is used as the updated last skip link (i.e., as the output of the last feature encoder).

[0069] Step 5: After multiple feature decoding and feature fusion processes, the prostate MRI segmentation model finally generates a high-resolution segmentation map with the same spatial resolution as the original input image.

[0070] In this example, the prostate MRI segmentation model gradually restores the spatial resolution of the feature map through feature decoding. At the same time, it concatenates the skip connection feature map from the encoding path with the feature map of the current decoding stage. This concatenation operation on the dimensional channel combines the high-resolution features of the encoding stage with the high-abstraction features of the decoding stage, enhancing the network's understanding of image details and context.

[0071] Step 6: Input the images from the test dataset into the trained prostate MRI segmentation model to obtain the prostate organ segmentation image for each MRI image.

[0072] In this example, the test set is ensured to be independent of the training and validation sets, and the proposed prostate MRI segmentation method based on multi-scale context modeling is evaluated using evaluation metrics such as Dice and HD95.

[0073] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A prostate MRI segmentation method based on multi-scale context modeling, characterized in that: It includes the following steps: Step 1: Obtain MRI image data with prostate organ range annotation and construct an MRI image dataset; Step 2: After preprocessing all the MRI image data in the MRI image dataset, construct a training dataset and a test dataset based on the MRI image dataset; Step 3: Construct a prostate MRI segmentation model based on a deep learning network, and use the training dataset to train the prostate MRI segmentation model. In the prostate MRI segmentation model: The input MRI image data undergoes a non-linear transformation through a Leaky ReLU layer, and the output of the Leaky ReLU layer is input into N layers of feature encoders. Among them, the input of the first layer of feature encoder is the output of the Leaky ReLU layer, and the input of the nth layer of feature encoder is the input of the (n - 1)th layer of feature encoder, where 1 < n ≤ N. At the same time, the encoded feature maps extracted by each layer of feature encoder are saved as the skip connections of the current layer. The first skip connection to the (N - 1)th skip connection are respectively fed into (N - 1) layers of feature fusion modules, and (N - 2) layers of context modeling modules are added between each pair of the first skip connection to the (N - 1)th skip connection; Except for the (N - 1)th layer of feature fusion module, the nth layer of feature fusion module fuses the output of the (n + 1)th layer of context modeling module and the nth layer of skip connection for feature fusion and channel compression, while the (N - 1)th layer of feature fusion module fuses the upsampled (N)th layer of encoded result and the (N - 1)th layer of skip connection for feature fusion and channel compression. Among them, the (N)th layer of encoded result is to fuse the (N)th layer of skip connection and multiple dataset-level features stored in the feature storage module, and these dataset-level features have the highest similarity with the current (N)th layer of skip connection; Each layer of context modeling module has two inputs, one of which is the result after upsampling the output of the current layer of feature fusion module, and the other input is the result after calculating the class probability distribution of the current layer of skip connection; The output of each layer of feature fusion module obtains the segmentation result of the current layer after passing through a segmentation head, and the segmentation result of the first layer is the segmentation result finally output by the prostate MRI segmentation model; Step 4: After using the test dataset to complete the evaluation and optimization of the trained prostate MRI segmentation model, deploy the prostate MRI segmentation model; Step 5: Input the real-time obtained MRI image data into the prostate MRI segmentation model, and perform real-time inference by the prostate MRI segmentation model to obtain the final segmentation result.

2. The prostate MRI segmentation method based on multi-scale context modeling according to claim 1, characterized in that: In Step 2, the preprocessing of the MRI image data includes the following steps: Step 201: Crop the non-zero regions of the MRI image data; Step 2 ​ 3. The prostate MRI segmentation method based on multi-scale context modeling according to claim 1, characterized in that: ​ 4. The prostate MRI segmentation method based on multi-scale context modeling according to claim 1, characterized in that: During the prostate MRI segmentation model training, the feature storage module is updated, including the following steps: Initializing the feature storage module; Obtaining multiple data set-level features having the highest similarity between the feature storage module and the current N-th layer jump link; The multiple dataset-level features obtained in the previous step are fused with the current N-th layer skip link to obtain new dataset-level features; While pushing the new dataset-level features obtained in the previous step into the feature storage module, the dataset-level features stored at the bottom of the feature storage module are popped out, completing an update of the feature storage module.

5. The prostate MRI segmentation method based on multi-scale context modeling according to claim 1, characterized in that: The two inputs of each layer of context modeling module are defined as features f sf and feature f seg , where the feature f seg is the result of interpolation calculation of the jump link of the current layer, feature f sf is the result of upsampling the output of the feature fusion module of the current layer, and each layer of context modeling module is based on the feature f sf and feature f seg The following steps are used to obtain the output R sf : Step 301: Calculate feature f using self-attention mechanism sf and feature f seg The similarity between sf : Where C is the number of channels, Represents matrix multiplication; Step 302: Calculate the similarity aggregation semantic features by the following formula to obtain the image level representation R sf As the output of each layer of context modeling module:

6. The prostate MRI segmentation method based on multi-scale context modeling according to claim 1, characterized in that: In the third step, the current layer jump link uses the following formula to realize the category probability distribution, and the obtained result is expressed as f seg , then: Where: l represents the image foreground and background category classification obtained through deep supervision, 0 represents the image background category, and 1 represents the image foreground category; Nor represents the normalization equation; D l represents the predicted probability that all pixels in the image corresponding to the current layer jump link belong to category l; R l Indicates that the deep supervised segmentation result of the image corresponding to the jump link of the current layer belongs to the true value of category l.