A muscle segmentation model method based on multimodal MRI images

By using two encoder and attention mechanism submodule with two structurally consistent parameters and unshared encoder and attention mechanism submodule in muscular tissue MRI image segmentation, the problem of insufficient accuracy and stability of muscle tissue segmentation in the prior art is solved, and the problem of insufficient muscle tissue segmentation is achieved. Higher segmentation accuracy and stability.

CN114511540BActive Publication Date: 2024-11-26PEKING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210123897.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-01-25
Filing Date
2022-02-10
Publication Date
2024-11-26
Estimated Expiration
2042-02-10

AI Technical Summary

Technical Problem

The prior art lacks accuracy and stability in muscle tissue MRI image segmentation, especially in multimodal image processing, the information complementary relationship between modes cannot be fully utilized, resulting in insufficient noise processing capability.

Method used

Two encoders with consistent structures and unshared parameters are used to encode FAT mode and WATER mode images respectively. The intermediate features are integrated through the attention mechanism submodule to generate muscle segmentation results, and the model is trained through the decoder and backpropagation to improve segmentation. accuracy.

Benefits of technology

The accuracy and stability of muscle segmentation of multimodal MRI images are improved, the utilization of image structure information and semantic information is enhanced, and the segmentation accuracy on data with high noise is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114511540B_ABST
    Figure CN114511540B_ABST
Patent Text Reader

Abstract

The present invention discloses a muscle segmentation model method based on multimodal MRI images, belonging to the technical field of medical image segmentation. The present invention concentrates the physical information of all MRI data, generates FAT modality and WATER modality image features through an encoder for each MRI data, integrates the information of the two modalities through an attention mechanism submodule, and finally generates the result of muscle segmentation through a decoding module, compares the result with the real annotation, trains the model by back propagation, and obtains a trained muscle segmentation model based on multimodal MRI images. In the case of multimodality, the present invention makes full use of the complementarity between each modality, deeply mines the structural information and semantic information of the image, and improves the accuracy and stability of muscle segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a muscle segmentation model method based on multimodal MRI images. Background Art

[0002] Fatty infiltration refers to the presence of fat cells in the stroma of tissues and organs that normally do not contain fat cells. It is a phenomenon of neuromuscular disorders and muscle metabolic disorders, and mainly occurs in tissues such as the heart, pancreas, and skeletal muscle. Clinically, the severity of fatty infiltration can reflect the progression of the disease. In recent years, the use of magnetic resonance imaging (MRI) for non-invasive assessment of muscle tissue has gradually attracted attention. Analysis of MRI images can measure the fat composition of various organs, including muscle tissue. Lipo-water decomposition MRI is a quantitative MRI technology that has been used in clinical practice and is widely used in the quantitative study of fatty infiltration. This technology can separate fat and water protons into FAT modal images and WATER modal images based on the chemical shift difference between them. Segmenting different muscle tissues based on the above two modal images is the first and key step in quantitative research.

[0003] To complete the segmentation of muscle tissue, the traditional method is to manually outline it, which is not only an extremely time-consuming and tedious process, but also prone to subjective bias. Recent related work has developed a variety of automatic segmentation algorithms, but most of them are based on only one modality image for segmentation, or simply sum or splice two modality images, without making good use of the complementary information relationship between the two modality images, and the ability to handle noise in the image is insufficient. The accuracy is insufficient for some noisy data. Therefore, the current muscle segmentation task on MRI images faces problems such as insufficient accuracy and insufficient stability.

[0004] In 2017, Ghosh from Canada proposed a solution (ICIP 2017P4437-4441): in each layer of the MRI image, a convolutional neural network is used to compress the image information into a feature vector, and then the decoding step of principal component analysis is used to decode the feature vector into muscle tissue segmentation results.

[0005] In 2020, Vardhanabhuti from Hong Kong, China proposed another solution (Insights into imaging 2020P1-11): after denoising the FAT modal images and WATER modal images of the thigh, they are input into a U-shaped architecture neural network together to obtain the segmentation results of each muscle tissue. Summary of the invention

[0006] The purpose of the present invention is to provide a muscle segmentation model method based on multimodal MRI images, which is used to make full use of the complementarity between the various modalities in a multimodal situation, deeply mine the structural information and semantic information of the image, and improve the accuracy and stability of muscle segmentation.

[0007] In order to solve the above technical problems, the technical solutions provided by the present invention are as follows:

[0008] A muscle segmentation model method based on multimodal MRI images, the specific steps include (such as Figure 1 shown):

[0009] A traverses every MRI image in the entire data set and parses the physical information of the MRI image;

[0010] B uses two encoders with consistent structures and unshared parameters to encode a pair of MRI images, which come from the FAT modality and the WATER modality respectively, and obtains multiple intermediate features of the two modalities;

[0011] C uses the attention mechanism submodule to splice the intermediate features of the above two modalities to obtain the integrated multimodal features;

[0012] D generates muscle segmentation results through the decoder, compares the results with the real annotations, trains the model through back propagation, and obtains a trained muscle segmentation model based on multimodal MRI images.

[0013] Further, step A is specifically as follows:

[0014] 1) Traverse each MRI image I in the entire dataset i , analyze the layer thickness information in the image (d i ,h i , w i ), where d i ,h i , w i Respectively represent the physical depth, height, and width of each voxel in the image;

[0015] 2) Calculate the average layer thickness of the data set All MRI images are scaled to this layer thickness;

[0016] 3) Traverse each scaled MRI image I′ i , parsing resolution information (D i , H i , W i ), where D i , H i , W i are the depth, height and width of the image respectively;

[0017] 4) Calculate the maximum resolution based on the resolution information of each image And pad each MRI image to this resolution.

[0018] Further, step B is specifically as follows:

[0019] Encoder 1 encodes the MRI image from the FAT modality, where the input MRI image first passes through the convolution submodule 1.1 to obtain intermediate feature 1; then the intermediate feature 1 is downsampled through the pooling layer and input to the convolution submodule 1.2 to obtain intermediate feature 2; and so on to obtain multiple intermediate features;

[0020] Encoder 2, encodes MRI images from the WATER modality, and the encoding process is the same as that of encoder 1.

[0021] Furthermore, the specific operation of each attention mechanism submodule in step C is:

[0022] 1) Use a 3D convolutional layer to calculate the relative weights of the two modalities;

[0023] 2) At each spatial position of the feature, read the relative weights w1 and w2 of the two modes at the current position from the corresponding position of the weight array, read the feature vector 1 and feature vector 2 of the current position from the two modes respectively, and obtain the new feature vector by weighted summation of the two feature vectors according to the weights.

[0024] Furthermore, in step D, the decoder is composed of alternating convolutional submodules and upsampling layers. Starting from the bottom-level features, the spatial dimensions of the features are gradually parsed using upsampling layers, and the convolutional submodules are used to fuse multiple layers of features, and finally the muscle segmentation results are output.

[0025] Furthermore, the encoder includes multiple stacked pooling layers and convolution submodules, and the convolution submodule takes the original image data or the output of the previous convolution submodule as input to obtain intermediate features.

[0026] Furthermore, the convolution submodule includes multiple convolutional layers and nonlinear layers. The image data will first pass through the convolutional layer to obtain features with unchanged array shape, and then use the nonlinear layer to set all negative numbers in the features to 0 to obtain the output intermediate features.

[0027] The muscle segmentation method of multimodal MRI images provided by the present invention improves the accuracy and stability of multimodal muscle segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a muscle segmentation model method based on multimodal MRI images of the present invention;

[0029] Figure 2 It is a convolution submodule framework diagram provided by the present invention;

[0030] Figure 3 This is a framework diagram of the attention mechanism submodule provided by the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below by way of examples:

[0032] (I) Pretreatment

[0033] Traverse each MRI image I in the entire dataset i , analyzing the physical information of the image (i.e., layer thickness, which identifies the physical depth, height, and width of each voxel in the image (d i ,h i , w i )). The module then calculates the average layer thickness And scale all MRI images to this layer thickness. Then, traverse each scaled MRI image I′ i , parsing resolution information (D i , H i , W i ), where D i , H i , W i are the depth, height, and width of the image respectively. Finally, the module calculates the maximum resolution based on the resolution information of each image. And pad each MRI image to this resolution.

[0034] (II) Muscle segmentation model

[0035] The first is the overall process from MRI images to segmentation prediction results. The format of the example data is two MRI images, which are from the FAT modality and the WATER modality respectively. The data structure of all images, features and segmentation results in the overall process is a 4-dimensional array. Next, take encoder 1 as an example to explain how the present invention encodes image features (the same applies to encoder 2). The array shape of the FAT modality image is (1, D, H, W). The input image first passes through the convolution submodule 1.1 to obtain intermediate feature 1, whose array shape is (32, D, H, W), where "32" and the image array shape "1" refer to the number of channels; then the intermediate feature 1 is downsampled through the pooling layer, and the array shape becomes (32, D / 2, H / 2, W / 2), and then input into the convolution submodule 1.2 to obtain intermediate feature 2, whose array shape is (64, D / 2, H / 2, W / 2); and so on to obtain intermediate features 3 and 4, whose array shapes are (128, D / 4, H / 4, W / 4) and (256, D / 8, H / 8, W / 8), respectively. For the two sets of intermediate features obtained from the two modalities, the present invention uses four attention mechanism sub-modules to fuse them to obtain fused intermediate features 1-4, whose array shapes are (32, D, H, W), (64, D / 2, H / 2, W / 2), (128, D / 4, H / 4, W / 4) and (256, D / 8, H / 8, W / 8), respectively. Based on these fused intermediate features, the decoder starts from the fused intermediate feature 4, and after the convolution submodule 3.4, the array shape becomes (128, D / 8, H / 8, W / 8), and then the upsampling layer is used for upsampling to obtain the decoded intermediate feature 4, whose array shape is (128, D / 4, H / 4, W / 4); the decoded intermediate feature 4 will be concatenated with the fused intermediate feature 3 to obtain a feature with an array shape of (256, D / 4, H / 4, W / 4), and then input into the convolution submodule 3.3 to perform similar operations as the convolution submodule 3.4; and so on to obtain the decoded intermediate feature 3 and the decoded intermediate feature 2 respectively. Finally, the decoded intermediate feature 2 and the fused intermediate feature 1 are concatenated and then input into the convolution submodule 3.1 to obtain the image segmentation result M. pred , its array shape is (D, H, W).

[0036] Next is the operation details of the convolution submodule. Assume that the number of input channels of the convolution submodule is C1, and the number of output channels is C2. Given an input feature, its array shape is (C1, D′, H′, W′), the feature will first pass through convolution layer 1 to obtain a feature with an array shape of (C2, D′, H′, W′), and then use a nonlinear layer to set all negative numbers in the feature to 0, and then pass through convolution layer 2 to obtain a feature with an unchanged array shape, and finally use a nonlinear layer to set all negative numbers in the feature to 0 to obtain the output feature.

[0037] Next, the operation details of the attention mechanism submodule. Given the input features of the FAT modality and the input features of the WATER modality, their array shapes are both (C, D′, H′, W′); each attention mechanism submodule first concatenates the two features to obtain a concatenated feature with an array shape of (2C, D′, H′, W′); then the concatenated feature is input to the 3D convolution layer, which calculates the relative weights of the two modalities and obtains a weight array with an array shape of (2, D′, H′, W′); finally, at each spatial position of the feature, the relative weights w1 and w2 of the two modalities at the current position are read from the corresponding position of the weight array, and the feature vector 1 and feature vector 2 of the current position are read from the two modalities respectively (the array shapes are both (C,)), and the two feature vectors are weighted and summed according to the weights to obtain a new feature vector with an array shape of (C,), and all positions are traversed to obtain the output feature with an array shape of (C, D′, H′, W′).

[0038] (III) Training Model

[0039] Given the predicted segmentation result M pred and gold standard manual annotation results M gt , their array shapes are all (D, H, W), and the value range of each position of both is {0, 1, ..., G}, where G represents the number of categories. The present invention calculates the following loss function:

[0040]

[0041] in, Represents the value of the predicted segmentation result at position (i, j, k), It represents the value of the gold standard manual annotation result at position (i, j, k). δ(x, y) is a judgment function. When the values ​​of x and y are consistent, δ(x, y) takes the value of 1, otherwise it is 0.

[0042] This loss function measures the consistency between the model segmentation results and the actual annotation results. The higher the consistency between the two, the lower the value of the loss function. The training goal of the model is to minimize the value of the loss function. In order to minimize the value of the loss function, the model is trained by starting from the loss function, passing the gradient back to each parameter in encoder 1, encoder 2, attention mechanism submodules 1-4 and decoder, and then letting each parameter update itself according to its corresponding gradient.

[0043] The above process is repeated until the segmentation prediction result of each MRI image in the training set can achieve a high degree of consistency with the gold standard manual annotation result of the image.

[0044] Using the adjusted parameters of encoder 1, encoder 2, attention mechanism sub-modules 1-4 and decoder, given a data set containing two MRI images, which are from the FAT modality and WATER modality respectively, encoder 1 is used to encode the features of the FAT modality image, encoder 2 is used to encode the features of the WATER modality, and then the attention mechanism sub-modules 1-4 are used to integrate the features of the two modalities. Finally, the decoder is used to decode and obtain the segmentation prediction result of the MRI image.

[0045] The embodiments described above are not intended to limit the present invention. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention is defined by the scope of the claims.

Claims

1. A muscle segmentation model method based on multimodal MRI images, It is characterized in that The specific steps include: A. Traverse every MRI image in the entire data set and parse the physical information of the MRI image; B. Two encoders with consistent structures and unshared parameters are used to encode a pair of MRI images, which come from the FAT modality and the WATER modality respectively, to obtain multiple intermediate features of the two modalities; C. Use the attention mechanism submodule to splice the intermediate features of the above two modalities to obtain the integrated multimodal features; D. Generate muscle segmentation results through the decoder, compare the results with the real annotations, train the model through back propagation, and obtain a trained muscle segmentation model based on multimodal MRI images. The loss function is calculated as: in, Represents the value of the predicted segmentation result at position (i, j, k), It represents the value of the gold standard manual annotation result at position (i, j, k). δ(x, y) is a judgment function. When the values ​​of x and y are consistent, δ(x, y) takes the value of 1, otherwise it is 0. The loss function measures the consistency between the model segmentation result and the true annotation result. The higher the consistency between the two, the lower the value of the loss function. The training goal of the model is to minimize the value of the loss function.

2. The muscle segmentation model method based on multimodal MRI images as claimed in claim 1, It is characterized in that Step A is specifically as follows: 1) Traverse each MRI image I in the entire dataset i , analyze the layer thickness information in the image (d i ,h i , w i ), where d i ,h i , w i Respectively represent the physical depth, height, and width of each voxel in the image; 2) Calculate the average layer thickness of the data set All MRI images are scaled to this layer thickness; 3) Traverse each scaled MRI image I′ i , parsing resolution information (D i , H i , W i ), where D i , H i , W i are the depth, height and width of the image respectively; 4) Calculate the maximum resolution based on the resolution information of each image And pad each MRI image to this resolution.

3. The muscle segmentation model method based on multimodal MRI images as claimed in claim 1, It is characterized in that Step B is specifically as follows: Encoder 1 encodes the MRI image from the FAT modality, where the input MRI image first passes through the convolution submodule 1.1 to obtain intermediate feature 1; then the intermediate feature 1 is downsampled through the pooling layer and input to the convolution submodule 1.2 to obtain intermediate feature 2; and so on to obtain multiple intermediate features; Encoder 2, encodes MRI images from the WATER modality, and the encoding process is the same as that of encoder 1.

4. The muscle segmentation model method based on multimodal MRI images as claimed in claim 1, It is characterized in that The specific operation of the attention mechanism submodule in step C is: 1) Use a 3D convolutional layer to calculate the relative weights of the two modalities; 2) At each spatial position of the feature, read the relative weight w of the two modes at the current position from the corresponding position of the weight array 1 and w 2 , read out the eigenvector 1 and eigenvector 2 of the current position from the two modes respectively, and obtain the new eigenvector by weighted summing up the two eigenvectors according to the weights.

5. The muscle segmentation model method based on multimodal MRI images as claimed in claim 1, It is characterized in that The encoder contains multiple stacked pooling layers and convolution sub-modules. The convolution sub-module takes the original image data or the output of the previous convolution sub-module as input to obtain intermediate features.

6. The muscle segmentation model method based on multimodal MRI images as claimed in claim 5, It is characterized in that The convolution submodule includes multiple convolutional layers and nonlinear layers. The image data will first pass through the convolutional layer to obtain features with unchanged array shape, and then use the nonlinear layer to set all negative numbers in the features to 0 to obtain the output intermediate features.

7. The muscle segmentation model method based on multimodal MRI images as claimed in claim 1, It is characterized in that The decoder consists of alternating convolutional submodules and upsampling layers. It starts from the bottom-level features, gradually uses upsampling layers to parse the spatial dimensions of the features, uses convolutional submodules to fuse multiple layers of features, and finally outputs the muscle segmentation results.

8. The muscle segmentation model method based on multimodal MRI images as claimed in claim 1, It is characterized in that The model is trained by starting with the loss function, passing the gradient back to each parameter in the two encoders, the attention mechanism module, and the decoder, and then updating each parameter according to its corresponding gradient.