Multimodal lung MRI image segmentation method based on deep learning and edge information fusion

Abstract

The present invention discloses a multimodal lung MRI image segmentation method based on deep learning and edge information fusion, which labels the lung proton MRI images and associates the lung proton MRI images with the corresponding lung hyperpolarization images. 129 XeMRI images are used as samples to construct a training set; a multimodal lung MRI image segmentation network is constructed at the same time; a total loss function is set; the multimodal lung MRI image segmentation network is trained using the training set; and the trained multimodal lung MRI image segmentation network is used to process the lung hyperpolarization 129 The XeMRI image is segmented to obtain a segmented image. The present invention provides a practical and effective method to significantly improve lung hyperpolarization 129 Accuracy of XeMRI image segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multimodal lung MRI image segmentation method that integrates deep learning and edge information, and is suitable for the segmentation of multimodal lung MRI images. Background Art

[0002] Magnetic resonance image segmentation technology is an important branch of magnetic resonance imaging (MRI), and its purpose is to separate and identify different tissues and organs in MRI images. By segmenting the lung contour, the volume and size of the lung change area can be calculated more accurately, and accurate input can be provided for computer-aided systems, thereby improving the efficiency of automated analysis and processing of medical images.

[0003] Traditional lung MRI image segmentation methods are mainly based on threshold segmentation, region growing and other technologies. Threshold segmentation technology is suitable for multi-image segmentation problems, especially for images with clear contrast between targets and backgrounds. It has a good segmentation effect and has a fast calculation speed. It is suitable for real-time processing and segmentation of large-scale image data. However, threshold segmentation technology is sensitive to noise in the image and is easily affected by noise and produces mis-segmentation. It also has poor segmentation effect when dealing with multi-target segmentation and overlapping target segmentation, making it difficult to perform effective segmentation. The region growing method has a good segmentation effect for images with strong local consistency and continuity, and can also effectively process some irregularly shaped targets. However, the region growing method is sensitive to the selection of initial seed points. For some targets with fuzzy boundaries, the region growing method is also prone to over-segmentation or under-segmentation. With the development of deep learning technology, MRI image segmentation technology based on deep learning has gradually become a research hotspot. The lung contour segmentation method based on deep learning is mainly based on convolutional neural network (CNN), recurrent neural network (RNN), etc. for feature image extraction and segmentation. By training a large amount of lung image data, the segmentation network can learn the characteristic representation of the lung contour, thereby realizing the automated segmentation process. A large number of experimental results show that the lung contour segmentation method based on deep learning has higher accuracy and robustness than the traditional method. The U-Net network model proposed by Ronneberger et al. based on the fully convolutional neural network (FCN) has become a popular segmentation network in recent years [Ronneberger O et al., MICCAI, 2015, 18: 234-241]. The model uses elastic deformation image augmentation to allow the network to learn more stable image features. In addition, Vaswani et al. proposed a model architecture called Transformer [Vaswani A et al, NIPS., 2017, 30: 6000-6010], which contains multiple layers of stacked encoders and decoders and expands the use of attention mechanisms.

[0004] Although deep learning has achieved certain results in lung contour segmentation, its segmentation effect still needs to be improved. First, due to different imaging methods, medical images of different modalities have many differences in reflecting tissue characteristics and physiological information. Using only a single modality image for segmentation often leads to the loss of some information. At the same time, single modality images are highly sensitive to noise and artifacts, which will interfere with the segmentation results and lead to poor segmentation effects. Vaidhya et al. first proposed the use of multi-modality for image segmentation [Vaidhya Ket al., IEEE Trans Med Imaging, 2016: 181-194], which improved the segmentation effect to a certain extent, but it will lead to grayscale fusion and easy mixing for areas with similar grayscale, affecting the local segmentation effect.

[0005] In addition, due to the limitations of imaging equipment, the similarity of tissue structure, and the influence of noise and artifacts, the edge of the contour is often the area with the worst segmentation effect in the segmentation operation, but edge information plays an important role in medical image segmentation and can help in positioning, quantification, feature extraction, detection and analysis, as well as visualization and interaction [Majumdar A et al., IEEE Eng Med Biol Soc, 2018: 583-587]. Therefore, the segmentation accuracy of the edge part still needs to be further improved during the segmentation process.

[0006] Therefore, in order to improve the above limitations of deep learning in lung contour segmentation, the present invention proposes a multimodal lung MRI image segmentation method that integrates deep learning with edge information, so as to improve the generalization ability and robustness of the segmentation model and improve lung hyperpolarization. 129 Accuracy of XeMRI image segmentation. Summary of the invention

[0007] The purpose of the present invention is to address the above-mentioned problems existing in the prior art and propose a multimodal lung MRI image segmentation method that integrates deep learning and edge information, which uses information from different modalities to complement each other and improve the prediction ability and accuracy of the model; combines semantic information with edge information to improve the accuracy of edge area segmentation, thereby improving lung hyperpolarization. 129 Segmentation effect of XeMRI images.

[0008] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0009] The multimodal lung MRI image segmentation method based on deep learning and edge information fusion includes the following steps:

[0010] Step 1: Label the lung proton MRI images and associate the lung proton MRI images with the corresponding lung hyperpolarization129 XeMRI images were used as samples to construct the training set;

[0011] Step 2: Construct a multimodal lung MRI image segmentation network;

[0012] Step 3: Set the total loss function L;

[0013] Step 4: According to the total loss function L, the multimodal lung MRI image segmentation network is trained using the training set, and the parameters of the multimodal lung MRI image segmentation network are saved;

[0014] Step 5: Use the trained multimodal lung MRI image segmentation network to process the lung hyperpolarization 129 XeMRI images are segmented to obtain the final lung hyperpolarization 129 Segmented image of the lung contour in a XeMRI image.

[0015] As mentioned above, step 1 also includes operations of normalizing the data of the samples and unifying the image size.

[0016] As mentioned above, the multimodal lung MRI image segmentation network includes a segmentation module and an edge information module.

[0017] The segmentation module includes an encoder and a decoder;

[0018] The encoder includes a convolutional neural network, a feature fusion block, and a Transformer block;

[0019] The decoder includes an upsampling layer and an output layer;

[0020] The edge information module includes a convolution block and an edge detection block.

[0021] As mentioned above, the edge information module includes 3 convolution blocks and 3 edge detection blocks;

[0022] The lung proton MRI image in the sample is input into the first convolution block, and the feature image output by the first convolution block is input into the second convolution block and the first edge detection block respectively; the feature image output by the second convolution block is input into the third convolution block and the second edge detection block respectively; the feature image output by the third convolution block is input into the third edge detection block.

[0023] As described above, the convolutional neural network includes an input layer and a first convolution module, a second convolution module, and a third convolution module, which are arranged in sequence. The lung proton MRI images and lung hyperpolarization in the sample 129 XeMRI images were input into the convolutional neural network to obtain lung proton MRI images and lung hyperpolarization 129 Feature images corresponding to XeMRI images.

[0024] As mentioned above, the feature fusion block includes convolutional layers and activation layers.

[0025] Proton MRI images of the lungs and lung hyperpolarization in the sample 129 The characteristic images obtained after the XeMRI images are processed by the encoder are subjected to matrix multiplication to obtain the lung proton MRI images and the corresponding lung hyperpolarization in the sample. 129 The relationship between each pixel in the XeMRI image is obtained by the activation layer to obtain a fused feature image, and the fused feature image is then passed through a convolution layer with a convolution kernel size of 1×1 to obtain a one-dimensional vector, which is input into multiple consecutive Transformer blocks to obtain a one-dimensional feature vector.

[0026] As mentioned above, the decoder part includes 4 upsampling layers and 1 output layer.

[0027] The feature image obtained by reshaping the one-dimensional feature vector is used as the input of the first upsampling layer. In the first upsampling layer, the feature image is processed by the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the first upsampling layer.

[0028] The two-modal feature images output by the third convolution module, the edge feature images output by the third edge detection block, and the feature images output by the first upsampling layer are used as the input of the second upsampling layer. In the second upsampling layer, the four input feature images are stacked according to the channels, and then multi-modal feature images are fused through the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the second upsampling layer.

[0029] The two-modal feature images output by the second convolution module, the edge feature images output by the second edge detection block, and the feature images output by the second upsampling layer are used as the input of the third upsampling layer. In the third upsampling layer, the four input feature images are stacked according to the channels, and then multi-modal feature images are fused through the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the third upsampling layer.

[0030] The two-modal feature images output by the first convolution module, the edge feature image output by the first edge detection block, and the feature image output by the third upsampling layer are used as the input of the fourth upsampling layer. In the fourth upsampling layer, the four input feature images are stacked according to the channel, and then the multi-modal feature images are fused through the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the fourth upsampling layer.

[0031] The feature image output by the fourth upsampling layer is input to the final output layer, and then passes through the convolution layer and ReLU activation layer in the final output layer to output the lung hyperpolarization. 129 Segmented image of the lung contour in a XeMRI image.

[0032] As mentioned above, the total loss function L is based on the following formula:

[0033] L=-(y t log(y p )+(1-y t )log(1-y p ))

[0034] Among them, y t Hyperpolarization of the lungs 129 The label of the XeMRI image sample, y p Represents the probability that the predicted segmented image of the lung contour obtained by the multimodal lung MRI image segmentation network is a positive example.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention uses lung proton MRI images and lung hyperpolarization 129 XeMRI images are fused with multimodal features to reduce the network model’s reliance on a single modality and reduce hyperpolarization 129 The influence of high noise and low resolution of XeMRI images enhances the robustness and stability of the model;

[0037] 2. Compared with the conventional multimodal image segmentation method, the present invention also combines the fusion of edge information and designs an edge detection block, which combines edge information with multimodal semantic information and uses edge information as prior information to aggregate context information, which can improve the accuracy of image segmentation and detail retention, making the segmentation result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the present invention;

[0039] Figure 2 It is a schematic diagram of the structure of the segmentation network of the present invention;

[0040] Figure 3 It is a structural diagram of the edge detection module;

[0041] Figure 4 This is a performance comparison chart of the present invention and other methods on the constructed test set. DETAILED DESCRIPTION

[0042] In order to facilitate those skilled in the art to understand and implement the present invention, the following embodiments and Figure 1-4 The present invention is further described in detail. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] The multimodal lung MRI image segmentation method based on deep learning and edge information fusion specifically includes the following steps:

[0044] Step 1: Data preprocessing. In this example, 1285 lung MRI images of two modalities were obtained from the MRI image data of 96 scanned subjects. The two modal lung MRI images are lung proton MRI images and lung hyperpolarization MRI images. 129 XeMRI images and label lung proton MRI images, and label lung proton MRI images and corresponding lung hyperpolarization 129 XeMRI images as samples, lung proton MRI images and lung hyperpolarization in each sample 129 The samples were divided into training set and test set in a ratio of 5:1, that is, there were 1028 lung proton MRI images and 1028 corresponding lung hyperpolarization images in the training set. 129 XeMRI images, in the test set there are 257 lung proton MRI images and 257 corresponding lung hyperpolarization 129 XeMRI images, and the two modal images correspond to each other in the training set and the test set. Then the samples in the training set and the test set are preprocessed by data normalization, cropping, and uniform image size to ensure that the lung proton MRI images in the samples and the corresponding lung hyperpolarization 129 The size of XeMRI images is 256×256 to reduce the differences in multimodal data at the input end.

[0045] Step 2: Construct a multimodal lung MRI image segmentation network based on deep learning and edge information fusion. The multimodal lung MRI image segmentation network includes a segmentation module and an edge information module, such as Figure 2 shown.

[0046] In this example, the segmentation module includes 1 encoder and 1 decoder;

[0047] The encoder consists of 1 convolutional neural network (CNN), 1 feature fusion block, and 12 Transformer blocks.

[0048] The decoder consists of 4 upsampling modules and an output layer.

[0049] The edge information module includes 3 convolution blocks and 3 edge detection blocks.

[0050] The edge information module takes the lung proton MRI image in the sample as the input of the module; when the data is input into the edge information module, the relevant feature image is first extracted through the first convolution block. The convolution kernel size in the first convolution block is 3×3, the input feature image size is 256×256, the number of input channels is 16, the output feature image size is 128×128, and the number of output channels is 64; then the feature image output by the first convolution block is input into the first edge detection block, and the extracted edge feature image is obtained through Sobel convolution, normalization, activation function, and 3×3 convolution kernel respectively. The edge feature image output by the first edge detection block has a size of 128×128 and an output channel number of 64.

[0051] The feature image output by the first convolution block is input into the second convolution block to extract the relevant feature image. The convolution kernel size in the second convolution block is 3×3, the input feature image size is 128×128, the number of input channels is 64, the output feature image size is 64×64, and the number of output channels is 128; then the feature image output by the second convolution block is input into the second edge detection block, and the extracted edge feature image is obtained through Sobel convolution, normalization, activation function, and 3×3 convolution kernel. The edge feature image size output by the second edge detection block is 64×64, and the number of output channels is 128.

[0052] The feature image output by the second convolution block is input into the third convolution block to extract the relevant feature image. The convolution kernel size in the third convolution block is 3×3, the input feature image size is 64×64, the number of input channels is 128, the output feature image size is 32×32, and the number of output channels is 256; then the feature image output by the third convolution block is input into the third edge detection block, and the extracted edge feature image is obtained through Sobel convolution, normalization, activation function, and 3×3 convolution kernel. The edge feature image size output by the third edge detection block is 32×32, and the number of output channels is 256.

[0053] The encoder consists of 1 convolutional neural network, 1 feature fusion block, and 12 Transformer blocks in order. 129When the XeMRI images are input into the convolutional neural network, they first pass through an input layer, which includes a convolution layer, a normalization layer, an activation layer, and a maximum pooling layer. The input channel number of this input layer is 3, the output channel number is 16, the input feature image size is 256×256, and the output feature image size is 128×128. The feature image output by the input layer is input into the three convolution modules designed for the convolutional neural network, and three feature images of different scales can be obtained; first, the feature image output by the input layer is input into the first convolution module, and passes through a convolution layer with a convolution kernel size of 1×1, a step size of 1, and a number of 16, a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a number of 16, and a convolution layer with a convolution kernel size of 1×1, a step size of 1, and a number of 64. The feature image size input by the first convolution module is 128×128, the number of input channels is 16, and the output feature image size is 128×128, and the number of output channels is 64; then the feature image output by the first convolution module is input into the second convolution module, and passes through a convolution layer with a convolution kernel size of 1×1, a step size of 1, and a number of 32, and a convolution layer A convolutional layer with a kernel size of 3×3, a stride of 2, and a number of 32 convolutional layers, a convolutional kernel size of 1×1, a stride of 1, and a number of 128 convolutional layers, the second convolutional module inputs a feature image size of 128×128, the number of input channels is 64, the output feature image size is 64×64, and the number of output channels is 128; finally, the feature image output by the second convolutional module is input into the third convolutional module, and passes through a convolutional layer with a convolutional kernel size of 1×1, a stride of 1, and a number of 64, a convolutional layer with a convolutional kernel size of 3×3, a stride of 2, and a number of 64, and a convolutional layer with a convolutional kernel size of 1×1, a stride of 1, and a number of 256 convolutional layers. The feature image size of the third convolutional module input is 64×64, the number of input channels is 128, the output feature image size is 32×32, and the number of output channels is 256.

[0054] The feature fusion block includes convolutional layers and activation layers, which combine the lung proton MRI images in the sample with the corresponding lung hyperpolarization 129 The characteristic images obtained after the XeMRI images are processed by the encoder are used to obtain the lung proton MRI images and the corresponding lung hyperpolarization images in the sample through matrix multiplication. 129The relationship between each pixel in the XeMRI image is then obtained through a softmax activation layer to obtain the fused feature image. The output fused feature image size is 16×16 and the number of output channels is 512. The obtained fused feature image is then passed through a convolution layer with a convolution kernel size of 1×1 to convert the input feature image into a one-dimensional vector with a size of 256×512. The one-dimensional vector is then input into the Transformer block and repeated 12 times to output a one-dimensional feature vector with a size of 256×512.

[0055] The decoder part includes 4 upsampling layers and 1 output layer. The 4 upsampling layers are the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer.

[0056] The first upsampling layer includes an upsampling module, a convolution layer, and an activation layer. The one-dimensional feature vector output by the last Transformer block is reshaped to obtain a feature image with a size of 16×16 and an output channel number of 512. After passing through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer, it is used as the input of the upsampling module in the first upsampling layer. The upsampling method used by the upsampling module in the first upsampling layer is bilinear interpolation. Therefore, the input feature image size of the first upsampling layer is 16×16, the output feature image size is 32×32, the number of input channels is 512, and the number of output channels is 256.

[0057] The second upsampling layer includes a fusion module, an upsampling module, a convolution layer and an activation layer; the feature images of two modes respectively output by the third convolution module designed in the encoder convolutional neural network, the edge feature image output by the third edge detection block in the edge information module and the feature image output by the first upsampling layer are used as the input of the fusion module in the second upsampling layer. The fusion module stacks the above four input feature images according to channels, and performs multi-modal feature image fusion through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer. The fused feature image is used as the input of the upsampling module in the second upsampling layer for upsampling operation. The upsampling method adopted by the upsampling module in the second upsampling layer is bilinear interpolation. After the feature image after the upsampling operation passes through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer, the output feature image of the second upsampling layer is obtained; therefore, the input feature image size of the second upsampling layer is 32×32, the output feature image size is 64×64, the number of input channels is 256, and the number of output channels is 128;

[0058] The third upsampling layer includes a fusion module, an upsampling module, a convolution layer and an activation layer; the feature images of the two modes respectively output by the second convolution module designed in the encoder convolutional neural network, the edge feature image output by the second edge detection block in the edge information module and the feature image output by the second upsampling layer are used as the input of the fusion module in the third upsampling layer. The fusion module stacks the above four input feature images according to channels, and performs multi-modal feature image fusion through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer. The fused feature image is used as the input of the upsampling module in the third upsampling layer for upsampling operation. The upsampling method adopted by the upsampling module in the third upsampling layer is bilinear interpolation. After the feature image after the upsampling operation passes through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer, the output feature image of the third upsampling layer is obtained; therefore, the input feature image size of the third upsampling layer is 64×64, the output feature image size is 128×128, the number of input channels is 128, and the number of output channels is 64;

[0059] The fourth upsampling layer includes a fusion module, an upsampling module, a convolution layer and an activation layer; the feature images of the two modes output by the first convolution module designed in the encoder convolutional neural network, the edge feature image output by the first edge detection block in the edge information module and the feature image output by the third upsampling layer are used as the input of the fusion module in the fourth upsampling layer. The fusion module stacks the above four feature images according to the channel, and performs multi-modal feature image fusion through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer. The fused feature image is used as the input of the upsampling module in the fourth upsampling layer for upsampling operation. The upsampling method adopted by the upsampling module in the fourth upsampling layer is bilinear interpolation. The feature image after the upsampling operation is passed through a convolution layer with a convolution kernel size of 3×3 and a ReLU activation layer to obtain the output feature image of the fourth upsampling layer; therefore, the input feature image size of the fourth upsampling layer is 128×128, the output feature image size is 256×256, the number of input channels is 64, and the number of output channels is 16; finally, the feature image output by the fourth upsampling layer is input to the final output layer, and then passes through a 3×3 convolution layer and a ReLU activation layer in the final output layer to output the lung hyperpolarization. 129 Segmented image of the lung contour in a XeMRI image.

[0060] Step 3: Set the total loss function L. As mentioned above, the total loss function L is:

[0061] L=-(y t log(y p )+(1-y t )log(1-y p))

[0062] The binary cross entropy loss is used during training, where y t Hyperpolarization of the lungs 129 The label of the XeMRI image sample, y p Represents the probability that the predicted lung contour segmentation image obtained by the multimodal lung MRI image segmentation network is a positive example if the lung is hyperpolarized 129 If the XeMRI image sample is a positive example, then the value is 1, otherwise, the value is 0; during the training process, the gradient of the multimodal lung MRI image segmentation network is calculated through the sample label and the generated segmentation image, and the parameters of the multimodal lung MRI image segmentation network are updated through gradient feedback.

[0063] Step 4. According to the total loss function set in step 3, the multimodal lung MRI image segmentation network constructed in step 2 is trained using the training set generated in step 1. The learning rate of the network is initialized to 0.0001, the batch size is set to 2, and the network is trained on the PyTroch platform using the Adam optimizer. The training is stopped after the total number of iterations reaches 150, and the corresponding parameters of the multimodal lung MRI image segmentation network are saved.

[0064] Step 5: Use the multimodal lung MRI image segmentation network trained in step 4 to hyperpolarize the multimodal image data in the test set generated in step 1 or the lung to be segmented. 129 XeMRI images are segmented to obtain the final lung hyperpolarization 129 Segmentation image of the lung contour of the XeMRI image.

[0065] The proposed method is compared with other lung contour segmentation methods. Figure 4 As shown in the figure, various methods are compared with labels, and the Dice coefficient is calculated. The results show that the average Dice obtained by using the U-Net segmentation network is 0.958, the average Dice obtained by the TransUNet segmentation network is 0.953, and the average Dice obtained by the Swin Trans segmentation network is 0.942. The average Dice coefficient obtained by the method proposed in the present invention is 0.965, which is the highest average Dice coefficient among the various segmentation methods. Therefore, the present invention provides a multimodal lung MRI image segmentation method by deep learning and fusion of edge information, which effectively utilizes the edge information in multimodal data and lung proton MRI images, and improves lung hyperpolarization. 129 Accuracy of XeMRI image segmentation.

[0066] A multimodal lung MRI image segmentation device based on deep learning and edge information fusion, comprising:

[0067] An image preprocessing module is used to implement the above step 1 to obtain a training set and a test set;

[0068] A multimodal lung MRI image segmentation network module, used to construct the multimodal lung MRI image segmentation network and total loss function in step 2 above;

[0069] The training module is used to implement the above step 4 and train the multimodal lung MRI image segmentation network based on the training set.

[0070] A terminal comprises a processor and a storage medium;

[0071] The storage medium is used to store instructions;

[0072] The processor is configured to operate according to the instructions to execute steps 2-5.

[0073] A computer-readable storage medium stores a computer program, which implements steps 2-5 when executed by a processor.

[0074] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A multimodal lung MRI image segmentation method based on deep learning and edge information fusion, characterized in that: The following steps are involved: Step 1: Label the lung proton MRI images and associate the lung proton MRI images with the corresponding lung hyperpolarization 129 XeMRI images were used as samples to construct the training set; Step 2: Construct a multimodal lung MRI image segmentation network; Step 3: Set the total loss function L; Step 4: According to the total loss function L, the multimodal lung MRI image segmentation network is trained using the training set, and the parameters of the multimodal lung MRI image segmentation network are saved; Step 5: Use the trained multimodal lung MRI image segmentation network to process the lung hyperpolarization 129 XeMRI images are segmented to obtain the final lung hyperpolarization 129 Segmented image of the lung contour in XeMRI images, The step 1 also includes the operations of normalizing the data and unifying the image size of the samples. The multimodal lung MRI image segmentation network includes a segmentation module and an edge information module. The segmentation module includes an encoder and a decoder; The encoder includes a convolutional neural network, a feature fusion block, and a Transformer block; The decoder includes an upsampling layer and an output layer; The edge information module includes a convolution block and an edge detection block. The edge information module includes 3 convolution blocks and 3 edge detection blocks; The lung proton MRI image in the sample is input into the first convolution block. The feature image output by the first convolution block is input into the second convolution block and the first edge detection block respectively. The feature image output by the second convolution block is input into the third convolution block and the second edge detection block respectively. The feature image output by the third convolution block is input into the third edge detection block. The convolutional neural network includes an input layer and a first convolutional module, a second convolutional module, and a third convolutional module arranged in sequence. The lung proton MRI image and lung hyperpolarization in the sample 129 XeMRI images were input into the convolutional neural network to obtain lung proton MRI images and lung hyperpolarization 129 The characteristic image corresponding to the XeMRI image, The feature fusion block includes a convolution layer and an activation layer. Proton MRI images of the lungs and lung hyperpolarization in the sample 129 The characteristic images obtained after the XeMRI images are processed by the encoder are subjected to matrix multiplication to obtain the lung proton MRI images and the corresponding lung hyperpolarization in the sample. 129 The relationship between each pixel in the XeMRI image is obtained by the activation layer to obtain a fused feature image, and the fused feature image is then passed through a convolution layer with a convolution kernel size of 1×1 to obtain a one-dimensional vector, which is input into multiple consecutive Transformer blocks to obtain a one-dimensional feature vector.

2. According to the multimodal lung MRI image segmentation method of deep learning fusion edge information according to claim 1, it is characterized in that: The decoder part includes 4 upsampling layers and 1 output layer. The feature image obtained by reshaping the one-dimensional feature vector is used as the input of the first upsampling layer. In the first upsampling layer, the feature image is processed by the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the first upsampling layer. The two-modal feature images output by the third convolution module, the edge feature images output by the third edge detection block, and the feature images output by the first upsampling layer are used as the input of the second upsampling layer. In the second upsampling layer, the four input feature images are stacked according to the channels, and then multi-modal feature images are fused through the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the second upsampling layer. The two-modal feature images output by the second convolution module, the edge feature images output by the second edge detection block, and the feature images output by the second upsampling layer are used as the input of the third upsampling layer. In the third upsampling layer, the four input feature images are stacked according to the channels, and then multi-modal feature images are fused through the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the third upsampling layer. The two-modal feature images output by the first convolution module, the edge feature image output by the first edge detection block, and the feature image output by the third upsampling layer are used as the input of the fourth upsampling layer. In the fourth upsampling layer, the four input feature images are stacked according to the channel, and then the multi-modal feature images are fused through the convolution layer and the activation layer, and then input into the upsampling module for upsampling to obtain the output of the fourth upsampling layer. The feature image output by the fourth upsampling layer is input to the final output layer, and then passes through the convolution layer and ReLU activation layer in the final output layer to output the lung hyperpolarization. 129 Segmented image of the lung contour in a XeMRI image.

3. According to claim 2, the multimodal lung MRI image segmentation method based on deep learning and edge information fusion is characterized in that: The total loss function L is based on the following formula: L=-(and t log(y p )+(1-and t )log(1-y p )) Among them, y t Hyperpolarization of the lungs 129 The label of the XeMRI image sample, y p Represents the probability that the predicted segmented image of the lung contour obtained by the multimodal lung MRI image segmentation network is a positive example.

Images