A magnetic resonance image enhancement method based on multi-task collaborative network
By constructing a magnetic resonance image enhancement method of multi-task collaborative network and combining a cluster structure convolutional neural network, the problems of high equipment requirements and poor enhancement effects in the existing technology are solved, and high-quality automation enhancement of 3T MRI images are achieved , reduces costs and improves diagnosis and treatment efficiency.
Patent Information
- Application Number
- CN202110237526.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-03
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2041-03-03
AI Technical Summary
The existing magnetic resonance image enhancement methods have high requirements for equipment and are not effective, so they cannot automatically complete magnetic resonance image enhancement. The 7T MRI equipment is expensive and limited in quantity, which limits the improvement of image quality.
Using the magnetic resonance image enhancement method based on multi-task collaborative network, the contrast enhancement, edge enhancement and signal-to-noise ratio enhancement module are constructed, combined with a convolutional neural network with a cluster structure, and the multi-task enhancement of images is achieved through the feature fusion module, and the model is trained to Minimize the total loss function updates the network parameters.
It effectively improves the contrast, edge clarity and signal-to-noise ratio of 3T magnetic resonance images, reduces the patient's diagnosis and treatment costs, realizes image reconstruction with a quality similar to 7T MRI, and has high model training efficiency, which is suitable for automation enhancement tasks.
Smart Images
Figure CN113096028B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of digital image processing methods, and in particular relates to a magnetic resonance image enhancement method based on a multi-task collaborative network. Background Art
[0002] Magnetic resonance imaging (MRI) is one of the most important imaging methods in the medical field. It can directly produce cross-sectional, sagittal, coronal and various oblique planes. It has the advantages of high soft tissue contrast, arbitrary azimuth imaging and no ionizing radiation. Therefore, magnetic resonance (MR) imaging is one of the most important medical images in clinical medical auxiliary diagnosis and is widely used in clinical diagnosis and academic research of diseases. The use of magnetic resonance imaging is of great significance for tumor type identification, tumor grading diagnosis and efficacy evaluation, and early diagnosis of neurological or psychiatric diseases such as Alzheimer's disease, epilepsy and depression. Currently, 1.5T and 3T magnetic resonance imaging are the most commonly used in clinical medicine, while 7T magnetic resonance imaging has clearer tissue contrast, which helps to diagnose brain, breast, liver and other diseases more accurately and at an early stage. However, 7T MRI scanners are currently very expensive and are used in small numbers in clinical and research settings. This has prompted researchers to propose a method to reconstruct images with similar quality to 7T MRI from 3T MRI. This method of post-enhancement software enhancement can not only improve image detail recognition, contrast and signal-to-noise ratio, but also reduce the cost of diagnosis and treatment for patients.
[0003] The deep learning-based methods are mainly supervised learning, including the use of 2DCNN and 3DCNN. 3D convolution performs well in terms of tissue boundaries, but cannot restore small voxel value differences and has high requirements for equipment. Considering the limitations of equipment and running time, most methods use 2D convolution. However, in terms of deep network modeling, the existing MRI enhancement models generally follow the network structure of natural image enhancement, ignoring the unique data structure characteristics of MRI images, and there is still a lot of room for improvement in the enhancement effect. The existing interactive MRI image processing software requires professional guidance and adjustment, and the processing results are highly volatile. It is impossible to automatically complete the MRI image enhancement task and the enhancement effect is not ideal. Therefore, the technical research on automatic enhancement of MRI images has broad application prospects and practical value, and has a far-reaching impact on promoting the development of precision medicine and smart medicine based on medical images. Summary of the invention
[0004] One of the purposes of the present invention is to provide a magnetic resonance image enhancement method based on a multi-task collaborative network to solve the problems in the background technology that the existing 7T imaging equipment is expensive and small in number, and the current software method for image enhancement has high equipment requirements and poor enhancement effect.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] A magnetic resonance image enhancement method based on a multi-task collaborative network, the method comprising the following steps:
[0007] Step 1: Build a multi-task enhancement model;
[0008] Step 2: Train the multi-task enhancement model;
[0009] Step 3: Input the magnetic resonance image to be enhanced into the trained multi-task enhancement model to obtain an enhanced magnetic resonance image.
[0010] Preferably, the step 1 comprises the following steps:
[0011] Step 1.1: Set up contrast enhancement module and edge enhancement module respectively;
[0012] Step 1.2: Setting an image difference processing module after the contrast enhancement module and the edge enhancement module;
[0013] Step 1.3: Setting a signal-to-noise ratio enhancement module after the image difference processing module;
[0014] Step 1.4: A feature fusion module is set after the signal-to-noise ratio enhancement module and the edge enhancement module.
[0015] Preferably, the contrast enhancement module comprises a pre-convolutional layer, a dense block, a cluster block and a post-convolutional layer connected in sequence.
[0016] Preferably, the edge enhancement module includes a pre-convolutional layer, a cluster, and a residual block connected in sequence, and the output feature map of the residual block and the output feature map of the pre-convolutional layer are added as input, and convolution is performed through a post-convolutional layer.
[0017] Preferably, the signal-to-noise ratio enhancement module includes a cluster block, a first residual block, and a second residual block connected in sequence. The output feature map of the cluster block, the output feature map of the first residual block, and the output feature map of the second residual block are added as input, and a convolution operation is performed through a post-convolution.
[0018] Preferably, in step 2, training the multi-task enhancement model comprises the following steps:
[0019] Step 2.1: Set the total loss function and network parameters of the multi-task enhancement model;
[0020] Step 2.2: construct a magnetic resonance sample image enhancement dataset, including a number of registration data pairs of sample images to be enhanced x and target sample images y;
[0021] Step 2.3: Use the sample image to be enhanced as the input of the multi-task enhancement model to obtain the output image;
[0022] Step 2.4: Based on the output image and the corresponding target sample image, update the network parameters with the goal of minimizing the loss value of the total loss function.
[0023] Preferably, the total loss function of the multi-task enhancement model is L total =L fusion +a×L edge , L fusion =||F(D(C(x)-E(x))+E(x))-y||1,L edge =||E(x)-y edge ||1, where L fusion represents the regional loss of the output image of the multi-task enhancement model, L edge represents the edge loss of the edge enhancement module, y edge represents the soft edge image extracted from the target sample image, a is a hyperparameter used to control the edge loss ratio, C(x) represents the contrast enhanced image obtained by the contrast enhancement module C() of the sample image to be enhanced, E(x) represents the edge enhanced image obtained by the edge enhancement module E() of the sample image to be enhanced, C(x)-E(x) represents the edge reduced image obtained by the contrast enhanced image C(x) and the edge enhanced image E(x) through the image difference processing module, D(C(x)-E(x)) represents the signal-to-noise ratio enhanced image obtained by the signal-to-noise ratio enhancement module D() of the edge reduced image, and F(D(C(x)-E(x))+E(x)) represents the output image of the multi-task enhancement model obtained by the feature fusion module F() of the signal-to-noise ratio enhanced image and the edge enhanced image.
[0024] Preferably, the sample image to be enhanced is a 3T plain scan magnetic resonance image, and the target sample image is a 7T magnetic resonance image or an enhanced magnetic resonance image after injection of a contrast agent.
[0025] Preferably, the soft edge image y edge The curvature formula y is used to extract eage =div(u m ,u n ),in m, n represent the horizontal and vertical coordinates of the target sample image respectively, It represents the gradient of the target sample image y in the horizontal or vertical direction, div represents the divergence operator, Represents the sum of the square of the horizontal gradient and the square of the vertical gradient of the target sample image y at the current coordinate point.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] (1) To address the problems of low tissue contrast, blurred boundaries, and low signal-to-noise ratio in common 3T MRI and other common MRI imaging, a MRI image enhancement model with three modules, namely collaborative contrast enhancement, edge enhancement, and signal-to-noise ratio enhancement, was constructed. The information interaction between the three modules was utilized to effectively improve the enhancement effect of MRI images.
[0028] (2) Different from the existing deep learning models based on convolutional neural networks or dense neural networks, the three module networks of the method of the present invention are combined with the clique structure with higher parameter efficiency in the clique network (CliqueNet), so that there are both forward connections and feedback connections between convolutional layers, which improves the detail preservation ability of deep network information transmission and improves the convergence speed of the loss function value in model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a flow chart of the present invention.
[0030] Figure 2 This is the structural diagram of the contrast enhancement module.
[0031] Figure 3 This is the structural diagram of the edge enhancement module.
[0032] Figure 4 This is the structural diagram of the signal-to-noise ratio enhancement module.
[0033] Figure 5 Schematic diagram of the structure of the cluster.
[0034] Figure 6 It is the effect diagram of the present invention. DETAILED DESCRIPTION
[0035] The present invention is further described in detail below in conjunction with examples and drawings, but the protection scope of the present invention is not limited thereto.
[0036] Figure 1 The flowchart of the magnetic resonance image enhancement method based on a multi-task collaborative network of the present invention includes the following three steps:
[0037] Step 1: Build a multi-task enhancement model, which includes 4 sub-steps:
[0038] Step 1.1: Set up contrast enhancement module and edge enhancement module respectively;
[0039] Step 1.2: Setting an image difference processing module after the contrast enhancement module and the edge enhancement module;
[0040] Step 1.3: Setting a signal-to-noise ratio enhancement module after the image difference processing module;
[0041] Step 1.4: A feature fusion module is set after the signal-to-noise ratio enhancement module and the edge enhancement module.
[0042] Step 2: Training the multi-task enhancement model, including 4 sub-steps:
[0043] Step 2.1: Set the total loss function and network parameters of the multi-task enhancement model;
[0044] Step 2.2: construct a magnetic resonance sample image enhancement dataset, including a number of registration data pairs of sample images to be enhanced x and target sample images y;
[0045] Step 2.3: Use the sample image to be enhanced as the input of the multi-task enhancement model to obtain the output image;
[0046] Step 2.4: Based on the output image and the corresponding target sample image, update the network parameters with the goal of minimizing the loss value of the total loss function.
[0047] Step 3: Input the magnetic resonance image to be enhanced into the trained multi-task enhancement model to obtain an enhanced magnetic resonance image.
[0048] The goal of the edge enhancement module is to learn the mapping function from the sample image to be enhanced to the edge of the target sample image, and reconstruct the clear edge of the high-quality target image from the input low-quality image. The goal of the signal-to-noise ratio enhancement module is to learn the mapping function from the low-SNR input image to the high-SNR target image, and realize the denoising of the input image.
[0049] In the present invention, the magnetic resonance image to be enhanced is processed by a contrast enhancement module to obtain an image with improved visual contrast, and is simultaneously processed by an edge enhancement module to obtain an enhanced edge image close to the edge of the target sample image. The signal-to-noise ratio of the edge-reduced image is improved by a signal-to-noise ratio enhancement module, and then the main content of the image is fused with the enhanced edge information by a feature fusion module to obtain the output image of the multi-task enhancement model.
[0050] In step 1.1, the contrast enhancement module and the edge enhancement module are set as follows:
[0051] A contrast enhancement module is constructed, including a pre-convolutional layer, a dense block, a cluster block and a post-convolutional layer connected in sequence.
[0052] In the contrast enhancement module of this embodiment, the dense block includes 6 convolutional layers, the cluster includes 4 convolutional layers, and the pre-convolutional layer, the convolutional layer in the dense block, the convolutional layer in the cluster, and the post-convolutional layer all use a 5*5 convolution kernel, and the activation function is ReLU.
[0053] A dense block inputs the features of each layer to all subsequent layers. In this embodiment, the feature map obtained by the first convolutional layer will be used as the input of the second convolutional layer. The feature map obtained by the first convolutional layer will be connected with the feature map output by the second convolutional layer in the channel dimension, and then used as the input of the third convolutional layer; the feature map obtained by the first convolutional layer and the feature map obtained by the second convolutional layer will be connected with the feature map obtained by the third convolutional layer in the channel dimension, and then used as the input of the fourth convolutional layer. And so on, the feature maps obtained by the first five convolutional layers will be connected in the channel dimension as the input of the sixth convolutional layer.
[0054] Then, an input image is first convolved through the pre-convolution layer to obtain the feature map C, which is used as the input of the dense block, and then through 6 densely connected convolution layers to obtain the feature map C′, which is used as the input of the cluster to obtain the feature map C″, and then the post-convolution operation is performed to obtain the feature map C (3) , and finally the feature map C is obtained through the activation function (4) .
[0055] The edge enhancement module includes a pre-convolution layer, a cluster, and a residual block connected in sequence. The output feature map of the residual block and the output feature map of the pre-convolution layer are added as input and convolved through a post-convolution layer.
[0056] In the edge enhancement module of this embodiment, the cluster includes 4 convolutional layers, the residual block includes 2 convolutional layers, and the convolutional layers inside the pre-convolutional layer, cluster and residual block all use a 3*3 convolution kernel, and the activation function is ReLU.
[0057] The input image is input to the front convolution layer for convolution operation to obtain the feature map E, which is used as the input of the cluster and output to obtain the feature map E′, which is used as the input of the residual block to obtain the feature map E″, and then the feature map E″ is added to E as the input of the back convolution layer to obtain the feature map E (3) , and finally the feature map E is obtained through the activation function (4) The input image here is the same as the input image in the contrast enhancement module.
[0058] In step 1.3 of the present invention, the signal-to-noise ratio enhancement module specifically includes: a cluster block, a first residual block, and a second residual block connected in sequence, adding the output feature map of the cluster block, the output feature map of the first residual block, and the output feature map of the second residual block as input, and performing a convolution operation through a post-convolution.
[0059] In the signal-to-noise ratio enhancement module of this embodiment, the cluster includes 4 convolutional layers, the first residual block and the second residual block each include 2 convolutional layers, and the convolutional layers and post-convolutional layers of the cluster, the first residual block and the second residual block all use a 3*3 convolution kernel, and the activation function is ReLU.
[0060] The edge-reduced image processed by the difference processing module is used as the input image, and the feature map D is obtained through the clustering. The feature map D is used as the input of the first residual block to obtain the feature map D′. The feature map D′ is used as the input of the second residual block to output the feature map D″. After adding the feature maps D, D′ and D″, a convolution operation is performed through post-convolution to obtain the feature map D (3) , and then the feature map D is obtained through the activation function (4) .
[0061] In step 1.4 of this embodiment, the feature fusion module is a convolutional layer with a kernel size of 1*1.
[0062] Any residual block mentioned in the present invention obtains a first feature map through a first convolution layer, uses the first feature map as input of a second convolution layer to obtain a second feature map, and adds the first feature map and the second feature map as an output feature map of the residual block.
[0063] like Figure 5 As shown, any cluster mentioned in the present invention is a block in the CliqueNet algorithm. In this block, each layer is both the input and output of other layers. Taking a cluster in the embodiment including 4 convolutional layers as an example, it includes two stages. The operation of the first stage is similar to that of DenseNet. The output and input image of the first 1-1 layer are used as the input of the lth layer to update the corresponding weights to obtain the latest layer: the input image X0 of the cluster is output through the first convolutional layer. The feature map The image X0 and the feature map As the input of the second convolutional layer, the output feature map Image X0, feature map And feature map As the input of the third convolutional layer, the output feature map And so on; in the second stage, the output of the other three layers is used as the input of another layer to update the corresponding weights: the output of the 2nd to 4th convolutional layers in the first stage As the input of the first convolutional layer after updating the weights, input the corresponding feature map The output of the 1st, 3rd, and 4th convolutional layers in the first stage As the input of the second convolutional layer after updating the weights, the corresponding features are output Figure, and so on.
[0064] In step 2, training the multi-task enhancement model specifically includes the following steps:
[0065] Step 2.1: Set the total loss function and network parameters of the multi-task enhancement model. The total loss function is L total =L fusion +a×L edge , L fusion =||F(D(C(x)-E(x))+E(x))-y||1,L edge =||E(x)-y edge ||1, where L fusion represents the regional loss of the output image of the multi-task enhancement model, L edge represents the edge loss of the edge enhancement module, y edge represents the soft edge image extracted from the target sample image, a is a hyperparameter used to control the edge loss ratio, C(x) represents the contrast enhanced image obtained by the contrast enhancement module C() of the sample image to be enhanced, E(x) represents the edge enhanced image obtained by the edge enhancement module E() of the sample image to be enhanced, C(x)-E(x) represents the edge reduced image obtained by the image difference processing module of the contrast enhanced image C(x) and the edge enhanced image E(x), D(C(x)-E(x)) represents the signal-to-noise ratio enhanced image obtained by the signal-to-noise ratio enhancement module D() of the edge reduced image, and F(D(C(x)-E(x))+E(x)) represents the output image of the multi-task enhancement model obtained by the feature fusion module F() of the signal-to-noise ratio enhanced image and the edge enhanced image.
[0066] The network parameters include learning rate, etc., which are common knowledge to ordinary technicians in this field, and those skilled in the art can set them according to actual conditions.
[0067] In this embodiment, the hyperparameter a used to control the edge loss ratio in the total loss function is set to 0.2, and the initial learning rate is 10 -4 .
[0068] In this embodiment, the soft edge image of the target sample image is extracted using the curvature formula y edge =div(u m ,u n ),in m, n represent the horizontal and vertical coordinates of the target sample image respectively, Represents the gradient of the target sample image y in the horizontal or vertical direction, that is, when i = m Represents the horizontal gradient value of the target sample image y at the current coordinate point, when i = n It represents the longitudinal gradient value of the target sample image y at the current coordinate point, div represents the divergence operator, Represents the sum of the square of the horizontal gradient and the square of the vertical gradient of the target sample image y at the current coordinate point.
[0069] Step 2.2: Construct a magnetic resonance sample image enhancement data set, including several registration data pairs of sample images to be enhanced x and target sample images y; the sample images to be enhanced can be 3T magnetic resonance plain scan images, and the target sample images can be 7T magnetic resonance images or enhanced magnetic resonance images after contrast agent injection.
[0070] In the present invention, the magnetic resonance sample image enhancement data set is divided into a training set and a test set, wherein the number of registered data pairs in the training set accounts for 70%, and the number of registered data pairs in the test set accounts for 30%.
[0071] In this embodiment, 15 registration data pairs are used, each of which includes a 3T magnetic resonance image and a 7T magnetic resonance image. Each registration data pair is linearly registered in the MIN standard space, and the data is normalized to the range of [0, 1].
[0072] In this embodiment, a leave-one-out cross-validation method is used, and one registered data pair in the test set is used for testing, four registered data pairs in the test set are used for validation, and nine registered data pairs in the training set are used for training.
[0073] In this embodiment, the Adam optimizer is used to update the network parameters. During training, the learning rate is halved every 100 epochs until the total loss function is minimized to complete the training, and the optimal network parameters including the corresponding learning rate are obtained. This is a conventional technical means in the field, and those skilled in the art can set it according to actual conditions.
[0074] The enhanced 3T image obtained by the embodiment of the present invention is as follows Figure 6 shown.
[0075] The multi-task enhancement model proposed by the method of the present invention mainly includes a contrast enhancement module, an edge enhancement module and a signal-to-noise ratio enhancement module. Each sub-module network can realize the enhancement function of a specific task. The overall model can realize end-to-end training and has good interpretability.
Claims
1. A magnetic resonance image enhancement method based on a multi-task collaborative network, characterized in that: The method comprises the following steps: Step 1: Build a multi-task enhancement model; Step 2: Train the multi-task enhancement model; Step 3: Input the magnetic resonance image to be enhanced into the trained multi-task enhancement model to obtain an enhanced magnetic resonance image; The step 1 comprises the following steps: Step 1.1: respectively setting a contrast enhancement module and an edge enhancement module; the contrast enhancement module comprises a pre-convolutional layer, a dense block, a cluster, and a post-convolutional layer connected in sequence; the edge enhancement module comprises a pre-convolutional layer, a cluster, and a residual block connected in sequence, adding an output feature map of the residual block and an output feature map of the pre-convolutional layer as input, and performing convolution through a post-convolutional layer; Step 1.2: Setting an image difference processing module after the contrast enhancement module and the edge enhancement module; Step 1.3: a signal-to-noise ratio enhancement module is set after the image difference processing module; the signal-to-noise ratio enhancement module includes a cluster block, a first residual block, and a second residual block connected in sequence, and the output feature map of the cluster block, the output feature map of the first residual block, and the output feature map of the second residual block are added as input, and a convolution operation is performed through a post-convolution; Step 1.4: A feature fusion module is set after the signal-to-noise ratio enhancement module and the edge enhancement module.
2. A magnetic resonance image enhancement method based on a multi-task collaborative network as claimed in claim 1, characterized in that: In step 2, training the multi-task enhancement model includes the following steps: Step 2.1: Set the total loss function and network parameters of the multi-task enhancement model; Step 2.2: construct a magnetic resonance sample image enhancement dataset, including a number of registration data pairs of sample images to be enhanced x and target sample images y; Step 2.3: Use the sample image to be enhanced as the input of the multi-task enhancement model to obtain the output image; Step 2.4: Based on the output image and the corresponding target sample image, update the network parameters with the goal of minimizing the loss value of the total loss function.
3. A magnetic resonance image enhancement method based on a multi-task collaborative network as claimed in claim 2, characterized in that: The sample image to be enhanced is a 3T plain scan magnetic resonance image, and the target sample image is a 7T magnetic resonance image or an enhanced magnetic resonance image after injection of contrast agent.
4. The magnetic resonance image enhancement method based on a multi-task collaborative network as claimed in claim 2, characterized in that: The total loss function of the multi-task enhancement model is L total =L fusion +a×L edge , L fusion =||F(D(C(x)-E(x))+E(x))-y||1,L edge =||E(x)-y edge ||1, where L fusion represents the regional loss of the output image of the multi-task enhancement model, L edge represents the edge loss of the edge enhancement module, y edge represents the soft edge image extracted from the target sample image, a is a hyperparameter used to control the edge loss ratio, C(x) represents the contrast enhanced image obtained by the contrast enhancement module C() of the sample image to be enhanced, E(x) represents the edge enhanced image obtained by the edge enhancement module E() of the sample image to be enhanced, C(x)-E(x) represents the edge reduced image obtained by the image difference processing module of the contrast enhanced image C(x) and the edge enhanced image E(x), D(C(x)-E(x)) represents the signal-to-noise ratio enhanced image obtained by the signal-to-noise ratio enhancement module D() of the edge reduced image, and F(D(C(x)-E(x))+E(x)) represents the output image of the multi-task enhancement model obtained by the feature fusion module F() of the signal-to-noise ratio enhanced image and the edge enhanced image.
5. The magnetic resonance image enhancement method based on a multi-task collaborative network as claimed in claim 4, characterized in that: The soft edge image y edge The curvature formula y is used to extract edge =div(u m ,u n ),in m, n represent the horizontal and vertical coordinates of the target sample image respectively, It represents the gradient of the target sample image y in the horizontal or vertical direction, div represents the divergence operator, Represents the sum of the square of the horizontal gradient and the square of the vertical gradient of the target sample image y at the current coordinate point.