A general approach to medical image restoration tasks guided by spatial information

Abstract

The present invention discloses a general method for medical image restoration tasks guided by spatial information. The method first trains a segmentation model based on an improved DenseUNet, and obtains the spatial information of the restoration target through the segmentation model. After segmentation, the spatial information is introduced into the image restoration network, and the introduced spatial information is connected to the features at all levels in the channel dimension to increase the network's sensitivity to the target area and improve the quality of image restoration. Experimental results show that this method can effectively enhance the image restoration effect of the segmented object, especially for targets such as blood vessels and tumors that are easily deformed after image restoration. The invention can also help doctors make diagnoses and greatly reduce their workload.

Description

Technical Field

[0001] The invention relates to a universal medical image restoration task method guided by spatial information, and belongs to the field of computer image processing. Background Art

[0002] Medical image restoration is an important branch of medical image processing. It aims to improve image quality, clarity and contrast by processing and enhancing medical images, thereby improving the visualization of medical images and helping doctors accurately diagnose and treat diseases.

[0003] However, various types of noise often exist in medical images, such as quantum noise accompanying the imaging process and electronic noise of the equipment itself. These noises will blur the detailed information in the image, resulting in a large amount of information loss in the final recovery results of some specific targets (blood vessels, tumors, etc.) during the image restoration process. Because they usually have similar density and contrast to the surrounding tissues, as well as their small and irregular shapes, they often have a large amount of information loss in the final recovery results, which greatly affects the doctor's ability to make a correct diagnosis.

[0004] In recent years, with the continuous advancement of computing, neural network-based methods have been widely used in medical image restoration. The models proposed by these methods can deeply extract image features and have achieved remarkable results in the general field of image restoration. However, most of these methods serve the image as a whole, resulting in oversmoothing or distortion in the restored results, which in turn affects the recognition and analysis of important structures in the image, especially the recognition of specific targets such as blood vessels and tumors. The models proposed by these methods do not provide a reasonable solution to the information loss problem mentioned above. Summary of the Invention

[0005] This paper addresses the problem of information loss of specific targets (blood vessels, tumors, etc.) in medical image restoration and proposes a universal spatial information-guided medical image restoration task method to improve the restoration quality of specific targets. The entire process of this method is as follows Figure 2 shown.

[0006] In order to achieve the above objectives, the technical solution of the present invention is as follows: The present invention provides a general medical image restoration task method guided by spatial information, such as Figure 1 The specific steps are as follows:

[0007] Step 1: Collect low-quality medical image data to be restored and the corresponding high-quality medical image data, and annotate the locations of specific targets (blood vessels, tumors, etc.) in the collected image data. The corresponding low-quality and high-quality image data and the segmentation labels of specific targets are constructed into a benchmark dataset for the spatial information-guided medical image restoration task framework.

[0008] Step 2: Build a medical image segmentation model suitable for a specific target (blood vessels, tumors, etc.) and a medical image restoration model suitable for that image modality;

[0009] Step 3: In the medical image segmentation model established in step 2, the low-quality medical image data collected in step 1 and the annotated segmentation label data are trained to obtain the training weights and predicted segmentation results of the image segmentation model;

[0010] Step 4: In the image restoration model established in step 2, the low-quality image data and high-quality data collected in step 1 are trained, and the predicted segmentation results corresponding to the low-quality images in step 3 are introduced into the model to obtain the training weights of the image restoration model;

[0011] In step 5, real low-quality clinical data is collected and first input into the medical image segmentation model. The medical image segmentation model weights obtained in step 3 are used to obtain the segmentation results of specific targets (such as blood vessels and tumors). Then, the data is input into the medical image restoration model and the training weights obtained in step 4 are used to obtain the image restoration results of the real clinical data.

[0012] In step 1, the high-quality image data collected comes from the 3D-IRCADb public dataset, which contains CT image data of 20 patients and annotates the location information of bones, portal vein, liver, liver tumor, venous system, and arteries. The low-quality image data comes from simulated projection. The TIGRE library is used to simulate the projection of the high-quality image data of 3D-IRCADb, and a 5×10 4 Poisson noise of high intensity is used to simulate the noise in real clinical data. Finally, back-projection is performed to obtain low-quality image data corresponding to the high-quality image data.

[0013] In step 2, the medical image segmentation model used is improved based on DenseUNet, such as Figure 2As shown in the figure, based on the original UNet model, the encoder-decoder structure is adopted. The downsampling module in UNet is replaced by a dense convolution module (DenseBlock), and the upsampling module in UNet is replaced by a residual module (ResBlock). During the encoding process, the features of the input image are encoded into the network using convolutional layers, the encoder is used to extract the image features, and then the feature map is downsampled in the maximum pooling layer, allowing the next encoder to extract deeper image features. Compared with the original upsampling module of UNet, each block of the decoder has a residual connection part, and the output of the encoder of the corresponding layer is received through skip connections. This structure combines local and global features of the image, which can better train the network.

[0014] In step 2, if Figure 4 As shown in (a), the main function of the dense convolution module (DenseBlock) in the model is to increase the reusability of features. The size of the convolution kernel is 3×3. Each dense convolution module contains 3 convolution layers. The specific process is as follows:

[0015] (1) Use the first convolution to operate on the input image I and obtain the result F1;

[0016] (2) Concatenate I and F1 and perform the second convolution operation to obtain the result F2;

[0017] (3) Concatenate I, F1, and F2 and perform the third convolution operation to obtain the result F3;

[0018] The formula used in the calculation process is:

[0019] F i =ReLU([I,F1,…,F i-1 ]),i=1,2,3…,n (1)

[0020] Among them, ReLU(·) is the activation function; i is the i-th convolutional layer; F i is the convolution operation result of the i-th convolutional layer in the dense convolution module; I is the input of the dense convolution module.

[0021] In step 2, if Figure 4 As shown in (b), the main function of the residual module (ResBlock) in the model is to establish local jump connections between each level of convolution in addition to the jump connections between the encoder and decoder. This helps to obtain a smooth loss curve and also helps to avoid gradient vanishing and exploding. The specific process is as follows:

[0022] (1) Concatenate the output feature I1 of the previous decoder layer and the feature I2 obtained by the jump connection in the feature dimension to obtain feature I3;

[0023] (2) Input feature I3 into one convolution layer and two convolution layers to obtain results F1 and F2;

[0024] (3) Perform a residual connection on F1 and F2 and obtain the output through ReLU to obtain the result F3;

[0025] The formula used in the calculation process is:

[0026] F1=ConvBNReLU([I1,I2]) (2)

[0027] F2=ConvBNReLU(ConvBNReLU([I1,I2])) (3)

[0028] F3=ReLU(F1+F2) (4)

[0029] Among them, ReLU(·) is the activation function; ConvBNReLU(·) refers to the continuous operation of Conv+BatchNormalization+ReLu.

[0030] In step 2, the medical image restoration model used is also improved based on DenseUNet, such as Figure 3 As shown in Figure 1, based on the medical image segmentation model, the spatial information of the specific target is introduced. In the encoding process, it can be expressed as:

[0031] Mask=Resize(Predict) (5)

[0032] D i+1 =DenseEncoder([D i ,D i *Mask]) (6)

[0033] Among them, Predict is the predicted segmentation result of the medical image segmentation model; D i is the output result of the i-th dense convolution module in the encoding process; Resize(·) is the nearest neighbor interpolation algorithm; DenseBlock(·) is the dense convolution module.

[0034] During the decoding process, the formula can be expressed as:

[0035] Mask=Resize(Predict) (7)

[0036] U i+1 =ResDecoder([Ui ,U i *Mask],D n-i+1 ]) (8)

[0037] Among them, U i It refers to the output result of the i-th dense convolution module during the decoding process.

[0038] In steps 3 and 4, the medical image segmentation model and the medical image restoration model are trained. The optimizer is Adam, and the loss function used uniformly is the mean square error loss function, which is formulated as follows:

[0039]

[0040] Where n is the number of samples. In the medical image segmentation model, y i is the true label, is the prediction result; in the medical image restoration model, y i is a high-quality image, is the restoration result of the image restoration model.

[0041] Compared with existing technologies, the advantages of this invention are as follows: (1) It can significantly improve the image restoration quality of specific targets. This method was primarily tested on blood vessels, and experiments have shown that this method can significantly improve the image restoration quality of such targets. (2) The method is universal. This method proposes a universal spatial information-guided medical image restoration task method that can be applied to medical image data of various modalities and various organs and lesions. (3) It runs quickly because both the segmentation and restoration algorithms run on the GPU. In actual testing, the segmentation and restoration algorithms combined only took 500ms. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the present invention.

[0043] Figure 2 Schematic diagram of the improved medical image segmentation network structure based on DenseUNet.

[0044] Figure 3 Schematic diagram of the medical image restoration network structure that introduces spatial information.

[0045] Figure 4 Schematic diagram of the structure of DenseEncoder and ResDecoder.

[0046] Figure 5 Visualize the results of the algorithm test.

[0047] Figure 6 This is a comparison chart of algorithm test indicators. DETAILED DESCRIPTION

[0048] The present invention is further illustrated below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0049] Example 1: The present invention provides a general spatial information guided medical image restoration task method, such as Figure 1 The specific steps are as follows:

[0050] Step 1: Collect low-quality medical image data to be restored and the corresponding high-quality medical image data, and annotate the locations of specific targets (blood vessels, tumors, etc.) in the collected image data. The corresponding low-quality and high-quality image data and the segmentation labels of specific targets are constructed into a benchmark dataset for the spatial information-guided medical image restoration task framework.

[0051] Step 2: Build a medical image segmentation model suitable for a specific target (blood vessels, tumors, etc.) and a medical image restoration model suitable for that image modality;

[0052] Step 3: In the medical image segmentation model established in step 2, the low-quality medical image data collected in step 1 and the annotated segmentation label data are trained to obtain the training weights and predicted segmentation results of the image segmentation model;

[0053] Step 4: In the image restoration model established in step 2, the low-quality image data and high-quality data collected in step 1 are trained, and the predicted segmentation results corresponding to the low-quality images in step 3 are introduced into the model to obtain the training weights of the image restoration model;

[0054] In step 5, real low-quality clinical data is collected and first input into the medical image segmentation model. The medical image segmentation model weights obtained in step 3 are used to obtain the segmentation results of specific targets (such as blood vessels and tumors). Then, the data is input into the medical image restoration model and the training weights obtained in step 4 are used to obtain the image restoration results of the real clinical data.

[0055] In step 1, the high-quality image data collected was from the 3D-IRCADb public dataset, which contains CT images of 20 patients and annotates the location information of bones, portal vein, liver, liver tumor, venous system, and arteries. The low-quality image data was obtained from simulated projections. The TIGRE library was used to simulate the projection of the high-quality image data of 3D-IRCADb, and a 5×10 4Poisson noise of high intensity is used to simulate the noise in real clinical data. Finally, back-projection is performed to obtain low-quality image data corresponding to the high-quality image data.

[0056] In step 2, the medical image segmentation model used is improved based on DenseUNet, such as Figure 2 As shown in the figure, based on the original UNet model, the encoder-decoder structure is adopted. The downsampling module in UNet is replaced by a dense convolution module (DenseBlock), and the upsampling module in UNet is replaced by a residual module (ResBlock). During the encoding process, the features of the input image are encoded into the network using convolutional layers, the encoder is used to extract the image features, and then the feature map is downsampled in the maximum pooling layer, allowing the next encoder to extract deeper image features. Compared with the original upsampling module of UNet, each block of the decoder has a residual connection part, and the output of the encoder of the corresponding layer is received through skip connections. This structure combines local and global features of the image, which can better train the network.

[0057] In step 2, if Figure 4 As shown in (a), the main function of the dense convolution module (DenseBlock) in the model is to increase the reusability of features. The size of the convolution kernel is 3×3. Each dense convolution module contains 3 convolution layers. The specific process is as follows:

[0058] (1) Use the first convolution to operate on the input image I and obtain the result F1;

[0059] (2) Concatenate I and F1 and perform the second convolution operation to obtain the result F2;

[0060] (3) Concatenate I, F1, and F2 and perform the third convolution operation to obtain the result F3;

[0061] The formula used in the calculation process is:

[0062] F i =ReLU([I,F1,…,F i-1 ]),i=1,2,3…,n (1)

[0063] Among them, ReLU(·) is the activation function; i is the i-th convolutional layer; F i is the convolution operation result of the i-th convolutional layer in the dense convolution module; I is the input of the dense convolution module.

[0064] In step 2, if Figure 4As shown in (b), the main function of the residual module (ResBlock) in the model is to establish local jump connections between each level of convolution in addition to the jump connections between the encoder and decoder. This helps to obtain a smooth loss curve and also helps to avoid gradient vanishing and exploding. The specific process is as follows:

[0065] (1) Concatenate the output feature I1 of the previous decoder layer and the feature I2 obtained by the jump connection in the feature dimension to obtain feature I3;

[0066] (2) Input feature I3 into one convolution layer and two convolution layers to obtain results F1 and F2;

[0067] (3) Perform a residual connection on F1 and F2 and obtain the output through ReLU to obtain the result F3;

[0068] The formula used in the calculation process is:

[0069] F1=ConvBNReLU([I1,I2]) (2)

[0070] F2=ConvBNReLU(ConvBNReLU([I1,I2])) (3)

[0071] F3=ReLU(F1+F2) (4)

[0072] Among them, ReLU(·) is the activation function;

[0073] ConvBNReLU(·) refers to the continuous operation of Conv+BatchNormalization+ReLu.

[0074] In step 2, the medical image restoration model used is also improved based on DenseUNet, such as Figure 3 As shown in Figure 1, based on the medical image segmentation model, the spatial information of the specific target is introduced. In the encoding process, it can be expressed as:

[0075] Mask=Resize(Predict) (5)

[0076] D i+1 =DenseEncoder([D i ,D i *Mask]) (6)

[0077] Among them, Predict is the predicted segmentation result of the medical image segmentation model; D i is the output result of the i-th dense convolution module in the encoding process; Resize(·) is the nearest neighbor interpolation algorithm; DenseBlock(·) is the dense convolution module.

[0078] During the decoding process, the formula can be expressed as:

[0079] Mask=Resize(Predict) (7)

[0080] U i+1 =ResDecoder([U i ,U i *Mask],D n-i+1 ]) (8)

[0081] Among them, U i It refers to the output result of the i-th dense convolution module during the decoding process.

[0082] In steps 3 and 4, the medical image segmentation model and the medical image restoration model are trained. The optimizer is Adam, and the loss function used uniformly is the mean square error loss function, which is formulated as follows:

[0083]

[0084] Where n is the number of samples. In the medical image segmentation model, y i is the true label, is the prediction result; in the medical image restoration model, y i is a high-quality image, is the restoration result of the image restoration model.

[0085] Example 2: Figure 1 As shown, the present invention provides a general medical image restoration task method guided by spatial information. The final test results are shown in Figure 5 The specific steps are as follows:

[0086] Step 1: 20 images from the 3D-IRCADb public dataset were used as high-quality images, and 5×10 4 Poisson noise images were used as low-quality images, and the vascular annotations in 3D-IRCADb were used as segmentation labels. The corresponding low-quality and high-quality image data and vascular segmentation labels were constructed as a benchmark dataset for the spatial information-guided medical image restoration task framework. The final dataset consisted of 1,653 training sets, 451 validation sets, and 527 test sets.

[0087] Step 2: Based on DenseUNet, build a medical image segmentation model and a medical image restoration model.

[0088] Step 3: In the medical image segmentation model established in step 2, the low-quality medical image data collected in step 1 and the annotated blood vessel segmentation label data are trained. The Adam optimizer is selected as the optimizer, the mean square error is selected as the loss function, and the batch size is selected as 2. The training weights of the image segmentation model and the predicted segmentation results are obtained;

[0089] Step 4: In the image restoration model established in step 2, the low-quality image data and high-quality data collected in step 1 are trained, and the predicted segmentation results corresponding to the low-quality images in step 3 are introduced into the model. Similarly, the Adam optimizer is selected as the optimizer, the mean square error is selected as the loss function, and the batch size is selected as 2 to obtain the training weights of the image restoration model;

[0090] In step 5, real low-quality clinical data is collected and first input into the medical image segmentation model. The medical image segmentation model weights obtained in step 3 are used to obtain the vessel segmentation results. Then, the data is input into the medical image restoration model and the training weights obtained in step 4 are used to obtain the image restoration results of the real clinical data.

[0091] Effect evaluation:

[0092] The present invention discloses a general medical image restoration task method guided by spatial information. The test results of this method are as follows: Figure 5 As shown in the figure, compared with the image restoration using only DenseUNet without introducing spatial information, the restoration results of the framework proposed in this paper can effectively reduce the problems of blood vessel deformation and information loss. The CNR calculated on the blood vessels improved by 0.339, and the PSNR of the entire image improved by 0.21dB. At the same time, the size of the Poisson noise simulated in this paper is 5×10 4 , which is quite severe noise, the proposed framework still successfully restores blood vessels, demonstrating its robustness to noise. Furthermore, the algorithm runs fast, with an average time of approximately 500ms, and achieves good overall image restoration, effectively enhancing the image restoration of segmented objects. Furthermore, this method proposes a general spatial information-guided approach for medical image restoration tasks, applicable to medical image data of various modalities, organs, and lesions.

[0093] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or replacements within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A general spatial information guided medical image restoration method, characterized in that: The steps include: Step 1: Collect low-quality medical image data and corresponding high-quality medical image data to be restored. Annotate the locations of specific targets in the collected image data. Construct the corresponding low-quality and high-quality image data and the segmentation labels of specific targets into a benchmark dataset for the spatial information-guided medical image restoration task framework. Step 2: Build a medical image segmentation model suitable for a specific target and a medical image restoration model suitable for the image modality; Step 3: In the medical image segmentation model established in step 2, the low-quality medical image data collected in step 1 and the annotated segmentation label data are trained to obtain the training weights and predicted segmentation results of the image segmentation model; Step 4: In the image restoration model established in step 2, the low-quality image data and high-quality data collected in step 1 are trained, and the predicted segmentation results corresponding to the low-quality images in step 3 are introduced into the model to obtain the training weights of the image restoration model; Step 5: Collect real low-quality clinical data and first input it into the medical image segmentation model. Use the medical image segmentation model weights obtained in step 3 to obtain the segmentation results of the specific target. Then input it into the medical image restoration model and use the training weights obtained in step 4 to obtain the image restoration results of the real clinical data. Among them, in step 2, the medical image restoration model used is also based on DenseUNet. On the basis of the medical image segmentation model, the spatial information of the specific target is introduced. In the encoding process, in addition to inputting the image features of the previous layer, the prediction mask of the medical image segmentation model is sampled to the same size as the image features of the previous layer through nearest neighbor interpolation, and then point multiplication is performed with the image feature matrix. Finally, the point product result and the image features of the previous layer are feature-connected in the channel dimension and input into the next encoder. In the decoding process, in addition to the output of the previous layer decoder and the output obtained by the skip connection structure of the corresponding layer encoder, the prediction mask of the medical image segmentation model is also sampled to the same size as the image features of the previous layer through nearest neighbor interpolation, and then point multiplication is performed with the image feature matrix. Finally, the feature connection is performed with the image feature matrix and input into the next decoder together with the features obtained by the skip connection. This structure combines the local and global features and spatial information of the image to better train the network. Introducing the spatial information of a specific target, during the encoding process, it is expressed as: Mask=Resize(Predict) (5) D i+1 =DenseEncoder([D i ,D i *Mask]) (6) Among them, Predict is the predicted segmentation result of the medical image segmentation model; Di is the output result of the i-th dense convolution module in the encoding process; Resize(·) is the nearest neighbor interpolation algorithm; DenseBlock(·) is the dense convolution module, In the decoding process, it is expressed as: Mask=Resize(Predict) (7) U i+1 =ResDecoder([U i ,U i *Mask],D n-i+1 ]) (8) Among them, U i It refers to the output result of the i-th dense convolution module during the decoding process.

2. A general spatial information guided medical image restoration method according to claim 1, characterized in that: In step 1, the high-quality image data collected comes from the 3D-IRCADb public dataset, which contains CT image data of 20 patients and annotates the location information of bones, portal vein, liver, liver tumor, venous system, and arteries.

3. A general spatial information guided medical image restoration method according to claim 1, characterized in that: In step 1, the low-quality image data comes from simulated projection. The TIGRE library is used to simulate the projection of the high-quality image data of 3D-IRCADb, and 5×10 4 Poisson noise of high intensity is used to simulate the noise in real clinical data, and finally back-projection is performed to obtain low-quality image data corresponding to the high-quality image data.

4. A general spatial information guided medical image restoration method according to claim 1, characterized in that: In step 2, the medical image segmentation model used is improved based on DenseUNet. It takes the original UNet model as the basis, adopts an encoder-decoder structure, and replaces the downsampling module in UNet with the dense convolution module DenseBlock, and uses the residual module ResBlock to replace the upsampling module in UNet. During the encoding process, the convolution layer is used to encode the features of the input image into the network, the encoder is used to extract the features of the image, and then the feature map is downsampled in the maximum pooling layer so that the next encoder can extract deeper image features. Compared with the original upsampling module of UNet, the decoder has a residual connection part in each block, and accepts the output of the corresponding layer encoder through the skip connection Skip-connection. This structure combines the local and global features of the image to better train the network.

5. A general spatial information guided medical image restoration method according to claim 4, characterized in that: The main function of the dense convolution module DenseBlock in the model is to increase the reusability of features. The size of the convolution kernel is 3×3. Each dense convolution module contains 3 convolution layers. The specific process is as follows: (1) Use the first convolution to operate on the input image I and obtain the result F1; (2) Concatenate I and F1 and perform the second convolution operation to obtain the result F2; (3) Concatenate I, F1, and F2 and perform the third convolution operation to obtain the result F3; The formula used in the calculation process is: F i =ReLU([I,F1,…,F i-1 ]),i=1,2,3…,n (1) Among them, ReLU(·) is the activation function; i is the i-th convolutional layer; F i is the convolution operation result of the i-th convolutional layer in the dense convolution module; i is the input of the dense convolution module.

6. A general spatial information guided medical image restoration method according to claim 4, characterized in that: The main function of the residual module (ResBlock) in the model is to establish local jump connections between the convolution at each level in addition to the jump connections between the encoder and decoder, which helps to obtain a smooth loss curve and also helps to avoid gradient vanishing and exploding. The specific process is as follows: (1) Concatenate the output feature i1 of the previous decoder layer and the feature i2 obtained by the jump connection in the feature dimension to obtain feature i3; (2) Input feature I3 into one convolution layer and two convolution layers to obtain results F1 and F2; (3) Perform a residual connection on F1 and F2 and obtain the output through ReLU to obtain the result F3; The formula used in the calculation process is: F1=ConvBNReLU([I1,I2]) (2) F2=ConvBNReLU(ConvBNReLU([I1,I2])) (3) F3=ReLU(F1+F2) (4) Among them, ReLU(·) is the activation function; ConvBNReLU(·) refers to the continuous operation of Conv+BatchNormalization+ReLu.

7. A general spatial information guided medical image restoration method according to claim 1, characterized in that: In steps 3 and 4, the medical image segmentation model and the medical image restoration model are trained. The optimizer is Adam, and the loss function used uniformly is the mean square error loss function, which is formulated as follows: Where n is the number of samples. In the medical image segmentation model, y i is the true label, is the predicted result; In the medical image restoration model, y i is a high-quality image, is the restoration result of the image restoration model.

Images