DTI super-resolution reconstruction method based on two-stage convolutional neural network

Abstract

The invention provides a DTI super-resolution reconstruction method based on a two-stage convolutional neural network, and belongs to the field of medical image processing. Firstly, an image is preprocessed, and a data set is constructed; then building a two-stage convolutional neural network, inputting the DTI in the training set into the one-stage convolutional neural network, and inputting the output DTI tensor feature value and the feature vector into the two-stage convolutional neural network for training to obtain a learned convolutional neural network model; and inputting the test DTI into the convolutional neural network model to obtain a reconstructed high-resolution image. According to the method, feature vector distortion is optimized by adopting a two-stage convolutional neural network, the shape and the direction of a reconstructed DTI tensor are ensured to be consistent with those of a real high-resolution DTI, and meanwhile, a reconstructed DTI black invalid region is kept pure; according to the method, the high-quality super-resolution DTI is generated, and the accuracy of clinical diagnosis can be improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. Background Art

[0002] In the field of neuroimaging, traditional super-resolution reconstruction of diffusion tensor magnetic resonance images (DT-MRI, DTI), referred to as super-resolution, relies on spatial interpolation or model-based reconstruction methods. Although these methods improve image resolution to a certain extent, they are often limited by low computational efficiency, insufficient noise resistance and sensitivity to initial conditions, and are difficult to meet the requirements of high precision and robustness.

[0003] Super-resolution technology based on convolutional neural networks is an effective super-resolution method. The existing DTI super-resolution based on convolutional neural networks first uses the interpolation method to enlarge the low-resolution image to the target resolution, and then uses a complex convolutional network for nonlinear mapping, and finally outputs a high-resolution DTI. However, this method is prone to a series of problems such as DTI tensor expansion and tensor non-positive definiteness, and it cannot guarantee that the shape and direction of the tensor after super-resolution are consistent with the real high-resolution DTI tensor. In addition, only the part of the brain DTI within the brain area is valid data, and the rest is black invalid data. The traditional super-resolution model cannot well maintain the clarity of the boundaries of the brain area of ​​the image and the absence of noise in the black invalid area. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes a DTI super-resolution reconstruction method based on a two-stage convolutional neural network; first, the image is preprocessed and a data set is constructed; then a two-stage convolutional neural network is built, the DTI in the training set is input into the first-stage convolutional neural network, and the output DTI tensor eigenvalues ​​and eigenvectors are input into the two-stage convolutional neural network for training to obtain a learned convolutional neural network model; then the test DTI is input into the convolutional neural network model to finally obtain a reconstructed super-resolution image. The present invention uses a two-stage convolutional neural network to optimize the eigenvector distortion, ensure that the shape and direction of the reconstructed DTI tensor are consistent with the true high-resolution DTI, and keep the black invalid area of ​​the super-resolution reconstructed DTI pure; it helps to improve the accuracy of clinical diagnosis.

[0005] The present invention proposes a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. The specific steps of the technical solution adopted are as follows:

[0006] Step 1: Select the developing Human Connectome Project (dHCP) dataset as the training dataset, test dataset, and migration training dataset;

[0007] Step 2: Preprocess the diffusion image in the dHCP dataset to obtain DTI;

[0008] Step 3: Construct a convolutional neural network model: The convolutional neural network model includes the first-stage convolutional neural network and the second-stage convolutional neural network;

[0009] Step 4: Train the convolutional neural network model;

[0010] Step 5: Test the convolutional neural network model;

[0011] Step 6: Comprehensively evaluate the super-resolution reconstructed DTI.

[0012] Further, in step 2, the pretreatment process is as follows:

[0013] Step 2.1, use the BET module in FSL tools to remove the skull from the diffusion image in the dHCP dataset;

[0014] Step 2.2, using the EDDY module to perform eddy current distortion correction and object motion correction on the diffuse image after removing the skull;

[0015] Step 2.3, using the dtifit command gradient file in the FDT module, perform fitting calculations on the diffusion tensor after skull removal and the corrected diffusion image to obtain the diffusion tensor eigenvalues ​​and eigenvectors;

[0016] Step 2.4, using the fsl_to_dtitk command in the DTI-TK software, the diffusion tensor eigenvalues ​​and eigenvectors are merged into DTI; the DTI is a low-resolution DTI and a high-resolution DTI.

[0017] Further, in step 3, the convolutional neural network model construction method is as follows:

[0018] Step 3.1, construct the first stage convolutional neural network: the first stage convolutional neural network includes two branches and three convolutional layers; of the two branches, the first branch includes a convolutional layer 1, four residual layers and a combination layer 1, and the combination layer 1 is a convolutional layer 2 plus an upsampling layer; the second branch includes a convolutional layer 3 and an upsampling layer; in the first branch, the output of the convolutional layer 1 and the output of the four residual layers are superimposed as the input of the convolutional layer 2; the outputs of the first branch and the second branch are superimposed and input to the three convolutional layers;

[0019] Step 3.2, construct the second stage convolutional neural network: the second stage convolutional neural network includes a convolutional layer 4, a combination layer 2 and a convolutional layer 5, the combination layer 2 includes three residual layers and a convolutional layer 6; the output of the convolutional layer 4 and the output of the convolutional layer 6 are superimposed and input into the convolutional layer 5;

[0020] Step 3.3, use the output of the first stage convolutional neural network as the input of the second stage convolutional neural network to obtain the convolutional neural network model.

[0021] Further, in step 4, the training steps of the convolutional neural network model are as follows;

[0022] Step 4.1: Input the training data set into the first stage convolutional neural network;

[0023] Step 4.2: Use the absolute value loss function as loss1 to calculate the error between the super-resolution reconstructed DTI and the high-resolution DTI output by the first-stage convolutional neural network;

[0024] Step 4.3: Use the Adam gradient optimization algorithm to update the network parameters of the first stage convolutional neural network;

[0025] Step 4.4: Decompose the DTI tensor output by the first-stage convolutional neural network into eigenvalues ​​and eigenvectors, and input them into the second-stage convolutional neural network;

[0026] Step 4.5: Using the L1 loss function as loss2, the eigenvector absolute cosine loss function as loss3, the tensor matrix loss function as loss4, the eigenvalue-eigenvector overlap error loss function as loss5, and the fractional anisotropy (FA) image loss function as loss6, calculate the error between the reconstructed DTI and the high-resolution DTI output by the second-stage convolutional neural network;

[0027] Step 4.6: Use the Adam gradient optimization algorithm to update the network parameters of the second-stage convolutional neural network;

[0028] Step 4.7: Repeat steps 4.1 to 4.6 for 20 rounds to complete the pre-training of the convolutional neural network model;

[0029] Step 4.8: Repeat steps 4.1 to 4.6, and perform 5 rounds of migration training on the convolutional neural network model obtained in step 4.7 by migrating the training data set to obtain a trained convolutional neural network model.

[0030] Further, in step 5, the testing steps of the convolutional neural network model are as follows:

[0031] Step 5.1: Input the low-resolution DTI in the test dataset into the convolutional neural network model trained in step 4, and output the super-resolution reconstructed DTI;

[0032] Step 5.2: Visually observe and compare the super-resolution reconstructed DTI and the high-resolution DTI to determine the clarity of the super-resolution reconstructed DTI and the surrounding black noise;

[0033] Step 5.3: Calculate eight indicators of super-resolution reconstructed DTI, including Peak Signal-to-Noise Ratio PSNR, Structural Similarity (SSIM), Tensor Euclidean Distance (TED), Deviatoric Tensor Euclidean Distance (DVED), Cross Correlation (CC), Overlap of eigenvalue eigenvectors (OVL), Coherence of primary eigenvectors (COH), and Cosine Similarity (CS).

[0034] Furthermore, the calculation method of the 8 indicators is as follows:

[0035] The peak signal-to-noise ratio: The peak signal-to-noise ratio is the ratio of the maximum power of the signal to the noise power of the signal, and is used to evaluate the quality of the processed image. In the super-resolution reconstruction task, the peak signal-to-noise ratio is a common evaluation indicator. The peak signal-to-noise ratio represents the logarithmic value of the ratio of the possible maximum pixel value of the image to the mean square error between the synthetic image and the real image, in dB. The peak signal-to-noise ratio calculation formula is as shown in formula (1):

[0036]

[0037] Among them, MAX is the maximum pixel value of the image, MSE is the average of the sum of squares of the pixel differences between the synthetic data and the real data; the larger the PSNR value, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality;

[0038] The structural similarity: structural similarity is a commonly used evaluation standard for measuring the similarity between images. Compared with the traditional error summation method, structural similarity is similar to the quality perception of the human visual perception system. The evaluation standard of structural similarity calculates the similarity of DTI in three aspects: image contrast, brightness and structure. It is a more comprehensive image similarity evaluation standard. The calculation formula of structural similarity is as shown in formula (2):

[0039]

[0040] in, is the brightness information of the two images, is the contrast information of the two images, is the structural comparison information of the two images, An element value of the tensor matrix output by the convolutional neural network model. is an element value on the upper diagonal of the real high-resolution DTI tensor matrix, and the value range of y is [1,6]; α, β, and γ are the weights of brightness information, contrast information, and structure contrast information, respectively, and the default values ​​of the three parameters are 1; the larger the value of SSIM, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality;

[0041] Tensor Euclidean distance: The tensor Euclidean distance between diffusion tensors is used to measure the closeness between the super-resolution reconstructed DTI and the true high-resolution DTI. The smaller the tensor Euclidean distance, the closer the spatial positions between the images are, and the better the DTI super-resolution reconstruction effect is. The tensor Euclidean distance is defined as the Frobenius norm of the difference between two diffusion tensor matrices. The calculation formula of the tensor Euclidean distance is as follows (3):

[0042]

[0043] Among them, ||·|| F represents the Frobenius norm of the matrix, D sr The DTI tensor matrix calculated for the output data of the convolutional neural network model, D hr is the real high-resolution DTI tensor matrix, i and j represent the coordinates of the corresponding values ​​in the tensor matrix, and tr represents the trace operation;

[0044] Partial tensor Euclidean distance: Partial tensor Euclidean distance is also used to measure the closeness between the super-resolution reconstructed DTI and the true high-resolution DTI. The smaller the partial tensor Euclidean distance, the closer the spatial positions between the images are, and the better the DTI super-resolution reconstruction effect is. The calculation formula of partial tensor Euclidean distance is as follows (4):

[0045]

[0046] in, Represents the partial tensor of the real image DTI tensor matrix, Represents the partial tensor of the high-resolution DTI tensor matrix;

[0047] Cross-correlation: The cross-correlation between FA images corresponding to DTI (FA cross-correlation for short). The FA cross-correlation calculation formula is as follows (5):

[0048]

[0049] Among them, FA sr (v) and FA hr (v) represents the FA value at coordinate v of the FA image corresponding to the DTI synthesized by the output data of the convolutional neural network model and the FA image corresponding to the real high-resolution DTI;

[0050] The cross-correlation between the Tr images corresponding to the DTI (abbreviated as Tr cross-correlation) is calculated as follows:

[0051]

[0052] Among them, Tr sr (v) and Tr hr (v) represents the Tr value at coordinate v of the Tr image corresponding to the DTI synthesized by the output data of the convolutional neural network model and the Tr image corresponding to the real high-resolution DTI; the greater the correlation between the images, the higher the matching degree between the images and the better the DTI super-resolution reconstruction effect;

[0053] Eigenvalue-eigenvector overlap ratio: The calculation formula of the eigenvalue-eigenvector overlap ratio between DTI images is as follows (7):

[0054]

[0055] in, and represents the kth eigenvalue corresponding to a pair of diffusion tensor matrices, and Represents the kth eigenvector corresponding to a pair of diffusion tensor matrices; the greater the eigenvalue-eigenvector overlap ratio between DTIs, the greater the overlap between images, the higher the alignment of white matter regions, and the better the super-resolution reconstruction effect of the convolutional neural network model;

[0056] Main eigenvector coherence: The main eigenvector coherence between DTIs is calculated as follows (8):

[0057]

[0058] Among them, τ1, τ2 and τ3 are the main eigenvalue, secondary eigenvalue and third eigenvalue of the average vector tensor, and τ1>τ2>τ3; the main eigenvector coherence is a parameter that represents the coherence relationship between DTIs. The greater the main eigenvector coherence between the super-resolution reconstructed DTI and the true high-resolution DTI, the closer the super-resolution reconstructed DTI is to the true high-resolution DTI, the higher the degree of alignment of the white matter regions between the images, and the better the super-resolution reconstruction effect of DTI;

[0059] Feature vector cosine similarity: Feature vector cosine similarity is a measure used to represent the feature vector similarity of two DTI data. The calculation formula of feature vector cosine similarity is as follows (9):

[0060]

[0061] in, represents the eigenvector corresponding to the super-resolution reconstructed DTI data, x represents the corresponding coordinate, y represents the number of the eigenvector, and the value range of x and y is [1,3]. Represents the eigenvector corresponding to the original high-resolution DTI data; the higher the cosine similarity of the eigenvector, the better the DTI super-resolution reconstruction effect.

[0062] Beneficial effects of the present invention:

[0063] (1) The DTI super-resolution reconstruction method of the present invention optimizes the existing convolutional neural network model, and in the two-stage convolutional neural network, the original DTI is not directly fed into the convolutional neural network model for training, but the eigenvalues ​​and eigenvectors describing the shape and direction of the DTI tensor are input into the convolutional neural network model for training. The obtained convolutional neural network model can comprehensively tune the eigenvalues ​​and eigenvectors of the DTI tensor obtained by the one-stage convolutional neural network, and finally complete the super-resolution reconstruction of the whole brain DTI;

[0064] (2) The present invention can not only ensure the positive definiteness of the tensor and the orthogonality of the eigenvector, but also make full use of the feature conversion characteristics and the map-guided characteristics of the convolutional neural network, effectively ensure the consistency of the shape and direction of the DTI tensor after super-resolution reconstruction with the real high-resolution DTI tensor and the purity of the black invalid area of ​​the image, which helps to improve the accuracy of clinical diagnosis;

[0065] (3) Compared with the existing methods, the reconstruction method of the present invention achieves better reconstruction effect. Intuitively, the recovered detail information and structural information are closer to the real high-resolution DTI. In addition, it is better than the existing methods in terms of eight quantitative indicators, including peak signal-to-noise ratio, structural similarity, tensor Euclidean distance, partial tensor Euclidean distance, cross-correlation, eigenvalue-eigenvector overlap rate, main eigenvector coherence and eigenvector cosine similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the present invention;

[0067] Figure 2 This is a network structure diagram of DTI super-resolution reconstruction based on a two-stage convolutional neural network of the present invention;

[0068] Figure 3 It is the specific internal structure of the residual layer;

[0069] Figure 4 It is the change of the loss function of the first stage with the number of training times during the training process;

[0070] Figure 5 It is the change of the second stage loss function with the number of training times during the training process;

[0071] Figure 6 It is the example result and comparison of super-resolution reconstruction by the method of the present invention;

[0072] Figure 6 In the figure, (a) is a low-resolution DTI, (b) is a high-resolution DTI, (c) is a super-resolution reconstruction result image by the interpolation method, (d) is a super-resolution result image by the existing super-resolution reconstruction CNN (SRCNN) method, and (e) is a super-resolution result image by the method of the present invention. DETAILED DESCRIPTION

[0073] The present invention proposes a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. The specific steps of the technical solution adopted are as follows:

[0074] like Figure 1 As shown,

[0075] Step 1: Preprocessing DTI: The DTI is low-resolution DTI and high-resolution DTI;

[0076] Step 2: Constructing datasets: Select the developing Human Connectome Project (dHCP) dataset as the training dataset, test dataset, and migration training dataset;

[0077] Step 3: construct a convolutional neural network model, wherein the convolutional neural network model includes a first-stage convolutional neural network and a second-stage convolutional neural network;

[0078] Step 4: Train the convolutional neural network model;

[0079] Step 5: Test the convolutional neural network model;

[0080] Step 6: Comprehensively evaluate the super-resolution reconstructed DTI.

[0081] like Figure 2 and Figure 3 As shown,

[0082] Furthermore, in step 3, the convolutional neural network model construction method is as follows:

[0083] Step 3.1, construct the first stage convolutional neural network: the first stage convolutional neural network includes two branches and three convolutional layers; of the two branches, the first branch includes a convolutional layer 1, four residual layers and a combination layer 1, and the combination layer 1 is a convolutional layer 2 plus an upsampling layer; the second branch includes a convolutional layer 3 and an upsampling layer; in the first branch, the output of the convolutional layer 1 and the output of the four residual layers are superimposed as the input of the convolutional layer 2; the outputs of the first branch and the second branch are superimposed and input to the three convolutional layers;

[0084] Step 3.2, construct the second stage convolutional neural network: the second stage convolutional neural network includes a convolutional layer 4, a combination layer 2 and a convolutional layer 5, the combination layer 2 includes three residual layers and a convolutional layer 6; the output of the convolutional layer 4 and the output of the convolutional layer 6 are superimposed and input into the convolutional layer 5;

[0085] Step 3.3, use the output of the first stage convolutional neural network as the input of the second stage convolutional neural network to obtain the convolutional neural network model.

[0086] Further, in step 4, the training steps of the convolutional neural network model are as follows;

[0087] Step 4.1: Input the training data set into the first stage convolutional neural network;

[0088] Step 4.2: Use the absolute value loss function as loss1 to calculate the error between the super-resolution reconstructed DTI and the high-resolution DTI output by the first-stage convolutional neural network;

[0089] Step 4.3: Use the Adam gradient optimization algorithm to update the network parameters of the first stage convolutional neural network;

[0090] Step 4.4: Decompose the DTI tensor output by the first-stage convolutional neural network into eigenvalues ​​and eigenvectors, and input them into the second-stage convolutional neural network;

[0091] Step 4.5: Using the L1 loss function as loss2, the eigenvector absolute cosine loss function as loss3, the tensor matrix loss function as loss4, the eigenvalue-eigenvector overlap error loss function as loss5, and the fractional anisotropy (FA) image loss function as loss6, calculate the error between the reconstructed DTI and the high-resolution DTI output by the second-stage convolutional neural network;

[0092] Step 4.6: Use the Adam gradient optimization algorithm to update the network parameters of the second-stage convolutional neural network;

[0093] Step 4.7: Repeat steps 4.1 to 4.6 for 20 rounds to complete the pre-training of the convolutional neural network model;

[0094] Step 4.8: Repeat steps 4.1 to 4.6, and perform 5 rounds of migration training on the convolutional neural network model obtained in step 4.7 by migrating the training data set to obtain a trained convolutional neural network model.

[0095] Further, in step 5, the testing steps of the convolutional neural network model are as follows:

[0096] Step 5.1: Input the low-resolution DTI in the test dataset into the convolutional neural network model trained in step 4, and output the super-resolution reconstructed DTI;

[0097] Step 5.2: Determine by visual observation whether the super-resolution reconstructed DTI is clear and whether there are black spots around it, and compare it with the corresponding high-resolution DTI in the test dataset;

[0098] Step 5.3: Calculate eight indicators of super-resolution reconstructed DTI, including Peak Signal-to-Noise Ratio PSNR, Structural Similarity (SSIM), Tensor Euclidean Distance (TED), Deviatoric Tensor Euclidean Distance (DVED), Cross Correlation (CC), Overlap of eigenvalue eigenvectors (OVL), Coherence of primary eigenvectors (COH), and Cosine Similarity (CS).

[0099] Furthermore, the calculation method of the 8 indicators is as follows:

[0100] The peak signal-to-noise ratio: The peak signal-to-noise ratio is the ratio of the maximum power of the signal to the noise power of the signal, and is used to evaluate the quality of the processed image. In the super-resolution reconstruction task, the peak signal-to-noise ratio is a common evaluation indicator. The peak signal-to-noise ratio represents the logarithmic value of the ratio of the possible maximum pixel value of the image to the mean square error between the synthetic image and the real image, in dB. The peak signal-to-noise ratio calculation formula is as shown in formula (1):

[0101]

[0102] Among them, MAX is the maximum pixel value of the image, MSE is the average of the sum of squares of the pixel differences between the synthetic data and the real data; the larger the PSNR value, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality;

[0103] The structural similarity: structural similarity is a commonly used evaluation standard for measuring the similarity between images. Compared with the traditional error summation method, structural similarity is similar to the quality perception of the human visual perception system. The evaluation standard of structural similarity calculates the similarity of DTI in three aspects: image contrast, brightness and structure. It is a more comprehensive image similarity evaluation standard. The calculation formula of structural similarity is as shown in formula (2):

[0104]

[0105] in, is the brightness information of the two images, is the contrast information of the two images, is the structural comparison information of the two images, An element value of the tensor matrix output by the convolutional neural network model. is an element value on the upper diagonal of the real high-resolution DTI tensor matrix, and the value range of y is [1,6]; α, β, and γ are the weights of brightness information, contrast information, and structure contrast information, respectively, and the default values ​​of the three parameters are 1; the larger the value of SSIM, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality;

[0106] Tensor Euclidean distance: The tensor Euclidean distance between diffusion tensors is used to measure the closeness between the super-resolution reconstructed DTI and the true high-resolution DTI. The smaller the tensor Euclidean distance, the closer the spatial positions between the images are, and the better the DTI super-resolution reconstruction effect is. The tensor Euclidean distance is defined as the Frobenius norm of the difference between two diffusion tensor matrices. The calculation formula of the tensor Euclidean distance is as follows (3):

[0107]

[0108] Among them, ||·|| F represents the Frobenius norm of the matrix, D sr The DTI tensor matrix calculated for the output data of the convolutional neural network model, D hr is the real high-resolution DTI tensor matrix, i and j represent the coordinates of the corresponding values ​​in the tensor matrix, and tr represents the trace operation;

[0109] Partial tensor Euclidean distance: Partial tensor Euclidean distance is also used to measure the closeness between the super-resolution reconstructed DTI and the true high-resolution DTI. The smaller the partial tensor Euclidean distance, the closer the spatial positions between the images are, and the better the DTI super-resolution reconstruction effect is. The calculation formula of partial tensor Euclidean distance is as follows (4):

[0110]

[0111] in, Represents the partial tensor of the real image DTI tensor matrix, Represents the partial tensor of the high-resolution DTI tensor matrix;

[0112] Cross-correlation: The cross-correlation between FA images corresponding to DTI (FA cross-correlation for short). The FA cross-correlation calculation formula is as follows (5):

[0113]

[0114] Among them, FA sr (v) and FA hr(v) represents the FA value at coordinate v of the FA image corresponding to the DTI synthesized by the output data of the convolutional neural network model and the FA image corresponding to the real high-resolution DTI;

[0115] The cross-correlation between the Tr images corresponding to the DTI (abbreviated as Tr cross-correlation) is calculated as follows:

[0116]

[0117] Among them, Tr sr (v) and Tr hr (v) represents the Tr value at coordinate v of the Tr image corresponding to the DTI synthesized by the output data of the convolutional neural network model and the Tr image corresponding to the real high-resolution DTI; the greater the correlation between the images, the higher the matching degree between the images and the better the DTI super-resolution reconstruction effect;

[0118] Eigenvalue-eigenvector overlap ratio: The calculation formula of the eigenvalue-eigenvector overlap ratio between DTI images is as follows (7):

[0119]

[0120] in, and represents the kth eigenvalue corresponding to a pair of diffusion tensor matrices, and Represents the kth eigenvector corresponding to a pair of diffusion tensor matrices; the greater the eigenvalue-eigenvector overlap ratio between DTIs, the greater the overlap between images, the higher the alignment of white matter regions, and the better the super-resolution reconstruction effect of the convolutional neural network model;

[0121] Main eigenvector coherence: The main eigenvector coherence between DTIs is calculated as follows (8):

[0122]

[0123] Among them, τ1, τ2 and τ3 are the main eigenvalue, secondary eigenvalue and third eigenvalue of the average vector tensor, and τ1>τ2>τ3; the main eigenvector coherence is a parameter that represents the coherence relationship between DTIs. The greater the main eigenvector coherence between the super-resolution reconstructed DTI and the true high-resolution DTI, the closer the super-resolution reconstructed DTI is to the true high-resolution DTI, the higher the degree of alignment of the white matter regions between the images, and the better the super-resolution reconstruction effect of DTI;

[0124] Feature vector cosine similarity: Feature vector cosine similarity is a measure used to represent the feature vector similarity of two DTI data. The calculation formula of feature vector cosine similarity is as follows (9):

[0125]

[0126] in, represents the eigenvector corresponding to the super-resolution reconstructed DTI data, x represents the corresponding coordinate, y represents the number of the eigenvector, and the value range of x and y is [1,3]. Represents the eigenvector corresponding to the original high-resolution DTI data; the higher the cosine similarity of the eigenvector, the better the DTI super-resolution reconstruction effect.

[0127] Depend on Figure 5 It can be seen that the main fiber bundles of the super-resolution image of the method of the present invention are basically the same as the fiber bundles of the real high-resolution DTI. Compared with other methods, the super-resolution image of the method has clearer edges and no noise, and the super-resolution result is closer to the real high-resolution DTI image.

[0128] Effects of the Invention

[0129] The internal structure of the residual layer includes a convolution layer, a weight normalization, an activation function, a convolution layer and a weight normalization in sequence.

[0130] Taking the two-stage training process as an example, during the training process, the training set error changes with the number of training times as shown in Figure 3 and Figure 4 As shown. The reason why loss functions 4 and 5 increase sharply after the first round of training is that their weights are increased after the first round of training, and then they remain stable during the training process. In general, the training process of the DTI super-resolution method based on the two-stage convolutional neural network proposed in the present invention is relatively stable, without the phenomenon of gradient disappearance.

[0131] Figure 4 The loss function of the first-stage convolutional neural network is learned on the training data set for 10 rounds of loss1 changes. From the change process of the curve, it can be seen that the first-stage network model can stably approach the target data with only one loss function. After 10 rounds of iterative training, the loss gap of the training set is smaller, which shows that the convolutional neural network proposed in the present invention can converge very effectively and quickly, and the fluctuation of the loss is very small.

[0132] Figure 5The loss function change process of the second-stage network model after six iterations on the training set is shown in the curve. It can be seen from the curve change that as the number of iterations increases, each loss function is steadily decreasing. The sharp increase in loss4 and loss5 after the first round of training is because their weights are increased after the first round of training. The sharp increase in the values ​​of these two losses in the second round did not affect the stability of loss2 and loss3. All loss functions did not fluctuate significantly within the design range. Through the change process of the five losses, it can be seen that the second-stage network designed by the present invention can converge very effectively and quickly.

[0133] Nine evaluation indicators including peak signal-to-noise ratio, structural similarity, tensor Euclidean distance, partial tensor Euclidean distance, FA cross-correlation, Tr cross-correlation, eigenvalue-eigenvector overlap rate, main eigenvector coherence and eigenvector cosine similarity are used to describe the similarity between the super-resolution results and the real high-resolution DTI images from different dimensions. The evaluation results are shown in Table 1:

[0134] Table 1 Comparison of three methods and nine evaluation indicators

[0135] Evaluation indicators Method of the present invention Interpolation CNN PSNR 33.699 30.644 32.562 SSIM 0.937 0.923 0.934 TED 0.022 0.033 0.025 DVED 0.019 0.023 0.020 CC(FA) 0.952 0.864 0.921 CC(Tr) 0.999 0.994 0.998 OVL 0.846 0.840 0.844 COH 0.896 0.891 0.892 CS 0.904 0.891 0.900

[0136] It can be seen from the results in Table 1 that, except for the TED and DVED indicators which are slightly lower than the other two comparison methods, the method of the present invention is superior to the comparison methods in most other indicators. Comprehensively considering the nine indicators, the method of the present invention is superior to the comparison methods as a whole, and the obtained super-resolution image results are closer to the real high-resolution DTI results.

Claims

1. A DTI super-resolution reconstruction method based on a two-stage convolutional neural network, characterized in that: The DTI super-resolution reconstruction method is as follows: Step 1: Select the dHCP dataset as the training dataset, test dataset, and migration training dataset; Step 2: Preprocess the diffusion image in the dHCP dataset to obtain DTI; Step 3: Construct a convolutional neural network model: The convolutional neural network model includes the first-stage convolutional neural network and the second-stage convolutional neural network; Step 4: Train the convolutional neural network model; Step 5: Test the convolutional neural network model; Step 6: Comprehensively evaluate the super-resolution reconstructed DTI.

2. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 1, characterized in that: In step 2, the pretreatment process is as follows: Step 2.1, use the BET module in FSL tools to remove the skull from the diffusion image in the dHCP dataset; Step 2.2, using the EDDY module to perform eddy current distortion correction and object motion correction on the diffuse image after removing the skull; Step 2.3, using the dtifit command gradient file in the FDT module, perform fitting calculations on the diffusion tensor after skull removal and the corrected diffusion image to obtain the diffusion tensor eigenvalues ​​and eigenvectors; Step 2.4, using the fsl_to_dtitk command in the DTI-TK software, the diffusion tensor eigenvalues ​​and eigenvectors are merged into DTI; the DTI is a low-resolution DTI and a high-resolution DTI.

3. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 1, characterized in that: In step 3, the convolutional neural network model construction method is as follows: Step 3.1, construct the first stage convolutional neural network: the first stage convolutional neural network includes two branches and three convolutional layers; of the two branches, the first branch includes a convolutional layer 1, four residual layers and a combination layer 1, and the combination layer 1 is a convolutional layer 2 plus an upsampling layer; the second branch includes a convolutional layer 3 and an upsampling layer; in the first branch, the output of the convolutional layer 1 and the output of the four residual layers are superimposed as the input of the convolutional layer 2; the outputs of the first branch and the second branch are superimposed and input to the three convolutional layers; Step 3.2, construct the second stage convolutional neural network: the second stage convolutional neural network includes a convolutional layer 4, a combination layer 2 and a convolutional layer 5, the combination layer 2 includes three residual layers and a convolutional layer 6; the output of the convolutional layer 4 and the output of the convolutional layer 6 are superimposed and input into the convolutional layer 5; Step 3.3, use the output of the first stage convolutional neural network as the input of the second stage convolutional neural network to obtain the convolutional neural network model.

4. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 1, characterized in that: In step 4, the training steps of the convolutional neural network model are as follows; Step 4.1: Input the training data set into the first stage convolutional neural network; Step 4.2: Use the absolute value loss function as loss1 to calculate the error between the super-resolution reconstructed DTI and the high-resolution DTI output by the first-stage convolutional neural network; Step 4.3: Use the Adam gradient optimization algorithm to update the network parameters of the first stage convolutional neural network; Step 4.4: Decompose the DTI tensor output by the first-stage convolutional neural network into eigenvalues ​​and eigenvectors, and input them into the second-stage convolutional neural network; Step 4.5: Using the L1 loss function as loss2, the eigenvector absolute cosine loss function as loss3, the tensor matrix loss function as loss4, the eigenvalue-eigenvector overlap error loss function as loss5, and the anisotropy fractional image loss function as loss6, calculate the error between the reconstructed DTI and the high-resolution DTI output by the second-stage convolutional neural network; Step 4.6: Use the Adam gradient optimization algorithm to update the network parameters of the second-stage convolutional neural network; Step 4.7: Repeat steps 4.1 to 4.6 for 20 rounds to complete the pre-training of the convolutional neural network model; Step 4.8: Repeat steps 4.1 to 4.6, and perform 5 rounds of migration training on the convolutional neural network model obtained in step 4.7 by migrating the training data set to obtain a trained convolutional neural network model.

5. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 1, characterized in that: In step 1, the testing steps of the convolutional neural network model are as follows: Step 5.1: Input the low-resolution DTI in the test dataset into the convolutional neural network model trained in step 4, and output the super-resolution reconstructed DTI; Step 5.2: Visually observe and compare the super-resolution reconstructed DTI and the high-resolution DTI to determine the clarity of the super-resolution reconstructed DTI and the surrounding black noise; Step 5.3: Calculate the peak signal-to-noise ratio, structural similarity, tensor Euclidean distance, partial tensor Euclidean distance, cross-correlation, eigenvalue-eigenvector overlap rate, main eigenvector coherence, and eigenvector cosine similarity of the super-resolution reconstructed DTI.

6. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 5, characterized in that: The calculation methods of the 8 indicators are as follows: The peak signal-to-noise ratio calculation formula is as follows: Among them, MAX is the maximum pixel value of the image, MSE is the average of the sum of squares of the pixel differences between the synthetic data and the real data; the larger the PSNR value, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality; The calculation formula of the structural similarity is as follows: in, is the brightness information of the two images, is the contrast information of the two images, is the structural comparison information of the two images, An element value of the tensor matrix output by the convolutional neural network model. is an element value on the upper diagonal of the real high-resolution DTI tensor matrix, and the value range of y is [1,6]; α, β, and γ are the weights of brightness information, contrast information, and structure contrast information, respectively, and the default values ​​of the three parameters are 1; the larger the value of SSIM, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality; The calculation formula of tensor Euclidean distance is as follows (3): Among them, ||·|| F represents the Frobenius norm of the matrix, D sr The DTI tensor matrix calculated for the output data of the convolutional neural network model, D hr is the real high-resolution DTI tensor matrix, i and j represent the coordinates of the corresponding values ​​in the tensor matrix, and tr represents the trace operation; The calculation formula of the partial tensor Euclidean distance is as follows (4): in, Represents the partial tensor of the real image DTI tensor matrix, Represents the partial tensor of the high-resolution DTI tensor matrix; The calculation formula of the cross-correlation between FA images corresponding to DTI is as follows (5): Among them, FA sr (v) and FA hr (v) represents the FA value at coordinate v of the FA image corresponding to the DTI synthesized by the output data of the convolutional neural network model and the FA image corresponding to the real high-resolution DTI; The calculation formula of the cross-correlation between Tr images corresponding to DTI is as follows (6): Among them, FA sr (v) and Tr hr (v) represents the Tr value at coordinate v of the Tr image corresponding to the DTI synthesized by the output data of the convolutional neural network model and the Tr image corresponding to the real high-resolution DTI; the greater the correlation between the images, the higher the matching degree between the images and the better the DTI super-resolution reconstruction effect; The formula for calculating the eigenvalue-eigenvector overlap ratio between DTI images is as follows (7): in, and represents the kth eigenvalue corresponding to a pair of diffusion tensor matrices, and represents the kth eigenvector corresponding to a pair of diffusion tensor matrices; The calculation formula of the main eigenvector coherence is as follows (8): Among them, τ1, τ2 and τ3 are the main eigenvalue, secondary eigenvalue and third eigenvalue of the average vector tensor, and τ1>τ2>τ3; The calculation formula of feature vector cosine similarity is as follows (9): in, represents the eigenvector corresponding to the super-resolution reconstructed DTI data, x represents the corresponding coordinate, y represents the number of the eigenvector, and the value range of x and y is [1,3]. Represents the feature vector corresponding to the original high-resolution DTI data.

7. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 6.

Images