A DTI super-resolution reconstruction method based on a two-stage convolutional neural network

Abstract

The application provides a DTI super-resolution reconstruction method based on a two-stage convolutional neural network, and belongs to the field of medical image processing. First, an image is preprocessed, and a data set is constructed; then, a two-stage convolutional neural network is built, DTI in a training set is input into a first-stage convolutional neural network, and output DTI tensor eigenvalues and eigenvectors are input into a second-stage convolutional neural network for training, so that a learned convolutional neural network model is obtained; next, test DTI is input into the convolutional neural network model, and a reconstructed high-resolution image is obtained. The two-stage convolutional neural network is adopted to optimize the distortion of the eigenvector, to ensure that the shape and direction of the reconstructed DTI tensor are consistent with the real high-resolution DTI, and to keep the reconstructed DTI black invalid area pure; and the application generates high-quality super-resolution DTI, which is helpful to improve the accuracy of clinical diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. Background Art

[0002] In the field of neuroimaging, traditional super-resolution reconstruction of Diffusion Tensor Magnetic Resonance Image (DT-MRI, DTI), also known as super-resolution, relies on spatial interpolation or model-based reconstruction methods. Although these methods improve image resolution to a certain extent, they are often limited by low computational efficiency, insufficient noise immunity, and sensitivity to initial conditions, making it difficult to meet the requirements of high precision and robustness.

[0003] Super-resolution technology based on convolutional neural networks is an effective super-resolution method. Existing DTI super-resolution based on convolutional neural networks first uses interpolation methods to scale low-resolution images to the target resolution, then uses complex convolutional networks for nonlinear mapping, ultimately outputting a high-resolution DTI. However, this method is prone to a series of problems such as DTI tensor expansion and tensor non-positive definiteness, and cannot guarantee that the shape and direction of the tensor after super-resolution are consistent with the true high-resolution DTI tensor. In addition, only the part of the brain DTI within the brain region is valid data, and the rest is black invalid data. Traditional super-resolution models cannot effectively maintain the clarity of the boundaries of the brain region in the image and the absence of noise in the black invalid area. Summary of the Invention

[0004] In response to the above technical problems, the present invention proposes a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. First, the image is preprocessed and a data set is constructed. Then, a two-stage convolutional neural network is built. The DTI in the training set is input into the first-stage convolutional neural network, and the output DTI tensor eigenvalues ​​and eigenvectors are input into the second-stage convolutional neural network for training to obtain a learned convolutional neural network model. The test DTI is then input into the convolutional neural network model to finally obtain a reconstructed super-resolution image. The present invention uses a two-stage convolutional neural network to optimize eigenvector distortion, ensuring that the shape and direction of the reconstructed DTI tensor are consistent with the true high-resolution DTI, while keeping the black invalid areas of the super-resolution reconstructed DTI pure, which helps to improve the accuracy of clinical diagnosis.

[0005] This paper proposes a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. The specific steps of the technical solution adopted are as follows:

[0006] Step 1: Select the developing Human Connectome Project (dHCP) dataset as the training dataset, test dataset, and transfer training dataset;

[0007] Step 2: Preprocess the diffusion image in the dHCP dataset to obtain DTI;

[0008] Step 3: Construct a convolutional neural network model: The convolutional neural network model includes the first-stage convolutional neural network and the second-stage convolutional neural network;

[0009] Step 4: Train the convolutional neural network model;

[0010] Step 5: Test the convolutional neural network model;

[0011] Step 6: Comprehensively evaluate the super-resolution reconstructed DTI.

[0012] Further, in step 2, the pre-processing process is as follows:

[0013] Step 2.1: Use the BET module in FSL tools to remove the skull from the diffusion image in the dHCP dataset.

[0014] Step 2.2: Use the EDDY module to perform eddy current distortion correction and object motion correction on the diffuse image after removing the skull;

[0015] Step 2.3: Use the dtifit command gradient file in the FDT module to fit the diffusion tensor after skull removal and the corrected diffusion image to obtain the diffusion tensor eigenvalues ​​and eigenvectors.

[0016] In step 2.4, the fsl_to_dtitk command in the DTI-TK software is used to merge the diffusion tensor eigenvalues ​​and eigenvectors into DTI; the DTI is low-resolution DTI and high-resolution DTI.

[0017] Furthermore, in step 3, the convolutional neural network model is constructed as follows:

[0018] Step 3.1: Construct the first-stage convolutional neural network. The first-stage convolutional neural network consists of two branches and three convolutional layers. The first branch consists of a convolutional layer 1, four residual layers, and a combination layer 1, where the combination layer 1 is a convolutional layer 2 plus an upsampling layer. The second branch consists of a convolutional layer 3 and an upsampling layer. In the first branch, the output of convolutional layer 1 and the output of the four residual layers are superimposed as the input of convolutional layer 2. The outputs of the first and second branches are superimposed and input to the three convolutional layers.

[0019] Step 3.2: Construct the second-stage convolutional neural network. The second-stage convolutional neural network includes a convolutional layer 4, a combination layer 2, and a convolutional layer 5. The combination layer 2 includes three residual layers and a convolutional layer 6. The output of convolutional layer 4 and the output of convolutional layer 6 are superimposed and input into convolutional layer 5.

[0020] In step 3.3, the output of the first-stage convolutional neural network is used as the input of the second-stage convolutional neural network to obtain the convolutional neural network model.

[0021] Furthermore, in step 4, the training steps of the convolutional neural network model are as follows;

[0022] Step 4.1: Input the training dataset into the first stage convolutional neural network;

[0023] Step 4.2: Use the absolute value loss function as loss1 to calculate the error between the super-resolution reconstructed DTI and the high-resolution DTI output by the first stage convolutional neural network;

[0024] Step 4.3: Use the Adam gradient optimization algorithm to update the network parameters of the first stage convolutional neural network;

[0025] Step 4.4: Decompose the DTI tensor output by the first-stage convolutional neural network into eigenvalues ​​and eigenvectors, and input them into the second-stage convolutional neural network;

[0026] Step 4.5: Calculate the error between the reconstructed DTI and the high-resolution DTI output by the second-stage convolutional neural network using the L1 loss function as loss2, the eigenvector absolute cosine loss function as loss3, the tensor matrix loss function as loss4, the eigenvalue-eigenvector overlap error loss function as loss5, and the fractional anisotropy (FA) image loss function as loss6.

[0027] Step 4.6: Use the Adam gradient optimization algorithm to update the network parameters of the second-stage convolutional neural network;

[0028] Step 4.7: Repeat steps 4.1 to 4.6 for 20 rounds to complete the pre-training of the convolutional neural network model.

[0029] Step 4.8: Repeat steps 4.1 to 4.6, and perform five rounds of migration training on the convolutional neural network model obtained in step 4.7 by migrating the training dataset to obtain a trained convolutional neural network model.

[0030] Furthermore, in step 5, the testing steps of the convolutional neural network model are as follows:

[0031] Step 5.1: Input the low-resolution DTI in the test dataset into the convolutional neural network model trained in step 4, and output the super-resolution reconstructed DTI;

[0032] Step 5.2: Visually compare the super-resolution reconstructed DTI and the high-resolution DTI to determine the clarity of the super-resolution reconstructed DTI and the surrounding black noise.

[0033] Step 5.3: Calculate eight indicators of super-resolution reconstructed DTI, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Tensor Euclidean Distance (TED), Deviatoric Tensor Euclidean Distance (DVED), Cross Correlation (CC), Overlap of eigenvalue eigenvectors (OVL), Coherence of primary eigenvectors (COH), and Cosine Similarity (CS).

[0034] Furthermore, the calculation method of the 8 indicators is as follows:

[0035] The peak signal-to-noise ratio (PSNR) is the ratio of the maximum power of the signal to the noise power of the signal, and is used to evaluate the quality of the processed image. In the super-resolution reconstruction task, the PSNR is a common evaluation indicator. The PSNR represents the logarithm of the ratio of the possible maximum pixel value of the image to the mean square error between the synthetic image and the real image, in dB. The PSNR calculation formula is as shown in formula (1):

[0036]

[0037] Among them, MAX is the maximum pixel value of the image, MSE is the average of the sum of squared differences between the corresponding pixels of the synthetic data and the real data; the larger the PSNR value, the more similar the output of the convolutional neural network model is to the real data, and the better the image quality;

[0038] The structural similarity: the structural similarity is a commonly used evaluation standard for measuring the similarity between images. Compared with the traditional error summation method, the structural similarity is similar to the quality perception of the human visual perception system; the evaluation standard of the structural similarity calculates the similarity of the DTI from three aspects of image contrast, brightness and structure, which is a relatively comprehensive image similarity evaluation standard; the calculation formula of the structural similarity is as shown in formula (2):

[0039]

[0040] Wherein, is the brightness information of two images, is the contrast information of two images, is the structural contrast information of two images, is an element value of the tensor matrix output by the convolutional neural network model, is an element value of the upper diagonal line of the real high-resolution DTI tensor matrix, the value range of y is [1, 6]; α, β and γ are the weight of brightness information, the weight of contrast information and the weight of structural contrast information respectively, and the default values of the three parameters are 1; the larger the value of SSIM is, the more similar the output result of the convolutional neural network model is to the real data, and the better the image quality is;

[0041] Tensor Euclidean distance: the tensor Euclidean distance between diffusion tensors is used to measure the closeness between the super-resolution reconstructed DTI and the real high-resolution DTI; the smaller the tensor Euclidean distance is, the closer the spatial position between images is, and the better the DTI super-resolution reconstruction effect is; the tensor Euclidean distance is defined as the Frobenius norm of the difference between two diffusion tensor matrices, and the calculation formula of the tensor Euclidean distance is as shown in formula (3):

[0042]

[0043] Wherein, ||·||F F represents the Frobenius norm of the matrix, D sr is the DTI tensor matrix calculated by the output data of the convolutional neural network model, D hr is the real high-resolution DTI tensor matrix, i and j represent the coordinates of the corresponding values in the tensor matrix, and tr represents the trace operation;

[0044] Partial tensor Euclidean distance: the partial tensor Euclidean distance is also used to measure the closeness between the super-resolution reconstructed DTI and the real high-resolution DTI; the smaller the partial tensor Euclidean distance is, the closer the spatial position between images is, and the better the DTI super-resolution reconstruction effect is; the calculation formula of the partial tensor Euclidean distance is as shown in formula (4):

[0045]

[0046] in, Represents the partial tensor of the real image DTI tensor matrix, Represents the partial tensor of the high-resolution DTI tensor matrix;

[0047] Cross-correlation: The cross-correlation between FA images corresponding to DTI (FA cross-correlation for short). The FA cross-correlation calculation formula is as follows (5):

[0048]

[0049] Among them, FA sr (v) and FA hr (v) represents the FA value at coordinate v of the FA image corresponding to the DTI synthesized by the convolutional neural network model output data and the FA image corresponding to the real high-resolution DTI;

[0050] The cross-correlation between the Tr images corresponding to the DTI (abbreviated as Tr cross-correlation) is calculated as follows:

[0051]

[0052] Among them, Tr sr (v) and Tr hr (v) represents the Tr value of the DTI image corresponding to the DTI synthesized by the convolutional neural network model output data and the Tr image corresponding to the real high-resolution DTI at coordinate v; the greater the correlation between the images, the higher the degree of matching between the images and the better the DTI super-resolution reconstruction effect;

[0053] Eigenvalue-eigenvector overlap ratio: The eigenvalue-eigenvector overlap ratio between DTI images is calculated as follows (7):

[0054]

[0055] in, and represents the kth eigenvalue corresponding to a pair of diffusion tensor matrices, and represents the kth eigenvector corresponding to a pair of diffusion tensor matrices; for The greater the eigenvalue-eigenvector overlap ratio between DTIs, the greater the degree of overlap between images, the higher the degree of alignment of white matter regions, and the better the super-resolution reconstruction effect of the convolutional neural network model.

[0056] Principal eigenvector coherence: The principal eigenvector coherence between DTIs is calculated as follows (8):

[0057]

[0058] Among them, τ1, τ2, and τ3 are the principal, secondary, and third eigenvalues ​​of the averaged dyadic tensor, respectively, and τ1>τ2>τ3; the principal eigenvector coherence is a parameter that represents the coherence relationship between DTIs. The greater the principal eigenvector coherence between the super-resolution reconstructed DTI and the true high-resolution DTI, the closer the super-resolution reconstructed DTI is to the true high-resolution DTI, the higher the degree of alignment of the white matter regions between the images, and the better the super-resolution reconstruction effect of DTI;

[0059] Eigenvector cosine similarity: Eigenvector cosine similarity is a measure of the similarity between the eigenvectors of two DTI data. The calculation formula of eigenvector cosine similarity is as follows (9):

[0060]

[0061] in, Represents the eigenvector corresponding to the super-resolution reconstructed DTI data, x represents the corresponding coordinate, y represents the number of the eigenvector, and the value range of x and y is [1,3]. Represents the eigenvector corresponding to the original high-resolution DTI data; the higher the cosine similarity of the eigenvector, the better the DTI super-resolution reconstruction effect.

[0062] Beneficial effects of the present invention:

[0063] (1) The DTI super-resolution reconstruction method of the present invention optimizes the existing convolutional neural network model. In the two-stage convolutional neural network, instead of directly feeding the original DTI into the convolutional neural network model for training, the eigenvalues ​​and eigenvectors describing the shape and direction of the DTI tensor are input into the convolutional neural network model for training. The resulting convolutional neural network model can comprehensively tune the eigenvalues ​​and eigenvectors of the DTI tensor obtained by the first-stage convolutional neural network, ultimately completing the super-resolution reconstruction of the DTI of the entire brain.

[0064] (2) The present invention can not only ensure the positive definiteness of the tensor and the orthogonality of the eigenvectors, but also make full use of the feature conversion characteristics and atlas guidance characteristics of the convolutional neural network, effectively ensuring the consistency of the shape and direction of the DTI tensor after super-resolution reconstruction with the true high-resolution DTI tensor and the purity of the black invalid area of ​​the image, which helps to improve the accuracy of clinical diagnosis;

[0065] (3) Compared with the existing methods, the reconstruction method of the present invention achieves better reconstruction effect. Intuitively, the recovered detail information and structural information are closer to the real high-resolution DTI, and are overall superior to the existing methods in eight quantitative indicators: peak signal-to-noise ratio, structural similarity, tensor Euclidean distance, partial tensor Euclidean distance, cross-correlation, eigenvalue-eigenvector overlap rate, main eigenvector coherence and eigenvector cosine similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the present invention;

[0067] Figure 2 This is a diagram of the DTI super-resolution reconstruction network structure based on a two-stage convolutional neural network of the present invention;

[0068] Figure 3 It is the specific internal structure of the residual layer;

[0069] Figure 4 It is the change of the loss function of the first stage with the number of training times during the training process;

[0070] Figure 5 It is the change of the second stage loss function with the number of training times during the training process;

[0071] Figure 6 This is the example result and comparison of super-resolution reconstruction by the method of the present invention;

[0072] Figure 6 In the figure, (a) is a low-resolution DTI, (b) is a high-resolution DTI, (c) is the super-resolution reconstruction result image of the interpolation method, (d) is the super-resolution result image of the existing super-resolution reconstruction CNN (Super Resolution CNN, SRCNN) method, and (e) is the super-resolution result image of the method of the present invention. DETAILED DESCRIPTION

[0073] This paper proposes a DTI super-resolution reconstruction method based on a two-stage convolutional neural network. The specific steps of the technical solution adopted are as follows:

[0074] like Figure 1 As shown,

[0075] Step 1: Preprocessing DTI: The DTI is low-resolution DTI and high-resolution DTI;

[0076] Step 2: Construct datasets: Select the developing Human Connectome Project (dHCP) dataset as the training dataset, test dataset, and transfer training dataset;

[0077] Step 3: Construct a convolutional neural network model, which includes a first-stage convolutional neural network and a second-stage convolutional neural network;

[0078] Step 4: Train the convolutional neural network model;

[0079] Step 5: Test the convolutional neural network model;

[0080] Step 6: Comprehensively evaluate the super-resolution reconstructed DTI.

[0081] like Figure 2 and Figure 3 As shown,

[0082] Furthermore, in step 3, the convolutional neural network model is constructed as follows:

[0083] Step 3.1: Construct the first-stage convolutional neural network. The first-stage convolutional neural network consists of two branches and three convolutional layers. The first branch consists of a convolutional layer 1, four residual layers, and a combination layer 1, where the combination layer 1 is a convolutional layer 2 plus an upsampling layer. The second branch consists of a convolutional layer 3 and an upsampling layer. In the first branch, the output of convolutional layer 1 and the output of the four residual layers are superimposed as the input of convolutional layer 2. The outputs of the first and second branches are superimposed and input to the three convolutional layers.

[0084] Step 3.2: Construct the second-stage convolutional neural network. The second-stage convolutional neural network includes a convolutional layer 4, a combination layer 2, and a convolutional layer 5. The combination layer 2 includes three residual layers and a convolutional layer 6. The output of convolutional layer 4 and the output of convolutional layer 6 are superimposed and input into convolutional layer 5.

[0085] In step 3.3, the output of the first-stage convolutional neural network is used as the input of the second-stage convolutional neural network to obtain the convolutional neural network model.

[0086] Furthermore, in step 4, the training steps of the convolutional neural network model are as follows;

[0087] Step 4.1: Input the training dataset into the first stage convolutional neural network;

[0088] Step 4.2: Use the absolute value loss function as loss1 to calculate the error between the super-resolution reconstructed DTI and the high-resolution DTI output by the first stage convolutional neural network;

[0089] Step 4.3: Use the Adam gradient optimization algorithm to update the network parameters of the first stage convolutional neural network;

[0090] Step 4.4: Decompose the DTI tensor output by the first-stage convolutional neural network into eigenvalues ​​and eigenvectors, and input them into the second-stage convolutional neural network;

[0091] Step 4.5: Calculate the error between the reconstructed DTI and the high-resolution DTI output by the second-stage convolutional neural network using the L1 loss function as loss2, the eigenvector absolute cosine loss function as loss3, the tensor matrix loss function as loss4, the eigenvalue-eigenvector overlap error loss function as loss5, and the fractional anisotropy (FA) image loss function as loss6.

[0092] Step 4.6: Use the Adam gradient optimization algorithm to update the network parameters of the second-stage convolutional neural network;

[0093] Step 4.7: Repeat steps 4.1 to 4.6 for 20 rounds to complete the pre-training of the convolutional neural network model.

[0094] Step 4.8: Repeat steps 4.1 to 4.6, and perform five rounds of migration training on the convolutional neural network model obtained in step 4.7 by migrating the training dataset to obtain a trained convolutional neural network model.

[0095] Furthermore, in step 5, the testing steps of the convolutional neural network model are as follows:

[0096] Step 5.1: Input the low-resolution DTI in the test dataset into the convolutional neural network model trained in step 4, and output the super-resolution reconstructed DTI;

[0097] Step 5.2: Visually inspect the super-resolution reconstructed DTI to determine if it is clear and whether there are any surrounding black spots. Compare it with the corresponding high-resolution DTI in the test dataset.

[0098] Step 5.3: Calculate eight indicators of super-resolution reconstructed DTI, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Tensor Euclidean Distance (TED), Deviatoric Tensor Euclidean Distance (DVED), Cross Correlation (CC), Overlap of eigenvalue eigenvectors (OVL), Coherence of primary eigenvectors (COH), and Cosine Similarity (CS).

[0099] Furthermore, the calculation method of the 8 indicators is as follows:

[0100] The peak signal-to-noise ratio (PSNR) is the ratio of the maximum power of the signal to the noise power of the signal, and is used to evaluate the quality of the processed image. In the super-resolution reconstruction task, the PSNR is a common evaluation indicator. The PSNR represents the logarithm of the ratio of the possible maximum pixel value of the image to the mean square error between the synthetic image and the real image, in dB. The PSNR calculation formula is as shown in formula (1):

[0101]

[0102] Among them, MAX is the maximum pixel value of the image, MSE is the average of the sum of squared differences between the corresponding pixels of the synthetic data and the real data; the larger the PSNR value, the more similar the output of the convolutional neural network model is to the real data, and the better the image quality;

[0103] The structural similarity: Structural similarity is a commonly used evaluation standard for measuring the similarity between images. Compared with the traditional error summation method, structural similarity is similar to the quality perception of the human visual perception system. The evaluation standard of structural similarity calculates the similarity of DTI in three aspects: image contrast, brightness and structure. It is a more comprehensive image similarity evaluation standard. The calculation formula of structural similarity is as shown in formula (2):

[0104]

[0105] in, is the brightness information of the two images, is the contrast information of the two images, is the structural comparison information of the two images, An element value of the tensor matrix output by the convolutional neural network model, is an element value on the upper diagonal of the true high-resolution DTI tensor matrix, and the value range of y is [1,6]. α, β, and γ are the weights of brightness information, contrast information, and structural contrast information, respectively. The default values ​​of the three parameters are 1. The larger the SSIM value, the more similar the output of the convolutional neural network model is to the real data, and the better the image quality.

[0106] Tensor Euclidean distance: The tensor Euclidean distance between diffusion tensors is used to measure the closeness between the super-resolution reconstructed DTI and the true high-resolution DTI. The smaller the tensor Euclidean distance, the closer the spatial positions between the images are, and the better the DTI super-resolution reconstruction effect. The tensor Euclidean distance is defined as the Frobenius norm of the difference between the two diffusion tensor matrices. The calculation formula of the tensor Euclidean distance is as follows (3):

[0107]

[0108] Among them, ||·|| F represents the Frobenius norm of the matrix, D sr The DTI tensor matrix calculated for the output data of the convolutional neural network model, D hr is the true high-resolution DTI tensor matrix, i and j represent the coordinates of the corresponding values ​​in the tensor matrix, and tr represents the trace operation;

[0109] Partial tensor Euclidean distance: Partial tensor Euclidean distance is also used to measure the closeness between the super-resolution reconstructed DTI and the true high-resolution DTI. The smaller the partial tensor Euclidean distance, the closer the spatial positions between the images are, and the better the DTI super-resolution reconstruction effect. The calculation formula of partial tensor Euclidean distance is as follows (4):

[0110]

[0111] in, Represents the partial tensor of the real image DTI tensor matrix, Represents the partial tensor of the high-resolution DTI tensor matrix;

[0112] Cross-correlation: The cross-correlation between FA images corresponding to DTI (FA cross-correlation for short). The FA cross-correlation calculation formula is as follows (5):

[0113]

[0114] Among them, FA sr (v) and FA hr(v) represents the FA value at coordinate v of the FA image corresponding to the DTI synthesized by the convolutional neural network model output data and the FA image corresponding to the real high-resolution DTI;

[0115] The cross-correlation between the Tr images corresponding to the DTI (abbreviated as Tr cross-correlation) is calculated as follows:

[0116]

[0117] Among them, Tr sr (v) and Tr hr (v) represents the Tr value of the DTI image corresponding to the DTI synthesized by the convolutional neural network model output data and the Tr image corresponding to the real high-resolution DTI at coordinate v; the greater the correlation between the images, the higher the degree of matching between the images and the better the DTI super-resolution reconstruction effect;

[0118] Eigenvalue-eigenvector overlap ratio: The eigenvalue-eigenvector overlap ratio between DTI images is calculated as follows (7):

[0119]

[0120] in, and represents the kth eigenvalue corresponding to a pair of diffusion tensor matrices, and Represents the kth eigenvector corresponding to a pair of diffusion tensor matrices; the greater the eigenvalue-eigenvector overlap ratio between DTIs, the greater the degree of overlap between images, the higher the degree of alignment of white matter regions, and the better the super-resolution reconstruction effect of the convolutional neural network model;

[0121] Principal eigenvector coherence: The principal eigenvector coherence between DTIs is calculated as follows (8):

[0122]

[0123] Among them, τ1, τ2, and τ3 are the principal, secondary, and third eigenvalues ​​of the averaged dyadic tensor, respectively, and τ1>τ2>τ3; the principal eigenvector coherence is a parameter that represents the coherence relationship between DTIs. The greater the principal eigenvector coherence between the super-resolution reconstructed DTI and the true high-resolution DTI, the closer the super-resolution reconstructed DTI is to the true high-resolution DTI, the higher the degree of alignment of the white matter regions between the images, and the better the super-resolution reconstruction effect of DTI;

[0124] Eigenvector cosine similarity: Eigenvector cosine similarity is a measure of the similarity between the eigenvectors of two DTI data. The calculation formula of eigenvector cosine similarity is as follows (9):

[0125]

[0126] in, Represents the eigenvector corresponding to the super-resolution reconstructed DTI data, x represents the corresponding coordinate, y represents the number of the eigenvector, and the value range of x and y is [1,3]. Represents the eigenvector corresponding to the original high-resolution DTI data; the higher the cosine similarity of the eigenvector, the better the DTI super-resolution reconstruction effect.

[0127] Depend on Figure 5 It can be seen that the main fiber bundles of the super-resolution image of the method of the present invention are basically the same as the fiber bundles of the real high-resolution DTI. Compared with other methods, the super-resolution image of this method has clearer edges and no noise, and the super-resolution result is closer to the real high-resolution DTI image.

[0128] Effects of the Invention

[0129] The internal structure of the residual layer includes a convolution layer, a weight normalization, an activation function, a convolution layer and a weight normalization in sequence.

[0130] Taking the two-stage training process as an example, during the training process, the training set error changes with the number of training times as follows: Figure 3 and Figure 4 As shown in Figure 3 , the sharp increase in loss functions 4 and 5 after the first round of training is due to the increase in their weights after the first round of training, but they remain stable during subsequent training. Overall, the DTI super-resolution method based on the two-stage convolutional neural network proposed in this paper undergoes a relatively stable training process, without vanishing gradients or other phenomena.

[0131] Figure 4 The loss function of the first-stage convolutional neural network is learned over 10 rounds on the training dataset. The curve shows that the first-stage network model can steadily approach the target data with only one loss function. After 10 rounds of iterative training, the loss gap on the training set is even smaller, demonstrating that the convolutional neural network proposed in this invention can converge very effectively and quickly, with very little loss fluctuation.

[0132] Figure 5This graph shows the evolution of the loss functions for the second-stage network model after six iterations on the training set. The curves show that each loss function steadily decreases with increasing iterations. The sharp increase in loss4 and loss5 after the first round of training is due to the increase in their weights after the first round of training. The sharp increase in these two losses in the second round did not affect the stability of loss2 and loss3. All loss functions remained within the designed range without significant fluctuations. The evolution of these five losses demonstrates that the second-stage network designed by this invention converges very effectively and quickly.

[0133] Nine evaluation indicators, including peak signal-to-noise ratio, structural similarity, tensor Euclidean distance, partial tensor Euclidean distance, FA cross-correlation, Tr cross-correlation, eigenvalue-eigenvector overlap rate, main eigenvector coherence, and special tensor cosine similarity, are used to describe the similarity between the super-resolution results and the true high-resolution DTI images from different dimensions. The evaluation results are shown in Table 1:

[0134] Table 1 Comparison of three methods and nine evaluation indicators

[0135] Evaluation indicators Method of the present invention Interpolation CNN PSNR 33.699 30.644 32.562 SSIM 0.937 0.923 0.934 TED 0.022 0.033 0.025 DVED 0.019 0.023 0.020 CC(FA) 0.952 0.864 0.921 CC(Tr) 0.999 0.994 0.998 OVL 0.846 0.840 0.844 COH 0.896 0.891 0.892 CS 0.904 0.891 0.900

[0136] The results in Table 1 show that, except for the TED and DVED indicators, the method of the present invention is slightly lower than the other two comparison methods. In terms of most other indicators, the method of the present invention is superior to the comparison methods as a whole. The obtained super-resolution image results are closer to the true high-resolution DTI results.

Claims

1. A DTI super-resolution reconstruction method based on a two-stage convolutional neural network, characterized in that: The DTI super-resolution reconstruction method is as follows: Step 1: Select the dHCP dataset as the training dataset, test dataset, and migration training dataset; Step 2: Preprocess the diffusion image in the dHCP dataset to obtain DTI; Step 3: Construct a convolutional neural network model: The convolutional neural network model includes the first-stage convolutional neural network and the second-stage convolutional neural network; The convolutional neural network model construction method is as follows: Step 3.1: Construct the first-stage convolutional neural network. The first-stage convolutional neural network consists of two branches and three convolutional layers. The first branch consists of a convolutional layer 1, four residual layers, and a combination layer 1, where the combination layer 1 is a convolutional layer 2 plus an upsampling layer. The second branch consists of a convolutional layer 3 and an upsampling layer. In the first branch, the output of convolutional layer 1 and the output of the four residual layers are superimposed as the input of convolutional layer 2. The outputs of the first and second branches are superimposed and input to the three convolutional layers. Step 3.2: Construct the second-stage convolutional neural network. The second-stage convolutional neural network includes a convolutional layer 4, a combination layer 2, and a convolutional layer 5. The combination layer 2 includes three residual layers and a convolutional layer 6. The output of convolutional layer 4 and the output of convolutional layer 6 are superimposed and input into convolutional layer 5. Step 3.3, use the output of the first-stage convolutional neural network as the input of the second-stage convolutional neural network to obtain a convolutional neural network model; Step 4: Train the convolutional neural network model; The training steps of the convolutional neural network model are as follows: Step 4.1: Input the training dataset into the first stage convolutional neural network; Step 4.2: Use the absolute value loss function as loss1 to calculate the error between the super-resolution reconstructed DTI and the high-resolution DTI output by the first stage convolutional neural network; Step 4.3: Use the Adam gradient optimization algorithm to update the network parameters of the first stage convolutional neural network; Step 4.4: Decompose the DTI tensor output by the first-stage convolutional neural network into eigenvalues ​​and eigenvectors, and input them into the second-stage convolutional neural network; Step 4.5: Calculate the error between the reconstructed DTI and the high-resolution DTI output by the second-stage convolutional neural network using the L1 loss function as loss2, the eigenvector absolute cosine loss function as loss3, the tensor matrix loss function as loss4, the eigenvalue-eigenvector overlap error loss function as loss5, and the fractional anisotropy image loss function as loss6. Step 4.6: Use the Adam gradient optimization algorithm to update the network parameters of the second-stage convolutional neural network; Step 4.7: Repeat steps 4.1 to 4.6 for 20 rounds to complete the pre-training of the convolutional neural network model. Step 4.8: Repeat steps 4.1 to 4.6, and perform five rounds of transfer training on the convolutional neural network model obtained in step 4.7 by transferring the training dataset to obtain a trained convolutional neural network model; Step 5: Test the convolutional neural network model; Step 6: Comprehensively evaluate the super-resolution reconstructed DTI.

2. A DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 1, characterized in that: In step 2, the pre-processing process is as follows: Step 2.1: Use the BET module in FSL tools to remove the skull from the diffusion image in the dHCP dataset. Step 2.2: Use the EDDY module to perform eddy current distortion correction and object motion correction on the diffuse image after removing the skull. Step 2.3: Use the dtifit command gradient file in the FDT module to fit the diffusion tensor after skull removal and the corrected diffusion image to obtain the diffusion tensor eigenvalues ​​and eigenvectors. In step 2.4, the fsl_to_dtitk command in the DTI-TK software is used to merge the diffusion tensor eigenvalues ​​and eigenvectors into DTI; the DTI is low-resolution DTI and high-resolution DTI.

3. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 1, characterized in that: In step 5, the testing steps of the convolutional neural network model are as follows: Step 5.1: Input the low-resolution DTI in the test dataset into the convolutional neural network model trained in step 4, and output the super-resolution reconstructed DTI; Step 5.2: Visually compare the super-resolution reconstructed DTI and the high-resolution DTI to determine the clarity of the super-resolution reconstructed DTI and the surrounding black noise. Step 5.3: Calculate the eight indicators of super-resolution reconstructed DTI, including peak signal-to-noise ratio, structural similarity, tensor Euclidean distance, partial tensor Euclidean distance, cross-correlation, eigenvalue-eigenvector overlap rate, main eigenvector coherence, and eigenvector cosine similarity.

4. The DTI super-resolution reconstruction method based on a two-stage convolutional neural network according to claim 3, characterized in that: The calculation methods of the eight indicators are as follows: The peak signal-to-noise ratio calculation formula is as follows: Among them, MAX is the maximum pixel value of the image, MSE is the average of the sum of squared differences between the corresponding pixels of the synthetic data and the real data; the larger the PSNR value, the more similar the output of the convolutional neural network model is to the real data, and the better the image quality; The calculation formula of the structural similarity is as follows: in, is the brightness information of the two images, is the contrast information of the two images, is the structural comparison information of the two images, An element value of the tensor matrix output by the convolutional neural network model, is an element value on the upper diagonal of the true high-resolution DTI tensor matrix, and the value range of y is [1,6]. α, β, and γ are the weights of brightness information, contrast information, and structural contrast information, respectively. The default values ​​of the three parameters are 1. The larger the SSIM value, the more similar the output of the convolutional neural network model is to the real data, and the better the image quality. The calculation formula of tensor Euclidean distance is as follows (3): Among them, ||·|| F represents the Frobenius norm of the matrix, D sr The DTI tensor matrix calculated for the output data of the convolutional neural network model, D hr is the true high-resolution DTI tensor matrix, i and j represent the coordinates of the corresponding values ​​in the tensor matrix, and tr represents the trace operation; The calculation formula of the partial tensor Euclidean distance is as follows (4): in, Represents the partial tensor of the real image DTI tensor matrix, Represents the partial tensor of the high-resolution DTI tensor matrix; The calculation formula for the cross-correlation between FA images corresponding to DTI is as follows (5): Among them, FA sr (v) and FA hr (v) represents the FA value at coordinate v of the FA image corresponding to the DTI synthesized by the convolutional neural network model output data and the FA image corresponding to the real high-resolution DTI; The calculation formula for the cross-correlation between Tr images corresponding to DTI is as follows (6): Among them, Tr sr (v) and Tr hr (v) represents the Tr value of the DTI image corresponding to the DTI synthesized by the convolutional neural network model output data and the Tr image corresponding to the real high-resolution DTI at coordinate v; the greater the correlation between the images, the higher the degree of matching between the images and the better the DTI super-resolution reconstruction effect; The formula for calculating the eigenvalue-eigenvector overlap ratio between DTI images is as follows (7): in, and represents the kth eigenvalue corresponding to a pair of diffusion tensor matrices, and represents the kth eigenvector corresponding to a pair of diffusion tensor matrices; for The transpose of The calculation formula of the main eigenvector coherence is as follows (8): Among them, τ1, τ2 and τ3 are the main eigenvalue, secondary eigenvalue and third eigenvalue of the average vector tensor, and τ1>τ2>τ3; The calculation formula of the feature vector cosine similarity is as follows (9): in, Represents the eigenvector corresponding to the super-resolution reconstructed DTI data, x represents the corresponding coordinate, y represents the number of the eigenvector, and the value range of x and y is [1,3]. Represents the feature vector corresponding to the original high-resolution DTI data.

5. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 4.

Images