An extreme sparse angle THz-CT reconstruction method based on multi-domain multi-scale feature fusion neural network

Abstract

The application belongs to the field of terahertz image processing, and discloses an extreme sparse angle terahertz computed tomography (THz-CT) reconstruction method based on a multi-domain multi-scale feature fusion neural network, which is used to solve the problem that a traditional reconstruction method cannot reconstruct a clear THz-CT image under an extreme sparse sampling angle. The reconstruction method of the application firstly performs sparse sampling on a sinusogram A obtained through terahertz time-domain spectrum acquisition to obtain a sinusogram B and make a mask graph C; then the sinusogram B and the mask graph C are input into a trained sinusogram image processing model to obtain an optimized sinusogram D; subsequently, the fast iterative shrinkage threshold algorithm (FISTA) is used to preliminarily reconstruct the sinusogram D into a CT image E; finally, a trained image domain processing model is used to optimize the reconstructed CT image to obtain a high-quality CT image. The method of the application can realize reconstruction of a THz-CT image under an extreme sparse angle sampling, and the reconstructed CT image can retain rich high-frequency details and edge information.

Description

Technical Field

[0001] This invention relates to a THz-CT image reconstruction method, specifically to an extremely sparse angle THz-CT reconstruction method based on a multi-domain, multi-scale feature fusion neural network. Background Technology

[0002] THz-CT is a non-contact imaging technique based on terahertz waves. As a type of non-ionizing radiation, terahertz waves are far less powerful than X-rays, thus avoiding ionizing damage to biological tissues or industrial materials. This makes it particularly suitable for in vivo detection, enabling safe imaging without harming patients or samples. However, when THz-TDS systems scan large areas of samples using a point-by-point scanning method, the sampling speed is slow, making it difficult to meet the demands of real-time imaging. One effective solution is to reduce the number of viewing angles sampled by the THz-TDS system, performing sparse-view reconstruction. However, because the full angle required for reconstruction is not reached, the reconstructed image produces numerous fringe artifacts, which worsen with decreasing viewing angles. Currently, widely used algorithms for sparse-view reconstruction include filtered back-projection and algebraic reconstruction algorithms. However, as the projection angle decreases, both algorithms generate a large number of artifacts and noise in the reconstructed image, accompanied by beam hardening, leading to a significant deterioration in image quality. Deep learning-based reconstruction algorithms, such as U-net, can simultaneously capture both global and local information of the image, preserving image details and structure while removing noise. However, for extremely sparse images, most networks suffer from severe loss of high-frequency and edge information, resulting in poor reconstructed image quality. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of current technology and provide an extreme sparse angle THz-CT reconstruction method based on a multi-domain, multi-scale feature fusion neural network. The THz-CT reconstruction algorithm can directly reconstruct the sine wave of an extreme sparse viewpoint, and can retain rich high-frequency details and edge information.

[0004] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0005] An extreme sparse angle THz-CT reconstruction method based on a multi-domain, multi-scale feature fusion neural network includes the following steps:

[0006] S1. The detector element of the terahertz time-domain spectroscopy (THz-TDS) system is moved at equal intervals to collect intensity data sine graphs from the corresponding viewing angles and placed into image set A.

[0007] S2, the images in the image set A are further extracted at equal intervals to obtain a set of extremely sparse sampling perspective sinusogram images B and a set of mask images C of the set of sinusogram images B;

[0008] S3, the images in the set of sinusogram images B are first put into a sinusoidal domain processing model to obtain an optimized image set New, then the set of sinusogram images B, the set of mask images C and the image set New are simultaneously input into a data consistency layer, the image set New and the set of mask images C are multiplied to obtain an optimized image of the unsampled part, and finally the optimized image of the unsampled part is added to the images in the set of extremely sparse sampling sinusogram images B to obtain an optimized set of sinusogram images D, and a loss function value between the image set D and the image set A is calculated;

[0009] S4, the FISTA algorithm built in Pytorch is used to initially reconstruct the optimized sinusogram in the image set D into a CT image set E;

[0010] S5, the CT image set E is input into an image domain processing model to obtain an optimized image set Image-After, then the CT image reconstructed from the sinusogram collected in step S1 is put into an image set Image-Label using the FISTA algorithm, and a loss function value between the image set Image-After and the image set Image-Label is calculated, the loss function value is combined with the loss function value in step S3, and the network weight and parameter are updated by back propagation, the optimizer for training is Adam, and when the set number of rounds is reached, the model weight and bias parameter are stopped from being updated, and a trained multi-domain and multi-scale feature fusion neural network is obtained.

[0011] Preferably, in step S1, the intensity data collected at equal intervals corresponding to the perspective is 1 / 5 of the 180 full-angle required in THz-CT reconstruction, i.e. 36.

[0012] Preferably, in step S2, the image set A is further extracted at equal intervals to obtain 1 / 15, 1 / 20, 1 / 30, 1 / 60 of the 180 full-angle required in THz-CT reconstruction, i.e. 12, 9, 6, 3.

[0013] Preferably, in step S3, the sine domain processing model comprises a sine domain image reconstruction network and a data consistency layer; the sine domain image reconstruction network divides the channel number into a sine domain branch and a Fourier sine domain branch; first, the feature map with a channel number of 1 is expanded to 128 through two channel expansions, and then is put into the sine domain image reconstruction network, the sine domain branch comprises multiple convolution layers and a channel spatial attention mechanism, wherein the convolution layer is composed of 4 3x3 convolution blocks; the Fourier sine domain branch uses a fast Fourier transform to extract features of the feature map and is composed of 1 3x3 convolution block; then, the outputs of the two branches are combined in the channel and a residual connection is added; finally, 1 3x3 convolution is used to restore to the input form.

[0014] Preferably, in step S5, the image domain processing model comprises an encoder, a decoder, a shallow extraction module, an image domain reconstruction network and a high-frequency reconstruction layer; the primary encoder down-samples the channel number of the CT graph in the form of 2 N increment; the high-level encoder up-samples the channel in the form of 2 N decrease, and N represents the number of sampling, wherein the CT graph with a primary encoder channel number of 64 captures features of different scales through the shallow extraction module; the image domain reconstruction network is similar to the sine domain image reconstruction network, and is divided into an image domain branch and an image Fourier domain branch, the image domain branch uses a GRU gated convolution unit to enhance the capture ability of the time sequence feature, and the image Fourier domain branch is consistent with the sine domain image reconstruction network; the low-level decoder up-samples the output of the image domain reconstruction network to be consistent with the high-level encoder; the high-level decoder, i.e. the high-frequency reconstruction layer, introduces a Laplace pyramid to extract the outline and details in the image, and fuses the low-frequency feature, the edge feature and the high-frequency feature to reconstruct the CT image with high quality.

[0015] Preferably, the loss function of the training network comprises three, which are as follows: the L1 loss in the sine domain image processing model and the multi-scale structural similarity loss, and the mean square error loss in the image domain processing model. The L1 loss is composed of and the multi-scale structural similarity loss is composed of. The mean square error loss is composed of.

[0016]

[0017] Wherein, y is the true value of the sine graph; is the predicted value of the sine graph; the multi-scale structural similarity loss function is:

[0018] The mean square error loss function is:

[0019]

[0020] Wherein, x is the true value of CT.​​​​ is the CT prediction value;

[0021] The final network loss function is:

[0022]

[0023] wherein z is the network output real value; is the network output prediction value; and both alpha and beta are 0.5.

[0024] Compared with the prior art, the present application has the following beneficial effects:

[0025] 1. The extreme sparse angle THz-CT reconstruction method based on the multi-domain multi-scale feature fusion neural network of the present application integrates the data consistency layer into the sine domain image processing model, ensures the fidelity of the optimized sine graph and the extreme sparse sampling data of the sine domain image reconstruction network, avoids the loss of the original successful sampling data due to excessive optimization of the network, and avoids the introduction of false information in the image generation process to cause distortion.

[0026] 2. The extreme sparse angle THz-CT reconstruction method based on the multi-domain multi-scale feature fusion neural network of the present application extends the dual-domain reconstruction method to the multi-domain reconstruction method, combines the Fourier sine domain and the Fourier image domain in the method of the present application, converts the images of different resolutions from the time domain to the frequency domain by using the fast Fourier transform, extracts the frequency components in the images, and uses the frequency domain information to supplement the high frequency components that cannot be completely captured in the time domain.

[0027] 3. The extreme sparse angle THz-CT reconstruction method based on the multi-domain multi-scale feature fusion neural network of the present application introduces the high frequency information reconstruction strategy into the image domain processing model, fuses the edge information and the high frequency features on multiple scales, and finely processes the edges and the high frequency regions in the last stage of image reconstruction, so as to ensure that the reconstructed image has clear boundaries and rich details. This is particularly important for the identification of microstructures in THz-CT image reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flow chart of the extreme sparse angle THz-CT reconstruction method based on the multi-domain multi-scale feature fusion neural network of the present application.

[0029] Figure 2 The sine domain image processing model schematic diagram of the extreme sparse angle THz-CT reconstruction method based on the multi-domain multi-scale feature fusion neural network of the present application.

[0030] Figure 3 The image domain processing model schematic diagram of the extreme sparse angle THz-CT reconstruction method based on the multi-domain multi-scale feature fusion neural network of the present application. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0032] See Figure 1 The extreme sparse angle THz-CT reconstruction method based on a multi-domain, multi-scale feature fusion neural network of the present invention includes the following steps:

[0033] S1. The detector element of the terahertz time-domain spectroscopy (THz-TDS) system is moved at equal intervals to collect intensity data sine graphs corresponding to the viewing angle and placed into image set A. The image resolution size is 36×367 (projection angle × number of detectors).

[0034] S2. Extract the images in image set A at equal intervals to obtain sine wave image set B with extremely sparse sampling perspective, and create mask image set C for sine wave image set B.

[0035] S3. Input the images from the sinusoidal image set B and the mask image set C into the trained sinusoidal domain image processing model to obtain the optimized sinusoidal image set D;

[0036] S4. Based on the FI STA algorithm built in PyTorch, the optimized sine wave in image set D is initially reconstructed into CT image set E with a resolution of 256×256.

[0037] S5. Input the CT image set E into the trained image domain processing model to obtain the optimized CT images.

[0038] See Figure 1 In step S3, the training of the sinusoidal domain image processing model includes the following steps:

[0039] S3-1. First, put the images in image set B into the sinusoidal domain image reconstruction network to obtain the optimized image set New. Then, simultaneously input the sinusoidal image set B, the mask image set C, and the image set New into the data consistency layer to obtain the image set Sino-After.

[0040] S3-2. Calculate the loss function value between the final output image set Sino-After and image set A of the sinusoidal domain image processing model.

[0041] See Figure 1 In step S5, the training of the image domain processing model includes the following steps:

[0042] S5-1. Using the FISTA algorithm, the CT image reconstructed from the sine wave acquired in S1 is placed into the image set Image-Label;

[0043] S5-2, construct an image domain processing model of a multi-level codec framework, which includes an image domain reconstruction network and a high-frequency reconstruction layer;

[0044] S5-3, calculate the loss function value between the final data image set Image-After of the image domain processing model and the image set A;

[0045] S5-4, combine the loss functions of S3-2 and S5-3 and update the network weights and parameters by back propagation, the optimizer for training is Adam, the initial momentum parameters of the optimizer are set to (0.9, 0.999). The initial learning rate is 0.0001, and the learning rate is decayed by 95% at intervals, the batch size is 2, and the iteration rounds are 50. The iteration number of the sinusoidal domain CT image reconstruction model and the FISTA algorithm is 10. When the set number of rounds is reached, stop updating the model weight and bias parameters to obtain a trained multi-domain multi-scale feature fusion neural network.

[0046] For extreme sparse angle sampling reconstruction of THz-CT images, the intensity data obtained is almost only the global artifact information, and it is difficult to calculate other information except the collected information using traditional iterative algorithms, which will introduce many error information in the process of reconstructing the image, thereby generating a large amount of artifacts and noise. The single-domain network for sparse angle reconstruction lacks data constraints on another domain and has a problem of generating false structures to some extent. The dual-domain reconstruction method is difficult to capture global high-frequency details in the face of extreme sparsity, limiting its ability to recover details.

[0047] Therefore, referring to Figure 2 The sinusoidal domain image processing model proposed in the present example. Among them, the sinusoidal domain image reconstruction network divides the channel number into a sinusoidal domain branch and a Fourier sinusoidal domain branch; first, the feature map with a channel number of 1 is expanded to 128 times through 2 channels, and then put into the sinusoidal domain image reconstruction network, the sinusoidal domain branch includes multiple convolution layers and channel spatial attention mechanisms, wherein the convolution layer is composed of 4 3x3 convolution blocks; the Fourier sinusoidal domain branch uses fast Fourier transform to extract the features of the feature map and uses a 3x3 convolution block; then the outputs of the two branches are combined in the channel and residual connection is added; finally, a 3x3 convolution is used to restore to the input form; the data consistency layer multiplies the optimized sinusoidal graph and the mask image to obtain the optimized image of the unsampled part, which is added to the image in the extreme sparse sampling sinusoidal image set B, and finally the optimized sinusoidal graph of the sinusoidal domain image processing model is obtained.

[0048] Referring to Figure 3 The image domain processing model proposed in the present example. It includes an encoder, a decoder, a shallow extraction module, an image domain reconstruction network and a high-frequency reconstruction layer; the primary encoder down-samples the channel number of the CT image by 2N in the form of increment; the high-level encoder up-sampling channel is in the form of decrement by 2 N , wherein the CT image with 64 primary encoder channels is captured by the shallow extraction module to capture features at different scales; the image domain reconstruction network is similar to the sinusoidal domain image reconstruction network, and is divided into an image domain branch and an image Fourier domain branch; the image domain branch adopts a GRU gated convolution unit to enhance the ability to capture time sequence features; the image Fourier domain branch is consistent with the sinusoidal domain image reconstruction network; the low-level decoder up-samples the output of the image domain reconstruction network to be consistent with the high-level encoder; the high-level decoder, i.e., the high-frequency reconstruction layer, introduces a Laplacian pyramid to extract the outlines and details in the image, and fuses low-frequency features, edge features and high-frequency features to perform high-quality reconstruction on the CT image.

[0049] In addition, the loss function of the training network includes three, which are as follows: the consisting of L1 loss and consisting of a multi-scale structural similarity loss, the consisting of a mean square error loss. The L1 loss function is:

[0050]

[0051] wherein y is the true value of the sinusoidal graph; is the predicted value of the sinusoidal graph; the multi-scale structural similarity loss function is:

[0052]

[0053] The mean square error loss function is:

[0054]

[0055] wherein x is the true value of the CT; is the predicted value of the CT;

[0056] The final network loss function is:

[0057]

[0058] wherein z is the true value of the network output; is the predicted value of the network output; and alpha and beta are both 0.5.

[0059] The following describes the THz-CT reconstruction method of the present application through a specific case:

[0060] First, sparse-view sinusoidal image data is acquired using a terahertz time-domain spectroscopy system and placed into image set A. Extremely sparse angle sampling is performed on the images in image set A, and the data is placed into image set B, along with a corresponding mask image set C. Then, image sets B and C are fed into a trained sinusoidal domain image processing model to obtain the optimized sinusoidal image set Sino-After. Next, the FISTA algorithm is used to perform CT reconstruction on image sets Sino-After and A, respectively, resulting in image set E and image set Image-Label. Image set E is then fed into a trained image domain processing model to obtain the CT image set Image-After. Finally, the loss is calculated for both the sinusoidal image set Sino-After and image set A, and simultaneously for both image sets Image-After and Image-Label. The loss values ​​are then used to update the network parameters.

[0061] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for extremely sparse angle THz-CT reconstruction based on a multi-domain, multi-scale feature fusion neural network, characterized in that, Includes the following steps: S1. The detector element of the terahertz time-domain spectroscopy (THz-TDS) system is moved at equal intervals to collect intensity data sine graphs from the corresponding viewing angles and placed into image set A. S2. Extract the images in image set A at equal intervals to obtain sine wave image set B with extremely sparse sampling perspective, and create mask image set C for sine wave image set B. S3. First, put the images in the sinusoidal image set B into the sinusoidal domain processing model to obtain the optimized image set New. Then, input the sinusoidal image set B, the mask image set C, and the image set New into the data consistency layer at the same time. First, multiply the image set New and the mask image set C to obtain the optimized image of the unsampled part. Finally, add the optimized image of the unsampled part to the image in the extremely sparsely sampled sinusoidal image set B to obtain the optimized sinusoidal image set D, and calculate the loss function value between image set D and image set A. S4. Based on the FISTA algorithm built in PyTorch, the optimized sine wave in image set D is initially reconstructed into CT image set E; S5. Input the CT image set E into the image domain processing model to obtain the optimized image set Image-After. Then, use the FISTA algorithm to put the CT images reconstructed from the sine wave acquired in step S1 into the image set Image-Label, and calculate the loss function value between the image set Image-After and the image set Image-Label. Combine this loss function value with the loss function value in step S3 and backpropagate to update the network weights and parameters. The optimizedr trained is Adam. When the set number of rounds is reached, stop updating the model weights and bias parameters to obtain a fully trained multi-domain multi-scale feature fusion neural network.

2. The method for extremely sparse angle THz-CT reconstruction based on a multi-domain, multi-scale feature fusion neural network according to claim 1, characterized in that, In step S1, intensity data of the corresponding viewpoints are collected at equal intervals, which is 1 / 5 of the 180 full-angle data required in THz-CT reconstruction, i.e., 36 data points.

3. The method for extremely sparse angle THz-CT reconstruction based on a multi-domain, multi-scale feature fusion neural network according to claim 1, characterized in that, In step S2, the image set A is further extracted at equal intervals to obtain 1 / 15, 1 / 20, 1 / 30, and 1 / 60 of the 180 full-angle values ​​required for THz-CT reconstruction, which are 12, 9, 6, and 3 respectively.

4. The method for extremely sparse angle THz-CT reconstruction based on a multi-domain, multi-scale feature fusion neural network according to claim 1, characterized in that, In step S3, the sinusoidal domain processing model includes a sinusoidal domain image reconstruction network and a data consistency layer. The sinusoidal domain image reconstruction network divides the number of channels into a sinusoidal domain branch and a Fourier sinusoidal domain branch. First, the feature map with 1 channel is expanded to 128 channels twice and then put into the sinusoidal domain image reconstruction network. The sinusoidal domain branch includes multiple convolutional layers and a channel spatial attention mechanism, where the convolutional layers consist of four 3×3 convolutional blocks. The Fourier sinusoidal domain branch uses Fast Fourier Transform to extract features from the feature map and uses one 3×3 convolutional block. Then, the outputs of the two branches are combined and residual connections are added. Finally, one 3×3 convolution is used to restore the input form.

5. The method for extremely sparse angle THz-CT reconstruction based on a multi-domain, multi-scale feature fusion neural network according to claim 1, characterized in that, In step S5, the image domain processing model includes an encoder, a decoder, a shallow extraction module, an image domain reconstruction network, and a high-frequency reconstruction layer; the primary encoder downsamples the CT image by 2... N The upsampling channels of the advanced encoder increase in increments of 2. N The form decreases, where the CT image with 64 channels in the primary encoder is captured by the shallow extraction module to capture features at different scales; Similar to the sinusoidal domain image reconstruction network, the image domain reconstruction network is divided into an image domain branch and an image Fourier domain branch. The image domain branch uses GRU gated convolutional units to enhance the ability to capture temporal features, while the image Fourier domain branch is consistent with the sinusoidal domain image reconstruction network. The low-level decoder upsamples the output of the image domain reconstruction network to match that of the high-level encoder; the high-level decoder, i.e., the high-frequency reconstruction layer, introduces the Laplacian pyramid to extract contours and details in the image, and fuses low-frequency features, edge features and high-frequency features to perform high-quality reconstruction of CT images.

6. The method for extremely sparse angle THz-CT reconstruction based on a multi-domain, multi-scale feature fusion neural network according to claim 1, characterized in that, The loss function for training the network includes three functions, as follows: (The loss function in the sinusoidal domain processing model is missing from the original text.) Composed of L1 loss and Composed of multi-scale structural similarity loss, in image domain processing models The loss consists of mean squared error; the L1 loss function is: Where y is the true value of the sine curve; It is the predicted value of the sine wave; the multi-scale structural similarity loss function is: The mean squared error loss function is: Where x is the true CT value; It is a CT prediction value; Finally, the network loss function is: Where z is the actual value output by the network; These are the network output predictions; α and β are both 0.5.

Images