A hybrid domain unsupervised limited-angle CT reconstruction method based on generative adversarial network

Abstract

The present invention relates to a hybrid domain unsupervised limited angle CT reconstruction method based on a generative adversarial network, comprising the following steps: step 1, constructing a CT reconstruction model: the constructed CT reconstruction module model comprises a first generative adversarial network for completing missing projection data and a second generative adversarial network for removing artifacts; step 2, training the first generative adversarial network and the second generative adversarial network separately to obtain the trained first generative adversarial network and the trained second generative adversarial network, that is, to obtain the trained CT reconstruction model; step 3, inputting any test sample in a test set into the trained CT reconstruction model to obtain a reconstructed CT image. The advantage is that the method further narrows the limited angle range, and the generated image has better performance in quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a hybrid domain unsupervised limited angle CT reconstruction method based on a generative adversarial network. Background Art

[0002] Computed tomography is a cross-sectional tomographic imaging technology. By performing X-ray imaging at different angles of the scanned object to generate a sinusoidal diagram, and then obtaining a reconstructed image through an algorithm, the internal information of the scanned object can be accurately obtained. However, in many application scenarios, due to the influence of the shape and structure of the object being detected, the limitation of the radiation dose of X-rays, and the pursuit of scanning speed and time, only projection data of limited angles (the amount of projection data is less than 180) can be obtained, such as: C-arm CT imaging, digital breast imaging, and dental imaging, which is called limited angle CT. Such data is incomplete and cannot meet the requirements of the Nyquist sampling theorem. The image reconstructed by the filtered back projection algorithm (FBP) has serious artifacts, which will affect the detection of industrial defects and the diagnosis of medical diseases. Limited angle projection image reconstruction is a typical inverse problem in mathematics. The solution obtained may be unstable, non-unique, or even non-existent. This is a challenging but common problem.

[0003] With the improvement of computer performance and the development of machine learning technology in recent years, many researchers have applied supervised convolutional neural network methods to the field of CT image reconstruction. For example, the stripe artifacts generated by the limited-angle sinusoidal image after reconstruction by the FBP algorithm have similar characteristics. Therefore, a neural network feature fitting framework based on the image domain is proposed to learn the mapping between the artifact image and the artifact-free image. Inspired by the research on image restoration in machine vision, some people believe that image completion in the projection domain to predict missing information is also an effective method. They use the context encoder (CE) to complete the limited-angle sinusoidal image, and then combine it with the iterative algorithm for reconstruction. The results show that the context encoder-based sinusoidal image completion method can effectively improve the efficiency and quality of iterative reconstruction. The above methods are processed in a single image domain or projection domain. For this reason, a hybrid domain convolutional neural network method is proposed to combine projection domain completion with image domain artifact suppression. The results are significantly better than the single domain FBPConvN method.

[0004] When the amount of training data is sufficient and the network is well trained, supervised deep learning methods can obtain good reconstruction results. However, in limited-angle CT reconstruction, it is difficult to obtain a large amount of paired data for supervised training. Therefore, some scholars have proposed a sine completion network (SIN) trained in an unsupervised manner. The network consists of two parts. The first part is a U-net-based generative adversarial network to complete the sinusoidal image. The second part is a U-net generative adversarial network with the same structure to learn the mapping from artifact images to artifact-free images. The two parts are connected by differentiable Radon and inverse Radon transform networks. The results are superior to the latest TV regularized simultaneous algebraic reconstruction technology (SART-TV) in both qualitative and quantitative aspects. However, its limitation is that it can only generate images of size 256×256, and the limit of the limited angle range can only be reduced to [0°,120°]. In addition, someone proposed another unsupervised tomographic reconstruction adversarial network based on hybrid domain optimization. Based on the sine completion network, it successfully expanded the size of the reconstructed image to 512×512 and significantly reduced the reconstruction time. However, this method learns the mapping from the simulated sinusoid to the real sinusoid in the projection domain, and does not add effective information in the projection domain. Its effect on improving the reconstruction quality is not obvious. Similarly, the minimum supported scanning angle limit is still [0°, 120°], and it will be powerless when the scanning angle is further reduced. Further improvements are needed for this. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a hybrid domain unsupervised limited angle CT reconstruction method based on a generative adversarial network to reduce the limited angle range and improve image quality in response to the above-mentioned prior art.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a hybrid domain unsupervised limited angle CT reconstruction method based on a generative adversarial network, characterized by comprising the following steps:

[0007] Step 1: construct a CT reconstruction model;

[0008] The constructed CT reconstruction model includes a first generative adversarial network for completing missing projection data and a second generative adversarial network for removing artifacts; wherein the first generative adversarial network includes a first generator G and a first discriminator D, and the first generator G is composed of an encoder module a, an attention module b, a decoder module c and a multi-scale decoding module d; the second generative adversarial network includes a second generator Gx2y, a second discriminator D_y, a third generator Gy2x and a third discriminator D_x;

[0009] Step 2: Train the first generative adversarial network and the second generative adversarial network separately to obtain the trained first generative adversarial network and the trained second generative adversarial network, that is, to obtain a trained CT reconstruction model;

[0010] The training process of the first generative adversarial network is:

[0011] Step 2-1, convert the original CT image into multiple sinusoidal images with an angle range of [0°, q°], 180<q≤360, and delete the projection data of [s°, t°] from each sinusoidal image of [0°, q°] to obtain the sinusoidal image of each [0°, q°] Figure 1 A corresponding projection data missing sinusoidal graph; s, t∈{0,q}; then constructed together into a training set and a test set;

[0012] Each training sample in the training set and each test sample in the test set includes a sinusoidal graph with an angle range of [0°, q°] and a projection data missing sinusoidal graph corresponding to the sinusoidal graph;

[0013] Step 2-2, inputting one or more training samples in the training set into the initialized first generative adversarial network;

[0014] The training process of inputting any one of the projection data missing sinusoidal graphs into the initialized first generative adversarial network is:

[0015] Step 2-21, inputting the projection data missing sinusoidal graph into the encoder module a, the encoder module a includes N sequentially connected downsampling layers, that is, obtaining the first downsampling feature graph, the second downsampling feature graph, ... the Nth downsampling feature graph in sequence; N is a positive integer;

[0016] Step 2-22, input the downsampled feature map output by each downsampling layer into the attention module b, which includes N-1 ATN networks;

[0017] Specifically, the Nth down-sampled feature map and the N-1th down-sampled feature map are input into the N-1th ATN network to obtain the N-1th patch feature map; and the i-th patch feature map and the i-1th down-sampled feature map are input into the i-1th ATN network to obtain the i-1th patch feature map, where i is N-1, N-2...2 in sequence, that is, the N-2th patch feature map, the N-3th patch feature map...the 1st patch feature map are obtained in sequence;

[0018] Step 2-23, input the downsampled feature map output by the encoder module a and the patched feature map output by the attention module b into the decoder module c in a cross-layer connection manner, wherein the decoder module c includes N sequentially connected upsampling layers and a convolutional layer with a channel number of 1;

[0019] Specifically, the Nth down-sampled feature map is input into the first up-sampled layer to obtain the first up-sampled feature map; the jth up-sampled feature map and the Njth patch feature map are stacked and input into the j+1th up-sampled layer to obtain the j+1th up-sampled feature map, where the value of j is 1, 2...N-1, that is, the second up-sampled feature map, the third up-sampled feature map...the Nth up-sampled feature map are obtained in sequence; the N up-sampled feature maps are passed through a convolutional layer with a channel number of 1 to obtain the final completed sinusoidal map;

[0020] Step 2-24, input the final completed sinusoidal graph outputted from the decoder module c into the first discriminator D, where the first discriminator D comprises M convolutional layers connected in sequence, and the number of channels of the last convolutional layer is 1, that is, outputting a probability result image block;

[0021] In addition, the upsampled feature maps output by the first N-1 upsampling layers in the decoder module c are input into the multi-scale decoding module d, which includes N-1 feature extraction layers with 1*1 convolution kernels with a channel number of 1;

[0022] Specifically, the first up-sampled feature map, the second up-sampled feature map, the third up-sampled feature map, and the N-1th up-sampled feature map are respectively input into a feature extraction layer of a 1*1 convolution kernel with a channel number of 1;

[0023] Step 2-25, calculate the total loss function L SI , and according to the total loss function L SI Update the initialized first generative adversarial network parameters, that is, obtain the first generative adversarial network after one training;

[0024] Step 2-3, selecting training samples of the training set multiple times and training the first generative adversarial network according to the same method as in the above step 2-2, that is, obtaining a trained first generative adversarial network;

[0025] The training process of the second generative adversarial network is:

[0026] Step 2-a, in the first generative adversarial network, the image output by the decoder module c is spliced ​​with the missing sinusoidal image of the projection data input to the encoder module a as the output result of the first generative adversarial network; the [0°, 180°] sinusoidal image is extracted from the output result of the first generative adversarial network, and the extracted [0°, 180°] sinusoidal image is reconstructed by FBP to obtain the generated domain CT image r_x;

[0027] Step 2-b, using the generated domain CT image r_x and the collected original CT image r_y as unpaired data sets;

[0028] Step 2-c, input the generated domain CT image r_x into the initialized second generator Gx2y to obtain the first image f_y, and input the obtained first image f_y into the second discriminator D_y;

[0029] Step 2-d, input the original CT image r_y into the initialized third generator Gy2x to obtain the second image f_x, and input the obtained second image f_x into the third discriminator D_x;

[0030] Step 2-e, calculate the total loss function L of the generated domain CT image r_x and the original CT image r_y AR , and according to the total loss function L AR Update the network parameters of the second generator Gx2y and the third generator Gy2x to obtain the second generator after one training;

[0031] Step 2-f, training the second generator Gx2y multiple times according to the same method as in steps 2-a to 2-e, to obtain the trained second generator Gx2y;

[0032] Step 3: Input any test sample in the test set into the trained CT reconstruction model to obtain a reconstructed CT image;

[0033] The specific process is:

[0034] Step 3-1, input any projection data missing sinusoidal graph in the test set into the trained encoder module a, and obtain the first complement sinusoidal graph outputted by the trained decoder module c according to the same method as in steps 2-21 to 2-23; and splice the first complement sinusoidal graph outputted by the trained decoder module c and the projection data missing sinusoidal graph inputted into the trained encoder module a into a final full-angle sinusoidal image;

[0035] Step 3-2, extracting a [0°, 180°] sinusoidal image from the final full-angle sinusoidal image, and performing FBP reconstruction on the extracted [0°, 180°] sinusoidal image to obtain a first generated domain CT image;

[0036] Step 3-3: input the first generated domain CT image into the trained second generator Gx2y to obtain a reconstructed CT image.

[0037] Preferably, the encoder module a in step 2-21 is composed of 6 downsampling layers connected in sequence, and each downsampling layer is a convolutional layer, the convolution kernel of each convolutional layer is 3×3, and the number of channels is 32, 64, 128, 256, 512 and 512 respectively. The first 5 convolutional layers use LeakyReLU as the activation function, and the slope is set to 0.2; the sixth convolutional layer uses ReLU as the activation function.

[0038] Preferably, the decoder module c in step 2-23 includes 6 upsampling layers consisting of "upsampling + convolution", the convolution kernel size of each layer is 3×3, the number of channels of the first 6 layers are 512, 256, 128, 64, 32 and 32 respectively, the activation function is ReLU, and the final completed sinusoidal graph is obtained by outputting the convolution layer with a channel number of 1, and the activation function of the last layer is Tanh.

[0039] Preferably, the first discriminator D in step 2-24 is composed of 5 layers of convolution, the first 4 layers of convolution kernels are 5×5, the step size is 2, the number of channels are 64, 128, 256, 512 respectively, the activation function is LeakyReLU, the slope is set to 0.2, and finally a convolution layer with 1 channel number, 5×5 convolution kernel and step size of 1 is used.

[0040] In the present invention, the total loss function L in step 2-25 is SI The calculation formula is:

[0041] L SI =λ1L GAN (G,D)+λ2L hole (G)+λ3L ms

[0042] Among them, λ1, λ2 and λ3 are L GAN (G,D),L hole (G) and L ms The learning rate L GAN (G,D) is the generation adversarial loss function, D(y) represents the output probability after the [0°, q°] sinusoidal graph y in any training sample is input to the first discriminator D; G(x) represents the image generated after the projection data missing sinusoidal graph x corresponding to the sinusoidal graph y is input to the first generator G; D(G(x)) represents the output probability after the image G(x) generated by the first generator G is input to the first discriminator D; represents the calculation of mathematical expectation; Pdata(y) represents the data distribution of sinusoidal graph y; Pdata(x) represents the data distribution of sinusoidal graph x with missing projection data; L hole (G) is the hole loss function, m is a mask, the value of the missing part of the projection data is 1, and the value of the rest is 0; ⊙ represents matrix dot product, ||||1 represents L1 norm; L ms is the multi-scale L1 loss, φ p is the upsampled feature map output by the pth layer of the decoder module c, f(φ p ) represents the up-sampled feature map φ p Encoded into a grayscale image of the same size with 1 channel, y p Indicates size and φ p Same original image.

[0043] In the present invention, the total loss function L in step 2-e is AR The calculation formula is:

[0044] L AR =α1L GAN +α2L cycle +α3L ide +α4L err

[0045] Among them, α1, α2, α3, and α4 are L GAN , L cycle , L ide and L err The coefficient of

[0046] L GAN =L GAN (Gx2y)+L GAN (Gy2x), L GAN (Gx2y) is the loss function of the second generator Gx2y,

[0047] y′~P data (r_y) indicates that image y′ follows the data distribution of the original CT image r_y; D_y(y′) indicates the output probability after image y′ is input to the second discriminator D_y; x′~P data (r_x) indicates that the image x′ obeys the data distribution of the generated domain CT image r_x; Gx2y(x′) indicates the image obtained by inputting the image x′ into the second generator Gx2y, and D_y(Gx2y(x′)) indicates the output probability after Gx2y(x′) is input into the second discriminator D_y; L GAN (Gy2x) is the loss function of the third generator Gy2x, D_x(x′) represents the output probability after image x′ is input to the third discriminator D_x; Gy2x(y′) represents the image obtained by inputting image y′ into the third generator Gy2x, and D_x(Gy2x(y′)) represents the output probability after Gy2x(y′) is input to the third discriminator D_x;

[0048]

[0049] Gy2x(Gx2y(x′)) represents the image output by inputting Gx2y(x′) into the third generator Gy2x; Gx2y(Gy2x(y′)) represents the image output by inputting Gy2x(y′) into the second generator Gx2y;

[0050]

[0051] Gy2x(x′) represents the image output when image x′ is input to the third generator Gy2x; Gx2y(y′) represents the image output when image y′ is input to the second generator Gx2y;

[0052]

[0053] Compared with the prior art, the advantages of the present invention are: the first generative adversarial network in the present invention can be effectively trained without paired sinusoidal-reconstructed image pairs, and through an unsupervised method, on the basis of ensuring the reconstruction size, the sinusoidal data missing from the limited-angle projection data is used to generate a reconstruction result equivalent to the full scan reconstruction. Compared with the current unsupervised deep learning hybrid domain processing method, this method further reduces the limited angle range, and at the same time, the generated image has better performance in quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of a first generative adversarial network in an embodiment of the present invention;

[0055] Figure 2 for Figure 1 Schematic diagram of the attention module;

[0056] Figure 3 This is a schematic diagram of a first generative adversarial network used for testing in an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of a second generative adversarial network in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention is further described in detail below with reference to the accompanying drawings.

[0059] A hybrid domain unsupervised limited angle CT reconstruction method based on a generative adversarial network in this embodiment includes the following steps:

[0060] Step 1: construct a CT reconstruction model;

[0061] The constructed CT reconstruction model includes a first generative adversarial network for completing missing projection data and a second generative adversarial network for removing artifacts; wherein the first generative adversarial network includes a first generator G and a first discriminator D, and the first generator G is composed of an encoder module a, an attention module b, a decoder module c and a multi-scale decoding module d; the second generative adversarial network includes a second generator Gx2y, a second discriminator D_y, a third generator Gy2x and a third discriminator D_x;

[0062] Step 2: Train the first generative adversarial network and the second generative adversarial network separately to obtain the trained first generative adversarial network and the trained second generative adversarial network, that is, to obtain a trained CT reconstruction model;

[0063] The training process of the first generative adversarial network is:

[0064] Step 2-1, convert the original CT image into multiple sinusoidal images with an angle range of [0°, q°], 180<q≤360, and delete the projection data of [s°, t°] from each sinusoidal image of [0°, q°] to obtain the sinusoidal image of each [0°, q°] Figure 1 A corresponding projection data missing sinusoidal graph; s, t∈{0,q}; then constructed together into a training set and a test set;

[0065] Each training sample in the training set and each test sample in the test set includes a sinusoidal graph with an angle range of [0°, q°] and a projection data missing sinusoidal graph corresponding to the sinusoidal graph;

[0066] In this embodiment, q=192, s=90, t=180;

[0067] Step 2-2, inputting one or more training samples in the training set into the initialized first generative adversarial network;

[0068] The training process of inputting any one of the projection data missing sinusoidal graphs into the initialized first generative adversarial network is:

[0069] Step 2-21, inputting the projection data missing sinusoidal graph into the encoder module a, the encoder module a includes N sequentially connected downsampling layers, that is, obtaining the first downsampling feature graph, the second downsampling feature graph, ... the Nth downsampling feature graph in sequence; N is a positive integer;

[0070] Step 2-22, input the downsampled feature map output by each downsampling layer into the attention module b, which includes N-1 ATN networks;

[0071] Specifically, the Nth down-sampled feature map and the N-1th down-sampled feature map are input into the N-1th ATN network to obtain the N-1th patch feature map; and the i-th patch feature map and the i-1th down-sampled feature map are input into the i-1th ATN network to obtain the i-1th patch feature map, where i is N-1, N-2...2 in sequence, that is, the N-2th patch feature map, the N-3th patch feature map...the 1st patch feature map are obtained in sequence;

[0072] Step 2-23, input the downsampled feature map output by the encoder module a and the patched feature map output by the attention module b into the decoder module c in a cross-layer connection manner, wherein the decoder module c includes N sequentially connected upsampling layers and a convolutional layer with a channel number of 1;

[0073] Specifically, the Nth down-sampled feature map is input into the first up-sampled layer to obtain the first up-sampled feature map; the jth up-sampled feature map and the Njth patch feature map are stacked and input into the j+1th up-sampled layer to obtain the j+1th up-sampled feature map, where the value of j is 1, 2...N-1, that is, the second up-sampled feature map, the third up-sampled feature map...the Nth up-sampled feature map are obtained in sequence; the N up-sampled feature maps are passed through a convolutional layer with a channel number of 1 to obtain the final completed sinusoidal map;

[0074] Step 2-24, input the final completed sinusoidal graph outputted from the decoder module c into the first discriminator D, where the first discriminator D comprises M convolutional layers connected in sequence, and the number of channels of the last convolutional layer is 1, that is, outputting a probability result image block;

[0075] In addition, the upsampled feature maps output by the first N-1 upsampling layers in the decoder module c are input into the multi-scale decoding module d, which includes N-1 feature extraction layers with 1*1 convolution kernels with a channel number of 1;

[0076] Specifically, the first up-sampled feature map, the second up-sampled feature map, the third up-sampled feature map, and the N-1th up-sampled feature map are respectively input into a feature extraction layer of a 1*1 convolution kernel with a channel number of 1;

[0077] Step 2-25, calculate the total loss function L SI , and according to the total loss function L SI Update the initialized first generative adversarial network parameters, that is, obtain the first generative adversarial network after one training;

[0078] The total loss function LSI The calculation formula is:

[0079] L SI =λ1L GAN (G,D)+λ2L hole (G)+λ3L ms

[0080] Among them, λ1, λ2 and λ3 are L GAN (G,D),L hole (G) and L ms The learning rate L GAN (G,D) is the generation adversarial loss function, D(y) represents the output probability after the [0°, q°] sinusoidal graph y in any training sample is input to the first discriminator D; G(x) represents the image generated after the projection data missing sinusoidal graph x corresponding to the sinusoidal graph y is input to the first generator G, corresponding to the image output by the decoder module c in the first generator G; D(G(x)) represents the output probability after the image G(x) generated by the first generator G is input to the first discriminator D; represents the calculation of mathematical expectation; Pdata(y) represents the data distribution of sinusoidal graph y, y~Pdata(y) corresponds to any one or a group of sinusoidal graphs y that satisfy Pdata(y); Pdata(x) represents the data distribution of sinusoidal graph x with missing projection data, x~Pdata(x) corresponds to any one or a group of sinusoidal graphs x with missing projection data that satisfy Pdata(x); L hole (G) is the hole loss function, m is a mask, the value of the missing part of the projection data is 1, and the value of the rest is 0. m is a matrix with the same size as the sinusoidal graph y and G(x); ⊙ represents matrix dot product, ||||1 represents L1 norm; L ms is the multi-scale L1 loss, φ p is the upsampled feature map output by the pth layer of the decoder module c, f(φ p ) represents the up-sampled feature map φ p Encoded into a grayscale image of the same size with 1 channel, y p Indicates size and φ p Same original image;

[0081] In this embodiment, the values ​​of λ1, λ2, and λ3 are: 0.05, 6, and 0.5 respectively;

[0082] The above generated adversarial loss function L GAN(G, D) attempts to learn the mapping of G(x)→y, and to generate realistic images that conform to the distribution of y through x as much as possible; in order to make the generator generate real data as much as possible in the missing area of ​​the image, the hole loss function L is introduced in this embodiment. hole (G) Constraining the optimization results; in addition, the output results of each layer of the decoder module are constrained by a multi-scale L1 loss function, so that the output results of each layer are as similar as possible to the original image of the same scale. Loss constraints of different scales can further improve the model effect;

[0083] Step 2-3, selecting training samples of the training set multiple times and training the first generative adversarial network according to the same method as in the above step 2-2, that is, obtaining a trained first generative adversarial network;

[0084] Each training sample of the first generative adversarial network includes a sinusoidal graph with an angle range of [0°, q°] and a projection data missing sinusoidal graph corresponding to the sinusoidal graph. However, in actual training, the completed sinusoidal graph is obtained by inputting the projection data missing sinusoidal graph into the first generative adversarial network. The sinusoidal graph with an angle range of [0°, q°] is used to play a constraint role in the calculation of the loss function.

[0085] like Figure 4 As shown, the training process of the second generative adversarial network is:

[0086] Step 2-a, in the first generative adversarial network, the image output by the decoder module c is spliced ​​with the missing sinusoidal image of the projection data input to the encoder module a as the output result of the first generative adversarial network; the [0°, 180°] sinusoidal image is extracted from the output result of the first generative adversarial network, and the extracted [0°, 180°] sinusoidal image is reconstructed by FBP to obtain the generated domain CT image r_x;

[0087] Step 2-b, using the generated domain CT image r_x and the collected original CT image r_y as unpaired data sets;

[0088] Step 2-c, input the generated domain CT image r_x into the initialized second generator Gx2y to obtain the first image f_y, and input the obtained first image f_y into the second discriminator D_y;

[0089] Step 2-d, input the original CT image r_y into the initialized third generator Gy2x to obtain the second image f_x, and input the obtained second image f_x into the third discriminator D_x;

[0090] Step 2-e, calculate the total loss function L of the generated domain CT image r_x and the original CT image r_y AR , and according to the total loss function LAR Update the network parameters of the second generator Gx2y and the third generator Gy2x to obtain the second generator after one training;

[0091] The total loss function L AR The calculation formula is:

[0092] L AR =α1L GAN +α2L cycle +α3L ide +α4L err

[0093] Among them, α1, α2, α3, and α4 are L GAN , L cycle , L ide and L err The coefficient of

[0094] L GAN =L GAN (Gx2y)+L GAN (Gy2x), L GAN (Gx2y) is the loss function of the second generator Gx2y,

[0095] y′~P data (r_y) indicates that image y′ follows the data distribution of the original CT image r_y; D_y(y′) indicates the output probability after image y′ is input to the second discriminator D_y; x′~P data (r_x) indicates that the image x′ obeys the data distribution of the generated domain CT image r_x; Gx2y(x′) indicates the image obtained by inputting the image x′ into the second generator Gx2y, and D_y(Gx2y(x′)) indicates the output probability after Gx2y(x′) is input into the second discriminator D_y; L GAN (Gy2x) is the loss function of the third generator Gy2x, D_x(x′) represents the output probability after image x′ is input to the third discriminator D_x; Gy2x(y′) represents the image obtained by inputting image y′ into the third generator Gy2x, and D_x(Gy2x(y′)) represents the output probability after Gy2x(y′) is input to the third discriminator D_x;

[0096]

[0097] Gy2x(Gx2y(x′)) represents the image output by inputting Gx2y(x′) into the third generator Gy2x; Gx2y(Gy2x(y′)) represents the image output by inputting Gy2x(y′) into the second generator Gx2y;

[0098]

[0099] Gy2x(x′) represents the image output when image x′ is input to the third generator Gy2x; Gx2y(y′) represents the image output when image y′ is input to the second generator Gx2y;

[0100]

[0101] In this embodiment, α1, α2, α3, and α4 are 1, 8, 4, and 2 respectively;

[0102] Step 2-f, training the second generator Gx2y multiple times according to the same method as in steps 2-a to 2-e, to obtain the trained second generator Gx2y;

[0103] Step 3: Input any test sample in the test set into the trained CT reconstruction model to obtain a reconstructed CT image;

[0104] The specific process is:

[0105] Step 3-1, input any projection data missing sinusoidal graph in the test set into the trained encoder module a, and obtain the first complement sinusoidal graph outputted by the trained decoder module c according to the same method as in steps 2-21 to 2-23; and splice the first complement sinusoidal graph outputted by the trained decoder module c and the projection data missing sinusoidal graph inputted into the trained encoder module a into a final full-angle sinusoidal image;

[0106] Step 3-2, extracting a [0°, 180°] sinusoidal image from the final full-angle sinusoidal image, and performing FBP reconstruction on the extracted [0°, 180°] sinusoidal image to obtain a first generated domain CT image;

[0107] Step 3-3: input the first generated domain CT image into the trained second generator Gx2y to obtain a reconstructed CT image.

[0108] The overall framework of the first generative adversarial network mentioned above is as follows Figure 1 As shown, the ATN network diagram in the attention module b is as follows Figure 2 As shown; when testing Figure 3As shown, only the trained encoder module a, attention module b and decoder module c are used to predict the missing part of the true limited-angle sinusoidal image and concatenate it with the input limited-angle sinusoidal image into the final full-angle sinusoidal image.

[0109] The encoder module a in the above-mentioned first generative adversarial network is mainly used to extract feature information of the finite-angle sinusoidal graph, downsample the low-level masks and images layer by layer, and learn a high-level compact potential feature; in this embodiment, the encoder module a is composed of 6 downsampling layers connected in sequence, and each downsampling layer is a convolutional layer, the convolution kernel of each convolutional layer is 3×3, and the number of channels is 32, 64, 128, 256, 512 and 512 respectively. The first 5 convolutional layers use LeakyReLU as the activation function, and the slope is set to 0.2; in order to make the values ​​of the feature maps output by the encoder positive numbers, the sixth convolutional layer uses ReLU as the activation function.

[0110] In order to overcome the limitation that the encoder module a cannot effectively observe distant image features due to layer-by-layer convolution, the present invention adds an attention module b. By repeatedly using the ATN network between adjacent layers of the encoder, the content of the high-level semantic feature layer is learned to fill the missing areas of the low-level feature layer. This cross-layer attention filling mechanism can realize the mapping of high-level semantic features to low-level features, and can achieve higher resolution and ensure the content consistency of the missing area. For the specific process of the ATN network, please refer to the paper: "Yu, JH, et al., Generative Image Inpainting with ContextualAttention. 2018 Ieee / Cvf Conference on Computer Vision and Pattern Recognition (Cvpr), 2018: p.5505-5514".

[0111] The decoder module c receives the feature images from the encoder module a and the attention module b simultaneously in a cross-layer connection mode, stacks the output feature map of the previous layer of the decoder and the output feature map of the corresponding layer of the attention module, and then inputs them into the next layer of the decoder, continuously advancing iteratively to complete the missing holes and form a complete sinusoidal image. In order to avoid the checkerboard phenomenon when using transposed convolution, this module replaces the transposed convolution layer with "upsampling + convolution". The convolution kernel size of each layer is 3×3, and the number of channels in the first 6 layers are 512, 256, 128, 64, 32 and 32 respectively. The activation function is ReLU, and then the convolution layer with a channel number of 1 outputs the final completed sinusoidal image. The activation function of the last layer is Tanh.

[0112] The multi-scale decoding module d generates images of different scales by convolving the feature maps of different layers of the decoder module c, and establishes a more stringent multi-scale loss to achieve better results. All convolutional layers use a 1×1 convolution kernel with a channel number of 1 and Tanh as the activation function.

[0113] The first discriminator D in this embodiment will output an M×N image block instead of a single value. Each pixel value in the image block represents the probability of the authenticity of the image in the corresponding receptive field of the original image. This region-aware discriminator can improve the similarity between the generated image and the original image in texture and semantics, so that the generated image has higher resolution and details. The first discriminator D in this embodiment consists of 5 layers of convolution. The first 4 layers have a convolution kernel of 5×5, a step size of 2, and the number of channels are 64, 128, 256, and 512 respectively. The activation function is LeakyReLU, and the slope is set to 0.2. Finally, a convolution layer with a channel number of 1, a convolution kernel of 5×5, and a step size of 1 is used to output the probability result image block, and the size of the image block is 59×19.

[0114] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A hybrid domain unsupervised limited angle CT reconstruction method based on generative adversarial network, characterized in that The steps include: Step 1: construct a CT reconstruction model; The constructed CT reconstruction model includes a first generative adversarial network for completing missing projection data and a second generative adversarial network for removing artifacts; The first generative adversarial network includes a first generator G and a first discriminator D, and the first generator G is composed of an encoder module a, an attention module b, a decoder module c and a multi-scale decoding module d; the second generative adversarial network includes a second generator Gx2y, a second discriminator D_y, a third generator Gy2x and a third discriminator D_x; Step 2: Train the first generative adversarial network and the second generative adversarial network separately to obtain the trained first generative adversarial network and the trained second generative adversarial network, that is, to obtain a trained CT reconstruction model; The training process of the first generative adversarial network is: Step 2-1, converting the original CT image into multiple sinusoidal images with an angle range of [0°, q°], 180<q≤360, and deleting the projection data of [s°, t°] from each sinusoidal image of [0°, q°] to obtain a projection data missing sinusoidal image corresponding to each sinusoidal image of [0°, q°]; s, t∈{0,q}; then construct them together into training set and test set; Each training sample in the training set and each test sample in the test set includes a sinusoidal graph with an angle range of [0°, q°] and a projection data missing sinusoidal graph corresponding to the sinusoidal graph; Step 2-2, inputting one or more training samples in the training set into the initialized first generative adversarial network; The training process of inputting any one of the projection data missing sinusoidal graphs into the initialized first generative adversarial network is: Step 2-21, inputting the projection data missing sinusoidal graph into the encoder module a, the encoder module a includes N sequentially connected downsampling layers, that is, obtaining the first downsampling feature graph, the second downsampling feature graph, ... the Nth downsampling feature graph in sequence; N is a positive integer; Step 2-22, input the downsampled feature map output by each downsampling layer into the attention module b, which includes N-1 ATN networks; Specifically, the Nth down-sampled feature map and the N-1th down-sampled feature map are input into the N-1th ATN network to obtain the N-1th patch feature map; and the i-th patch feature map and the i-1th down-sampled feature map are input into the i-1th ATN network to obtain the i-1th patch feature map, where i is N-1, N-2...2 in sequence, that is, the N-2th patch feature map, the N-3th patch feature map...the 1st patch feature map are obtained in sequence; Step 2-23, input the downsampled feature map output by the encoder module a and the patched feature map output by the attention module b into the decoder module c in a cross-layer connection manner, wherein the decoder module c includes N sequentially connected upsampling layers and a convolutional layer with a channel number of 1; Specifically, the Nth down-sampled feature map is input into the first up-sampled layer to obtain the first up-sampled feature map; the jth up-sampled feature map and the Njth patched feature map are stacked and input into the j+1th up-sampled layer to obtain the j+1th up-sampled feature map, where the value of j is 1, 2...N-1, that is, the second up-sampled feature map, the third up-sampled feature map...the Nth up-sampled feature map are obtained in sequence; the N up-sampled feature maps are passed through a convolutional layer with a channel number of 1 to obtain the final completed sinusoidal map; Step 2-24, input the final completed sinusoidal graph outputted from the decoder module c into the first discriminator D, where the first discriminator D comprises M convolutional layers connected in sequence, and the number of channels of the last convolutional layer is 1, that is, outputting a probability result image block; In addition, the upsampled feature maps output by the first N-1 upsampling layers in the decoder module c are input into the multi-scale decoding module d, which includes N-1 feature extraction layers with 1*1 convolution kernels with a channel number of 1; Specifically, the first up-sampled feature map, the second up-sampled feature map, the third up-sampled feature map, and the N-1th up-sampled feature map are respectively input into a feature extraction layer of a 1*1 convolution kernel with a channel number of 1; Step 2-25, calculate the total loss function L SI , and according to the total loss function L SI Update the initialized first generative adversarial network parameters, that is, obtain the first generative adversarial network after one training; Step 2-3, selecting training samples of the training set multiple times and training the first generative adversarial network according to the same method as in the above step 2-2, that is, obtaining a trained first generative adversarial network; The training process of the second generative adversarial network is: Step 2-a, in the first generative adversarial network, the image output by the decoder module c and the missing sinusoidal graph of the projection data input into the encoder module a are concatenated as the output result of the first generative adversarial network; Extract the [0°, 180°] sinusoidal image from the output of the first generative adversarial network, and perform FBP reconstruction on the extracted [0°, 180°] sinusoidal image to obtain the generated domain CT image r_x; Step 2-b, using the generated domain CT image r_x and the collected original CT image r_y as unpaired data sets; Step 2-c, input the generated domain CT image r_x into the initialized second generator Gx2y to obtain the first image f_y, and input the obtained first image f_y into the second discriminator D_y; Step 2-d, input the original CT image r_y into the initialized third generator Gy2x to obtain the second image f_x, and input the obtained second image f_x into the third discriminator D_x; Step 2-e, calculate the total loss function L of the generated domain CT image r_x and the original CT image r_y AR , and according to the total loss function L AR Update the network parameters of the second generator Gx2y and the third generator Gy2x to obtain the second generator Gx2y after one training; Step 2-f, training the second generator Gx2y multiple times according to the same method as in steps 2-a to 2-e, to obtain the trained second generator Gx2y; Step 3: Input any test sample in the test set into the trained CT reconstruction model to obtain a reconstructed CT image; The specific process is: Step 3-1, input any projection data missing sinusoidal graph in the test set into the trained encoder module a, and obtain the first complement sinusoidal graph outputted by the trained decoder module c according to the same method as in steps 2-21 to 2-23; and splice the first complement sinusoidal graph outputted by the trained decoder module c and the projection data missing sinusoidal graph inputted into the trained encoder module a into a final full-angle sinusoidal image; Step 3-2, extracting a [0°, 180°] sinusoidal image from the final full-angle sinusoidal image, and performing FBP reconstruction on the extracted [0°, 180°] sinusoidal image to obtain a first generated domain CT image; Step 3-3: input the first generated domain CT image into the trained second generator Gx2y to obtain a reconstructed CT image.

2. The hybrid domain unsupervised limited angle CT reconstruction method based on generative adversarial network according to claim 1, characterized in that: The encoder module a in the step 2-21 is composed of 6 downsampling layers connected in sequence, and each downsampling layer is a convolutional layer. The convolution kernel of each convolutional layer is 3×3, and the number of channels is 32, 64, 128, 256, 512 and 512 respectively. The first 5 convolutional layers use LeakyReLU as the activation function, and the slope is set to 0.2; the sixth convolutional layer uses ReLU as the activation function.

3. The hybrid domain unsupervised limited angle CT reconstruction method based on generative adversarial network according to claim 1, characterized in that: The decoder module c in step 2-23 includes 6 upsampling layers consisting of "upsampling + convolution", the convolution kernel size of each layer is 3×3, the number of channels of the first 6 layers are 512, 256, 128, 64, 32 and 32 respectively, the activation function is ReLU, and then the final completed sinusoidal graph is output through a convolution layer with a channel number of 1, and the activation function of the last layer is Tanh.

4. The hybrid domain unsupervised limited angle CT reconstruction method based on generative adversarial network according to claim 1, characterized in that: In the step 2-24, the first discriminator D is composed of 5 layers of convolution. The first 4 layers have convolution kernels of 5×5, a step size of 2, and the number of channels are 64, 128, 256, and 512 respectively. The activation function is LeakyReLU, and the slope is set to 0.

2. Finally, a convolution layer with 1 channel, a convolution kernel of 5×5, and a step size of 1 is used.

5. The hybrid domain unsupervised limited angle CT reconstruction method based on generative adversarial network according to claim 1, characterized in that: The total loss function L in step 2-25 SI The calculation formula is: L SI =λ1L GAN (G,D)+λ2L hole (G)+λ3L ms Among them, λ1, λ2 and λ3 are L GAN (G, D), L hole (G) and L ms The learning rate L GAN (G, D) is the generation adversarial loss function, D(y) represents the output probability after the [0°, q°] sinusoidal graph y in any training sample is input to the first discriminator D; G(x) represents the image generated after the projection data missing sinusoidal graph x corresponding to the sinusoidal graph y is input to the first generator G; D(G(x)) represents the output probability after the image G(x) generated by the first generator G is input to the first discriminator D; represents the calculation of mathematical expectation; Pdata(y) represents the data distribution of sinusoidal graph y; Pdata(x) represents the data distribution of sinusoidal graph x with missing projection data; L hole (G) is the hole loss function, m is a mask, the value of the missing part of the projection data is 1, and the value of the rest is 0; ⊙ represents matrix dot product, || ||1 represents L1 norm; L ms is the multi-scale L1 loss, φ p is the upsampled feature map output by the pth layer of the decoder module c, f(φ p ) represents the up-sampled feature map φ p Encoded into a grayscale image of the same size and with 1 channel, y p Indicates size and φ p Same original image.

6. The hybrid domain unsupervised limited angle CT reconstruction method based on generative adversarial network according to claim 1, characterized in that: The total loss function L in step 2-e AR The calculation formula is: L AR =α1L GAN +ɑ2L cycle +ɑ3L ide +α4L err Among them, α1, α2, α3, and α4 are L GAN , L cycle , L ide and L err The coefficient of L GAN =L GAN (Gx2y)+L GAN (Gy2x), L GAN (Gx2y) is the loss function of the second generator Gx2y, y′~P data (r_y) indicates that image y′ follows the data distribution of the original CT image r_y; D_y(y′) indicates the output probability after image y′ is input to the second discriminator D_y; x′~P data (r_x) indicates that the image x′ obeys the data distribution of the generated domain CT image r_x; Gx2y(x′) indicates the image obtained by inputting the image x′ into the second generator Gx2y, and D_y(Gx2y(x′)) indicates the output probability after Gx2y(x′) is input into the second discriminator D_y; L GAN (Gy2x) is the loss function of the third generator Gy2x, D_x(x′) represents the output probability after image x′ is input to the third discriminator D_x; Gy2x(y′) represents the image obtained by inputting image y′ into the third generator Gy2x, and D_x(Gy2x(y′)) represents the output probability after Gy2x(y′) is input to the third discriminator D_x; Gy2x(Gx2y(x′)) represents the image output by inputting Gx2y(x′) into the third generator Gy2x; Gx2y(Gy2x(y′)) represents the image output by inputting Gy2x(y′) into the second generator Gx2y; Gy2x(x′) represents the image output when image x′ is input to the third generator Gy2x; Gx2y(y′) represents the image output when image y′ is input to the second generator Gx2y;

Images