Image compressive sensing reconstruction method and system based on transformer generative adversarial network
By constructing a deep generative adversarial network (GAN) that includes sampling, generation, and discrimination networks, the image compressed sensing method based on Transformer generative adversarial networks solves the problems of high time complexity and poor reconstruction effect of traditional compressed sensing reconstruction algorithms, and achieves efficient and accurate image reconstruction, especially in the field of medical MRI.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-28
- Publication Date
- 2026-03-20
AI Technical Summary
Traditional iterative optimization compressed sensing reconstruction algorithms have high time complexity and poor reconstruction results at low sampling rates. Deep learning-based compressed sensing models offer limited improvement in image reconstruction performance.
An image compression sensing method based on Transformer generative adversarial networks is adopted. By constructing a deep generative adversarial network that includes a sampling network, a generation network, and a discrimination network, the quality of image reconstruction is improved by utilizing the attention mechanism of Transformer Block and sub-pixel convolutional blocks.
It significantly improves image reconstruction quality, shortens medical imaging time, enhances imaging speed and quality, preserves image detail information, and is suitable for the medical MRI field.
Smart Images
Figure CN116228520B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image processing, and particularly relates to an image compressive sensing reconstruction method and system based on a Transformer generative adversarial network. BACKGROUND
[0002] With the advent of the big data information era, the shortcomings of relying on the traditional sampling theorem for data sampling become increasingly apparent. As an advanced data sampling theory, compressive sensing is based on the compressibility of signals and realizes the reconstruction of signals from low-dimensional measurements. However, the traditional iterative optimization compressive sensing reconstruction algorithm has high time complexity and the reconstruction effect is not ideal at a low sampling rate. With the development of deep learning, the proposal of a compressive sensing model based on deep learning greatly reduces the time complexity of the reconstruction algorithm and improves the reconstruction effect. SUMMARY
[0003] In view of the problems in the prior art, the application provides an image compressive sensing reconstruction method and system based on a Transformer generative adversarial network, which uses a Transformer Block based on an attention mechanism to construct a deep generative adversarial network, thereby significantly improving the quality of the reconstructed image.
[0004] To achieve the above-mentioned purposes, the application adopts the following technical solutions:
[0005] The application provides an image compressive sensing reconstruction method based on a Transformer generative adversarial network, comprising the following steps:
[0006] An image sample set is obtained, which is divided into a training set and a test set in proportion, and the images are preprocessed;
[0007] A Transformer generative adversarial network model is constructed according to a sampling rate;
[0008] The hyperparameters of the Transformer generative adversarial network model are set, and a loss function and an optimization method are selected;
[0009] The image dataset is used to train the network model at different sampling rates, the optimal parameters of the network model are trained and learned through the loss function and the optimization method, and the Transformer generative adversarial network model trained at different sampling rates is obtained;
[0010] The Transformer generative adversarial network model trained is used for image compressive sensing reconstruction, and an evaluation index is used to verify the performance of the network.
[0011] Further, before training, the image size is adjusted to 64x64 pixels for each batchsize.
[0012] Further, the Transformer generative adversarial network model comprises a sampling network, a generation network and a discrimination network.
[0013] Further, the sampling network generates the measurement value by using a convolution layer with a convolution kernel size of 32*32 and a step of 32, and the output channel number of the convolution layer is set according to the sampling rate.
[0014] Further, the generation network comprises a Flatten layer, a full connection layer, a first layer hidden layer and a second layer hidden layer, the measurement value generated by the sampling network is first flattened into one dimension by the Flatten layer, then the nodes are expanded to 24567 by the full connection layer, and then the output of the full connection layer is adjusted to a specified image size by a reshape operation; the first layer hidden layer is a Transformer Block, wherein the Transformer Block comprises an original image and a corresponding position encoding, a PixelNorm normalization layer, a multi-head self-attention mechanism, a PixelNorm normalization layer and an MLP layer; the second layer hidden layer is a sub-pixel convolution block, and the sub-pixel convolution block comprises a convolution layer with a size of 3*3, a batch normalization layer, a SELU activation function layer, a sub-pixel convolution layer and a SELU activation function layer.
[0015] Further, the discrimination network judges whether the image generated by the generation network is true or not, and comprises a plurality of convolution layers, a batch normalization layer, a Flatten layer and a full connection layer.
[0016] Further, the hyperparameters of the Transformer generative adversarial network model are set, including setting an initial learning rate as 0.001 and setting the number of network iterations as 20 times.
[0017] Further, the parameters of the generation network are trained and updated by using an Adam optimization algorithm, and the parameters of the discrimination network are trained and updated by using an RMSProp optimization algorithm.
[0018] Further, the performance of the network is verified by using evaluation indexes of a peak signal-to-noise ratio (PSNR) and a structural similarity (SSIM).
[0019] The application further provides an image compressive sensing reconstruction system based on a Transformer generative adversarial network.
[0020] An image sample set acquisition module is configured to acquire an image sample set, divide the image sample set into a training set and a test set according to a proportion, and pre-process the image.
[0021] A network model construction module is configured to construct a Transformer generative adversarial network model according to a sampling rate.
[0022] The super parameter setting module is used for setting the super parameters of the Transform generative adversarial network model, selecting a loss function and an optimization method;
[0023] The training module is used for training the network model at different sampling rates using an image data set, training the optimal parameters of the learning network model through the loss function and the optimization method, and obtaining the trained Transform generative adversarial network model at different sampling rates.
[0024] The image reconstruction module is used for image compressive sensing reconstruction using the trained Transform generative adversarial network model, and verifying the performance of the network using an evaluation index.
[0025] Compared with the prior art, the present application has the following advantages:
[0026] 1. The present application uses a convolutional layer as a sampling network to simulate the measurement of an image after a traditional compressive sensing measurement process, thereby improving the correlation between the measurement value and the image. Compared with the traditional block-based compressive sensing algorithm, the present application solves the image block effect problem existing in the prior art based on block sampling. The measurement value is subjected to a full connection layer and a reshape operation of the convolutional neural network, thereby completing the preliminary reconstruction from the measurement vector to the original image. A deep generative adversarial network is constructed using a Transform Block based on an attention mechanism, the preliminary reconstructed image is fed into the deep generative adversarial network, the attention mechanism in the Transform Block can obtain the global information of the initial reconstructed image, and the content dependency between the initial reconstructed image and the attention weight is ensured, the receptive field of the network is increased to capture more context information, and the quality of the reconstructed image is iteratively improved through the generative adversarial manner.
[0027] 2. The image compressive sensing reconstruction method of the present application can be applied to the medical MRI field. Compared with the traditional scanning method, the imaging speed is greatly accelerated, the imaging quality is improved, and the scanning time is shortened. At a low time cost, accurate and efficient images are obtained, and the image detail information is preserved; while reducing the scanning time of patients, it is also beneficial to rapid diagnosis by doctors. BRIEF DESCRIPTION OF DRAWINGS
[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0029] Figure 1is a flowchart of an image compressive sensing reconstruction method based on a Transformer generative adversarial network according to an embodiment of the present application;
[0030] Figure 2 is a structural diagram of a Transformer generative adversarial network model according to an embodiment of the present application;
[0031] Figure 3 is a structural diagram of a sampling network and a generative network according to an embodiment of the present application;
[0032] Figure 4 is a structural diagram of a discriminative network according to an embodiment of the present application. DETAILED DESCRIPTION
[0033] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.
[0034] As shown in Figure 1 , the present embodiment provides an image compressive sensing reconstruction method based on a Transformer generative adversarial network, comprising the following steps:
[0035] Step S1, an image sample set is obtained, which is divided into a training set and a test set according to a proportion, and the images are preprocessed.
[0036] Step S2, a Transformer generative adversarial network model is constructed according to a sampling rate.
[0037] Step S3, the hyperparameters of the Transformer generative adversarial network model are set, and a loss function and an optimization method are selected.
[0038] Step S4, the network model is trained under different sampling rates using an image data set, the optimal parameters of the network model are trained and learned through the loss function and the optimization method, and the trained Transformer generative adversarial network model under different sampling rates is obtained.
[0039] Step S5, the trained Transformer generative adversarial network model is used for image compressive sensing reconstruction, and an evaluation index is used to verify the performance of the network.
[0040] In step S1, 202599 images of size 178x218 of CelebA in the public training dataset are divided into 162770 as a training set, 19867 as a validation set, and 19962 as a test set. Before training, the image size is adjusted to 64x64 pixels per batch size.
[0041] Further, the Transformer generative adversarial network model in step S2 includes a sampling network, a generation network, and a discrimination network, as shown in Figure 2
[0042] As shown in Figure 3 , the sampling network generates measurement values using a convolutional layer with a kernel size of 32x32 and a step size of 32, and the output channel number is set according to the sampling rate. For an image of size MxN, a convolutional layer with a kernel size of BxBxl and a step size of BxB is used to simulate the sampling operation, and the final measurement value has a size of
[0043] As shown in Figure 3 , the generation network includes a Flatten layer, a fully connected layer, a first hidden layer, and a second hidden layer. The measurement values generated by the sampling network are first flattened into one dimension by the Flatten layer, then expanded to 24567 nodes by the fully connected layer, and then reshaped to a feature map with an image size of 8*8 and a channel number of 384.
[0044] The first hidden layer is a Transformer Block, which uses the Transformer Block to reconstruct the image with high quality. The Transformer Block includes the original image and its corresponding position encoding, a PixelNorm normalization layer, a multi-head self-attention mechanism, a PixelNorm normalization layer, and an MLP layer. Its structure can be represented as: [Embedded Patches-Pixel Norm-Multi head self Attention-Pixel Norm-MLP], and the output channel number is 384. The multi-head self-attention mechanism in the Transformer Block can block the image and extract the global information of the image within the block. The attention mechanism can capture more context information and calculate the weight value of the texture complexity of each part of the image, and allocate computing resources according to the weight value. By matching sub-pixel convolution, the image size is gradually expanded, and the reconstruction quality is continuously improved through generative adversarial.
[0045] The second layer hidden layer is a sub-pixel convolution block. The sub-pixel convolution block is used for multiple image size promotion. The sub-pixel convolution block includes a 3*3 convolution layer, a batch normalization layer, a SELU activation function layer, a sub-pixel convolution layer and a SELU activation function layer, and the structure can be represented as [Conv 3×3 -BN-SeLU-subpixelConv 3×3 ]. The output image size of the Transformer Block is 8*8, and the output channel number is 384. After sub-pixel convolution, the image size becomes 16*16, and the channel number becomes 96. After multiple sub-pixel convolution and Transformer Block combination, the image size becomes 64*64, and the channel number becomes 6. After a convolution layer, the channel number of the final reconstructed image is 3, and the image size is 64*64.
[0046] As shown in Figure 4 , the discrimination network judges whether the image generated by the generation network is true or not. The discrimination network includes multiple convolution layers, batch normalization layers, Flatten layers and fully connected layers, and the structure can be represented as [Conv 3×3 -LreLU-BN-…-Conv 3×3 -BN-Flatten-DenseLayer]. The channel number of the input image is 3, and then the channel number is increased to 512 through multiple convolution layers. Then, the channel number is further expanded to 1024 through a fully connected layer, and finally, a true / false result is output. Through mutual confrontation between the generation network and the discrimination network, iterative training is performed. The discrimination network can help the generation network to better optimize parameters, thereby improving the quality of the reconstructed image.
[0047] In step S3, the images in the training set are input into the network model established in step S2 according to the batchsize. The appropriate batchsize is set according to the hardware condition. In this example, the batchsize is set to 16. The hyperparameters of the Transformer generative adversarial network model are set as follows: the initial learning rate is set to 0.001, and the network iteration number is set to 20.
[0048] (1) The target loss function of the generation network is set as follows:
[0049]
[0050] Wherein, n is the number of training images in the training set, x i is the original image, is the reconstructed image output by the generation network. The Adam optimization algorithm is used to train and update the parameters of the generation network.
[0051] (2) The target loss function of the discrimination network is set as:
[0052]
[0053] wherein n is the number of training images in the training set, x i is the original image, is the reconstructed image output by the generation network. The parameters of the discrimination network are trained and updated by using the RMSProp optimization algorithm.
[0054] Specifically, step S4 specifically includes the following steps:
[0055] Step S401, the number of channels of the sampling convolution layer is set according to the sampling rate.
[0056] Step S402, the trained model is saved in the format of.npz.
[0057] Specifically, in step S5, the evaluation indexes peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to verify the performance of the network, and the specific steps include the following steps:
[0058] Step S501, select an image in the test set and input it into the trained Transformer generative adversarial network model, obtain a measurement value through the sampling network, and input the measurement value into the generation network to finally output a reconstructed image.
[0059] Step S502, PSNR is used to measure the reconstruction effect of the network model, wherein the larger the PSNR is, the better the reconstruction effect is, and the calculation formula is as follows:
[0060] Mean square error (MSE):
[0061]
[0062] MSE represents the mean square error of the current image and the reference image f(i,j); M and N are the height and width of the image, respectively.
[0063] Peak signal-to-noise ratio (PSNR):
[0064]
[0065] n is the number of bits per pixel, generally taking 8, that is, the pixel gray scale is 256, and the unit is dB.
[0066] Step S503, SSIM is used to measure the reconstruction effect of the network model, wherein the larger the SSIM is, the better the reconstruction effect is, and for a given image x and y, the SSIM calculation formula of the two images is as follows:
[0067]
[0068] wherein μ x is the mean of x, μ y is the mean of y, is the variance of x, is the variance of t, σ xy is the covariance of x, y, c1=(k1L) 2 ,c2=(k2L) 2 is a constant used to maintain stability, L is the dynamic range of pixel values, k1=0.01, k2=0.03.
[0069] The network model used by the present application has an average increase of 2.0018dB in PSNR and an average increase of 0.0609 in SSIM on the MNIST data set compared with the sub-pixel convolutional adversarial neural network SCGAN. On the Fashion-MNIST data set, the average increase in PSNR is 1.0031dB, and the average increase in SSIM is 0.0513. On the CelebA data set, the average increase in PSNR is 1.2301dB, and the average increase in SSIM is 0.1123. The experimental results show that the present method has a better reconstruction effect than the current advanced deep compressive sensing algorithm.
[0070] Corresponding to the above-mentioned image compressive sensing reconstruction method based on the Transformer generative adversarial network, the present embodiment also provides an image compressive sensing reconstruction system based on the Transformer generative adversarial network, comprising an image sample set acquisition module, a network model construction module, a hyperparameter setting module, a training module and an image reconstruction module.
[0071] The image sample set acquisition module is used to acquire an image sample set, divide the image sample set into a training set and a test set according to a proportion, and pre-process the image;
[0072] The network model construction module is used to construct a Transformer generative adversarial network model according to a sampling rate;
[0073] The hyperparameter setting module is used to set the hyperparameters of the Transformer generative adversarial network model, select a loss function and an optimization method;
[0074] The training module is used to train the network model under different sampling rates using the image data set, train the optimal parameters of the network model through the loss function and the optimization method, and obtain the trained Transformer generative adversarial network model under different sampling rates;
[0075] An image reconstruction module is configured to perform image compressive sensing reconstruction using the trained GAN model, and to verify the performance of the network using evaluation indexes.
[0076] It should be noted that, in this document, the terms "comprise", "comprising", or any other variant thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements, but can also include other elements not expressly listed, or inherent to such processes, methods, articles, or apparatuses.
[0077] Finally, it should be noted that the above description is only a preferred embodiment of the present application, and is only used to illustrate the technical solutions of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. An image compressed sensing reconstruction method based on Transformer generative adversarial networks, characterized in that, Includes the following steps: S1: Obtain an image sample set, divide it into a training set and a test set according to the ratio, and preprocess the images; S2: Based on the sampling rate, construct a Transformer generative adversarial network model, which includes a sampling network, a generation network, and a discrimination network. The sampling network uses convolutional layers with a kernel size of 32×32 and a stride of 32 to generate measurement values, and the number of output channels is set according to the sampling rate; for a size of The image was processed using a convolution kernel with a size of [missing information]. Step size is The convolutional layer simulates the sampling operation, and the final measured value is [value missing]. ; The generative network includes a Flatten layer, a fully connected layer, a first hidden layer, and a second hidden layer. The measurements generated by the sampling network are first flattened to one dimension by the Flatten layer, then expanded to 24567 nodes by the fully connected layer, and then the output of the fully connected layer is adjusted to the specified image size by the reshape operation. The first hidden layer is a Transformer Block, which contains the original image and its corresponding position encoding, a PixelNorm normalization layer, a multi-head self-attention mechanism, a PixelNorm normalization layer, and an MLP layer. The second hidden layer is a subpixel convolutional block, which includes a 3×3 convolutional layer, a batch normalization layer, a SELU activation function layer, a subpixel convolutional layer, and a SELU activation function layer. The discrimination network determines whether the image generated by the generator network is real, and includes multiple convolutional layers, batch normalization layers, Flatten layers, and fully connected layers. S3: Set the hyperparameters of the Transformer generative adversarial network model, and select the loss function and optimization method; S4: Train the network model using an image dataset at different sampling rates, and learn the optimal parameters of the network model through loss function and optimization method to obtain the Transformer generative adversarial network model trained at different sampling rates; S5: Use the trained Transformer generative adversarial network model for image compressed perceptual reconstruction, and use evaluation metrics to verify the network's performance.
2. The image compressed sensing reconstruction method based on Transformer generative adversarial networks according to claim 1, characterized in that, Before training, each batch size resizes the images to 64×64 pixels.
3. The image compressed sensing reconstruction method based on Transformer generative adversarial networks according to claim 1, characterized in that, Setting the hyperparameters of the Transformer generative adversarial network model includes: setting the initial learning rate to 0.001 and the number of network iterations to 20.
4. The image compressed sensing reconstruction method based on Transformer generative adversarial networks according to claim 1, characterized in that, The parameters of the generator network are trained and updated using the Adam optimization algorithm, and the parameters of the discriminator network are trained and updated using the RMSProp optimization algorithm.
5. The image compressed sensing reconstruction method based on Transformer generative adversarial networks according to claim 1, characterized in that, The performance of the network was verified using evaluation metrics such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).
6. An image compressed sensing reconstruction system based on Transformer generative adversarial networks, characterized in that, For implementing the method as described in any one of claims 1-5, the system comprises: The image sample set acquisition module is used to acquire an image sample set, divide it into a training set and a test set according to a certain ratio, and preprocess the images. The network model construction module is used to construct a Transformer generative adversarial network model based on the sampling rate. The hyperparameter setting module is used to set the hyperparameters of the Transformer generative adversarial network model, and to select the loss function and optimization method; The training module is used to train the network model using an image dataset at different sampling rates. It learns the optimal parameters of the network model through loss functions and optimization methods, and obtains the Transformer generative adversarial network model trained at different sampling rates. The image reconstruction module is used to perform compressed-sensory reconstruction of images using a trained Transformer generative adversarial network model, and uses evaluation metrics to verify the network's performance.