Image super-resolution reconstruction method based on SAUGAN network

Abstract

The invention discloses an image super-resolution reconstruction method based on a SAUGAN network, and relates to the field of image and machine vision technology of artificial intelligence. The invention integrates a GAN network, a self-attention mechanism, an RRDB+ module, and a U‑Net architecture. A multi-scale feature extraction module is designed in the shallow feature extraction part of the generator, which can retain more fine details of the original image; an RRDB+ module is designed in the deep feature extraction part of the generator, and combined with a self-attention mechanism module, the characteristics of the residual network are fully utilized, the global information of the image is paid attention to, and the global feature information is better utilized for super-resolution image reconstruction, so as to obtain an image with better perceived quality; the discriminator adopts a U‑Net architecture to provide global and local feedback to the generator, improve the local details of the image, and maintain the global and local sense of reality. And spectral normalization regularization is used to stabilize the training process, and finally a good effect of local enhancement and suppression of artifacts is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image and machine vision technology, and specifically to an image super-resolution reconstruction method based on a SAUGAN network. Background Art

[0002] Image super-resolution (SR) reconstruction is a key technology in image processing and an important part of computer vision. It aims to generate high-resolution (HR) images containing high-frequency details from low-resolution (LR) images. Super-resolution reconstruction technology has been widely used in many fields, such as remote sensing technology, biometric imaging, public security, and medical imaging. For example, in the medical field, obtaining higher quality images can help doctors accurately detect diseases. Therefore, the study of image super-resolution reconstruction is of great significance to both academia and industry.

[0003] Image super-resolution reconstruction technology is generally divided into two categories: traditional methods and deep learning methods. Among the traditional methods, the most commonly used are interpolation-based, reconstruction-based, and learning-based methods. However, due to the fast speed and simple calculation of traditional interpolation methods, they usually lead to blurred edges and textures and artifacts. Compared with interpolation-based super-resolution reconstruction methods, reconstruction-based and learning-based super-resolution reconstruction methods can better highlight the texture details of the image, and the reconstructed image quality is also better, but the amount of calculation is greatly increased, the convergence speed is slow, and it cannot be well applied to real-life scenes.

[0004] In recent years, deep learning has developed rapidly and has achieved excellent performance in the field of image super-resolution reconstruction, mainly by two mainstream methods: convolutional neural networks (CNNs) and generative adversarial networks (GANs). For example, Dong et al. proposed a pixel mapping method based on super-resolution convolutional neural network (SRCNN) in 2014. This method only uses three convolutional layers and obtains stronger learning ability than some popular machine learning methods in image super-resolution. Although SRCNN has good SR effect, it still has the problems of shallow structure and high complexity; then some improved networks based on SRCNN are increasing, such as the VDSR network introduces the residual idea and the idea of ​​adaptive gradient clipping to increase convergence stability, while using high learning rate to accelerate convergence, and using gradient clipping to prevent gradient explosion. However, an important disadvantage of VDSR is that the image is interpolated and enlarged before entering the network, which increases the network parameters and the amount of calculation, requiring more computing time and storage space; in order to solve the above problems, generative adversarial networks (GANs) are proposed to generate clearer and more realistic samples through game confrontation. Christian et al. proposed a generative adversarial network (SRGAN) for image super-resolution in 2017. However, the images generated by the SRGAN model are still not clear, and when the statistical data of the training set and the test set are very different, the batch normalization layer will limit the generalization ability of the model, thereby introducing artifacts; the ESRGAN network is based on the SRGAN network, removes the batch normalization layer, and uses the residual dense block (RRDB) to replace the basic block of SRGAN, thereby mining more detailed information of the image and further improving the quality of image generation. It has achieved good results in the image super-resolution reconstruction task, but the ESRGAN model introduces additional information while generating rich texture details. In summary, in the field of applied technology, the super-resolution reconstruction task still has some limitations:

[0005] (1) The problem of small-scale texture detail information being lost during the reconstruction process. There is a lot of small-scale texture detail information in the image, and this information often plays a key role. However, in convolutional neural networks, texture detail information is concentrated in the low-level network. Existing algorithms mostly use high-level semantic information in the deep layer of the network for reconstruction, and do not fully utilize the feature information in the low-level network. Therefore, how to enhance the reconstruction effect of texture detail information in the image is one of the difficulties;

[0006] (2) Natural images have richer color information points and the entire image contains valid information, while some images are grayscale images with meaningless black backgrounds and valid information distributed in certain areas. Therefore, there are large differences in the importance of information contained in each channel and each spatial area of ​​the feature map extracted from the grayscale image. That is, the enhancement effect is not obvious on grayscale images with less texture detail information, and there is a lack of capture of the long-range dependencies necessary for areas with complex anatomical structures.

[0007] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0008] The object of the present invention is to provide an image super-resolution reconstruction method based on a SAUGAN network to solve the technical problems described in the background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: an image super-resolution reconstruction method based on a SAUGAN network, comprising at least the following steps:

[0010] S1: Batch data preprocessing of collected computed tomography MRI data;

[0011] S2: Divide the data into non-overlapping data sets;

[0012] S3: Designing loss functions for super-resolution reconstruction;

[0013] S4: Design a super-resolution reconstruction network based on the SAUGAN network;

[0014] S5: Train the super-resolution reconstruction model and test it. The model training is completed.

[0015] Preferably, the S1 at least comprises the following steps:

[0016] Image segmentation, the data image is in NIFTI format, and the image size is 240×240×N 3D image, so it is segmented along the horizontal plane;

[0017] Image selection: select brain MR images from slices 44 to 73, which have complete brain structures and provide more detailed brain structure information;

[0018] Get a low-resolution image, take the original image as a high-resolution image with a size of 240×240, and downsample it by 4 times to get a low-resolution image with a pixel size of 60×60;

[0019] All original HR images are resized to 120×120 and LR images are resized to 30×30 to reduce system consumption.

[0020] Preferably, S2 at least includes the following steps: randomly shuffling and dividing the patients, selecting MR images of 500 subjects as a training set and MR images of 70 subjects as a test set, wherein the data of any patient cannot overlap between the training set and the test set.

[0021] Preferably, the loss function in S3 at least includes the following: (1) structural similarity loss; (2) downsampling loss; (3) edge loss;

[0022] Among them, the structural similarity loss and edge loss ensure that the SR image has a similar structure to the original HR image, and the downsampling loss ensures that the SR image has a similar intensity distribution to the input LR image.

[0023] The formula for structural similarity loss is:

[0024] in, and are the grayscale mean of the super-resolution reconstructed image and the original HR image, and Represent the grayscale variance of the super-resolution reconstructed image and the original HR image, and represents the grayscale covariance between two images, and are constants to avoid instability, is the SR image, is the original HR image;

[0025] The formula for downsampling loss is:

[0026] in, is the SR image after downsampling, is the input LR image, for Norm,Downsampling loss calculates the pixel-level loss between the SR image and the LR image, which can prevent the SR image from having intensity changes relative to the input LR image.

[0027] The formula for edge loss is:

[0028] in, for norm, Indicates that the Sobel operator is used to extract the edge of the SR image. Indicates that the Sobel operator is used to extract the edge of the HR image.

[0029] Fighting Loss:

[0030] According to the description of adversarial loss in ESRGAN, the standard discriminator is replaced by the relative average discriminator RaD, denoted as , the standard discriminator can be expressed as ,in is the sigmoid function, is the non-transformation discriminator output, then the expression of RaD is ,in represents the operation of averaging all fake data in a mini-batch, so the loss function of the discriminator is:

[0031]

[0032] The adversarial loss of the generator is:

[0033]

[0034] in, , represents the input LR image;

[0035] The formula for perceptual loss is:

[0036]

[0037] in and They respectively describe the dimensions of the respective feature maps in the VGG network, represents the j-th convolution before the i-th maxpooling layer in the VGG19 network, and the feature map obtained before activation, Represents the features of the SR image obtained after the generator reconstructs the input LR image;

[0038] Content loss:

[0039] In order to evaluate the 1-norm distance between the generated SR image and the original HR image, L1 loss is used as the content loss, and the loss term is:

[0040]

[0041] in, represents the SR image generated by the generator, represents the original HR image;

[0042] The loss function of the generator is composed of adversarial loss, perceptual loss, content loss, SSIM loss, downsampling loss and edge loss. Therefore, the total loss function of the generator can be expressed as:

[0043]

[0044] in, , , , as well as is the coefficient for balancing different loss terms.

[0045] Preferably, the S5 at least includes the following steps:

[0046] Training is performed using the joint loss function formulated in S3;

[0047] Use the generative adversarial network to update the model weights during the training process by back-propagating the gradient;

[0048] Finally, after the model training is completed, the super-resolution reconstruction effect is evaluated on the test set.

[0049] Preferably, the steps of calculating the super-resolution reconstruction evaluation index involved in S5 are as follows: using peak signal-to-noise ratio, structural similarity (and no-reference image quality evaluation index) for quantitative evaluation;

[0050] PSNR is the most common and widely used objective image evaluation indicator, which is based on the error between pixels and is defined as follows:

[0051]

[0052] in, The number of bits representing the image pixels, MSE, is the mean square error. The lower the value, the higher the PSNR value;

[0053] MSE is shown as:

[0054]

[0055] in and They represent the SR reconstructed image with a height of H and a width of W and the real HR image at the pixel point The grayscale value at , the unit of PSNR is dB, the larger the PSNR value is, the smaller the distortion of the reconstructed image is, and the better the quality is.

[0056] The value range of SSIM is [0,1]. The closer the value is to 1, the smaller the image distortion is and the more similar the two images are.

[0057] The NIQE indicator uses the distance between the NSS feature model and the MVG extracted from the distorted image features to represent the quality of the distorted image. The lower the value, the better the image quality:

[0058] in, Represent the mean vector and covariance matrix of the natural MVG model and the distorted image MVG model respectively.

[0059] Preferably, the training super-resolution reconstruction model in S5 includes at least two parts: a generator and a discriminator. The generator part includes three contents: shallow feature extraction, deep feature extraction and upsampling. The main architecture of the discriminator part is a U-Net network.

[0060] The generator model consists of a multi-scale feature extraction module, a self-attention mechanism module, an RRDB+ module, and sub-pixel convolution. The shallow feature extraction part mainly uses the multi-scale feature extraction module to calculate shallow features at different scales through different convolution kernels, and then connects the channel numbers of the three scales into a feature vector. The final result will be used as the input for the next step of deep feature extraction.

[0061] The deep feature extraction part includes at least a self-attention mechanism module and a RRDB+ module, a residual is added to the dense block of the RRDB module, the characteristics of the residual network are fully utilized to obtain an image with better perceptual quality, and the output of the self-attention mechanism module is jump-connected to the output of the RRDB+ module to form a residual connection;

[0062] The discriminator model is composed of a U-Net architecture. The downsampling encoder module undergoes four layers of downsampling to gradually reduce the input samples and capture the global image context, while the upsampling decoder uses progressive upsampling to match the output resolution to the input resolution, and uses spectral normalization regularization to stabilize the training process, ultimately achieving the effect of local enhancement and artifact suppression.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. The present invention integrates the GAN network, self-attention mechanism, RRDB+ module and U-Net architecture. A multi-scale feature extraction module is designed in the shallow feature extraction part of the generator, which can retain more fine details of the original image; the RRDB+ module is designed in the deep feature extraction part of the generator, and combined with the self-attention mechanism module, it fully utilizes the characteristics of the residual network, pays attention to the global information of the image, and better utilizes the global feature information to reconstruct the super-resolution image, so as to obtain images with better perceived quality; the discriminator adopts the U-Net architecture to provide global and local feedback to the generator, improve the local details of the image, and maintain the global and local sense of reality. And the spectral normalization regularization is used to stabilize the training process, and finally achieve the good effect of local enhancement and suppression of artifacts;

[0065] 2. The present invention designs a new loss function, which is composed of structural similarity loss, downsampling loss and edge loss, which ensures that the SR image has a similar structure to the original HR image, and also ensures that the SR image has a similar intensity distribution to the input LR image, so that the generated SR image has the same edge structure and texture information as the original HR image.

[0066] 3. The present invention can effectively reconstruct super-resolution images for both medical images and natural images. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0068] Figure 1 This is a flow chart of the overall image super-resolution reconstruction algorithm provided by the present invention.

[0069] Figure 2 Schematic diagram of the overall architecture of the generator in the SAUGAN network provided by the present invention.

[0070] Figure 3 Schematic diagram of the shallow feature extraction module in the generator provided by the present invention.

[0071] Figure 4 Schematic diagram of the deep feature extraction module in the generator provided by the present invention.

[0072] Figure 5 Schematic diagram of the self-attention mechanism module in the deep feature extraction module provided by the present invention.

[0073] Figure 6 This is a schematic diagram of the identifier module provided by the present invention.

[0074] Figure 7 This is a visualization comparison of the super-resolution results of different images on the IXI dataset provided by the present invention.

[0075] Figure 8 This is a visualization comparison diagram of the texture detail results of a small area of ​​an image provided by the present invention.

[0076] Fig. 9 This is a visual comparison chart of the module ablation experiment results provided by the present invention.

[0077] Fig.10 This is a visual comparison chart of the loss function ablation experiment results provided by the present invention. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0079] Embodiment 1:

[0080] See also Figure 1-Figure 10 , the image super-resolution reconstruction method based on the SAUGAN network at least includes the following steps:

[0081] S1: Batch data preprocessing of collected computed tomography MRI data;

[0082] S2: Divide the data into non-overlapping data sets;

[0083] S3: Designing loss functions for super-resolution reconstruction;

[0084] S4: Design a super-resolution reconstruction network based on the SAUGAN network;

[0085] S5: Train the super-resolution reconstruction model and test it. The model training is completed.

[0086] S1 includes at least the following steps:

[0087] Image segmentation, the data image is in NIFTI format, and the image size is 240×240×N 3D image, so it is segmented along the horizontal plane;

[0088] Image selection: select brain MR images from slices 44 to 73, which have complete brain structures and provide more detailed brain structure information;

[0089] Get a low-resolution image, take the original image as a high-resolution image with a size of 240×240, and downsample it by 4 times to get a low-resolution image with a pixel size of 60×60;

[0090] All original HR images were resized to 120×120 and LR images were resized to 30×30 to reduce system consumption;

[0091] S2 at least includes the following steps: randomly shuffling and dividing the patients, selecting MR images of 500 subjects as a training set and MR images of 70 subjects as a test set, wherein the data of any patient cannot overlap between the training set and the test set;

[0092] S3 proposes a new loss function to address the problem of unclear texture details and incomplete structure when low-resolution images are generated. This loss function retains the three loss functions in the ESRGAN model, namely adversarial loss, perceptual loss, and content loss. However, the loss function used in the traditional ESRGAN cannot guarantee that the generated SR image has the same structure and texture information as the original HR image. Therefore, while retaining the loss function in ESRGAN, a new set of loss functions is designed. This new set of loss functions includes at least the following: (1) structural similarity (SSIM) loss; (2) downsampling loss; (3) edge loss;

[0093] Among them, the structural similarity loss and edge loss ensure that the SR image has a similar structure to the original HR image, and the downsampling loss ensures that the SR image has a similar intensity distribution to the input LR image.

[0094] The formula for structural similarity loss is:

[0095] in, and are the grayscale mean of the super-resolution reconstructed image and the original HR image, and Represent the grayscale variance of the super-resolution reconstructed image and the original HR image, and represents the grayscale covariance between two images, and are constants to avoid instability, is the SR image, is the original HR image;

[0096] The formula for downsampling loss is:

[0097] in, is the SR image after downsampling, is the input LR image, for Norm,Downsampling loss calculates the pixel-level loss between the SR image and the LR image, which can prevent the SR image from having intensity changes relative to the input LR image.

[0098] The formula for edge loss is:

[0099] in, for norm, Indicates that the Sobel operator is used to extract the edge of the SR image. Indicates that the Sobel operator is used to extract the edge of the HR image.

[0100] Fighting Loss:

[0101] According to the description of adversarial loss in ESRGAN, the standard discriminator is replaced by the relative average discriminator RaD, denoted as , the standard discriminator can be expressed as ,in is the sigmoid function, is the non-transformation discriminator output, then the expression of RaD is ,in represents the operation of averaging all fake data in a mini-batch, so the loss function of the discriminator is:

[0102] The adversarial loss of the generator is:

[0103] in, , represents the input LR image;

[0104] The formula for perceptual loss is:

[0105] in and They respectively describe the dimensions of the respective feature maps in the VGG network, represents the feature map obtained before activation of the jth convolution before the i-th maxpooling layer in the VGG19 network, Represents the features of the SR image obtained after the generator reconstructs the input LR image;

[0106] Content loss:

[0107] In order to evaluate the 1-norm distance between the generated SR image and the original HR image, L1 loss is used as the content loss, and the loss term is:

[0108] in, represents the SR image generated by the generator, represents the original HR image;

[0109] The loss function of the generator is composed of adversarial loss, perceptual loss, content loss, SSIM loss, downsampling loss and edge loss. Therefore, the total loss function of the generator can be expressed as:

[0110] in, , , , as well as To balance the coefficients of different loss terms;

[0111] S5 at least includes the following steps:

[0112] Training is performed using the joint loss function developed by S3;

[0113] Use the generative adversarial network to update the model weights during the training process by back-propagating the gradient;

[0114] Finally, after the model training is completed, the super-resolution reconstruction effect is evaluated on the test set;

[0115] The steps for calculating the super-resolution reconstruction evaluation index involved in S5 are as follows: quantitative evaluation is performed using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and no-reference image quality evaluation index (NIQE);

[0116] PSNR is the most common and widely used objective image evaluation indicator, which is based on the error between pixels and is defined as follows:

[0117] in, Indicates the number of bits of image pixels, usually 8, that is, the number of pixel grayscale levels is 256. MSE is the mean square error. The lower the value, the higher the PSNR value.

[0118] MSE is shown as:

[0119] in and They represent the SR reconstructed image with a height of H and a width of W and the real HR image at the pixel point The grayscale value at , the unit of PSNR is dB, the larger the PSNR value is, the smaller the distortion of the reconstructed image is, and the better the quality is.

[0120] The value range of SSIM is [0,1]. The closer the value is to 1, the smaller the image distortion is and the more similar the two images are.

[0121] The NIQE indicator uses the distance between the NSS feature model and the MVG extracted from the distorted image features to represent the quality of the distorted image. The lower the value, the better the image quality:

[0122] in, Represent the mean vector and covariance matrix of the natural MVG model and the distorted image MVG model respectively;

[0123] The training super-resolution reconstruction model in S5 consists of at least two parts: the generator and the discriminator. The generator part contains three contents: shallow feature extraction, deep feature extraction, and upsampling. The overall architecture is shown in Figure 2 ; The main architecture of the discriminator is the U-Net network, and the overall architecture is shown in Figure 6 ;

[0124] The generator model consists of a multi-scale feature extraction module, a self-attention mechanism module, an RRDB+ module, and sub-pixel convolution. The shallow feature extraction part mainly consists of the multi-scale feature extraction module calculating shallow features at different scales through different convolution kernels, and then connecting the channels of the three scales into a feature vector. The final result will be used as the input for the next step of deep feature extraction.

[0125] The deep feature extraction part includes at least a self-attention mechanism module and an RRDB+ module. A residual is added to the dense block of the RRDB module to make full use of the characteristics of the residual network to obtain images with better perceptual quality. The output of the self-attention mechanism module is jump-connected to the output of the RRDB+ module to form a residual connection.

[0126] The discriminator model is composed of a U-Net architecture. The downsampling encoder module undergoes four layers of downsampling to gradually reduce the input samples and capture the global image context, while the upsampling decoder uses progressive upsampling to match the output resolution to the input resolution, and uses spectral normalization regularization to stabilize the training process, ultimately achieving the effect of local enhancement and artifact suppression.

[0127] Shallow feature extraction module structure:

[0128] The multi-scale feature extraction module performs shallow feature extraction on the image. The internal structure is as follows: Figure 3 , convolution kernels with sizes of 3, 5, and 7 are used to calculate shallow features at different scales. Figure 3 As shown in the figure, the feature extraction for each scale consists of two blocks, each block contains two convolutional layers, the number of channels is 64, the stride is 1, each convolutional layer is followed by a ReLU activation function, and a jump connection is added between the first block and the second block, so that the features of the first block and the features of the second block are added element by element. After extracting features at different scales, the number of channels at the three scales are connected to a feature vector, and the final result will be used as the input for the next step of deep feature extraction. Using multi-scale feature extraction can retain more fine details of the original image. The method of shallow feature extraction using multi-scale method can be expressed as:

[0129]

[0130]

[0131]

[0132] in, represents the input LR image, and Represent the convolution and ReLU activation functions respectively, It represents the output features of the first block after the convolution kernel size is 3, 5, and 7. It represents the output features of the second block after the convolution kernel size is 3, 5, and 7. represents pixel-by-pixel addition, represents the sum of the number of feature channels, Represents the output features of the multi-scale shallow feature extraction module;

[0133] The internal structure of the deep feature extraction module is as follows: Figure 4 , mainly by adding a residual to the dense block of the RRDB module. Make full use of the characteristics of the residual network to obtain images with better perceptual quality. Secondly, for convolution, the setting of the convolution kernel limits the size of the receptive field, resulting in the network often requiring multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is that its attention is global, and it can obtain the global spatial information of the feature map through simple query and assignment. Therefore, we also added a self-attention mechanism to the generator, which can better utilize the global feature information for super-resolution image reconstruction. After the convolution operation, the convolution feature is convolved by 1×1 convolution to obtain three branches, namely f(x), g(x) and h(x). The size of the feature map remains unchanged, the number of output channels of f(x) and g(x) changes, and the number of output channels of h(x) remains unchanged. Then the output of f(x) is transposed and multiplied with the output matrix of h(x). The attention maps are obtained by softmax normalization, and finally multiplied with the output of h(x), and the final feature map is obtained by 1×1 convolution. Among them,

[0134] For the model in the synthesis Region The output of the attention layer is expressed as:

[0135] Among them, C is the number of channels, N is the feature number position of the previous hidden layer feature,

[0136] In the above formula, , , and is the learned weight matrix, which is implemented by 1×1 convolution. In addition, the output of the attention layer is further multiplied by the scale parameter and added back to the input feature map, and the final output is:

[0137] in, is a pre-set coefficient, and its initial value is 0. The network first relies on clues from the local neighborhood, and then gradually learns to assign more weight to the non-local domain. The structure of the self-attention module is as follows Figure 5 The specific structure of the deep feature extraction part is shown in Figure 4 shown.

[0138] The internal structure of the identification module is as follows Figure 6 ,Since the discriminator needs stronger discrimination ability for complex training outputs, we use the U-Net network as the discriminator to provide global and local feedback to the generator, improve the local details of the image, and maintain the global and local realism. Figure 6 As shown in the figure, our discriminator network consists of a downsampling encoder module and an upsampling decoder module. The downsampling encoder module undergoes four layers of downsampling. The first layer is convolution and LReLU activation function. The second to fourth layers are convolution, LReLU activation function and spectral normalization. It gradually reduces the input samples and captures the global image context. The upsampling decoder uses progressive upsampling, which matches the output resolution to the input resolution, just like the encoder structure. Finally, a good effect of local enhancement and suppression of artifacts is achieved.

[0139] Embodiment 2:

[0140] The present invention takes brain MR images as an example. After data preprocessing, 15,000 2D MR brain images are obtained as a training set, and 2,100 2D MR brain images are used as a test set. The following indicators are used to evaluate the segmentation model when training and testing the super-resolution reconstruction performance of the model: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and natural image quality evaluation (NIQE).

[0141] The SAUGAN algorithm is compared with a variety of excellent super-resolution reconstruction algorithms. The final results on the public dataset IXI are shown in Table 1. The visual comparison effect is shown in Figure 7 In addition, we also conducted comparative experiments on each module. The comparison results are shown in Table 2 and visualized as follows: Figure 8 As shown; Finally, the comparison is made from the perspective of loss function, as shown in Table 3, and the visualization results are as follows Fig. 9 All experiments show that the SAUGAN network has the best super-resolution reconstruction performance in this implementation case, which improves the overall quality of the image while maintaining image details.

[0142] Table 1 Model comparison results on dataset IXI (bold indicates the best, underline indicates the second best)

[0143] Table 2 Comparison results of module ablation experiments

[0144] Table 3. Comparison results of loss function ablation experiments

[0145] In response to the problems that the super-resolution images generated by most super-resolution reconstruction models are not clear, the texture details are not prominent, the edge contours are blurred, and the generated artifacts introduce additional information, this patent proposes a SAUGAN network for image super-resolution reconstruction, which is mainly composed of two parts: a generator and a discriminator. On the one hand, the proposed multi-scale feature extraction module, self-attention mechanism, and RRDB+ network are combined as a generator to focus on the global information of the image during the feature extraction process to obtain images with higher perceptual quality; on the other hand, U-net is used to form a discriminator network to provide detailed pixel-by-pixel feedback to the generator for distinguishing between the original HR image and the reconstructed SR image, ultimately achieving good results in local enhancement and suppression of artifacts. In addition, this patent also proposes a new loss function to make the texture details of the SR image closest to the original HR image.

[0146] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An image super-resolution reconstruction method based on a SAUGAN network, characterized by: At least the following steps are included: S1: Batch data preprocessing of collected computed tomography MRI data; S2: Divide the data into non-overlapping data sets; S3: Designing loss functions for super-resolution reconstruction; The loss function in S3 at least includes the following: structural similarity loss; downsampling loss; edge loss; Among them, the structural similarity loss and edge loss ensure that the SR image has a similar structure to the original HR image, and the downsampling loss ensures that the SR image has a similar intensity distribution to the input LR image; The formula for structural similarity loss is: ， in, and are the grayscale mean of the super-resolution reconstructed image and the original HR image, and Represent the grayscale variance of the super-resolution reconstructed image and the original HR image, and represents the grayscale covariance between two images, and are constants to avoid instability, is the SR image, is the original HR image; The formula for downsampling loss is: ， in, is the SR image after downsampling, is the input LR image, for Norm,downsampling loss calculates the pixel-level loss between the SR image and the LR image,,which can prevent the SR image from having intensity changes relative to the input LR image; The formula for edge loss is: , in, for norm, Indicates that the Sobel operator is used to extract the edge of the SR image. Indicates that the Sobel operator is used to extract the edge of the HR image; Fighting Loss: According to the description of adversarial loss in ESRGAN, the standard discriminator is replaced by the relative average discriminator RaD, denoted as , the standard discriminator is expressed as ,in is the sigmoid function, is the non-transformation discriminator output, then the expression of RaD is: ， in represents the operation of averaging all fake data in a mini-batch, so the loss function of the discriminator is: ， The adversarial loss of the generator is: ， in, , represents the input LR image; The formula for perceptual loss is: ， in and They respectively describe the dimensions of the respective feature maps in the VGG network, represents the feature map obtained before activation of the jth convolution before the i-th maxpooling layer in the VGG19 network, Represents the features of the SR image obtained after the generator reconstructs the input LR image; Content loss: In order to evaluate the 1-norm distance between the generated SR image and the original HR image, L1 loss is used as the content loss, and the loss term is: ， in, represents the SR image generated by the generator, represents the original HR image; The loss function of the generator is composed of adversarial loss, perceptual loss, content loss, SSIM loss, downsampling loss and edge loss. Therefore, the total loss function of the generator is expressed as: ， in, , , , as well as To balance the coefficients of different loss terms; S4: Design a super-resolution reconstruction network based on the SAUGAN network; S5: Train the super-resolution reconstruction model and test it. The model training is completed. The S5 at least comprises the following steps: Training is performed using the joint loss function formulated in S3; Use the generative adversarial network to update the model weights during the training process by back-propagating the gradient; Finally, after the model training is completed, the super-resolution reconstruction effect is evaluated on the test set; The steps of calculating the super-resolution reconstruction evaluation index involved in S5 are as follows: using peak signal-to-noise ratio, structural similarity and no-reference image quality evaluation index for quantitative evaluation, where the no-reference image quality evaluation index is NIQE; PSNR is the most common and widely used objective image evaluation indicator, which is based on the error between pixels and is defined as follows: ， in, Indicates the number of bits of image pixels. MSE is the mean square error. The lower the value, the higher the PSNR value. MSE is shown as: ， in and They represent the SR reconstructed image with a height of H and a width of W and the real HR image at the pixel point The grayscale value at the pixel, the unit of PSNR is dB. The larger the PSNR value, the smaller the distortion of the reconstructed image and the better the quality. The value range of SSIM is [0,1]. If the value is closer to 1, it means that the image distortion is smaller and the two images are more similar; The NIQE indicator uses the distance between the NSS feature model and the MVG extracted from the distorted image features to represent the quality of the distorted image. The lower the value, the better the image quality: ， in, Represent the mean vector and covariance matrix of the natural MVG model and the distorted image MVG model respectively; The training super-resolution reconstruction model in S5 includes at least two parts: a generator and a discriminator. The generator part includes three contents: shallow feature extraction, deep feature extraction, and upsampling; the discriminator part is structured as a U-Net network; The generator model consists of a multi-scale feature extraction module, a self-attention mechanism module, an RRDB+ module, and sub-pixel convolution. The shallow feature extraction part is composed of a multi-scale feature extraction module that calculates shallow features at different scales through different convolution kernels, and then connects the channel numbers of the three scales into a feature vector. The final result will be used as the input for the next step of deep feature extraction. The deep feature extraction part includes at least a self-attention mechanism module and an RRDB+ module, a residual is added to the dense block of the RRDB module, the characteristics of the residual network are used to obtain an image with better perceptual quality, and the output of the self-attention mechanism module is jump-connected to the output of the RRDB+ module to form a residual connection; The discriminator model is composed of a U-Net architecture. The downsampling encoder module undergoes four layers of downsampling to gradually reduce the input samples and capture the global image context, while the upsampling decoder uses progressive upsampling to match the output resolution to the input resolution, and uses spectral normalization regularization to stabilize the training process, ultimately achieving the effect of local enhancement and artifact suppression.

2. The image super-resolution reconstruction method based on the SAUGAN network according to claim 1, characterized in that: The S1 at least comprises the following steps: Image segmentation, the data image is in NIFTI format, and the image size is 240×240×N 3D image, so it is segmented along the horizontal plane; Image selection: select brain MR images from slices 44 to 73, which have complete brain structures and provide more detailed brain structure information; Get a low-resolution image, take the original image as a high-resolution image with a size of 240×240, and downsample it by 4 times to get a low-resolution image with a pixel size of 60×60; All original HR images are resized to 120×120 and LR images are resized to 30×30 to reduce system consumption.

3. The image super-resolution reconstruction method based on the SAUGAN network according to claim 1, characterized in that: The S2 at least includes the following steps: randomly shuffling and dividing the patients, selecting MR images of 500 subjects as a training set and MR images of 70 subjects as a test set, wherein the data of any patient cannot overlap between the training set and the test set.

Images