Image super-resolution method based on novel network convolution architecture
By introducing a new network convolutional architecture and layer normalization module into image super-resolution networks, the problem of noise and degradation model processing in synthetic aperture radar images is solved, and a higher quality image reconstruction and more efficient training process is achieved.
Patent Information
- Application Number
- CN202411438542.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-03-14
AI Technical Summary
The prior art is difficult to effectively process the coherent spot noise and complex degradation models inherent in synthetic aperture radar images, resulting in insufficient clarity in texture details and edge information of the reconstructed image.
Using an image super-resolution network based on a new network convolutional architecture, the network's expression ability and image reconstruction quality are enhanced by introducing a new convolutional network structure into the generator and discriminator. Specific measures include using a new layer normalization module to replace BatchNorm, introducing deep convolutional layers, layer normalization, linear convolutional layers and residual connection modules, and improving the capabilities of feature extraction and image reconstruction.
The detail recovery ability of synthetic aperture radar images is significantly improved, and the reconstructed images have significantly improved in clarity and detail, and the training process is more efficient and adaptable.
Smart Images
Figure CN119624770A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and deep learning, and in particular to an image super-resolution method based on a novel network convolution architecture, which aims to improve the spatial resolution and detail restoration capability of an image. Background Art
[0002] Synthetic aperture radar is an active microwave remote sensing imaging technology that achieves high-resolution target detection by transmitting linear frequency modulation signals and processing the received echo signals. Synthetic aperture radar systems have all-day and all-weather observation capabilities and can penetrate clouds, vegetation and the ground surface. They are widely used in military reconnaissance, earth remote sensing, resource exploration, marine environment monitoring and disaster forecasting. The resolution of synthetic aperture radar images is a key indicator to measure the performance of the imaging system, which determines the ability to distinguish adjacent targets in the image.
[0003] However, the resolution of synthetic aperture radar images is often reduced due to a variety of factors, including but not limited to platform motion, meteorological interference, system noise, image blur, defocus, and geometric distortion. Although high-resolution images can provide richer detail information, due to the hardware limitations of the synthetic aperture radar system itself and the complex factors in the imaging process, synthetic aperture radar images often have blur and geometric distortion, resulting in actual image resolution lower than the theoretical resolution of the system. These problems make it extremely challenging to recover high-quality detail information from synthetic aperture radar images.
[0004] In recent years, deep learning technology has made significant progress in the field of image super-resolution reconstruction. Although traditional super-resolution methods perform well in processing optical images, they still face many challenges in processing synthetic aperture radar images. Existing super-resolution methods based on deep learning are usually unable to effectively deal with the coherent speckle noise and complex degradation models inherent in synthetic aperture radar images, resulting in unclear texture details and edge information in the reconstructed image.
[0005] There is a new type of convolutional network architecture that shows superior performance in visual tasks through optimized convolution operations and network design. Compared with traditional convolutional networks, this network has significant advantages in image feature extraction and representation capabilities, and can better capture image details and edge information. Combining the new convolutional network architecture and the technology of super-resolution generative adversarial networks, especially in the super-resolution task of processing synthetic aperture radar images, it is expected to significantly improve the effect of super-resolution reconstruction. Therefore, it is urgent to improve the existing technology to give full play to the advantages of this new convolutional network architecture and improve the super-resolution reconstruction effect of synthetic aperture radar images. Summary of the invention
[0006] The purpose of the invention is to improve the performance of super-resolution generative adversarial networks and solve the problem of poor super-resolution image generation effect in the prior art. The present invention provides a method for image super-resolution reconstruction based on an image super-resolution network based on a novel network convolutional architecture.
[0007] Technical solution: To solve the above technical problems, the present invention proposes an image super-resolution network based on a novel network convolution architecture. The network introduces a novel convolution network structure in the generator and discriminator to enhance the network's expression ability and image reconstruction quality. The specific implementation steps are as follows:
[0008] Step 1: Obtain a high-resolution synthetic aperture radar image dataset, and perform preprocessing and degradation processing on the image to generate its corresponding low-resolution image;
[0009] Step 2: Build an image super-resolution network, which uses the super-resolution generative adversarial network as the baseline model and improves it using a new network convolutional architecture;
[0010] Step 3: taking the synthetic aperture radar image as a high-resolution image and the corresponding generated degraded blurred image as a low-resolution image, constructing a data pair set, and inputting the data pair set into the improved image super-resolution network for training until the network model converges, so as to obtain a trained image super-resolution network model;
[0011] Step 4: Apply the trained image super-resolution network model, and input the image to be super-resolved into the test network for super-resolution processing to generate an image with higher resolution.
[0012] Furthermore, in step 1, the image preprocessing includes random cropping, grayscale processing, pixel value normalization processing, and converting the processed image into a PyTorch tensor for easy input into the neural network. The degradation processing uses the bicubic interpolation method to downsample the image to a low resolution to obtain the low-resolution image required for training.
[0013] Furthermore, in step 2, in the construction of the image super-resolution network based on the novel network convolutional architecture, the BatchNorm in the original network is replaced by a new layer normalization module, and normalization is performed independently on samples without relying on the statistical information of the entire batch.
[0014] Furthermore, in step 2, in the construction of the image super-resolution network based on the novel network convolution architecture, a new module is introduced into the network, and the new module includes the following parts:
[0015] Deep convolution layer: Use Conv2d to construct a deep convolution layer and perform spatial convolution operations to maintain the dimension of the feature map;
[0016] Layer Normalization: Use the new layer normalization module to normalize feature maps to enhance training stability and keep feature maps invariant in spatial dimensions;
[0017] Linear convolution layer: Use Conv2d to perform linear convolution operations to increase the number of channels of the feature map, then activate the feature map through the activation function GELU, and finally restore the number of feature map channels to the original dimension through Conv2d;
[0018] Residual connection: Add the input feature map to the feature map after the above processing to form a residual connection to improve the expressiveness of the model.
[0019] Furthermore, in step 2, in the construction of the image super-resolution network based on the novel network convolution architecture, the improvement of the generator network structure includes:
[0020] The new modules introduced in the multi-level use process at feature dimensions of 64, 128, 256, and 512 respectively to capture feature information at different levels;
[0021] In the upsampling module, PixelShuffle is used to rearrange pixels to improve the spatial resolution of the feature map, and the upsampled feature map is further processed by Conv2d and PReLU to restore the spatial resolution of the original image.
[0022] Furthermore, in step 2, in the construction of the image super-resolution network based on the novel network convolution architecture, the improvement of the discriminator network structure includes:
[0023] The initial convolutional layer uses Conv2d and a new layer normalization module to perform preliminary feature extraction and normalization on the input image to improve the discriminative ability;
[0024] The introduced new modules are applied at different levels to extract features of different scales through multiple downsampling operations to enhance the discrimination ability of high-resolution images;
[0025] The fully connected layer performs final classification through Linear and LeakyReLU to determine the authenticity of the image.
[0026] Furthermore, in step three, the synthetic aperture radar image super-resolution network training includes: inputting the images and related information in the data set into the improved generator network, and the improved discriminator network classifies the generated images to optimize the performance of the generator; during the training process, calculating the loss function according to the difference between the generated image and the real image, and updating the parameters of the generator and the discriminator according to back propagation, thereby optimizing the network parameters to reduce the reconstruction error.
[0027] Improved effect: Compared with the prior art, the present invention realizes effective reconstruction of synthetic aperture radar image super-resolution by combining the feature extraction capability of the novel convolutional network architecture with the advantages of the generative adversarial network, and has the following technical advantages:
[0028] 1) Higher reconstruction quality: This invention introduces a new convolutional network architecture. The optimized network structure significantly improves the detail recovery capability of synthetic aperture radar images, and the reconstructed images are significantly improved in clarity and detail.
[0029] 2) Powerful feature expression capability: The new convolutional network architecture enhances the model’s ability to capture complex textures and edges, helping to restore key information in blurred images.
[0030] 3) Efficient training process: The improved network structure improves training efficiency. Compared with traditional high-resolution generative adversarial networks, it can converge to the ideal state faster and reduce the consumption of computing resources.
[0031] 4) Strong adaptability: The present invention exhibits good adaptability in various synthetic aperture radar image types and is suitable for different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings of the present invention constitute a part of the specification, and are intended to provide readers with a deeper understanding of the present invention. These illustrative embodiments and descriptions do not constitute a limitation of the present invention.
[0033] Figure 1 is a flow chart of the steps of the method of the present invention;
[0034] Figure 2 It is the structural diagram of the new module introduced;
[0035] Figure 3 This is the structure diagram of the generator in the improved super-resolution network;
[0036] Figure 4 This is the discriminator structure diagram in the improved super-resolution network. DETAILED DESCRIPTION
[0037] The present invention is further described below by way of examples. It should be noted that the purpose of publishing the examples is to help further understand the present invention, but those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the examples, and the scope of the present invention to be protected shall be subject to the scope defined in the claims.
[0038] Embodiment: Super-resolution processing is performed on a low-resolution synthetic aperture radar image.
[0039] like Figure 1 As shown, the present invention proposes an image super-resolution method based on a novel convolutional network architecture. The method mainly includes the following steps:
[0040] Step 1: Obtain a high-resolution synthetic aperture radar image dataset, and perform preprocessing and degradation processing on the image to generate its corresponding low-resolution image;
[0041] Step 2: Build an image super-resolution network, which uses the super-resolution generative adversarial network as the baseline model and improves it using a new network convolutional architecture;
[0042] Step 3: taking the synthetic aperture radar image as a high-resolution image and the corresponding generated degraded blurred image as a low-resolution image, constructing a data pair set, and inputting the data pair set into the improved image super-resolution network for training until the network model converges, so as to obtain a trained image super-resolution network model;
[0043] Step 4: Apply the trained image super-resolution network model, and input the image to be super-resolved into the test network for super-resolution processing to generate an image with higher resolution.
[0044] In step 1, the synthetic aperture radar image dataset is preprocessed, and the high-resolution image is randomly cropped to a size of 256×256. The image is converted to grayscale mode, and then converted to a tensor, and the value range of the image is normalized from [0,255] to [0, 1]. After conversion to a tensor, the dimension of the image is converted from (H, W, C) to (C, H, W). Image degradation is performed on each image of the synthetic aperture radar image dataset, and the high-resolution image is reduced to 1 / 4 of the original size using bicubic interpolation, that is, the image size is adjusted to 64x64 pixels to generate a low-resolution image.
[0045] In step 2, an image super-resolution network based on a novel convolutional network architecture is constructed, where the super-resolution generative adversarial network is used as the baseline model. The improvements include:
[0046] First, a new layer normalization model is used instead of BatchNorm because the new layer normalization model is normalized independently by sample, while BatchNorm relies on the statistics of the entire batch. For the input tensor x, the mean and variance of each sample are first calculated. Assuming the input shape is [N, C, H, W] (batch size, number of channels, height, width), the calculation is performed on the feature dimension:
[0047] Mean:
[0048] variance:
[0049] Each feature is normalized using the calculated mean and variance to obtain the normalized output:
[0050]
[0051] in, is a small constant to avoid division by zero errors.
[0052] In order to maintain the expressiveness of the model, learnable parameters are introduced and :
[0053]
[0054] in, and They are used for scaling and offset respectively, so that the network can adapt to different tasks.
[0055] In this embodiment, the input tensor x is rearranged into a shape of [N, H, W, C] so that the new layer normalization model normalizes on the last dimension (the number of channels). After normalization, the tensor is converted back to its original shape [N, C, H, W] for subsequent network processing.
[0056] Secondly, new modules Block are introduced, such as Figure 2 As shown in the figure, the module includes: the input feature map is transformed through deep convolution, layer normalization and linear convolution layer, and then passes through the activation function, and is finally added to the original input feature map to form a residual connection.
[0057] In this embodiment, the deep convolution module uses Conv2d to construct a deep convolution layer, sets the number of input and output channels to dim, the convolution kernel size to 7x7, sets groups to dim, and applies the convolution kernel to each channel separately, thereby realizing deep separable convolution. The layer normalization module uses a new normalization model to normalize the feature map so that the mean of each channel is 0 and the variance is 1. The linear convolution layer module uses Conv2d to perform linear convolution operations. Specifically, the number of channels is first increased to 4*dim to enrich the feature representation, the convolution kernel is 1x1, and the GELU function is used after activation to enhance the nonlinear expression capability, and then the number of channels is restored to the original dim, so as to facilitate subsequent addition to the input feature map. The residual connection module adds the input feature map to the feature map after the above processing to form a residual connection.
[0058] Next, the generator network structure is improved. The improved generator network structure is as follows: Figure 3As shown in the figure, features are gradually extracted and processed through multiple convolutional layers, normalization, and residual blocks, and finally a high-resolution output image is generated.
[0059] In this embodiment, a multi-level Block structure is introduced to process feature dimensions of 64, 128, 256, and 512 respectively. The first convolution layer of the upsampling module increases the number of channels from 512 to 1024, uses PixelShuffle to increase the spatial resolution of the feature map by 2 times, and introduces nonlinearity using the PReLU activation function. Another convolution layer increases the number of channels from 256 to 512, and uses PixelShuffle and PReLU again. The upsampled feature map is further processed by Conv2d, and a 7x7 convolution layer is used to restore the number of channels to 1 to generate the final output image.
[0060] Finally, the discriminator network structure is improved. The improved discriminator network structure is as follows: Figure 4 As shown in the figure, features are gradually extracted through multiple convolutional layers, downsampling, normalization, and residual blocks, and finally a probability value is output to determine whether the input image is a real image or a generated image.
[0061] In this embodiment, the initial convolution layer uses Conv2d to convert the number of input channels from 1 to 96, and uses a 7x7 convolution kernel and a convolution layer with a stride of 2 to reduce the size of the input feature map by half. At the same time, a new layer normalization module is used to normalize the 96 channels to enhance the stability of training. Next, feature extraction is performed through multiple blocks. Through multiple downsampling operations (2x2 convolution and layer normalization), the number of channels increases from 96 to 192, from 192 to 384, and then from 384 to 768, and the size of the feature map is gradually reduced, thereby extracting features of different scales and enhancing the ability to discriminate high-resolution images. In the final stage of the model, the spatial size of the feature map is reduced to 1x1, the multidimensional tensor is flattened using Flatten, the fully connected layer converts the number of channels from 768 to 1024, and nonlinearity is introduced through the LeakyReLU activation function. Finally, another fully connected layer is used to reduce the output dimension to 1, and the output is mapped to the range of (0, 1) through the Sigmoid activation function, indicating the authenticity probability of the input image.
[0062] In step three, training the synthetic aperture radar image super-resolution network includes: inputting images and related information in the data set into the improved generator network, using the improved generator network to generate high-resolution images, and the improved discriminator network classifies the generated images, calculating the loss function according to the difference between the generated images and the real images, updating the parameters of the generator and the discriminator according to back propagation, thereby optimizing the network parameters to reduce the reconstruction error, and finally obtaining a trained synthetic aperture radar image super-resolution network.
[0063] In this embodiment, the training process sets the batch size to 1, the learning rate to 1e-4, and the number of training rounds to 100. Content loss (mean square error) and adversarial loss (binary cross entropy) are used as loss functions to effectively evaluate the difference between the generated image and the real image. The Adam optimizer is used to update the parameters of the generator and discriminator to improve the model performance.
[0064] Finally, a trained synthetic aperture radar image super-resolution network model is obtained. Low-resolution synthetic aperture radar images can be input into the model for super-resolution to obtain high-resolution images.
[0065] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An image super-resolution method based on a novel network convolutional architecture, characterized in that: The method comprises the following steps: Step 1: Obtain a high-resolution image dataset, and perform preprocessing and degradation processing on the image to generate its corresponding low-resolution image; Step 2: Build an image super-resolution network, which uses the super-resolution generative adversarial network as the baseline model and improves it using a new network convolutional architecture; Step 3: taking the image as a high-resolution image and the corresponding generated degraded blurred image as a low-resolution image, constructing a data pair set, and inputting the data pair set into the improved image super-resolution network for training until the network model converges, so as to obtain a trained image super-resolution network model; Step 4: Apply the trained image super-resolution network model, and input the image to be super-resolved into the test network for super-resolution processing to generate an image with higher resolution.
2. The image super-resolution method based on a novel network convolution architecture according to claim 1, characterized in that: In step 2, in the construction of the improved image super-resolution network, the batch normalization in the original network is replaced by a new layer normalization module, so that the network can perform independent normalization for a single sample and avoid relying on the statistical information of the entire batch.
3. The image super-resolution method based on a novel network convolution architecture according to claim 1, characterized in that: In step 2, in the construction of the improved image super-resolution network, a new module is introduced into the network, and the new module includes the following parts: The deep convolutional layer module constructs deep convolutional layers and performs spatial convolution operations to maintain the dimension of the feature map. The layer normalization module uses a new layer normalization method to normalize the feature map to enhance training stability and keep the feature map unchanged in spatial dimensions. The linear convolution layer module performs linear convolution operations to increase the number of channels of the feature map, then activates the feature map through the activation function, and finally restores the number of channels of the feature map to the original dimension; in the residual connection module, the input feature map is added to the feature map after the above processing to form a residual connection to improve the expressiveness of the model.
4. The image super-resolution method based on a novel network convolution architecture according to claim 1, characterized in that: In step 2, in the construction of the improved image super-resolution network, the improvement of the generator network structure includes: using the new modules introduced at multiple levels to process on multiple feature dimensions respectively to capture feature information at different levels; firstly rearranging pixels in the upsampling module to improve the spatial resolution of the feature map, and then further processing the upsampled feature map to restore the spatial resolution of the original image.
5. The image super-resolution method based on a novel network convolution architecture according to claim 1, characterized in that: In step 2, in the construction of the improved image super-resolution network, the improvement of the discriminator network structure includes: the initial convolution layer uses deep convolution and new layer normalization to perform preliminary feature extraction and normalization on the input image to improve the discrimination ability; the introduced new modules are applied at different levels to extract features of different scales through multiple downsampling operations to enhance the discrimination ability of high-resolution images; the fully connected layer performs the final classification to judge the authenticity of the image.