Real world image super-resolution method and device
By building a neural network including shallow and deep feature extraction modules, combining efficient supergrouping and multi-head self-attention layer, the problems of performance degradation and lack of details in the existing Transformer method are solved, and richer texture details and computing efficiency are achieved balance.
Patent Information
- Application Number
- CN202410984824.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-29
AI Technical Summary
The existing Transformer-based super-resolution method has problems of performance degradation, excessive smoothing, and lack of detailed textures. Most methods achieve lightweight by reducing the number of intermediate feature map channels and the number of feature extraction blocks, resulting in insufficient refinement.
Adopt a neural network structure including shallow feature extraction module, deep feature extraction module and high-quality super-segment reconstruction module. Through efficient super-grouping and multi-head self-attention layer combining local features, the feedforward neural network is enhanced, and global and local features are constructed and localized. The channel dimensions are used to balance performance and efficiency.
The detailed texture information richness of super-resolution results is improved, the amount of network parameters is reduced, the calculation redundancy is alleviated, and the visual effect is maintained.
Smart Images

Figure CN119048347A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a real-world image super-resolution method and device. Background Art
[0002] Image super-resolution is a classic ill-posed problem in the field of image processing, the goal is to reconstruct or generate a corresponding high-resolution image from one or more low-resolution images. Image super-resolution has been widely used due to its high practicality in enhancing image details and textures, such as medical image processing, satellite remote sensing image analysis, and video surveillance.
[0003] Most current deep learning-based image super-resolution methods predefine a downsampling process that is easy to mathematically model, such as bicubic downsampling, and then train the network model on the dataset generated by this downsampling process. However, most low-resolution images in the real world experience mixed distortion of unknown types and degrees, and the downsampling process is difficult to mathematically model. When facing low-resolution images in the real world, the trained models often have obvious performance degradation.
[0004] Relying on the powerful local feature extraction capabilities of convolutional neural networks, researchers began to try to use convolutional neural networks in real-world image super-resolution tasks. In 2021, Xintao Wang et al. proposed the Real-ESRGAN method in "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data", which improved the degradation modeling method, added noise, blur, JPEG compression distortion, ringing artifacts and other degradations to the synthesized low-resolution images, and better simulated the complex degradation in the real world. Adversarial training is used to make its super-resolution results contain rich detailed texture information, but the degradation modeling of this method cannot adapt well to images with low degree of degradation in the real world, and the network structure parameters are huge. In recent years, the Transformer architecture has gradually become the mainstream architecture in the field of natural language processing due to its excellent performance. Researchers have conducted extensive research on real-world image super-resolution using the Transformer architecture. In 2021, Hanting Chen et al. proposed in "Pre-Trained Image Processing Transformer" that by fine-tuning the pre-trained Transformer model, it can acquire the ability to adapt to various underlying computer vision tasks. However, this method is only a simple migration of the Transformer. It not only relies on a large number of model parameters and large-scale data sets, but also does not make targeted improvements to the network in the face of different tasks. Therefore, it has limited effect in real-world super-resolution tasks. In 2021, Jingyun Liang et al. proposed the SwinIR network architecture in "SwinIR: Image Restoration Using Swin Transformer" for image super-resolution tasks, combining residual connections to limit the self-attention calculation to the divided window, reducing the number of parameters while improving the quality of super-resolution images. It is worth noting that many subsequent image super-resolution works are based on this network architecture for improvement. In 2023, Yupeng Zhou et al. proposed the SRFormer method in "SRFormer: Permuted Self-Attention for Single Image Super-Resolution". This method sacrifices some channel information and embeds more spatial information into a fixed-size window through tensor dimension rearrangement and tensor reshaping operations. While the window size remains unchanged, more spatial information can be used to further improve the real-world super-resolution effect.In 2023, Song Zhanghan proposed an image super-resolution reconstruction method based on multi-scale window self-attention in "Research on Image Super-Resolution Reconstruction Based on Self-Attention Mechanism" to address the fixed window limitations of SwinIR. This method groups windows according to different spatial sizes, calculates window self-attention in parallel, and then fuses them, so that the model can learn a variety of features and improve the real-world super-resolution effect. In 2023, Liu Heng et al. used unpaired high-resolution and low-resolution image pairs in the patent "An Unsupervised Real-World Image Super-Resolution Method Based on Knowledge Distillation" and adopted an unsupervised method to train a knowledge distillation model and adversarial generative network to reduce the domain distance between different images, thereby processing low-resolution images in the real world. However, unsupervised methods usually have the problem of poor super-resolution results.
[0005] Existing Transformer-based real image super-resolution methods have the following disadvantages: (1) Most Transformer-based real image super-resolution methods perform lightweight operations by only reducing the number of intermediate feature map channels and the number of feature extraction blocks, which is not refined enough and leads to a serious performance degradation. (2) Due to the self-attention mechanism, Transformer-based real image super-resolution methods can well construct global features within the window, but ignore the construction of local features, which leads to the problem of over-smoothing and lack of detailed texture in the super-resolution results, which seriously affects the subjective experience. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the existing Transformer-based real-world image super-resolution method and to propose a real-world image super-resolution method and device.
[0007] The technical solution for implementing the present invention is as follows:
[0008] In a first aspect, the present application provides a real-world image super-resolution method, the specific process of which is as follows:
[0009] Get real-world raw images for neural network training;
[0010] Construct a neural network including a shallow feature extraction module, a deep feature extraction module and a high-quality super-resolution reconstruction module; wherein the deep feature extraction module is composed of n efficient super-grouping layers, a LayerNorm layer and a convolutional layer; the input of the shallow feature extraction module is the original image, and the output is the shallow feature map; the input of the deep feature extraction module is the shallow feature map, and the output is the deep feature map; the input of the high-quality super-resolution reconstruction module is the pre-super-resolution feature map, and the output is the super-resolution image; the pre-super-resolution feature map is obtained by reshaping the deep feature map from two-dimensional to three-dimensional tensor and adding the corresponding elements of the shallow feature map;
[0011] The original image is used to train a neural network, and the trained neural network is used to obtain a super-resolution image of a real-world image.
[0012] Furthermore, the efficient super-grouping of the present invention is composed of m efficient super-blocks and a 3×3 convolutional layer; assuming that the input of the efficient super-grouping is the i-th level image feature map F i , the i-th level image feature map F i First, the global and local features are obtained through m efficient super-block modeling, and then refined through a 3×3 convolutional layer. The output of the convolutional layer is superimposed with the input of the efficient super-block to obtain the i+1th level image feature map F i+1 .
[0013] Furthermore, the efficient super-blocking of the present invention is composed of 1 channel splitting layer, 2 LayerNorm layers, 1 self-attention layer, 1 local feature enhanced feedforward neural network and 1 channel rearrangement layer;
[0014] Assume that the input feature map F of the jth block of the i-th group i,j First, F i,j Perform channel splitting to obtain activation feature map And straight-through characteristic diagram Activation feature map First, it will be processed by the first LayerNorm layer, and then the global features will be modeled by the self-attention layer. The input of the first LayerNorm layer and the output of the self-attention layer will be superimposed. The superposition result will be processed by the second LayerNorm layer, and finally the local features will be extracted and enhanced by the local feature enhanced feedforward neural network. The input of the second LayerNorm layer and the output of the local feature enhanced feedforward neural network will be superimposed to obtain the processed activation feature map. After the processed activation feature map and the through feature map are spliced in the channel dimension, they are input into the channel rearrangement layer for channel shuffling and rearrangement to obtain the output F i,j+1 .
[0015] Furthermore, the multi-head self-attention layer of the present invention is composed of two linear layers and one Sofmax calculation. The self-attention layer receives the normalized feature map F from the first LayerNorm layer. in As input, first, the number of channels is tripled by the first linear layer. Then, the output tensor is divided into three equal parts in the channel dimension, called Q, K, and V respectively. Then Q and K are combined. TThe autocorrelation coefficient is calculated by matrix multiplication, and then normalized by the Softmax function to obtain the self-attention feature map Attn. Attn is then matrix multiplied with V to obtain the feature map processed by self-attention. Finally, a linear layer is used for weighting and bias processing to obtain the final output feature map F. out .
[0016] Furthermore, the local feature enhanced feedforward neural network of the present invention is composed of 2 linear layers, 3 GELU activation functions, 1 multi-scale convolution layer group, 1 global average pooling, 1 global maximum pooling, two convolution layers and 1 Sigmoid calculation layer;
[0017] Local feature enhanced feedforward neural network with feature map F after the second LayerNorm layer fin As input, it first passes through a linear layer to double the number of channels of the input channel, and then passes through the GELU activation function to increase nonlinearity to obtain the intermediate feature map F 1 ; then F 1 After the tensor is reshaped from two dimensions to three dimensions, the channel is evenly split into n parts in the channel dimension, and the convolution layers with different kernel sizes in the multi-scale convolution layer group are passed to extract local features at different scales; then these n groups of output feature maps are connected in the channel dimension to obtain F 2 ; then F 2 Input into the global average pooling and global maximum pooling, compress its spatial scale into 1×1, add the resulting elements, and get the coarse channel attention vector C'; then pass this coarse channel attention vector C' through a 1×1 convolution layer, a GELU activation function, a 1×1 convolution layer, and a Sigmoid calculation layer in sequence to get a refined channel attention vector C, and add this refined channel attention vector C to F 2 Perform element-wise multiplication to obtain the intermediate feature map F with local feature enhancement 3 ; F 3 After the GELU activation function enhances the nonlinearity, the three-dimensional to two-dimensional tensor is reshaped and then connected to F through a jump connection. 1 Add the elements together; finally, pass a linear layer to return the number of channels to the same as F fin Consistent, get F fout .
[0018] On the other hand, an embodiment of the present application provides a real-world image super-resolution device, comprising a deep and shallow layer feature extraction module, a layer feature extraction module, and a high-quality super-resolution reconstruction module; wherein the deep layer feature extraction module is composed of n efficient super-grouping layers, 1 LayerNorm layer, and 1 convolution layer;
[0019] The input of the shallow feature extraction module is the original image, and the output is the shallow feature map;
[0020] The input of the deep feature extraction module is a shallow feature map, and the output is a deep feature map;
[0021] The input of the high-quality super-resolution reconstruction module is a pre-super-resolution feature map, and the output is a super-resolution image; wherein the pre-super-resolution feature map is obtained by reshaping the deep feature map from two-dimensional to three-dimensional tensor and adding the corresponding elements of the shallow feature map.
[0022] Furthermore, the efficient super-grouping of the present invention is composed of m efficient super-blocks and one 3×3 convolutional layer;
[0023] The efficient super grouping is used to input feature map F i Perform efficient super-block modeling to obtain global and local features, and output them to the convolutional layer;
[0024] The 3×3 convolution layer is used to perform convolution processing on the global features and local features of the input;
[0025] The feature map of the output of the efficient super grouping is the superposition of the convolution processing result and the efficient super grouping input F i+1 .
[0026] Furthermore, the efficient super-blocking of the present invention is composed of 1 channel splitting layer, 2 LayerNorm layers, 1 self-attention layer, 1 local feature enhanced feedforward neural network and 1 channel rearrangement layer;
[0027] Channel splitting layer is used to transform the input feature map F i,j Image split into activation feature maps And straight-through characteristic diagram
[0028] The first LayerNorm layer is used to activate the feature map Perform normalization and transfer the processing results to the self-attention layer;
[0029] Self-attention layer, used to model the global characteristics of the activation feature map;
[0030] The second LayerNorm layer is used to normalize the superposition of the global feature and the activation feature map, and transmit the processing result to the local feature enhanced feedforward neural network;
[0031] Local feature enhancement feedforward neural network is used to extract and enhance local features to obtain the processed activation feature map;
[0032] The channel rearrangement layer is used to shuffle and rearrange the channels of the activated feature map and the direct feature map in the channel dimension to obtain the global feature and local feature F i,j+1 .
[0033] Furthermore, the multi-head self-attention layer of the present invention is composed of 2 linear layers and 1 Sofmax calculation layer;
[0034] The first linear layer is used to change the number of channels of the feature map output after normalization by the first LayerNorm layer to three times the number of input channels, and divide the output tensor into three equal parts in the channel dimension, which are defined as And Q and K T Calculate autocorrelation coefficient by matrix multiplication;
[0035] The Sofmax calculation layer is used to normalize the autocorrelation coefficient to obtain a self-attention feature map Attn;
[0036] The second linear layer is used to weight and bias the result of multiplying the self-attention feature map Attn with the V matrix to obtain the global feature.
[0037] Furthermore, the local feature enhanced feedforward neural network of the present invention is composed of 2 linear layers, 3 GELU activation functions, 1 multi-scale convolution layer group, 1 global average pooling, 1 global maximum pooling, 2 convolution layers and 1 Sigmoid calculation layer; wherein 1 convolution layer, 1 GELU activation function, another convolution layer and the Sigmoid calculation layer are connected in series in sequence to form a fine channel attention processing layer;
[0038] The first linear layer is used to transform the input feature map F fin The number of channels becomes twice that of the input channels, and the output is sent to the first GELU;
[0039] The first GELU activation function is used to increase the nonlinearity of the input feature map and obtain the intermediate feature map F 1 ;
[0040] Channel splitting, used to reshape F after the two-dimensional to three-dimensional tensor 1 , split the channel into multiple parts in the channel dimension and input the multi-scale convolution layer group;
[0041] Multi-scale convolutional layer group, used to extract local features at different scales;
[0042] The global average pooling layer and the global maximum pooling layer are used to connect the results F obtained by connecting the local feature channels at different scales. 2 , to perform spatial scale compression;
[0043] A fine channel processing layer is used to process the coarse channel attention vector C' to obtain a refined channel attention vector C, where C' is obtained by adding the elements of the spatial scale compression result;
[0044] The second GELU activation function is used to enhance the intermediate feature map F 3 The nonlinearity of F 3 Reshape the tensor from three dimensions to two dimensions, and the intermediate feature map is the attention vector C and F 2 Multiply the elements together to get;
[0045] The second linear layer is used to transform the intermediate feature map F 3 and the intermediate feature map F 1 The number of channels of the element addition result is changed back to the same number of channels of the input feature map, and the activated feature map F is obtained. fout .
[0046] Beneficial effects:
[0047] First, while using the self-attention mechanism to construct global features, the method of the present invention uses the powerful local feature extraction ability of the multi-scale convolution group and the channel attention to enhance the local features, enriching the detailed texture information of the super-resolution results. Compared with the existing Transformer-based real-world image super-resolution method, the super-resolution results have richer texture details.
[0048] Second, the present invention splits the feature map into an activation feature map and a direct feature map in the channel dimension, only allows the activation feature map to extract features through the self-attention mechanism and the feedforward neural network, and then rearranges them after connecting them in the channel dimension. By adjusting the number of channels of the activation feature map and the direct feature map, a balance between performance and efficiency can be achieved. Compared with the existing Transformer-based real-world image super-resolution method, it reduces the amount of network parameters and alleviates computational redundancy while maintaining good visual effects of the super-resolution results.
[0049] Third, the local feature enhancement feedforward neural network of the present invention uses a multi-scale convolution group to extract local features at different scales, and uses channel attention to enhance the extracted local features, thereby enriching the detailed texture information of the super-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0051] Figure 1A real-world image super-resolution method based on local feature enhancement;
[0052] Figure 2 This is the structural diagram of the deep feature extraction module;
[0053] Figure 3 This is the structural diagram of the high-quality super-resolution reconstruction module;
[0054] Figure 4 It is a graph of efficient super grouping structure;
[0055] Figure 5 It is an efficient super-block structure diagram;
[0056] Figure 6 It is the self-attention structure diagram;
[0057] Figure 7 Diagram of the feedforward neural network architecture for local feature enhancement. DETAILED DESCRIPTION
[0058] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0059] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.
[0060] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0061] For the convenience of description, it is assumed that the acquired image is subjected to 4 times super-resolution. The embodiment of the present application provides an efficient real-world image super-resolution method, and the specific process is as follows:
[0062] (1) Obtain the original image.
[0063] The original image refers to an image obtained in a real-world scene, in which there may be unknown non-ideal downsampling processes; in addition, there will be mixed distortions of unknown types and degrees in the original image, such as Gaussian blur, Gaussian noise, and JPEG compression distortion. Due to alignment problems, it is very difficult to collect high-quality clear-distorted image pairs. Considering that there is currently no real dataset that meets the requirements, the present invention refers to the DASR degradation modeling method proposed by Jie Liang et al. in "Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution" in 2022, and uses the DASR method on the three datasets of DIV2K, Flickr2K, and OST to generate a synthetic dataset for network training.
[0064] This method divides the entire degradation space into three subspaces according to the degree of degradation: mild, standard, and severe. Each subspace contains five major degradation categories: blur, noise, JPEG compression distortion, ringing artifacts, and resizing. Among them, blur is divided into six subcategories: generalized anisotropic, generalized isotropic, narrow anisotropic, narrow isotropic Gaussian blur kernel, plateau-shaped isotropic, and plateau-shaped anisotropic blur kernel; noise is divided into three subcategories: Gaussian noise, Poisson noise, and grayscale noise; resizing includes three subcategories: size increase, size unchanged, and size reduction. It is worth noting that in each iteration, one of the degradation subspaces will be randomly selected, and the blur type, blur degree, noise type, noise degree, JPEG compression distortion degree, ringing artifact degree, resizing type, and resizing degree will continue to be randomly selected in this degradation subspace according to the preset probability.
[0065] (2) The original image is input into the shallow feature extraction module to obtain the shallow feature map.
[0066] The shallow feature extraction module consists of a 3×3 convolutional layer. Original image After being processed by the shallow feature extraction module, a shallow feature map with unchanged spatial size is obtained. And the number of channels is expanded to C, where W, H and C represent the width, height and number of channels of the shallow feature map respectively. The shallow feature map will be used as the input of the deep feature extraction module.
[0067] (3) The shallow feature map is input into the deep feature extraction module to obtain the deep feature map.
[0068] The deep feature extraction module uses the shallow feature map F s , extract more abstract deep features of the image, and reshape the tensor from three-dimensional to two-dimensional to obtain the deep feature map Where N = W × H, C represents the sequence length and the number of channels respectively. The deep feature extraction module consists of (4≤n≤6) efficient super-grouping, 1 LayerNorm layer and 1 3×3 convolution layer. The specific structure is as follows Figure 2 shown.
[0069] First, the shallow feature map F s After the tensor is reshaped, a series of efficient super-grouping is performed to extract the deep features of the image. Secondly, after refinement by 1 LayerNorm layer and 1 3×3 convolution layer, a deep feature map is generated. Finally, the deep feature map is reshaped from 2D to 3D tensor and then added to the corresponding elements of the shallow feature map to obtain the pre-super-resolved feature map. Where H, W and C represent the width, height and number of channels of the pre-super-resolution feature map, respectively, and F s Be consistent.
[0070] (4) The pre-super-resolution feature map is input into the high-quality super-resolution reconstruction module for super-resolution reconstruction to obtain a super-resolution image.
[0071] The high-quality super-resolution reconstruction module uses the obtained pre-super-resolution feature map F SR , perform super-resolution reconstruction to obtain a super-resolution image Where W and H represent the width and height of the super-resolution image, respectively, which are consistent with the original image. Figure 3 As shown in Figure 1, the high-quality super-resolution reconstruction module first uses a 3×3 convolutional layer to refine the pre-super-resolution feature map and changes the number of channels to 48; then uses a PixelShuffle upsampling layer to increase the spatial size of the feature map by 4 times and the number of channels to 3, thereby obtaining the super-resolution image I SR .
[0072] The following focuses on the efficient super grouping in the deep feature extraction module:
[0073] like Figure 4 As shown in Figure 1, the efficient super-grouping consists of m (4≤m≤6) efficient super-blocks and a 3×3 convolutional layer. The input of the efficient super-grouping is the i-th level image feature map The output is the image feature map of level i+1 Where N, C and i represent the sequence length, number of channels and order of efficient super grouping of the image feature map respectively. i First, the global and local features are modeled through m efficient super-blocks, and then refined through a 3×3 convolutional layer. Finally, the skip connection is used to ensure the training stability, and the image feature map F of the i+1th level is obtained. i+1 .
[0074] like Figure 5As shown in Figure 1, the efficient super-block consists of 1 channel splitting layer, 2 LayerNorm layers, 1 self-attention layer, 1 local feature enhanced feedforward neural network and 1 channel rearrangement layer. Assume that the input feature map of the jth block of the i-th group Where N and C represent the sequence length and number of channels of the image feature map respectively. First, i,j Perform channel splitting to obtain activation feature map And straight-through characteristic diagram Among them, C 1 and C 2 Respectively represent the number of channels of the activated feature map and the direct feature map, and C 1 +C 2 = C. Activation feature map First, it will be processed by a LayerNorm layer, then the global features will be modeled by the self-attention layer, and the skip connection will be used to ensure the stability of the training; then it will be processed by a LayerNorm layer, and finally the local feature enhancement feedforward neural network will be used to extract and enhance the local features, and the skip connection will be used to ensure the stability of the training; thus, the processed activation feature map will be obtained. After the processed activation feature map and the through feature map are spliced in the channel dimension, they are input into the channel rearrangement layer for channel shuffling and rearrangement to obtain the output
[0075] like Figure 6 As shown in Figure 1, the self-attention layer refers to the design of the multi-head self-attention layer proposed by Ashish Vaswani et al. in "Attention is all you need" in 2017, which consists of 2 linear layers and 1 Sofmax calculation. The self-attention layer receives the normalized feature map from the LayerNorm layer. As input, it first passes through a linear layer to triple the number of channels of the input channel, and then divides the output tensor into three equal parts in the channel dimension, respectively called Then Q and K T The autocorrelation coefficient is calculated by matrix multiplication, and then normalized by the Softmax function to obtain the self-attention feature map. Then, Attn is matrix-multiplied with V to obtain the feature map after self-attention processing, and finally refined by a linear layer, that is, weighted and biased processing is introduced to obtain the final output feature map. In this embodiment, in order to prevent the problem of gradient disappearance or gradient explosion, more learnable weights and biases are introduced by using linear layers, so that the model can learn more complex representations and enhance model performance, thereby achieving refined processing.
[0076] like Figure 7As shown in Figure 1, the local feature enhancement feedforward neural network consists of 2 linear layers, 3 GELU activation functions, 1 multi-scale convolution layer group, 1 global average pooling, 1 global maximum pooling, two convolution layers and 1 Sigmoid calculation layer. The local feature enhancement feedforward neural network is composed of the feature map after the LayerNorm layer. As input, it first passes through a linear layer to double the number of channels of the input channel, and then passes through the GELU activation function to increase nonlinearity to obtain the intermediate feature map Then F 1 After the tensor is reshaped from two dimensions to three dimensions, the channel is evenly split into n (2≤n≤4) parts in the channel dimension (the figure takes splitting into 4 parts as an example), and then it is passed through convolution layers with different kernel sizes in the multi-scale convolution layer group to extract local features at different scales; then these n groups of output feature maps are connected in the channel dimension to obtain Where W and H represent the width and height of the original image respectively, and W×H=N; then F 2 Input to the global average pooling and global maximum pooling, compress its spatial scale to 1×1, add the resulting elements together to get the coarse channel attention vector Then, this coarse channel attention vector C' is sequentially passed through a 1×1 convolution layer, a GELU activation function, a 1×1 convolution layer, and a Sigmoid calculation layer to obtain a refined channel attention vector Combine this refined channel attention vector with F 2 Perform element-wise multiplication to obtain an intermediate feature map with local feature enhancement F 3 After the GELU activation function enhances the nonlinearity, F is reshaped from three-dimensional to two-dimensional tensors. 3 The tensor shape is changed back to N×2C, and then connected to F through a skip connection 1 Add the elements to ensure training stability; finally, use a linear layer to return the number of channels to the same as F. fin Consistent, get
[0077] This method defines the super-resolution image as High resolution images are The loss function is
[0078]
[0079] Among them, N is the batch size during training, and Adam is selected as the training iterator.
[0080] On the other hand, an embodiment of the present application provides an efficient real-world image super-resolution device, comprising a deep and shallow layer feature extraction module, a layer feature extraction module, and a high-quality super-resolution reconstruction module; wherein the deep layer feature extraction module is composed of n efficient super-grouping layers, 1 LayerNorm layer, and 1 convolution layer;
[0081] The input of the shallow feature extraction module is the original image, and the output is the shallow feature map;
[0082] The input of the deep feature extraction module is a shallow feature map, and the output is a deep feature map;
[0083] The input of the high-quality super-resolution reconstruction module is a pre-super-resolution feature map, and the output is a super-resolution image; wherein the pre-super-resolution feature map is obtained by reshaping the deep feature map from two-dimensional to three-dimensional tensor and adding the corresponding elements of the shallow feature map.
[0084] In order to improve computational efficiency, the method in the embodiment of the present application splits the intermediate feature map into an activation feature map and a direct feature map in the channel dimension, and only allows the activation feature map to pass through the feature extraction layer with a large computational workload, thereby reducing the number of network parameters and the amount of computation.
[0085] In order to avoid the problem that only certain channels repeatedly undergo feature extraction, the embodiment of the present application also disrupts the order of channels after the activation feature map and the through feature map are connected in the channel dimension, so that each channel can undergo feature extraction.
[0086] In order to extract rich local feature information, the embodiment of the present application embeds a multi-scale convolution group into a feed-forward neural network to extract multi-scale local features, and further enhances the extracted features using channel attention.
[0087] This method alleviates computational redundancy and improves the utilization efficiency of computing resources; and reduces the severity of the problem of super-resolution images being too smooth and lacking in texture details in the real-world image super-resolution problem.
[0088] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A real-world image super-resolution method, characterized in that: The specific process is: Get real-world raw images for neural network training; Construct a neural network including a shallow feature extraction module, a deep feature extraction module and a high-quality super-resolution reconstruction module; wherein the deep feature extraction module is composed of n efficient super-grouping layers, a LayerNorm layer and a convolutional layer; the input of the shallow feature extraction module is the original image, and the output is the shallow feature map; the input of the deep feature extraction module is the shallow feature map, and the output is the deep feature map; the input of the high-quality super-resolution reconstruction module is the pre-super-resolution feature map, and the output is the super-resolution image; the pre-super-resolution feature map is obtained by reshaping the deep feature map from two-dimensional to three-dimensional tensor and adding the corresponding elements of the shallow feature map; The original image is used to train a neural network, and the trained neural network is used to obtain a super-resolution image of a real-world image.
2. The real-world image super-resolution method according to claim 1, characterized in that: The efficient super grouping consists of m efficient super blocks and a 3×3 convolutional layer. Assume that the input of the efficient super grouping is the i-th level image feature map F i , the i-th level image feature map F i First, the global and local features are obtained through m efficient super-block modeling, and then refined through a 3×3 convolutional layer. The output of the convolutional layer is superimposed with the input of the efficient super-block to obtain the i+1th level image feature map F i+1 .
3. The real-world image super-resolution method according to claim 2, characterized in that: The efficient super-blocking consists of 1 channel splitting layer, 2 LayerNorm layers, 1 self-attention layer, 1 local feature enhanced feedforward neural network and 1 channel rearrangement layer; Assume that the input feature map F of the jth block of the i-th group i,j First, F i,j Perform channel splitting to obtain activation feature map And straight-through characteristic diagram Activation feature map First, it will be processed by the first LayerNorm layer, and then the global features will be modeled by the self-attention layer. The input of the first LayerNorm layer and the output of the self-attention layer will be superimposed. The superposition result will be processed by the second LayerNorm layer, and finally the local features will be extracted and enhanced by the local feature enhanced feedforward neural network. The input of the second LayerNorm layer and the output of the local feature enhanced feedforward neural network will be superimposed to obtain the processed activation feature map. After the processed activation feature map and the through feature map are spliced in the channel dimension, they are input into the channel rearrangement layer for channel shuffling and rearrangement to obtain the output F i,j+1 .
4. The real-world image super-resolution method according to claim 3, characterized in that: The multi-head self-attention layer consists of two linear layers and one Sofmax calculation. The self-attention layer receives the normalized feature map F from the first LayerNorm layer. in As input, first, the number of channels is tripled by the first linear layer. Then, the output tensor is divided into three equal parts in the channel dimension, called Q, K, and V respectively. Then Q and K are combined. T The autocorrelation coefficient is calculated by matrix multiplication, and then normalized by the Softmax function to obtain the self-attention feature map Attn. Attn is then matrix multiplied with V to obtain the feature map processed by self-attention. Finally, a linear layer is used for weighting and biasing to obtain the final output feature map F. out .
5. The real-world image super-resolution method according to claim 3, characterized in that: The local feature enhanced feedforward neural network consists of 2 linear layers, 3 GELU activation functions, 1 multi-scale convolution layer group, 1 global average pooling, 1 global maximum pooling, two convolution layers and 1 Sigmoid calculation layer; Local feature enhanced feedforward neural network with feature map F after the second LayerNorm layer fin As input, it first passes through a linear layer to double the number of channels of the input channel, and then increases nonlinearity through the GELU activation function to obtain the intermediate feature map F1; then F1 is reshaped from a two-dimensional to a three-dimensional tensor, and the channels are evenly split into n parts in the channel dimension, and then passed through convolution layers with different kernel sizes in the multi-scale convolution layer group to extract local features at different scales; Then, these n groups of output feature maps are connected in the channel dimension to obtain F2; then F2 is input into global average pooling and global maximum pooling, and its spatial scale is compressed to 1×1. The resulting elements are added to obtain a coarse channel attention vector C'; this coarse channel attention vector C' is then passed through a 1×1 convolution layer, a GELU activation function, a 1×1 convolution layer, and a Sigmoid calculation layer in sequence to obtain a refined channel attention vector C, and this refined channel attention vector C is element-wise multiplied with F2 to obtain an intermediate feature map F3 with local feature enhancement; after F3 is enhanced nonlinearly by the GELU activation function, it is reshaped using a three-dimensional to two-dimensional tensor and then added to the F1 element through a jump connection; finally, a linear layer is used to convert the number of channels back to the same as F fin Consistent, get F fout .
6. A real-world image super-resolution device, characterized in that: It includes a deep and shallow layer feature extraction module, a layer feature extraction module and a high-quality super-resolution reconstruction module; wherein the deep layer feature extraction module is composed of n efficient super-grouping layers, 1 LayerNorm layer and 1 convolution layer; The input of the shallow feature extraction module is the original image, and the output is the shallow feature map; The input of the deep feature extraction module is a shallow feature map, and the output is a deep feature map; The input of the high-quality super-resolution reconstruction module is a pre-super-resolution feature map, and the output is a super-resolution image; wherein the pre-super-resolution feature map is obtained by reshaping the deep feature map from two-dimensional to three-dimensional tensor and adding the corresponding elements of the shallow feature map.
7. The real-world image super-resolution device according to claim 6, characterized in that: The efficient super-grouping consists of m efficient super-blocks and a 3×3 convolutional layer; The efficient super grouping is used to input feature map F i Perform efficient super-block modeling to obtain global and local features, and output them to the convolutional layer; The 3×3 convolution layer is used to perform convolution processing on the global features and local features of the input; The feature map of the output of the efficient super grouping is the superposition of the convolution processing result and the efficient super grouping input F i+1 .
8. The real-world image super-resolution device according to claim 7, characterized in that: The efficient super-blocking consists of 1 channel splitting layer, 2 LayerNorm layers, 1 self-attention layer, 1 local feature enhanced feedforward neural network and 1 channel rearrangement layer; Channel splitting layer is used to transform the input feature map F i,j Image split into activation feature maps And straight-through characteristic diagram The first LayerNorm layer is used to activate the feature map Perform normalization and transfer the processing results to the self-attention layer; Self-attention layer, used to model the global characteristics of the activation feature map; The second LayerNorm layer is used to normalize the superposition of the global feature and the activation feature map, and transmit the processing result to the local feature enhanced feedforward neural network; Local feature enhancement feedforward neural network is used to extract and enhance local features to obtain the processed activation feature map; The channel rearrangement layer is used to shuffle and rearrange the channels of the activated feature map and the direct feature map in the channel dimension to obtain the global feature and local feature F i,j+1 .
9. The real-world image super-resolution device according to claim 8, characterized in that: The multi-head self-attention layer consists of 2 linear layers and 1 Sofmax calculation layer; The first linear layer is used to change the number of channels of the feature map output after normalization by the first LayerNorm layer to three times the number of input channels, and divide the output tensor into three equal parts in the channel dimension, which are defined as And Q and K T Calculate autocorrelation coefficient by matrix multiplication; The Sofmax calculation layer is used to normalize the autocorrelation coefficient to obtain a self-attention feature map Attn; The second linear layer is used to weight and bias the result of multiplying the self-attention feature map Attn with the V matrix to obtain the global feature.
10. The real-world image super-resolution device according to claim 8, characterized in that: The local feature enhanced feedforward neural network consists of 2 linear layers, 3 GELU activation functions, 1 multi-scale convolution layer group, 1 global average pooling, 1 global maximum pooling, 2 convolution layers and 1 Sigmoid calculation layer; wherein 1 convolution layer, 1 GELU activation function, another convolution layer and the Sigmoid calculation layer are connected in series in sequence to form a fine channel attention processing layer; The first linear layer is used to transform the input feature map F fin The number of channels becomes twice that of the input channels, and the output is sent to the first GELU; The first GELU activation function is used to increase the nonlinearity of the input feature map and obtain the intermediate feature map F1; Channel splitting is used to split F1, which has been reshaped from two-dimensional to three-dimensional tensors, into multiple channels in the channel dimension and input them into a multi-scale convolutional layer group; Multi-scale convolutional layer group, used to extract local features at different scales; The global average pooling layer and the global maximum pooling layer are used to compress the spatial scale of the result F2 obtained by connecting the local feature channel dimensions at different scales; A fine channel processing layer is used to process the coarse channel attention vector C' to obtain a refined channel attention vector C, where C' is obtained by adding the elements of the spatial scale compression result; The second GELU activation function is used to enhance the nonlinearity of the intermediate feature map F3, and then reshape F3 from three dimensions to two dimensions. The intermediate feature map is obtained by multiplying the attention vector C by the F2 element. The second linear layer is used to change the number of channels of the result of adding the elements of the intermediate feature map F3 and the intermediate feature map F1 back to the same number of channels as the input feature map, and obtain the activated feature map F fout .
Citation Information
Cited By
Defect identification method and system for gearbox shell of electro-tricycle
CN119991624A
