Nonlinear perceptual multiscale super-resolution image generation system and method
By using a nonlinear perceptual multi-scale super-resolution convolutional neural network, the problem of image detail loss caused by single-scale or linear fusion in existing technologies is solved, achieving higher quality image super-resolution reconstruction and restoring the structural and texture information of the image.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2022-06-29
- Publication Date
- 2026-06-02
AI Technical Summary
Existing super-resolution methods based on convolutional neural networks typically only consider a single scale or fuse multi-scale features in a linear manner, resulting in the loss of image detail and texture and an inability to effectively reconstruct high-frequency information.
A nonlinear perceptual multi-scale super-division convolutional neural network is adopted. Through shallow feature extraction, multiple cascaded residual nesting groups and reconstruction sub-networks, nonlinear perceptual modules and cross-channel second-order attention mechanisms are used to perform nonlinear fusion of multi-scale features to obtain more discriminative image information.
It achieves competitive results on all datasets at all scales, recovers image structural information, and improves image reconstruction quality, especially performing well at high magnification.
Smart Images
Figure CN115375537B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer image processing, and particularly relates to a method for super-resolution generation of images or videos. Background Technology
[0002] Natural images contain various types of features, and different types of features correspond to different scales. For example, smooth regions and strong edge regions correspond to larger scales, while textured regions correspond to smaller scales.
[0003] Super-resolution (SR) technology refers to the reconstruction of a corresponding high-resolution image from an observed low-resolution image. It has important applications in fields such as surveillance equipment, satellite imagery, and medical imaging. Image super-resolution refers to the recovery of a corresponding high-resolution (HR) image from a low-resolution (LR) image.
[0004] Existing super-resolution methods based on convolutional neural networks often employ only large or small receptive fields, extracting features at a single scale. This fails to capture multi-scale features and makes it difficult to reconstruct high-frequency information in images. Furthermore, most existing multi-scale super-resolution learning networks use a linear fusion approach, directly stacking or adding feature maps. The extracted feature information has high redundancy and cannot truly simulate the powerful adaptive integration of visual information by neurons, severely limiting the network's representational capabilities and super-resolution image reconstruction performance. Summary of the Invention
[0005] To address the problem that most current end-to-end super-resolution networks only consider a single scale or use a linear approach to fuse multi-scale features, which easily leads to the loss of details and textures in the restored image, this invention proposes a nonlinear perceptual image super-resolution network image generation system and method. Based on a high-order attention mechanism, it models the correlation of multi-scale features to obtain more discriminative multi-scale information, realizes nonlinear fusion of multi-scale features, and achieves the reconstruction of super-resolution images.
[0006] This invention is achieved using the following technical solution:
[0007] A nonlinear perceptual multi-scale super-resolution image generation system includes a nonlinear perceptual multi-scale super-resolution convolutional neural network, which is further composed of a shallow feature extraction sub-network, a multi-cascaded residual nested group sub-network, and a reconstruction sub-network; wherein:
[0008] The training dataset is used as input to the nonlinear perceptual multi-scale super-resolution convolutional neural network. The shallow feature extraction sub-network consists of a single convolutional layer, used to coarsely extract the input features to obtain a feature map. The multi-cascaded residual nested group sub-network connects several residual nested groups through a global multi-cascaded mechanism. It takes the feature map as input and learns further nonlinear mappings to extract nonlinear multi-scale features deep within the image. The multi-cascaded residual nested group connects several residual nested groups through a global multi-cascaded mechanism. Each residual nested group contains multiple residual blocks and a convolutional layer, and a residual structure is formed by skip connections. Each residual block is equipped with a nonlinear perceptual module, which uses convolutional kernels of different sizes to obtain features of different scales of the image. By assigning weights to the feature maps of the branches of each different convolutional kernel, the total output is obtained, which is the multi-scale feature map selected by the nonlinear perceptual module.
[0009] The reconstruction subnetwork consists of subpixel convolutional layers and reconstruction convolutional layers, which produce a super-resolution image of a specified size.
[0010] A nonlinear perceptual multi-scale super-resolution image generation method, the process of which specifically includes the following steps:
[0011] Step 1: Obtain the total training set and test set;
[0012] Step 2: Construct a nonlinear perceptual multi-scale super-resolution convolutional neural network to achieve image super-resolution processing. The specific process is as follows:
[0013] In the nonlinear perceptual multi-scale super-resolution convolutional neural network, the training dataset is used as input. A shallow feature extraction sub-network consisting of a conv3×3 convolutional layer completes the coarse extraction of input features. The obtained feature map is input into a multi-cascaded residual nesting group sub-network to learn further nonlinear mapping and extract nonlinear multi-scale features deep in the image. The multi-cascaded residual nesting group connects several residual nesting groups through a global multi-cascaded mechanism. Each residual nesting group contains multiple residual blocks and a conv1×1 convolutional layer. Skip connections are used to form the residual structure. Each residual block is equipped with a nonlinear perceptual module, which uses convolutional kernels of different sizes to obtain features of different scales of the image. By assigning weights to the feature maps of the branches of each different convolutional kernel, the total output is obtained, which is the multi-scale feature map selected by the nonlinear perceptual module.
[0014] A super-resolution image of a specified size is obtained by using a reconstruction subnetwork consisting of subpixel convolutional layers and reconstruction convolutional layers.
[0015] Compared with the prior art, the present invention can achieve the following beneficial technical effects:
[0016] 1) This invention has achieved competitive results on all scales and all datasets. Compared with advanced super-resolution networks in recent years, it has achieved significant improvements in objective metrics (objective evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity (SSIM) value, and the number of parameters of algorithms with similar performance to this model).
[0017] 2) This invention employs a nonlinear perceptual learning method, which can effectively recover the structural information of images, demonstrating its superiority in all datasets with large magnification.
[0018] 3) This invention can better restore the texture details of the image, and the overall quality of the reconstructed image is higher. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the architecture of the nonlinear sensing multi-scale super-resolution image generation system of the present invention;
[0020] Figure 2 This is a schematic diagram of the overall process of the nonlinear sensing multi-scale super-resolution image generation method of the present invention;
[0021] Figure 3 This is a schematic diagram of a nonlinear perceptual multi-scale super-resolution convolutional neural network (NPMNet).
[0022] Figure 4 This is a schematic diagram of a multi-cascaded nested residual group.
[0023] Figure 5 This is a schematic diagram of a nonlinear sensing module;
[0024] Figure 6 A detailed comparison diagram of the "Comic" image;
[0025] Figure 7 A comparative illustration of the super-resolution image generation effect for "ground" images;
[0026] Figure 8 A comparative illustration of the super-resolution image generation effect for the "architecture" image. Detailed Implementation
[0027] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0028] The basic principles of this invention are briefly described below:
[0029] LR image I x The model represents the output of the degradation process, expressed as follows:
[0030]
[0031] in, For the degenerate mapping function, I y For the corresponding HR image, δ represents a parameter of the degradation process (such as a scaling factor or noise). Generally, the degradation process (i.e., γ and δ) is unknown, and only the LR image is provided. In this case, also known as blind SR, researchers need to obtain the HR image from the LR image. x Recovering true ground HR images y HR approximation as follows:
[0032]
[0033] in, For the super-resolution model, θ is The parameters.
[0034] like Figure 1 The diagram shown illustrates the architecture of a nonlinear perceptual multi-scale super-resolution image generation system according to the present invention. The system includes a nonlinear perceptual multi-scale super-resolution convolutional neural network, which is further composed of a shallow feature extraction sub-network 100, a multi-level cascaded residual nested group sub-network 200, and a reconstruction sub-network 300.
[0035] The training dataset is used as input to the nonlinear perceptual multi-scale super-resolution convolutional neural network. The shallow feature extraction sub-network 100 consists of a conv3×3 convolutional layer, which is used to perform coarse extraction of input features to obtain feature maps. The multi-cascaded residual nested group sub-network 200 connects several residual nested groups through a global multi-cascaded mechanism. It takes the feature maps as input and extracts nonlinear multi-scale features deep in the image by learning further nonlinear mappings. The reconstruction sub-network 300 obtains a super-resolution image of a specified size and consists of sub-pixel convolutional layers and reconstruction convolutional layers.
[0036] The multi-cascaded residual nested group connects several residual nested groups through a global multi-cascaded mechanism. Each residual nested group contains multiple residual blocks and a convolutional layer conv1×1, and uses skip connections to form the residual structure. The output of a convolutional layer conv1×1 in each residual block performs feature map stacking operation and connects to the input of the next residual block.
[0037] Each residual block is equipped with a nonlinear perception module, which uses convolutional kernels of different sizes to acquire features of different scales of the image. Through a cross-channel second-order attention mechanism, it scales multi-scale features more discriminatively, completes the nonlinear fusion of multi-scale information, and finally obtains the multi-scale feature map after selection by the nonlinear perception module to obtain the total output of the convolutional layer.
[0038] like Figure 2The diagram shown illustrates the flowchart of the nonlinear sensing multi-scale super-resolution image processing method of the present invention. The flowchart specifically includes the following steps:
[0039] Step 1: Obtain the total training and test sets: For the training dataset, the DIV2K dataset is used. DIV2K is a newly proposed dataset containing high-quality images, including 800 training images, enabling the network to learn rich features. For the test set, four benchmark datasets—Set5, Set14, BSD100, and Ubran100—are used to evaluate the network's super-resolution reconstruction performance. Bicubic interpolation is used to downsample high-resolution images to a specified size to create a low-resolution dataset.
[0040] Step 2: Construct a nonlinear perceptual multi-scale super-resolution convolutional neural network (NPMNet), and use this convolutional neural network to generate super-resolution images:
[0041] like Figure 3 The diagram shown illustrates a nonlinear perceptual multi-scale super-resolution convolutional neural network. Figure 4 The diagram shown is a schematic of a multi-cascaded nested residual group.
[0042] The goal of super-resolution networks is to recover more useful information in order to build more effective residual learning mechanisms.
[0043] The nonlinear perceptual multi-scale super-resolution convolutional neural network (NPMNet) of this invention consists of three sub-networks. First, the training dataset is used as input, and a shallow feature extraction sub-network consisting of a conv3×3 convolutional layer performs coarse extraction of the input features. Next, the feature maps are input into a multi-cascaded residual nested group (MC-RNG) sub-network to learn further nonlinear mappings and extract nonlinear multi-scale features deep within the image. The circle with the letter C indicates the feature map stacking operation.
[0044] The multi-cascaded residual nested group (LRNG) connects several local residual nested groups (LRNGs) through a global multi-cascade mechanism. Each LRNG employs a local residual nesting mechanism. Each LRNG contains multiple residual blocks (residual block 1, ..., residual block d, ..., residual block D) and a conv1×1 convolutional layer (as a bottleneck layer), using skip connections to form the residual structure. The output of the conv1×1 convolutional layer in each residual block is used for feature map stacking and connected to the input of the next residual block. Each residual block contains the residual structure of a nonlinear perceptual module (NPMM).
[0045] The output of a multi-cascaded nested residual group (LRNG) includes the output of the first nested residual group (LRNG) and the output of the nth nested residual group (LRNG), which are represented as follows:
[0046]
[0047]
[0048] in, This represents the output of the first Residual Nested Group (LRNG), whose input is the output F0 of the shallow feature extraction network. H represents the output of the nth Residual Nested Group (LRNG), whose input is the output of the preceding convolutional layer with a 1×1 kernel. LRNG,n (·) represents the mapping relationship contained in the nth residual nesting group (LRNG), which is mainly composed of residual nesting mechanism, and N represents the total number of residual nesting groups (LRNG);
[0049] The output of the d-th residual block in the nth nested residual group (LRNG) is represented as:
[0050]
[0051] The input of the d-th residual block in the n-th nested residual group (LRNG) is: This is the output of the d-th residual block in the nth Residual Nested Group (LRNG). The operation of residual blocks, where D represents the total number of residual blocks in each Residual Nesting Group (LRNG).
[0052] The Residual Nested Group (LRNG) uses skip connections to obtain the output of the residual block, represented as:
[0053]
[0054] Among them, W n The parameters of the last convolutional layer in the nth Residual Nested Group (LRNG) are used to make the backbone network focus more on more useful feature information.
[0055] Unlike the dense connections of convolutional layer features in DenseNet, the global multi-cascade mechanism reduces the computational burden of the network by reusing the hierarchical features of the integrated residual blocks.
[0056] like Figure 5The diagram shown illustrates the nonlinear perception module. For this module, features at different scales of the image are first acquired using convolutional kernels of different sizes. Then, a cross-channel second-order attention mechanism is used to scale the multi-scale features more discriminatively, completing the nonlinear fusion of multi-scale information.
[0057] Taking a Nonlinear Perception Module (NPMM) with two branches containing convolutional kernels of different sizes as an example, assuming the input is X with C channels, first, different convolution operations Conv3×3 and Conv5×5 are performed on the two branches to obtain two sets of feature maps of size H×W containing C channels. and It contains feature information from two different scales of images. The two branch feature maps are merged by element-wise addition to obtain the channel feature map M. Then, global covariance pooling is used to generate the global second-order statistic S of each channel feature map. GCP .
[0058] A 1×1 convolutional layer is used to learn the mapping relationship feature vector Z between two scales of features, improving efficiency through dimensionality reduction. Then, the mapping relationship feature vector Z is passed through a fully connected layer to learn two sets of feature vectors with different scales and receptive fields. A softmax operation is performed on the two branches of feature vectors in the same dimension to obtain weight vectors a and b. Weights are assigned to the feature maps of each branch with different convolutional kernels to obtain the c-th feature map output O, as shown below:
[0059]
[0060] Among them, a c Let a and b be the c-th weight vectors. c Let b be the c-th weight vector;
[0061] By assigning weights to the feature maps of the branches of different convolutional kernels, the total output O = [O1, O2, ..., O2] is obtained. C This yields the multi-scale feature map after selection by the nonlinear sensing module.
[0062] Finally, a high-resolution image of the specified size is obtained by passing through a reconstruction subnetwork consisting of subpixel convolutional layers and reconstruction convolutional layers.
[0063] The aforementioned nonlinear perception module (NPMM) was used to achieve adaptive extraction of multi-scale information from images and to perform cross-channel learning of multi-scale information using a second-order attention mechanism, thereby enhancing key multi-scale features.
[0064] In step 2, the algorithm for image super-resolution processing based on the convolutional neural network uses the L1 loss function to guide network training. The L1 or MSE loss makes the model focus more on the average loss at the pixel level in order to achieve higher PSNR and SSIM indices.
[0065] The results of the verification of the beneficial effects of the present invention are as follows:
[0066] 1) Objective Performance Comparison: Objective evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) value, and parameter count comparison with algorithms of similar performance to our model. The average PSNR and SSIM values for each method were calculated on four benchmark datasets: Set5, Set14, BSD100, and Ubran100. The results are shown in Tables 1 and 2. Table 1 provides an example of objective evaluation metrics. Table 2 provides an example of parameter count comparison at scale 4.
[0067] Table 1
[0068]
[0069]
[0070] Table 2
[0071]
[0072] The comparative results show that, with fewer than 2 million parameters, NPMNet achieves competitive results across all scales and datasets, demonstrating significant improvements in objective metrics compared to advanced super-resolution networks in recent years. In particular, NPMNet significantly outperforms existing methods, even very deep models, in the SSIM metric. This indicates that NPMNet, employing a non-linear perceptual learning method, can effectively recover the structural information of images. Furthermore, NPMNet achieves the highest metrics across all datasets when magnified by a factor of 4, demonstrating its superiority at higher magnification levels.
[0073] To more intuitively compare the reconstruction effects of various super-resolution algorithms, subjective visual comparisons were conducted using images magnified 3 times from Set14, BSD100, and Urban100, which are considered more difficult to recover. For example... Figure 6 The image shown is a detailed comparison diagram of the "Comic" image. This subjective comparison demonstrates that other networks produce relatively blurry high-frequency detail restorations of the fingers and nails, while NPMNet restores the texture more clearly and completely than most comparison algorithms.
[0074] like Figure 7The image shown illustrates the contrast effect of the "ground" image. The texture restored by NPMNet is clearer than most contrast algorithms. Figure 8 The image shown is a comparison of the "building" image. It is clearly observed that NPMNet recovers more horizontal stripes; specifically, it recovers two more horizontal stripes than other models. In summary, the proposed NPMNet can better restore the texture details of the image, resulting in a higher overall image quality.
[0075] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A nonlinear perceptual multi-scale super-resolution image generation system, characterized by, The system includes a nonlinear perceptual multi-scale super-resolution convolutional neural network, which is further composed of a shallow feature extraction sub-network, a multi-cascaded residual nested group sub-network, and a reconstruction sub-network; wherein: The training dataset is used as input to the nonlinear perceptual multi-scale super-resolution convolutional neural network. The shallow feature extraction sub-network consists of a single convolutional layer, used to coarsely extract the input features to obtain a feature map. The multi-cascaded residual nested group sub-network connects several residual nested groups through a global multi-cascaded mechanism. It takes the feature map as input and learns further nonlinear mappings to extract nonlinear multi-scale features deep within the image. The multi-cascaded residual nested group connects several residual nested groups through a global multi-cascaded mechanism. Each residual nested group contains multiple residual blocks and a convolutional layer, and a residual structure is formed by skip connections. Each residual block is equipped with a nonlinear perceptual module, which uses convolutional kernels of different sizes to obtain features of different scales of the image. By assigning weights to the feature maps of the branches of each different convolutional kernel, the total output is obtained, which is the multi-scale feature map selected by the nonlinear perceptual module. The reconstruction subnetwork consists of subpixel convolutional layers and reconstruction convolutional layers, which produce a super-resolution image of a specified size.
2. The nonlinear perceptual multi-scale super-resolution image generation system of claim 1, wherein, The multi-cascaded residual nested group connects several residual nested groups through a global multi-cascaded mechanism. Each residual nested group contains multiple residual blocks and a convolutional layer conv1×1, and uses skip connections to form the residual structure. The output of a convolutional layer conv1×1 in each residual block performs feature map stacking operation and connects to the input of the next residual block.
3. The nonlinear perceptual multi-scale super-resolution image generation system as described in claim 1, characterized in that, The output of a convolutional layer conv1×1 in each residual block is used for feature map stacking and then connected to the input of the next residual block.
4. A nonlinear sensing multi-scale super-resolution image generation method, characterized in that, The process specifically includes the following steps: Step 1: Obtain the total training set and test set; Step 2: Construct a nonlinear perceptual multi-scale super-resolution convolutional neural network to achieve image super-resolution processing. The specific process is as follows: In the nonlinear perceptual multi-scale super-resolution convolutional neural network, the training dataset is used as input. A shallow feature extraction sub-network consisting of a 3×3 convolutional layer completes the coarse extraction of input features. The obtained feature map is input into a multi-cascaded residual nesting group sub-network to learn further nonlinear mapping and extract nonlinear multi-scale features deep in the image. The multi-cascaded residual nesting group connects several residual nesting groups through a global multi-cascaded mechanism. Each residual nesting group contains multiple residual blocks and a 1×1 convolutional layer. Skip connections are used to form the residual structure. Each residual block is equipped with a nonlinear perceptual module, which uses convolutional kernels of different sizes to obtain features of different scales of the image. By assigning weights to the feature maps of the branches of each different convolutional kernel, the total output is obtained, which is the multi-scale feature map selected by the nonlinear perceptual module. A super-resolution image of a specified size is obtained by using a reconstruction subnetwork consisting of subpixel convolutional layers and reconstruction convolutional layers.
5. The nonlinear perceptual multi-scale super-resolution image generation method as described in claim 4, characterized in that, The output of a convolutional layer conv1×1 in each residual block is used for feature map stacking and then connected to the input of the next residual block.
6. The nonlinear perceptual multi-scale super-resolution image generation method as described in claim 4 or 5, characterized in that, The method includes the following specific steps: The output of the nested residual group includes the output of the first nested residual group and the output of the second nested residual group. n The outputs of each nested residual group are represented as follows: in, This represents the output of the first nested residual group, whose input is the output of the shallow feature extraction network. , Indicates the first n The output of each nested residual group is the output of the preceding convolutional layer with a 1×1 kernel. Indicates the first n The mapping relationships contained in each nested residual group are mainly composed of a nested residual mechanism. N Indicates the total number of nested residual groups; No. n The first residual nested group d The output of each residual block is represented as: Among them, the n The first residual nested group d The input of each residual block is , For the first n The first residual nested group d The output of each residual block Operations on residual blocks, D This represents the total number of residual blocks in each nested residual group; The nested residual group uses skip connections to obtain the output of the residual block, represented as: in, For the first n The parameters of the last convolutional layer in the nested residual group.
7. The nonlinear sensing multi-scale super-resolution image generation method as described in claim 4, characterized in that, In the nonlinear sensing module, the input is X It contains C Each channel is first subjected to different convolution operations on branches with convolution kernels of different sizes, resulting in multiple sets of data corresponding to the number of branches. C The multi-scale feature maps of each channel contain feature information of images at different scales. The feature maps of each branch are merged by adding each branch element by element to obtain the channel feature maps. Then, global covariance pooling is used to generate global second-order statistics for each channel feature map. A convolutional layer with a kernel size of 1×1 is used to learn the mapping relationship between features at two scales in the feature vector. Then the mapping relationship feature vector Two sets of feature vectors with different scales and receptive fields are obtained through fully connected layers. The feature vectors of each branch are then processed along the same dimension. softmax The operation yields the weight vector. and Weights are assigned to the feature maps of the branches of different convolutional kernels to obtain the first... c Each feature map output , means as follows: in, For the first c Weight vectors , For the first c Weight vectors , and These are two sets of feature maps of size H×W, each containing C channels; The multi-scale feature maps selected by the nonlinear perception module are used to obtain the total output of the convolutional layer. .