A method for constructing a neural network-based image super-resolution model

Abstract

The application is a kind of construction method of image super-resolution model based on neural network. It includes the following steps: (1) preparing data set; (2) training data preprocessing; (3) constructing light weight deep learning model DLGSANet: constructing local information extraction submodel with dynamic weight generation network; constructing global information extraction submodel with sparse attention mechanism; combining local information extraction submodel and global information extraction submodel to form basic module HDTB; stacking basic module to form hybrid dynamic Transformer module group RHDTG; formulating proposed residual hybrid dynamic Transformer module group RHDTG as end-to-end deep CNN model DLGSANet; (4) deep learning model parameter training; (5) model packaging. The application has very small parameter quantity and very small calculation quantity, can effectively fuse local information and global information, and can achieve very good image super-resolution performance under limited cost.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a method for constructing an image super-resolution model based on neural networks. Background Technology

[0002] Image super-resolution is a crucial component of computer vision, specifically within the fundamental field of visual processing. It involves using algorithms to fill in and enhance pixels of a low-resolution input image, generating a super-resolution image with more detail. Image super-resolution has wide applications in numerous practical problems. For example, in security cameras, it performs super-resolution operations to obtain clearer, more detailed high-resolution images; in mobile devices, it performs super-resolution operations on photographed images to obtain higher-resolution images; and in network applications, it uses super-resolution algorithms to perform super-resolution operations on compressed low-resolution images transmitted over the network to obtain high-resolution images.

[0003] In recent years, with the development of deep learning and machine learning, image super-resolution technology has also seen unprecedented progress. However, generating higher-resolution images from the limited information provided by low-resolution images has always been a challenging problem. Traditional image super-resolution methods generate high-resolution images through interpolation methods (nearest neighbor interpolation, bicubic interpolation, etc.). With the introduction of deep learning methods, deeper and more powerful convolutional neural networks can generate high-resolution images from low-resolution input images, and the quality of the generated high-resolution images is significantly improved. However, the learning ability of convolutional neural networks is limited by their fixed weight matrix. Therefore, with the development of computer vision, the Transformer with dynamic weights has been introduced into computer vision. Its dynamic weights can effectively extract more structural and high-frequency information from the limited information provided by low-resolution images, and can better restore details and generate higher-quality high-resolution images. However, both convolutional neural network methods and Transformer methods face the challenge of maintaining the model performance for generating high-resolution images through a large number of parameters and computational costs.

[0004] Meanwhile, existing methods are almost all limited to extracting local or global information, without considering the fusion of local and global information, thus facing certain performance bottlenecks.

[0005] Moreover, with the development of technology, mobile devices have reached an unprecedented scale, and the demand for lightweight and efficient deep learning models is increasing. Traditional high-performance image super-resolution deep learning algorithms cannot well meet the requirements of lightweight and efficiency in terms of model parameter quantity and computational cost. Summary of the Invention

[0006] The purpose of this invention is to provide a method for constructing an image super-resolution model based on a neural network.

[0007] The technical solution for achieving the objective of this invention is: a method for constructing an image super-resolution model based on a neural network, comprising the following steps:

[0008] Step (1): Prepare the dataset;

[0009] Step (2): Training data preprocessing;

[0010] Step (3): Constructing the lightweight deep learning model DLGSANet: Constructing the local information extraction sub-model MHDLSA with a dynamic weight generation network; constructing the global information extraction sub-model SparseGSA with a sparse attention mechanism; combining the local information extraction sub-model MHDLSA and the global information extraction sub-model SparseGSA to form the basic module HDTB; stacking the basic module HDTB to form the hybrid dynamic Transformer module group RHDTG, and defining the proposed hybrid dynamic Transformer module group RHDTG as an end-to-end deep CNN model, i.e., the lightweight deep learning model DLGSANet;

[0011] Step (4): Deep learning model parameter training: Use the training data processed in step (2) to train the deep learning model DLGSANet constructed in step (3);

[0012] Step (5): Model encapsulation.

[0013] Furthermore, in step (1), an open-source dataset is used to train and validate the dataset. The open-source dataset consists of data pairs. For high-resolution images, the images are downsampled using bicubic interpolation to generate low-resolution images at the corresponding downsampling scale.

[0014] Furthermore, in step (2), the data pairs in the training dataset in step (1) are cropped to form sub-data pairs of the same size, thus forming the training dataset.

[0015] Furthermore, step (3) "constructing the local information extraction sub-model MHDLSA with a dynamic weight generation network" specifically means:

[0016] Given features Consists of a normalization layer and a Convolutional generation: Design a squeeze and activation network SENet without any normalization layers and non-linear activations as a dynamic weight generation network; use deep convolutional layers in SENet;

[0017] Dynamic weight generation networks are implemented in the following ways:

[0018]

[0019]

[0020]

[0021] in Represents the real number field. Indicates the length and width range of the image. Indicates the number of channels. Indicates the compression factor; Indicates the filter size is Depth convolution of pixels; Indicates the filter size is Pixel convolution; Indicates the refactoring function; Indicates pixel index, For the generated dynamic weights; each pixel has an associated weight. Dynamic kernels are used for dynamic convolution;

[0022] Use the generated pixel-level weights Aggregation features are obtained through the following methods. :

[0023]

[0024] in This indicates dynamic convolution, which has a weight-sharing mechanism for each channel in each group;

[0025] Apply the improved feedforward network to aggregate features :

[0026]

[0027] in This indicates a FeedForward Network. These are the features of the feedforward network.

[0028] Furthermore, step (3) "constructing a global information extraction sub-model SparseGSA with a sparse attention mechanism" specifically means:

[0029] Given normalized features generated by the MHDLSA module ,use Convolution, and then Depth convolution generates query key , and value ;

[0030] For each query ,key Sum Applying the reconstruction function, we get , ,and ;

[0031] Self-attention is calculated in the following way:

[0032]

[0033] in Represents a nonlinear activation unit. These are learnable parameters, and the attention can be estimated. Generate output aggregate features Then the feedforward network is applied To generate output ;

[0034] The SparseGSA sub-model, which extracts global information, preserves useful features for high-resolution image reconstruction.

[0035] Furthermore, in step (3), "combining the local information extraction sub-model MHDLSA and the global information extraction sub-model SparseGSA to form the basic module HDTB; stacking the basic module HDTB to form the hybrid dynamic Transformer module group RHDTG, and formulating the proposed hybrid dynamic Transformer module group RHDTG as an end-to-end deep CNN model, i.e., the lightweight deep learning model DLGSANet" specifically means:

[0036] Develop a hybrid dynamic Transformer base module HDTB that includes the global information extraction sub-models MHDLSA and SparseGSA; based on HDTB, develop a hybrid dynamic Transformer module group RHDTG to reduce training difficulty;

[0037] Given input features The proposed RHDTG is implemented in the following way:

[0038]

[0039]

[0040] in Indicates the first One HDTB, For the first The intermediate features generated by HDTB are used to formulate the proposed RHDTG as an end-to-end deep CNN model, DLGSANet, to solve the image super-resolution problem.

[0041] Compared with the prior art, the significant advantages of this invention are:

[0042] 1. Compared with existing methods, this invention solves the above problems from a novel perspective, and can better meet the requirements of lightweight and high efficiency; compared with existing convolutional neural network methods, this invention can achieve higher image super-resolution results than existing convolutional neural network methods with only about 10% of the number of parameters and computational overhead; compared with existing Transformer methods, this invention can achieve higher image super-resolution results than existing convolutional neural network methods with only about 50% of the number of parameters and computational overhead.

[0043] 2. Compared with existing methods, this invention has innovatively invented HDTB, which uses an efficient fusion of local information extraction and global information extraction to improve the performance of our deep learning model, thereby compressing the overall model parameter count and reducing the model's computational load, thus achieving the requirements of lightweight and high efficiency.

[0044] 3. Regarding the local information extraction sub-model MHDLSA, this invention innovatively uses a novel convolutional neural network model based on dynamic weights. Under the design of this invention, compared with the existing convolutional neural network method model with fixed weights, this local information extraction sub-model can effectively integrate the convolution operation characteristics of convolutional neural networks and the dynamic weight characteristics of Transformers, and can extract local feature information very efficiently under certain parameter and computational requirements.

[0045] 4. Regarding the global information extraction sub-model SparseGSA, this invention innovatively designs a Transformer structure sub-model with a sparse attention mechanism. Under the design of this invention, compared with the existing Transformer method that retains all attention, this global information extraction sub-model has a sparse attention mechanism, which can retain effective attention information, remove redundant attention information, and efficiently fuse global information.

[0046] 5. Compared to existing methods that are limited to using local information or global information alone, this invention cleverly integrates these two sub-models into a single module, HDTB, which serves as the foundation module of the deep learning model designed in this invention. This module innovatively integrates local and global information, resulting in very high feature extraction capabilities. Furthermore, this invention uses this foundation module to design an end-to-end deep learning neural network model, DLGSANet, which has a very small model size and very low computational overhead, and can generate very high-quality, high-resolution images.

[0047] 6. Specifically, compared to existing methods that have a large number of parameters and computational overhead, this invention innovatively proposes a lightweight and efficient deep learning model for image super-resolution. This model can effectively fuse local and global information and achieve very good image super-resolution performance with limited overhead. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the overall model DLGSANet constructed in this invention.

[0049] In the diagram, LR stands for Low Resolution (low-resolution image input); SR stands for Super Resolution (super-resolution image output); Shallow Feature Extraction; Deep Feature Extraction; HR stands for High Resolution Image Reconstruction; Reconstruction is a high-resolution image reconstruction; pixel shuffle is pixel shuffle; DWConv stands for DepthWise Convolution; Norm stands for Normalize; Dynamic Weight is dynamic weight; FFN stands for FeedForward Network; Dynamic Convolution is dynamic convolution; Matrix Multiplication is matrix multiplication; Element-wise Multiplication is element-wise multiplication; Element-wise Addition is element-wise addition; Activation is activation function; and Reshape is reshaping.

[0050] Figure 2This is a schematic diagram of the local information extraction sub-model of the present invention.

[0051] Figure 3 This is a schematic diagram of the global information extraction sub-model of the present invention. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings.

[0053] This method innovatively designs a foundational module, the Hybrid Dynamic Transformer Block (HTDB). This module combines the local information extraction sub-model Multi-Head Dynamic Local Self-Attention (MHDLSA) and the global information extraction sub-model Sparse Global Self-Attention (SparseGSA) from this method, efficiently fusing local and global information to achieve highly efficient feature extraction capabilities. The method also innovatively designs the local information extraction sub-model Multi-Head Dynamic LocalSelf-Attention (MHDLSA). This sub-model effectively combines the convolutional operation characteristics of convolutional neural networks with the dynamic weight characteristics of Transformers, efficiently extracting local feature information under certain parameter and computational requirements. Furthermore, this method innovatively designs the global information extraction sub-model Sparse GlobalSelf-Attention (SparseGSA). This global information extraction sub-model possesses a sparse attention mechanism, preserving effective attention information, eliminating redundant attention information, and efficiently fusing global information. Finally, this method designs a lightweight deep learning model, Dynamic Local and Global Self-Attention Networks (DLGSANet). This model consists of the basic module (HDTB) of this method, possessing a very small number of parameters and minimal computational cost. It effectively fuses local and global information, achieving excellent image super-resolution performance with limited overhead. The specific method is as follows:

[0054] Step 1: Prepare the dataset. We use an open-source dataset for our training and validation datasets. This type of dataset consists of data pairs. For high-resolution images, we use bicubic interpolation to downsample the images and generate low-resolution images at the corresponding downsampling scale.

[0055] Step 2: Training Data Preprocessing. The data pairs in the training dataset from Step 1 are cropped into sub-data pairs of the same size to form our training dataset. This preprocessing operation aims to reduce the training overhead of the model in our method.

[0056] Step 3: Construct the lightweight deep learning model DLGSANet based on this method. The local information extraction sub-model MHDLSA from this method is used as the local information extraction model. This sub-model has a dynamic weight generation network that extracts corresponding weights for the input features, thus achieving different dynamic weights for different inputs, resulting in very high local information feature extraction performance. The global information extraction sub-model SparseGSA from this method is used as the global information extraction model. This global information extraction model retains effective attention information and eliminates redundant attention information through the cleverly designed sparse attention mechanism of this method, thus achieving very high global feature information extraction performance. Combining the local and global information extraction sub-models from this method forms the basic module HDTB of this method, which fuses the extracted local and global information. Finally, the basic modules are stacked to form a Residual Hybrid Dynamic Transformer Group (RHDTG), and combined with commonly used components to form our lightweight deep learning model DLGSANet.

[0057] in:

[0058] MHDLSA Local Information Extraction Submodule:

[0059] Specifically, given a feature Consists of a normalization (norm) layer and a For convolution generation, we first designed a Squeeze and Excitation Network (SENet) without any normalization layers or nonlinear activations as our dynamic weight generation network. To ensure that the generated dynamic weights better simulate local information, we further used deep convolutional layers in SENet, as deep convolution operations can model local attention.

[0060] The proposed dynamic weight generation is achieved through the following method:

[0061]

[0062]

[0063]

[0064] in Represents the real number field. Indicates the length and width range of the image. Indicates the number of channels. Indicates the compression factor; Indicates the filter size is Depth convolution of pixels; Indicates the filter size is Pixel convolution; Indicates the refactoring function; Indicates pixel index, For the generated dynamic weights. Each pixel has an associated weight. Dynamic kernels are used for dynamic convolution.

[0065] Use the generated pixel-level weights We obtain the aggregation features in the following way. :

[0066]

[0067] in This represents dynamic convolution, which has a weight-sharing mechanism for each channel in each group.

[0068] Since feedforward networks are widely used in Transformers to obtain better feature representation capabilities, we further apply an improved feedforward network to aggregate features. :

[0069]

[0070] in This indicates a FeedForward Network. Features of the feedforward network

[0071] SparseGSA Global Information Extraction Submodule:

[0072] Given normalized features generated by the MHDLSA module We first use Convolution, and then Depth-wise convolution generates queries. key , and value .

[0073] We will separately query ,key Sum Applying the reconstruction function, we get , , and .

[0074] To retain the most useful attention for feature aggregation, we compute self-attention as follows:

[0075]

[0076] in Represents a nonlinear activation unit. These are learnable parameters. With the estimated attention... We generate output aggregate features Then the feedforward network is applied. To generate output .

[0077] With the proposed SparseGSA, we can preserve the most useful features for high-resolution image reconstruction.

[0078] RHDTG:

[0079] By exploring MHDLSA and SparseGSA, we developed a Hybrid Dynamic Transformer Base Module (HDTB) incorporating both MHDLSA and SparseGSA for local and global feature estimation. Building upon HDTB, we further developed a Hybrid Dynamic Transformer module group RHDTG to reduce training difficulty.

[0080] Given input features The proposed RHDTG is implemented in the following way:

[0081]

[0082]

[0083] in Indicates the first One HDTB, For the first We generate intermediate features from HDTB. Finally, we formulate the proposed RHDTG as an end-to-end deep CNN model, DLGSANet, to solve the image super-resolution problem.

[0084] Step 4: Deep learning model parameter training. Using the training data processed in Step 2, we train the deep learning model constructed in Step 3. Through multiple training iterations and optimizations, we obtain the model with the best performance.

[0085] Step 5: Model encapsulation. The deep learning model obtained through steps 2-4 is encapsulated, compressed, and encrypted to form algorithm components.

Claims

1. A method for constructing an image super-resolution model based on a neural network, characterized in that, Includes the following steps: Step (1): Prepare the dataset; Step (2): Training data preprocessing; Step (3): Constructing the lightweight deep learning model DLGSANet: Constructing the local information extraction sub-model MHDLSA with a dynamic weight generation network; constructing the global information extraction sub-model SparseGSA with a sparse attention mechanism; combining the local information extraction sub-model MHDLSA and the global information extraction sub-model SparseGSA to form the basic module HDTB; stacking the basic module HDTB to form the hybrid dynamic Transformer module group RHDTG, and defining the proposed hybrid dynamic Transformer module group RHDTG as an end-to-end deep CNN model, i.e., the lightweight deep learning model DLGSANet; Step (4): Deep learning model parameter training: Use the training data processed in step (2) to train the deep learning model DLGSANet constructed in step (3); Step (5): Model encapsulation; The "construction of the local information extraction sub-model MHDLSA with a dynamic weight generation network" in step (3) specifically refers to: Given features Consists of a normalized norm layer and a Convolutional generation: Design a squeeze and activation network SENet without any normalization layers and non-linear activations as a dynamic weight generation network; use deep convolutional layers in SENet; Dynamic weight generation networks are implemented in the following ways: ， ， ， in Represents the real number field. Indicates the length and width range of the image. Indicates the number of channels. Indicates the compression factor; Indicates the filter size is Depth convolution of pixels; Indicates the filter size is Pixel convolution; Indicates the refactoring function; Indicates pixel index, For the generated dynamic weights; each pixel has an associated weight. Dynamic kernels are used for dynamic convolution; Use the generated pixel-level weights Aggregation features are obtained through the following methods. : ， in This indicates dynamic convolution, which has a weight-sharing mechanism for each channel in each group; Apply the improved feedforward network to aggregate features : ， in This indicates a FeedForward Network. Features derived from the feedforward network; The "construction of a global information extraction sub-model SparseGSA with a sparse attention mechanism" in step (3) specifically refers to: Given normalized features generated by the MHDLSA module ,use Convolution, and then Depth convolution generates query key , and value ; For each query ,key Sum Applying the reconstruction function, we get , , and ; Self-attention is calculated in the following way: ， in Represents a nonlinear activation unit. These are learnable parameters, and the attention can be estimated. Generate output aggregate features ; Then the feedforward network is applied To generate output ; The global information extraction sub-model SparseGSA preserves useful features for high-resolution image reconstruction. Step (3) "combining the local information extraction sub-model MHDLSA and the global information extraction sub-model SparseGSA to form the basic module HDTB; stacking the basic module HDTB to form the hybrid dynamic Transformer module group RHDTG, and formulating the proposed hybrid dynamic Transformer module group RHDTG as an end-to-end deep CNN model, i.e., the lightweight deep learning model DLGSANet" specifically means: Develop a hybrid dynamic Transformer base module HDTB that includes the global information extraction sub-models MHDLSA and SparseGSA; based on HDTB, develop a hybrid dynamic Transformer module group RHDTG to reduce training difficulty; Given input features The proposed RHDTG is implemented in the following way: ， ， in Indicates the first One HDTB, For the first The intermediate features generated by HDTB are used to formulate the proposed RHDTG as an end-to-end deep CNN model DLGSANet to solve the image super-resolution problem.

2. The method according to claim 1, characterized in that, In step (1), an open-source dataset is used to train and validate the dataset. The open-source dataset consists of data pairs. For high-resolution images, the images are downsampled using bicubic interpolation to generate low-resolution images at the corresponding downsampling scale.

3. The method according to claim 2, characterized in that, In step (2), the data pairs in the training dataset from step (1) are cropped to form sub-data pairs of the same size, thus forming the training dataset.

Images