Image super-resolution reconstruction method based on double-domain dynamic coding

Abstract

The invention discloses an image super-resolution reconstruction method based on double-domain dynamic coding. The method mainly comprises the following steps: respectively training convolutional neural network models with different magnification times; and taking a low-resolution image as input, and obtaining a final reconstruction result through the network model trained in the previous step. According to the method, a high-quality high-resolution image can be reconstructed from a low-resolution image, and the method is an effective image super-resolution reconstruction method.

Description

Technical Field

[0001] The present invention relates to an image resolution enhancement technology, and in particular to a lightweight super-resolution reconstruction method based on dual-domain dynamic coding, and belongs to the field of digital image processing. Background Art

[0002] Image super-resolution technology is a branch of image processing and computer vision that aims to improve the resolution and quality of digital images. It addresses the limitations of low-resolution images by generating high-resolution versions with improved details and clarity. This technology is crucial in cases where imaging equipment is poor, or when processing existing low-resolution images. Image super-resolution technology has a wide range of applications in many fields: In medical scenarios, super-resolution technology helps to produce high-resolution images from low-resolution medical scans, making analysis and diagnosis more accurate. In military reconnaissance scenarios, high-quality remote sensing images help analyze battlefield situations and realize military applications such as military force deployment reconnaissance and strike effect assessment. Super-resolution is also widely used in surveillance systems, autonomous driving, video tracking, and other aspects.

[0003] Convolutional networks are widely used in various vision tasks, including image super-resolution. The inherent inductive bias properties of the convolution operation, such as translation invariance, local connectivity, and sliding window strategy, make it inherently suitable for vision tasks such as image super-resolution. However, convolutional networks have two major limitations. First, spatial locality limits their ability to model long-range dependencies. Second, during inference, the interactions between spatial locations are fixed by static convolution kernels, limiting their flexibility to adapt to different input contents. In recent years, Transformers have provided a solution to these limitations. By introducing the self-attention mechanism in Transformers, global interactions can be explicitly modeled, and the importance of each pixel during inference can be dynamically adjusted by the attention scores between global pixel pairs. However, the computational complexity of Transformers grows quadratically with the sequence length (or spatial resolution), posing a challenge for practical applications on resource-constrained hardware. This leads to a natural question: how to effectively combine the advantages of convolutional networks and Transformers to develop a lightweight and high-performance network suitable for resource-constrained devices? Summary of the invention

[0004] The purpose of the present invention is to combine the advantages of convolutional networks and Transformers, realize long-range correlation and multi-order interactivity through dual-domain dynamic coding, and then construct an efficient image super-resolution reconstruction method suitable for resource-limited devices.

[0005] The image super-resolution reconstruction method based on dual domain dynamic encoding (Dual Domain Dynamic Encoding Network, DDDEN) proposed in the present invention mainly includes the following steps:

[0006] (1) Train neural network models with different magnifications respectively;

[0007] (2) The low-resolution image is used as input and the final reconstruction result is obtained through the network model trained in step (1). BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a principle block diagram of the image super-resolution reconstruction method based on dual-domain dynamic encoding of the present invention. Among them, (a) is the overall reconstruction block diagram of DDDEN of the present invention, (b) is the structure diagram of the dual-domain dynamic encoding (Dual Domain DynamicEncoding, DDDE) layer in (a), (c) is the structure diagram of the spatial domain dynamic encoding (Spatial Domain DynamicEncoding, SDDE) layer in (b), (d) is the structure diagram of the channel domain dynamic encoding (Channel Domain DynamicEncoding, CDDE) layer in (b), and (e) is the structure diagram of the dynamic adaptation (Dynamic Adaptation Layer, DAL) layer in (d).

[0009] Figure 2 This is a comparison of the reconstruction results of the test image "Barbara" by the present invention and the other eight methods. Among them, (a) is the original high-resolution image, (b) is the Bicubic processing result, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention.

[0010] Figure 3 This is a comparison chart of the reconstruction results of the test image "ShimatteIkouze_vol01" by the present invention and the other eight methods. Among them, (a) is the original high-resolution image, (b) is the Bicubic processing result, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention.

[0011] Figure 4 This is a comparison chart of the reconstruction results of the test image "img_12" by the present invention and the other eight methods. Among them, (a) is the original high-resolution image, (b) is the Bicubic processing result, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention.

[0012] Figure 5This is a comparison chart of the reconstruction results of the test image "img_62" by the present invention and the other eight methods. Among them, (a) is the original high-resolution image, (b) is the Bicubic processing result, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention.

[0013] Figure 6 This is a comparison chart of the reconstruction results of the test image "img_76" by the present invention and the other eight methods. Among them, (a) is the original high-resolution image, (b) is the Bicubic processing result, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be further described below in conjunction with the accompanying drawings:

[0015] Figure 1 In the present invention, the image super-resolution reconstruction method based on dual-domain dynamic coding can be divided into the following steps:

[0016] (1) Train convolutional neural network models with different magnifications respectively;

[0017] (2) Taking the low-resolution image as input, the network model trained in step (1) is used to obtain the final reconstruction result;

[0018] Specifically, in step (1), the convolutional neural network model to be trained is as follows: Figure 1 (a) As shown. The network structure mainly includes three parts: feature extraction, nonlinear mapping and image reconstruction. First, the input image is feature extracted through the first layer of the network and the image is transferred into the feature domain. Then, the high-dimensional features are refined through the nonlinear mapping part. Finally, the reconstruction result is obtained through the convolution layer and the sub-pixel sampling layer. The core nonlinear mapping part contains 16 dual domain dynamic encoding layers (Dual Domain Dynamic Encoding, DDDE), each DDDE contains the same structure: normalization layer (LayerNorm, LN) - spatial domain dynamic encoding layer (Spatial Domain Dynamic Encoding, SDDE) - normalization layer (Layer Norm, LN) - channel domain dynamic encoding layer (Channel Domain Dynamic Encoding, CDDE).

[0019] For any low-resolution input The output is r represents the amplification factor. The overall implementation of the network model is based on the following process:

[0020] F SHALLOW =fEXT (I) (1)

[0021]

[0022] I SR =f REC (F SHALLOW +F DEEP ) (3)

[0023] Among them, f EXT (·) represents the initial feature extraction layer, and its output feature is represented as F SHALLOW ; It represents the nth DDDE layer, and its output feature is represented as F DEEP ;f REC (·) denotes the final reconstruction layer, which consists of a convolutional layer and a sub-pixel sampling layer.

[0024] The implementation of the core unit DDDE is based on the following process:

[0025] For a given input feature

[0026]

[0027] Among them, X′ and X″ represent the intermediate process output and final output result of DDDE respectively, and LN(·) represents the layer normalization operation: LayerNorm.

[0028] The implementation of SDDE is based on the following process:

[0029] 1. Implicit encoding. For a given input feature First, X is transferred into the latent space through two feature conversion layers to generate latent features and potential characteristics

[0030] Q=Linear(X),V=Linear(X) (5)

[0031] 2. Spatial perception. In order to effectively extract fine features from coarse features with local connections and spatial redundancy characteristics, it is necessary to introduce structural bias. To achieve this, a depth-wise convolution (DWConv) with a kernel size of 5×5 is used to extract more valuable features from the initial coarse features. This process can be mathematically represented as a transformation of the input feature Q, allowing subsequent feature refinement:

[0032] F REGION =DWConv(Q) (6)

[0033] 3. Long-range spatial relationship modeling and multi-order context information interaction. In order to effectively obtain long-range spatial relationships and multi-order context information in visual tasks, local and global features must be considered simultaneously. To achieve this, three parallel branches including depth-wise dilation convolution (DW-D-Conv) are used to achieve multi-order interaction and long-range modeling. Specifically, the input feature It is split into three sub-parts along the channel dimension: low-order Intermediate and high-level And C L +C M +C H = C. Next, for feature F L and F H Apply DWDConv separately 5×5,k＝2 (·) and DWDConv 7×7,k＝3 The convolution operation of (·) realizes long-range correlation modeling, where DWDConv represents the depth-separable dilated convolution operation, 5×5 and 7×7 represent the size of the convolution kernel, and k∈{2,3} represents the dilation rate. Finally, the multi-branch responses are aggregated to obtain long-range correlation and multi-order interaction features. This process can be expressed as follows:

[0034] F MULTI =Concat(DWDConv 5×5,k＝2 (F L ),F M ,DWDConv 7×7,k＝3 (F H )) (7)

[0035] 4. Feature aggregation. The extracted high-quality long-range and multi-order feature representations are fed into the last projection layer for two purposes: (1) cross-channel information integration; (2) estimating the importance of each point and generating attention weights. This is achieved using a standard point-convolution (PConv) operation.

[0036] F INTERGRATION =PConv(F MULTI ) (8)

[0037] 5. Gated activation. A gating mechanism is designed to capture long-range spatial statistical features from aggregated information. To this end, the Sigmoid Linear Unit (SiLU) gating is used, which is an advanced version of Sigmoid with self-stabilizing properties. The SiLU gating function is defined as: x·Sigmoid(x). Finally, the generated spatial convolution modulation weights are described by the formula:

[0038] A(F INTERGRATION )=SiLU(F INTERGRATION ) (9)

[0039] Finally, the overall process of SDDE can be expressed as follows:

[0040] SDDE(X)=Linear(A(F INTERGRATION )⊙V) (10)

[0041] SDDE achieves the advantages of Transformer through large kernel convolution and convolution modulation: long-range correlation, multi-order interactivity, and input content adaptability. It is worth noting that our method, like most convolutional networks, only requires linear complexity, rather than the quadratic complexity of Transformer.

[0042] The implementation of CDDE is based on the following process:

[0043] For any input feature The proposed CDDE layer can be expressed as follows:

[0044] CDDE(Y)=DAL(Linear(σ(SAL(Linear(Y))))) (11)

[0045] Among them, Linear(·) represents the linear layer, SAL(·) represents the spatial perception layer, which is used to introduce spatial locality, σ(·) represents the nonlinear activation function such as Gaussian Error Linear Unit (GELU), and the core module DAL(·) represents the Dynamic Adaptation Layer (DAL), which is used to dynamically weigh the importance of different channels, allowing the network to focus on the most relevant features while ignoring redundant or irrelevant information, thereby improving the diversity of features and explicitly modeling feature relationships to achieve channel redundancy reduction. The specific implementation of DAL(·) is as follows: for any input feature feature

[0046] DAL(Z)=(S(Z)·C(Z))·Z (12)

[0047]

[0048] S(Z)=Sigmoid(Linear 2c / r→c (Concat(σ(Linear c→c / r (Z)),F SQUEEZE ))) (14)

[0049] Among them, S(·) and C(·) represent the spatial relationship modeling branch and the channel relationship modeling branch respectively; GAP(·) represents the global average pooling operation; Linear m→n denotes the channel transformation dimension from m to n; Sigmoid(·) denotes the activation function Sigmoid, which is used to transform the extracted feature map into attention probability and Concat(a,b) denotes the concatenation of features a and b. It is worth noting that the proposed DAL layer achieves adaptability not only in the spatial dimension but also in the channel dimension. These improve the efficiency and performance of the network in modeling long-range dependencies and local information. By doing so, the proposed CDDE layer is able to flexibly and effectively modulate the feature representation, facilitating the modeling of complex image patterns with high adaptability and representation capabilities.

[0050] In the step (2), the low resolution is input into the network model trained in the step (1) to obtain the final reconstruction result.

[0051] In order to better illustrate the effectiveness of the present invention, comparative experiments were conducted on the commonly used test sets "Set5", "Set14", "Urban100", "Manga109" and "BSDS100". The generation method of simulating low-resolution images: using bicubic interpolation to perform 2, 3 and 4 times downsampling respectively. In the experiment, bicubic interpolation and 8 typical image super-resolution algorithms were selected as comparison methods.

[0052] The compared super-resolution reconstruction algorithms are:

[0053] Method 1: The method proposed by Kim et al., reference "Kim J, Lee JK, Lee K M. Accurate image super-resolution using very deep convolutional networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1646-1654."

[0054] Method 2: The method proposed by Kim et al., reference “Kim J, Lee JK, Lee K M. Deeply-recursive convolutional network for image super-resolution[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:1637-1645.”

[0055] Method 3: The method proposed by Ahn et al., reference “Ahn N, Kang B, Sohn K A. Fast, accurate, and lightweight super-resolution with cascading residual network[C] / / Proceedings of the European conference on computer vision(ECCV).2018:252-268.”

[0056] Method 4: The method proposed by Lim et al., reference "Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 136-144."

[0057] Method 5: The method proposed by Chen et al., reference “Chen H, Gu J, Zhang Z. Attention inattention network for image super-resolution[J].arXiv preprint arXiv:2104.09497,2021.”

[0058] Method 6: The method proposed by Wan et al., reference “Wan J, Yin H, Liu Z, et al. Lightweight image super-resolution by multi-scale aggregation[J]. IEEE Transactions on Broadcasting, 2020, 67(2): 372-382.”

[0059] Method 7: Liang et al., reference “Liang J, Cao J, Sun G, et al. Swinir: Image restoration using swin transformer [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 1833-1844.”

[0060] The contents of the comparative experiment are as follows:

[0061] Experiment 1, Bicubic, methods 1 to 18 and the present invention are used to reconstruct the low-resolution images simulated by the commonly used test sets "Set5", "Set14", "Urban100", "Manga109" and "BSDS100" by 2, 3 and 4 times. The super-resolution reconstruction results are shown in Figure 1. Figure 2 , Figure 3 and Figure 4 The objective evaluation parameters of the reconstruction results are shown in Table 1. The objective evaluation parameters are PSNR (Peak Signal to Noise Ratio) and SSIM (Structure Similarity Index). The higher the values ​​of the two parameters, the better the reconstruction effect. The number of parameters and the amount of calculation are used to measure the complexity of the storage burden of the model. The lower the value, the smaller the burden, which is more conducive to actual deployment.

[0062] Table 1

[0063]

[0064]

[0065] Obviously, at lower complexity, the patented invention (parameter amount / computation amount @PSNR / SSIM: 860K / 186.3G@38.26 / 0.9619) achieves better PSNR / SSIM results than recent methods, such as the latest method 7: parameter amount / computation amount @PSNR / SSIM: 878K / 243.7G@38.14 / 0.9611.

[0066] Compared with other methods, the patent of this invention shows better reconstruction effect of grid, stripe and text patterns with minimal blur and artifacts. This observation confirms the practicality and effectiveness of the patent of this invention. Figure 2 For example, the patented method accurately generates stripes with the correct orientation and minimal blur, while other methods produce incorrect stripes with noticeable blur over a wide range.

[0067] In summary, compared with the comparison method, the reconstruction results of the present invention have great advantages in both subjective and objective evaluation. Therefore, the present invention is an effective image super-resolution reconstruction method.

Claims

1. Image super-resolution reconstruction method based on Dual Domain Dynamic Encoding Network (DDDEN), characterized by The following steps are involved: Step 1: Train convolutional neural network models with different magnifications respectively; Step 2: Take the low-resolution image as input and obtain the final reconstruction result through the network model trained in step 1.

2. The image super-resolution reconstruction method based on dual-domain dynamic coding according to claim 1 is characterized in that The convolutional neural network model described in step 1; the network model mainly includes three parts: feature extraction, nonlinear mapping and image reconstruction; first, the input image is feature extracted through the first layer of the network and the image is transferred to the feature domain; then, the high-dimensional features are refined through the nonlinear mapping part; finally, the reconstruction result is obtained through the convolution layer and the sub-pixel sampling layer; the core nonlinear mapping part contains 16 dual domain dynamic encoding layers (Dual Domain Dynamic Encoding, DDDE), each DDDE contains the same structure: normalization layer (Layer Norm, LN) - spatial domain dynamic encoding layer (Spatial Domain Dynamic Encoding, SDDE) - normalization layer (Layer Norm, LN) - channel domain dynamic encoding layer (Channel Domain Dynamic Encoding, CDDE), the implementation of the core unit DDDE is based on the following process: For a given input Among them, X′ and X″ represent the intermediate process output and final output result of DDDE respectively, and LN(·) represents the layer normalization operation: Layer Norm.

3. The image super-resolution reconstruction method based on dual-domain dynamic coding according to claim 1 is characterized in that The convolutional neural network model described in step 1; the implementation of the core unit SDDE is based on the following process: for a given input —— Implicit coding generates Q and V —— Spatial perception generates F REGION ——Long-range modeling and spatial interaction generation F MULTI ——Feature aggregation generates F INTERGRATION ——Gated activation A(F INTERGRATION ): Q=Linear(X),V=Linear(X) (2) F REGION =DWConv(Q) (3) F MULTI =Concat(DWDConv 5×5,k＝2 (F L ),F M ,DWDConv 7×7,k＝3 (F H )) (5) F INTERGRATION =PConv(F MULTI ) (6) A(F INTERGRATION )=SiLU(F INTERGRATION ) (7) Finally, the overall process of SDDE can be expressed as follows: SDDE(X)=Linear(A(F INTERGRATION )⊙V) (8) Among them, DWConv(·) represents depth-wise convolution (DWConv); point convolution (PConv); SiLU(·) represents the gating function SiLU (Sigmoid Linear Unit, SiLU); ⊙ represents the point multiplication operation.

4. The image super-resolution reconstruction method based on dual-domain dynamic coding according to claim 1 is characterized in that The convolutional neural network model described in step 1; the implementation of the core unit CDDE is based on the following process: For any input The proposed CDDE layer can be expressed as follows: DAL(Z)=(S(Z)·C(Z))·Z (10) S(Z)=Sigmoid(Linear 2c / r→c (Concat(σ(Linear c→c / r (Z)),F SQUEEZE ))) (12) Among them, Linear(·) represents the linear layer, SAL(·) represents the spatial perception layer, which is used to introduce spatial locality, σ(·) represents the nonlinear activation function such as Gaussian Error Linear Unit (GELU), and the core module DAL(·) represents the dynamic adaptation layer (DAL); S(·) and C(·) represent the spatial relationship modeling branch and the channel relationship modeling branch respectively; GAP(·) represents the global average pooling operation; Linear m→n Indicates that the channel transformation dimension is from m to n; Sigmoid(·) represents the activation function Sigmoid, which is used to convert the extracted feature map into attention probability and Concat(a,b) represents the concatenation of features a and b.

Images