Image super-resolution reconstruction method based on dual-domain dynamic coding

Abstract

The application discloses an image super-resolution reconstruction method based on double-domain dynamic coding. Mainly includes the following steps: training a convolutional neural network model with different magnification; taking a low-resolution image as input, and obtaining a final reconstruction result through the network model trained in the previous step. The method can reconstruct a high-quality high-resolution image from a low-resolution image, and is an effective image super-resolution reconstruction method.

Description

TECHNICAL FIELD

[0001] The present application relates to image resolution enhancement technology, in particular to a lightweight super-resolution reconstruction method based on dual-domain dynamic coding, and belongs to the field of digital image processing. BACKGROUND

[0002] Image super-resolution technology is a branch of image processing and computer vision, aiming to improve the resolution and quality of digital images. It solves the limitations of low-resolution images by generating high-resolution versions with improved details and clarity. This technology is crucial in situations where imaging devices are poor, or when dealing with existing low-resolution images. Image super-resolution technology has a wide range of applications in many fields: in medical scenarios, super-resolution technology helps to produce high-resolution images from low-resolution medical scans, making analysis and diagnosis more accurate. In military reconnaissance scenarios, high-quality remote sensing images help to analyze battlefield situations, achieve military deployment reconnaissance, and evaluate the effectiveness of strikes. Super-resolution is also widely used in surveillance systems, autonomous driving, and video tracking.

[0003] Convolutional networks are widely used in various visual tasks, including image super-resolution. The inherent inductive bias characteristics of convolution operations, such as translational invariance, local connectivity, and sliding window strategy, make them inherently suitable for visual tasks such as image super-resolution. However, convolutional networks have two main limitations. First, spatial locality limits their ability to model long-range correlations. Second, the interaction between spatial positions is fixed through static convolution kernels during inference, limiting their flexibility to adapt to different input content. In recent years, Transformers have provided a solution to these limitations. By introducing a self-attention mechanism in Transformers, global interactions can be explicitly modeled, and the importance of each pixel during inference can be dynamically adjusted through attention scores between global pixel pairs. However, the computational complexity of Transformers grows quadratically with the sequence length (or spatial resolution), posing a challenge for practical applications on resource-limited hardware. This leads to a natural question: how to effectively combine the advantages of convolutional networks and Transformers to develop a lightweight high-performance network suitable for resource-limited devices? SUMMARY

[0004] The purpose of the present application is to combine the advantages of convolutional networks and Transformers, to realize long-range correlation and multi-order interaction through dual-domain dynamic coding, and to construct an efficient image super-resolution reconstruction method suitable for resource-limited devices.

[0005] The application provides a dual domain dynamic encoding network (DDDEN) based on image super-resolution reconstruction.

[0006] (1) training neural network models of different magnifications respectively;

[0007] (2) taking a low-resolution image as input, and obtaining a final reconstruction result through the network model trained in step (1). BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a principle block diagram of the dual domain dynamic encoding network based on image super-resolution reconstruction provided by the application. Wherein, (a) is a general reconstruction block diagram of the DDDEN, (b) is a dual domain dynamic encoding (DDDE) layer structure diagram in (a), (c) is a spatial domain dynamic encoding (SDDE) layer structure diagram in (b), (d) is a channel domain dynamic encoding (CDDE) layer structure diagram in (b), and (e) is a dynamic adaptation layer (DAL) layer structure diagram in (d).

[0009] Figure 2 is a comparison diagram of reconstruction results of a test picture "Barbara" by the application and other eight methods. Wherein, (a) is an original high-resolution image, (b) is a Bicubic processing result, (c) to (i) are experimental results of method 1 to method 7, and (j) is the result of the application.

[0010] Figure 3 is a comparison diagram of reconstruction results of a test picture "ShimatteIkouze_vol01" by the application and other eight methods. Wherein, (a) is an original high-resolution image, (b) is a Bicubic processing result, (c) to (i) are experimental results of method 1 to method 7, and (j) is the result of the application.

[0011] Figure 4 is a comparison diagram of reconstruction results of a test picture "img_12" by the application and other eight methods. Wherein, (a) is an original high-resolution image, (b) is a Bicubic processing result, (c) to (i) are experimental results of method 1 to method 7, and (j) is the result of the application.

[0012] Figure 5This is a comparison chart of the reconstruction results of the test image "img_62" by the present invention and eight other methods. Among them, (a) is the original high-resolution image, (b) is the result of Bicubic processing, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention.

[0013] Figure 6 This is a comparison chart of the reconstruction results of the test image "img_76" by the present invention and eight other methods. Among them, (a) is the original high-resolution image, (b) is the result of Bicubic processing, (c) to (i) are the experimental results of methods 1 to 7, and (j) is the result of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings:

[0015] Figure 1 In this paper, the image super-resolution reconstruction method based on dual-domain dynamic coding can be divided into the following steps:

[0016] (1) Train convolutional neural network models with different magnification factors respectively;

[0017] (2) Using the low-resolution image as input, the final reconstruction result is obtained through the network model trained in step (1);

[0018] Specifically, in step (1), the convolutional neural network model being trained is as follows: Figure 1 As shown in (a), the network structure mainly consists of three parts: feature extraction, nonlinear mapping, and image reconstruction. First, the first layer of the network extracts features from the input image, transforming the image into the feature domain. Then, the high-dimensional features are refined through the nonlinear mapping part. Finally, the reconstruction result is obtained through convolutional layers and sub-pixel sampling layers. The core nonlinear mapping part contains 16 Dual Domain Dynamic Encoding (DDDE) layers, each DDDE containing the same structure: Layer Norm (LN) – Spatial Domain Dynamic Encoding (SDDE) – Layer Norm (LN) – Channel Domain Dynamic Encoding (CDDE).

[0019] For any low-resolution input The output is r represents the amplification factor. The overall implementation of the network model is based on the following process:

[0020] F SHALLOW =fEXT (I) (1)

[0021]

[0022] I SR = f REC (F SHALLOW +F DEEP ) (3)

[0023] where f EXT (·) denotes the initial feature extraction layer, whose output is denoted as F SHALLOW ; denotes the nth DDDE layer, whose output is denoted as F DEEP ; f REC (·) denotes the final reconstruction layer, which contains a convolutional layer and a sub-pixel sampling layer.

[0024] The implementation of the core unit DDDE is based on the following process:

[0025] For a given input feature X

[0026]

[0027] where X' and X" denote the intermediate process output and the final output result of DDDE, respectively, and LN(·) denotes the layer normalization operation: LayerNorm.

[0028] The implementation of SDDE is based on the following process:

[0029] 1. Implicit encoding. For a given input feature X First, X is converted into latent space by two feature conversion layers to generate latent feature and latent feature

[0030] Q = Linear(X), V = Linear(X) (5)

[0031] 2. Spatial perception. In order to effectively extract fine features from coarse features with local connections and spatial redundancy characteristics, it is necessary to introduce structural bias. To achieve this, a depth-wise convolution (DWConv) with a kernel size of 5x5 is used to extract more valuable features from the initial coarse features. This process can be mathematically represented as a transformation of the input feature Q, allowing subsequent feature refinement:

[0032] F REGION = DWConv(Q) (6)

[0033] 3. Long-range spatial relation modeling and multi-order context information interaction. To effectively capture the long-range spatial relation and multi-order context information in visual tasks, both local and global features must be considered. To achieve this, three parallel branches containing depth-wise dilation convolution (DW-D-Conv) are used to realize multi-order interaction and long-range modeling. Specifically, the input feature is split into three sub-parts along the channel dimension: low-order mid-order and high-order and C L +C M +C H =C. Next, the convolution operations of DWDConv L (·) and DWDConv H (·) are applied to features F 5×5,k＝2 and F 7×7,k＝3 respectively to realize long-range correlation modeling, where DWDConv represents the depth separable dilation convolution operation, 5x5 and 7x7 represent the size of the convolution kernel, and k e {2, 3} represents the dilation rate. Finally, the multi-branch responses are aggregated to obtain long-range correlation and multi-order interaction features. This process can be represented by the following formula:

[0034] F MULTI = Concat(DWDConv 5×5,k＝2 (F L ), F M , DWDConv 7×7,k＝3 (F H )) (7)

[0035] 4. Feature aggregation. The extracted high-quality long-range and multi-order feature representations are sent to the last projection layer for two purposes: (1) cross-channel information integration; (2) estimating the importance of each point and generating attention weights. This is achieved using a standard point convolution (PConv) operation.

[0036] F INTERGRATION = PConv(F MULTI ) (8)

[0037] 5. Gating activation. A gating mechanism is designed to capture long-range spatial statistical features from aggregated information. For this purpose, a sigmoid linear unit (SiLU) gate is used, which is an advanced version of Sigmoid with self-stabilizing properties. The SiLU gating function is defined as: x Sigmoid(x). Finally, the generated spatial convolution modulation weights are described by the formula:

[0038] A(F INTERGRATION )=SiLU(F INTERGRATION ) (9)

[0039] Finally, the overall procedure of SDDE can be represented as follows:

[0040] SDDE(X)=Linear(A(F INTERGRATION )⊙V) (10)

[0041] SDDE achieves the advantages of Transformer, i.e., long-range correlation, multi-order interaction, and input content adaptivity, through large kernel convolution and convolution modulation. It is worth noting that our method, like most convolutional networks, only requires linear complexity, rather than quadratic complexity of Transformer.

[0042] The implementation of CDDE is based on the following procedure:

[0043] For any input feature The proposed CDDE layer can be represented as follows:

[0044] CDDE(Y)=DAL(Linear(σ(SAL(Linear(Y))))) (11)

[0045] where Linear(·) represents a linear layer, SAL(·) represents a spatial-aware layer for introducing spatial locality, σ(·) represents a nonlinear activation function such as Gaussian Error Linear Unit (GELU), and the core module DAL(·) represents a dynamic adaptation layer (DAL) for dynamically weighting the importance of different channels, allowing the network to focus on the most relevant features while ignoring redundant or irrelevant information, thereby improving feature diversity and explicitly modeling feature relationships to achieve channel redundancy reduction. DAL(·) is implemented as follows: for any input feature

[0046] DAL(Z)=(S(Z)·C(Z))·Z (12)

[0047]

[0048] S(Z)=Sigmoid(Linear 2c / r→c (Concat(σ(Linear c→c / r (Z)),F SQUEEZE ))) (14)

[0049] where S(·) and C(·) represent the spatial relation modeling branch and the channel relation modeling branch, respectively; GAP(·) represents the global average pooling operation; Linear represents the linear transformation; and Sigmoid(·) represents the activation function Sigmoid. m→n where GAP(·) represents the global average pooling operation; Linear represents the linear transformation; and Sigmoid(·) represents the activation function Sigmoid.

[0050] In step (2), the network model trained in step (1) is input with a low-resolution image to obtain a final reconstruction result.

[0051] To better illustrate the effectiveness of the present application, comparative experiments were performed on commonly used test sets "Set5", "Set14", "Urban100", "Manga109" and "BSDS100". The generation of simulated low-resolution images was as follows: bicubic interpolation was used for 2, 3 and 4 times down-sampling, respectively. In the experiment, bicubic interpolation and 8 typical image super-resolution algorithms were selected as comparative methods.

[0052] The comparative super-resolution reconstruction algorithms were as follows:

[0053] Method 1: the method proposed by Kim et al., reference: Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1646-1654.

[0054] Method 2: Method proposed by Kim et al., reference “Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1637-1645.”

[0055] Method 3: Method proposed by Ahn et al., reference “Ahn N, Kang B, Sohn K A. Fast, accurate, and lightweight super-resolution with cascading residual network [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 252-268.”

[0056] Method 4: Method proposed by Lim et al., reference “Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 136-144.”

[0057] Method 5: Method proposed by Chen et al., reference “Chen H, Gu J, Zhang Z. Attention inattention network for image super-resolution [J]. arXiv preprint arXiv:2104.09497, 2021.”

[0058] Method 6: Method proposed by Wan et al., reference “Wan J, Yin H, Liu Z, et al. Lightweight image super-resolution by multi-scale aggregation[J]. IEEE Transactions on Broadcasting, 2020, 67(2):372-382.”

[0059] Method 7: Method proposed by Liang et al., reference “Liang J, Cao J, Sun G, et al. Swinir: Image restoration using swin transformer[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021:1833-1844.”

[0060] The contents of the comparative experiments are as follows:

[0061] Experiment 1, Bicubic, methods 1 to 18 and the present application are respectively used to simulate the generation of low-resolution images of commonly used test sets “Set5”, “Set14”, “Urban100”, “Manga109” and “BSDS100” for 2, 3 and 4 times reconstruction. The super-resolution reconstruction results are shown in Figure 2 、 Figure 3 and Figure 4 , and the objective evaluation parameters of the reconstruction results are shown in Table 1. Among them, the objective evaluation parameters adopt PSNR (Peak Signal to Noise Ratio) and SSIM (Structure Similarity Index), and the higher the values of the two parameters, the better the reconstruction effect. The parameter quantity and the calculation quantity are used to measure the complexity of the storage burden of the model, and the lower the value, the smaller the burden, which is more conducive to actual deployment.

[0062] Table 1

[0063]

[0064]

[0065] Obviously, at a lower complexity, the present patent (Parameter amount / Computational amount@PSNR / SSIM: 860K / 186.3G@38.26 / 0.9619) obtains better PSNR / SSIM results compared with the latest method, i.e., the latest method 7: Parameter amount / Computational amount@PSNR / SSIM: 878K / 243.7G@38.14 / 0.9611.

[0066] Compared with other methods, the present patent shows better reconstruction results for grid, stripe and text patterns, and has the least blurring and artifacts. This observation confirms the practicability and effectiveness of the present patent. For example, the present patent accurately generates a stripe with a correct direction and the least blurring, while other methods produce an incorrect stripe and obvious blurring in a large range. Figure 2

[0067] In summary, compared with the comparative methods, the reconstruction results of the present patent have great advantages in subjective and objective evaluations. Therefore, the present patent is an effective image super-resolution reconstruction method.​

Claims

1. An image super-resolution reconstruction method based on dual-domain dynamic coding (DDDEN), characterized in that... Convolutional neural network models with different magnification ratios were trained separately. Low-resolution images were used as input, and the final reconstruction result was obtained through training a complete network model. This network model mainly consists of three parts: feature extraction, nonlinear mapping, and image reconstruction. First, the first layer of the network extracts features from the input image, transferring the image to the feature domain. Then, the nonlinear mapping part refines the high-dimensional features. Finally, the reconstruction result is obtained through convolutional layers and sub-pixel sampling layers. The core nonlinear mapping part contains 16 dual-domain dynamic coding layers (DDDEs), each DDDE containing the same structure: normalization layer LN – spatial domain dynamic coding layer SDDE – normalization layer LN – channel domain dynamic coding layer CDDE. The implementation of the core unit DDDE is based on the following process: Given input Where X′ and X″ represent the intermediate output and final output of DDDE, respectively, and LN(·) represents the layer normalization operation: Layer Normalization; The implementation of the core unit SDDE is based on the following process: for a given input —Implicit encoding generates Q and V—Spatial perception generates F REGION ——Long-range modeling and spatial interactive generation F MULTI —Feature aggregation to generate F INTERGRATION —Gated activation A(F) INTERGRATION ): Q=Linear(X),V=Linear(X) (2) F REGION =DWConv(Q) (3) F MULTI =Concat(DWDConv 5×5,k＝2 (F L ),F M ,DWDConv 7×7,k＝3 (F H )) (5) F INTERGRATION =PConv(F MULTI ) (6) A(F INTERGRATION )=SiLU(F INTERGRATION ) (7) Finally, the overall SDDE process can be represented by the following formula: SDDE(X)=Linear(A(F INTERGRATION )⊙V) (8) Where DWConv(·) represents depthwise convolution DWConv; pointwise convolution PConv; SiLU(·) represents the gate function SiLU; and ⊙ represents the dot product operation. The implementation of the core unit CDDE is based on the following process: for any input The proposed CDDE layer can be represented by the following formula: DAL(Z)=(S(Z)·C(Z))·Z (10) S(Z)=Sigmoid(Linear 2c / r→c (Concat(σ(Linear c→c / r (Z)),F SQUEEZE ))) (12) Wherein, Linear(·) represents a linear layer, SAL(·) represents a spatially aware layer used to introduce spatial locality, σ(·) represents a nonlinear activation function such as a Gaussian error linear unit (GELU), and the core module DAL(·) represents a dynamic adaptive layer (DAL); S(·) and C(·) represent the spatial relationship modeling branch and the channel relationship modeling branch, respectively; GAP(·) represents the global average pooling operation; Linear m→n The channel transformation dimension is from m to n; Sigmoid(·) represents the activation function Sigmoid, which is used to transform the extracted features into attention probabilities; and Concat(a,b) represents concatenating features a and b.

Images