A lightweight image super-resolution method and system based on dynamic reparameterization

Abstract

A lightweight image super-resolution method based on dynamic reparameterization includes: S1 inputting a preprocessed low-resolution original image into a CNN-based lightweight image super-resolution backbone network and image reconstruction module to obtain a super-resolution image; S2 calculating a loss function based on the super-resolution image and the corresponding high-resolution original image after preprocessing, and updating the network weights through backpropagation; S3 expanding the convolutional layer with the largest contribution score in the iteration cycle into an enhanced edge-oriented reparameterization branch module according to the iteration cycle and initialization conditions, and pruning all reparameterization branches with the smallest contribution score that have been expanded into reparameterization branch modules in the iteration cycle; repeating steps S1-S3 until the backbone network converges. This method improves reconstruction performance without changing the lightweight network parameters of the backbone network and maintaining fast inference speed. Compared to other reparameterization methods, it can significantly reduce training time and memory consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a lightweight image super-resolution method based on dynamic re-parameterization and an application thereof. Background Art

[0002] The goal of lightweight image super-resolution (ESR) is to minimize the running time and memory consumption during inference while maintaining a certain reconstruction quality. Common improvement methods include re-parameterization (Rep), knowledge distillation, pruning, etc. The re-parameterization method enhances the representation ability of the network by introducing operations that are more conducive to mining feature information during training. In the inference stage, complex operations can be equivalently converted into simpler operations, thereby improving the inference speed of the network. Re-parameterization has become the most mainstream method in lightweight image super-resolution research. However, the current super-resolution re-parameterization methods convert all convolutional layers of the feature extraction part into re-parameterized branch structures, which will lead to the problem of over-parameterization, resulting in excessive consumption of computing resources during training and complex network inference. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a lightweight image super-resolution method based on dynamic re-parameterization (DyRepSR, A Efficient Image Super-resolution Method based on Dynamic Re-parameterization). The dynamic re-parameterization method is adopted to improve the reconstruction performance without changing the lightweight network parameters of the backbone network and maintaining a fast inference speed. Compared with other re-parameterization methods, the training time and memory consumption can be greatly reduced.

[0004] The first aspect of the present invention provides:

[0005] A lightweight image super-resolution method based on dynamic reparameterization, comprising:

[0006] S1 inputs the preprocessed low-resolution original image into the CNN-based lightweight image super-resolution backbone network and image reconstruction module to obtain a super-resolution image;

[0007] S2 calculates the loss function based on the super-resolution image and the true value image preprocessed by the corresponding high-resolution original image, and back-propagates to update the network weights;

[0008] S3 expands the convolution layer with the largest contribution score in the iteration cycle into an enhanced edge-oriented reparameterized branch module according to the iteration cycle and the initialization conditions, and prunes all the reparameterized branches with the smallest contribution score in the reparameterized branch module that have been expanded in the iteration cycle;

[0009] Repeat the above steps S1-S3 until the backbone network converges, and re-parameterize the trained DyRepSR model to convert it into the original CNN-based lightweight image super-resolution backbone network.

[0010] Furthermore, small images of uniform size are randomly captured from the original low-resolution images used for training, and the images are preprocessed by random horizontal flipping, random vertical flipping, and random rotation of 90°. The same operation is performed on the corresponding high-resolution images to obtain the corresponding true value images.

[0011] Furthermore, the preprocessed image is input into a CNN-based lightweight image super-resolution backbone network for feature extraction to obtain deep features.

[0012] Furthermore, the deep features are input into the image reconstruction module to obtain a super-resolution image and complete the forward propagation.

[0013] Furthermore, the mean absolute error loss between the super-resolution image and the true value image is calculated, and the losses of the same batch of images are summed and averaged to obtain the final loss.

[0014] Furthermore, back propagation is performed based on the loss result obtained in step 4 to update the network weights.

[0015] Furthermore, the weights and gradients of the convolutional layer in the deep feature extraction subnet of the backbone network after this iteration are recorded, and their contribution scores to reducing the loss are calculated.

[0016] Furthermore, when the iteration cycle meets the requirements of dynamic reparameterization setting, the convolution layer with the largest contribution index in the cycle is expanded to be an enhanced edge-oriented reparameterized branch module. In addition, all branches with the smallest weight contribution index in the reparameterized branch module that have been expanded are pruned to obtain a rich form.

[0017] Furthermore, the above steps are repeated to make the loss approach zero until the network converges. After training all images each time, the network model obtained in the last iteration is used for verification, and the network model with the best verification effect is saved for testing.

[0018] Furthermore, the trained DyRepSR model is reparameterized and converted into the original CNN-based lightweight image super-resolution backbone network.

[0019] The second aspect of the present invention provides: a lightweight image super-resolution system based on dynamic re-parameterization: the system includes at least one processor; and a memory storing instructions, wherein the processor is configured to:

[0020] Inputting the low-resolution image to be super-resolved into the lightweight image super-resolution backbone network as described in the first aspect of the present invention;

[0021] According to the size multiple of the super-resolution, a super-resolution image with its length and width enlarged to a size multiple corresponding to the super-resolution image is output.

[0022] The beneficial effect of the present invention is that it avoids the problem of over-parameterization, thereby achieving the purpose of improving the reconstruction performance without changing the lightweight network parameters of the backbone network and maintaining a fast inference speed. Compared with other heavy parameterization methods, it can significantly reduce the training time and memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of an image super-resolution backbone network in an embodiment of the present invention;

[0024] Figure 2 Flow chart of dynamic re-parameter training according to an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of residual local feature block according to an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of an enhanced spatial attention module according to an embodiment of the present invention;

[0027] Figure 5 Schematic diagram of an enhanced edge-oriented re-parameterized branch module according to an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of dynamic expansion in dynamic reparameterization in an embodiment of the present invention;

[0029] Figure 7 Schematic diagram of dynamic pruning in dynamic reparameterization according to an embodiment of the present invention

[0030] Figure 8 A schematic diagram of a low-resolution image according to an embodiment of the present invention;

[0031] Fig. 9 Schematic diagram of super-resolution image according to an embodiment of the present invention;

[0032] Fig.10 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following examples further illustrate the content of the present invention, but should not be construed as limiting the present invention. Without departing from the spirit and substance of the present invention, modifications or substitutions made to the methods, steps or conditions of the present invention all fall within the scope of the present invention.

[0034] In some embodiments, Figure 1-7 As shown:

[0035] A lightweight image super-resolution method based on dynamic reparameterization, comprising:

[0036] S1 inputs the preprocessed low-resolution original image into the CNN-based lightweight image super-resolution backbone network and image reconstruction module to obtain a super-resolution image;

[0037] S2 calculates the loss function based on the super-resolution image and the true value image preprocessed by the corresponding high-resolution original image, and back-propagates to update the network weights;

[0038] S3 expands the convolution layer with the largest contribution score in the iteration cycle into an enhanced edge-oriented reparameterized branch module according to the iteration cycle and the initialization conditions, and prunes all the reparameterized branches with the smallest contribution score in the reparameterized branch module that have been expanded in the iteration cycle;

[0039] Repeat the above steps S1-S3 until the backbone network converges, and re-parameterize the trained DyRepSR model to convert it into the original CNN-based lightweight image super-resolution backbone network.

[0040] In these embodiments:

[0041] Randomly extract small images of uniform size from the original low-resolution images used for training, and preprocess the images by randomly flipping horizontally, randomly flipping vertically, and randomly rotating 90°. Perform the same operation on the corresponding high-resolution images to obtain the corresponding true value images.

[0042] The preprocessed image is input into the CNN-based lightweight image super-resolution backbone network for feature extraction to obtain deep features.

[0043] The deep features are input into the image reconstruction module to obtain the super-resolution image and complete the forward propagation.

[0044] Calculate the mean absolute error loss between the super-resolution image and the true value image, and average the losses of the same batch of images to get the final loss.

[0045] Back propagation is performed based on the obtained loss results to update the network weights.

[0046] Record the weights and gradients of the convolutional layer in the deep feature extraction subnet of the backbone network after this iteration, and calculate its contribution score to reducing the loss.

[0047] When the iteration cycle meets the requirements of dynamic reparameterization setting, the convolution layer with the largest contribution index in the cycle is expanded to an enhanced edge-oriented reparameterized branch module. In addition, all branches with the smallest weight contribution index in the reparameterized branch module that have been expanded are pruned to obtain a rich form.

[0048] Repeat the above steps to make the loss approach zero until the network converges. After training all the images each time, use the network model obtained in the last iteration for verification, and save the network model with the best verification effect for testing.

[0049] The trained DyRepSR model is reparameterized and converted into the original CNN-based lightweight image super-resolution backbone network.

[0050] In some embodiments, Figure 1 , Figure 5 , Figure 6 As shown:

[0051] The lightweight image super-resolution method based on dynamic reparameterization expands the convolutional layer with the largest contribution score in the iteration cycle e into an enhanced edge-oriented reparameterized branch module, including:

[0052] The convolutional layer is converted into a re-parameterized branch module (or also called a re-parameterized branch structure) with randomly initialized weights, and the outputs of all re-parameterized branches in the re-parameterized branch module and the output of the re-parameterized branch module as a whole are calculated;

[0053] The output of the expanded re-parameterized branch module is as follows:

[0054]

[0055] in:

[0056] F (1) =F (ori) -(F (2) +…+F (6) ) (2)

[0057] Where: F (ori) represents the convolutional layer weight, b (ori) Represents the convolution layer bias, I is the input feature map, O is the output feature map, Represents the convolution operation, and the expanded re-parameterized branch module contains {o (1) ,...,o (7)},o (1) is a 3×3 convolution branch, {o(2) ,...,o (6)} is a heavily parameterized branch with random weight initialization and bias initialization to zero, o (7) is a residual connection branch, in which the sum of the input feature maps is the residual operation, and this residual operation has no weight value; {F (1) ,...,F 6)} is the weight corresponding to each branch after expansion, where F (1) is the weight of the 3×3 convolution branch, {F (2) ,...,F (6)} is the weight of the heavy parameter branch with random weight initialization and bias initialization to zero.

[0058] In these embodiments:

[0059] During the expansion process, a single convolution is converted into a re-parameterized branch with randomly initialized weights, and the outputs of all branches and the output of the re-parameterized branch module as a whole are calculated. (ori) , bias is b (ori) The convolutional layer is expanded into a set of 7 branches {o (1) ,...,o (7)}, where o (1) is a 3×3 convolution branch, {o (2) ,...,o (6)} is a heavily parameterized branch with random weight initialization and bias initialization to zero, o (7) is the residual connection branch. The expanded heavy parameter structure output is as follows:

[0060]

[0061] in:

[0062] F (1) =F (ori) -(F (2) +…+F (6) ) (2)

[0063] I is the input feature map, O is the output feature map, represents the convolution operation, then F (1) Represents the weight of the 3×3 convolution branch in the expanded re-parameterized branch.

[0064] In some embodiments, Figure 1 and Figure 7 As shown:

[0065] According to the lightweight image super-resolution method based on dynamic reparameterization, all reparameterized branches with the smallest weight contribution scores in the reparameterized branch module that have been expanded are pruned, including:

[0066] The weights and biases of the pruned branches are reparameterized and added to a 3×3 convolution branch;

[0067] After pruning, the output of the re-parameterized branch module is as follows:

[0068]

[0069] in:

[0070] F (1_new) =F (1) +F (cut) (4)

[0071] b (1_new) =b (1) +b (cut) (5)

[0072] Where: I represents the input feature map, represents the convolution operation, O represents the output feature map, F (1_new) 、b (1_new) represents the weight and bias of the 3×3 convolution branch in the re-parameterized branch after pruning, F (1) 、b (1) represents the weight and bias of the 3×3 convolution branch in the re-parameterized branch before pruning, F (cut) 、b (cut) represents the weight and bias of the branch to be pruned in the re-parameterized branch before pruning, {F (2) ,...,F (6)}{b (2) ,...,b (6)} are the weights and biases of the heavily parameterized branches whose weights are randomly initialized and biases are initialized to zero before pruning.

[0073] In these embodiments:

[0074] When cut=true is satisfied, the re-parameterized branch is pruned. At the same time, the weights and biases of the pruned branch are re-parameterized and added to the 3×3 convolution branch to ensure that the overall weights and bias values ​​of the re-parameterized branch remain unchanged. (1) ,...,o (cut) ,...,o (7)}, where o (1) is a 3×3 convolution branch, o (cut) It is the branch that needs to be pruned. (7) is the residual connection branch. After pruning, the output of the re-parameterized branch module is as follows:

[0075]

[0076] in:

[0077] F (1_new) =F (1) +F (cut) (4)

[0078] b (1_new) =b (1) +b (cut) (5)

[0079] where F (1_new) 、b (1_new) represents the weight and bias of the 3×3 convolution branch in the re-parameterized branch after pruning, F (1) 、b (1) represents the weight and bias of the 3×3 convolution branch in the re-parameterized branch before pruning, F (cut) 、b (cut) represents the weight and bias of the branch to be pruned in the re-parameterized branch before pruning, {F (2) ,...,F (6)}{b (2) ,...,b (6)} represents the weight and bias of the heavily parameterized branch whose weights are randomly initialized and biases are initialized to zero before pruning.

[0080] In some embodiments, Figure 2 , Figure 6 , Figure 7 As shown:

[0081] A lightweight image super-resolution method based on dynamic re-parameterization, the method further comprising:

[0082] Initialization: Dynamic re-parameter adjustment minimum period E min , dynamic re-parameter adjustment minimum period E max , dynamic re-parameter update interval t, dynamic pruning flag cut, dynamic pruning contribution threshold λ;

[0083] When the iteration cycle meets E min <e<E max When e%t=0, dynamic re-parameter adjustment is performed, and at least one dynamic expansion is performed during the adjustment;

[0084] When the iteration cycle satisfies E min <e<E max And e%t=0, and cut=true and S o_min <λ, a dynamic pruning is performed.

[0085] In these embodiments:

[0086] For dynamic parameterization, several related parameters need to be initialized before network training, including the minimum period of dynamic re-parameter adjustment Emin, the minimum period of dynamic re-parameter adjustment Emax, the dynamic re-parameter update interval t, the dynamic pruning flag cut, and the dynamic pruning contribution threshold λ. min <e<E max When e%t=0, dynamic re-parameter adjustment is performed. During adjustment, dynamic expansion is performed. o_min <λ, dynamic pruning will be performed. During dynamic expansion, this method only considers 3×3 convolution operations with the same number of input and output channels in the deep feature extraction subnet, and expands the convolution operations that meet the conditions and have the highest contribution score in this cycle. During dynamic pruning, from all re-parameter branch modules that have undergone at least one network training, the branch with the smallest weight contribution index is selected for pruning. When pruning, the 3×3 convolution operations and residual connection branches in the original re-parameter branches are retained, and pruning is considered for other branches. After dynamic re-parameter adjustment, the adjusted network structure is used for subsequent training, and the expanded network structure is recorded.

[0087] In some embodiments, Figure 5 As shown:

[0088] The lightweight image super-resolution method based on dynamic reparameterization, the enhanced edge-oriented reparameterization branch module includes:

[0089] 3×3 convolution branch, depth-wise separable convolution branch, 1×1﹣3×3﹣1×1 convolution sequence branch, sobel-x edge extraction branch, sobel-y edge extraction branch, Laplacian edge extraction branch and residual connection branch; all 7 branches can be equivalently exchanged with the 3×3 convolution operation; the sum of the branch feature maps output by all branches is the output feature map of the heavy parameter branch module.

[0090] In these embodiments:

[0091] The enhanced edge-oriented reparameterized branch module designed by this method contains a total of 7 branches, namely 3×3 convolution branch, depth-separable convolution branch, 1×1-3×3-1×1 convolution sequence branch, sobel-x edge extraction branch, sobel-y edge extraction branch, Laplacian edge extraction branch and residual connection branch. The sum of the branch feature maps output by all branches is the output feature map of the reparameterized branch module. Among them, all 7 branches can be equivalently exchanged with 3×3 convolution operations. Introducing the depth-separable convolution branch in the reparameterized branch structure can allow the network to perform feature extraction similar to 3×3 convolution with less parameters and computational cost. The stacked small convolution operations in the 1×1-3×3-1×1 convolution sequence branch can help the network learn more complex information. In addition, the edge extraction branch is added to allow the network to extract edge and texture information more effectively.

[0092] In some embodiments, Figure 2 As shown:

[0093] Lightweight image super-resolution method based on dynamic re-parameterization, contribution scores include:

[0094] Sum the contribution scores of all parameters in the convolution layer to get the contribution score S of the entire convolution operation o ;

[0095] Sum the contribution scores of all reparameterized branch convolutional layers participating in back propagation on the same reparameterized branch as the overall contribution score S of the reparameterized branch. b ;

[0096]

[0097] Where: θ i represents the i-th parameter weight of the convolution operation, Represents the loss function of the neural network when the parameter is θ, θ i ∈θ,S p represents the contribution score of each parameter, ⊙ represents the Hadamard product operator, and || represents the absolute value.

[0098] In these embodiments:

[0099] After the iteration cycle is completed (an epoch ends), the weights and gradients of the convolutional layer in the deep feature extraction subnet of the backbone network are recorded, and their contribution score to reducing the loss is calculated.

[0100] This method uses a contribution scoring method that combines weight size and gradient information. It calculates the contribution of a certain operation in each iteration within a cycle and then takes the average as the contribution score of the operation in this cycle.(j) , whose contribution score is S o The specific calculation is as follows: the contribution score of the entire convolution operation is obtained by summing the contribution scores of all parameters in the convolution.

[0101]

[0102] Among them, θ i is the i-th parameter weight of the convolution operation, is the loss function of the neural network when the parameter is θ, θ i ∈θ,S p is the contribution score of each parameter, ⊙ is the Hadamard product operator, and || is the absolute value.

[0103] For the heavily parameterized branched structure, this method is based on S o The contribution score of all convolutional layers involved in back propagation on the same branch is summed up as the overall contribution score S of the branch. b .

[0104]

[0105] In some embodiments, Figure 1 , 3 , as shown in 4:

[0106] Based on the lightweight image super-resolution method of dynamic re-parameterization, the backbone network is configured as follows:

[0107] The preprocessed low-resolution original image is input into a 3×3 convolutional layer to extract shallow features and obtain a shallow feature map;

[0108] Input the shallow feature map into N residual local feature blocks and a 3×3 convolutional layer in sequence, and then add it to the shallow feature map to obtain a deep feature map of the same size as the shallow feature map;

[0109] The residual local feature block is configured as:

[0110] The shallow feature map is sequentially input into three stacked 3×3 convolutional layers and LeakyReLU activation function, and then added to the shallow feature map to obtain an intermediate feature map;

[0111] Input the intermediate feature map into a 1×1 convolutional layer of aggregated features to obtain an input feature map;

[0112] The input feature map is input into the enhanced spatial attention module to obtain a new feature map with the same size as the intermediate feature map.

[0113] In other embodiments, Figure 1 ,3 , as shown in 4:

[0114] The enhanced spatial attention module is configured as:

[0115] The input feature map is sequentially input into a 1×1 convolution layer, a convolution layer with a step size of 2, a maximum pooling module, a 3×3 convolution layer, an upsampling module for restoring the spatial size, and then combined with a residual branch, and input into a 1×1 convolution layer module to restore the number of channels;

[0116] Finally, a feature matrix is ​​generated through the Sigmoid function module, and the feature matrix is ​​multiplied by the input feature map to obtain a residual local feature map of the same size as the shallow feature map.

[0117] In these embodiments:

[0118] Step 2: Input the preprocessed image into the CNN-based lightweight image super-resolution backbone network for feature extraction to obtain deep features.

[0119] 1) First, a 3×3 convolutional layer is used to extract shallow features. An image of size (b×3×64×64) is input into the convolutional layer, and the output is a shallow feature map of size (b×c×64×64).

[0120] 2) The shallow feature map is input into the deep feature extraction subnet, and N residual local feature blocks and a 3×3 convolutional layer are used to extract deep features.

[0121] i. Input the feature map of size (b×c×64×64) into the residual local feature block. Specifically, first extract features through three stacked 3×3 convolutional layer modules, each followed by a LeakyReLU activation function. Add the feature map extracted by the last convolutional layer to the original input feature map to obtain the intermediate feature map (b×c×64×64).

[0122] ii. The intermediate feature map (b×c×64×64) is first passed through a 1×1 convolutional layer module and then input into the enhanced spatial attention module. Specifically, the number of channels is first reduced to 16 through a 1×1 convolutional layer module, and a convolutional layer module with a step size of 2 and a maximum pooling module are used to expand the receptive field and reduce the spatial dimension. Then a 3×3 convolutional layer is used to extract features. The spatial size is restored through the interpolation upsampling module, and the number of channels is restored to c through a 1×1 convolutional layer module combined with the residual branch connection. Finally, the feature matrix is ​​generated through the Sigmoid function module, and the feature matrix is ​​multiplied by the input feature to obtain a new feature map, which is still (b×c×64×64).

[0123] iii. Repeat steps i and ii N times. The residual local feature map obtained in the last step is extracted through a 3×3 convolutional layer to obtain a deep feature of size (b×c×64×64).

[0124] In some embodiments, Figure 1 As shown:

[0125] Based on the lightweight image super-resolution method of dynamic re-parameterization, the image reconstruction module is configured as follows:

[0126] The deep feature map is sequentially input into a 3×3 convolution layer and a sub-pixel convolution layer that upsamples the deep feature map to obtain a super-resolution image.

[0127] In these embodiments:

[0128] Step 3: Input the deep features into the image reconstruction module to obtain the super-resolution image and complete the forward propagation.

[0129] The feature map of size (b×c×64×64) is input into the 3×3 convolutional layer to adjust the channel. At this time, the output feature map size is (b×3*s*s×64×64).

[0130] Use the sub-pixel convolution layer to upsample the feature map of size (b×3*s*s×64×64) to (b×3×64s×64s)

[0131] Further: Step 4: Calculate each super-resolution image I SR and the true value image I HR The mean absolute error loss

[0132] Step 5: Perform back propagation based on the loss result obtained in step 4 and update the network weights.

[0133] In some embodiments, Figure 1 As shown:

[0134] A lightweight image super-resolution method based on dynamic reparameterization, which makes the loss approach zero until the network converges, also includes:

[0135] After training all images each time, the network model obtained in the last iteration is used for verification, and the network model with the best verification effect is saved for testing;

[0136] The trained DyRepSR model is reparameterized and converted into the original CNN-based lightweight image super-resolution backbone network.

[0137] In these embodiments:

[0138] Step 8: Repeat the above steps to make the loss approach zero until the network converges. After training all the images each time, use the network model obtained in the last iteration for verification, and save the network model with the best verification effect for testing.

[0139] Step 9: Reparameterize the trained DyRepSR model and convert it into the original CNN-based lightweight image super-resolution backbone network.

[0140] In some embodiments, Figure 1 , Figure 8 , Fig. 9 As shown:

[0141] Application of a lightweight image super-resolution method based on dynamic reparameterization:

[0142] Inputting the low-resolution image to be super-resolved into the lightweight image super-resolution backbone network as described in the above embodiment;

[0143] According to the size multiple of the super-resolution, a super-resolution image with its length and width enlarged to a size multiple corresponding to the super-resolution image is output.

[0144] In these embodiments:

[0145] Step 10: Input the low-resolution image to be super-resolved (c×h×w) into the above-mentioned lightweight image super-resolution backbone network, and select the super-resolution size multiple of ×2, ×3 or ×4.

[0146] Step 11: The obtained super-resolution image is enlarged to a corresponding multiple (c×hs×ws).

[0147] In some embodiments, in combination with the above embodiments, Figures 1 to 9 As shown:

[0148] 1. The training process of the DyRepSR model (scaling factor s=4) includes the following steps:

[0149] Step 1: Randomly extract images of size 64×64 from the original low-resolution images used for training, and preprocess the images by randomly flipping horizontally, randomly flipping vertically, and randomly rotating 90° according to probability p=0.5. For the same batch of 64 images, perform the same operations on the corresponding high-resolution images to obtain the corresponding high-resolution true value images.

[0150] Step 2: Input the preprocessed image into the CNN-based lightweight image super-resolution backbone network for feature extraction to obtain deep features.

[0151] 1) First, a 3×3 convolutional layer is used to extract shallow features. An image of size (64×3×64×64) is input into the convolutional layer, and the output is a shallow feature map of size (64×52×64×64).

[0152] 2) The shallow feature map is input into the deep feature extraction subnet, and 6 residual local feature blocks and a 3×3 convolutional layer are used to extract deep features.

[0153] i. Input the feature map of size (64×52×64×64) into the residual local feature block. Specifically, first extract features through three stacked 3×3 convolutional layer modules, each followed by a LeakyReLU activation function. Add the feature map extracted by the last convolutional layer to the original input feature map to obtain the intermediate feature map (64×52×64×64).

[0154] ii. The intermediate feature map (64×52×64×64) is first passed through a 1×1 convolutional layer module and then input into the enhanced spatial attention module. Specifically, the number of channels is first reduced to 16 through a 1×1 convolutional layer module, and a convolutional layer module with a step size of 2 and a maximum pooling module are used to expand the receptive field and reduce the spatial dimension. Then a 3×3 convolutional layer is used to extract features. The spatial size is restored through an interpolation upsampling module, and the number of channels is restored to 52 through a 1×1 convolutional layer module combined with residual branch connection. Finally, a feature matrix is ​​generated through a Sigmoid function module, and the feature matrix is ​​multiplied by the input feature to obtain a new feature map, which is still (64×52×64×64).

[0155] iii. Repeat steps i and ii for a total of 6 times. The residual local feature map obtained in the last step is extracted through a 3×3 convolutional layer to obtain a deep feature of size (64×52×64×64).

[0156] Step 3: Input the deep features into the image reconstruction module to obtain the super-resolution image and complete the forward propagation.

[0157] The feature map of size (64×52×64×64) is input into the 3×3 convolutional layer to adjust the channel. At this time, the output feature map size is (64×48×64×64).

[0158] The sub-pixel convolution layer is used to upsample the feature map of size (64×48×64×64) to (64×3×256×256)

[0159] Step 4: Calculate each super-resolution image I SR and the true value image I HR The mean absolute error loss

[0160] Step 5: Perform backpropagation based on the loss result obtained in Step 4 to update the network weights.

[0161] Step 6: After this iteration cycle (referring to the end of one epoch), record the weights and gradients of the convolutional layers in the deep feature extraction subnet of the backbone network, and calculate its contribution score to reducing the loss.

[0162] Step 7: Initialize relevant parameters, including the minimum cycle for dynamic reparameterization adjustment Emin = 40, the maximum cycle for dynamic reparameterization adjustment Emax = 140, the dynamic reparameterization update interval t = 10, the flag for dynamic pruning or not cut = true, and the dynamic pruning contribution threshold λ = 1×10-5.

[0163] When the iteration cycle e satisfies 40 < e < 140 and e is a multiple of 10, expand the 3×3 convolutional layer with the largest contribution index in this cycle into an enhanced edge-oriented reparameterization branch module. Additionally, prune the branch with the smallest weight contribution index among all the branches that have been expanded into reparameterization branch modules.

[0164] Step 8: Repeat the above steps to make the loss approach zero until the network converges. After each training of all the images (1 epoch), use the network model obtained in the last iteration for verification, and save the network model with the best verification effect for testing. The learning rate is initialized to 5×10-4, halved for the first time at the 250th cycle, and then halved every 200 epochs, and fixed after 1050 epochs.

[0165] Step 9: Reparameterize the trained DyRepSR model and convert it into the original lightweight image super-resolution backbone network based on CNN.

[0166] The process of using the DyRepSR model for image super-resolution specifically includes the following steps:

[0167] Step 10: Input the low-resolution image to be super-resolved (3×256×170) into the above lightweight image super-resolution backbone network, and select the super-resolution size multiple ×4.

[0168] Step 11: Obtain the super-resolution image with the length and width enlarged to the corresponding multiple (3×1024×680).

[0169] The technical effects of the present invention are multi-faceted.

[0170] This method innovatively adopts the idea of ​​dynamic reparameterization to train lightweight image super-resolution networks, so that the network focuses more on operations that contribute more to reducing losses. During the training process, this method only expands the convolutional layers with large contributions into reparameterized branches, and prunes the branches with small contributions in the reparameterized branch structure, so as to smoothly evolve a network suitable for super-resolution tasks. An enhanced edge-oriented reparameterized branch module is proposed, which combines convolution sequence branches, depthwise separable convolution branches and enhanced edge operator branches, which can improve the representation ability of reparameterized branches while retaining the edge details of features. The dynamic reparameterization method and enhanced edge-oriented reparameterized branch module proposed in this method can be used to improve any CNN-based reparameterized image super-resolution network. Compared with the current over-constructed reparameterized lightweight image super-resolution network, this method can significantly reduce the training time and memory consumption during training while maintaining similar reconstruction quality.

[0171] The processes and logic flows described in this specification may be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus may also be implemented as special purpose logic circuits.

[0172] Although the present invention has been described in detail above by general description, specific implementation methods and experiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A lightweight image super-resolution method based on dynamic reparameterization, characterized in that: include: S1 inputs the preprocessed low-resolution original image into the CNN-based lightweight image super-resolution backbone network and image reconstruction module to obtain a super-resolution image; S2 calculates the loss function based on the super-resolution image and the true value image preprocessed by the corresponding high-resolution original image, and back-propagates to update the network weights; S3, according to the iteration cycle e and the initialization conditions, expands the convolution layer with the largest contribution score in the iteration cycle e into an enhanced edge-oriented reparameterized branch module, and prunes all the reparameterized branches with the smallest contribution score in the reparameterized branch module that have been expanded in the iteration cycle e; Repeat the above steps S1-S3 until the backbone network converges; The convolutional layer with the largest contribution score in the iteration cycle e is expanded into an enhanced edge-oriented re-parameterized branch module, including: Converting the convolutional layer into a re-parameterized branch module with randomly initialized weights, and calculating the outputs of all re-parameterized branches in the re-parameterized branch module and the output of the re-parameterized branch module as a whole; The expanded re-parameterized branch structure output is as follows: in: F (1) =F (ori) -(F (2) +…+F (6) ) (2) Where: F (ori) represents the convolutional layer weight, b (ori) Represents the convolution layer bias, I is the input feature map, O is the output feature map, Represents the convolution operation, and the expanded re-parameterized branch module contains {o (1) ,...,o (7) }, where o (1) is a 3×3 convolution branch, {o (2) ,...,o (6) } is a heavily parameterized branch with random weight initialization and bias initialization to zero, o (7) is the residual connection branch, corresponding to {F (1) ,...,F (6) } is the weight corresponding to each branch after expansion, where F (1) is the weight of the 3×3 convolution branch, {F (2) ,...,F (6) } is the weight of the heavy parameter branch with random weight initialization and bias initialization to zero; Prune all reparameterized branches that have been expanded to have the smallest weight contribution score in the reparameterized branch module, including: The weights and biases of the pruned branches are reparameterized and added to a 3×3 convolution branch; After pruning, the output of the re-parameterized branch structure is as follows: in: F (1_new) =F (1) +F (cut) (4) b (1 _ new) =b (1) +b (cut) (5) Where: I represents the input feature map, represents the convolution operation, O represents the output feature map, F (1_new) 、b (1_new) represents the weight and bias of the 3×3 convolution branch in the re-parameterized branch after pruning, F (1) 、b (1) represents the weight and bias of the 3×3 convolution branch in the re-parameterized branch before pruning, {F (2) ,...,F (6) }{b (2) ,...,b (6) } represents the weight and bias of the heavily parameterized branch with random weight initialization and bias initialization to zero before pruning, F (cut) 、b (cut) Represents the weight and bias of the branch to be pruned in the re-parameterized branch before pruning The method according to the iteration period e and the initialization condition includes: When the iteration cycle meets E min <e<E max When e%t=0, the convolution layer is dynamically re-parameterized, and at least one dynamic expansion is performed during the adjustment; When the iteration cycle meets E min <e<E max And e%t=0, and cut=true and S o_min <λ, the branch module that has been expanded to be re-parameterized is dynamically pruned; Where: E min Indicates the minimum period of dynamic re-parameter adjustment, E max It represents the maximum period of dynamic re-parameter adjustment, t represents the dynamic re-parameter update interval, cut represents the dynamic pruning flag, and λ represents the dynamic pruning contribution threshold.

2. The lightweight image super-resolution method based on dynamic reparameterization according to claim 1, characterized in that: The enhanced edge-oriented re-parameterized branch module includes: 3×3 convolution branch, depth-wise separable convolution branch, 1×1﹣3×3﹣1×1 convolution sequence branch, sobel-x edge extraction branch, sobel-y edge extraction branch, Laplacian edge extraction branch and residual connection branch; all 7 branches can be equivalently exchanged with the 3×3 convolution operation; the sum of the branch feature maps output by all branches is the output feature map of the heavy parameter branch module.

3. The lightweight image super-resolution method based on dynamic reparameterization according to claim 1, characterized in that: Contribution scores include: Sum the contribution scores of all parameters in the convolution layer to get the contribution score S of the entire convolution operation o ; Sum the contribution scores of all reparameterized branch convolutional layers participating in back propagation on the same reparameterized branch as the overall contribution score S of the reparameterized branch. b ; Where: θ i represents the i-th parameter weight of the convolution operation, Represents the loss function of the neural network when the parameter is θ, θ i ∈θ,S p represents the contribution score of each parameter, ⊙ represents the Hadamard product operator, and || represents the absolute value.

4. The lightweight image super-resolution method based on dynamic reparameterization according to claim 1, characterized in that: The backbone network is configured as follows: The preprocessed low-resolution original image is input into a 3×3 convolutional layer to extract shallow features and obtain a shallow feature map; Input the shallow feature map into N residual local feature blocks and a 3×3 convolutional layer in sequence, and then add it to the shallow feature map to obtain a deep feature map of the same size as the shallow feature map; The residual local feature block is configured as: The shallow feature map is sequentially input into three stacked 3×3 convolutional layers and LeakyReLU activation function, and then added to the shallow feature map to obtain an intermediate feature map; Input the intermediate feature map into a 1×1 convolutional layer of aggregated features to obtain an input feature map; The input feature map is input into the enhanced spatial attention module to obtain a new feature map with the same size as the intermediate feature map.

5. The lightweight image super-resolution method based on dynamic reparameterization according to claim 4, characterized in that: The image reconstruction module is configured as follows: The deep feature map is sequentially input into a 3×3 convolution layer and a sub-pixel convolution layer to obtain a super-resolution image.

6. The lightweight image super-resolution method based on dynamic reparameterization according to claim 4, characterized in that: The enhanced spatial attention module is configured as follows: The input feature map is sequentially input into a 1×1 convolution layer, a convolution layer with a step size of 2, a maximum pooling module, a 3×3 convolution layer, an upsampling module for restoring the spatial size, and then combined with a residual branch, and input into a 1×1 convolution layer module to restore the number of channels; Finally, a feature matrix is ​​generated through the Sigmoid function module, and the feature matrix is ​​multiplied by the input feature map to obtain a residual local feature map of the same size as the shallow feature map.

7. A lightweight image super-resolution system based on dynamic reparameterization, characterized in that The system comprises at least one processor; and a memory storing instructions, wherein the processor is configured to: Inputting the low-resolution image to be super-resolved into the lightweight image super-resolution backbone network as described in claims 1-6; According to the size multiple of the super-resolution, a super-resolution image with its length and width enlarged to a size multiple corresponding to the super-resolution image is output.

Images