Lightweight super-resolution network method based on path dynamic selection

Abstract

The invention discloses a lightweight super-resolution network method based on path dynamic selection, and belongs to the technical field of image processing and deep learning. According to the method, a strategy network, reinforcement learning and a super-resolution network are combined, and challenges faced by edge equipment with limited computing resources during high-precision image super-resolution processing are effectively overcome. According to the method, features are effectively extracted from a low-resolution image through convolutional layers of a strategy network and a super-division network, the strategy network further generates an optimized path according to the features, the super-division network is guided to accurately process the image according to the path, and a high-resolution image is generated. The core innovation of the method is that the super-division network and a path dynamic selection mechanism are combined, the processing speed is remarkably improved on the premise that the image quality is not sacrificed, and efficient image processing in a calculation-limited environment is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and deep learning, and specifically relates to a lightweight super-resolution network method based on dynamic path selection. Background Art

[0002] Super-resolution technology, especially in the field of image processing, has always been a hot topic of research. Its goal is to reconstruct a high-resolution image from a low-resolution image to improve the clarity and details of the image. In the past few decades, this field has experienced a transition from traditional algorithms to deep learning methods. Early super-resolution techniques mainly relied on interpolation algorithms such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. These methods are simple and effective, but they often fail to achieve the desired clarity and detail restoration effects when processing complex or low-quality images.

[0003] With the rise of deep learning, super-resolution methods based on convolutional neural networks have become popular. These methods can automatically extract important features and achieve more accurate image reconstruction by learning a large amount of image data. However, these deep learning-based methods must use a lot of computing resources, and the model structure is complex, which limits their use in mobile devices or real-time processing applications. In addition, the existing deep learning super-resolution methods use a fixed network structure, which means that the same processing flow is used for all images, ignoring the characteristic differences between different images. This "one-size-fits-all" approach often fails to achieve the best results when processing diverse image data. Therefore, how to design an efficient super-resolution method that can adapt to different image features and run under limited computing resources has become an important research topic in this field. The ideal solution can dynamically adjust the network structure according to the characteristics of the image and the available computing resources to achieve the best super-resolution effect. Summary of the invention

[0004] In order to solve the problems of high computing requirements and insufficient flexibility in implementing image super-resolution processing in resource-constrained environments, we proposed a lightweight super-resolution network method based on dynamic path selection. This method not only significantly reduces the computational burden by combining reinforcement learning and super-resolution network design, but also can customize processing according to the specific characteristics of different images by dynamically adjusting the processing strategy. This allows the method to significantly accelerate the inference speed of the super-resolution network model while maintaining high image quality, and is particularly suitable for mobile devices with limited computing resources and real-time processing applications. The technical problem proposed by the present invention is solved as follows:

[0005] Specific content:

[0006] A lightweight super-resolution network method based on dynamic path selection includes the following steps:

[0007] Step 1: Collect paired low-resolution and high-resolution image datasets, where each pair represents two resolutions of the same scene, perform preliminary processing on these images, and divide them into training and test sets;

[0008] Step 2, preprocessing the obtained low-resolution image and performing normalization processing;

[0009] Step 3: Input the original low-resolution data into the backbone network (i.e., EDSR super-resolution network) and the strategy network respectively. The strategy network generates a strategy path based on the extracted features. The backbone network processes the image based on this path. The backbone network performs super-resolution on the low-resolution image based on the convolution kernel size and the strategy path.

[0010] Step 4: Integrate the super-resolution results and the selected path in step 3, and feed the calculated reward value back to the strategy network to optimize the super-resolution strategy through reinforcement learning method;

[0011] Step 5, train and improve the lightweight super-resolution network constructed in steps 3 and 4 on paired low-resolution and high-resolution image datasets;

[0012] Step 1 of the present invention comprises:

[0013] Step 1-1, obtain paired low-resolution and high-resolution datasets. First, perform a preliminary screening of the dataset, remove irrelevant images, and unify the image sizes to ensure data consistency;

[0014] Step 1-2: The preliminarily processed data set is further divided into a training set and a test set to provide data support for subsequent network training and performance evaluation;

[0015] Step 2 of the present invention comprises:

[0016] Step 2-1, use imageio and opencv libraries to read the image, load the image into memory according to the specified file name and path, and the preset image size parameters, and store it in array format;

[0017] Step 2-2, batch processing the image data loaded in step 2-1, first use the shuffle function to randomly shuffle the data index to enhance the randomness of the data, and then read the image data in order of these random indexes;

[0018] Step 2-3, data preprocessing in the backbone network. This mainly includes normalizing the image data read in step 2-2 to make it conform to the standard format of network input;

[0019] Step 3 of the present invention comprises:

[0020] Step 3-1, use the following formula to initialize the policy network. The specific process is as follows:

[0021] m=f p (x, w)

[0022] Among them, x represents the input low-resolution image, w represents the weight of the policy network randomly initialized, and the policy network f p It includes three ResNet blocks (forming a structure similar to ResNet-8), each of which consists of a 3x3 convolutional layer, a normalization layer, and a ReLU activation layer. The output m is the policy path generated by the policy network based on the input image.

[0023] Step 3-2: For the policy path generated in step 3-1, its specific form is as follows:

[0024] m=(m1,m2,…m k )

[0025] Among them, m is a k-dimensional (i.e. 32-dimensional) vector, m1, m2, …m k The value of each element is between 0 and 1, and each element m k When it is greater than 0.5, the corresponding ResNet block is executed in the backbone network, and when it is less than 0.5, the block is not executed;

[0026] Step 3-3, use the probability model to quantify the selection probability of each potential operation of the policy path defined in step 3-2, the specific form of which is as follows:

[0027]

[0028] Among them, x, w, k and m k Same as steps 3-1 and 3-2, π ω represents the probability distribution of selecting action u under the condition of input x. Here, u k =1 indicates the execution of the corresponding ResNet block, u k =0 means no execution;

[0029] Step 4 of the present invention comprises:

[0030] Step 4-1, in order to find the optimal strategy path, that is, to minimize the use of ResNet blocks while maximizing the image quality, the present invention maximizes the reward function through a reinforcement learning method, realizes the joint optimization of the strategy network and the backbone network, thereby improving the lightweight effect of the super-resolution network;

[0031] Step 4-2, using the policy path generated by the policy network and the performance indicators of the backbone network to perform reinforcement learning optimization, the specific form is as follows:

[0032]

[0033] Where u and k are the same as step 3-3, p represents the PSNR value obtained after the backbone network executes the policy path, t represents the set performance benchmark threshold (set to 10), γ represents the penalty factor imposed when the actual performance p does not reach the threshold t (set to -100), R (u，p) represents the reward value calculated based on strategy u and performance p;

[0034] Step 4-3, use the policy gradient method to update the policy network parameters, the specific form is as follows:

[0035]

[0036] Among them, x, u, k, p, m k Same as step 3-1, step 3-2 and step 4-2, represents the gradient of the objective function J with respect to the policy network parameter θ, E represents the expectation operator, which is used to calculate the expectation of the random variable, logπ θ (u|x) represents the logarithm of the probability of generating a policy u based on the policy network parameters θ when given an input x. This is a key part of the policy gradient algorithm, which connects the probability distribution of the policy and the gradient of the parameters.

[0037] Step 5 of the present invention comprises:

[0038] Step 5-1, jointly train the policy network and the backbone network, the parameters of the backbone network are updated by the gradient descent method, and the parameters of the policy network are updated by the policy gradient method described in step 4-3. After the parameter optimization is completed, the backbone network will execute the policy path generated by the policy network to effectively reconstruct the low-resolution image into a high-resolution image;

[0039] The present invention proposes a lightweight super-resolution network method based on dynamic path selection, which has the following advantages and beneficial effects compared with the prior art:

[0040] The present invention combines policy networks, reinforcement learning and super-resolution network models, significantly improving the efficiency and quality of super-resolution processing of low-resolution images. Compared with the prior art, the lightweight method of the present invention utilizes the exploration ability of reinforcement learning and the path dynamic selection mechanism of the policy network to select the most appropriate processing path according to the specific characteristics of the low-resolution image, thereby accelerating the inference speed while maintaining excellent image quality. This technology is particularly effective in resource-constrained environments and can achieve personalized processing for images of different difficulty levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An overall flow chart of a lightweight super-resolution network for dynamic path selection according to an embodiment of the present invention;

[0042] Figure 2 A flow chart of a generation strategy according to an embodiment of the present invention;

[0043] Figure 3 The overall network structure diagram of the backbone network and the policy network of the embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a low-resolution input image according to an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a high-resolution output image corresponding to a low-resolution input image according to an embodiment of the present invention; DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0047] The present invention is implemented including:

[0048] The overall process of the method proposed by the present invention is as follows Figure 1 As shown, the specific steps of constructing the lightweight super-resolution network with dynamic path selection in the embodiment of the present invention are as follows:

[0049] Step 1: Collect paired low-resolution and high-resolution image datasets, where each pair represents two resolutions of the same scene, perform preliminary processing on these images, and divide them into training and test sets;

[0050] Step 2, preprocessing the obtained low-resolution image and performing normalization processing;

[0051] Step 3: Input the original low-resolution data into the backbone network (i.e., EDSR super-resolution network) and the strategy network respectively. The strategy network generates a strategy path based on the extracted features. The backbone network processes the image based on this path. The backbone network performs super-resolution on the low-resolution image based on the convolution kernel size and the strategy path.

[0052] Step 4: Integrate the super-resolution results and the selected path in step 3, and feed the calculated reward value back to the strategy network to optimize the super-resolution strategy through reinforcement learning method;

[0053] Step 5: Train and improve the lightweight super-resolution network constructed in steps 3 and 4 on paired low-resolution and high-resolution image datasets.

[0054] Step 1 of the present invention comprises:

[0055] Step 1-1, obtain paired low-resolution and high-resolution datasets. First, perform a preliminary screening of the dataset, remove irrelevant images, and unify the image sizes to ensure data consistency;

[0056] In step 1-2, the preliminarily processed data set is further divided into a training set and a test set to provide data support for subsequent network training and performance evaluation.

[0057] Step 2 of the present invention comprises:

[0058] Step 2-1, use imageio and opencv libraries to read the image, load the image into memory according to the specified file name and path, and the preset image size parameters, and store it in array format;

[0059] Step 2-2, batch processing the image data loaded in step 2-1, first use the shuffle function to randomly shuffle the data index to enhance the randomness of the data, and then read the image data in order of these random indexes;

[0060] Step 2-3, data preprocessing in the backbone network, mainly includes normalizing the image data read in step 2-2 to make it conform to the standard format of network input.

[0061] Step 3 of the present invention comprises:

[0062] Step 3-1, use the following formula to initialize the policy network. The specific process is as follows:

[0063] m=f p (x, w)

[0064] Among them, x represents the input low-resolution image, w represents the weight of the policy network randomly initialized, and the policy network f p It includes three ResNet blocks (forming a structure similar to ResNet-8), each ResNet block consists of a 3x3 convolution layer, a normalization layer and a ReLU activation layer, and the output m is the policy path generated by the policy network based on the input image.

[0065] Step 3-2: For the policy path generated in step 3-1, its specific form is as follows:

[0066] m=(m1,m2,…m k )

[0067] Among them, m is a k-dimensional (i.e. 32-dimensional) vector, m1, m2, …m kThe value of each element is between 0 and 1, and each element m k When it is greater than 0.5, the corresponding ResNet block is executed in the backbone network, and when it is less than 0.5, the block is not executed.

[0068] Step 3-3, use the probability model to quantify the selection probability of each potential operation of the policy path defined in step 3-2, the specific form of which is as follows:

[0069]

[0070] Among them, x, w, k and m k Same as steps 3-1 and 3-2, π ω represents the probability distribution of selecting action u under the condition of input x. Here, u k =1 indicates the execution of the corresponding ResNet block, u k =0 means not executed.

[0071] Step 4 of the present invention comprises:

[0072] Step 4-1, in order to find the optimal strategy path, that is, to minimize the use of ResNet blocks while maximizing the image quality, the present invention maximizes the reward function through a reinforcement learning method, realizes the joint optimization of the strategy network and the backbone network, thereby improving the lightweight effect of the super-resolution network.

[0073] Step 4-2, use the policy path generated by the policy network and the performance indicators of the backbone network to perform reinforcement learning optimization. The specific form is as follows:

[0074]

[0075] Where u and k are the same as step 3-3, p represents the PSNR value obtained after the backbone network executes the policy path, t represents the set performance benchmark threshold (set to 10), γ represents the penalty factor imposed when the actual performance p does not reach the threshold t (set to -100), R (u，p) Represents the reward value calculated based on strategy u and performance p.

[0076] Step 4-3, use the policy gradient method to update the policy network parameters, the specific form is as follows:

[0077]

[0078] Among them, x, u, k, p, m k Same as step 3-1, step 3-2 and step 4-2, represents the gradient of the objective function J with respect to the policy network parameter θ, E represents the expectation operator, which is used to calculate the expectation of the random variable, logπθ (u|x) represents the logarithm of the probability of the policy u generated according to the policy network parameters θ when the input x is given. This is a key part of the policy gradient algorithm, which connects the probability distribution of the policy and the gradient of the parameters.

[0079] Step 5 of the present invention comprises:

[0080] Step 5-1, jointly train the policy network and the backbone network, the parameters of the backbone network are updated by the gradient descent method, and the parameters of the policy network are updated by the policy gradient method described in step 4-3. After the parameter optimization is completed, the backbone network will execute the policy path generated by the policy network to effectively reconstruct the low-resolution image into a high-resolution image.

[0081] The present invention combines policy networks, reinforcement learning and super-resolution network models, significantly improving the efficiency and quality of super-resolution processing of low-resolution images. Compared with the prior art, the lightweight method of the present invention utilizes the exploration ability of reinforcement learning and the path dynamic selection mechanism of the policy network to select the most appropriate processing path according to the specific characteristics of the low-resolution image, thereby accelerating the inference speed while maintaining excellent image quality. This technology is particularly effective in resource-constrained environments and can achieve personalized processing for images of different difficulty levels.

[0082] The present invention provides a lightweight super-resolution network method based on dynamic path selection. There are many methods and ways to implement the technical solution. The above is the preferred implementation scheme of the present invention. For ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and polishing must be made, and these improvements and polishing should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A lightweight super-resolution network method based on dynamic path selection, characterized in that: The following steps are involved: Step 1: Collect paired low-resolution and high-resolution image datasets, where each pair represents two resolutions of the same scene, perform preliminary processing on these images, and divide them into training and test sets; Step 2, preprocessing the obtained low-resolution image and performing normalization processing; Step 3: Input the original low-resolution data into the backbone network and the policy network respectively. The policy network generates a policy path based on the extracted features. The backbone network processes the image based on this path. The backbone network performs super-resolution on the low-resolution image based on the convolution kernel size and the policy path. Step 4: Integrate the super-resolution results and the selected path in step 3, and feed the calculated reward value back to the strategy network to optimize the super-resolution strategy through reinforcement learning method; Step 5: Train and improve the lightweight super-resolution network constructed in steps 3 and 4 on paired low-resolution and high-resolution image datasets.

2. The method according to claim 1, characterized in that Step 2 includes: Step 2-1, use imageio and opencv libraries to read the image, load the image into memory according to the specified file name and path, and the preset image size parameters, and store it in array format; Step 2-2, batch processing the image data loaded in step 2-1, first use the shuffle function to randomly shuffle the data index to enhance the randomness of the data, and then read the image data in order of these random indexes; Step 2-3, data preprocessing in the backbone network, mainly includes normalizing the image data read in step 2-2 to make it conform to the standard format of network input.

3. The method according to claim 1, characterized in that Step 3 includes: Step 3-1, use the following formula to initialize the policy network. The specific process is as follows: m=f p (x,w)。 4. Among them, x represents the input low-resolution image, w represents the randomly initialized weight of the policy network, and f p represents the policy network, and m represents the policy path generated by the policy network according to the input image; Step 3-2: For the policy path generated in step 3-1, its specific form is as follows: m=(m1,m2,…m k )。 5. Among them, m is a k-dimensional vector, m1, m2, …m k The value of each element is between 0 and 1, and each element m k When it is greater than 0.5, the corresponding ResNet block is executed in the backbone network, and when it is less than 0.5, the block is not executed; Step 3-3, use the probability model to quantify the selection probability of each potential operation of the policy path defined in step 3-2, the specific form of which is as follows:

6. Among them, x, w, k, and m k Same as steps 3-1 and 3-2, π ω represents the probability distribution of selecting action u under the condition of input x. Here, u k =1 indicates the execution of the corresponding ResNet block, u k =0 means not executed.

7. The method according to claim 1, characterized in that Step 4 includes: Step 4-1, in order to find the optimal strategy path, that is, to minimize the use of ResNet blocks while maximizing the image quality, the present invention maximizes the reward function through a reinforcement learning method, realizes the joint optimization of the strategy network and the backbone network, thereby improving the lightweight effect of the super-resolution network; Step 4-2, using the policy path generated by the policy network and the performance indicators of the backbone network to perform reinforcement learning optimization, the specific form is as follows:

8. Where u and k are the same as in step 3-3, p represents the PSNR value obtained after the backbone network executes the policy path, t represents the set performance benchmark threshold, γ represents the penalty factor imposed when the actual performance p does not reach the threshold t, and R (u，p) represents the reward value calculated based on strategy u and performance p; Step 4-3, use the policy gradient method to update the policy network parameters, the specific form is as follows:

9. Among them, x,u,k,p,m k Same as step 3-1, step 3-2 and step 4-2, represents the gradient of the objective function J with respect to the policy network parameter θ, E represents the expectation operator, which is used to calculate the expectation of the random variable, log π θ (u|x) represents the logarithm of the probability of the policy u generated according to the policy network parameters θ when the input x is given. This is a key part of the policy gradient algorithm, which connects the probability distribution of the policy and the gradient of the parameters.

10. The method according to claim 1, characterized in that Step 5 includes: Step 5-1, jointly train the policy network and the backbone network, the parameters of the backbone network are updated by the gradient descent method, and the parameters of the policy network are updated by the policy gradient method described in step 4-3. After the parameter optimization is completed, the backbone network will execute the policy path generated by the policy network to effectively reconstruct the low-resolution image into a high-resolution image.

Images