Image super-division and model training method and device, equipment, medium and product

Abstract

The invention provides an image super-division and model training method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning and the like. The image super-division method comprises the following steps: carrying out feature extraction on an original image with a first resolution to obtain an initial feature map; processing the initial feature map based on a basic block to obtain an intermediate feature map; obtaining a target feature map based on the initial feature map and the intermediate feature map; obtaining a target image of a second resolution based on the target feature map; the second resolution is higher than the first resolution; wherein the basic block is a residual network, and a residual path of the residual network comprises a convolutional layer and an attention layer.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, deep learning and other technical fields, and in particular to an image super-resolution and model training method, device, equipment, medium and product. Background Art

[0002] Image super-resolution is an image processing technique that aims to reconstruct a high-resolution image from a low-resolution image.

[0003] How to achieve image super-resolution efficiently and lightweight is a problem that needs to be solved. Summary of the invention

[0004] The present invention provides an image super-resolution method, apparatus, device, medium and product.

[0005] According to one aspect of the present disclosure, an image super-resolution method is provided, comprising: performing feature extraction on an original image of a first resolution to obtain an initial feature map; based on a basic block, processing the initial feature map to obtain an intermediate feature map; based on the initial feature map and the intermediate feature map, obtaining a target feature map; based on the target feature map, obtaining a target image of a second resolution; the second resolution is higher than the first resolution; wherein the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0006] According to another aspect of the present disclosure, a method for training an image super-resolution model is provided, wherein the image super-resolution model includes a basic block, and the method includes: performing feature extraction on an image sample of a first resolution to obtain an initial feature map; based on the basic block, processing the initial feature map to obtain an intermediate feature map; based on the initial feature map and the intermediate feature map, obtaining a target feature map; based on the target feature map, obtaining a predicted image of a second resolution; the second resolution is higher than the first resolution; constructing a loss function based on the predicted image and the real image corresponding to the image sample; based on the loss function, adjusting the initial model parameters of the current training stage to obtain the target model parameters of the current training stage; wherein the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0007] According to another aspect of the present disclosure, an image super-resolution device is provided, comprising: an extraction module for performing feature extraction on an original image of a first resolution to obtain an initial feature map; a processing module for processing the initial feature map according to a basic block to obtain an intermediate feature map; an acquisition module for acquiring a target feature map based on the initial feature map and the intermediate feature map; a prediction module for acquiring a target image of a second resolution based on the target feature map; the second resolution is higher than the first resolution; wherein the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0008] According to another aspect of the present disclosure, there is provided an image super-resolution model training device, wherein the image super-resolution model comprises a basic block, and the device comprises: an extraction module, configured to perform feature extraction on an image sample of a first resolution to obtain an initial feature map; a processing module, configured to process the initial feature map according to the basic block to obtain an intermediate feature map; an acquisition module, configured to obtain a target feature map according to the initial feature map and the intermediate feature map; a prediction module, configured to obtain a predicted image of a second resolution according to the target feature map; the second resolution is higher than the first resolution; a construction module, configured to construct a loss function according to the predicted image and a real image corresponding to the image sample; an adjustment module, configured to adjust the initial model parameters of the current training stage according to the loss function to obtain the target model parameters of the current training stage; wherein the basic block is a residual network, and the residual path of the residual network comprises: a convolutional layer and an attention layer.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.

[0012] According to the embodiments of the present disclosure, image super-resolution can be achieved efficiently and light-weight.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of an application scenario for implementing an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of an image super-resolution model provided according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of a basic block provided according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram according to a third embodiment of the present disclosure;

[0021] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0022] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0023] Fig. 9 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0024] Fig.10 It is a schematic diagram of an electronic device used to implement the image super-resolution method or the image super-resolution model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] In order to achieve image super-resolution efficiently and lightly, the present disclosure provides the following embodiments.

[0027] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure. This embodiment provides an image super-resolution method, such as Figure 1 As shown, the method includes:

[0028] 101. Perform feature extraction on an original image of a first resolution to obtain an initial feature map.

[0029] 102. Based on a basic block, the initial feature map is processed to obtain an intermediate feature map; the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0030] 103. Obtain a target feature map based on the initial feature map and the intermediate feature map.

[0031] 104. Based on the target feature map, obtain a target image with a second resolution; the second resolution is higher than the first resolution.

[0032] The original image refers to the image before processing.

[0033] The target image refers to the processed image.

[0034] In the image super-resolution scenario, the original image refers to a low-resolution image, and the target image refers to a high-resolution image. That is, the resolution of the original image is represented by a first resolution, and the resolution of the target image is represented by a second resolution, and the second resolution is higher than the first resolution. For example, the first resolution is 128*128, and the second resolution is 256*256.

[0035] The initial feature map refers to the feature map obtained after feature extraction of the original image. Specifically, the original image can be processed using a convolutional network to obtain the initial feature map.

[0036] The intermediate feature map refers to the feature map after the initial feature map is processed by the basic block.

[0037] The basic block is the basic unit for processing feature maps, and the specific number can be set according to actual needs.

[0038] Residual Network (ResNet) is a network architecture in deep learning, usually composed of two parts, namely shortcut path and residual path. The shortcut path usually does not contain nonlinear transformations; the residual path usually contains nonlinear transformations such as multiple convolutional layers. The output of the residual path is added to the output of the shortcut path to get the final output. It can be expressed as:

[0039] H(x)=F(x)+A(x)

[0040] Among them, x is the input of the residual network, H(x) is the output of the residual network, F(x) is the output of the residual path, and A(x) is the output of the shortcut path. If the shortcut path is a direct path, then A(x) = x.

[0041] In this embodiment, the basic block is a residual network, in which the attention layer and the convolutional layer constitute the residual path, and the shortcut path is the direct path.

[0042] The target feature map refers to the feature map obtained by processing the initial feature map and the intermediate feature map. Afterwards, the target feature map can be used to reconstruct the image.

[0043] An image reconstructed based on a target feature map may be called a reconstructed image. For example, a convolutional network may be used to process the target feature map to obtain a reconstructed image.

[0044] In order to obtain a high-resolution target image, the reconstructed image can be upsampled to obtain the target image.

[0045] In this embodiment, an intermediate feature map is obtained through processing of the above-mentioned basic blocks, a target feature map is obtained based on the intermediate feature map and the initial feature map, and a target image is obtained based on the target feature map, which can effectively balance accuracy and efficiency, thereby realizing image super-resolution in an efficient and lightweight manner.

[0046] In order to better understand the present disclosure, the application scenarios involved in the present disclosure are described as follows:

[0047] Figure 2 It is a schematic diagram of an application scenario for implementing the embodiment of the present disclosure.

[0048] like Figure 2 As shown, an application (Application, APP) for image super-resolution can be installed on the user terminal 201. The user can upload the original image (such as a low-resolution image) through the APP. Taking the image super-resolution performed locally on the user terminal as an example, after the user terminal obtains the original image, the pre-deployed image super-resolution model can be used to process the original image to obtain the target image (such as a high-resolution image). After obtaining the target image, it can be displayed to the user through the APP.

[0049] The above takes the local deployment of the image super-resolution model in the user terminal as an example. It can be understood that the image super-resolution model can also be deployed on the server. The user terminal interacts with the server and sends the original image to the server. The server enhances the original image based on the image super-resolution model to obtain the target image, and returns it to the user terminal for display.

[0050] Figure 3It is a schematic diagram of an image super-resolution model provided according to an embodiment of the present disclosure.

[0051] The image super-resolution model includes: a feature extraction network 301, a feature processing network 302, a reconstruction network 303 and an upsampling module 304.

[0052] The feature extraction network 301 is used to extract features from the input original image, and the output is an initial feature map.

[0053] The feature extraction network may specifically be a convolutional network, specifically a 3x3 convolutional block, represented by the first Conv-3.

[0054] The feature processing network 302 is used to process the initial feature map to obtain a target feature map.

[0055] like Figure 3 As shown, the feature processing network is a residual network, and its shortcut path is a direct path, that is, a path with the same input and output, represented by a straight line. Specifically, the output of the direct path is the initial feature map.

[0056] Its residual path includes: basic blocks.

[0057] The number of basic blocks can be set according to actual needs.

[0058] In order to make the network lightweight enough to adapt to hardware with very limited resources, in this embodiment, the number of basic blocks is 3 and the network width is selected to be 16, which can reduce network redundant parameters and thus ensure the real-time performance of the model.

[0059] When there are multiple basic blocks, such as Figure 3 As shown, the residual path of the feature processing network also includes a concatenation module and a dimension conversion module. The dimension conversion module can specifically be a 1x1 convolution block (Conv-1).

[0060] The basic block is used to process the feature map, that is, to obtain the output feature map according to the input feature map.

[0061] In the overall process of obtaining the target feature map based on the initial feature map, the output feature map of the basic block can also be called an intermediate feature map.

[0062] When there is one basic block, the input feature map of the basic block is the initial feature map, and the output feature map is the output feature map of the residual path of the feature processing network.

[0063] When there are multiple basic blocks, the input feature map of the first basic block is the initial feature map, and the input feature map of the non-first basic block is the output feature map of the previous basic block.

[0064] For example, taking three basic blocks as an example, the three basic blocks are respectively called the first basic block, the second basic block and the third basic block. Figure 3 As shown, the input feature map of the first basic block is the initial feature map. Assuming that its output feature map is called the first feature map, then the first feature map is input into the second basic block, and after being processed by the second basic block, the output is the second feature map; the second feature map is input into the third basic block, and after being processed by the third basic block, the output is the third feature map.

[0065] The splicing module is used to splice multiple intermediate feature maps. For example, if they are spliced ​​by channel, the feature map obtained is called a spliced ​​feature map.

[0066] The dimension conversion module (Conv-1) is used to perform dimension conversion on the concatenated feature map to obtain a dimension-converted feature map whose dimension is the same as that of the initial feature map. After that, the initial feature map is added to the dimension-converted feature map to obtain a target feature map.

[0067] The reconstruction network 303 is used to obtain an image according to the feature map. In this embodiment, the input target feature map is reconstructed to obtain a reconstructed image. The reconstruction network can be a convolutional network, specifically a 3x3 convolutional block, represented by the second Conv-3.

[0068] The up-sampling module 304 is used to up-sample the reconstructed image to obtain a target image. The resolution of the target image is higher than the resolution of the original image. For example, the first resolution of the original image is 128*128, and the second resolution of the target image is 256*256.

[0069] The upsampling module can be specifically implemented using a sub-pixel convolution (PixelShuffle) operator.

[0070] The PixelShuffle operator is an operation used in fields such as image super-resolution, which can convert low-resolution images into high-resolution images. Compared with traditional interpolation methods, the PixelShuffle operator has better reconstruction effects and higher computational efficiency. Its working principle mainly includes: first, the input low-resolution image is nonlinearly transformed to learn a residual image. Then, the residual image is added pixel by pixel to the original low-resolution image to obtain the final high-resolution image. This process can be implemented in a neural network through a series of convolutional layers, deconvolutional layers, and pixel reorganization operations. The working principle is simple and efficient, so it can simply and efficiently realize the conversion of low-resolution images into high-resolution images.

[0071] Figure 4 is a schematic diagram of a basic block provided according to an embodiment of the present disclosure.

[0072] like Figure 4 As shown, the basic block is a residual network, whose shortcut path is a direct path, represented by a straight line, and its residual path includes: convolutional layer and attention layer.

[0073] like Figure 4 As shown, the convolution layer can specifically be a 3x3 convolution layer, denoted by Conv-3.

[0074] When there are multiple convolutional layers, such as three in this embodiment, for non-last convolutional layers, an activation layer is usually set after the corresponding convolutional layer.

[0075] The activation function of the activation layer may specifically be an Lrelu function. The Lrelu (Leakyrelu) function is a variant of the rectified linear unit (Relu) function, and compared with the Relu function, it can improve the nonlinear representation ability of the network.

[0076] like Figure 4 As shown, the attention layer can specifically adopt a channel attention layer.

[0077] Channel attention layer processes feature maps based on the channel attention mechanism.

[0078] The channel attention mechanism assigns a weight to each feature map channel. This weight reflects the importance of the channel to the final task. The input feature map is multiplied by the corresponding weight to obtain a weighted feature map. In this way, important channel features are amplified, while unimportant channel features are suppressed.

[0079] Compared with the Relu activation function, the Lrelu activation function can improve the nonlinear representation ability of the network. At the same time, it uses the channel attention mechanism (ChannelAttention) to focus on the significant channels in the feature map, thereby improving the representation ability while making the network lightweight.

[0080] For each basic block, its input feature map is processed by the residual path to obtain a residual feature map, and the input feature map and the residual feature map are added to obtain the output feature map of the basic block.

[0081] refer to Figure 3 , the input feature map of the first basic block (the first basic block) is the initial feature map, and the input feature map of other basic blocks (non-first basic blocks) is the output feature map of the previous basic block.

[0082] The output feature map of the basic block is also called the intermediate feature map. Figure 3, multiple intermediate feature maps can be concatenated to obtain a concatenated feature map; then the concatenated feature map is dimensionally transformed to obtain a dimensionally transformed feature map, and the dimensionally transformed feature map is added to the initial feature map to obtain a target feature map. Afterwards, image reconstruction and upsampling are performed based on the target feature map to obtain a target image.

[0083] In combination with the above application scenarios, the present disclosure also provides the following embodiments.

[0084] Figure 5 is a schematic diagram according to the second embodiment of the present disclosure, which provides an image super-resolution method.

[0085] This embodiment takes image super-resolution based on an image super-resolution model as an example.

[0086] like Figure 5 As shown, the method includes:

[0087] 501. Use a feature extraction network in an image super-resolution model to extract features from an original image of a first resolution to obtain an initial feature map.

[0088] For example, see Figure 3 , a 3x3 convolution block can be used to convolve the original image to obtain the initial feature map.

[0089] 502. Use the basic blocks in the image super-resolution model to process the initial feature map to obtain an intermediate feature map.

[0090] Among them, the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0091] Specifically, Figure 4 As shown, the attention layer can be a channel attention layer, and the convolution layer is a 3x3 convolution layer.

[0092] The channel attention mechanism assigns a weight to each feature map channel. This weight reflects the importance of the channel to the final task. The input feature map is multiplied by the corresponding weight to obtain a weighted feature map. In this way, important channel features are amplified, while unimportant channel features are suppressed.

[0093] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability and further improving the image super-resolution effect.

[0094] 503. Obtain a target feature map based on the initial feature map and the intermediate feature map.

[0095] Among them, reference Figure 3The feature processing network is also a residual network. At this time, the output feature map of the residual path of the feature processing network can be obtained based on the intermediate feature map, and then the initial feature map and the output feature map are added to obtain the target feature map.

[0096] Specifically, if there is one basic block, the output feature map of the basic block can be used as the output feature map of the residual path; or,

[0097] If there are multiple basic blocks, such as Figure 3 As shown, multiple intermediate feature maps (output feature maps of basic blocks) are concatenated to obtain a concatenated feature map, and then the concatenated feature map is dimensionally transformed to obtain the output feature map of the residual path.

[0098] 504. Reconstruct the target feature map using a reconstruction network in the image super-resolution model to obtain a reconstructed image.

[0099] 505. Use an upsampling module in the image super-resolution model to upsample the reconstructed image to obtain a target image with a second resolution; the second resolution is higher than the first resolution.

[0100] For example, refer to Figure 3 , a 3x3 convolution block is used to convolve the target feature map, and the output image is called the reconstructed image. The PixelShuffle operator is then used to upsample the reconstructed image to obtain the target image.

[0101] In this embodiment, a reconstructed image is obtained based on a target feature map, and the reconstructed image is upsampled to obtain a target image, so that the target image can be obtained simply and efficiently.

[0102] Figure 6 is a schematic diagram according to the third embodiment of the present disclosure, which provides an image super-resolution method.

[0103] This embodiment takes multiple basic blocks as an example and illustrates the specific processing flow of each basic block.

[0104] Since both the basic block and the feature processing network are residual networks, in order to distinguish them, the output feature map of the residual path of the basic block is called the first residual feature map, and the output feature map of the feature processing network is called the second residual feature map.

[0105] like Figure 6 As shown, the method includes:

[0106] 601. Perform feature extraction on an original image of a first resolution to obtain an initial feature map.

[0107] For example, see Figure 3, a 3x3 convolution block can be used to convolve the original image to obtain the initial feature map.

[0108] 602. Use the residual path in the basic block to process the input feature map to obtain a first residual feature map.

[0109] 603. Add the input feature map and the first residual feature map to obtain an output feature map, and use the output feature map as the intermediate feature map.

[0110] Among them, there are multiple basic blocks. If the basic block is the first basic block, the input feature map is the initial feature map; if the basic block is not the first basic block, the input feature map is the output feature map of the previous basic block.

[0111] Cross-layer connections are not friendly to user-side hardware device reasoning, so excessive use of residual connections should be avoided during network design.

[0112] In this embodiment, in the basic block, a residual connection can be used to simplify the residual structure and improve the reasoning speed.

[0113] 604. Concatenate multiple intermediate feature maps to obtain a concatenated feature map.

[0114] For example, refer to Figure 3 , Taking three intermediate feature maps as an example, the three intermediate feature maps are spliced ​​to obtain a spliced ​​feature map.

[0115] In this way, by splicing multiple intermediate feature maps to obtain a spliced ​​feature map, the feature information of the shallow, middle and deep layers can be integrated to improve the feature representation ability and thus improve the image super-resolution effect.

[0116] 605. Perform dimension conversion on the concatenated feature map to obtain a second residual feature map.

[0117] 606. Add the initial feature map and the second residual feature map to obtain a target feature map.

[0118] In this way, the target feature map is obtained by adding the initial feature map and the second residual feature map. The target feature map can be obtained based on the initial feature map using a primary residual structure, thereby improving processing efficiency and enhancing image super-resolution performance.

[0119] 607. Reconstruct the target feature map to obtain a reconstructed image.

[0120] 608. Upsample the reconstructed image to obtain a target image with a second resolution; the second resolution is higher than the first resolution.

[0121] Through the above-mentioned overall image super-resolution process, lightweight and efficient image super-resolution can be achieved, and the real-time performance of the image super-resolution process is improved, which is suitable for user-side processing scenarios.

[0122] After experiments, verification was carried out on public data sets such as Set5, Set14, Urban100 and BSD100. When the above-mentioned image super-resolution model was used for x2 (such as the resolution of the original image is 128*128, and the resolution of the target image is 256*256) and x4 (such as the resolution of the original image is 64*64, and the resolution of the target image is 256*256) image super-resolution, the number of parameters and the inference time were both small, indicating that the computational complexity was reduced, the processing efficiency was improved, the inference time was reduced, and thus the real-time performance was improved; PSNR and SSIM were improved compared to related algorithms, indicating that the image super-resolution effect was improved. Compared with the ordinary bilinear interpolation algorithm and other lightweight super-resolution algorithms, such as Super-Resolution Convolutional Neural Networks (SRCNN) and Fast Super-Resolution CNN (FSRCNN), the inference time is shorter under x2 and x4 on the 1080Ti graphics card, which shows that the model improves performance indicators while improving real-time performance.

[0123] Among them, Set5, Set14, Urban100 and BSD100 are public data sets, PSNR is peak signal-to-noise ratio, and SSIM is structural similarity index.

[0124] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure. This embodiment provides an image super-resolution model training method. The image super-resolution model includes a basic block. The method includes:

[0125] 701. Perform feature extraction on an image sample of a first resolution to obtain an initial feature map.

[0126] 702. Based on the basic block, process the initial feature map to obtain an intermediate feature map.

[0127] 703. Obtain a target feature map based on the initial feature map and the intermediate feature map.

[0128] 704. Based on the target feature map, obtain a predicted image with a second resolution.

[0129] 705. Construct a loss function based on the predicted image and the real image corresponding to the image sample.

[0130] 706. Based on the loss function, adjust the initial model parameters of the current training stage to obtain the target model parameters of the current training stage.

[0131] The basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer;

[0132] The second resolution is higher than the first resolution.

[0133] Among them, high-resolution images, such as 256*256 images, can be obtained from the existing sample set, and these high resolution images are downsampled to obtain low-resolution image samples, such as image samples with a resolution of 128*128. The above high-resolution images are used as real images corresponding to the image samples.

[0134] The process of obtaining the predicted image during model training is similar to the process of obtaining the target image during image super-resolution, and will not be repeated here.

[0135] After obtaining the predicted image, a loss function can be constructed based on the predicted image and the real image, such as using the L1 loss function and the L2 loss function alternately. The L1 loss function calculates the average of the absolute difference between the predicted value and the real value, and the L2 loss function is also called the Mean Squared Error (MSE) loss function, which calculates the average of the square of the difference between the predicted value and the real value.

[0136] When using L1 loss function and L2 loss function alternately, L1loss can be used first and then L2loss. For example, L1loss is used in the first 600 training cycles (epochs) and L2loss is used in the next 400 epochs. L1loss focuses on optimizing the details of the image, while L2loss makes the image smoother as a whole.

[0137] After constructing the loss function, you can use a parameter optimization algorithm, such as the Adam algorithm, to adjust the model parameters until the preset end condition is reached to minimize the loss function. Figure 3 , specifically, the parameters of the first Conv-3, the basic block, Conv-1 and the second Conv-3 can be adjusted.

[0138] Adam (Adaptive Moment Estimation) is an optimization algorithm based on gradient descent, which combines the characteristics of momentum and adaptive learning rate. The main idea is to dynamically adjust the learning rate of each parameter according to the gradient of the parameter.

[0139] For example, the learning rate is initialized to 1×10 -3 , every 2×10 5 The learning rate is halved for 1×10 iterations, and the loss function is minimized using the Adam optimizer. 6 Iterations.

[0140] In this embodiment, an intermediate feature map is obtained through processing of the above-mentioned basic blocks, and a target feature map is obtained based on the intermediate feature map and the initial feature map. Then, model training is performed based on the target feature map, which can effectively balance the accuracy and efficiency of the model, thereby realizing image super-resolution in an efficient and lightweight manner.

[0141] The training process can be divided into one or more training stages. For example, if one training stage is used, the model that reaches the preset end condition (such as reaching the preset number of iterations) is used as the final model for the image super-resolution processing process in the inference stage.

[0142] If multiple training stages are used, each training stage can be used as the current training stage. At this time, if the current training stage is not the first training stage, the target model parameters of the previous training stage are used as the initial model parameters of the current training stage.

[0143] For example, if there are three training stages, the target model parameters of the first training stage are used as the initial model parameters of the second training stage, the target model parameters of the second training stage are used as the initial model parameters of the third training stage, and the target model parameters of the third training stage are used as the final model parameters for the inference stage. The initial model parameters of the first training stage may be randomly generated.

[0144] The configuration parameters of each training stage, such as image samples, number of iterations, initial value and variation rule of learning rate, etc., can be the same.

[0145] In this embodiment, by using the target model parameters of the previous training stage as the initial model parameters of the current training stage, cyclic training can be implemented to achieve the purpose of warm-up, thereby making the training close to the global optimum and stabilizing the training process.

[0146] After experiments, taking 2x resolution super-resolution as an example, as the number of training stages increases, the PSNR and SSIM indicators of the model also improve accordingly, which can effectively promote the improvement of model performance without incurring additional reasoning overhead.

[0147] In some embodiments, the processing of the initial feature map based on the basic block to obtain an intermediate feature map includes:

[0148] Processing the input feature map using the residual path to obtain a first residual feature map;

[0149] Adding the input feature map and the first residual feature map to obtain an output feature map, and using the output feature map as the intermediate feature map;

[0150] Among them, there are multiple basic blocks. If the basic block is the first basic block, the input feature map is the initial feature map; if the basic block is not the first basic block, the input feature map is the output feature map of the previous basic block.

[0151] In this embodiment, in the basic block, the residual structure can be simplified through a residual connection, thereby simplifying the model structure, and further improving the reasoning speed when using the model for reasoning.

[0152] In some embodiments, the attention layer is a channel attention layer.

[0153] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability, thereby improving the image super-resolution model and the image super-resolution effect.

[0154] In some embodiments, obtaining a target feature map based on the initial feature map and the intermediate feature map includes:

[0155] Based on the intermediate feature map, obtaining a second residual feature map;

[0156] The initial feature map and the second residual feature map are added to obtain the target feature map.

[0157] In this embodiment, the target feature map is obtained by adding the initial feature map and the second residual feature map. A primary residual structure can be used to obtain the target feature map based on the initial feature map, simplifying the model structure, and then using the model for reasoning to improve processing efficiency and enhance image super-resolution performance.

[0158] In some embodiments, the intermediate feature map is multiple;

[0159] The obtaining a second residual feature map based on the intermediate feature map includes:

[0160] Splicing the multiple intermediate feature maps to obtain a spliced ​​feature map;

[0161] Perform dimension transformation on the concatenated feature map to obtain the second residual feature map.

[0162] In this embodiment, by splicing multiple intermediate feature maps to obtain a spliced ​​feature map, the feature information of the shallow, middle and deep layers can be integrated to improve the feature representation capability and thus improve the image super-resolution effect.

[0163] In some embodiments, acquiring a predicted image with a second resolution based on the target feature map includes:

[0164] Reconstructing the target feature map to obtain a reconstructed image;

[0165] The reconstructed image is upsampled to obtain the target image.

[0166] In this embodiment, a reconstructed image is obtained based on a target feature map, and the reconstructed image is upsampled to obtain a target image, so that the target image can be obtained simply and efficiently.

[0167] Figure 8 It is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides an image super-resolution device, and the device 800 includes: an extraction module 801, a processing module 802, an acquisition module 803 and a prediction module 804.

[0168] The extraction module 801 is used to extract features from image samples of the first resolution to obtain an initial feature map; the processing module 802 is used to process the initial feature map according to the basic block to obtain an intermediate feature map; the acquisition module 803 is used to obtain a target feature map according to the initial feature map and the intermediate feature map; the prediction module 804 is used to obtain a predicted image of the second resolution according to the target feature map; the second resolution is higher than the first resolution; the construction module is used to construct a loss function according to the predicted image and the real image corresponding to the image sample; the adjustment module is used to adjust the initial model parameters of the current training stage according to the loss function to obtain the target model parameters of the current training stage; wherein the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0169] In this embodiment, an intermediate feature map is obtained through processing of the above-mentioned basic blocks, a target feature map is obtained based on the intermediate feature map and the initial feature map, and a target image is obtained based on the target feature map, which can effectively balance accuracy and efficiency, thereby realizing image super-resolution in an efficient and lightweight manner.

[0170] In some embodiments, the processing module 802 is further configured to:

[0171] Processing the input feature map using the residual path to obtain a first residual feature map;

[0172] Adding the input feature map and the first residual feature map to obtain an output feature map, and using the output feature map as the intermediate feature map;

[0173] Among them, there are multiple basic blocks. If the basic block is the first basic block, the input feature map is the initial feature map; if the basic block is not the first basic block, the input feature map is the output feature map of the previous basic block.

[0174] In this embodiment, in the basic block, a residual connection can be used to simplify the residual structure and improve the reasoning speed.

[0175] In some embodiments, the attention layer is a channel attention layer.

[0176] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability and further improving the image super-resolution effect.

[0177] In some embodiments, the acquisition module 803 is further used to:

[0178] Based on the intermediate feature map, obtaining a second residual feature map;

[0179] The initial feature map and the second residual feature map are added to obtain the target feature map.

[0180] In this embodiment, the target feature map is obtained by adding the initial feature map and the second residual feature map. A primary residual structure can be used to obtain the target feature map based on the initial feature map, thereby improving processing efficiency and enhancing image super-resolution performance.

[0181] In some embodiments, there are multiple intermediate feature maps; the acquisition module 803 is further used to:

[0182] Splicing the multiple intermediate feature maps to obtain a spliced ​​feature map;

[0183] Perform dimension transformation on the concatenated feature map to obtain the second residual feature map.

[0184] In this embodiment, by splicing multiple intermediate feature maps to obtain a spliced ​​feature map, the feature information of the shallow, middle and deep layers can be integrated to improve the feature representation capability and thus improve the image super-resolution effect.

[0185] In some embodiments, the prediction module 804 is further configured to:

[0186] Reconstructing the target feature map to obtain a reconstructed image;

[0187] The reconstructed image is upsampled to obtain the target image.

[0188] In this embodiment, a reconstructed image is obtained based on a target feature map, and the reconstructed image is upsampled to obtain a target image, so that the target image can be obtained simply and efficiently.

[0189] Fig. 9 It is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides an image super-resolution model training device, and the image super-resolution model includes basic blocks. The device 900 includes: an extraction module 901, a processing module 902, an acquisition module 903, a prediction module 904, a construction module 905 and an adjustment module 906.

[0190] The extraction module 901 is used to extract features from image samples of the first resolution to obtain an initial feature map; the processing module 902 is used to process the initial feature map according to the basic block to obtain an intermediate feature map; the acquisition module 903 is used to obtain a target feature map according to the initial feature map and the intermediate feature map; the prediction module 904 is used to obtain a predicted image of the second resolution according to the target feature map; the second resolution is higher than the first resolution; the construction module 905 is used to construct a loss function according to the predicted image and the real image corresponding to the image sample; the adjustment module 906 is used to adjust the initial model parameters of the current training stage according to the loss function to obtain the target model parameters of the current training stage; wherein the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

[0191] In some embodiments, if there are multiple training stages, for the current training stage in the multiple training stages:

[0192] If the current training stage is not the first training stage, the target model parameters of the previous training stage are used as the initial model parameters of the current training stage.

[0193] In this embodiment, by using the target model parameters of the previous training stage as the initial model parameters of the current training stage, cyclic training can be implemented to achieve the purpose of warm-up, thereby making the training close to the global optimum and stabilizing the training process.

[0194] In some embodiments, the processing module 902 is further configured to:

[0195] Processing the input feature map using the residual path to obtain a first residual feature map;

[0196] Adding the input feature map and the first residual feature map to obtain an output feature map, and using the output feature map as the intermediate feature map;

[0197] Among them, there are multiple basic blocks. If the basic block is the first basic block, the input feature map is the initial feature map; if the basic block is not the first basic block, the input feature map is the output feature map of the previous basic block.

[0198] In this embodiment, in the basic block, the residual structure can be simplified through a residual connection, thereby simplifying the model structure, and further improving the reasoning speed when using the model for reasoning.

[0199] In some embodiments, the attention layer is a channel attention layer.

[0200] In this embodiment, a channel attention layer is used to focus on important channels of the feature map, thereby improving the feature representation capability, thereby improving the image super-resolution model and the image super-resolution effect.

[0201] In some embodiments, the acquisition module 903 is further used to:

[0202] Based on the intermediate feature map, obtaining a second residual feature map;

[0203] The initial feature map and the second residual feature map are added to obtain the target feature map.

[0204] In this embodiment, the target feature map is obtained by adding the initial feature map and the second residual feature map. A primary residual structure can be used to obtain the target feature map based on the initial feature map, simplifying the model structure, and then using the model for reasoning to improve processing efficiency and enhance image super-resolution performance.

[0205] In some embodiments, there are multiple intermediate feature maps; the acquisition module 903 is further used to:

[0206] Splicing the multiple intermediate feature maps to obtain a spliced ​​feature map;

[0207] Perform dimension transformation on the concatenated feature map to obtain the second residual feature map.

[0208] In this embodiment, by splicing multiple intermediate feature maps to obtain a spliced ​​feature map, the feature information of the shallow, middle and deep layers can be integrated to improve the feature representation capability and thus improve the image super-resolution effect.

[0209] In some embodiments, the prediction module 904 is further configured to:

[0210] Reconstructing the target feature map to obtain a reconstructed image;

[0211] The reconstructed image is upsampled to obtain the target image.

[0212] In this embodiment, a reconstructed image is obtained based on a target feature map, and the reconstructed image is upsampled to obtain a target image, so that the target image can be obtained simply and efficiently.

[0213] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0214] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0215] It is understandable that unless there is any special limitation on the sequence of steps in the process, it means that the timing relationship between these steps is not limited.

[0216] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0217] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0218] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0219] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0220] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0221] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as an image super-resolution method or an image super-resolution model training method. For example, in some embodiments, the image super-resolution method or the image super-resolution model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1009. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the image super-resolution method or the image super-resolution model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the image super-resolution method or the image super-resolution model training method in any other appropriate manner (for example, by means of firmware).

[0222] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0223] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable task processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0224] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0225] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0226] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0227] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0228] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0229] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image super-resolution method, comprising: Performing feature extraction on the original image of the first resolution to obtain an initial feature map; Based on the basic block, the initial feature map is processed to obtain an intermediate feature map; Based on the initial feature map and the intermediate feature map, obtaining a target feature map; Based on the target feature map, acquiring a target image of a second resolution; The second resolution is higher than the first resolution; Among them, the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

2. The method according to claim 1, wherein: The step of processing the initial feature map based on the basic block to obtain an intermediate feature map includes: Processing the input feature map using the residual path to obtain a first residual feature map; Adding the input feature map and the first residual feature map to obtain an output feature map, and using the output feature map as the intermediate feature map; Among them, there are multiple basic blocks. If the basic block is the first basic block, the input feature map is the initial feature map; if the basic block is not the first basic block, the input feature map is the output feature map of the previous basic block.

3. The method according to claim 1, wherein: The attention layer is a channel attention layer.

4. The method according to claim 1, wherein: The step of acquiring a target feature map based on the initial feature map and the intermediate feature map includes: Based on the intermediate feature map, obtaining a second residual feature map; The initial feature map and the second residual feature map are added to obtain the target feature map.

5. The method according to claim 4, wherein: There are multiple intermediate feature maps; The obtaining a second residual feature map based on the intermediate feature map includes: Splicing the multiple intermediate feature maps to obtain a spliced ​​feature map; Perform dimension transformation on the concatenated feature map to obtain the second residual feature map.

6. The method according to claim 1, wherein: The step of acquiring a target image with a second resolution based on the target feature map includes: Reconstructing the target feature map to obtain a reconstructed image; The reconstructed image is upsampled to obtain the target image.

7. A method for training an image super-resolution model, wherein the image super-resolution model includes a basic block, and the method includes: Performing feature extraction on the image samples of the first resolution to obtain an initial feature map; Based on the basic block, processing the initial feature map to obtain an intermediate feature map; Based on the initial feature map and the intermediate feature map, obtaining a target feature map; Based on the target feature map, obtaining a predicted image with a second resolution; The second resolution is higher than the first resolution; Constructing a loss function based on the predicted image and the real image corresponding to the image sample; Based on the loss function, adjusting the initial model parameters of the current training stage to obtain the target model parameters of the current training stage; Among them, the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

8. The method according to claim 7, wherein: If there are multiple training stages, for the current training stage in the multiple training stages: If the current training stage is not the first training stage, the target model parameters of the previous training stage are used as the initial model parameters of the current training stage.

9. The method according to claim 7 or 8, wherein: The step of processing the initial feature map based on the basic block to obtain an intermediate feature map includes: Processing the input feature map using the residual path to obtain a first residual feature map; Adding the input feature map and the first residual feature map to obtain an output feature map, and using the output feature map as the intermediate feature map; Among them, there are multiple basic blocks. If the basic block is the first basic block, the input feature map is the initial feature map; if the basic block is not the first basic block, the input feature map is the output feature map of the previous basic block.

10. The method according to claim 7 or 8, wherein: The attention layer is a channel attention layer.

11. The method according to claim 7 or 8, wherein: The step of acquiring a target feature map based on the initial feature map and the intermediate feature map includes: Based on the intermediate feature map, obtaining a second residual feature map; The initial feature map and the second residual feature map are added to obtain the target feature map.

12. The method according to claim 11, wherein: There are multiple intermediate feature maps; The obtaining a second residual feature map based on the intermediate feature map includes: Splicing the multiple intermediate feature maps to obtain a spliced ​​feature map; Perform dimension transformation on the concatenated feature map to obtain the second residual feature map.

13. The method according to claim 7 or 8, wherein: The step of acquiring a predicted image with a second resolution based on the target feature map includes: Reconstructing the target feature map to obtain a reconstructed image; The reconstructed image is upsampled to obtain the target image.

14. An image super-resolution device, comprising: An extraction module, used for performing feature extraction on the original image of the first resolution to obtain an initial feature map; A processing module, configured to process the initial feature map according to the basic block to obtain an intermediate feature map; An acquisition module, used for acquiring a target feature map based on the initial feature map and the intermediate feature map; A prediction module, used for acquiring a target image of a second resolution based on the target feature map; The second resolution is higher than the first resolution; Among them, the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

15. A device for training an image super-resolution model, the image super-resolution model comprising a basic block, the device comprising: An extraction module, used for performing feature extraction on the image sample of the first resolution to obtain an initial feature map; A processing module, configured to process the initial feature map according to the basic block to obtain an intermediate feature map; An acquisition module, used for acquiring a target feature map according to the initial feature map and the intermediate feature map; A prediction module, configured to obtain a prediction image of a second resolution according to the target feature map; the second resolution is higher than the first resolution; A construction module, used to construct a loss function according to the predicted image and the real image corresponding to the image sample; An adjustment module, used to adjust the initial model parameters of the current training stage according to the loss function to obtain the target model parameters of the current training stage; Among them, the basic block is a residual network, and the residual path of the residual network includes: a convolutional layer and an attention layer.

16. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images