An image super-resolution reconstruction method based on attribution-oriented data augmentation

Abstract

The application discloses an image super-resolution reconstruction method based on attribution-oriented data augmentation, comprising the following steps: reading an input image and generating a reference image sequence; inputting the reference image sequence into a pre-training network for attribution analysis to generate a saliency map; using the saliency map to find a pixel with the maximum saliency value and generate a corresponding binary mask image; using the binary mask image to optimize and mix multiple data augmentation methods to obtain an attribution-oriented data augmentation method; using the attribution-oriented data augmentation method to obtain a training image set for training an image super-resolution network; and using the trained image super-resolution network to perform image super-resolution reconstruction. The application realizes a general and efficient image super-resolution network data augmentation method, can effectively optimize the super-resolution network performance, and has high use value.

Description

TECHNICAL FIELD

[0001] The present application relates to an image super-resolution reconstruction method, in particular to an image super-resolution reconstruction method based on attribution-oriented data augmentation. BACKGROUND

[0002] The part provided in this part is only background information related to the present disclosure, which does not necessarily constitute prior art.

[0003] Data augmentation method is one of the most practical and effective methods to enhance the performance and generalization ability of deep neural network. In advanced visual tasks such as image recognition, semantic segmentation, early ordinary augmentation methods such as MIXUP, CUTMIX lay the foundation. In recent years, the augmentation method based on saliency has attracted much attention and has shown superior performance to ordinary augmentation methods in advanced visual tasks.

[0004] Although the data augmentation method has developed rapidly in advanced visual tasks, it is very challenging to directly migrate the saliency-based augmentation method to low-level visual tasks. These difficulties make the current data augmentation method in low-level visual tasks limited to ordinary augmentation methods such as flip, rotation, MIXUP, CUTBLUR, CUTMIB.

[0005] The ordinary data augmentation method in low-level visual tasks (such as image super-resolution) faces the bottleneck of information loss, which seriously damages the quality of the reconstructed image. The ordinary data augmentation method often generates flat and simple edge regions for augmented images, which have little contribution to the image super-resolution task, resulting in serious information loss. On the contrary, the saliency augmentation method can provide more key information for the reconstruction. This drives the development of a new augmentation method that focuses on more meaningful regions in the image super-resolution task. To provide more meaningful information for augmented images, alleviate the problem of information loss and help the training of super-resolution network.

[0006] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The present application aims to solve the technical problems of the prior art and provides an image super-resolution reconstruction method based on attribution-oriented data augmentation.

[0008] In order to solve the above technical problems, the present application discloses an image super-resolution reconstruction method based on attribution-oriented data augmentation, comprising the following steps:

[0009] Step 1, read the input image and generate a reference image sequence;

[0010] Step 2, input the reference image sequence into the pre-trained network for attribution analysis to generate a saliency map;

[0011] Step 3, using the saliency map, find the pixel with the maximum saliency value and generate a corresponding binary mask map;

[0012] Step 4, using the binary mask map, optimize and mix a plurality of data augmentation methods to obtain an attribution-oriented data augmentation method;

[0013] Step 5, using the attribution-oriented data augmentation method, obtain a training image set for training an image super-resolution network;

[0014] Step 6, using the trained image super-resolution network for image super-resolution reconstruction, complete the attribution-oriented data augmentation based image super-resolution reconstruction.

[0015] Further, the generating reference image sequence in step 1 specifically includes:

[0016] Step 1-1, generate a truncated normal distribution function, and sample to obtain reference image parameter points subject to normal distribution;

[0017] Step 1-2, using the reference image parameter points, generate a reference image sequence.

[0018] Further, the generating truncated normal distribution function in step 1-1, sampling to obtain reference image parameter points subject to normal distribution, specifically includes:

[0019] Step 1-1-1, randomly generate a truncated normal distribution Subscript indicates that the function is a truncated function, where is a random normal distribution parameter, a min ,a max is the lower limit and upper limit of the truncation, subscript min represents the minimum value, and max represents the maximum value;

[0020] Step 1-1-2, set the lower limit parameter and upper limit parameter a min ,a max and the distribution parameter Then randomly sample k parameters from the truncated normal distribution function to generate k reference image parameter points

[0021] The generating reference image sequence for input in step 1-2 specifically includes:

[0022] Using the k reference map parameter points described in step 1-1-2 Generate the corresponding reference image sequence Among them, the i-th reference map parameter point is used. Generate the i-th reference image The method is as follows:

[0023] Select reference graph parameter points but:

[0024]

[0025] in, The input image is ω(σ), which represents a Gaussian blur kernel function with a kernel width of σ. It is a convolution operation.

[0026] Furthermore, the generation of the saliency map in step 2 specifically includes:

[0027] Step 2-1: Input the reference image sequence into the pre-trained network to generate a high-resolution reference image corresponding to each reference image in the reference image sequence;

[0028] Step 2-2: Use a global feature detector to obtain global features of the high-resolution reference image;

[0029] Steps 2-3: Use the global features of the high-resolution reference image to backpropagate the pre-trained network and calculate the backpropagation gradient of the corresponding reference image to the input.

[0030] Steps 2-4: Calculate the difference between all reference maps and their adjacent reference maps, and use integral accumulation to calculate the sum of the products of the backpropagation gradients of all reference maps and the differences to obtain the saliency map.

[0031] Furthermore, the specific method for generating the high-resolution reference image corresponding to each reference image in the reference image sequence as described in step 2-1 includes:

[0032]

[0033] in, It is a pre-trained image super-resolution network. This is the generated high-resolution reference image, where the superscript ↑ indicates that the image is a high-resolution image.

[0034] Furthermore, the method for obtaining the global features of the high-resolution reference image as described in step 2-2 includes:

[0035] Step 2-2-1: Generate the SOBEL operator along the x-axis direction of the high-resolution reference image. Perform calculations to obtain the image gradient along the x-axis. Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript x indicates that the gradient is calculated along the x-axis.

[0036] Step 2-2-2: Generate the SOBEL operator along the y-axis direction on the high-resolution reference image. Perform calculations to obtain the image gradient along the y-axis. Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript y indicates that the gradient is calculated along the y-axis.

[0037] Step 2-2-3: Calculate the approximate values ​​of the combined horizontal and vertical gradients at each pixel of the high-resolution reference image. Specifically as follows:

[0038]

[0039] Step 2-2-4: Calculate the global features of the high-resolution reference image. That is, to calculate the second normal form of the gradient of all pixels in the high-resolution reference image, as follows:

[0040]

[0041] Among them, h0, w0 and h max ,w max These represent the minimum and maximum positions of pixels in the high-resolution reference image, respectively. The subscripts 0 and max represent the specific coordinates of the minimum and maximum values, respectively. GD indicates that the gradient is a global gradient value.

[0042] Furthermore, the calculation of the backpropagation gradient with respect to the corresponding reference graph in steps 2-3 specifically includes the following methods:

[0043] Calculate the reference graph sequence sequentially The backpropagation gradient values ​​of all reference graphs in the i-th reference graph. The method for calculating the backpropagation gradient value is as follows;

[0044] Step 2-3-1, select the i-th reference image.

[0045] Step 2-3-2: Calculate the global features of the i-th high-resolution reference image relative to the reference image. The backpropagation gradient values ​​are as follows:

[0046]

[0047] in, This represents the i-th reference image. This represents the i-th high-resolution reference image.

[0048] Further, the method of obtaining the saliency map in step 2-4 includes:

[0049] Step 2-4-1, calculate the difference between the input i-th and i+1-th adjacent reference maps, specifically as follows:

[0050]

[0051] Step 2-4-2, calculate the product of the difference in step 2-4-1 and the backpropagation gradient in step 2-3-2, the difference between the i-th and i+1-th adjacent reference maps and the backpropagation gradient product are:

[0052]

[0053] Step 2-4-3, accumulate the difference and backpropagation gradient product between all k adjacent reference maps, the integral of the input image saliency map Specifically as follows:

[0054]

[0055] Where subscript s represents that the map is a saliency map, and superscript represents that the map size is a low resolution map.

[0056] Further, the method of generating a corresponding binary mask map in step 3 includes:

[0057] Step 3-1, using the saliency map obtained in step 2-4-3 search for the position of the pixel with the maximum saliency value (x max ,y max ), specifically as follows:

[0058]

[0059] Where (x, y) represents the coordinate position, subscript max represents that the coordinate has the maximum saliency value, represents finding the maximum value;

[0060] Step 3-2, in the saliency map with a height of H and a width of W, centering on the position of the pixel with the maximum saliency value, select a block with a random size of h in height and w in width and obtain the binary mask map M of the block, specifically as follows:

[0061]

[0062] wherein M e {0, 1} H×W denotes a binary mask map with value 0 or 1, height H, and width W.

[0063] Further, the step 4 uses the binary mask map to optimize and mix a plurality of data augmentation methods, and the specific method comprises:

[0064] Step 4-1, randomly extract the first image pair and the second image pair from the input training image data. Optimize the data augmentation method CUTMIX using the binary mask map to generate a new sample pair for training, i.e. low resolution image and high resolution image The specific method is as follows:

[0065]

[0066] wherein ⊙ denotes a pixel-wise Hadamard product operation, and denote the input low resolution image, and denote the input high resolution image, the subscript and denote the image number, and the superscript and respectively denote that the image is a low resolution and high resolution image, and the subscript cutmix denotes that the augmented image pair is generated by the optimized CUTMIX;

[0067] Step 4-2, optimize the data augmentation method CUTMIXUP using the binary mask map, and the specific method is as follows:

[0068] Step 4-2-1, respectively cut the image blocks for pasting from the low resolution image and the high resolution image:

[0069]

[0070] wherein P denotes that the image is an image block, the subscript mix denotes that the image block is a mixed image block, and λ is a weighting coefficient;

[0071] Step 4-2-2, generate a new sample pair for training, i.e. low resolution image and high resolution image The specific method is as follows:

[0072]

[0073] wherein and denote the image sequence number, superscript and denote that the image is a low-resolution and high-resolution image respectively, subscript cutmixup denotes that the augmented image pair is generated by the optimized CUTMIXUP;

[0074] Step 4-3, the data augmentation method CUTBLUR is optimized using a binary mask image, specifically as follows:

[0075] Step 4-3-1, cut down the image blocks from the low-resolution image and the high-resolution image respectively and

[0076]

[0077] Step 4-3-2, the cut-down low-resolution image block is s times bicubic interpolation BICUBIC up-sampling to obtain The cut-down high-resolution image block is s times bicubic interpolation BICUBIC down-sampling to obtain

[0078] Step 4-3-3, a new sample pair for training is generated, that is, the high-resolution image and the low-resolution image Specifically as follows:

[0079]

[0080] Wherein, ⊙ denotes a pixel-by-pixel Hadamard product operation, and denote the image sequence number, superscript and denote that the image is a high-resolution and low-resolution image respectively, denote that the augmented image pair is generated by the optimized CUTBLUR;

[0081] Step 4-4, the data augmentation methods described in steps 4-1 to 4-3 are mixed, that is, the probability p is set for selecting the data augmentation method.

[0082] Beneficial effects:

[0083] 1. The present application adopts an attribution analysis method to extract a saliency map, optimizes existing ordinary data augmentation methods and performs fusion, and augments the input image of the training image super-resolution network. For any low-level visual task, as long as attribution analysis can be performed, the method proposed in the present application can be used.

[0084] 2、The application completely operates on the data level, can adapt to neural networks of any structure and design, and has strong compatibility, and can be applied to various low-level vision tasks including image super resolution, and effectively improves network performance and generalization capability. BRIEF DESCRIPTION OF DRAWINGS

[0085] The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0086] Figure 1 It is a schematic diagram of the overall process of the application.

[0087] Figure 2 It is a schematic diagram of the attribution analysis process.

[0088] Figure 3 It is a data augmentation operation flowchart.

[0089] Figure 4 It is a binary mask generation flowchart.

[0090] Figure 5 It is a result schematic diagram in an embodiment. DETAILED DESCRIPTION

[0091] The application proposes an image super resolution reconstruction method based on attribution-oriented data augmentation, and the core is a data augmentation method based on attribution analysis suitable for low-level vision tasks. The method completely operates on the data level, is compatible with any structure of neural network, augments different structure and type of model, and has strong compatibility.

[0092] The application discloses an image super resolution reconstruction method based on attribution-oriented data augmentation, as shown in the figure, comprising the following steps: Figure 1

[0093] Step 1, generating a reference image for inputting a pre-trained network for attribution analysis;

[0094] Step 2, generating a truncated normal distribution function, and sampling to obtain reference image parameter points subject to normal distribution;

[0095] Step 3, using the sampled reference image parameter points to generate a reference image for input;

[0096] Step 4, inputting the sampled reference image into the pre-trained network to obtain a super resolution image of the reference image;

[0097] Step 5, using a global feature detector to obtain global features of the high-resolution reference image;​

[0098] Step 6: Backpropagate the pre-trained network using the global features of the high-resolution reference image to calculate the gradient of the super-resolution network with respect to the input reference image.

[0099] Step 7: Calculate the difference between all adjacent reference maps, and use integral accumulation to calculate the sum of the products of all gradients and the difference;

[0100] Step 8: Using the saliency map, find the pixel with the largest saliency value and generate the corresponding binary mask image;

[0101] Step 9: Optimize CUTMIX using a binary mask image;

[0102] Step 10: Optimize CUTMIXUP using a binary mask image;

[0103] Step 11: Optimize CUTBLUR using a binary mask image;

[0104] Step 12: Combine the above methods to train the image super-resolution network;

[0105] Step 13: Use the trained image super-resolution network to perform image super-resolution reconstruction.

[0106] like Figure 2 As shown, step 1 specifically includes the following steps:

[0107] This step reads the input images and generates corresponding reference images for subsequent input into the pre-trained network for attribution analysis. First, based on the dataset used, input image samples for training. Here, we use DIV2K as an example. The DIV2K training dataset contains 800 high-resolution 2K images, and the validation dataset contains 100 high-resolution 2K images. The images in the training dataset are numbered 1-800, and the images in the validation dataset are numbered 801-900. Since there is no open test dataset for DIV2K, the common practice is not to perform validation on the validation dataset, but to use the validation dataset as the test dataset. The corresponding low-resolution images are obtained by bicubic interpolation (BICUBIC) downsampling of the high-resolution images, forming low-resolution-high-resolution data pairs. Before training begins, the 800 training data pairs are first read into memory, and then a batch of training images of size BATCH is randomly extracted each time. The batch size is usually set to 16. Next, a corresponding reference image is generated for all images in this batch. Here, we take any input image from the batch as an example. Let's take an example to illustrate. Step 1-1,

[0108] For the current input image Reference image Some missing features in the current input image I are satisfied. In high-level vision tasks, a pure black image (i.e., all pixel values are 0) is often used as a reference image, which is missing features such as objects in the input image. However, in low-level vision tasks, high-frequency information (edges, textures) has a more significant contribution to the model than low-frequency information (color, brightness). Therefore, here the high-frequency information (edges, textures) is taken as the missing feature, and the reference image is obtained by blurring the input image, thereby removing the high-frequency information therein. The operation is specifically to traverse the entire input image from left to right and from top to bottom using a blur kernel to obtain the corresponding blurred reference image Therefore, the reference image is generated in the manner that:

[0109]

[0110] where ω (σ) represents a Gaussian blur kernel function with a kernel width σ, is a convolution operation. There are various blur kernels in image super-resolution, and considering that Gaussian blur is a relatively common blur function, a Gaussian blur kernel is used here. The kernel width σ of the Gaussian blur kernel is set to 5 to control the blur intensity to be relatively moderate.

[0111] Although the reference image is directly generated in one step reflects the missing features in the input image, but such feature changes are too drastic and cannot reflect the missing situation of different features and analyze them. For example, the image directly blurred after the image, all textures and details are completely missing, and it is also impossible to analyze the effect of the missing part of the texture on the model. Therefore, it is necessary to gradually control the intensity of the blur kernel to obtain reference images with different blur degrees, reflecting the missing situation of different features. Therefore, the parameter a is used to gradually change the kernel width σ of the Gaussian blur kernel function, and the reference images with different blur degrees are generated smoothly

[0112]

[0113] where a ∈ [0, 1], and the corresponding and can be obtained Thus, when a = 0, the reference image generated is the most blurred image with all features, and when a = 1, the reference image generated is the least blurred image with features, i.e., the original input image. By controlling the parameter a, reference images with different blur degrees can be generated smoothly.

[0114] Step 2 specifically includes the following steps:

[0115] Sample from the parameter a in step 1, obtain the reference map in which the missing features are more reflected. Considering that reference maps of different blur degrees can reflect the missing features of the input map, but part of the adjacent reference maps are actually too similar, so the contribution to the missing features is small. Because too many steps will affect the efficiency of integration, here the parameter a is sampled, and only some important parameters are taken. Generally speaking, the reference map when a is close to 0 tends to reflect the missing features of the input map, and when a is larger and closer to 1, it is easy to fall into the problem of gradient saturation. Therefore, a truncated normal distribution is considered here to generate a partial normal distribution that meets the a parameter value, more sampling when a is small, and less sampling when a is large. The resulting reference map can not only accurately reflect the missing features, but also reduce the number of a sampling to speed up the calculation of integration. Therefore, a truncated normal distribution is randomly generated wherein is a random normal distribution parameter, a min ,a max is the lower limit and upper limit of truncation. Here, a min = 0, a max = 1 is set to meet the sampling range of a, The value of is a hyperparameter that needs to be adjusted constantly during the training process.

[0116] Then sample using the truncated normal distribution function, assuming that i reference map parameter points are sampled from it. First, set the relevant parameters, the lower limit and upper limit parameters a min ,a max and the distribution parameter Then sample using the truncated normal distribution function to generate reference map parameter points The reference map parameter points are still arranged from small to large, and according to the setting of the lower limit and upper limit a min ,a max of truncation, the range of all reference map parameter points is between 0 and 1.

[0117] Step 3 specifically includes the following steps:

[0118] Generate a reference map according to the reference map parameter points sampled. Apply the reference map generation method described in step 1 Use the reference map parameter points sampled in step 2 to generate a reference map for input

[0119]

[0120] wherein, is an input image, and ω(σ) represents a Gaussian blur kernel function with a kernel width of σ. is a convolution operation. Through this operation, k reference maps can be obtained, which respectively represent the missing features of different input maps.

[0121] Step 4 specifically includes the following steps:

[0122] All k reference maps are input into the pre-trained network, and the pre-trained network is used for super-resolution to output the high-resolution images corresponding to the reference maps. Here, the i-th reference map parameter point is used The i-th reference map is generated The method is as follows:

[0123]

[0124] wherein, is a pre-trained image super-resolution network, is a generated high-resolution reference map, wherein the superscript ^ represents that the image is a high-resolution image. The pre-trained model used here is consistent with the model used in the training process. For example, if the model used in the training is EDSR, the pre-trained model used here should also use EDSR. The super-resolution up-sampling multiple is also consistent with the up-sampling multiple used in the training.

[0125] The pre-trained image super-resolution network mentioned is either Residual Channel Attention Network (RCAN) (reference: Zhang, Yulun, et al. "Image super-resolution using very deep residual channel attention networks." Proceedings of the European conference on computer vision (ECCV). 2018.), Enhanced Deep Residual Network (EDSR) (reference: Lim, Bee, et al. "Enhanced deep residual networks for single image super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.), or SwinIR (Swinir: Image restoration using swin transformer). (reference: Liang, Jingyun, et al. "Swinir: Image restoration using swin transformer." Proceedings of the IEEE / CVF international conference on computer vision. 2021.).

[0126] Step 5 specifically includes the following steps:

[0127] Calculate the high-resolution reference image in step 4. The global features are obtained by traversing the entire image using the SOBEL operator (see: Kanopoulos, N., Vasanthavada, N. & Baker, RL, 1988. Design of an image edge detection filter using the Sobel operator. IEEE Journal of solid-state circuits, 23(2), pp. 358–367.). The SOBEL operator can make the change trend of global features between different reference images smoother, alleviate gradient saturation, and make the sampled points of the truncated normal distribution more accurate.

[0128] First, generate the SOBEL operator along the x-axis direction on the image. Perform calculations to obtain the image gradient along the x-axis.

[0129] Next, the SOBEL operator is generated along the y-axis direction for the image. Perform calculations to obtain the image gradient along the y-axis.

[0130] Using the gradient results along the x and y axes, calculate the approximate gradient value at each pixel:

[0131]

[0132] in and These are the gradients calculated by the SOBEL operator along the x-axis and y-axis of the image, respectively. Includes reference images The overall gradient result for each pixel.

[0133] Next, the global features of the high-resolution reference image are calculated. Transform the gradients of all pixels in the entire image into a comprehensive feature value, and then calculate the second normal form of the gradients of all pixels:

[0134]

[0135] Among them, h0, w0 and h max ,w max These represent the minimum position (0,0) at the top left corner and the maximum position (max,max) at the bottom right corner of the pixel in the high-resolution reference image, respectively. max ,w max These are the height and width values ​​of the high-resolution reference image, respectively.

[0136] Step 6 specifically includes the following steps:

[0137] For the i-th sampling parameter a i The obtained reference image Calculate the corresponding global features respectively This global feature is then backpropagated in the pre-trained model to calculate the gradient magnitude with respect to the input reference map. Here, the i-th sampling parameter a is used as an example. i The obtained reference image For example, backpropagation is performed.

[0138] Calculate the backpropagation gradient of the global features of the i-th high-resolution reference map to the input reference map:

[0139]

[0140] in, This represents the i-th reference image. The size of the calculated backpropagation gradient value map is consistent with the size of the input reference image, where the value of each pixel reflects the contribution of the corresponding reference image pixel to the model.

[0141] Step 7 specifically includes the following steps:

[0142] Integrate the product of the backpropagation gradients of all reference maps and the differences between adjacent reference maps. First, calculate the difference between the i-th and (i+1)-th adjacent reference maps in the input:

[0143]

[0144] Then calculate the product of the difference obtained in the previous step and the backpropagation gradient of the i-th reference image. The product of the difference between the i-th and (i+1)-th adjacent reference images and the backpropagation gradient of the i-th reference image is:

[0145]

[0146] This yields a relatively accurate contribution value of the pixel in the i-th reference image to the model. To measure the contribution of all missing features in the input image, the pixel contribution values ​​of all reference images are then integrated to calculate the contribution of all pixels in the image.

[0147] The input image is obtained by integrating the product of the differences between all k neighboring reference images and the backpropagation gradient. Saliency plot

[0148]

[0149] in of This indicates that the size of the graph is the same as the low-resolution input image, and s indicates that the graph is the saliency graph of the input image.

[0150] like Figure 4 As shown, step 8 specifically includes the following steps:

[0151] Using the saliency map of the input image, a binary mask is generated for cropping the image patch. This binary mask should be the region with the highest saliency in the saliency map. Although summing the saliency values ​​of all pixels in this region would be more accurate, it requires traversing the entire saliency map repeatedly, resulting in excessive computational complexity. Therefore, the calculation is simplified by searching for the pixel with the highest saliency value in the saliency map as the center of the cropping patch, generating the corresponding binary mask.

[0152] The saliency map obtained in step 7 Find the location of the pixel with the highest saliency value (x).max ,y max ):

[0153]

[0154] Here, argmax represents the function that evaluates the pixel to obtain the pixel coordinates with the maximum saliency value. (x, y) represents the coordinate position on the saliency map. This represents the significance value located at coordinates (x, y) in the significance graph.

[0155] Next, a saliency plot with height H and width W is plotted. In the process, taking the pixel position with the maximum saliency value as the center, select a block of random size with height h and width w, and obtain the binary mask image M of this block:

[0156]

[0157] Where M∈{0,1} H×W This represents a binary mask with a value of 0 or 1, a height of H, and a width of W. The size of this block is a hyperparameter that needs to be adjusted during training; it is typically 0.3-0.7 times the size of the input image. max The x-coordinate of the pixel with the maximum significance value, y max The ordinate represents the pixel with the highest saliency value. The size of the binary mask M is the same as the input low-resolution image, where a pixel with a value of 1 represents a cropped pixel, and a pixel with a value of 0 represents an uncropped pixel.

[0158] like Figure 3 As shown, step 9 specifically includes the following steps:

[0159] The CUTMIX method for general data augmentation is optimized using the binary mask image obtained in step 8. The first image is randomly selected from a batched image of the input. Image pairs and the Image pairs Generate new augmented image pairs (low-resolution images) for training. and high-resolution images ).

[0160] First, starting from the... Low-resolution images of image pairs Above, use a binary mask image M to cut out image blocks and paste them into the low-resolution image of the j-th image pair. superior:

[0161]

[0162] Then from the first High-resolution images of image pairs Above, use a binary mask image M to cut out image patches and paste them into the high-resolution image of the j-th image pair. superior:

[0163]

[0164] Where ⊙ represents the Hadamard product operation pixel by pixel. and Indicates the image sequence number. The resulting new image is a comparison of the low-resolution image. and high-resolution images This refers to the augmented image pair, where the subscript cutmix indicates that the augmented image pair was generated by the optimized CUTMIX.

[0165] Step 10 specifically includes the following steps:

[0166] The CUTMIXUP method for general data augmentation is optimized using the binary mask image obtained in step 8. The first image is randomly selected from the input BATCH image. Image pairs and the Image pairs Generate new augmented image pairs (low-resolution images) for training. and high-resolution images ).

[0167] First, from the first The first low-resolution image and the first In each low-resolution image, image patches are cropped using a binary mask M, and then weighted based on coefficients λ to obtain the blended low-resolution image patches.

[0168] Then, from the first The high-resolution image and the first In a high-resolution image, image patches are cropped using a binary mask image M, and then weighted based on coefficient λ to obtain a blended high-resolution image patch.

[0169] Using the binary mask image M from the first A low-resolution image Remove the image patch from the top, and the remaining part is mixed with the resulting low-resolution image patch. Add them together to generate new low-resolution sample images for training.

[0170] Using the binary mask image M from the first a high resolution image the remaining part is added to the resulting high resolution mixed image block to generate a new training sample high resolution image

[0171] wherein, and denote the image index, the subscript cutmixup indicates that the augmented image pair is generated by the optimized CUTMIXUP.

[0172] Step 11 specifically comprises the following steps:

[0173] The binary mask map obtained in step 8 is used to optimize the general data augmentation CUTBLUR method. From the input BATCH of images, the first image pair is randomly extracted. A new augmented image pair (a high resolution image and a low resolution image ) for training is generated.

[0174] From the first low resolution image , the low resolution image block is cut out using the binary mask map M.

[0175] From the first high resolution image , the high resolution image block is cut out using the binary mask map M.

[0176] The cut-out low resolution image block is s times bicubic up-sampled (reference: R.C. Gonzalez and R.E. Woods, Digital Image Processing, 3rd ed. Pearson, 2007.) to a high resolution image block The cut-out high resolution image block is s times bicubic down-sampled to a low resolution image block

[0177] The binary mask map M is used to remove the image block from the first high resolution image , and the remaining part is added to the resulting high resolution mixed image block to generate a new training sample high resolution image

[0178]

[0179] Using the binary mask map M to remove the image blocks from the first low resolution image The remaining part is added to the resulting low resolution mixed image block to generate a new sample low resolution image for training

[0180]

[0181] where ⊙ represents the pixel-wise Hadamard product operation, and represents the image sequence number, represents that the augmented image is a high resolution image, represents that the augmented image is a low resolution image, represents that the augmented image block is generated by the optimized CUTBLUR method.

[0182] Step 12 specifically includes the following steps:

[0183] The methods of steps 10, 11 and 12 are included in a unified framework, and in each training cycle, any method has a probability p of being applied to the augmented input image. The augmented input image will be used together with the original input image to train the image super-resolution network.

[0184] Embodiment:

[0185] In one embodiment, the application of the method of the present application to the image is enlarged by 4 times of super-resolution effect as shown in Figure 5 The present embodiment includes the following parts:

[0186] Step 1 extracts batch data:

[0187] ​This step reads the input image and generates the corresponding reference map for subsequent input into the pre-trained network for attribution analysis. First, according to the data set used, the image samples for training are input. Here, DIV2K is taken as an example for illustration. The training data set of DIV2K contains 800 high-definition images of 2K resolution, and the verification data set contains 100 high-definition images of 2K resolution. The image serial numbers in the training data set are 1-800, and the image serial numbers in the verification data set are 801-900. Since the DIV2K data does not open a test data set, the general practice is not to verify on the verification data set, but to use the verification data set as the test data set. The corresponding low-resolution image is obtained by bicubic down-sampling the high-definition image, and is composed of a low-definition-high-definition data pair. Before training begins, first read the 800 training data pairs into memory, and then randomly extract a batch of training images from them each time. Here, the batch size is set to 16, that is, a batch of training images contains 16 low-definition-high-definition image pairs Next, from this batch of images, any low-definition image is selected The corresponding reference map is generated. Step 1-1,

[0188] For the current input image Reference map satisfies some missing features in the current input image I. In advanced visual tasks, a pure black image (i.e., all pixel values are 0) is often used as a reference map, and the pure black image lacks the features of objects in the input image. However, in low-level visual tasks, high-frequency information (edges, textures) has a more significant contribution to the model than low-frequency information (color, brightness). Therefore, here the high-frequency information (edges, textures) is taken as the missing feature, and the reference map is accordingly a blur operation on the input image, thereby removing the high-frequency information therein. The operation is specifically to traverse the entire input image from left to right and from top to bottom using a blur kernel to perform convolution operation to obtain the corresponding blurred reference map Therefore, the reference map is generated in the following manner:

[0189]

[0190] where ω(σ) represents a Gaussian blur kernel function with kernel width σ, is a convolution operation. In this embodiment, the kernel width σ of the Gaussian blur kernel is set to 5 to control the blur intensity to be relatively moderate.

[0191] Although the reference map is directly generated in one step The missing features in the input image are reflected, but the feature changes are too drastic and cannot reflect the missing of different features and analyze them. For example, after direct blurring, all textures and details are completely missing, and it is also impossible to analyze the effect of the missing part of the texture on the model. Therefore, the intensity of the blur kernel needs to be gradually controlled to obtain reference images with different blur levels to reflect the missing of different features. Therefore, the parameter a is used to gradually change the kernel width σ of the Gaussian blur kernel function to generate reference images with different blur levels

[0192]

[0193] where a ∈ [0, 1], and the corresponding and Thus, when a = 0, the generated reference image is the most blurred image When a = 1, the generated reference image is the least blurred image, i.e., the original input image By controlling the parameter a, reference images with different blur levels can be generated smoothly and gradually. The specific value of a in this example will be explained in step 2.

[0194] Step 2 specifically includes the following steps:

[0195] From the parameter a in step 1, the reference images that better reflect the missing features are obtained. Considering that reference images with different blur levels can all reflect the missing features of the input image, but some adjacent reference images are actually too similar and therefore contribute less to the missing features. Since too many steps will affect the efficiency of integration, here the parameter a is sampled and only some important parameters are taken. Generally, the reference image when a is close to 0 tends to better reflect the missing features of the input image, and when a is larger and closer to 1, it is easy to fall into the problem of gradient saturation. Therefore, a truncated normal distribution is considered to generate a partial normal distribution that satisfies the a parameter value, more sampling when a is small, and less sampling when a is large. The obtained reference image not only accurately reflects the missing features, but also reduces the number of a samples and speeds up the calculation of integration. Therefore, a truncated normal distribution is randomly generated where is a random normal distribution parameter, a min ,a max is the lower and upper limits of truncation. Here, a min = 0, a max = 1 to meet the sampling range of a, and the normal distribution parameter is set to to control more sampling when a is small and less sampling when a is large.

[0196] Then sample from the truncated normal distribution function, here 10 reference map parameter points are sampled. min = 0, a max = 1 and distribution parameters Then and use the truncated normal distribution function to sample from, to generate reference map parameter points All parameter points are The reference map parameter points are still arranged from small to large, according to the settings of the lower limit and upper limit a min = 0, a max = 1, all reference map parameter points range between 0 and 1.

[0197] Step 3 specifically includes the following steps:

[0198] Generate a reference map according to the sampled reference map parameter points. Apply the reference map generation method described in step 1 Use the reference map parameter points sampled in step 2 to generate the reference map applied to the input

[0199]

[0200] wherein, is the input image, ω(σ) represents a Gaussian blur kernel function with kernel width σ = 5, is a convolution operation, is the i-th reference map parameter point. Through this operation, 10 adjacent reference maps respectively represent different input graph missing features.

[0201] Step 4 specifically includes the following steps:

[0202] Input all 10 reference maps into the pre-trained enhanced deep residual network EDSR, use the pre-trained enhanced deep residual network EDSR for super resolution, and output 10 high-resolution images corresponding to the reference maps The i-th reference map parameter point is used to generate the i-th reference map The method is described as follows:

[0203]

[0204] wherein, is the pre-trained enhanced deep residual network EDSR, is the ith high-resolution reference image. The pre-trained model type used here is consistent with the model used in the training process, and the model used in the training is also an enhanced deep residual network (EDSR). The super-resolution upsampling factor is also consistent with the upsampling factor used in the training, which is 4 in this example.

[0205] Step 5 specifically includes the following steps:

[0206] The global features of the 10 high-resolution reference images in step 4 are calculated. Here, the SOBEL operator is used to traverse the entire image to obtain the global features. Using the SOBEL operator can make the trend of global feature changes between different reference images more gentle, alleviate gradient saturation, and make the sampled sample points of the truncated normal distribution more accurate. Here, any image in the 10 high-resolution reference images is taken as an example.

[0207] First, a 3x3 SOBEL operator is generated to calculate the image along the x-axis direction, obtaining the image gradient

[0208] Second, a 3x3 SOBEL operator is generated to calculate the image along the y-axis direction, obtaining the image gradient

[0209] Using the gradient results in the x-axis and y-axis directions, the approximate value of the comprehensive gradient at each pixel point is calculated:

[0210]

[0211] where and are the gradients calculated by the SOBEL operator along the x-axis and y-axis of the image, respectively. contains the comprehensive gradient results of each pixel point of the high-resolution reference image .

[0212] Next, the global feature of the high-resolution reference image is calculated The gradients of all pixels on the entire image are converted into a comprehensive feature value, and the gradients of all pixel points are squared:

[0213]

[0214] where h0, w0, and h max , w max represent the minimum position (0, 0) of the top left corner of the high-resolution reference image pixel and the maximum position (max, max) of the bottom right corner, respectively. h​max = 2040, w maz = 1440 are the height and width values of the high resolution reference image respectively.

[0215] Step 6 specifically includes the following steps:

[0216] The reference image obtained by the 10 sampling parameters The corresponding global feature is calculated respectively And the global feature is back propagated in the pre-trained model to calculate the gradient size of the input reference image. Here, the high resolution reference image obtained by the ith sampling parameter is taken as an example for back propagation.

[0217] The back propagation gradient value of the global feature of the ith high resolution reference image to the input reference image is calculated as:

[0218]

[0219] Wherein, The size of the calculated back propagation gradient value image is consistent with the size of the input reference image, wherein the value of each pixel point reflects the contribution of the corresponding reference image pixel to the model.

[0220] Step 7 specifically includes the following steps:

[0221] The product of the back propagation gradient of all reference images and the difference value of adjacent reference images is integrated. First, the difference value between the input ith and ith+1 adjacent reference images is calculated:

[0222]

[0223] Then, the product of the difference value obtained in the last step and the back propagation gradient of the ith reference image is calculated. The product of the difference value between the ith and ith+1 adjacent reference images and the back propagation gradient of the ith reference image is:

[0224]

[0225] Here, the relatively accurate contribution value size of the pixel on the ith reference image to the model is obtained. In order to measure the contribution of all missing features in the input image, the pixel contribution values of all reference images are integrated to obtain the contribution size of all pixels in the image.

[0226] The product of the difference value between all 10 adjacent low resolution reference images and the back propagation gradient is accumulated, and the saliency map of the input image is obtained by integration.

[0227]

[0228] wherein the indicates that the size of the map is equal to the low resolution input image, s indicates that the map is the saliency map of the input image.

[0229] As Figure 4 shown, step 8 specifically includes the following steps:

[0230] Using the saliency map of the input image, a binary mask map for the cropped image block is generated. The binary mask map should be the largest saliency region in the saliency map. Although it is more accurate to count the sum of saliency values of all pixels in the region, the computational complexity is too high to traverse the entire saliency map one by one. Therefore, the calculation is simplified, and the pixel with the largest saliency value in the saliency map is searched as the center of the cropped block, and the corresponding binary mask map is generated.

[0231] Using the saliency map obtained in step 7 , the position (x max , y max ) of the pixel with the largest saliency value is searched:

[0232]

[0233] wherein, argmax represents a function parameter function, and the pixel coordinates with the largest saliency value are obtained. (x, y) represents the coordinate position on the saliency map, and S(x, y) represents the saliency value of the saliency map at the coordinate (x, y).

[0234] Then, in the saliency map with a height of H and a width of W , a block with a random size of h height and w width is selected with the position of the pixel with the largest saliency value as the center, and a binary mask map M of the block is obtained:

[0235]

[0236] wherein, M ∈ {0, 1} H×W represents a value of 0 or 1, a binary mask map with a height of H and a width of W. The size of the block is selected as a hyperparameter that needs to be adjusted in the training process. In this example, the multiple of the super resolution is set to 4, so the height H of the saliency map is 510, the width W is 360, and the size of the block is set to 0.7 of the input size, so the height h of the block is 357, and the width w is 252. x max represents the maximum value of the horizontal coordinate, and y maxThis represents the maximum value of the ordinate. The size of the binary mask M is equivalent to the input low-resolution image, where a pixel with a value of 1 represents a cropped pixel, and a pixel with a value of 0 represents a non-cropped pixel.

[0237] like Figure 3 As shown, step 9 specifically includes the following steps:

[0238] The CUTMIX method for general data augmentation is optimized using the binary mask image obtained in step 8. The first image is randomly selected from a batch of input images. Image pairs and the Image pairs Generate new augmented image pairs (low-resolution images) for training. and high-resolution images In this example, the batch contains 16 images, therefore

[0239] First, starting from the... Low-resolution images of image pairs Above, use a binary mask image M to cut out image blocks and paste them into the low-resolution image of the j-th image pair. superior:

[0240]

[0241] Then from the first High-resolution images of image pairs Above, use the binary mask image M to cut out the image block and paste it to the first... High-resolution images of image pairs superior:

[0242]

[0243] Where ⊙ represents the Hadamard product operation pixel by pixel. and Indicates the image sequence number. The resulting new image is a comparison of the low-resolution image. and high-resolution images This refers to the augmented image pair, where the subscript cutmix indicates that the augmented image pair was generated by the optimized CUTMIX.

[0244] Step 10 specifically includes the following steps:

[0245] The CUTMIXUP method for general data augmentation is optimized using the binary mask image obtained in step 8. The first image is randomly selected from a batch of input images. Image pairs and the Image pairs generate new augmented image pairs (low resolution images and high resolution images ) for training.

[0246] First, from the first low resolution image and the second low resolution image, respectively, the image blocks are cut out using the binary mask map M and weighted based on the coefficient λ to obtain the mixed low resolution image blocks

[0247] Then, from the first high resolution image and the second high resolution image, respectively, the image blocks are cut out using the binary mask map M and weighted based on the coefficient λ to obtain the mixed high resolution image blocks The coefficient λ is randomly sampled from the beta distribution β(1.0, 1.0):

[0248]

[0249] The image blocks are removed from the first low resolution image using the binary mask map M, and the remaining part is added to the obtained low resolution mixed image blocks to generate a new sample low resolution image

[0250] The image blocks are removed from the first high resolution image using the binary mask map M, and the remaining part is added to the obtained high resolution mixed image blocks to generate a new sample high resolution image

[0251] wherein and denote the image number, and the subscript cutmixup indicates that the augmented image pair is generated by the optimized CUTMIXUP.

[0252] Step 11 specifically comprises the following steps:

[0253] The binary mask map obtained in step 8 is used to optimize the general data augmentation CUTBLUR method. From an input batch of images, the first image pair generate new augmented image pairs (high resolution images and low resolution images ).

[0254] From the first low resolution image , cut out the low resolution image patch using the binary mask map M

[0255] From the first high resolution image , cut out the high resolution image patch using the binary mask map M

[0256] Upsample the cut-out low resolution image patch by s = 4 times bicubic interpolation (BICUBIC) to a high resolution image patch Downsample the cut-out high resolution image patch by s = 4 times bicubic interpolation (BICUBIC) to a low resolution image patch

[0257] Remove the image patch from the first high resolution image using the binary mask map M, and add the remaining part to the resulting high resolution mixed image patch to generate a new sample high resolution image for training

[0258]

[0259] Remove the image patch from the first low resolution image using the binary mask map M, and add the remaining part to the resulting low resolution mixed image patch to generate a new sample low resolution image for training

[0260]

[0261] where ⊙ denotes the pixel-wise Hadamard product operation, and denote the image index, denote that the augmented image is a high resolution image, denote that the augmented image is a low resolution image, denote that the augmented image patch is generated by the optimized CUTBLUR method.

[0262] Step 12 specifically includes the following steps:

[0263] The steps 10, 11 and 12 are incorporated into a unified framework, in each training cycle, any method has a probability p = 1.0 to be applied to augment the input image. The augmented input image will be used together with the original input image to train the image super-resolution network. Finally, the trained image super-resolution network is used to calculate the super-resolution result of the input image.

[0264] In this example, the image super-resolution network in step 12 can use residual channel attention network RCAN (reference: Zhang, Yulun, et al. "Image super-resolution using very deep residual channel attention networks." Proceedings of the European conference on computer vision (ECCV). 2018.), enhanced deep residual network EDSR (reference: Lim, Bee, et al. "Enhanced deep residual networks for single image super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.) or sliding window attention image restoration network SwinIR (reference: Liang, Jingyun, et al. "Swinir: Image restoration using swin transformer." Proceedings of the IEEE / CVF international conference on computer vision. 2021.). The image super-resolution network used for training and the pre-trained image super-resolution network used for attribution analysis are consistent. In this example, the DIV2K dataset is used and trained for 700,000 steps. In each step, a batch of images (16 image pairs) are randomly selected from the dataset and corresponding augmented images are generated, which are input into the image super-resolution network for training. In this example, after training, the optimal network is selected and tested on six datasets, Set5, Set14, B100, Urban100, Manga109 and RealSR, to calculate the peak signal-to-noise ratio (PSNR) of the output image.

[0265] In this example, the results obtained by the enhanced deep residual network EDSR trained using the augmented data are compared with the results obtained by the original enhanced deep residual network EDSR network. The peak signal-to-noise ratio (PSNR) index of the six data sets is improved by 0.08, 0.12, 0.20, 0.18, 0.11, and 0.28, respectively (in the field of super-resolution, the value of PSNR changes in the order of 10 -2 magnitudes is a significant improvement. This shows the significance of the image super-resolution reconstruction method based on the attribution-oriented data augmentation.

[0266] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the invention content and part or all of the steps in each embodiment of the image super-resolution reconstruction method based on the attribution-oriented data augmentation when executed by the data processing unit. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0267] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be realized by means of a computer program and its corresponding general hardware platform. Based on this understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes a plurality of instructions for causing a device (which can be a personal computer, a server, a single-chip microcomputer, an MCU, or a network device) containing a data processing unit to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0268] The present application provides a method and approach for the image super-resolution reconstruction method based on the attribution-oriented data augmentation. There are many methods and approaches to implement the technical solutions, and the above description is only the preferred embodiment of the present application. It should be noted that for ordinary skilled persons in the technical field, without departing from the principles of the present application, some improvements and refinements can be made, which should also be considered as the protection scope of the present application. The components not explicitly described in the embodiments can be implemented by using existing technology.

Claims

1. An image super-resolution reconstruction method based on attribution-guided data augmentation, characterized in that, Includes the following steps: Step 1: Read the input image and generate a reference image sequence; Step 2: Input the reference image sequence into the pre-trained network for attribution analysis to generate a saliency map; Step 3: Using the saliency map, find the pixel with the largest saliency value and generate the corresponding binary mask image; Step 4: Optimize multiple data augmentation methods using binary mask images and then mix them to obtain an attribution-oriented data augmentation method. Step 5: Use the attribution-guided data augmentation method to obtain a training image set for training the image super-resolution network; Step 6: Use the trained image super-resolution network to perform image super-resolution reconstruction, thus completing the image super-resolution reconstruction based on attribution-oriented data augmentation; Specifically, generating the saliency map in step 2 includes: Step 2-1: Input the reference image sequence into the pre-trained network to generate a high-resolution reference image corresponding to each reference image in the reference image sequence; Step 2-2: Use a global feature detector to obtain global features of the high-resolution reference image; Steps 2-3: Use the global features of the high-resolution reference image to backpropagate the pre-trained network and calculate the backpropagation gradient of the corresponding reference image to the input. Steps 2-4: Calculate the difference between all reference maps and their adjacent reference maps, and use integral accumulation to calculate the sum of the products of the backpropagation gradients of all reference maps and the differences to obtain the saliency map; The specific method for generating a high-resolution reference image corresponding to each reference image in the reference image sequence as described in step 2-1 includes: in, It is a pre-trained image super-resolution network. This is the generated high-resolution reference image, where the superscript ↑ indicates that the image is a high-resolution image; It is the i-th reference image; The method for obtaining the global features of the high-resolution reference image as described in step 2-2 includes: Step 2-2-1: Generate the SOBEL operator along the x-axis direction of the high-resolution reference image. Perform calculations to obtain the image gradient along the x-axis. Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript x indicates that the gradient is calculated along the x-axis. Step 2-2-2: Generate the SOBEL operator along the y-axis direction on the high-resolution reference image. Perform calculations to obtain the image gradient along the y-axis. Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript y indicates that the gradient is calculated along the y-axis. Step 2-2-3: Calculate the approximate values ​​of the combined horizontal and vertical gradients at each pixel of the high-resolution reference image. The subscript xy indicates that the gradient is a combined approximation of the gradients in the x and y directions, as follows: Step 2-2-4: Calculate the global features of the high-resolution reference image. That is, to calculate the second normal form of the gradient of all pixels in the high-resolution reference image, as follows: Among them, h0, w0 and h max ,w max These represent the minimum and maximum positions of pixels in the high-resolution reference image, respectively. The subscripts 0 and max represent the specific coordinates of the minimum and maximum values, respectively. GD indicates that the gradient is a global gradient value. The calculation of the backpropagation gradient with respect to the corresponding reference graph in steps 2-3 includes the following specific methods: Calculate the reference graph sequence sequentially The backpropagation gradient values ​​of all reference graphs in the i-th reference graph. The method for calculating the backpropagation gradient value is as follows; Step 2-3-1, select the i-th reference image. Step 2-3-2: Calculate the global features of the i-th high-resolution reference image relative to the reference image. The backpropagation gradient values ​​are as follows: in, This represents the i-th reference image. This represents the i-th high-resolution reference image; The specific methods for obtaining the saliency map as described in steps 2-4 include: Step 2-4-1: Calculate the difference between the i-th and (i+1)-th adjacent reference images, as follows: Step 2-4-2: Calculate the product of the difference in step 2-4-1 and the backpropagation gradient in step 2-3-2. The product of the difference between the i-th and (i+1)-th adjacent reference maps and the backpropagation gradient is: Steps 2-4-3: Accumulate the product of the differences and backpropagation gradients between all k adjacent reference images, and integrate to obtain the input image. Saliency plot Specifically as follows: Wherein, the subscript 's' indicates that the graph is a saliency graph, and the superscript 's' indicates that the graph is a saliency graph. This indicates that the image size is a low-resolution image.

2. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 1, characterized in that, The generation of the reference graph sequence in step 1 specifically includes: Step 1-1: Generate the truncated normal distribution function and sample reference map parameter points that follow a normal distribution; Steps 1-2: Use the reference map parameter points to generate a reference map sequence.

3. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 2, characterized in that, The method for generating the truncated normal distribution function and sampling reference graph parameter points that follow a normal distribution, as described in step 1-1, includes: Step 1-1-1: Randomly generate a truncated normal distribution. This indicates that the function follows a random normal distribution, and the subscript t indicates that the function is a cutoff function. Let a be the parameter of the random normal distribution. min ,a max The lower and upper limits are the cutoff values, where min represents the minimum value and max represents the maximum value; Step 1-1-2: Set the lower limit parameter and upper limit parameter 'a' for truncation. min ,a max and distribution parameters Then from the truncated normal distribution function k parameters are randomly sampled from the data to generate k reference map parameter points. in This indicates that the parameter point is sampled from a truncated normal distribution, a k This represents the k-th parameter point, where the index k is the index of the parameter point. The specific method for generating the reference graph sequence for input as described in steps 1-2 includes: Using the k reference map parameter points described in step 1-1-2 Generate the corresponding reference image sequence γ indicates that this graph is a reference graph, and the subscript tb indicates that the reference graph is generated from parameter points sampled from a truncated normal distribution. The i-th reference graph parameter point is used. Generate the i-th reference image The method is as follows: Select reference graph parameter points but: in, The input image is ω(σ), which represents a Gaussian blur kernel function with a kernel width of σ. It is a convolution operation.

4. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 3, characterized in that, The specific method for generating the corresponding binary mask image in step 3 includes: Step 3-1: Use the significance plot obtained in Steps 2-4-3. Find the location of the pixel with the highest saliency value (x). max ,y max The details are as follows: Where (x, y) represents the coordinate position, and the subscript max indicates that the coordinate has the maximum significance value. Indicates that it was found The maximum value; Step 3-2, saliency plot with height H and width W In the process, taking the position of the pixel with the maximum saliency value as the center, a block of random size with height h and width w is selected, and the binary mask image M of this block is obtained, as follows: Where M∈{0,1} H×W A binary mask with a value of 0 or 1, a height of H, and a width of W.

5. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 4, characterized in that, Step 4 describes optimizing and combining multiple data augmentation methods using binary mask images. The specific methods include: Step 4-1: Randomly select the i-th image pair from the input training image data. and the j-th image pair The CUTMIX data augmentation method is optimized using binary mask images to generate new sample pairs for training, namely low-resolution images. and high-resolution images Specifically as follows: Where ⊙ represents the Hadamard product operation pixel by pixel. and This represents the input low-resolution image. and This represents the input high-resolution image, where the subscripts i and j represent the image index, and the superscript... and These indicate that the image is a low-resolution image and a high-resolution image, respectively. The subscript cutmix indicates that the augmented image pair was generated by the optimized CUTMIX method. Step 4-2, optimize the data augmentation method CUTMIXUP using a binary mask image, as follows: Step 4-2-1: Cut out the image blocks for pasting from both the low-resolution and high-resolution images: Where P represents an image patch, the subscript mix indicates that the image patch is a mixed image patch, and λ is a weighting coefficient; Step 4-2-2: Generate new sample pairs for training, i.e., low-resolution images. and high-resolution images Specifically as follows: Where i and j represent image numbers, and the superscript... and These indicate that the image is a low-resolution image and a high-resolution image, respectively. The subscript cutmixup indicates that the augmented image pair was generated by the optimized CUTMIXUP method. Step 4-3: Optimize the data augmentation method CUTBLUR using a binary mask image, as detailed below: Step 4-3-1: Cut out image patches from the low-resolution image and the high-resolution image respectively. and Step 4-3-2: Perform s-fold bicubic interpolation (BICUBIC) upsampling on the cropped low-resolution image patch to obtain... The high-resolution image patch was downsampled by s-fold bicubic interpolation (BICUBIC) to obtain... Step 4-3-3: Generate new sample pairs for training, i.e., high-resolution images. and low-resolution images Specifically as follows: Where ⊙ represents the pixel-by-pixel Hadamard product operation, i and j represent the image index, and the superscript... and The superscript LR(s×↑) indicates that the image is obtained by upsampling the low-resolution image by a factor of s using bicubic interpolation, and the superscript HR(s×↓) indicates that the image is obtained by downsampling the high-resolution image by a factor of s using bicubic interpolation. This indicates that the augmented image pair was generated by the optimized CUTBLUR method; Step 4-4: Combine the data augmentation methods described in steps 4-1 to 4-3, i.e., set a probability p to select the data augmentation method.

Images