Image super-resolution reconstruction method based on attribution-oriented data augmentation

Abstract

The invention discloses an image super-resolution reconstruction method based on attribution-oriented data augmentation, and the method comprises the steps: reading an input image, and generating a reference image sequence; inputting the reference graph sequence into a pre-training network for attribution analysis, and generating a saliency graph; using the saliency map to find the pixel with the maximum saliency value in the saliency map, and generating a corresponding binary mask map; respectively optimizing and mixing the plurality of data augmentation methods by using the binary mask graph to obtain an attribution-oriented data augmentation method; obtaining a training image set for training an image super-resolution network by using the attribution-oriented data augmentation method; and performing image super-resolution reconstruction by using the trained image super-resolution network. The image super-resolution network data augmentation method is universal and efficient, super-resolution network performance can be effectively optimized, and the image super-resolution network data augmentation method has high use value.

Description

Technical Field

[0001] The invention relates to an image super-resolution reconstruction method, in particular to an image super-resolution reconstruction method based on attribution-oriented data augmentation. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Data augmentation is one of the most practical and effective methods to enhance the performance and generalization ability of deep neural networks. In advanced visual tasks, such as image recognition and semantic segmentation, early common augmentation methods such as MIXUP and CUTMIX laid the foundation. In recent years, saliency-based augmentation methods have received a lot of attention and have shown superior performance in advanced visual tasks than common augmentation methods.

[0004] Although data augmentation methods have developed rapidly in high-level vision tasks, it is very challenging to directly transfer saliency-based augmentation methods to low-level vision tasks. These difficulties limit the current data augmentation methods in low-level vision tasks to common augmentation methods such as flip, rotate, MIXUP, CUTBLUR, and CUTMIB.

[0005] Ordinary data augmentation methods in low-level vision tasks such as image super-resolution face the bottleneck of information loss, which seriously damages the quality of reconstructed images. Ordinary data augmentation methods often generate flat and simple edge regions for augmented images, which contribute little to the image super-resolution task, resulting in severe information loss. In contrast, saliency augmentation methods can provide regions with more critical information for reconstruction. This drives the development of a new augmentation method that focuses on more meaningful regions in image super-resolution tasks. Providing more meaningful information for augmented images alleviates the information loss problem and helps the training of super-resolution networks.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] Purpose of the invention: The technical problem to be solved by the present invention is to provide an image super-resolution reconstruction method based on attribution-guided data augmentation in view of the shortcomings of the prior art.

[0008] In order to solve the above technical problems, the present invention discloses an image super-resolution reconstruction method based on attribution-guided data augmentation, comprising the following steps:

[0009] Step 1: read the input image and generate a reference image sequence;

[0010] Step 2, inputting the reference image sequence into a pre-trained network for attribution analysis to generate a saliency map;

[0011] Step 3: Use the saliency map to find the pixel with the maximum saliency value and generate the corresponding binary mask map;

[0012] Step 4, using the binary mask image to optimize multiple data augmentation methods respectively and mix them to obtain an attribution-oriented data augmentation method;

[0013] Step 5, using the attribution-guided data augmentation method to obtain a training image set for training an image super-resolution network;

[0014] Step 6: Use the trained image super-resolution network to perform image super-resolution reconstruction to complete the image super-resolution reconstruction based on attribution-guided data augmentation.

[0015] Furthermore, the step of generating a reference image sequence in step 1 specifically includes:

[0016] Step 1-1, generate a truncated normal distribution function, and sample to obtain reference graph parameter points that obey the normal distribution;

[0017] Step 1-2, using the reference image parameter points to generate a reference image sequence.

[0018] Furthermore, the truncated normal distribution function is generated in step 1-1, and the reference graph parameter points that obey the normal distribution are obtained by sampling. The specific method includes:

[0019] Step 1-1-1, randomly generate truncated normal distribution Subscript Indicates that the function is a truncated function, where is the random normal distribution parameter, a min ,a max The lower and upper limits of the truncation, the subscript min indicates the minimum value, and max indicates the maximum value;

[0020] Step 1-1-2, set the lower limit parameter and upper limit parameter a of the truncation min ,a max , and the distribution parameters Then from the truncated normal distribution function Randomly sample k parameters from the graph to generate k reference graph parameter points

[0021] The method for generating a reference image sequence for input described in step 1-2 includes:

[0022] Use the k reference map parameter points described in step 1-1-2 Generate the corresponding reference image sequence Among them, the i-th reference graph parameter point is used Generate the i-th reference image The method is as follows:

[0023] Select reference image parameter points but:

[0024]

[0025] in, is the input image, ω(σ) represents the Gaussian blur kernel function with kernel width σ, is a convolution operation.

[0026] Furthermore, the generating of the saliency map described in step 2 specifically includes:

[0027] Step 2-1, inputting a reference image sequence into a pre-trained network to generate a high-resolution reference image corresponding to each reference image in the reference image sequence;

[0028] Step 2-2, using a global feature detector to obtain global features of a high-resolution reference image;

[0029] Step 2-3, use the global features of the high-resolution reference image to back-propagate the pre-trained network and calculate the back-propagation gradient of the corresponding reference image of the input;

[0030] Step 2-4, calculate the difference between all reference images and adjacent reference images, use integral accumulation to calculate the sum of the back-propagation gradients of all reference images and the product of the difference to obtain a saliency map.

[0031] Furthermore, the step 2-1 of generating a high-resolution reference image corresponding to each reference image in the reference image sequence comprises:

[0032]

[0033] in, is a pre-trained image super-resolution network, is the generated high-resolution reference image, where the superscript ↑ indicates that the image is a high-resolution image.

[0034] Furthermore, the specific method of obtaining the global features of the high-resolution reference image in step 2-2 includes:

[0035] Step 2-2-1, generate a SOBEL operator, and perform a multiplication of the high-resolution reference image along the x-axis direction. Calculate and obtain the image gradient in the x-axis direction Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript x indicates that the gradient is calculated along the x-axis direction;

[0036] Step 2-2-2, generate the SOBEL operator and perform a multiplication of the high-resolution reference image along the y-axis direction. Calculate and obtain the image gradient in the y-axis direction Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript y indicates that the gradient is calculated along the y-axis direction;

[0037] Step 2-2-3, calculating the approximate horizontal and vertical comprehensive gradients at each pixel of the high-resolution reference image The details are as follows:

[0038]

[0039] Step 2-2-4, calculate the global features of the high-resolution reference image That is, the gradient of all pixels in the high-resolution reference image is calculated in the second normal form, as follows:

[0040]

[0041] Among them, h0, w0 and h max ,w max They represent the minimum and maximum positions of the pixels in the high-resolution reference image respectively. The subscripts 0 and max represent the minimum and maximum specific coordinate values ​​respectively. GD indicates that the gradient is the global gradient value.

[0042] Furthermore, the calculation of the back propagation gradient of the corresponding reference image of the input described in step 2-3 includes:

[0043] Calculate the reference image sequence sequentially The back propagation gradient values ​​of all reference images in , where the i-th reference image The calculation method of the back propagation gradient value is as follows;

[0044] Step 2-3-1, select the i-th reference image

[0045] Step 2-3-2, calculate the global features of the i-th high-resolution reference image for the reference image The back propagation gradient value is as follows:

[0046]

[0047] in, represents the i-th reference image, represents the i-th high-resolution reference image.

[0048] Furthermore, the specific method of obtaining the saliency map described in steps 2-4 includes:

[0049] Step 2-4-1, calculate the difference between the input i-th and i+1-th adjacent reference images, as follows:

[0050]

[0051] Step 2-4-2, calculate the product of the difference in step 2-4-1 and the back-propagated gradient in step 2-3-2, the difference between the i-th and i+1-th adjacent reference images and the back-propagated gradient product is:

[0052]

[0053] Step 2-4-3, accumulate the difference between all k adjacent reference images and the product of the back-transferred gradient, and integrate to get the input image The saliency map of The details are as follows:

[0054]

[0055] The subscript s indicates that the map is a saliency map, and the superscript Indicates that the image size is a low-resolution image.

[0056] Furthermore, the method for generating the corresponding binary mask image in step 3 includes:

[0057] Step 3-1, use the saliency map obtained in step 2-4-3 Search for the position of the pixel with the maximum saliency value (x max ,y max ), as follows:

[0058]

[0059] Among them, (x, y) represents the coordinate position, and the subscript max indicates that the coordinate has the maximum significance value. Indicates found The maximum value of

[0060] Step 3-2, in the saliency map with a height of H and a width of W In , taking the position of the pixel with the maximum saliency value as the center, a random-sized block with a height of h and a width of w is selected and the binary mask map M of the block is obtained as follows:

[0061]

[0062] Where M∈{0,1} H×W Represents a binary mask with a value of 0 or 1, a height of H, and a width of W. .

[0063] Furthermore, the binary mask image described in step 4 is used to optimize and mix multiple data augmentation methods respectively, and the specific method includes:

[0064] Step 4-1: randomly extract the first Image pairs and Image pairs Use the binary mask map to optimize the data augmentation method CUTMIX to generate new sample pairs for training, i.e. low-resolution images and high-resolution images The details are as follows:

[0065]

[0066] Where ⊙ represents the pixel-by-pixel Hadamard product operation, and represents the input low-resolution image, and represents the input high-resolution image, subscript and Indicates the image number, superscript and They indicate that the image is a low-resolution and a high-resolution image respectively, and the subscript cutmix indicates that the augmented image pair is generated by the optimized CUTMIX;

[0067] Step 4-2, use the binary mask image to optimize the data augmentation method CUTMIXUP, as follows:

[0068] Step 4-2-1, cut the image blocks for pasting from the low-resolution image and the high-resolution image respectively:

[0069]

[0070] Wherein, P indicates that the image is an image block, the subscript mix indicates that the image block is a mixed image block, and λ is a weighting coefficient;

[0071] Step 4-2-2, generate new sample pairs for training, i.e. low-resolution images and high-resolution images The details are as follows:

[0072]

[0073] in, and Indicates the image number, superscript and They indicate that the image is a low-resolution and a high-resolution image respectively, and the subscript cutmixup indicates that the augmented image pair is generated by the optimized CUTMIXUP;

[0074] Step 4-3, use the binary mask image to optimize the data augmentation method CUTBLUR, as follows:

[0075] Step 4-3-1: Cut out image blocks from the low-resolution image and the high-resolution image respectively and

[0076]

[0077] Step 4-3-2: Perform s-fold bicubic interpolation BICUBIC upsampling on the cut low-resolution image block to obtain The cut high-resolution image block is downsampled by s times bicubic interpolation BICUBIC to obtain

[0078] Step 4-3-3, generate new sample pairs for training, i.e. high-resolution images and low-resolution images The details are as follows:

[0079]

[0080] Where ⊙ represents the pixel-by-pixel Hadamard product operation, and Indicates the image number, superscript and Respectively indicate that the image is a high-resolution image and a low-resolution image. Indicates that the augmented image pair is generated by the optimized CUTBLUR;

[0081] Step 4-4, mixing the data augmentation methods described in steps 4-1 to 4-3, that is, setting the probability p for selecting the data augmentation method.

[0082] Beneficial effects:

[0083] 1. The present invention uses an attribution analysis method to extract saliency maps, optimizes and fuses existing common data augmentation methods, and augments the input image of the training image super-resolution network. For any low-level visual task, as long as attribution analysis can be performed, the method proposed by the present invention can be used.

[0084] 2. The present invention operates entirely at the data level and can adapt to neural networks of any structure and design. In summary, the present invention has strong compatibility and can be applied to a variety of low-level visual tasks including image super-resolution, effectively improving network performance and generalization ability. Therefore, the image super-resolution reconstruction method based on attribution-guided data augmentation proposed in the present invention has high use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0086] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0087] Figure 2 Schematic diagram of the attribution analysis process.

[0088] Figure 3 Flowchart of data augmentation operations.

[0089] Figure 4 Generate a flow graph for binary masks.

[0090] Figure 5 It is a result schematic diagram in one embodiment. DETAILED DESCRIPTION

[0091] This paper proposes an image super-resolution reconstruction method based on attribution-guided data augmentation, the core of which is a set of data augmentation methods based on attribution analysis suitable for low-level visual tasks. This method operates completely at the data level and is compatible with neural networks of any structure, augmenting models of different structures and types, and has strong compatibility.

[0092] The present invention discloses an image super-resolution reconstruction method based on attribution-guided data augmentation, such as Figure 1 As shown, the following steps are included:

[0093] Step 1: Generate a reference graph for input into the pre-trained network for attribution analysis;

[0094] Step 2, generate a truncated normal distribution function, and sample to obtain reference graph parameter points that obey the normal distribution;

[0095] Step 3, using the sampled reference image parameter points to generate a reference image for input;

[0096] Step 4, input the sampled reference image into the pre-trained network to obtain a super-resolution image of the reference image;

[0097] Step 5, using a global feature detector to obtain global features of the high-resolution reference image;

[0098] Step 6, back-propagate the pre-trained network using the global features of the high-resolution reference image to calculate the gradient of the super-resolution network to the input reference image;

[0099] Step 7, calculate the difference between all adjacent reference images, and use integral accumulation to calculate the sum of all gradients multiplied by the difference;

[0100] Step 8, using the saliency map, find the pixel with the maximum saliency value and generate the corresponding binary mask map;

[0101] Step 9, optimize CUTMIX using the binary mask map;

[0102] Step 10, optimize CUTMIXUP using the binary mask image;

[0103] Step 11, optimize CUTBLUR using the binary mask image;

[0104] Step 12, mixing the above methods to train an image super-resolution network;

[0105] Step 13: Use the trained image super-resolution network to perform image super-resolution reconstruction.

[0106] like Figure 2 As shown, step 1 specifically includes the following steps:

[0107] This step reads the input image and generates the corresponding reference image for subsequent input into the pre-trained network for attribution analysis. First, input the image sample for training according to the dataset used. Here, DIV2K is used as an example. The training dataset of DIV2K contains 800 high-definition images with 2K resolution, and the verification dataset contains 100 high-definition images with 2K resolution. The image numbers in the training dataset are 1-800, and the image numbers in the verification dataset are 801-900. Since the DIV2K data does not have an open test dataset, the general practice is not to verify on the verification dataset, but to use the verification dataset as a test dataset. The corresponding low-resolution images are obtained by downsampling the high-definition images by bicubic interpolation (BICUBIC) and form low-definition-high-definition data pairs. Before the training starts, first read the 800 training data pairs into the memory, and randomly extract a training image set of BATCH size each time. The BATCH size is usually set to 16. Then generate corresponding reference images for all images in this BATCH. Here, any input image in the BATCH is Take this as an example. Step 1-1,

[0108] For the current input image Reference Image Satisfy some missing features in the current input image I. In high-level visual tasks, pure black images (i.e., all pixel values ​​are 0) are often used as reference images. Pure black images lack features such as objects in the input image. However, in low-level visual tasks, high-frequency information (edges, textures) contributes more significantly to the model than low-frequency information (color, brightness). Therefore, here high-frequency information (edges, textures) is used as missing features, and the reference image is blurred accordingly to remove the high-frequency information. This operation is specifically to use the blur kernel to traverse the entire input image from left to right and from top to bottom. Perform convolution operation to obtain the corresponding blurred reference image Therefore, the reference figure The generation method can be specifically expressed as:

[0109]

[0110] Among them, ω(σ) represents the Gaussian blur kernel function with kernel width σ, is a convolution operation. There are various blur kernels in image super-resolution. Considering that Gaussian blur is a more common blur function, the Gaussian blur kernel is used here. The kernel width of the Gaussian blur kernel is set to 5 to control the blur intensity to be relatively moderate.

[0111] Although the reference graph is generated directly in one step It reflects the missing features in the input image, but this feature change is too drastic to reflect the missing features and analyze them. For example, in a directly blurred image, all textures and details are completely missing, and it is impossible to analyze the effect of a certain part of the texture on the model when only a certain part of the texture is missing. Therefore, it is necessary to gradually control the intensity of the blur kernel to obtain reference images with different blur levels to reflect the missing features. Therefore, the parameter a is used to gradually change the kernel width σ of the Gaussian blur kernel function to smoothly generate reference images with different blur levels.

[0112]

[0113] where a∈[0,1], and we can get and Under such control, when a=0, the generated reference image is the one with the most blurred features, and when a=1, the generated reference image is the one with the least blurred features, that is, the original input image. By controlling the parameter a, reference images with different blur levels can be generated smoothly and gradually.

[0114] Step 2 specifically includes the following steps:

[0115] Sampling is performed from the parameter a in step 1 to obtain a reference image that can better reflect the missing features. Considering that reference images with different degrees of blur can reflect the missing features of the input image, some adjacent reference images are actually too close, so their contribution to the missing features is small. However, since too many steps will affect the efficiency of integration, the parameter a is sampled here, and only some of the more important parameters are taken. Generally speaking, the reference image is often more able to reflect the missing features of the input image when a is close to 0, and the larger a is and the closer it is to 1, the more likely it is to fall into the gradient saturation problem. Therefore, the use of truncated normal distribution is considered here to generate a partial normal distribution that satisfies the parameter value of a, with more sampling when a is smaller and less sampling when a is larger. The obtained reference image can not only more accurately reflect the missing features, but also reduce the number of sampled a, thereby accelerating the calculation of the integral. Therefore, a truncated normal distribution is randomly generated. in is the random normal distribution parameter, a min ,a max is the lower and upper limits of the truncation. Here we set a min =0,a max =1 to match the sampling range of a, The value of is a hyperparameter and needs to be adjusted during the training process.

[0116] Then, the truncated normal distribution function is used for sampling. Here, it is assumed that i reference image parameter points are sampled from it. First, set the relevant parameters, the lower limit and upper limit parameters a min ,a max With distribution parameters Then and using the truncated normal distribution function Sampling is performed from it to generate reference graph parameter points Reference diagram parameter points Still arrange from small to large, according to the lower limit and upper limit a of the truncation min ,a max Settings, all reference map parameter points The range is between 0 and 1.

[0117] Step 3 specifically includes the following steps:

[0118] Generate a reference image based on the sampled reference image parameter points. Apply the reference image generation method described in step 1 Use the reference image parameter points sampled in step 2 Generate reference graph for input

[0119]

[0120] in, is the input image, ω(σ) represents the Gaussian blur kernel function with kernel width σ, is a convolution operation. Through this operation, k reference images can be obtained, each representing a different missing feature of the input image.

[0121] Step 4 specifically includes the following steps:

[0122] Input all k reference images into the pre-trained network, use the pre-trained network for super-resolution, and output the high-resolution image corresponding to the reference image. Generate the i-th reference image The method is as follows:

[0123]

[0124] in, is a pre-trained image super-resolution network, is the generated high-resolution reference image, where the superscript ↑ indicates that the image is a high-resolution image. The type of pre-trained model used here is consistent with the model used in the training process. For example, the model used in training is EDSR, and the pre-trained model used here should also use EDSR. The upsampling multiple of the super-resolution is also consistent with the upsampling multiple used in training.

[0125] The pre-trained image super-resolution network is a residual channel attention network RCAN (reference: Zhang, Yulun, et al. "Image super-resolution using very deep residual channel attentionnetworks." Proceedings of the European conference on computer vision (ECCV). 2018.), an enhanced deep residual network EDSR (reference: Lim, Bee, et al. "Enhanced deep residualnetworks for single image super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.) or a sliding window attention image restoration network SwinIR (reference: Liang, Jingyun, et al. "Swinir: Imagerestoration using swin transformer." Proceedings of the IEEE / CVF international conference on computer vision. 2021.).

[0126] Step 5 specifically includes the following steps:

[0127] Calculate the high-resolution reference image in step 4 Here, the SOBEL operator (reference: Kanopoulos, N., Vasanthavada, N. & Baker, RL, 1988. Design of an image edge detection filter using the Sobel operator. IEEE Journal of solid-state circuits, 23 (2), pp. 358–367.) is used to traverse the entire image to obtain global features. The use of the SOBEL operator can make the change trend of global features between different reference images smoother, alleviate gradient saturation, and make the sample points sampled by the truncated normal distribution more accurate.

[0128] First, generate the SOBEL operator to calculate the image along the x-axis direction. Calculate and obtain the image gradient in the x-axis direction

[0129] Secondly, generate the SOBEL operator to calculate the image along the y-axis direction Calculate and obtain the image gradient in the y-axis direction

[0130] Using the gradient results on the x-axis and y-axis, calculate the approximate comprehensive gradient at each pixel:

[0131]

[0132] in and They are the gradients calculated by the SOBEL operator along the x-axis and y-axis of the image respectively. Reference images included The comprehensive gradient result of each pixel.

[0133] Then calculate the global features of the high-resolution reference image Convert the gradient of all pixels in the entire image into a comprehensive feature value, and find the second normal form of the gradient of all pixels:

[0134]

[0135]

[0136] Among them, h0, w0 and h max ,w max They represent the minimum position (0,0) of the upper left corner and the maximum position (max,max) of the lower right corner of the high-resolution reference image pixel. max ,w max They are the height and width of the high-resolution reference image, respectively.

[0137] Step 6 specifically includes the following steps:

[0138] For the i-th sampling parameter a i The reference image obtained Calculate the corresponding global features respectively The global feature is back-propagated in the pre-trained model to calculate the gradient of the input reference image. i The reference image obtained Take back propagation as an example.

[0139] Calculate the back-propagation gradient value of the global feature of the i-th high-resolution reference image to the input reference image:

[0140]

[0141] in, Represents the i-th reference image. The size of the calculated reverse gradient propagation value map is consistent with the size of the input reference image, where the value of each pixel reflects the contribution of the corresponding reference image pixel to the model.

[0142] Step 7 specifically includes the following steps:

[0143] Integrate the product of the back-propagated gradients of all reference images and the difference between adjacent reference images. First, calculate the difference between the i-th and i+1-th adjacent reference images of the input:

[0144]

[0145] Then calculate the product of the difference obtained in the previous step and the back-propagation gradient of the i-th reference image. Then the product of the difference between the i-th and i+1-th adjacent reference images and the back-propagation gradient of the i-th reference image is:

[0146]

[0147] Here we get a relatively accurate contribution value of the pixel on the i-th reference image to the model. In order to measure the contribution of all missing features in the input image, we need to integrate the pixel contribution values ​​of all reference images to find the contribution of all pixels in the image.

[0148] Accumulate the difference between all k adjacent reference images and the product of the back-propagated gradient, and integrate to get the input image The saliency map of

[0149]

[0150]

[0151] in of It indicates that the image size is equal to the low-resolution input image, and s indicates that the image is the saliency map of the input image.

[0152] like Figure 4 As shown, step 8 specifically includes the following steps:

[0153] The saliency map of the input image is used to generate a binary mask map for cutting the image block. The binary mask map should be the maximum saliency region in the saliency map. Although it is more accurate to calculate the sum of the saliency values ​​of all pixels in the region, it is necessary to traverse the entire saliency map one by one, and the computational complexity is too high. Therefore, the calculation is simplified, and the pixel with the maximum saliency value in the saliency map is searched as the center of the cut block to generate the corresponding binary mask map.

[0154] Use the saliency map obtained in step 7 Search for the position of the pixel with the maximum significance value (x max ,y max ):

[0155]

[0156] Among them, argmax represents the function parameterization function, which finds the pixel coordinates with the maximum saliency value. (x, y) represents the coordinate position on the saliency map. Represents the saliency value at coordinate (x,y) in the saliency map.

[0157] Next, in this saliency map with a height of H and a width of W In , taking the pixel position with the maximum saliency value as the center, select a random block with a height of h and a width of w and obtain the binary mask map M of the block:

[0158]

[0159] Where M∈{0,1} H×W The value is 0 or 1, the height is H, and the width is W. The size of this block is a hyperparameter that needs to be adjusted during training, usually 0.3-0.7 of the input image size. max represents the horizontal coordinate of the pixel with the maximum saliency value, y max Represents the ordinate of the pixel with the maximum saliency value. The size of the binary mask map M is equivalent to the input low-resolution map, where a pixel with a value of 1 represents the clipped pixel, and a pixel with a value of 0 represents the non-clipped pixel.

[0160] like Figure 3 As shown, step 9 specifically includes the following steps:

[0161] Use the binary mask image obtained in step 8 to optimize the common data augmentation CUTMIX method. Randomly extract the first Image pairs and Image pairs Generate new augmented image pairs for training (low-resolution images and high-resolution images ).

[0162] First, from Low-resolution images of image pairs On the left, use the binary mask image M to cut the image patch and paste it into the low-resolution image of the j-th image pair. superior:

[0163]

[0164] Then from High-resolution images of image pairs On the left, use the binary mask image M to cut the image patch and paste it to the high-resolution image of the j-th image pair superior:

[0165]

[0166] Where ⊙ represents the pixel-by-pixel Hadamard product operation, and Represents the image sequence number. The new image is the low-resolution image and high-resolution images That is, the augmented image pair, where the subscript cutmix indicates that the augmented image pair is generated by the optimized CUTMIX.

[0167] Step 10 specifically includes the following steps:

[0168] Use the binary mask image obtained in step 8 to optimize the common data augmentation CUTMIXUP method. Randomly extract the first Image pairs and Image pairs Generate new augmented image pairs for training (low-resolution images and high-resolution images ).

[0169] First, from the low-resolution images and In the low-resolution images, the binary mask map M is used to cut the image blocks respectively, and the mixed low-resolution image blocks are obtained by weighting based on the coefficient λ

[0170] Then, from the High-resolution images and In the high-resolution images, the binary mask map M is used to cut the image blocks respectively, and the mixed high-resolution image blocks are obtained by weighting based on the coefficient λ

[0171] Use the binary mask map M from the Low-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained low-resolution image block Add together to generate new sample low-resolution images for training

[0172] Use the binary mask map M from the High-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained high-resolution image block Add together to generate new sample high-resolution images for training

[0173] in, and Represents the image sequence number, and the subscript cutmixup indicates that the augmented image pair is generated by the optimized CUTMIXUP.

[0174] Step 11 specifically includes the following steps:

[0175] Use the binary mask image obtained in step 8 to optimize the common data augmentation CUTBLUR method. Randomly extract the first Image pairs Generate new augmented image pairs for training (high-resolution images and low-resolution images ).

[0176] From Low-resolution images In the example, the low-resolution image block is cut using the binary mask M.

[0177] From High-resolution images In the example, the binary mask M is used to cut out the high-resolution image patch.

[0178] The low-resolution image patch under cutting Perform s-fold bicubic interpolation (BICUBIC) upsampling (reference: RC Gonzalez and RE Woods, Digital Image Processing, 3rd ed. Pearson, 2007.) to a high-resolution image patch The high-resolution image patch under cutting Perform s-fold bicubic interpolation (BICUBIC) downsampling to obtain low-resolution image blocks

[0179] Use the binary mask map M from the High-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained high-resolution image block Add together to generate new sample high-resolution images for training

[0180]

[0181] Use the binary mask map M from the Low-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained low-resolution image block Add together to generate new sample low-resolution images for training

[0182]

[0183] Where ⊙ represents the pixel-by-pixel Hadamard product operation, and Indicates the image number, Indicates that the augmented image is a high-resolution image. Indicates that the augmented image is a low-resolution image. Indicates that the augmented image block is generated by the optimized CUTBLUR method.

[0184] Step 12 specifically includes the following steps:

[0185] The methods of steps 10, 11, and 12 are incorporated into a unified framework. In each training cycle, any method has a probability p of being applied to the augmented input image. The augmented input image will be used together with the original input image to train the image super-resolution network.

[0186] Example:

[0187] In one embodiment, the super-resolution effect of applying the method of the present invention to magnify the image by 4 times is as follows: Figure 5 This embodiment includes the following parts:

[0188] Step 1 Extract batch data:

[0189] This step reads the input image and generates the corresponding reference image for subsequent input into the pre-trained network for attribution analysis. First, according to the dataset used, input the image samples for training. Here we take DIV2K as an example. The training dataset of DIV2K contains 800 high-definition images with 2K resolution, and the verification dataset contains 100 high-definition images with 2K resolution. The image numbers in the training dataset are 1-800, and the image numbers in the verification dataset are 801-900. Since the DIV2K data does not have an open test dataset, the general practice is not to verify on the verification dataset, but to use the verification dataset as a test dataset. The corresponding low-resolution images are obtained by downsampling the high-definition images using bicubic interpolation (BICUBIC) and form low-definition-high-definition data pairs. Before the training starts, first read the 800 training data pairs into the memory, and then randomly extract a batch of training images from them each time. Here the batch size is set to 16, that is, a batch of training images contains 16 low-definition-high-definition image pairs. Then, from this batch of images, select any low-resolution image Generate the corresponding reference image. Step 1-1,

[0190] For the current input image Reference Image Satisfy some missing features in the current input image I. In high-level visual tasks, pure black images (i.e., all pixel values ​​are 0) are often used as reference images. Pure black images lack features such as objects in the input image. However, in low-level visual tasks, high-frequency information (edges, textures) contributes more significantly to the model than low-frequency information (color, brightness). Therefore, here high-frequency information (edges, textures) is used as missing features, and the reference image is blurred accordingly to remove the high-frequency information. This operation is specifically to use the blur kernel to traverse the entire input image from left to right and from top to bottom. Perform convolution operation to obtain the corresponding blurred reference image Therefore, the reference figure The generation method can be specifically expressed as:

[0191]

[0192] Among them, ω(σ) represents the Gaussian blur kernel function with kernel width σ, is a convolution operation. In this embodiment, the kernel width σ of the Gaussian blur kernel is set to 5 to control the blur intensity to be relatively moderate.

[0193] Although the reference graph is generated directly in one step It reflects the missing features in the input image, but this feature change is too drastic to reflect the missing features and analyze them. For example, in a directly blurred image, all textures and details are completely missing, and it is impossible to analyze the effect of a certain part of the texture on the model when only a certain part of the texture is missing. Therefore, it is necessary to gradually control the intensity of the blur kernel to obtain reference images with different blur levels to reflect the missing features. Therefore, the parameter a is used to gradually change the kernel width σ of the Gaussian blur kernel function to smoothly generate reference images with different blur levels.

[0194]

[0195] where a∈[0,1], and we can get and Under such control, when a=0, the generated reference image is the one that blurs all features to the maximum extent. When a=1, the generated reference image is the one with the minimum fuzzy features, that is, the original input image. By controlling the parameter a, reference images with different blur levels can be smoothly and gradually generated. The specific value of a in this example will be described in step 2.

[0196] Step 2 specifically includes the following steps:

[0197] Sampling is performed from the parameter a in step 1 to obtain a reference image that can better reflect the missing features. Considering that reference images with different degrees of blur can reflect the missing features of the input image, some adjacent reference images are actually too close, so their contribution to the missing features is small. However, since too many steps will affect the efficiency of integration, the parameter a is sampled here, and only some of the more important parameters are taken. Generally speaking, the reference image is often more able to reflect the missing features of the input image when a is close to 0, and the larger a is and the closer it is to 1, the more likely it is to fall into the gradient saturation problem. Therefore, the use of truncated normal distribution is considered here to generate a partial normal distribution that satisfies the parameter value of a, with more sampling when a is smaller and less sampling when a is larger. The obtained reference image can not only more accurately reflect the missing features, but also reduce the number of sampled a, thereby accelerating the calculation of the integral. Therefore, a truncated normal distribution is randomly generated. in is the random normal distribution parameter, a min ,a max is the lower and upper limits of the truncation. Here we set a min =0,a max = 1 to match the sampling range of a, and the normal distribution parameters are set to To control more sampling when a is small and less sampling when a is large.

[0198] Then use the truncated normal distribution function to sample, and here we sample 10 reference map parameter points. First set the relevant parameters, the lower limit and upper limit parameters a min =0,a max = 1 and distribution parameter Then and using the truncated normal distribution function Sampling is performed from it to generate reference graph parameter points All parameter points are The reference graph parameter points are still arranged from small to large, according to the lower and upper limits of the truncation a min =0,a max = 1, all reference map parameter points The range is between 0 and 1.

[0199] Step 3 specifically includes the following steps:

[0200] Generate a reference image based on the sampled reference image parameter points. Apply the reference image generation method described in step 1 Use the reference image parameter points sampled in step 2 Generate corresponding reference images for input

[0201]

[0202] in, is the input image, ω(σ) represents the Gaussian blur kernel function with kernel width σ, kernel width σ=5, is the convolution operation, is the parameter point of the i-th reference image. Through this operation, 10 adjacent reference images can be obtained. They represent different missing features of the input graph.

[0203] Step 4 specifically includes the following steps:

[0204] All 10 reference images are input into the pre-trained enhanced deep residual network EDSR, and the pre-trained enhanced deep residual network EDSR is used for super-resolution to output 10 high-resolution images corresponding to the reference images. Take the i-th reference image parameter point Generate the i-th reference image The method is described as an example:

[0205]

[0206] in, It is a pre-trained enhanced deep residual network EDSR, is the i-th high-resolution reference image. The pre-trained model type used here is consistent with the model used in the training process, and the model used in the training is also the enhanced deep residual network EDSR. The upsampling multiple of the super-resolution is also consistent with the upsampling multiple used in the training, which is 4 times in this example.

[0207] Step 5 specifically includes the following steps:

[0208] Calculate the 10 high-resolution reference images in step 4 Here, the SOBEL operator is used to traverse the entire image to obtain the global features. The use of the SOBEL operator can make the change trend of the global features between different reference images smoother, alleviate gradient saturation, and make the sample points sampled by the truncated normal distribution more accurate. Here, any image in the 10 high-resolution reference images is used. Provide explanation.

[0209] First, generate a SOBEL operator with a kernel size of 3×3 and apply it to the image along the x-axis. Calculate and obtain the image gradient in the x-axis direction

[0210] Next, generate a SOBEL operator with a kernel size of 3×3 and apply it to the image along the y-axis. Calculate and obtain the image gradient in the y-axis direction

[0211] Using the gradient results on the x-axis and y-axis, calculate the approximate comprehensive gradient at each pixel:

[0212]

[0213] in and They are the gradients calculated by the SOBEL operator along the x-axis and y-axis of the image respectively. High-resolution reference images included The comprehensive gradient result of each pixel.

[0214] Then calculate the global features of the high-resolution reference image Convert the gradient of all pixels in the entire image into a comprehensive feature value, and find the second normal form of the gradient of all pixels:

[0215]

[0216] Among them, h0, w0 and h max ,w max They represent the minimum position (0,0) of the upper left corner and the maximum position (max,max) of the lower right corner of the high-resolution reference image pixel.max =2040,w maz =1440 are the height and width values ​​of the high-resolution reference image respectively.

[0217] Step 6 specifically includes the following steps:

[0218] Reference graphs obtained for the 10 sampling parameters Calculate the corresponding global features respectively The global feature is back-propagated in the pre-trained model to calculate the gradient of the input reference image. Here, the high-resolution reference image obtained by the i-th sampling parameter Take back propagation as an example.

[0219] Calculate the back-propagation gradient value of the global feature of the i-th high-resolution reference image to the input reference image:

[0220]

[0221] in, Represents the low-resolution reference image of the i-th input. The size of the calculated reverse gradient propagation value map is consistent with the size of the input reference image, where the value of each pixel reflects the contribution of the corresponding reference image pixel to the model.

[0222] Step 7 specifically includes the following steps:

[0223] Integrate the product of the back-propagated gradients of all reference images and the difference between adjacent reference images. First, calculate the difference between the i-th and i+1-th adjacent reference images of the input:

[0224]

[0225] Then calculate the product of the difference obtained in the previous step and the back-propagation gradient of the i-th reference image. Then the product of the difference between the i-th and i+1-th adjacent reference images and the back-propagation gradient of the i-th reference image is:

[0226]

[0227] Here we get a relatively accurate contribution value of the pixel on the i-th reference image to the model. In order to measure the contribution of all missing features in the input image, we need to integrate the pixel contribution values ​​of all reference images to find the contribution of all pixels in the image.

[0228] Cumulatively collect all 10 adjacent low-resolution reference images The difference between them and the back-propagated gradient product are integrated to obtain the input image The saliency map of

[0229]

[0230] in of It indicates that the image size is equal to the low-resolution input image, and s indicates that the image is the saliency map of the input image.

[0231] like Figure 4 As shown, step 8 specifically includes the following steps:

[0232] The saliency map of the input image is used to generate a binary mask map for cutting the image block. The binary mask map should be the maximum saliency region in the saliency map. Although it is more accurate to calculate the sum of the saliency values ​​of all pixels in the region, it is necessary to traverse the entire saliency map one by one, and the computational complexity is too high. Therefore, the calculation is simplified, and the pixel with the maximum saliency value in the saliency map is searched as the center of the cut block to generate the corresponding binary mask map.

[0233] Use the saliency map obtained in step 7 Search for the position of the pixel with the maximum saliency value (x max ,y max ):

[0234]

[0235] Among them, argmax represents the function parameterization function, which finds the pixel coordinates with the maximum saliency value. (x, y) represents the coordinate position on the saliency map. Represents the saliency value at coordinate (x,y) in the saliency map.

[0236] Next, in this saliency map with a height of H and a width of W In , taking the position of the pixel with the maximum saliency value as the center, select a random block with a height of h and a width of w and obtain the binary mask map M of the block:

[0237]

[0238] Where M∈{0,1} H×W Represents a binary mask map with a value of 0 or 1, a height of H, and a width of W. The size of this block is a hyperparameter that needs to be adjusted during training. In this example, the super-resolution factor is set to 4, so the height of the saliency map is H = 510, the width is W = 360, and the block size is set to the input 0.7 of the size, so the block has a height of h = 357 and a width of w = 252. max Indicates the maximum value of the horizontal axis, y maxIndicates the maximum value of the ordinate. The size of the binary mask image M is equivalent to the input low-resolution image, where pixels with a value of 1 represent the clipped pixels, and pixels with a value of 0 represent the non-clipped pixels.

[0239] like Figure 3 As shown, step 9 specifically includes the following steps:

[0240] Use the binary mask obtained in step 8 to optimize the common data augmentation CUTMIX method. Randomly extract the first Image pairs and Image pairs Generate new augmented image pairs for training (low-resolution images and high-resolution images ). In this example, the number of images in a batch is 16, so

[0241] First, from Low-resolution images of image pairs On the left, use the binary mask image M to cut the image patch and paste it into the low-resolution image of the j-th image pair. superior:

[0242]

[0243] Then from High-resolution images of image pairs On the left, use the binary mask image M to cut the image block and paste it to the High-resolution images of image pairs superior:

[0244]

[0245] Where ⊙ represents the pixel-by-pixel Hadamard product operation, and Represents the image sequence number. The new image is the low-resolution image and high-resolution images That is, the augmented image pair, where the subscript cutmix indicates that the augmented image pair is generated by the optimized CUTMIX.

[0246] Step 10 specifically includes the following steps:

[0247] Use the binary mask obtained in step 8 to optimize the common data augmentation CUTMIXUP method. Randomly extract the first Image pairs and Image pairs Generate new augmented image pairs for training (low-resolution images and high-resolution images ).

[0248] First, from the low-resolution images and In the low-resolution images, the binary mask map M is used to cut the image blocks respectively, and the mixed low-resolution image blocks are obtained by weighting based on the coefficient λ

[0249] Then, from the High-resolution images and In the high-resolution images, the binary mask map M is used to cut the image blocks respectively, and the mixed high-resolution image blocks are obtained by weighting based on the coefficient λ The coefficient λ is randomly sampled from a Beta distribution β(1.0,1.0):

[0250]

[0251] Use the binary mask map M from the Low-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained low-resolution image block Add together to generate new sample low-resolution images for training

[0252] Use the binary mask map M from the High-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained high-resolution image block Add together to generate new sample high-resolution images for training

[0253] in, and Represents the image sequence number, and the subscript cutmixup indicates that the augmented image pair is generated by the optimized CUTMIXUP.

[0254] Step 11 specifically includes the following steps:

[0255] Use the binary mask obtained in step 8 to optimize the common data augmentation CUTBLUR method. Randomly extract the first Image pairs Generate new augmented image pairs for training (high-resolution images and low-resolution images ).

[0256] From Low-resolution images In the example, the low-resolution image block is cut using the binary mask M.

[0257] From High-resolution images In the example, the binary mask M is used to cut out the high-resolution image patch.

[0258] The low-resolution image patch under cutting Perform s=4 times bicubic interpolation (BICUBIC) upsampling to high-resolution image blocks The high-resolution image patch under cutting Perform s=4 times bicubic interpolation (BICUBIC) downsampling to obtain a low-resolution image block

[0259] Use the binary mask map M from the High-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained high-resolution image block Add together to generate new sample high-resolution images for training

[0260]

[0261] Use the binary mask map M from the Low-resolution images The image block is removed from the image, and the remaining part is mixed with the obtained low-resolution image block Add together to generate new sample low-resolution images for training

[0262]

[0263] Where ⊙ represents the pixel-by-pixel Hadamard product operation, and Indicates the image number, Indicates that the augmented image is a high-resolution image. Indicates that the augmented image is a low-resolution image. Indicates that the augmented image block is generated by the optimized CUTBLUR method.

[0264] Step 12 specifically includes the following steps:

[0265] The methods of steps 10, 11, and 12 are incorporated into a unified framework. In each training cycle, any method has a probability p = 1.0 of being applied to the augmented input image. The augmented input image will be used together with the original input image to train the image super-resolution network. Finally, the trained image super-resolution network is used to calculate the super-resolution result of the input image.

[0266] In this example, the image super-resolution network in step 12 can use the residual channel attention network RCAN (reference: Zhang, Yulun, et al. "Image super-resolution using very deep residual channel attention networks." Proceedings of the European conference on computer vision (ECCV). 2018.), the enhanced deep residual network EDSR (reference: Lim, Bee, et al. "Enhanced deep residual networks for single image super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.) or the sliding window attention image restoration network SwinIR (reference: Liang, Jingyun, et al. "Swinir: Image restoration using swin transformer." Proceedings of the IEEE / CVF international conference on computer vision. 2021.), and the image super-resolution network used for training is consistent with the pre-trained image super-resolution network used for attribution analysis. In this example, the DIV2K dataset is used and trained for 700,000 steps. At each step, a batch of images (16 image pairs) are randomly selected from the dataset and the corresponding augmented images are generated and input into the image super-resolution network for training. In this example, after the training is completed, the optimal network is selected and tested on six datasets: Set5, Set14, B100, Urban100, Manga109, and RealSR, and the peak signal-to-noise ratio (PSNR) of the output image is calculated.

[0267] In this example, the results obtained by using the enhanced deep residual network EDSR trained with augmented data are compared with the results obtained by the original enhanced deep residual network EDSR network. The peak signal-to-noise ratio (PSNR) of the six data sets is improved by 0.08, 0.12, 0.20, 0.18, 0.11, and 0.28 respectively (in the field of super-resolution, the value of PSNR is 10 -2 A change in magnitude is a significant improvement, which shows the significance of the image super-resolution reconstruction method based on attribution-guided data augmentation.

[0268] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of an image super-resolution reconstruction method based on attribution-guided data augmentation provided by the present invention and some or all steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0269] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0270] The present invention provides an idea and method for image super-resolution reconstruction based on attribution-guided data augmentation. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. An image super-resolution reconstruction method based on attribution-guided data augmentation, characterized in that: The following steps are involved: Step 1: read the input image and generate a reference image sequence; Step 2, inputting the reference image sequence into a pre-trained network for attribution analysis to generate a saliency map; Step 3: Use the saliency map to find the pixel with the maximum saliency value and generate the corresponding binary mask map; Step 4, using the binary mask image to optimize multiple data augmentation methods respectively and mix them to obtain an attribution-oriented data augmentation method; Step 5, using the attribution-guided data augmentation method to obtain a training image set for training an image super-resolution network; Step 6: Use the trained image super-resolution network to perform image super-resolution reconstruction to complete the image super-resolution reconstruction based on attribution-guided data augmentation.

2. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 1, characterized in that: The generation of the reference image sequence described in step 1 specifically includes: Step 1-1, generate a truncated normal distribution function, and sample to obtain reference graph parameter points that obey the normal distribution; Step 1-2, using the reference image parameter points to generate a reference image sequence.

3. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 2, characterized in that: The truncated normal distribution function is generated in step 1-1, and the reference graph parameter points that obey the normal distribution are obtained by sampling. The specific method includes: Step 1-1-1, randomly generate truncated normal distribution Indicates that the function follows a random normal distribution, the subscript Indicates that the function is a truncated function, where is the random normal distribution parameter, a min ,a max The lower and upper limits of the truncation, the subscript min indicates the minimum value, and max indicates the maximum value; Step 1-1-2, set the lower limit parameter and upper limit parameter a of the truncation min ,a max , and the distribution parameters Then from the truncated normal distribution function Randomly sample k parameters from the graph to generate k reference graph parameter points in Indicates that the parameter point is sampled from a truncated normal distribution, a k represents the kth parameter point, where the subscript k is the serial number of the parameter point; The method for generating a reference image sequence for input described in step 1-2 includes: Use the k reference map parameter points described in step 1-1-2 Generate the corresponding reference image sequence γ indicates that the image is a reference image, and the subscript tb indicates that the reference image is generated by sampling parameter points from a truncated normal distribution, where the i-th reference image parameter point is used. Generate the i-th reference image The method is as follows: Select reference image parameter points but: in, is the input image, ω(σ) represents the Gaussian blur kernel function with kernel width σ, is a convolution operation.

4. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 3, characterized in that: Generating a saliency map as described in step 2 specifically includes: Step 2-1, inputting a reference image sequence into a pre-trained network to generate a high-resolution reference image corresponding to each reference image in the reference image sequence; Step 2-2, using a global feature detector to obtain global features of the high-resolution reference image; Step 2-3, use the global features of the high-resolution reference image to back-propagate the pre-trained network and calculate the back-propagation gradient of the corresponding reference image of the input; Step 2-4, calculate the difference between all reference images and adjacent reference images, use integral accumulation to calculate the sum of the back-propagation gradients of all reference images and the product of the difference to obtain a saliency map.

5. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 4, characterized in that: The specific method of generating a high-resolution reference image corresponding to each reference image in the reference image sequence in step 2-1 includes: in, is a pre-trained image super-resolution network, is the generated high-resolution reference image, where the superscript ↑ indicates that the image is a high-resolution image.

6. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 5, characterized in that: The specific method of obtaining the global features of the high-resolution reference image described in step 2-2 includes: Step 2-2-1, generate a SOBEL operator, and perform a multiplication of the high-resolution reference image along the x-axis direction. Calculate and obtain the image gradient in the x-axis direction Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript x indicates that the gradient is calculated along the x-axis direction; Step 2-2-2, generate the SOBEL operator and perform a multiplication of the high-resolution reference image along the y-axis direction. Calculate and obtain the image gradient in the y-axis direction Where Sobel indicates that the gradient is generated by the SOBEL operator, and the subscript y indicates that the gradient is calculated along the y-axis direction; Step 2-2-3, calculating the approximate horizontal and vertical comprehensive gradients at each pixel of the high-resolution reference image The subscript xy indicates that the gradient is a comprehensive approximation of the gradient in the x direction and the gradient in the y direction, as follows: Step 2-2-4, calculate the global features of the high-resolution reference image That is, the gradient of all pixels in the high-resolution reference image is calculated in the second normal form, as follows: Among them, h0, w0 and h max ,w max They represent the minimum and maximum positions of the pixels in the high-resolution reference image respectively. The subscripts 0 and max represent the minimum and maximum specific coordinate values ​​respectively. GD indicates that the gradient is the global gradient value.

7. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 6, characterized in that: The calculation described in step 2-3 is to back propagate the gradient of the corresponding reference image of the input, and the specific method includes: Calculate the reference image sequence sequentially The back propagation gradient values ​​of all reference images in , where the i-th reference image The calculation method of the back propagation gradient value is as follows; Step 2-3-1, select the i-th reference image Step 2-3-2, calculate the global features of the i-th high-resolution reference image for the reference image The back propagation gradient value is as follows: in, represents the i-th reference image, represents the i-th high-resolution reference image.

8. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 7, characterized in that: The specific method of obtaining the saliency map described in step 2-4 includes: Step 2-4-1, calculate the difference between the input i-th and i+1-th adjacent reference images, as follows: Step 2-4-2, calculate the product of the difference in step 2-4-1 and the back-propagated gradient in step 2-3-2, the difference between the i-th and i+1-th adjacent reference images and the back-propagated gradient product is: Step 2-4-3, accumulate the difference between all k adjacent reference images and the product of the back-transferred gradient, and integrate to get the input image The saliency map of The details are as follows: The subscript s indicates that the map is a saliency map, and the superscript Indicates that the image size is a low-resolution image.

9. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 8, characterized in that: The method for generating the corresponding binary mask image in step 3 includes: Step 3-1, use the saliency map obtained in step 2-4-3 Search for the position of the pixel with the maximum significance value (x max ,y max ), as follows: Among them, (x, y) represents the coordinate position, and the subscript max indicates that the coordinate has the maximum significance value. Indicates found The maximum value of Step 3-2, in the saliency map with a height of H and a width of W In , taking the position of the pixel with the maximum saliency value as the center, a random-sized block with a height of h and a width of w is selected and the binary mask map M of the block is obtained as follows: Where M∈{0,1} H×W Represents a binary mask with a value of 0 or 1, a height of H, and a width of W.

10. The image super-resolution reconstruction method based on attribution-guided data augmentation according to claim 9, characterized in that: The binary mask map described in step 4 is used to optimize and mix multiple data augmentation methods separately, including: Step 4-1: randomly extract the first Image pairs and Image pairs Use the binary mask map to optimize the data augmentation method CUTMIX to generate new sample pairs for training, i.e. low-resolution images and high-resolution images The details are as follows: Where ⊙ represents the pixel-by-pixel Hadamard product operation, and represents the input low-resolution image, and represents the input high-resolution image, subscript and Indicates the image number, superscript and They indicate that the image is a low-resolution and a high-resolution image, respectively. The subscript cutmix indicates that the augmented image pair is generated by the optimized CUTMIX method. Step 4-2, use the binary mask image to optimize the data augmentation method CUTMIXUP, as follows: Step 4-2-1, cut the image blocks for pasting from the low-resolution image and the high-resolution image respectively: Wherein, P indicates that the image is an image block, the subscript mix indicates that the image block is a mixed image block, and λ is a weighting coefficient; Step 4-2-2, generate new sample pairs for training, i.e. low-resolution images and high-resolution images The details are as follows: in, and Indicates the image number, superscript and They indicate that the image is a low-resolution and a high-resolution image, respectively. The subscript cutmixup indicates that the augmented image pair is generated by the optimized CUTMIXUP method. Step 4-3, use the binary mask image to optimize the data augmentation method CUTBLUR, as follows: Step 4-3-1: Cut out image blocks from the low-resolution image and the high-resolution image respectively and Step 4-3-2: Perform s-fold bicubic interpolation BICUBIC upsampling on the cut low-resolution image block to obtain The cut high-resolution image block is downsampled by s times bicubic interpolation BICUBIC to obtain Step 4-3-3, generate new sample pairs for training, i.e. high-resolution images and low-resolution images The details are as follows: Where ⊙ represents the pixel-by-pixel Hadamard product operation, and Indicates the image number, superscript and They represent high-resolution and low-resolution images respectively. The superscript LR (s×↑) indicates that the image is obtained by upsampling the low-resolution image by s times bicubic interpolation. The superscript HR (s×↓) indicates that the image is obtained by downsampling the high-resolution image by s times bicubic interpolation. Indicates that the augmented image pair is generated by the optimized CUTBLUR method; Step 4-4, mixing the data augmentation methods described in steps 4-1 to 4-3, that is, setting the probability p for selecting the data augmentation method.

Images