Arbitrary scale image downsampling method based on meta-learning
Through a meta-learning-based method, combining jump connection networks and forward neural networks, dynamically adjusting the convolution kernel parameters, the problem that the existing technology cannot realize decimal image downsampling and single-scale limitation is solved, and arbitrary image is realized Scale downsampling and information retention.
Patent Information
- Application Number
- CN202111344541.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-11-12
AI Technical Summary
The existing learning-based image downsampling method cannot implement image downsampling tasks of decimal times, and can only handle downsampling tasks of single scales, and cannot adapt to display devices of different resolutions.
A meta-learning-based method is used to build an arbitrary-scale image downsampling model, combining jump connection networks and forward neural networks, using single-scale downsampling networks and bicubital interpolation methods, and predicting network dynamic adjustments through feature extraction networks and convolution kernel parameters Convolution kernel parameters, introducing space to depth and mask selection methods to achieve decimal-fold image downsampling.
It realizes downsampling of the image at any scale, retains feature information, adapts to display devices with different resolutions, and improves the image downsampling effect.
Smart Images

Figure CN114092328B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an arbitrary scale image down-sampling method based on meta-learning. Background Art
[0002] Image downsampling has different degrees of application in the fields of image compression, image relocation, etc. Existing image downsampling methods can be divided into interpolation-based methods, sub-pixel-based methods and learning-based methods in the spatial domain. Interpolation-based downsampling algorithms have high computational efficiency, are simple to use, and have a wide range of applications. However, the interpolation-based algorithm causes the low-resolution image obtained by downsampling to have different degrees of information loss, resulting in problems such as aliasing and blurring of the image in vision. The sub-pixel-based downsampling algorithm reduces the image resolution according to the specified method, which improves the apparent resolution of the image, but also brings about the problem of chromatic distortion. With the continuous development of deep learning and hardware technology, researchers use learning-based methods to reduce image resolution. However, most of the current learning-based image downsampling algorithms cannot achieve decimal multiple image downsampling tasks, but can only achieve integer multiple image downsampling tasks, and can only achieve single-scale downsampling tasks, that is, a network can only handle tasks with a fixed downsampling scale. For downsampling tasks of other integer multiple scales, only another set of network parameters can be trained to achieve the purpose, which makes these algorithms unable to adapt to display devices with different resolutions. Summary of the invention
[0003] The purpose of the present invention is to provide an arbitrary scale image downsampling method based on meta-learning to solve the technical problems existing in the existing learning-based image downsampling methods and to be able to downsample images to any scale.
[0004] To achieve the above object, the present invention provides the following solutions:
[0005] A method for downsampling an image of any scale based on meta-learning comprises the following steps:
[0006] An arbitrary scale image downsampling model is constructed based on meta-learning; the image downsampling model based on meta-learning includes a skip connection network and a forward neural network; the skip connection network includes a single-scale downsampling network and a bicubic interpolation method (bicubic); the forward neural network includes a feature extraction network and a convolution kernel parameter prediction network based on meta-learning; the outputs of the skip connection network and the forward neural network are element-wise added to obtain a low-resolution image;
[0007] The constructed image downsampling model is trained, and image downsampling is performed using the trained image downsampling model;
[0008] The single-scale downsampling network is a residual network; the skip connection part of the network uses a bicubic interpolation method to downsample the input high-resolution image, and the forward part uses a feature extraction network and a parameter prediction network to extract features and predict convolution kernel parameters of the input high-resolution image respectively, convolve the extracted feature map with the predicted convolution kernel, and add the output to the output of the skip connection part element by element;
[0009] The feature extraction network consists of 10 convolution kernels, all of which have a size of 3×3, a stride and padding of 1, but the first convolution kernel has 3 input channels and 64 output channels, and the other 9 convolution kernels have 64 input and output channels.
[0010] The convolution kernel parameter prediction network consists of two fully connected layers and a Relu activation layer, and dynamically predicts the convolution kernel parameters according to the specific input downsampling scale.
[0011] Preferably, the single-scale downsampling network and the forward neural network also include space-to-depth and mask selection methods. The space-to-depth method is used to perform channel and spatial transformations on the output of the feature extraction network; the mask selection method is used to screen the rows and columns of the output of the convolution of the feature extraction network with the dynamically predicted convolution kernel to complete the decimal image downsampling task.
[0012] Preferably, during the training process of the arbitrary-scale image downsampling model, it is necessary to first train a single-scale image downsampling network and then train the entire model.
[0013] Preferably, the specific method for training the arbitrary scale image downsampling model includes:
[0014] The acquired image dataset is divided into a training set and a test set. Before the training set images are input into the network, they are randomly cropped, rotated, and horizontally flipped.
[0015] The training set images are input into the arbitrary scale image downsampling model to train the arbitrary scale image downsampling model. During the training process, the loss function is used as the evaluation index of the arbitrary scale image downsampling model. The loss function L of the arbitrary scale image downsampling model is total The calculation of is as follows:
[0016] L total =a×L rec +b×L reg (1)
[0017]
[0018]
[0019] Among them, L reg represents the regularization loss of low-resolution image LR and target, L rec represents the reconstruction loss of high-resolution images HR and Rec_HR, L t represents the final loss of the model for downsampling images of any scale, and parameters a and b represent the loss function L respectively. reg With L rec The weight of
[0020] The test set is input into the trained arbitrary-scale image downsampling model to verify the effect of the arbitrary-scale image downsampling model.
[0021] The present invention discloses the following technical effects:
[0022] (1) The present invention introduces a space-to-depth method to transform the spatial size and number of channels of the feature map, compress the spatial size by integer multiples, and increase the number of channels accordingly, so that the feature map after processing has the same number of elements as the feature map before processing, and retains the information of the feature map before processing as much as possible. This allows the subsequent convolution operation to extract better feature information.
[0023] (2) The present invention designs a convolution kernel parameter prediction network to dynamically predict the convolution kernel parameters of different scale downsampling tasks. Predicting different convolution kernel parameters for each different downsampling scale enables our method to convolve with the convolution kernel at different downsampling scales, thereby extracting a feature map that better matches the downsampling scale and is richer in information.
[0024] (3) The present invention introduces a mask selection method to select integer multiple rows and columns of the feature map to obtain the final low-resolution image LR, thereby achieving fractional multiple image downsampling. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is the specific structure of the arbitrary-scale image downsampling model based on meta-learning in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] Reference Figure 1 As shown, this embodiment provides an arbitrary scale image downsampling method based on meta-learning, which specifically includes the following steps:
[0028] Based on meta-learning, an arbitrary scale image downsampling model is constructed; the image downsampling model includes a skip connection network and a forward neural network; the skip connection network includes a single-scale downsampling network and a bicubic interpolation method; the forward neural network includes a feature extraction network and a convolution kernel parameter prediction network based on meta-learning; the outputs of the skip connection network and the forward neural network are element-wise added to obtain a low-resolution image; the specific structure of the arbitrary scale image downsampling model based on meta-learning is as follows Figure 1 shown.
[0029] The constructed image downsampling model is trained, and image downsampling is performed using the trained image downsampling model;
[0030] The single-scale downsampling network is a residual network; the skip connection part of the network uses a bicubic interpolation method to downsample the input high-resolution image, and the forward part uses a feature extraction network and a parameter prediction network to extract features and predict convolution kernel parameters of the input high-resolution image respectively, convolve the extracted feature map with the predicted convolution kernel, and add the output to the output of the skip connection part element by element;
[0031] The feature extraction network consists of 10 convolution kernels, all of which have a size of 3×3, a stride and padding of 1, but the first convolution kernel has 3 input channels and 64 output channels, and the other 9 convolution kernels have 64 input and output channels.
[0032] The convolution kernel parameter prediction network consists of two fully connected layers and a Relu activation layer, and dynamically predicts the convolution kernel parameters according to the specific input downsampling scale.
[0033] In a specific application example, the skip connection network and the forward neural network also include a space to depth method and a mask selection method. The space to depth method maps the feature map extracted by the feature extraction network to a low-resolution scale space. Assuming that the input and output of the space to depth method are represented by input and output respectively, the element coordinate relationship between the two can be expressed by formula (4).
[0034] output(k,j,i)=input(k / n 2 , n×j+(k / n)%n, n×i+k%n) (4)
[0035] Among them, output(k, j, i) means the value of the element whose output is in channel k, whose horizontal coordinate is j, and whose vertical coordinate is i, and input(k / n 2, n×j+(k / n)%n, n×i+k%n) means the input is in channel k / n 2 , the horizontal axis is n×j+(k / n)%n, the vertical axis is the element value of n×i+k%n, and n is the space compression ratio between input and output.
[0036] The mask selection method filters the elements of the input feature map (the feature map extracted by the feature extraction network and the predicted convolution kernel are convolved) according to the Boolean mask matrix mask. When the value of an element in the mask is 0, the corresponding element in the input feature map is discarded. When the value of an element in the mask is 1, the corresponding element in the input feature map is retained. After all elements are filtered, the final output is obtained.
[0037] The calculation process of the Boolean mask matrix mask is as follows. First, design a matrix mask with the same spatial size as the input feature map, and initialize the element values to 0. Then, traverse the interval [0, W) with a step size of 1 (W is the width of the output of the arbitrary scale image downsampling model). For each traversed value i, multiply it with r (r is the spatial compression ratio between the input feature map of the mask selection method and the output of the arbitrary scale image downsampling model), and round down the result and map it back to the input feature map. Add 1 to any element in the matrix mask with the same column subscript as the result. Then, traverse the interval [0, H) with a step size of 1 (H is the height of the output of the arbitrary scale image downsampling model). For each traversed value i, multiply i with r, round down the result and map it back to the input feature map. Add 1 to any element in the matrix mask with the same row subscript as the result. Finally, compare the element values in the matrix mask with the integer 2, set the elements with values equal to 2 to 1, and set the elements with values not equal to 2 to 0, and obtain the final matrix yan.
[0038] S2, training the constructed arbitrary scale image downsampling model, and performing image downsampling through the trained arbitrary scale image downsampling model; specifically including:
[0039] In this embodiment, the DIV2K dataset is used to train the arbitrary scale image downsampling model, and the Set5, Set14, BSD100 and Urban100 datasets are used to verify the arbitrary scale image downsampling model.
[0040] First, the training set is processed to obtain a low-resolution image target map.
[0041] The DIV2K training set is processed with a trained 2x single-scale image downsampling network CRCNN1 to obtain the corresponding low-resolution image that is reduced by 2 times. Then, the low-resolution image is interpolated and downsampled using the bicubic interpolation method to obtain a low-resolution image with a downsampling scale range of (2, 3) and a step size of 0.1 corresponding to the original high-resolution image of DIV2K. In addition, a pre-trained 3x single-scale image downsampling network CRCNN2 is processed in the same way to obtain a low-resolution image with a downsampling scale range of (3, 4) and a step size of 0.1. Finally, we obtain a low-resolution image with a downsampling scale range of [2, 4) and a step size of 0.1 corresponding to the original high-resolution image of DIV2K.
[0042] Secondly, the training set sample data is input into the arbitrary scale image downsampling model to train the arbitrary scale image downsampling model;
[0043] In model training, the input training set images are randomly cropped to obtain 16 40×40 image blocks as a batch of image data, which are normalized and input into the model for training. The initial learning rate of the arbitrary scale image downsampling model is set to 10 -4 , and the Adam optimizer is used to optimize the model. In the training process of the downsampling model of any scale image, the most important indicator is the loss function.
[0044] The loss function refers to the error between the model output image and the real image. The greater the difference between the two, the greater the error. In this embodiment, since the DIV2K dataset only has the original lossless high-resolution image and no corresponding lossless low-resolution image, this paper uses the linear combination of the regularization loss of the low-resolution image LR output by the model and the real low-resolution image target and the reconstruction loss of the high-resolution image HR and Rec_HR (obtained by upsampling the low-resolution image LR output by the model using the bicubic interpolation method bicubic) as the final loss function of the model, as shown below.
[0045] L total =a×L rec +b×L reg (1)
[0046]
[0047]
[0048] Among them, L reg represents the regularization loss of low-resolution image LR and target, L rec represents the reconstruction loss of high-resolution images HR and Rec_HR, Lt represents the final loss of the model for downsampling images of any scale, and parameters a and b represent the loss function L respectively. reg With L rec The weight of .
[0049] The test set is input into the trained arbitrary-scale image downsampling model to verify the effect of the arbitrary-scale image downsampling model.
[0050] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.
Claims
1. A method for downsampling images of arbitrary scales based on meta-learning, characterized in that: The steps include: An arbitrary scale image downsampling model is constructed based on meta-learning; the image downsampling model includes a skip connection network and a feed-forward neural network; the skip connection network includes a single-scale downsampling network and a bicubic interpolation method; the feed-forward neural network includes a feature extraction network and a convolution kernel parameter prediction network based on meta-learning; the outputs of the skip connection network and the feed-forward neural network are element-wise added to obtain a low-resolution image; The constructed image downsampling model is trained, and image downsampling is performed using the trained image downsampling model; The single-scale downsampling network is a residual network; the skip connection part of the network uses a bicubic interpolation method to downsample the input high-resolution image, and the forward part uses a feature extraction network and a parameter prediction network to extract features and predict convolution kernel parameters of the input high-resolution image respectively, convolve the extracted feature map with the predicted convolution kernel, and add the output to the output of the skip connection part element by element; The feature extraction network consists of 10 convolution kernels, all of which have a size of 3×3, a stride and padding of 1, but the first convolution kernel has 3 input channels and 64 output channels, and the other 9 convolution kernels have 64 input and output channels. The convolution kernel parameter prediction network consists of two fully connected layers and a Relu activation layer, and dynamically predicts the convolution kernel parameters according to the specific input downsampling scale.
2. The method for downsampling an image of any scale based on meta-learning according to claim 1, wherein: The single-scale downsampling network and the forward neural network also include space-to-depth and mask selection methods. The space-to-depth method is used to perform channel and spatial transformations on the output of the feature extraction network; the mask selection method is used to perform row and column screening on the output of the convolution of the feature extraction network with the dynamically predicted convolution kernel, thereby completing the decimal multiple image downsampling task.
3. The method for downsampling an image of any scale based on meta-learning according to claim 1, characterized in that: During the training process of the arbitrary-scale image downsampling model, it is necessary to first train a single-scale image downsampling network and then train the entire model.
4. The method for downsampling an image of any scale based on meta-learning according to claim 1, characterized in that: The specific methods for training the downsampling model of arbitrary scale images include: The acquired image dataset is divided into a training set and a test set. Before the training set images are input into the network, they are randomly cropped, rotated, and horizontally flipped. The training set images are input into the arbitrary scale image downsampling model to train the arbitrary scale image downsampling model. During the training process, the loss function is used as the evaluation index of the arbitrary scale image downsampling model. The loss function L of the arbitrary scale image downsampling model is total The calculation of is as follows: L total =a×L rec +b×L reg (1) Among them, L reg represents the regularization loss of low-resolution image LR and target, L rec represents the reconstruction loss of high-resolution images HR and Rec_HR, L t represents the final loss of the model for downsampling images of any scale, and parameters a and b represent the loss function L respectively. reg With L rec The weight of The test set is input into the trained arbitrary-scale image downsampling model to verify the effect of the arbitrary-scale image downsampling model.
