Texture-guided gradient weight loss and attention mechanism super-resolution method

Abstract

The present invention discloses a texture-guided super-resolution method, which mainly solves the problem in the prior art that high-frequency details are seriously lost when generating high-resolution images from low-resolution images. The main steps are: (1) obtaining paired high-resolution and low-resolution image training data. (2) constructing an image super-resolution network, in particular using an attention mechanism to measure the weights between each channel to fuse features, training the generator network, and using gradient weight loss constraints to constrain the training in texture guidance. (3) using the generator trained in step (2), training a relative discriminator to fine-tune the generator network to obtain the final image super-resolution network model. (4) inputting a low-resolution image, passing through the image super-resolution model, and outputting a high-resolution generated image. The present invention can more accurately recover high-frequency detail information from low-resolution images, and can be used in target recognition, image classification and other fields.

Description

Technical Field

[0001] The present invention belongs to a super-resolution method in image processing technology, which can be classified as single-image super-resolution, and is used to improve the quality of images, and can be used in target recognition, image classification and other fields. Background Art

[0002] With the rapid development of image processing technology, more ultra-high-definition display devices have emerged, and people's demand for higher-resolution images or videos is growing. Image super-resolution is a task that aims to generate high-resolution images from low-resolution images. As a basic visual task, it has always attracted people's attention.

[0003] In recent years, super-resolution methods can be mainly divided into three types: interpolation-based methods, reconstruction-based methods, and learning-based methods. Among them, interpolation-based methods such as Bicubic and Lanczos interpolation methods have the advantages of being fast and direct, but interpolation-based methods will lose high-frequency detail information of the image, and the restored high-resolution image will be accompanied by unnatural details such as artifacts, and the image quality is poor; reconstruction-based methods use prior knowledge to limit the possible solution space to generate clearer details. However, as the magnification factor increases, the performance of reconstruction-based super-resolution methods will decrease, and the quality of the restored high-resolution image will also decrease. In addition, reconstruction-based methods usually take a lot of time and computational costs. Learning-based methods usually use machine learning algorithms to obtain nonlinear mapping models between low-resolution images and high-resolution images, such as Markov random fields, neighborhood embedding methods, sparse coding, random forests, and other methods. In recent years, compared with other super-resolution methods, learning-based methods have received widespread attention due to their excellent performance.

[0004] Super-resolution methods based on deep learning are a type of learning-based method. By using convolutional neural networks (CNNs) to obtain a nonlinear mapping relationship model between low-resolution images and high-resolution images, a clearer high-resolution image with higher image quality is generated. The pioneering work in applying deep learning to the field of super-resolution is the proposal of SRCNN. Using CNN networks to solve image super-resolution problems is more advantageous than traditional methods because CNN networks can learn richer features from a large number of data sets. After SRCNN was proposed, VDSR further deepened the depth of the network to solve the problem of single-image super-resolution. EDSR proposed to delete the batch normalization (BN) layer in the network because it will introduce a transformation of features, which may have an adverse effect on the final performance. However, the objective functions of these methods are mainly focused on minimizing the mean square reconstruction error, which results in the lack of high-frequency information in the generated super-resolution images. In order to solve this problem, a super-resolution method using a generative adversarial network (SRGAN) was proposed, which can reconstruct finer texture details to a certain extent. Although the scores of evaluation indicators such as peak signal-to-noise ratio and structural similarity are not high, these images are more visually acceptable.

[0005] Existing super-resolution methods do not take into account the texture relationship and aggregation of global features between low-resolution images and high-resolution images, and ignore the fact that low-resolution images may generate more detailed textures through texture guidance and aggregation of global features. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned prior art, the present invention proposes a super-resolution method based on texture-guided gradient weight loss and attention mechanism to improve the detail information of the generated image.

[0007] The technical solution to achieve the above purpose is:

[0008] A texture-guided gradient weight loss and attention mechanism super-resolution method is proposed. A CNN-based image super-resolution network model is constructed, including a generator for generating images and a discriminator for identifying the authenticity of the generated images. The generator extracts features from the input low-resolution image and performs nonlinear feature mapping. The attention mechanism is used to calculate the correlation between feature maps, and the weight of each feature map is redistributed. A high-resolution image is obtained through an image reconstruction module of the attention mechanism. The gradient weight loss constrains the training process in the texture-guided training process. The method specifically includes the following steps:

[0009] (1) Constructing a dataset: Obtain high-resolution images in the dataset, where low-resolution images are obtained by sampling high-resolution images, and obtain paired training data. The specific formula is as follows:

[0010] x=F(y)

[0011] F() represents the downsampling operation, x represents the low-resolution image, and y represents the high-resolution image;

[0012] (2) Construct an image super-resolution model and train the generator: The model consists of two branch networks: the generator and the discriminator, including: a shallow feature extraction module, a nonlinear mapping module, an image reconstruction module of the attention mechanism, and a discriminator module; the shallow feature extraction module consists of a convolutional layer, the nonlinear mapping module consists of a residual dense network, and the image reconstruction module of the attention mechanism consists of an attention module and an upsampling module; feature extraction and mapping are achieved through CNN, and finally the acquired features are weighted and reconstructed to obtain a high-resolution image;

[0013] Constructing the target equation: The generator uses the generative adversarial network loss L G And the gradient weight loss L gw Constraints, the specific formula is as follows:

[0014] L=L G +L gw

[0015] Among them, the specific formula for generating adversarial network loss is as follows:

[0016]

[0017] In the generative adversarial network loss, L1 = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image, λ represents the loss parameter, and L per is the perceptual loss, and the specific formula is:

[0018] L per =‖φ(y)-φ[f(x)]‖

[0019] Among them, φ(y), φ(f(x)) represent the features of the real image and the generated image extracted by CNN;

[0020] is the adversarial loss of the generator, and the specific formula is:

[0021]

[0022] is the discriminator, X r ,X fRepresent the real image and the generated image respectively, and the output is the probability of being identified as a real image; represents the expectation of the function distribution;

[0023] The specific formula of gradient weight loss is as follows:

[0024]

[0025] Where L1 = ||yf(x)||, f(x) represents the generated image, y represents the real image; where D gw =(1+αD x )(1+αD y ); D x ,D y represents the gradient difference map between the generated image and the high resolution in the horizontal and vertical directions, represents the gradient image of the generated image, Represents the gradient image of the real image, α is the weight coefficient in the loss function;

[0026] (3) Training the discriminator: After the initial training of the generator is completed, the discriminator is trained to fine-tune the generator. The relative discriminator is used instead of the standard discriminator to generate more realistic and better quality images. The following is the discriminator loss function:

[0027]

[0028] in, is the discriminator, which inputs the real image and the generated image, and outputs the probability of being judged as the real image. Represents the expectation of the distribution of a function.

[0029] (4) High-resolution image output: Input a low-resolution image, and output a high-resolution image through the trained image super-resolution network model. The formula is as follows:

[0030] Y=F SR (x)

[0031] Where x is the input low-resolution image, F SR () is the trained image super-resolution model, and Y is the output high-resolution image.

[0032] More specifically, the specific steps of step (2) are:

[0033] (2a) The input image passes through the shallow feature extraction module, i.e., two convolutional layers, to obtain the feature map m;

[0034] (2b) The feature map m is nonlinearly mapped through multiple residual dense network modules to obtain the feature map m1;

[0035] (2c) Construct an image reconstruction module with an attention mechanism and obtain the global features of the feature map u in the entire spatial feature through global pooling. The formula is as follows:

[0036]

[0037] Where H and W represent the height and width of the feature map, u c (i, j) is the eigenvalue of the feature map u corresponding to the position (i, j). After obtaining the global feature description of the feature map u, the relationship between the channels is obtained using the following formula:

[0038] z=F ex (z,W)=σ(g(z,W))=σ(W2ReLU(W1z))

[0039] in r is the dimension reduction coefficient, ReLU is the activation function, σ() is the Sigmoid activation function, and z is the final output scalar;

[0040] Finally, the activation value of each channel, that is, the weight, is multiplied by the original feature map u:

[0041]

[0042] where F scale (u c ,s c ) is the feature map u c and the scalar z c Corresponding channel product;

[0043] (2d) The feature map obtained in (2c) is passed through a convolution upsampling module to obtain a higher resolution result;

[0044] (2e) The generator network is trained, and the loss function uses the generative adversarial network loss and the gradient weight loss. The specific formula is as follows:

[0045] L=L G +L gw

[0046] Among them, the specific formula of generator loss is as follows:

[0047]

[0048] L1 in the generator loss = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image, and L peris the perceptual loss, λ is the loss parameter, and the loss is calculated using the feature map output by the trained VGG16 network relu3_3 on the ImageNet dataset. The specific formula is: per =‖φ(y)-φ[f(x)]‖, φ(y), φ(f(x)) represent the features of the real image and the generated image extracted by CNN, is the adversarial loss of the generator network, and the specific formula is:

[0049]

[0050] is the discriminator, X r ,X f is the probability of the real image and the generated image being judged as the real image. Represents the expectation of the function distribution, where the specific formula of the gradient weight loss is as follows:

[0051]

[0052] Where L1 = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image; where D gw =(1+αD x )(1+αD y ); D x ,D y represents the texture difference map between the generated image and the high resolution in the horizontal and vertical directions, represents the gradient image of the generated image, represents the gradient image of the real image, α is the weight coefficient in the loss function, which is 4 in the example of the present invention.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] In the process of generating a high-resolution image based on an input low-resolution image, more attention should be paid to the guiding role of texture. The introduction of the attention mechanism emphasizes the relationship between different channels of the feature map, and learns the correlation between channels. Since more attention is paid to the texture edges in the image, the image generated by the example of the present invention is more realistic, and its texture details are clearer, which effectively improves the visual perception quality of the generated image; and the gradient weight loss and attention mechanism module used in the example of the present invention have a low computational cost, but the effect improvement is more obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of the image super-resolution network of an example of the present invention;

[0056] Figure 2 A generator network structure diagram of an example of the present invention;

[0057] Figure 3 It is a residual dense network structure diagram of an example of the present invention;

[0058] Figure 4 A diagram showing the structure of a discriminator network according to an example of the present invention;

[0059] Figure 5 This is a comparison diagram of the effects of the examples of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in conjunction with the accompanying drawings. The specific implementation described herein is only used to more clearly illustrate the present invention and cannot be used to limit the protection scope of the present invention.

[0061] See also Figure 1 The specific implementation steps of the present invention are as follows:

[0062] Step 1. Build the dataset.

[0063] Input a high-resolution image y and a corresponding downsampled low-resolution image x respectively: In the example of the present invention, images in the DIV2K dataset are used, the input high-resolution image size is 128*128, and the input low-resolution image size is 32*32, where the low-resolution image is obtained by downsampling the high-resolution image to a low-resolution image using the Bicubic interpolation method to obtain paired training data. The specific formula is as follows:

[0064] x=F(y)

[0065] F() represents the downsampling operation, x represents the low-resolution image, and y represents the high-resolution image. Get paired training data.

[0066] Step 2. Build an image super-resolution network and train the generator.

[0067] An image super-resolution model is constructed, which consists of two branch networks, a generator and a discriminator, including: a shallow feature extraction module, a nonlinear mapping module, an image reconstruction module, and a discriminator module, wherein the nonlinear mapping module is composed of a residual dense network, and the image reconstruction module is composed of an attention module and an upsampling module. The image super-resolution network constructed by the present invention extracts and maps the features of low-resolution images and high-resolution images by constructing an image super-resolution network containing multiple convolutional network modules, and finally performs weighted fusion on the extracted features to obtain the final high-resolution image. A target equation is constructed to train the generator, and a gradient weight loss is additionally added to constrain the training process. The generator network structure is as follows Figure 2 As shown,

[0068] (2a) The low-resolution image undergoes shallow feature extraction, i.e., two layers of convolution to obtain a feature map m with a size of 32*32*32.

[0069] (2b) The feature map is nonlinearly mapped through the constructed nonlinear mapping module. The specific structure of the nonlinear mapping module is as follows: Figure 3 As shown. This module consists of multiple residual dense networks, the number of residual blocks is 16, the number of residual dense networks is 23, each residual dense network module is composed of three dense network modules after residual scaling fusion, the feature map is mapped from 32*32*32 to 64*32*32, the dense network module is composed of convolution layer, LeakyReLU activation layer, convolution layer, LeakyReLU activation layer, convolution layer, LeakyReLU activation layer, convolution layer, LeakyReLU activation layer, convolution layer, convolution layer, and convolution layer, and is built in a dense connection. Finally, the nonlinear mapping feature map m1 is obtained, the size is 64*32*32, and it is input to the image reconstruction module of the attention mechanism.

[0070] (2c) Obtain the global features of feature map u in the entire spatial features through global pooling:

[0071]

[0072] Where H and W represent the height and width of the feature map, u c (i,j) is the eigenvalue of the feature map u corresponding to the (i,j) position.

[0073] (2d) After obtaining the global feature description of feature map u, the relationship between channels is obtained using the following formula:

[0074] z=F ex (z,W)=σ(g(z,W))=σ(W2ReLU(W1z))

[0075] in r is the dimensionality reduction coefficient, ReLU is the activation function, σ() is the Sigmoid activation function, and z is the final output scalar.

[0076] (2e) Finally, the activation value (i.e., weight) of each channel is multiplied by the original feature map u:

[0077]

[0078] where F scale (u c ,s c ) is the feature map u c and the scalar z c Corresponding channel product.

[0079] (2f) After the feature map obtained in (2e) is upsampled by convolution, the size of the feature map becomes 64*128*128. After the final convolution to generate the image, the size changes from 64*128*128 to 3*128*128.

[0080] (2g) Use the training samples generated in step 1 and the Adam stochastic gradient descent algorithm to train the network. The loss function uses the generative adversarial network loss and the gradient weight loss. The specific formula is as follows:

[0081] L=L G +L gw

[0082] Among them, the specific formula of generator loss is as follows:

[0083]

[0084] L1 in the generator loss = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image, and L per is the perceptual loss, λ is the loss parameter, and the loss is calculated using the feature map output by the trained VGG16 network relu3_3 on the ImageNet dataset. The specific formula is: pre =‖φ(y)-φ[f(x)]‖, φ(y), φ(f(x)) represent the features of the real image and the generated image extracted by CNN, is the adversarial loss of the generator network, and the specific formula is:

[0085]

[0086] is the discriminator, X r ,X f is the probability of the real image and the generated image being judged as the real image. Represents the expectation of the distribution of a function.

[0087] Among them, the specific formula of gradient weight loss is as follows:

[0088]

[0089] Where L1 = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image; where D gw =(1+αD x )(1+αD y ); D x ,D y Represents the texture difference map between the generated image and the high resolution in the horizontal and vertical directions. represents the gradient image of the generated image, represents the gradient image of the real image, α is the weight coefficient in the loss function, which is 4 in the example of the present invention.

[0090] Step 3. Train the discriminator and fine-tune the generator.

[0091] Using the generator of the image super-resolution network trained in step 2, the discriminator is trained using the preliminarily trained generator, and the generator is fine-tuned to obtain the final image super-resolution network model. The discriminator network structure is as follows: Figure 4 As shown in the figure, it contains 8 convolutional layers. As the number of network layers increases, the number of features continues to increase, and the feature size continues to decrease. The activation function is selected as LeakyReLU. Finally, the probability of predicting a natural image is obtained through two fully connected layers and the final Sigmoid activation function, making the generated image closer to the real picture. The relative discriminator is used instead of the standard discriminator to generate more realistic and better quality images. The following is the discriminator network loss function:

[0092]

[0093] is the discriminator, X r ,X f It is the probability of the real image and the generated image, and the output is judged as the real image. Represents the expectation of the distribution of a function.

[0094] Step 4. Input a low-resolution image and output a high-resolution generated image.

[0095] Input a low-resolution image, and after the final image super-resolution network model obtained in step (3), output a high-resolution image. The specific formula is as follows:

[0096] Y=F SR (x)

[0097] Where x is the input low-resolution image, F SR () is the final image super-resolution model, and Y is the output high-resolution image.

[0098] In order to verify the effect of the present invention, the method of the present invention is compared with other existing image super-resolution network methods, and the evaluation indicators are as follows:

[0099] PI (perceptual index) is a comprehensive image perception quality evaluation standard proposed in the PIRM super-resolution competition. It is currently the mainstream super-resolution quality evaluation indicator. The specific formula is as follows:

[0100]

[0101] Among them, Ma uses statistics in the spatial domain and frequency domain as features to represent the image. Each set of extracted features is trained in a separate ensemble regression tree, and a linear regression model is used to predict the quality score from a large number of visual perception scores. The larger the Ma index, the better the visual effect of the image.

[0102] NIQE (natural image quality evaluator): Based on quality perception features, the larger the NIQE, the better the image quality. The smaller the perception index PI obtained by combining the two quality evaluation methods, the better the image quality and visual quality.

[0103] The results of the present invention and other super-resolution methods based on deep learning under the same training conditions are shown in the following examples: Figure 5 As shown, the high-resolution image generated by the present invention has a better restoration effect on texture details and a better perception index than other similar super-resolution methods.

Claims

1. A texture-guided gradient weight loss and attention mechanism super-resolution method, constructing a CNN-based image super-resolution network model, including a generator for generating images and a discriminator for identifying the authenticity of the generated images. The generator extracts features from the input low-resolution image and performs nonlinear feature mapping. The attention mechanism is used to calculate the correlation between feature maps, and the weight of each feature map is redistributed. A high-resolution image is obtained through an image reconstruction module of the attention mechanism. The gradient weight loss constrains the training process in the texture-guided way. The method specifically includes the following steps: (1) Constructing a dataset: Obtain high-resolution images in the dataset, where low-resolution images are obtained by sampling high-resolution images, and obtain paired training data. The specific formula is as follows: x=F(y) F() represents the downsampling operation, x represents the low-resolution image, and y represents the high-resolution image; (2) Build an image super-resolution model and train the generator: The model consists of two branch networks: the generator and the discriminator, including: Shallow feature extraction module, nonlinear mapping module, image reconstruction module of attention mechanism, and discriminator module; among them, the shallow feature extraction module is composed of convolutional layers, the nonlinear mapping module is composed of residual dense network, and the image reconstruction module of attention mechanism is composed of attention module and upsampling module; CNN is used to extract and map features, and finally the acquired features are weighted and reconstructed to obtain high-resolution images; Constructing the target equation: The generator uses the generative adversarial network loss L G And the gradient weight loss Constraints, the specific formula is as follows: Among them, the specific formula for generating adversarial network loss is as follows: In the generative adversarial network loss, L1 = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image, λ represents the loss parameter, and L per is the perceptual loss, and the specific formula is: L per =‖φ(y)-φ[f(x)]‖ Among them, φ(y), φ(f(x)) represent the features of the real image and the generated image extracted by CNN; is the adversarial loss of the generator, and the specific formula is: is the discriminator, X r ,X f Represent the real image and the generated image respectively, and the output is the probability of being identified as a real image; represents the expectation of the function distribution; The specific formula of gradient weight loss is as follows: Among them, L1=||yf(x)||, f(x) represents the generated image, y represents the real image; D x ,D y represents the gradient difference map between the generated image and the high resolution in the horizontal and vertical directions, represents the gradient image of the generated image, Represents the gradient image of the real image, α is the weight coefficient in the loss function; (3) Training the discriminator: After the initial training of the generator is completed, the discriminator is trained to fine-tune the generator. The relative discriminator is used instead of the standard discriminator to generate more realistic and better quality images. The following is the discriminator loss function: in, is the discriminator, which inputs the real image and the generated image, and outputs the probability of being judged as the real image. represents the expectation of the function distribution; (4) High-resolution image output: Input a low-resolution image, and output a high-resolution image through the trained image super-resolution network model. The formula is as follows: Y=F SR (x) Where x is the input low-resolution image, F SR () is the trained image super-resolution model, and Y is the output high-resolution image.

2. The texture-guided gradient weight loss and attention mechanism super-resolution method according to claim 1, characterized in that: The specific steps of step (2) are: (2a) The input image passes through the shallow feature extraction module, i.e., two convolutional layers, to obtain the feature map m; (2b) The feature map m is nonlinearly mapped through multiple residual dense network modules to obtain the feature map m1; (2c) Construct an image reconstruction module with an attention mechanism and obtain the global features of the feature map u in the entire spatial feature through global pooling. The formula is as follows: Where H and W represent the height and width of the feature map, u c (i, j) is the eigenvalue of the feature map u corresponding to the position (i, j). After obtaining the global feature description of the feature map u, the relationship between the channels is obtained using the following formula: from=F ex (z,W)=σ(g(z,W))=σ(W2ReLU(W1z)) in r is the dimension reduction coefficient, ReLU is the activation function, σ() is the Sigmoid activation function, and z is the final output scalar; Finally, the activation value of each channel, that is, the weight, is multiplied by the original feature map u: where F scale (u c ,s c ) is the feature map u c and the scalar z c Corresponding channel product; (2d) The feature map obtained in (2c) is passed through a convolution upsampling module to obtain a higher resolution result; (2e) The generator network is trained, and the loss function uses the generative adversarial network loss and the gradient weight loss. The specific formula is as follows: Among them, the specific formula of generator loss is as follows: L1 in the generator loss = ||yf(x)||, f(x) represents the generated image, y represents the high-resolution image, and L per is the perceptual loss, λ is the loss parameter, and the loss is calculated using the feature map output by the trained VGG16 network relu3_3 on the ImageNet dataset. The specific formula is: per =‖φ(y)-φ[f(x)]‖, φ(y), φ(f(x)) represent the features of the real image and the generated image extracted by CNN, is the adversarial loss of the generator network, and the specific formula is: is the discriminator, X r ,X f is the probability of the real image and the generated image being judged as the real image. represents the expectation of the function distribution, Among them, the specific formula of gradient weight loss is as follows: Where L1 = ||yf(x)||, f(x) represents the generated image, and y represents the high-resolution image; D x ,D y represents the texture difference map between the generated image and the high resolution in the horizontal and vertical directions, represents the gradient image of the generated image, Represents the gradient image of the true image, and α is the weight coefficient in the loss function.

Images