An image deblurring method based on local features and non-local features

Abstract

The present invention discloses an image deblurring method based on local features and non-local features, comprising the following steps: (1) building a generative adversarial network model and setting parameters; (2) setting image semantic content constraints, image structure reconstruction constraints and a discrimination target loss function; (3) introducing an IAM network module to predict clear image features and suppress fuzzy image features; (4) for a discriminant network, inputting a generated image and a label image into the generative network, and the discriminant network is required to discriminate the authenticity of the image content; (5) the discriminant network feeds back the discrimination result to the generative network, and the generative network updates the network parameters according to the feedback of the discriminant network and enters the next iterative training. The generative network and the discriminant network continuously compete for training in the manner described above until the network training converges; (7) loading the fuzzy image into the generative network whose training converges, thereby obtaining a generated image with a significant structure.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing and computer vision, and particularly relates to an image deblurring method based on local features and non-local features. Background Art

[0002] As a medium for recording and transmitting information about the external world, images have long been the primary source and means for humans to acquire and discern this information. However, image blur often occurs during image capture, often due to camera shake or object motion. Blurred images lose their clear structure and rich texture information, making it difficult to extract clear content and detailed information. Therefore, how to sharpen motion-blurred images for better application in advanced image processing (image detection, image recognition) and other fields has become a research hotspot.

[0003] Research on the problem of sharpening blurred images has focused on two main approaches: traditional methods and deep learning-based image deblurring methods. Traditional image deblurring methods rely on manually extracting priors or statistical information from the image, building an optimization equation based on this information, and iteratively solving the optimization equation to obtain the restored image. Because traditional methods only extract priors from a limited number of images, they only achieve good deblurring results on specific blurred images and have poor generalization to other blurred images. Furthermore, iteratively solving the optimization function is time-consuming, making these methods inadequate for real-time performance. Deep learning-based image deblurring extracts features from large datasets and iteratively selects weights that are more suitable for image restoration during network model training to restore the underlying labeled image. While some progress has been made in image deblurring, the restored images are often less than satisfactory. For example, some deep learning-based methods suffer from excessive network parameters and large network models, which undoubtedly place higher demands on the hardware configuration for network training. Other methods are only applicable to synthetic blurred images and have poor generalization and robustness to real-world blurred images. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology. By comprehensively utilizing the adversarial generative model and the structural attention mechanism, we study and provide an image deblurring method that can effectively reduce the size of the network model, meet the real-time processing requirements of image deblurring, and solve problems such as image texture and detail loss.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] An image deblurring method based on local features and non-local features comprises the following steps:

[0007] (1) Prepare data, including blurred images and labeled images; build a generative adversarial network model and set parameters;

[0008] (2) Setting up the network structure framework of the generative network and the discriminative network; the network structure framework of the generative network adopts a U-shaped network structure, which includes an encoder and a decoder. The encoder downsamples and extracts useful detail features in the image for encoding, and the decoder is used to upsample and decode the image; a dense expansion network is also set between the encoder and the decoder. The dense expansion network consists of a dense expansion module and an attention module, which are respectively used to learn the local features of the image and the prediction of the clear features of the image by the non-local features. A threshold mechanism is introduced in the attention module; specifically:

[0009] The blurred image is fed into the designed generative network. During the optimization training of the generative network, constraints are added to the semantic objective loss function and the structural objective loss function, so that the generated image has the same semantic and structural information as the labeled image. The discriminant network feeds back the discrimination results to the generative network, driving the generative network to generate a generated image with a relatively complete image structure. The generative network updates the parameters of the generative network based on the feedback from the discriminant network and enters the next iterative training. The generative network and the discriminant network continue to compete and train in the manner described above until the generative adversarial network training converges.

[0010] (3) Setting the target loss function of the generative adversarial network model: The target loss function in the generative network includes the image semantic target loss function L content , image structure reconstruction target loss function L gradient Among them, L content To ensure that the images before and after restoration can maintain the same semantic content, L gradient The generated image and the clear image are constrained to be consistent in image structure; the target loss function L in the discriminant network adv , L adv It is used to distinguish between generated images and labeled images, so as to improve the discriminant network's ability to distinguish between true and false images;

[0011] (4) The blurred image and the labeled image are fed into the generative adversarial network model, and the discriminant network is used to determine the consistency of the content between the generated image and the labeled image. The discriminant network feeds back the judgment result to the generative network, and the generative network and the discriminant network compete for learning until a Nash equilibrium is reached and the generative adversarial network model converges. The blurred image is loaded into the generative network that has converged after training, and finally a structurally complete generated image is obtained.

[0012] Furthermore, in step (3), the target loss function of the generative adversarial network model is weightedly expressed as:

[0013] L(G,D)=βL content +λL gradient +αL adv

[0014] Among them, β, λ and α are L content , L gradient and L adv The weight coefficient of each constraint item is as follows: β = 10, λ = 12, α = 1. The larger the value, the more important the corresponding part.

[0015] 10. The image deblurring method based on an improved attention mechanism according to claim 1, characterized in that, for the generator network, the encoder includes a convolutional layer for mapping the dimensions of the input image to 64×64; three downsampling layers are used to downsample and encode the image, and three residual blocks are added after the second and third downsampling layers, respectively, and the image resolution is reduced from 256×256 to 64×64; after the above operations are completed, a dense dilation network is introduced to learn the prediction of the image's sharp features using local and non-local features; correspondingly, the decoder includes three upsampling layers for upsampling and decoding the image, and three residual blocks are added before each upsampling layer, and the image resolution is reduced from 64×64 to 256×256; finally, the deblurred image is reconstructed by a Tanh layer and a convolutional layer;

[0016] For the discriminant network, a block-based generative adversarial network is adopted as the network structure of the discriminant network, which includes a flat convolution layer, three downsampling convolution layers, and a feature reconstruction block; the downsampling convolution layer is used to reduce the resolution of the input image and encode local features; each convolution layer is followed by a sample normalization layer and a leaky rectified linear unit activation function Leaky ReLU, and the convolution kernel size of each convolution layer is 4×4.

[0017] Furthermore, the dense expansion network is divided into three parts: a pre-feature conversion module, a deep feature extraction module, and a post-feature conversion module; the deep feature extraction module consists of 5 dense expansion modules, 1 attention module and 1 1×1 kernel convolution layer; the outputs of the 5 dense expansion modules and 1 attention module are element-wise summed, and the output features obtained by the summation are subjected to a 1×1 convolution operation to obtain the output features of the deep feature extraction module, and a residual connection is established between this feature and the input features of the deep feature extraction module; the dense expansion module is used to enhance the correlation of features and construct high-dimensional complex features, and the attention module is used to realize global feature fusion and residual learning.

[0018] Furthermore, the pre-feature conversion module acts as a buffer to map features to high-dimensional features. The pre-feature conversion module consists of two cascaded convolutional layers with a convolution kernel size of 3×3.

[0019] U -1 =FT1(U n-1 ),

[0020] U0=FT2(U -1 ),

[0021] Among them, FT1 and FT2 are the first and second convolutional layers in the front feature conversion module respectively, and feature U-1 is the output feature of convolutional layer FT1; on the one hand, feature U0 is passed as input to the deep feature extraction module; on the other hand, feature U0 establishes a global residual connection with the output of the deep feature extraction module;

[0022] The mathematical expression of the dense expansion module is:

[0023] U n =DDB n (U0),

[0024] Among them, U0 represents the input feature of the dense expansion module, DDB n Represents the nth dense expansion module. For each dense expansion module, the output of its mth convolutional layer Expressed as:

[0025]

[0026] in, represents the mth convolutional layer of the nth dense dilation module. The number of convolutional layers m is set to 4, and (·) represents the element-wise summation operation;

[0027] The attention module is used to focus on the contextual information between channel features. A threshold mechanism is introduced in the attention module. The front end of the attention module is divided into two convolutional modules, each of which consists of two convolutional layers.

[0028] The features processed by the dense expansion module are first input into the two-way convolution module in the attention module. After the input features pass through the two-way convolution modules, the features are fused through the summation operation.

[0029] A residual connection is established between the fused features and the input features of the attention module to promote gradient propagation and network optimization training;

[0030] The features of the residual connection are then passed through two sets of cascaded convolutional layers. A residual connection is established between each set of cascaded convolutional layers and the input of the cascaded convolutional layers to obtain continuously enhanced local features for network optimization and accelerated training.

[0031] The continuously enhanced local features are output as re-weighted non-local features through threshold control, and the continuously enhanced local features are multiplied with the re-weighted non-local features to obtain non-local features;

[0032] A global residual connection is constructed between the input features of the attention module and the non-local features; the feature representation after the global residual connection is passed as input to the subsequent post-feature conversion module;

[0033] The post-feature conversion module acts as a buffer to map features to high-dimensional features. The post-feature conversion module consists of two cascaded convolutional layers with convolution kernel sizes of 1×1 and 3×3 respectively. It uses a global feature fusion operation to fuse local features from the deep feature extraction module through the following operations;

[0034] U GF =GE(U1,U2,...,U n )

[0035] Among them, (·) represents the element-wise summation operation, GE represents the global feature fusion operation; finally, between GE and U -1 A global residual connection is established between them to enhance network optimization and gradient propagation.

[0036] Furthermore, the threshold mechanism works as follows:

[0037] 1) Use global average pooling operation GP to control each feature map u entering the threshold C ∈R H×W The dimension is compressed to obtain a global distribution scalar z describing the local feature channel C =GP(u C ), where H×W represents the size of the feature map, and the vector z=[z1,z2,...,z C ]∈R C Indicates the global distribution of each channel and can adaptively predict the importance of each feature channel; in order to utilize z C =GP(u C ), the second operation is performed below to fully capture the dependency of feature channels, learn the nonlinear interaction between feature channels, and be able to learn a non-exclusive relationship to ensure that the importance of multiple feature channels is emphasized.

[0038] 2) In order to limit the complexity of the model and promote the generalization of the model, the gating mechanism is parameterized. The specific operation is as follows: the global distribution scalar z describing the local feature channel obtained in step 1) is converted to C and Perform convolution operation to convert z C Compression is performed and the soft threshold function Activate and get the compressed feature F U ; Compressed feature F U and Perform convolution operation to restore feature F U to its original size and activated by the Sigmoid function, outputting the vector F IAM Each element of is used as a channel threshold to recalibrate the importance of each channel;

[0039]

[0040] Among them, α represents the Sigmoid activation function, ConvU represents a trainable upsampling weight matrix, represents the soft threshold, R represents the real number domain, C represents the feature channel, r represents the scaling ratio, Represents the feature dimension, ConvR represents the trainable downsampling weight matrix, and GP represents the global pooling operation;

[0041] 3) Feature map u entering threshold control C The local features and non-local features F IAM Perform dot multiplication on the features of the corresponding feature channel is the reweighted non-local feature.

[0042] Furthermore, the generative adversarial network model was run on a computer; the number of batch training images was 2, the learning rate of the generator network and the discriminator network was 0.0001; the slope of the activation function Leaky ReLU was 0.2; the generative adversarial network used the Adam optimizer, and its momentum parameters were β1 = 0.5 and β2 = 0.999, respectively.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the steps of the image deblurring method based on local features and non-local features are implemented.

[0044] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image deblurring method based on local features and non-local features are implemented.

[0045] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0046] 1. Compared with existing image deblurring algorithms, the restored images produced by this method have clearer structures and richer details. Compared with previous methods that rely on manual feature extraction or prior knowledge from limited images, the purely data-driven reconstruction approach based on the generative adversarial network in this invention avoids the cumulative errors caused by separately estimating model parameters, thereby improving the accuracy and robustness of the algorithm.

[0047] 2. The present invention proposes an image deblurring method based on local features and non-local features. Compared with existing methods, this method can generate deblurred aspects with significant structure and clear details.

[0048] 3. Dense connections enable different features to interact with different receptive fields. Compared with the original convolutional layer, the dense expansion module used in this invention can mine local features from multiple scales; in addition, the dense expansion module can also promote local feature fusion and residual learning.

[0049] 4. The attention module of the present invention focuses on the contextual information between channel features and introduces a threshold mechanism to promote the performance of image de-roughening. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the process framework of the method of the present invention;

[0051] Figure 2 To generate network layout and parameter graphs for the network;

[0052] Figure 3 This is a schematic diagram of the structure of the dense dilated block network (DDBNet);

[0053] Figure 4 This is the network structure diagram of the dense dilated block (DDB);

[0054] Figure 5 This is the network structure diagram of the Improved Attention Module (IAM);

[0055] Figure 6 Network layout and parameter diagram for the discriminant network. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] like Figure 1 As shown, this embodiment provides an image deblurring method based on local features and non-local features, which includes the following steps:

[0058] Step 1: Prepare data, including blurred images and label images; build a generative adversarial network model and set parameters; the label image is a clear image without blur degradation, and the generative adversarial network model includes two sub-networks: the generative network and the discriminative network;

[0059] Step 2: Construct the network structure of the generation network and the discrimination network; Figure 2 and Figure 6 As shown in the figure, the generative network is used to learn the image sharpening process, while the discriminative network discriminates and provides feedback on the deblurring results learned by the generative network.

[0060] (1) Constructing the network structure of the generated network G

[0061] This embodiment adopts a U-shaped network as the network structure of the generated network, such as Figure 2 As shown. The U-shaped network consists of two parts: the encoder and the decoder. The main function of the encoder stage is to spatially compress and encode the image, and the main function of the decoder stage is to construct the features of the image content and domain. In addition, a jump connection is established between the upsampling layer and the downsampling layer of the corresponding scale to associate the low-dimensional features learned in the encoder with the decoded features, guiding the recovery of the local and details of the image, which is critical for learning the image structure and details. The encoder part includes a convolutional layer to map the feature dimension to 64×64; three downsampling layers are used to downsample and encode the image, and the following are added after the second and third downsampling layers. Figure 2 The three residual blocks shown in the figure reduce the image resolution from 256×256 to 64×64. A dense dilated block network (DDBNet) structure is then introduced, as detailed in step (2).

[0062] Correspondingly, the decoder consists of three upsampling layers to upsample and decode the image. Three residual blocks are added before each upsampling layer, and the image resolution is reduced from 64×64 to 256×256. Finally, the deblurred image is reconstructed by a Tanh layer and a convolution layer with a 7×7 kernel.

[0063] (2) Build DDBNet;

[0064] DDBNet consists of three parts: a pre-feature conversion module, a deep feature extraction module, and a post-feature conversion module. The deep feature extraction module consists of five dense expansion modules, an improved attention module, and a 1×1 kernel convolutional layer. The DDB module can enhance feature correlation and construct high-dimensional complex features, and the IAM module can achieve global feature fusion and residual learning.

[0065] (201) The pre-feature conversion module acts as a buffer to map features to high-dimensional features. The pre-feature conversion module consists of two cascaded convolutional layers with a convolution kernel size of 3×3;

[0066] U -1 =FT1(U n-1 ),

[0067] U0=FT2(U -1 ),

[0068] Among them, FT1 and FT2 are the first and second convolution layers in the front feature conversion module respectively, and feature U-1 is the output feature of convolution layer FT1; on the one hand, feature U0 is passed as input to the deep feature extraction module; on the other hand, feature U0 establishes a global residual connection with the output of the deep feature extraction module; (202) The deep feature extraction module is as follows Figure 3 As shown in Figure 2, it mainly consists of a dense dilated block (DDB) and an improved attention module (IAM).

[0069] It consists of two parts. It should be noted that there are three differences between the deep feature extraction module and DenseNet (i.e., Densely Connected Convolutional Networks, see https: / / arxiv.org / abs / 1608.06993 for details): 1) Each part of the convolution layer is increased layer by layer; 2) After the 5 convolution layers, an improved attention module is added; 3) The first 4 flat convolution layers are replaced by dilated convolution layers with dilation rates of 1, 2, 3, and 3 respectively. For the current dilated convolution layer, the receptive field of all previous convolution layers is RF n-1 , the convolution kernel size of the current convolution layer is f k , the expansion rate is D, and the receptive field of the current convolutional layer can be expressed as RF n =RF n-1 +(((f k -1)×D+1)-1); the structure of the DDB module is as follows Figure 4As shown in Figure 2, dense connections enable different features to interact with different receptive fields. Compared with the original convolutional layer, local features can be mined from multiple scales. In addition, the dense expansion module can also promote local feature fusion and residual learning. The mathematical expression of the dense expansion module is: U n =DDB n (U0),

[0070] Among them, U0 represents the input feature of the dense expansion module, DDB n Represents the nth dense expansion module. For each dense expansion module, the output of its mth convolutional layer Expressed as:

[0071]

[0072] in, represents the mth convolutional layer of the nth dense dilation module. The number of convolutional layers m is set to 4, and (·) represents the element-wise summation operation;

[0073] For blurred images, most image deblurring methods utilize contextual information to separate blurry areas from clean image regions. The contextual information constructed by the channel attention module has demonstrated outstanding performance in image recognition tasks. Considering that CNN-based image deblurring methods rely heavily on the receptive field, which treats image feature information equally, this embodiment focuses on the contextual information between channel features, first introducing a threshold mechanism and then further introducing the IAM module to improve image deblurring performance.

[0074] See Figure 5 The attention module is used to focus on the contextual information between channel features. A threshold mechanism is introduced in the attention module. The front end of the attention module is divided into two convolution modules, and each convolution module consists of two convolution layers.

[0075] The features processed by the dense expansion module are input into the two-way convolution module in the attention module. After the input features pass through the two-way convolution modules respectively, the features are fused through the sum operation;

[0076] The fused features flow into the next convolutional layer, and on the other hand, a residual connection is established between the features and the input, which promotes the propagation of gradients and network optimization training.

[0077] The features of the residual connection are then passed through two sets of cascaded convolutional layers. A residual connection is established between each set of cascaded convolutional layers and the input of the cascaded convolutional layers to obtain continuously enhanced local features for network optimization and accelerated training.

[0078] The continuously enhanced local features are output as re-weighted non-local features through threshold control, and the continuously enhanced local features are multiplied with the re-weighted non-local features to obtain non-local features;

[0079] A global residual connection is constructed between the input features of the attention module and the non-local features; the feature representation after the global residual connection is passed as input to the subsequent post-feature conversion module.

[0080] The significance of the threshold mechanism is as follows: capturing long-range non-local features is of great significance in deep neural networks. Generally, the filters of each channel of a convolutional neural network are obtained by capturing the local receptive field. These features reflect the local spatial relationship within the image neighborhood. Only when local features are repeatedly used can long-range non-local dependencies be captured. For image deblurring tasks, methods that accurately distinguish blurry areas mostly require contextual information from the clear areas of the image. It is very necessary to capture non-local features that represent the overall data distribution of the image. The model directly models the dependencies between channels and introduces non-local contextual information across feature channels. Specifically, the threshold mechanism aggregates the spatial information of all feature channels, learns specific weights for each channel through a gating mechanism based on channel dependencies, and re-weights and sorts the feature channels according to the weights of the features themselves. The local features that are continuously enhanced before the threshold mechanism are point-multiplied with the re-weighted non-local features after the threshold mechanism. The network relies on local and non-local features to predict the clear features of the image and suppress the blurry features, so that the restored image has good visual effects.

[0081] Specifically, the threshold mechanism works as follows:

[0082] 1) Use global average pooling operation GP to control each feature map u entering the threshold C ∈R H×W The dimension is compressed to obtain a global distribution scalar z describing the local feature channel C =GP(u C ), where H×W represents the size of the feature map, and the vector z=[z1,z2,...,z C ]∈R C Indicates the global distribution of each channel and can adaptively predict the importance of each feature channel; in order to utilize z C =GP(u C ), the second operation is performed below to fully capture the dependency of feature channels, learn the nonlinear interaction between feature channels, and be able to learn a non-exclusive relationship to ensure that the importance of multiple feature channels is emphasized.

[0083] 2) In order to limit the complexity of the model and promote the generalization of the model, the gating mechanism is parameterized. The specific operation is as follows: the global distribution scalar z describing the local feature channel obtained in step 1) is converted to C and Perform convolution operation to convert z C Compression is performed and the soft threshold function Activate and get the compressed feature F U ; Compressed feature F U and Perform convolution operation to restore feature F U to its original size and activated by the Sigmoid function, outputting the vector F IAM Each element of is used as a channel threshold to recalibrate the importance of each channel;

[0084]

[0085] Among them, α represents the Sigmoid activation function, ConvU represents a trainable upsampling weight matrix, represents the soft threshold, R represents the real number domain, C represents the feature channel, r represents the scaling ratio, Represents the feature dimension, ConvR represents the trainable downsampling weight matrix, and GP represents the global pooling operation;

[0086] 3) Feature map u entering threshold control C The local features and non-local features F IAM Perform dot multiplication on the features of the corresponding feature channel For the reweighted non-local features, a more specific description is, Refers to the non-local features of the reweighted correction channels.

[0087] (203) The post-feature conversion module acts as a buffer to map features to high-dimensional features. The post-feature conversion module consists of convolutional layers, which are connected in a cascaded manner. The sizes of the convolution kernels are 1×1 and 3×3, respectively. A global feature fusion operation is used to fuse local features from the deep feature extraction module through the following operations:

[0088] U GF =GE(U1,U2,...,U n )

[0089] Among them, (·) represents the element-wise summation operation, GE represents the global feature fusion operation; finally, between GE and U -1 A global residual connection is established between them to enhance network optimization and gradient propagation.

[0090] (3) Construct the discriminant network D network structure

[0091] In order to distinguish the real label image from the image generated by the generative network, this embodiment uses PatchGAN as the discriminant network, such as Figure 6 As shown in the figure, the discriminant network consists of a convolutional layer; three downsampling convolutional layers reduce the resolution of the input image and encode important local features for classification. The image resolution after the three downsampling convolutional layers is reduced from 256×256 to 32×32; and a convolutional layer activated by a Sigmoid function is used to obtain the final classification response. Each convolutional layer is followed by a sample normalization layer and a Leaky ReLU activation function, and the convolution kernel size of all convolutional layers is 4×4. For the generative network, during the optimization training process of the generative network, the constraints of the semantic objective loss function and the structural objective loss function are added to it, so that the generated image has the same semantic and structural information as the labeled image. It is expected that the discriminant network cannot distinguish whether the input image is a synthetic image or a real labeled image. For the discriminant network, it feeds back the discrimination results to the generative network, driving the generative network to generate a generated image with significant image structure; the generative network updates the network parameters according to the feedback of the discriminant network and enters the next iterative training. The generative network and the discriminant network continue to compete and train in the manner described above until the network training converges. The blurred image is loaded into the generative network with converged training to obtain a deblurred image with significant structure.

[0092] Step 3: Construct the target loss function of the generative adversarial network model;

[0093] Based on the structure discrimination mechanism of the present invention, by introducing the image semantic content constraint term L content , image structure reconstruction constraint L gradient , and the discriminant loss function L adv , respectively regularize the training of the generative network G and the discriminative network D. Therefore, according to the importance of each part, the overall objective loss function of the generative adversarial network can be expressed as follows:

[0094] L(G,D)=βL content +λL gradient +αL adv ,

[0095] Among them, β, λ and α are L content , L gradient and L adv The weight coefficients of the constraints are as follows: β = 10, λ = 12, α = 1, where a larger value indicates that the corresponding part is more important.

[0096] Step 301: Construct image semantic content constraint L content

[0097] The purpose of image deblurring is to ensure that the restored image retains the same semantic content. In this implementation, a pre-trained VGG19 model is used to extract high-level features of the generated image and the labeled image, respectively, with the expectation that the distance between the two will be infinitely close. The semantic learning process of the generated image is constrained by solving the L2 norm. The semantic constraint function is expressed as follows:

[0098]

[0099] Among them, L content (s,G θG (b)) represents the gap between the semantic similarity of the generated image and the label image, W and H represent the width and height of the input image respectively, φ i,j Represents the features obtained after activation of the i-th pooling layer and the j-th convolution layer of the VGG19 model, G θG represents the generation network and all parameters, φ i,j (s) x,y represents the semantic content of the label image, φ i,j (G θG (b)) x,y Represents the semantic content of the generated image.

[0100] Step 302: Construct image structure reconstruction constraint L gradient

[0101] Salient structure is one of the qualitative indicators for measuring image clarity and is also one of the important priors in traditional methods. Therefore, this embodiment constrains the difference in image gradients between the generated image and the labeled image in the horizontal and vertical directions, so that the generated image has significant structural features. Specifically, the structural features of the image are constrained by solving the norm L1. The structural objective loss function is expressed as follows:

[0102]

[0103] in, Represents the gradient information of the image in the horizontal direction, Represents the gradient information in the vertical direction of the image.

[0104] Step 203: Discriminant loss function L adv .

[0105] As described in step 1, blur the image b i ∈P data (b) Input into the generative network to obtain the cleared image G θG (b), label image s i ∈P data (s), where P data(b) represents the data distribution of blurred images, P data (s) represents the data distribution of the label image. In the method of the present invention, the goal of training the discriminant network D is to assign the correct label to the cleared image G to the maximum extent possible. θG (b), Label image s with clear structure i ∈P data (s), guiding the generative network to convert the blurred image into a labeled image. In addition, the present invention adopts the optimization architecture based on bulldozer distance and gradient penalty (WGAN-GP) as the discriminant network. Therefore, the constraint function of the structure-driven discriminant network is expressed as follows:

[0106]

[0107] in, The term is the discriminant network D discriminant label image s i For true, The term is the discriminant network D that discriminates the generated image G(b i ) is false, represents the gradient penalty term, λ is the coefficient term, Indicates that the label image s i and generate data G(b i ) is a sample distribution obtained by uniformly sampling on the line connecting random values ​​between .

[0108] Step 4: Training and testing the generative adversarial network model

[0109] In this embodiment, 2103 pairs of labeled images / blurred images in the GOPRO dataset are selected as the training set of the present invention. i ∈P data (b), label image s i ∈P data (s) Randomly cropped images of size 256×256 are input to the generative network to learn the image sharpening process, and this process is determined by the target loss function L content , L gradient Constraints are imposed. The discriminant network needs to be completed to generate the image G θG (b) and label image s i ∈P data (s) discrimination task. The generator and discriminator networks maintain a competitive learning state until the network training converges. When testing, simply input the blurred image into the trained and converged G to obtain a generated image with significant structure.

[0110] In this example, the generative adversarial network model was run on a desktop computer equipped with an Intel(R) Core(TM) i7 CPU (16GB RAM) at 3.60GHz and an NVIDIA GeForce GTX 1080Ti GPU. The batch size (number of training images per batch) was 2, and the G and D learning rates were 0.0001. The slope of the Leaky ReLU activation function was 0.2. The network used the Adam optimizer with momentum parameters β1 = 0.5 and β2 = 0.999.

[0111] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps of the image deblurring method based on local features and non-local features in the above embodiments. The electronic device specifically includes the following contents:

[0112] Processor, memory, communications interface, and bus;

[0113] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0114] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all steps of the image deblurring method based on local features and non-local features in the above embodiment are implemented.

[0115] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all steps of the image deblurring method based on local features and non-local features in the above embodiment. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all steps of the image deblurring method based on local features and non-local features in the above embodiment.

[0116] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, those skilled in the art may make various specific modifications based on the teachings of the present invention, all of which fall within the scope of protection of the present invention.

[0120] Finally, it should be noted that the above examples are intended only to illustrate the calculation process of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the above examples, those skilled in the art will appreciate that the calculation process described in the above examples may be modified or some parameters may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the calculation method of the present invention.

[0121] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, those skilled in the art may make various specific modifications based on the teachings of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. An image deblurring method based on local features and non-local features, characterized in that: The following steps are involved: (1) Prepare data, including blurred images and labeled images; build a generative adversarial network model and set parameters; (2) Setting up the network structure framework of the generative network and the discriminative network; the network structure framework of the generative network adopts a U-shaped network structure, which includes an encoder and a decoder. The encoder downsamples and extracts useful detail features in the image for encoding, and the decoder is used to upsample and decode the image; a dense expansion network is also set between the encoder and the decoder. The dense expansion network consists of a dense expansion module and an attention module, which are respectively used to learn the local features of the image and the prediction of the clear features of the image by the non-local features. A threshold mechanism is introduced in the attention module; specifically: The blurred image is fed into the designed generative network. During the optimization training of the generative network, constraints are added to the semantic objective loss function and the structural objective loss function, so that the generated image has the same semantic and structural information as the labeled image. The discriminant network feeds the discrimination results back to the generative network, driving the generative network to generate a generated image with a relatively complete image structure. The generative network updates its parameters based on the feedback from the discriminative network and enters the next iterative training. The generative network and the discriminative network continue to compete and train in the manner described above until the generative adversarial network training converges. (3) Setting the target loss function of the generative adversarial network model: The target loss function in the generative network includes the image semantic target loss function L content , image structure reconstruction target loss function L gradient Among them, L content To ensure that the images before and after restoration can maintain the same semantic content, L gradient The generated image and the clear image are constrained to be consistent in image structure; the target loss function L in the discriminant network adv , L adv It is used to distinguish between generated images and labeled images, so as to improve the discriminant network's ability to distinguish between true and false images; (4) The blurred image and the labeled image are fed into the generative adversarial network model, and the discriminant network is used to determine the consistency of the content between the generated image and the labeled image; the discriminant network feeds back the judgment result to the generative network, and the generative network and the discriminant network compete for learning until a Nash equilibrium is reached and the generative adversarial network model converges; the blurred image is loaded into the generative network that has converged in training, and finally a generated image with a complete structure is obtained.

2. The image deblurring method based on the improved attention mechanism according to claim 1, characterized in that: In step (3), the target loss function of the generative adversarial network model is weighted as: L(G,D)=βL content +λL gradient +αL adv Among them, β, λ and α are L content , L gradient and L adv The weight coefficient of each constraint item is as follows: β = 10, λ = 12, α = 1. The larger the value, the more important the corresponding part.

3. The image deblurring method based on the improved attention mechanism according to claim 1, characterized in that: For the generative network, the encoder includes a convolutional layer for mapping the dimension of the input image to 64×64; three downsampling layers are used to downsample and encode the image, and three residual blocks are added after the second and third downsampling layers, and the image resolution is reduced from 256×256 to 64×64; after the above operations are completed, a dense expansion network is introduced to learn the prediction of the local features and non-local features of the image for the clear features of the image; correspondingly, the decoder includes three upsampling layers for upsampling and decoding the image, and three residual blocks are added before each upsampling layer, and the image resolution is reduced from 64×64 to 256×256; finally, the deblurred image is reconstructed by a Tanh layer and a convolutional layer; For the discriminant network, a block-based generative adversarial network is adopted as the network structure of the discriminant network, which includes a flat convolution layer, three downsampling convolution layers, and a feature reconstruction block; the downsampling convolution layer is used to reduce the resolution of the input image and encode local features; each convolution layer is followed by a sample normalization layer and a leaky rectified linear unit activation function Leaky ReLU, and the convolution kernel size of each convolution layer is 4×4.

4. The image deblurring method based on local features and non-local features according to claim 1, characterized in that: The dense expansion network is divided into three parts: a front feature conversion module, a deep feature extraction module, and a post feature conversion module; The deep feature extraction module consists of 5 dense expansion modules, 1 attention module and 1 1×1 kernel convolution layer. The outputs of the 5 dense expansion modules and 1 attention module are summed element by element. The output features obtained by the summation are subjected to a 1×1 convolution operation to obtain the output features of the deep feature extraction module. This feature then establishes a residual connection with the input features of the deep feature extraction module. The dense expansion module is used to enhance the correlation of features and construct high-dimensional complex features. The attention module is used to achieve global feature fusion and residual learning.

5. The image deblurring method based on local features and non-local features according to claim 4, characterized in that: The pre-feature conversion module acts as a buffer to map features to high-dimensional features. The pre-feature conversion module consists of two cascaded convolutional layers with a convolution kernel size of 3×3. IN -1 =FT1(U n-1 ), U0=FT2(U -1 ), Among them, FT1 and FT2 are the first and second convolutional layers in the front feature conversion module respectively, and feature U-1 is the output feature of convolutional layer FT1; on the one hand, feature U0 is passed as input to the deep feature extraction module; on the other hand, feature U0 establishes a global residual connection with the output of the deep feature extraction module; The mathematical expression of the dense expansion module is: U n =DDB n (U0), Among them, U0 represents the input feature of the dense expansion module, DDB n Represents the nth dense expansion module. For each dense expansion module, the output of its mth convolutional layer Expressed as: in, represents the mth convolutional layer of the nth dense dilation module. The number of convolutional layers m is set to 4, and (·) represents the element-wise summation operation; The attention module is used to focus on the contextual information between channel features. A threshold mechanism is introduced in the attention module. The front end of the attention module is divided into two convolutional modules, each of which consists of two convolutional layers. The features processed by the dense expansion module are first input into the two-way convolution module in the attention module. After the input features pass through the two-way convolution modules, the features are fused through the summation operation. A residual connection is established between the fused features and the input features of the attention module to promote gradient propagation and network optimization training; The features of the residual connection are then passed through two sets of cascaded convolutional layers. A residual connection is established between each set of cascaded convolutional layers and the input of the cascaded convolutional layers to obtain continuously enhanced local features for network optimization and accelerated training. The continuously enhanced local features are output as re-weighted non-local features through threshold control, and the continuously enhanced local features are multiplied with the re-weighted non-local features to obtain non-local features; A global residual connection is constructed between the input features of the attention module and the non-local features; the feature representation after the global residual connection is passed as input to the subsequent post-feature conversion module; The post-feature conversion module acts as a buffer to map features to high-dimensional features. The post-feature conversion module consists of two cascaded convolutional layers with convolution kernel sizes of 1×1 and 3×3 respectively. It uses a global feature fusion operation to fuse local features from the deep feature extraction module through the following operations; U GF =GE(U1,U2,...,U n ) Among them, (·) represents the element-wise summation operation, GE represents the global feature fusion operation; finally, between GE and U -1 A global residual connection is established between them to enhance network optimization and gradient propagation.

6. The image deblurring method based on local features and non-local features according to claim 5, characterized in that: The threshold mechanism works as follows: 1) Use global average pooling operation GP to control each feature map u entering the threshold C ∈R H×W The dimension is compressed to obtain a global distribution scalar z describing the local feature channel C =GP(u C ), where H×W represents the size of the feature map, and the vector z=[z1,z2,...,z C ]∈R C Indicates the global distribution of each channel and can adaptively predict the importance of each feature channel; in order to utilize z C =GP(u C ), the second operation is performed below to fully capture the dependency of feature channels, learn the nonlinear interaction between feature channels, and be able to learn a non-exclusive relationship to ensure that the importance of multiple feature channels is emphasized; 2) In order to limit the complexity of the model and promote the generalization of the model, the gating mechanism is parameterized. The specific operation is as follows: the global distribution scalar z describing the local feature channel obtained in step 1) is converted to C and Perform convolution operation to convert z C Compression is performed and the soft threshold function Activate and get the compressed feature F U ; Compressed feature F U and Perform convolution operation to restore feature F U to its original size and activated by the Sigmoid function, outputting the vector F IAM Each element of is used as a channel threshold to recalibrate the importance of each channel; Among them, α represents the Sigmoid activation function, ConvU represents a trainable upsampling weight matrix, represents the soft threshold, R represents the real number domain, C represents the feature channel, r represents the scaling ratio, Represents the feature dimension, ConvR represents the trainable downsampling weight matrix, and GP represents the global pooling operation; 3) Feature map u entering threshold control C The local features and non-local features F IAM Perform dot multiplication on the features of the corresponding feature channel is the reweighted non-local feature.

7. The image deblurring method based on local features and non-local features according to claim 1, characterized in that: The generative adversarial network model was run on a computer; the number of batch training images was 2, the learning rate of the generator network and the discriminator network was 0.0001, the slope of the activation function Leaky ReLU was 0.2, and the generative adversarial network used the Adam optimizer with momentum parameters β1 = 0.5 and β2 = 0.999, respectively.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the image deblurring method based on local features and non-local features according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image deblurring method based on local features and non-local features according to any one of claims 1 to 6 are implemented.

Images