Image deblurring method based on multi-scale feature extraction and fusion
Through the multi-scale feature extraction and fusion technology of the generative model and structural attention mechanism, the generalization and real-time problems of image defuzzing in the existing technology are solved, and efficient and accurate image defuzzing effect is achieved, which is suitable for a variety of reasons. Blurry image.
Patent Information
- Application Number
- CN202411003536.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art has problems such as low generalization, long iteration solution time, and too many network parameters in terms of image defuzzing, and especially difficult to achieve efficient and robust defuzzing effects in actual blurred images.
Using multi-scale feature extraction and fusion method based on adversarial generation model and structural attention mechanism, through competitive training of the generative network and discriminative network, combining image semantic target loss function and structural target loss function, reduce network model size, improve generalization ability and Real-time processing capability.
It realizes the recovery of images with clear structure and rich details, improves the accuracy and robustness of the algorithm, is suitable for a variety of blurred images, reduces hardware configuration requirements, and has a wide range of application prospects.
Smart Images

Figure CN119006334A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing and computer vision technology, and particularly relates to an image deblurring method based on multi-scale feature extraction and fusion. Background Art
[0002] As a carrier for recording and transmitting information about the external objective world, images have always been the main source and means for humans to obtain and identify information about the objective world. However, image blur caused by camera shake or object motion often occurs during image capture. Since blurred images lose their clear structure and rich texture information, it is difficult for people to obtain clear content and fine information from them. Therefore, how to clarify motion blurred images so that they can be better applied to advanced image processing (image detection, image recognition) and other fields has become a research hotspot.
[0003] Regarding the problem of how to clarify blurred images, people have mainly conducted research from the following two aspects: image deblurring methods based on traditional methods and image deblurring methods based on deep learning. Image deblurring methods based on traditional methods rely on manually extracting prior information or statistical information from images, and on this basis, modeling optimization equations, and obtaining restored images by iteratively solving the optimization equations. Since traditional methods only extract prior information on limited images, such methods only obtain good deblurring results on specific blurred images, but have low generalization on other blurred images. In addition, it takes a lot of time to iteratively solve the optimization function, so such methods cannot meet the real-time requirements of the algorithm well. Image deblurring problems based on deep learning extract features from a large number of data sets, and continuously iterate to select weights that are more suitable for image restoration during the training of the network model, thereby restoring the potential label image. Although some achievements have been made in the image deblurring problem, the restored images are not very satisfactory. For example, some methods based on deep learning have problems such as too many network parameters and too large network models, which undoubtedly puts higher requirements on the hardware configuration of network training; other methods are only applicable to synthetic blurred images, and have weak generalization and robustness in actual blurred images. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art, and to study and provide an image deblurring method that can effectively reduce the size of the network model, meet the real-time processing requirements of image deblurring, and solve the problems of image texture and detail loss by comprehensively utilizing the adversarial generative model and the structural attention mechanism.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] An image deblurring method based on multi-scale feature extraction and fusion comprises the following steps:
[0007] (1) Prepare data, including blurred images and labeled images; build a generative adversarial network model and set parameters;
[0008] (2) Setting a network structure framework of a generative network and a discriminative network; the network structure framework of the generative network adopts a U-shaped network structure, and the generative network includes an encoder and a decoder. The encoder downsamples and extracts useful detail features in the image for encoding, and the decoder is used to upsample and decode the image; a multi-scale feature extraction and fusion module is also set between the encoder and the decoder, specifically:
[0009] The blurred image is sent to the generative network. During the optimization training of the generative network, constraints of the image semantic target loss function and the image structure target loss function are added to the generative network, so that the generated image has semantic information and structural information consistent with the label image; the discriminant network feeds back the discriminant result to the generative network, driving the generative network to generate a generated image with a complete image structure; the generative network updates the parameters of the generative network according to the feedback of the discriminant network and enters the next iterative training. The generative network and the discriminant network continuously compete and train in the manner described above until the training of the generative adversarial network model converges;
[0010] (3) Setting the target loss function of the generative adversarial network model: The target loss function in the generative network includes the image semantic target loss function L content , image structure objective loss function L gradient Among them, L content To ensure that the restored images can maintain the same semantic content, L gradient The generated image is constrained to be consistent with the clear image in terms of image structure; the target loss function L in the discriminant network adv It is used to distinguish between generated images and labeled images, so as to improve the discriminant network's ability to distinguish true and false images;
[0011] (4) The blurred image and the labeled image are fed into the generative adversarial network model, and the discriminant network is used to determine the consistency of the content between the generated image and the labeled image. The discriminant network feeds back the judgment result to the generative network, and the generative network and the discriminant network compete for learning until a Nash equilibrium is reached and the generative adversarial network model converges. The blurred image is loaded into the generative network that has been trained and converged, and finally a generated image with a complete structure is obtained.
[0012] Furthermore, in step (3), the objective loss function of the generative adversarial network model is weighted as:
[0013] L(G,D)=βL content +λLgradient +αL adv
[0014] Among them, β, λ and α are L content , L gradient and L adv The weight coefficient of each constraint item is as follows: β=10, λ=12, α=1. The larger the value, the more important the corresponding part.
[0015] Further, for the generative network, the encoder includes a convolutional layer for mapping the dimension of the input image to 64×64; five downsampling layers are used to downsample and encode the image, and the image resolution is reduced from 256×256 to 16×16; after the above operations are completed, a multi-scale feature extraction and fusion module MSFEFM is introduced to extract more complex and abstract features to capture potential patterns and relationships in the data, reduce the dependence of the generative adversarial network model on training data, and improve the generalization ability; correspondingly, the decoder includes five upsampling layers for upsampling and decoding the image, and the image resolution is increased from 16×16 to 256×256; finally, the deblurred image is reconstructed by a Tanh layer and a convolutional layer;
[0016] For the discriminant network, a block-based generative adversarial network is adopted as the network structure of the discriminant network, including a flat convolution layer, three downsampling convolution layers, and a feature reconstruction block; the downsampling convolution layer is used to reduce the resolution of the input image and encode local features; a sample normalization layer and a leaky rectified linear unit activation function Leaky ReLU are added after each convolution layer, and the convolution kernel size of each convolution layer is 4×4.
[0017] Furthermore, the multi-scale feature extraction and fusion module is divided into two parts, including two parallel multi-scale feature extraction modules and a multi-scale feature fusion module; the multi-scale feature extraction module is used to extract features of different scales and levels of input data; the multi-scale feature fusion module forms a rich and comprehensive feature representation by fusing features from different scales; features of different scales contain information at different levels. Through fusion, the generative adversarial network model can simultaneously utilize low-level detail features and high-level abstract features.
[0018] Furthermore, the features compressed by the encoder are input into two parallel multi-scale feature extraction modules, and non-overlapping patches are aggregated and shifted in the spatial dimension according to different scale values, as follows:
[0019] Input: feature x, patch; the feature with dimension (B, H, W, C) is input into the multi-scale feature extraction module. The working steps of the multi-scale feature extraction module are as follows:
[0020] Step 401. Exchange the dimensions of the feature tensor from (B, H, W, C) to (B, W, C, H); B, H, W, C refer to batch size, height, width, and number of channels, respectively;
[0021] Step 402. Get the batch size, height, width and number of channels of the dimensions;
[0022] Step 403. Generate non-overlapping patches according to different patch sizes P to perform local processing on the input features;
[0023] Step 404: The dimensions of the new patch are reorganized to form a unified patch dimension format to ensure that subsequent operations can proceed smoothly;
[0024] Step 405: average the features of the patch to obtain a mean feature patch to extract the overall features of the patch;
[0025] Step 406: Perform nonlinear transformation on the features of the patch through a multi-layer perceptron (MLP) to extract complex and abstract features;
[0026] Step 407: Perform layer normalization on the features of the patch to improve the stability and training efficiency of the generative adversarial network model;
[0027] Step 408: Perform a further nonlinear transformation on the features of the patch through a multi-layer perceptron (MLP) to extract deeper features;
[0028] Step 409: Calculate the attention distribution, and convert the input value into a value representing the probability distribution through the softmax function;
[0029] Step 410: Perform a weighted multiplication operation on the mean feature patch and the attention distribution calculated in step 409, and weightedly multiply the mean feature patch and the attention distribution through the attention mechanism to obtain a weighted feature representation; for each mean feature patch, give different importance weights through the attention distribution, and then add these weighted mean feature patches to obtain the final feature representation;
[0030] Step 411. Normalize the output features along the last dimension so that the length of each vector is 1;
[0031] Step 412: Generate a mask for filtering and selecting important feature information;
[0032] Step 413. Apply the generated mask to the output features, and each feature value will be scaled according to the mask value;
[0033] Step 414: Adjust the dimension of the output features.
[0034] Furthermore, the workflow of the multi-scale feature fusion module is as follows:
[0035] Step 501: Use a 1x1 convolution kernel to perform a convolution operation on the input feature x, without changing the spatial size, but only changing the number of channels;
[0036] Step 502: Apply two multi-scale feature extraction modules using different attention mechanisms and patch values to capture features at more levels; perform three convolution operations on the input feature x in sequence to extract features, and add and fuse all the extracted features;
[0037] Step 503: Apply the channel attention module ECA to enhance the features of important channels; apply the spatial attention module to enhance the features of important spatial positions and improve the feature expression capability.
[0038] Furthermore, the generative adversarial network model is run on a computer; the number of batch training images is 2, the learning rate of the generative network and the discriminative network is 0.0001; the slope of the activation function Leaky ReLU is 0.2; the generative adversarial network model uses the Adam optimizer, and its momentum parameters are β1=0.5 and β2=0.999 respectively.
[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the image deblurring method based on multi-scale feature extraction and fusion are implemented.
[0040] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image deblurring method based on multi-scale feature extraction and fusion are implemented.
[0041] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0042] 1. Compared with existing image deblurring algorithms, the method of the present invention can restore images with clearer structures and richer details. Traditional methods rely on manually extracting feature prior knowledge from limited images, while the present invention avoids the cumulative error caused by separately estimating model parameters through a purely data-driven reconstruction method of the adversarial generative network, thereby improving the accuracy and robustness of the algorithm.
[0043] 2. The method of the present invention can generate deblurred images with significant structures and clear details, which is significantly better than traditional methods. The multi-scale feature extraction and fusion module can effectively capture the different scale features of the image, ensuring that the overall structure and local details of the image can be fully restored.
[0044] 3. The present invention adopts the attention mechanism to fuse the feature information extracted by the multi-scale network structure to promote the performance of image deblurring. The attention mechanism can highlight the important feature areas in the image, thereby improving the deblurring effect and making the restored image clearer and more realistic.
[0045] 4. By comprehensively utilizing the adversarial generative model and the structural attention mechanism, the method of the present invention can effectively reduce the size of the network model and meet the needs of real-time image deblurring. Compared with some deep learning-based methods that have problems such as too many network parameters and too large models, the present invention reduces the requirements for hardware configuration while maintaining high performance, making it more practical and economical.
[0046] 5. The method of the present invention is not only applicable to synthetic blurred images, but also shows strong generalization and robustness in actual blurred images. This means that no matter what kind of blurred environment, the method can effectively restore clear images and has broad application prospects.
[0047] In summary, the present invention proposes an efficient, accurate and robust image deblurring method by comprehensively utilizing the generative adversarial model, structural attention mechanism and multi-scale feature extraction and fusion technology, which effectively overcomes the shortcomings of the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a schematic diagram of the process framework of the method of the present invention;
[0049] Figure 2 Network layout and parameter diagrams for generating networks;
[0050] Figure 3 It is a schematic diagram of the structure of the Multi-Scale Feature Extraction and Fusion Module (MSFEFM);
[0051] Figure 4 Network layout and parameter diagram for the discriminant network. DETAILED DESCRIPTION
[0052] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0053] like Figure 1 As shown, this embodiment provides an image deblurring method based on multi-scale feature extraction and fusion, which includes the following steps:
[0054] Step 1. Prepare data, including blurred images and label images; build a generative adversarial network model and set parameters; the label image is a clear image without blur degradation, and the generative adversarial network model includes two sub-networks: the generative network and the discriminative network;
[0055] Step 2. Construct the network structure of the generating network and the discriminating network; Figure 2 and Figure 4 As shown, the generative network is used to learn the image sharpening process, while the discriminative network discriminates and provides feedback on the deblurring results learned by the generative network.
[0056] (1) Constructing the network structure of the generated network G
[0057] This embodiment uses a U-shaped network as the network structure of the generated network, such as Figure 2 As shown. The U-shaped network consists of two parts: the encoder and the decoder. The main function of the encoder stage is to spatially compress and encode the image, and the main function of the decoder stage is to construct the features of the image content and domain. In addition, a jump connection is established between the upsampling layer and the downsampling layer of the corresponding scale to associate the low-dimensional features learned in the encoder with the decoded features, guiding the recovery of the local and detailed parts of the image, which is critical for learning the image structure and details. The encoder part includes a convolutional layer that maps the feature dimension to 64; five downsampling layers are used to downsample and encode the image, and the image resolution is reduced from 256×256 to 16×16. A multi-scale feature extraction and fusion module is then introduced, see step (2) for details.
[0058] Correspondingly, the decoder includes five upsampling layers to upsample and decode the image, and the image resolution increases from 16×16 to 256×256. Finally, the deblurred image is reconstructed by a Tanh layer and a convolution layer with a convolution kernel of 7×7. A multi-scale feature extraction and fusion module is also set between the encoder and decoder.
[0059] (2) Construct a multi-scale feature extraction and fusion module MSFEFM, see Figure 3 .
[0060] The multi-scale feature extraction and fusion module is divided into two parts, including two parallel multi-scale feature extraction modules and a multi-scale feature fusion module; the role of the multi-scale feature extraction module is to extract features of different scales and levels of input data in the neural network. Different feature scales can capture detailed information and global information in the image; smaller scales can capture details (such as edges, textures), while larger scales can capture larger structures and patterns; by using filters or feature extraction methods of different scales, the network can have different receptive fields and can perceive input data at multiple scales; extracting features at different scales and fusing these features can form a richer and more comprehensive feature representation, which helps to improve the expression ability of the model; the multi-scale feature fusion module, by fusing features from different scales, the model can form a richer and more comprehensive feature representation, which helps to better describe complex input data; features of different scales contain information at different levels. Through fusion, the model can simultaneously utilize low-level detail features and high-level abstract features. The details are as follows:
[0061] (201) The features compressed by the encoder are input into the parallel dual multi-scale feature extraction module MSFEM. Non-overlapping patches are aggregated and displaced in the spatial dimension according to different scale values. The specific working process of the multi-scale feature extraction module is as follows:
[0062] Input: feature x, patch size P = 2, 4; Below, the features of dimension [4, 16, 16, 1024] are input to the MSFEM module, and each step is given in detail and the output results are calculated;
[0063] a. Exchange dimensions. Exchange the dimensions of the feature tensor and transform the dimensions from (B, H, W, C) to (B, W, C, H) through x←x.permute(0,2,3,1). The input dimensions are [4, 16, 16, 1024] and the output dimensions are [4, 16, 1024, 16], so that the arrangement order meets the requirements of subsequent operations.
[0064] b. Get the batch size, height, width and number of channels through B, H, W, C←x.shape, B=4, H=16, W=1024, C=16;
[0065] c. Generate non-overlapping patches according to different patch sizes P. Specifically, through patches←x.unfold(1,P,P).unfold(2,P,P), taking P=4 as an example, the dimension of the obtained patch block is [4,4,256,16,4,4], and non-overlapping patches are generated to facilitate local processing of input features; dividing the input features into small blocks helps capture local information and details. The features within each patch can represent the local pattern of the image or data, which helps to extract finer features; dividing the input features into patches can effectively reduce the dimension of the data. By processing each patch, the calculation can be simplified and the computational complexity can be reduced; through local processing, the model can better cope with local changes and noise in the input data, improving the robustness and generalization ability of the model;
[0066] d. Rearrange the dimensions of the patches through patches←patches.reshape(B,-1,P*P,C), where B is the batch size, -1 indicates the automatically calculated dimension, P*P is the number of pixels in each patch, and C is the number of channels. The reorganized dimension is [4,1024,16,16]; this is done to adjust the generated patches to a shape and format suitable for subsequent processing operations. Converting the patch tensor to a four-dimensional form (B,-1,P*P,C) can unify the dimensional form of the patches and ensure that subsequent operations can proceed smoothly;
[0067] c. Calculating the mean of patches by patches←patches.mean(dim=-1) is to average the features of the patches, thereby simplifying the data and extracting the overall features of the patches. The feature dimension after calculation is [4,1024,16]. By averaging the features of the patches, the dimension of the data can be effectively reduced. The original patch contains the feature information of all pixels. Averaging these features can compress high-dimensional features into low-dimensional features. Averaging the features of each patch can extract the overall features of the patch and help the model capture local patterns. This is very important for understanding the local information of images or other data.
[0068] f. The first MLP operation patches←mlp1(patches) aims to perform nonlinear transformation on patch features through multi-layer perceptron (MLP) to extract more complex and abstract features. MLP can perform nonlinear transformation on input features to help extract more complex features and patterns. Through the combination of multiple fully connected layers, MLP can capture high-order relationships in the data. Using MLP to process patch features can increase the expressiveness of the model, enabling it to learn more abstract and advanced features. Assume that the output dimension of mlp1 is 8, and the feature dimension after MLP remains unchanged, which is [4,1024,8].
[0069] g. Layer normalization patches←layernorm(patches), its purpose is to perform layer normalization on patch features to improve the stability of the model and training efficiency; layer normalization normalizes the features of each patch to make its mean zero and variance one, thereby reducing internal covariate shift; this helps stabilize the model training process and improve the convergence speed; after layer normalization, the feature dimension remains unchanged, still [4,1024,8];
[0070] h. The second MLP operation patches←mlp2(patches) can extract deeper features through further nonlinear transformation and nonlinear function activation. This helps capture more complex and advanced patterns and relationships in the data; the second MLP operation can further fuse and combine the features processed by the first MLP and layer normalization, so that the model can better understand and utilize the relationship between local features; assuming that the output dimension of mlp2 is 16, the feature dimension after the second MLP is [4,1024,16];
[0071] i. Calculate the attention distribution through attention←F.softmax(patches,dim=-1). The softmax function is usually used to convert the input value into a value representing the probability distribution. Here, patches is a tensor containing various regions or features. Dim=-1 means that the softmax operation is performed on the last dimension. The purpose of this operation is to calculate the weight of each region or feature in the attention mechanism, that is, the overall importance of different regions or features, so that the model can pay more attention to the important parts when performing tasks. The feature dimension after softmax remains unchanged, which is still [4,1024,16];
[0072] j. Perform a weighted multiplication operation on the feature patch after averaging and the attention distribution calculated in step 9 through Out←patches×attention, and multiply the mean feature patch by the attention distribution in a weighted manner through the attention mechanism to obtain a weighted feature representation; specifically, for each feature patch, different importance weights are given through the attention distribution, and then these weighted mean feature patches are added together to obtain the final feature representation; this operation can help the model focus on the parts that are important for the current task and improve the model's representation ability and performance; the feature dimension after the weighted multiplication operation remains unchanged, and is still [4,1024,16];
[0073] k. Normalize the output features along the last dimension through F.normalize(local_out, dim=-1) so that the length of each vector is 1; Normalize the prompt vector through F.normalize(self.prompt[None,...,None], dim=1) and add a dimension to the dimension of the prompt vector for broadcasting; Use matrix multiplication @ to calculate the dot product between the normalized output features and the normalized prompt vector to obtain the cosine similarity. The feature dimension after cosine similarity calculation is [4,1024,16]; Calculate the cosine similarity between the output feature and a prompt vector to generate a mask; Cosine similarity is a commonly used similarity measure, especially suitable for high-dimensional vector space. By calculating cosine similarity, the degree of similarity between the output feature and the prompt vector in direction can be determined;
[0074] l. Generate a mask through mask←clamp(cos sim,0,1) to filter and select important feature information; specifically, clamp(cos sim,0,1) limits the value of cosine similarity to the range of 0 to 1, filtering out negatively correlated or irrelevant features: by limiting negative values (values below 0) to 0, negatively correlated or irrelevant features can be ignored, and only positively correlated features are retained; the feature dimension after generating the mask remains unchanged, still [4,1024,1];
[0075] m. Apply the mask through out←out*mask, which applies the generated mask to the output feature out. Each output feature value will be scaled according to the mask value; the feature dimension after applying the mask is [4,1024,16]; out←out@self.top_down_transform, which transforms the output feature with the mask applied through a linear transformation matrix self.top_down_transform. Here, matrix multiplication @ is used to achieve this. After applying the linear transformation, the output feature is [4,1024,16];
[0076] n.out←out.reshape(B,H / / P,W / / P,self.output_dim) is to resize the dimension of feature out to [Batch size,Height / Patch size,Width / Patch size,Output dimension], and the output dimension of out is [4,4,256,16]; out←out.permute(0,3,1,2) is to adjust the dimension order and adjust the shape of out to [Batch size,Output dimension,Height / Patch size,Width / Patch size], and the output dimension of out is [4,16,4,256]; out←F.interpolate(local_out,size=(H,W),mode='bilinear',align_corners=False) is to use bilinear interpolation to restore the spatial size of feature out from [Height / Patch size,Width / Patch size] to the original [Height,Width], and the output dimension of out is [4,16,16,1024].
[0077] (202) The multi-scale feature fusion module is as follows:
[0078] A.x_skip = self.skip(x), performs a convolution operation on the input x, using a 1x1 convolution kernel, without changing the spatial size, only the number of channels. self.skip has no activation function, which means it is just a linear transformation, and its main function is to provide a shortcut path for subsequent feature fusion, similar to the skip connection of ResNet;
[0079] B.x_lga2 = self.lga2(x_skip), apply a multi-scale feature extraction module, with a patch value of P = 2. The purpose of this step is to enhance the feature expression capability by combining local and global features; x_lga4 = self.lga4(x_skip), apply another multi-scale feature extraction module, with a patch value of P = 4, similar to the previous step, but using a different attention mechanism to capture more layers of features; x1 = self.c1(x), perform the first 3x3 convolution operation on the input x, use the activation function for nonlinear transformation, and extract local features; x2 = self. c2(x1), perform a second 3x3 convolution operation on x1, use the activation function for nonlinear transformation, and self.c2 further extracts and processes features; x3=self.c3(x2), perform a third 3x3 convolution operation on x2, use the activation function for nonlinear transformation, and self.c3 further extracts and processes features; x=x1+x2+x3+x_skip+x_lga2+x_lga4, add and fuse all the extracted features (convolutional layer features, skip connection features, and multi-scale features). This step combines features of different levels and types to form a stronger feature expression;
[0080] Cx=self.cn(x), apply channel attention module (ECA) to enhance the features of important channels. This step improves the feature expression ability by focusing on important channel features; x=self.sa(x), apply spatial attention module (SpatialAttentionModule), enhance the features of important spatial positions. This step improves the feature expression ability by focusing on important spatial position features; x=self.drop(x), apply dropout, randomly discard some features to prevent overfitting, this step increases the generalization ability of the model; x=self.bn1(x), apply batch normalization (BatchNorm2d), normalize feature data, reduce internal covariate shift, stabilize the training process, this step can make the model training faster and more stable; x=self.relu(x), apply ReLU activation function to increase nonlinearity. This step makes the features have nonlinear expression ability, which helps the model learn complex patterns; thus finally obtaining the features processed by MSFEFM.
[0081] (3) Construct the discriminant network D network structure
[0082] In order to distinguish the real label image from the image generated by the generative network, this embodiment uses PatchGAN as the discriminant network, such as Figure 4As shown. The discriminant network includes a convolution layer; three downsampling convolution layers reduce the resolution of the input image and encode important local features for classification. The image resolution after the three downsampling convolution layers decreases from 256×256 to; a convolution layer activated by the Sigmoid function is used to obtain the final classification response. A sample normalization layer and a Leaky ReLU activation function are added after each convolution layer, and the convolution kernel size of all convolution layers is 4×4. For the generative network, during the optimization training of the generative network, the constraints of the semantic target loss function and the structural target loss function are added to it, so that the generated image has the same semantics and structural information as the label image. It is expected that the discriminant network cannot distinguish whether the input image is a synthetic image or a real label image. For the discriminant network, it feeds back the discrimination result to the generative network, driving the generative network to generate a generated image with a significant image structure; the generative network updates the network parameters according to the feedback of the discriminant network to enter the next iterative training. The generative network and the discriminant network continuously compete for training in the manner described above until the network training converges, and the blurred image is loaded into the generative network with converged training to obtain a deblurred image with a significant structure.
[0083] Step 3. Construct the target loss function of the generative adversarial network model;
[0084] Based on the structure discrimination mechanism of this embodiment, by introducing the image semantic target loss function L in the generative network content , image structure objective loss function L gradient , and the target loss function L in the discriminant network adv , respectively regularize the training of the generative network G and the discriminative network D. Therefore, according to the importance of each part, the overall objective loss function of the generative adversarial network can be expressed weightedly as:
[0085] L(G,D)=βL content +λL gradient +αL adv ,
[0086] Among them, β, λ and α are L content , L gradient and L adv The weight coefficients of each constraint are as follows: β = 10, λ = 12, α = 1, and a larger value means that the corresponding part is more important.
[0087] S301. Construct image semantic target loss function L content
[0088] The purpose of image deblurring is to ensure that the images before and after restoration can maintain the same semantic content. In this implementation, the pre-trained VGG19 model is used to extract high-order features of the generated image and the label image, respectively, in the expectation that the distance between the two is infinitely close, and the semantic learning process of the generated image is constrained by solving the two-norm L2. The semantic constraint function is expressed as follows:
[0089]
[0090] Among them, L content (s,G θG (b)) represents the gap between the semantic similarity of the generated image and the labeled image, W and H represent the width and height of the input image respectively, φ i,j represents the features obtained after the activation of the i-th pooling layer and the j-th convolution layer of the VGG19 model, G θG represents the generation network and all parameters, φ i,j (s) x,y represents the semantic content of the label image, φ i,j (G θG (b)) x,y Represents the semantic content of the generated image.
[0091] S302. Construct image structure target loss function L gradient
[0092] The significant structure is one of the qualitative indicators to measure whether the image is clear, and it is also one of the important priors in the traditional method. Therefore, this embodiment constrains the difference in image gradient between the generated image and the label image in the horizontal and vertical directions of the image, so that the generated image has significant structural features. Specifically, the structural features of the image are constrained by solving a norm L1, and the structural objective loss function is expressed as follows:
[0093]
[0094] Among them, s represents the label image, b represents the blurred image, Represents the gradient information of the image in the horizontal direction, ▽ v Represents the gradient information of the image in the vertical direction.
[0095] S302. Discriminant loss function L adv .
[0096] Blurred image b i ∈P data (b) Input into the generative network to obtain the generated image G θG (b), label image s i ∈P data (s), where P data(b) represents the data distribution of blurred images, P data (s) represents the data distribution of the label image. In the present embodiment, the goal of training the discriminant network D is to assign the correct label to the cleared image G to the maximum extent possible. θG (b), Label image s with clear structure i ∈P data (s), guiding the generative network to convert the blurred image into a label image. In addition, this embodiment adopts the optimization architecture based on bulldozer distance and gradient penalty term (WGAN-GP) as the discriminant network. Therefore, the constraint function of the structure-driven discriminant network is expressed as follows:
[0097]
[0098] in, The term is the discriminant network D discriminates the label image s i For true, The term is the discriminant network D discriminates the generated image G(b i ) is false, represents the gradient penalty term, λ is the coefficient term, Indicates that between the label image si and the generated data G(b i ) is a sample distribution obtained by uniformly sampling on the line connecting random values between them.
[0099] Step 4. Training and testing of the generative adversarial network model
[0100] In this embodiment, 2103 pairs of labeled images / blurred images in the GOPRO dataset are selected as the training set of the present invention. i ∈P data (b), label image s i ∈P data (s) Randomly cropped images of size 256×256 are input into the generative network to learn the image sharpening process, and this process is determined by the objective loss function L content , L gradient Constraints are imposed. The discriminant network needs to be completed to generate the image G θG (b) and label image s i ∈P data The generation network and the discrimination network maintain a competitive learning state until the network training converges. When testing, you only need to input the blurred image into the G that has been trained and converged to obtain a generated image with a significant structure.
[0101] In this embodiment, the generative adversarial network model is run on a desktop computer configured with an Intel(R) Core(TM) i7 CPU (16GB RAM) 3.60GHz CPU and an NVIDIA RTX 4090 GPU. The batch size (the number of batch training images) is 2, and the G and D learning rates are 0.0001. The slope of the activation function Leaky ReLU is 0.2. The network uses the Adam optimizer, and its momentum parameters are β1=0.5 and β2=0.999 respectively.
[0102] The embodiment of the present application also provides a specific implementation of an electronic device capable of implementing all steps in the image deblurring method based on multi-scale feature extraction and fusion in the above embodiment, and the electronic device specifically includes the following contents:
[0103] Processor, memory, communications interface and bus;
[0104] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.
[0105] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the image deblurring method based on multi-scale feature extraction and fusion in the above embodiment are implemented.
[0106] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all steps of the image deblurring method based on multi-scale feature extraction and fusion in the above embodiment. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, all steps of the image deblurring method based on multi-scale feature extraction and fusion in the above embodiment are implemented.
[0107] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0108] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0109] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0110] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.
[0111] Finally, it should be pointed out that the above examples are only used to illustrate the calculation process of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above examples, a person skilled in the art should understand that the calculation process described in the above examples can still be modified, or some parameters can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding calculation method deviate from the spirit and scope of the calculation method of the present invention.
[0112] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.
Claims
1. An image deblurring method based on multi-scale feature extraction and fusion, characterized in that: The following steps are involved: (1) Prepare data, including blurred images and labeled images; build a generative adversarial network model and set parameters; (2) Setting a network structure framework of a generative network and a discriminative network; the network structure framework of the generative network adopts a U-shaped network structure, and the generative network includes an encoder and a decoder. The encoder downsamples and extracts useful detail features in the image for encoding, and the decoder is used to upsample and decode the image; a multi-scale feature extraction and fusion module is also set between the encoder and the decoder, specifically: The blurred image is sent to the generative network. During the optimization training of the generative network, constraints of the image semantic target loss function and the image structure target loss function are added to the generative network, so that the generated image has the same semantic information and structural information as the label image. The discriminant network feeds back the discriminant result to the generative network, driving the generative network to generate a generated image with a complete image structure. The generator network updates the parameters of the generator network according to the feedback from the discriminator network and enters the next iterative training. The generator network and the discriminator network continue to compete and train in the manner described above until the training of the generative adversarial network model converges; (3) Setting the target loss function of the generative adversarial network model: The target loss function in the generative network includes the image semantic target loss function L content , image structure objective loss function L gradient Among them, L content To ensure that the restored images can maintain the same semantic content, L gradient The generated image is constrained to be consistent with the clear image in terms of image structure; the target loss function L in the discriminant network adv It is used to distinguish between generated images and labeled images, so as to improve the discriminant network's ability to distinguish true and false images; (4) The blurred image and the labeled image are fed into the generative adversarial network model, and the discriminant network is used to determine the consistency of the generated image and the labeled image in terms of content; The discriminant network feeds back the judgment results to the generative network. The generative network and the discriminant network compete and learn until the Nash equilibrium is reached and the generative adversarial network model converges. The blurred image is loaded into the generative network that has converged in training, and finally a generated image with a complete structure is obtained.
2. The image deblurring method based on multi-scale feature extraction and fusion according to claim 1, characterized in that: In step (3), the objective loss function of the generative adversarial network model is weighted as: L(G,D)=βL content +λL gradient +αL adv Among them, β, λ and α are L content , L gradient and L adv The weight coefficient of each constraint item is as follows: β=10, λ=12, α=1. The larger the value, the more important the corresponding part.
3. The image deblurring method based on multi-scale feature extraction and fusion according to claim 1, characterized in that: For the generative network, the encoder includes a convolutional layer for mapping the dimension of the input image to 64×64; five downsampling layers are used to downsample and encode the image, and the image resolution is reduced from 256×256 to 16×16; after the above operations are completed, a multi-scale feature extraction and fusion module MSFEFM is introduced to extract more complex and abstract features to capture potential patterns and relationships in the data, reduce the dependence of the generative adversarial network model on training data, and improve the generalization ability; correspondingly, the decoder includes five upsampling layers for upsampling and decoding the image, and the image resolution is increased from 16×16 to 256×256; finally, the deblurred image is reconstructed by a Tanh layer and a convolutional layer; For the discriminant network, a block-based generative adversarial network is adopted as the network structure of the discriminant network, including a flat convolution layer, three downsampling convolution layers, and a feature reconstruction block; the downsampling convolution layer is used to reduce the resolution of the input image and encode local features; a sample normalization layer and a leaky rectified linear unit activation function Leaky ReLU are added after each convolution layer, and the convolution kernel size of each convolution layer is 4×4.
4. The image deblurring method based on multi-scale feature extraction and fusion according to claim 1, characterized in that: The multi-scale feature extraction and fusion module is divided into two parts, including two parallel multi-scale feature extraction modules and a multi-scale feature fusion module; The multi-scale feature extraction module is used to extract features of different scales and levels of the input data; the multi-scale feature fusion module forms a rich and comprehensive feature representation by fusing features from different scales; features of different scales contain information at different levels. Through fusion, the generative adversarial network model can simultaneously utilize low-level detail features and high-level abstract features.
5. The image deblurring method based on multi-scale feature extraction and fusion according to claim 4, characterized in that: The features compressed by the encoder are input into two parallel multi-scale feature extraction modules, and non-overlapping patches are aggregated and shifted according to different scale values in the spatial dimension as follows: Input: feature x, patch; the feature with dimension (B, H, W, C) is input into the multi-scale feature extraction module. The working steps of the multi-scale feature extraction module are as follows: Step 401. Exchange the dimensions of the feature tensor from (B, H, W, C) to (B, W, C, H); B, H, W, C refer to batch size, height, width, and number of channels, respectively; Step 402. Get the batch size, height, width and number of channels of the dimensions; Step 403. Generate non-overlapping patches according to different patch sizes P so as to perform local processing on the input features; Step 404: The dimensions of the new patch are reorganized to form a unified patch dimension format to ensure that subsequent operations can proceed smoothly; Step 405: average the features of the patch to obtain a mean feature patch to extract the overall features of the patch; Step 406: Perform nonlinear transformation on the features of the patch through a multi-layer perceptron (MLP) to extract complex and abstract features; Step 407: Perform layer normalization on the features of the patch to improve the stability and training efficiency of the generative adversarial network model; Step 408: Perform a further nonlinear transformation on the features of the patch through a multi-layer perceptron (MLP) to extract deeper features; Step 409: Calculate the attention distribution, and convert the input value into a value representing the probability distribution through the softmax function; Step 410: Perform a weighted multiplication operation on the mean feature patch and the attention distribution calculated in step 409, and weightedly multiply the mean feature patch and the attention distribution through the attention mechanism to obtain a weighted feature representation; for each mean feature patch, give different importance weights through the attention distribution, and then add these weighted mean feature patches to obtain the final feature representation; Step 411. Normalize the output features along the last dimension so that the length of each vector is 1; Step 412: Generate a mask for filtering and selecting important feature information; Step 413. Apply the generated mask to the output features, and each feature value will be scaled according to the mask value; Step 414: Adjust the dimension of the output features.
6. The image deblurring method based on multi-scale feature extraction and fusion according to claim 4, characterized in that: The workflow of the multi-scale feature fusion module is as follows: Step 501: Use a 1x1 convolution kernel to perform a convolution operation on the input feature x, without changing the spatial size, but only changing the number of channels; Step 502: Apply two multi-scale feature extraction modules using different attention mechanisms and patch values to capture features at more levels; perform three convolution operations on the input feature x in sequence to extract features, and add and fuse all the extracted features; Step 503: Apply the channel attention module ECA to enhance the features of important channels; apply the spatial attention module to enhance the features of important spatial positions and improve the feature expression capability.
7. The image deblurring method based on multi-scale feature extraction and fusion according to claim 1, characterized in that: The generative adversarial network model is run on a computer; the number of batch training images is 2, the learning rate of the generative network and the discriminative network is 0.0001; the slope of the activation function Leaky ReLU is 0.2; the generative adversarial network model uses the Adam optimizer, and its momentum parameters are β1=0.5 and β2=0.999 respectively.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the image deblurring method based on multi-scale feature extraction and fusion according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image deblurring method based on multi-scale feature extraction and fusion as claimed in any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Space and texture collaborative image generation method for small sample torch combustion state
CN121459100A
A spatial and texture collaborative image generation method for small sample torch combustion state
CN121459100B
