A Hybrid Distorted Image Restoration Method Based on Edge Enhancement and Distortion Estimation
By combining edge detection and distortion estimation with feature fusion block methods, the problems of poor compatibility and low efficiency of image restoration methods in the prior art are solved, and high-quality hybrid distortion image restoration is achieved, especially the enhancement of edge details.
Patent Information
- Application Number
- CN202310018848.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-06
- Publication Date
- 2025-07-22
- Estimated Expiration
- 2043-01-06
AI Technical Summary
The existing hybrid distortion image restoration methods have poor compatibility when processing single distorted images, and the existing methods are inefficient or have poor restoration effects, so they cannot accurately describe the mutual interference between the degree of distortion and the processing of distortion.
By extracting the image feature map, edge detection and distortion estimation are performed, the image is reconstructed using edge images and distortion vectors, and image restoration is performed by combining feature fusion blocks to achieve enhanced edge features and accurate estimation of distortion information.
Improve image restoration quality, can stabilize the restoration of mixed distorted images of different types and degrees, avoid interference between multiple distortions, and enhance image edge details.
Smart Images
Figure CN116128755B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and particularly relates to a hybrid distortion image restoration method based on edge enhancement and distortion estimation. Background Art
[0002] Image restoration refers to analyzing a distorted image to reconstruct a high-quality clear image, which can effectively improve the accuracy of advanced vision tasks such as target detection and image segmentation in the subsequent image utilization process, and can be applied to various fields such as industrial production, public safety, and medical engineering.
[0003] Distorted images usually include various distortion methods, including image noise, image blur, compression distortion, etc. Existing hybrid distortion image restoration methods usually include hybrid distortion restoration with known distortion types and hybrid distortion restoration with unknown distortion types. The former operates on the distorted image in a pipeline manner using methods such as denoising, super-resolution, and demosaicing, and the latter uses hybrid multi-distortion enhancement algorithms with dual priors (distortion prior and image prior) learning, SVM to determine the distortion type + pipeline processing method, image restoration methods that integrate distortion detection and restoration, etc.
[0004] However, the hybrid distortion image restoration method with known distortion types cannot handle single-distortion images and has poor compatibility. The hybrid multi-distortion enhancement algorithm with dual prior learning has an extremely long processing time, and the restoration effect is related to the number of iterations, resulting in extremely high usage costs and low efficiency; the SVM to determine the distortion type + pipeline processing method does not consider the mutual interference between distortion types, and new distortion interference will be introduced while removing one distortion, resulting in poor image quality of the restored image; the image restoration method that integrates distortion detection and restoration cannot accurately describe the distortion degree of various distortions, and the connection between distortion detection and restoration is loose, resulting in poor restoration quality. Summary of the Invention
[0005] In view of this, the present invention provides a hybrid distortion image restoration method based on edge enhancement and distortion estimation, which can first extract the image feature map of the input image, estimate the distortion information of the image to obtain a distortion vector, and at the same time use the image feature map to detect the edges of the image to obtain an edge image, and perform reconstruction based on the image feature map, distortion vector, and edge image to obtain the restored image, which can make full use of the edge features of the image, enrich the edge details in the image restoration process, and improve the image restoration quality.
[0006] The technical solution of the present invention is implemented as follows:
[0007] A hybrid distortion image restoration method based on edge enhancement and distortion estimation, comprising:
[0008] Obtain the image to be restored; wherein, the image to be restored includes one or more distortions;
[0009] Input the image to be restored including one or more distortions into an image restoration model, the image restoration model includes a feature extraction module, an edge detection module, a distortion estimation module and an image reconstruction module, wherein:
[0010] The image restoration model inputs the image to be restored into the feature extraction module to obtain the image feature map of the image to be restored;
[0011] Input the image feature map of the image to be restored into the edge detection module, and obtain an edge image according to the output of the edge detection module;
[0012] Input the image feature map of the image to be restored into the distortion estimation module, estimate the distortion information in the image to be restored, and obtain the distortion vector of the image to be restored;
[0013] Input the image feature map, the edge image and the distortion vector into the image reconstruction module, and obtain the restored image of the image to be restored according to the output of the image reconstruction module.
[0014] Optionally, the feature extraction module consists of 1 convolutional layer and m residual blocks, each residual block contains 2 3×3 convolutional layers, and the LeakyReLU activation function is used for activation between the 2 convolutional layers; wherein, 5 < m < 10.
[0015] Optionally, the edge detection module consists of n residual blocks, 2 convolutional layers and 1 LeakyReLU activation function; wherein, 5 < n < 10.
[0016] Optionally, the distortion estimation module consists of 1 downsampling layer, 1 global average pooling layer, 3 fully connected layers, 2 LeakyReLU activation functions and 1 Sigmoid activation function; wherein, the downsampling layer uses a 2×2 convolutional layer with a stride of 2.
[0017] Optionally, the image reconstruction module consists of 3 convolutional layers, p feature fusion blocks and 1 LeakyReLU activation function; wherein, 20 < p < 40.
[0018] Optionally, each of the feature fusion blocks consists of 4 convolutional layers, 2 LeakyReLU activation functions, 1 fully connected layer and 1 Sigmoid activation function.
[0019] Optionally, during the training process of the image restoration model, it further includes:
[0020] Take the sample image as the input of the image restoration model to be trained. According to the comparison between the output of the image restoration model to be trained and the original image, determine the model loss function as:
[0021] where N is the batch size of the sample images, is the image restored by the image restoration model to be trained, is the sample image, λ is the first balance factor, is the edge image of the sample image estimated by the distortion estimation module, is the true edge image of the sample image, μ is the second balance factor, is the distortion vector of the sample image estimated by the distortion estimation module, is the true distortion vector of the sample image;
[0022] Perform iterative training on the image restoration model to be trained;
[0023] According to the training results of the iterative training, determine the final image restoration model.
[0024] Beneficial effects:
[0025] (1) The present invention estimates the edge information and distortion information of the distorted image, and reconstructs the image using the obtained edge image and distortion vector. On the one hand, through the processing branch of the edge feature map in the feature fusion block, the edge details of the restored image are enhanced, and the restoration effect is better; on the other hand, a regulation variable is generated based on the distortion vector, and weights are assigned to the processing branches of the image feature maps in each feature fusion block. By jointly training each module, the network can restore mixed-distortion images of different types and degrees, achieving global optimality, with stable restoration results and avoiding interference between multiple distortions.
[0026] (2) The distortion estimation module of the present invention can judge the type of distortion contained in the image to be restored and estimate the corresponding degree of distortion, so as to more accurately describe the distortion information of the image.
[0027] (3) The present invention fuses the edge feature map into the image feature map, makes full use of the edge information of the image, and gradually enhances the edge features of the image during the image reconstruction process, making the edge details of the restored image richer. Description of the Drawings
[0028] Figure 1 is a schematic diagram of the main process of the hybrid distortion image restoration method based on edge enhancement and distortion estimation according to an embodiment of the present invention.
[0029] Figure 2 is a schematic diagram of the structure of the image restoration model according to an embodiment of the present invention.
[0030] Figure 3 It is a schematic diagram of the main process of the method for using the image restoration model according to an embodiment of the present invention.
[0031] Figure 4 It is a schematic diagram of the structure of the feature extraction module according to an embodiment of the present invention.
[0032] Figure 5 It is a schematic diagram of the structure of the edge detection module according to an embodiment of the present invention.
[0033] Figure 6 It is a schematic diagram of the structure of the distortion estimation module according to an embodiment of the present invention.
[0034] Figure 7 It is a schematic diagram of the structure of the image reconstruction module according to an embodiment of the present invention.
[0035] Figure 8 It is a schematic diagram of the structure of the feature fusion block of the image reconstruction module according to an embodiment of the present invention. Detailed implementation manners
[0036] The present invention will be described in detail below with reference to the accompanying drawings and by way of examples.
[0037] The present invention provides a method for restoring a mixed-distortion image based on edge enhancement and distortion estimation. As Figure 1 shown, the method for restoring a mixed-distortion image based on edge enhancement and distortion estimation of the present invention includes the following steps:
[0038] Step 11: Obtain the image to be restored; wherein, the image to be restored includes one or more types of distortions.
[0039] In an embodiment of the present invention, the image to be restored has mixed distortions of unknown types and unknown degrees, and the distortion modes may be fog, Gaussian blur, Gaussian noise, JPEG compression distortion, etc.
[0040] Furthermore, the image to be restored may include one or more types of distortions, or the image to be restored may not contain any distortions. The image restoration model of the present invention can restore an image without distortions or with mixed distortions of unknown types and unknown degrees.
[0041] Step 12: Input the image to be restored including one or more types of distortions into the image restoration model. The image restoration model to be trained uses the edge detection module to perform edge detection on the image feature map of the image to be restored, and uses the distortion estimation module to calculate the distortion vector of the image feature map of the image to be restored.
[0042] In an embodiment of the present invention, the image restoration model is used to restore the image to be restored. As Figure 2As shown in the figure, it includes a feature extraction module, an edge detection module, a distortion estimation module, and an image reconstruction module. The feature extraction module is used to extract the image feature map of the image to be restored. The edge detection module performs edge detection on the image feature map to obtain an edge image. The distortion estimation module estimates the distortion degree of the image feature map to obtain a distortion vector. The image feature map, the edge image, and the distortion vector are input into the image reconstruction module for feature fusion to reconstruct the restored image after restoration of the image to be restored.
[0043] In the embodiment of the present invention, as Figure 3 shown, the method for using the image restoration model of the present invention includes the following steps:
[0044] Step 31: Input the image to be restored into the feature extraction module to obtain the image feature map of the image to be restored.
[0045] In the embodiment of the present invention, the feature extraction module of the image restoration model is composed of 1 convolutional layer and m (5 < m < 10) residual blocks. As Figure 4 shown, the convolutional kernel of the convolutional layer can be set as needed. For example, 3×3, 5×5, 7×7, etc. Each residual block contains 2 convolutional layers of 3×3. The LeakyReLU function is used for activation between the 2 convolutional layers, and a skip connection is used to connect the input and the output. After the image to be restored is processed by the feature extraction module, an image feature map F of W×H×C is obtained, where W, H, and C respectively represent the width, height, and number of channels of the image feature map.
[0046] Step 32: Input the image feature map of the image to be restored into the edge detection module, and obtain an edge image according to the output of the edge detection module.
[0047] In the embodiment of the present invention, the edge detection module of the image restoration model is composed of n (5 < n < 10) residual blocks, 2 convolutional layers, and 1 LeakyReLU activation function. As Figure 5 shown, the convolutional kernel of the convolutional layer can be set as needed. For example, 3×3, 5×5, 7×7, etc. The edge detection module uses the obtained image feature map F to estimate the edge of the image to be restored to obtain an edge image. Specifically:
[0048] First, the image feature map F extracts the edge features of the image through n residual blocks; then, the edge features are processed by 2 convolutional layers of 3×3 using the LeakyReLU activation function to generate an edge image The edge image can be used as prior information for the image reconstruction module to assist in the image reconstruction process.
[0049] Step 33: Input the image feature map of the image to be restored into the distortion estimation module to estimate the distortion information in the image to be restored, and obtain the distortion vector of the image to be restored.
[0050] In the embodiment of the present invention, the distortion estimation module of the image restoration model is composed of 1 downsampling layer, 1 global average pooling layer, 3 fully connected layers, 2 LeakyReLU activation functions and 1 Sigmoid activation function. As Figure 6 shown, the downsampling layer uses a 2×2 convolutional layer with a stride of 2. The distortion estimation module uses the obtained image feature map to estimate the distortion degree of various distortions contained in the image to be restored, and obtains a distortion vector. The distortion vector D includes distortion factors d of various distortions. The distortion factor d represents the distortion degree of the distortion, and d ∈ [0, 1]. The larger d is, the greater the distortion degree of the corresponding type of distortion. d = 0 means that there is no corresponding type of distortion. For example, the distortion methods added to the sample image include fog, Gaussian blur, Gaussian noise, and JPEG compression distortion. The distortion vector D = [d1, d2, d3, d4], where d1, d2, d3, and d4 respectively represent the distortion factors of fog, Gaussian blur, Gaussian noise, and JPEG compression distortion. Among them, and Specifically:
[0051] First, use a 2×2 convolutional layer with a stride of 2 as the downsampling layer to downsample the input image feature map F, and at the same time expand the number of channels to 4 times the original; second, use the global average pooling layer to compress the downsampled feature map in the spatial dimension to obtain a one-dimensional vector; then, use 3 fully connected layers to calculate this one-dimensional vector, and use the LeakyReLU function to activate between every 2 fully connected layers; finally, use the Sigmoid function to map the calculation result to [0, 1] to obtain the distortion vector D. The distortion vector D can be used as prior information for the image reconstruction module to adjust the image reconstruction process.
[0052] Step 34: Input the image feature map, the edge image, and the distortion vector into the image reconstruction module, and obtain the restored image of the image to be restored according to the output of the image reconstruction module.
[0053] In the embodiment of the present invention, the image reconstruction module of the image restoration model is composed of 3 convolutional layers, p (20 < p < 40) feature fusion blocks and 1 LeakyReLU activation function. As Figure 7As shown, the convolution kernels of the convolutional layer can be set as needed. For example, 3×3, 5×5, 7×7, etc. The image reconstruction module uses the image feature map output by the feature extraction module, the edge image output by the edge detection module, and the distortion vector output by the distortion estimation module to perform image reconstruction. The distortion vector can adjust the image reconstruction process to adapt to the reconstruction of images with different distortion types and degrees; the edge image can guide the image reconstruction process, enhance the edge features of the image, and reconstruct a restored image with clearer edge details. Specifically:
[0054] First, the edge image E generates an image edge feature map K through a 3×3 convolutional layer; then, the image feature map F, the image edge feature map K, and the distortion vector D are input into a series of feature fusion blocks; finally, the image feature map and the image edge feature map processed by the series of feature fusion blocks are fused, and after being processed by 2 3×3 convolutional layers using the LeakyReLU activation function, a restored image is generated. Through the adjustment of the distortion vector and the guidance of the edge image, the image restoration module can reconstruct a high-quality restored image.
[0055] Furthermore, each feature fusion block consists of 4 convolutional layers, 2 LeakyReLU activation functions, 1 fully connected layer, and 1 Sigmoid activation function. As Figure 8 shown, the input of the feature fusion block is the i-th level image feature map F i , the image edge feature map K i and the distortion vector D, and the output is the (i + 1)-th level image feature map F i+1 and the image edge feature map K i+1 , where i represents the order of the feature fusion block. Each feature fusion block includes an image feature map processing branch, an image edge feature map processing branch, and a distortion vector processing branch. The image feature map F i and the image edge feature map K i are respectively processed by the image feature map processing branch and the image edge feature map processing branch, and the distortion vector D is processed by the distortion vector processing branch. The image feature map processing branch and the image edge feature map processing branch respectively consist of 2 3×3 convolutional layers and 1 LeakyReLU activation function, and the distortion vector processing branch consists of 1 fully connected layer and 1 Sigmoid activation function. Taking the feature fusion block in Figure 8 as an example, the i-th level image edge feature map K i enters the image edge feature map processing branch, and after being processed by 2 3×3 convolutional layers using the LeakyReLU activation function, it is fused with the image edge feature map K i to obtain the (i + 1)-th level image edge feature map K i+1 , that is, K i+1= f K (K i ) + K i , f K represents the image edge feature map processing branch. The distortion vector D enters the distortion vector processing branch, and after being processed by a fully connected layer and a Sigmoid activation function, the adjustment variable α is obtained. The image feature map F of the i-th level i enters the image feature map processing branch, and after being processed by 2 3×3 convolutional layers activated by the LeakyReLU activation function, it is multiplied by the adjustment variable α. The multiplication result is fused with the image feature map F of the i-th level i and then fused with the image edge feature map K of the (i + 1)-th level i+1 to obtain the image feature map F of the (i + 1)-th level i+1 , that is, F i+1 = f F (F i ) × α + F i + K i , f F represents the image feature map processing branch.
[0056] In an embodiment of the present invention, during the training process of the image restoration model, the training data includes a sample image and a corresponding original image. The original image is a clear image obtained under different scenarios, and the sample image is obtained by adding one or more different degrees of distortion processing to the original image.
[0057] Further, the sample image is a mixed distortion image with any one, two, three, or four of the four distortion methods of adding fog, Gaussian blur, Gaussian noise, and JPEG compression distortion, or without distortion. The four distortion methods can be added to the original image according to different distortion degrees to obtain the sample image. For example, fog is added to the original image according to the atmospheric scattering model, the atmospheric light value is A = 0.9, and the atmospheric scattering coefficient β is randomly taken within the range of [0, 0.2]. For another example, Gaussian blur is added to the original image, and the standard deviation σ1 of the Gaussian blur is randomly taken within the range of [0, 3]. For yet another example, Gaussian noise is added to the original image, and the standard deviation σ2 of the Gaussian noise is randomly taken within the range of [0, 30]. For still another example, JPEG compression distortion is added to the original image, and the quality factor q of the JPEG compression distortion is randomly taken within the range of [10, 1000]. The smaller the quality factor q, the greater the compression degree and the higher the JPEG compression distortion degree. Correspondingly, the sample images include 16 categories, namely: sample images with all four types of mixed distortion of fog, Gaussian blur, Gaussian noise, and JPEG compression distortion, a total of 1 category; sample images with any three types of mixed distortion among the above four types of distortion, a total of 4 categories; sample images with any two types of mixed distortion among the above four types of distortion, a total of 6 categories; sample images with any one type of single distortion among the above four types of distortion, a total of 4 categories; sample images without distortion (same as the original image), a total of 1 category.
[0058] In the embodiment of the present invention, during the training process of the image restoration model, the image feature map processing branch in each feature fusion block can be used as a functional unit. The image reconstruction module includes a total of p functional units. By adjusting the parameters of the fully connected layer of the distortion vector processing branch multiple times, the adjustment variable α can be adjusted, and the best weighting coefficients can be assigned to each functional unit, so that the image reconstruction module can restore mixed distortion images of different types and degrees.
[0059] In the embodiment of the present invention, according to the method for restoring mixed distortion images based on edge enhancement and distortion estimation of the present invention, the feature extraction module, distortion estimation module, edge detection module, and image reconstruction module of the image restoration model are jointly trained. The sample image is used as the input of the image restoration model to be trained. According to the comparison between the output of the image restoration model to be trained and the original image, the image restoration model to be trained is iteratively trained; according to the training results of the iterative training, the final image restoration model is determined. Specifically:
[0060] Define the image restored by the image restoration model to be trained as The sample image is The estimated edge image is The true edge image corresponding to the sample image is The estimated distortion vector is The true distortion vector corresponding to the sample image is The model loss function is shown as follows:
[0061]
[0062] In the above formula, N is the batch size of the training data, λ is the first balance factor, and μ is the second balance factor. Respectively, λ = 0.5 and μ = 0.005 are taken.
[0063] The training iterator selects the Adam optimizer to perform iterative training on the image restoration model to be trained until Ltotal is stable, and determines the final image restoration model for restoring single-distortion or mixed-distortion images during subsequent use.
[0064] Step 13: Obtain the restored image of the image to be restored according to the output of the image restoration model.
[0065] In the embodiment of the present invention, the restored image of the image to be restored is determined according to the output of the image restoration model.
[0066] In summary, the above is only a preferred embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A hybrid distortion image restoration method based on edge enhancement and distortion estimation, characterized in that Including: Obtain an image to be restored; wherein, the image to be restored includes one or more distortions; Input the image to be restored including one or more distortions into an image restoration model, the image restoration model including a feature extraction module, an edge detection module, a distortion estimation module, and an image reconstruction module, wherein: The image restoration model inputs the image to be restored into the feature extraction module to obtain an image feature map of the image to be restored; Input the image feature map of the image to be restored into the edge detection module, and obtain an edge image according to the output of the edge detection module; Input the image feature map of the image to be restored into the distortion estimation module, estimate the distortion information in the image to be restored, and obtain a distortion vector of the image to be restored; Input the image feature map, the edge image, and the distortion vector into the image reconstruction module, and obtain a restored image of the image to be restored according to the output of the image reconstruction module; During the training process of the image restoration model, it further includes: Take a sample image as the input of the image restoration model to be trained, and determine the model loss function according to the comparison between the output of the image restoration model to be trained and the original image as: where N is the batch size of the sample images, is the image restored by the image restoration model to be trained, is the sample image, λ is the first balance factor, is the edge image of the sample image estimated by the distortion estimation module, is the true edge image of the sample image, μ is the second balance factor, is the distortion vector of the sample image estimated by the distortion estimation module, is the true distortion vector of the sample image; Perform iterative training on the image restoration model to be trained; determine the final image restoration model according to the training results of the iterative training.
2. The method according to claim 1, characterized in that, The feature extraction module is composed of 1 convolutional layer and m residual blocks, each residual block contains 2 3×3 convolutional layers, and a LeakyReLU activation function is used for activation between the 2 convolutional layers; wherein, 5 < m < 10.
3. The method according to claim 1, characterized in that, The edge detection module is composed of n residual blocks, 2 convolutional layers, and 1 LeakyReLU activation function; wherein, 5 < n < 10.
4. The method according to claim 1, characterized in that, The distortion estimation module is composed of 1 downsampling layer, 1 global average pooling layer, 3 fully connected layers, 2 LeakyReLU activation functions, and 1 Sigmoid activation function; wherein, the downsampling layer uses a 2×2 convolutional layer with a stride of 2.
5. The method according to claim 1, characterized in that, The image reconstruction module is composed of 3 convolutional layers, p feature fusion blocks, and 1 LeakyReLU activation function; wherein, 20 < p < 40.
6. The method according to claim 5, characterized in that, Each of the feature fusion blocks is composed of 4 convolutional layers, 2 LeakyReLU activation functions, 1 fully connected layer, and 1 Sigmoid activation function.