Image enhancement method and system based on learnable curvature map

Abstract

The present invention proposes an image enhancement method and system based on a learnable curvature map, which belongs to the technical field of computer science image processing. The image enhancement method includes: step 1, constructing a two-stage denoising model; step 2, transferring the image data to be analyzed to the denoising model; step 3, in the first stage, performing feature extraction using the denoising model to obtain noise level estimation and noise distribution estimation; step 4, calculating Gaussian curvature based on the obtained noise level estimation and noise distribution estimation; step 5, in the second stage, using the denoising model based on Gaussian curvature to combine local information and global information to obtain denoised image data; step 6, outputting denoised image data. The present invention can not only effectively remove Gaussian noise from an image, but also improve the denoising effect of different noise types through model training, and has strong robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of computer science image processing, and in particular to an image enhancement method and system based on a learnable curvature map. Background Art

[0002] With the rapid development of imaging technology and higher bandwidth, people's demand for high-precision and high-definition images is also increasing. However, during the shooting and transmission process, cameras are polluted by various unknown noises caused by hardware, environment, ISP processing, etc., resulting in a decrease in image quality.

[0003] Image denoising is to remove noise and restore the real image. The problem of denoising a known image is mathematically an inverse process, and its solution is not unique. Since noise, texture, and edges are all high-frequency information, some detailed features will inevitably be lost during the denoising process. Therefore, restoring meaningful features from existing information to achieve high-quality images is an important issue that is being actively addressed in various fields today.

[0004] A large number of experiments have shown that feature extraction based on convolutional neural networks has good performance in noise removal, but the research is mainly focused on Gaussian noise, the noise form is single and lacks generalization; at the same time, the frontier uses Gaussian distribution or Poisson distribution to estimate the noise level, which has achieved relatively accurate research. However, the noise types in actual images are complex and unknown. Adding convolutional layers to improve network training capabilities will increase the computational burden, and adding feature supervision modules will lead to the loss of high-frequency details of the image, all of which increase the technical difficulty of image denoising. Summary of the invention

[0005] Purpose of the invention: Propose an image enhancement method and system based on a learnable curvature map to solve the above problems existing in the prior art. By classifying the noise based on a two-stage denoising model, while increasing feature supervision, curvature supervision is added to achieve greatly improved control of detail texture effects while denoising.

[0006] Technical solution: In the first aspect, a method for image enhancement based on a learnable curvature map is proposed, and the method comprises the following steps:

[0007] Step 1: Construct a two-stage denoising model; the construction process includes:

[0008] Step 1.1, construct a noise estimation subnetwork for performing feature extraction and obtaining noise level estimation and noise distribution estimation;

[0009] Step 1.2: Construct a denoising subnetwork for performing a denoising operation on a real image based on the output data of the noise estimation subnetwork.

[0010] Step 2: The image data to be analyzed is transferred to the denoising model;

[0011] Step 3: In the first stage, feature extraction is performed using the denoising model to obtain noise level estimation and noise distribution estimation;

[0012] Step 4: Calculate Gaussian curvature based on the obtained noise level estimate and noise distribution estimate;

[0013] Step 5: In the second stage, the denoising model is used based on Gaussian curvature to combine local information and global information to obtain denoised image data;

[0014] Step 6: Output the denoised image data.

[0015] In some implementations of the first aspect, the noise estimation subnetwork includes, in order: a convolutional layer, an attention mechanism, a convolutional layer, a noise output module, and a Gaussian curvature module; the denoising subnetwork includes: a deep network, a shallow network, a residual module, and a multi-scale module; the deep network is used to extract the noise residual features of the input mixed image; and the shallow network is used for multi-feature extraction.

[0016] In the first stage, the noise estimation subnetwork uses a 4-layer neural network model with a 3*3 convolution kernel in the convolution layer, and uses padding operation to obtain network features of equal size in each layer; then, the attention mechanism is used before the last convolution layer to perform weighted extraction on the noise distribution estimate; finally, the Gaussian curvature is used as a priori to calculate the Gaussian curvature.

[0017] In the second stage, the obtained Gaussian curvature, noise level estimation, and noisy image are input into the denoising subnetwork, and the noise at the specific position obtained in the first stage is processed at the same time.

[0018] In some implementations of the first aspect, after the two-stage denoising model is constructed, the method further includes step 1.3 of performing performance optimization training on the denoising model. The process of performing performance optimization training includes the following steps:

[0019] Step 1.3.1, build an image test set;

[0020] Step 1.3.2: Use the image test set to train the noise estimation subnetwork to obtain optimized network parameters, noise level estimation, and Gaussian curvature;

[0021] Step 1.3.3, input the noise level estimation and Gaussian curvature into the denoising sub-network for training;

[0022] Step 1.3.4, negatively feed back the training result in step 1.3.3 to the noise estimation subnetwork;

[0023] Step 1.3.5, the noise estimation subnetwork performs adjustment control on relevant parameters according to the received results;

[0024] Step 1.3.6: Construct a loss function and determine the degree of model performance optimization based on the calculation results of the loss function;

[0025] Step 1.3.7: When the model performance optimization degree meets the training completion conditions, the final network model parameters are obtained and the performance optimization training is ended.

[0026] Among them, the loss function L1 used in the noise estimation subnetwork training includes: the noise distribution loss obtained by training and the noise estimation loss, that is:

[0027]

[0028] In the formula, y nl Represents the noise level of the real image; represents the estimated noise level; y nd Represents the noise level of the real image; Represents the estimated noise level.

[0029] The loss function L2 used in the denoising sub-network training includes: the difference between the predicted value output by the denoising sub-network and the clean image, that is:

[0030]

[0031] In the formula, represents the output predicted image; x represents the clean image.

[0032] In the second aspect, a learnable curvature map-based image enhancement system is proposed, which is used to implement an image enhancement method based on a learnable curvature map. The system includes the following modules:

[0033] The model building module is set to build a two-stage denoising model according to actual application requirements;

[0034] A first data analysis module is configured to perform feature extraction on the image data to be analyzed and obtain relevant parameters;

[0035] A second data analysis module is configured to perform image denoising processing based on the output data of the first data analysis module;

[0036] The data output module is configured to output the processing result of the second data analysis module.

[0037] In some implementable methods of the second aspect, when performing image enhancement, first, according to the image requirements in the actual application process, a two-stage denoising model is constructed using a model building module; then, a first data analysis module is used to perform feature extraction on the image data to be analyzed to obtain noise level estimation and noise distribution estimation, and Gaussian learning curvature is introduced to obtain Gaussian curvature; then, based on the output result of the first data analysis module, a second data analysis module is used to combine local information and global information to obtain denoised image data; finally, the denoised and enhanced image data is output through the data output module.

[0038] Beneficial effects: The present invention proposes an image enhancement method and system based on a learnable curvature map. In view of the fact that real noise level denoising is prone to detail loss and difficult to accurately estimate, curvature learning is used to estimate the noise distribution to obtain more accurate denoising strength and texture details; at the same time, the residual network and multi-scale module are combined for denoising to obtain a high-quality denoised image. The present invention can not only effectively remove Gaussian noise from the image, but also improve the denoising effect of different noise types through model training, and has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the image enhancement method based on learnable curvature map of the present invention.

[0040] Figure 2 Schematic diagram of the image enhancement method based on learnable curvature map of the present invention. DETAILED DESCRIPTION

[0041] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0042] Embodiment 1

[0043] In one embodiment, in view of the fact that real noise level denoising is prone to detail loss and difficult to accurately estimate, an image enhancement method based on a learnable curvature map is proposed. The data processing process is divided into two processing stages. By estimating the image noise distribution, more accurate denoising strength and texture details are obtained, thereby obtaining more representative image features. The process of performing image enhancement to obtain high-quality image data includes the following steps:

[0044] Step 1: Build a two-stage denoising model based on deep learning;

[0045] Specifically, the denoising model includes: a noise estimation subnetwork and a denoising subnetwork. The noise estimation subnetwork is used to perform feature extraction, obtain noise level estimation and noise distribution estimation, and calculate Gaussian curvature; the denoising subnetwork is used to perform image denoising operations based on the data obtained by the noise estimation subnetwork.

[0046] The noise estimation subnetwork includes: 4 layers of convolutional layers with 3*3 convolution kernels, attention mechanism, convolutional layer, noise output module, Gaussian curvature module in order; the denoising subnetwork includes: deep network, shallow network, residual module, multi-scale module. The deep network is used to extract the noise residual features of the input mixed image, and the shallow network is used to perform multi-feature extraction to obtain more comprehensive image feature extraction.

[0047] Step 2: The image data to be analyzed is transferred to the denoising model;

[0048] Step 3. In the first stage, the denoising model is used to perform feature extraction to obtain noise level estimation and noise distribution estimation; that is, in the first stage, the noise estimation subnetwork in the denoising model is used to perform feature extraction based on the attention mechanism to obtain noise level estimation and noise distribution estimation, and the Gaussian curvature is calculated by estimating the clean image of the input noisy image after removing the noise distribution.

[0049] Specifically, the noise estimation subnetwork uses a 4-layer neural network model with a 3*3 convolution kernel in the convolution layer, and uses padding operation to obtain network features of equal size in each layer; then, the attention mechanism is used before the last convolution layer to perform weighted extraction on the noise distribution estimation, thereby increasing the attention of the noise estimation subnetwork to meaningful information, thereby enhancing network performance.

[0050] Step 4: Calculate Gaussian curvature based on the obtained noise level estimate and noise distribution estimate;

[0051] Specifically, since the traditional encoder-decoder lacks the ability to preserve spatial image details and texture structures, based on the idea of ​​Gaussian curvature, this embodiment uses a Gaussian curvature module to take the learnable Gaussian curvature as a priori to enhance the ability of real images to retain edge details in terms of noise distribution estimation.

[0052] Based on the obtained noise level estimate and noise distribution estimate, the Gaussian curvature is calculated by estimating a clean image of the input noisy image with the noise distribution removed.

[0053] Step 5: In the second stage, the denoising model is used based on Gaussian curvature to combine local information and global information to obtain denoised image data;

[0054] Specifically, the obtained Gaussian curvature, noise level estimation, and noisy image are input into the denoising sub-network, and the noise at the specific position obtained in the first stage is processed. The denoising network is mainly divided into a deep network and a shallow network. The deep network extracts the noise residual features of the input mixed image, and the shallow network is used for multi-feature extraction to obtain a more comprehensive image feature extraction.

[0055] Step 6: Output the denoised image data.

[0056] In actual application scenarios, the noise model in real image data corresponds to the expression:

[0057] y=x+q(x)

[0058] In the formula, y represents the real noisy image; x represents the clean image; q(x) represents the noise introduced in the real environment. In order to remove the noise q(x) in the real image, the unknown noise distribution of each part of the image and its noise level estimation pose great difficulties for the mapping learning of the denoising model. Therefore, in order to obtain high-quality image data, the noise distribution estimation is performed before the denoising module to implement the data enhancement operation on the real image containing noise. Based on the image enhancement method proposed in the present invention, for the received real image, a two-stage denoising model is first constructed, and the noise level estimation is performed in the first stage of data processing, and the image denoising is performed in the second stage.

[0059] Specifically, Figure 2 As shown in the figure, first, the noise estimation sub-network uses a 3*3 convolution kernel to build a 4-layer neural network model, and uses padding to obtain network features of equal size. Then, feature extraction is performed based on the attention mechanism, and the noise level estimation and noise distribution estimation values ​​are output. The Gaussian curvature is calculated by estimating the clean image of the input noisy image and removing the noise distribution. Then, the mixed image of the noise image, noise level estimation and Gaussian curvature is used as the input of the denoising sub-network. The denoising network uses a deep network to extract the noise residual features of the input mixed image, and the shallow network is used for multi-feature extraction. The detailed information and global information are combined to enrich the features in image restoration. The two paths are combined and sent to the CSAM (Curvature Supervised Attention Module) module, and finally enter the convolution layer for reconstruction to generate a residual image, which is connected to the original image and outputs the final denoised image.

[0060] This embodiment effectively enhances image details and improves the quality of the final image data during image data processing by introducing the concept of learning curvature. First, the noise estimation subnetwork performs feature extraction based on the attention mechanism to obtain noise level estimation and noise distribution estimation, and calculates Gaussian curvature by estimating the clean image of the input noisy image and removing the noise distribution; the second stage model is constructed based on the convolution layer, and the mixed image of the noise image, noise level estimation and Gaussian curvature is used as the input of the denoising subnetwork. While considering the detail information and global information, the curvature supervised attention module is used for effective denoising prior modeling, and finally the residual connection is introduced to obtain a high-quality denoised image.

[0061] Embodiment 2

[0062] In a further embodiment based on the first embodiment, in order to improve the image enhancement performance of the denoising model, performance optimization training is performed on it, and the corresponding training process includes the following steps:

[0063] Step 1: Build an image test set;

[0064] Step 2: Use the image test set to train the noise estimation subnetwork to obtain optimized network parameters, noise level estimation, and Gaussian curvature;

[0065] Step 3: Input the noise level estimation and Gaussian curvature into the denoising sub-network for training;

[0066] Step 4: Negatively feed back the training result in step 3 to the noise estimation subnetwork;

[0067] Step 5: The noise estimation subnetwork performs adjustment control on relevant parameters according to the received results;

[0068] Step 6: Construct a loss function and determine the degree of model performance optimization based on the calculation results of the loss function;

[0069] Step 7: When the model performance optimization degree meets the training completion conditions, the final network model parameters are obtained and the performance optimization training is ended.

[0070] Among them, the loss function L1 used in the end-to-end noise estimation subnetwork training includes: the noise distribution loss obtained by training and the noise estimation loss, that is:

[0071]

[0072] In the formula, y nl Represents the noise level of the real image; represents the estimated noise level; y nd Represents the noise level of the real image; Represents the estimated noise level.

[0073] The loss function L2 used in the denoising sub-network training includes: the difference between the predicted value output by the denoising sub-network and the clean image, that is:

[0074]

[0075] In the formula, represents the output predicted image; x represents the clean image.

[0076] In a preferred embodiment, when constructing an image test set, the BSD500 and Urban100 datasets are used to test network performance, including Ground Truth for image segmentation and contour extraction, involving a variety of noise types including Gaussian noise, salt and pepper noise, exponential noise, impulse noise, etc. Clean images are selected to add noise distribution and noise level, which are used as training inputs for the network model, forming a pair of training and validation samples with the sample dataset.

[0077] The noise estimation subnetwork is trained using the image test set to determine the network parameters, obtain the noise level estimation and Gaussian curvature, input them into the denoising subnetwork for training, and negatively feedback them to the noise estimation network to adjust and control the relevant parameters. When the loss function meets the requirements, the network training is completed and the final two-stage network parameters are obtained.

[0078] Based on the constructed loss function, the Adam optimizer is used for end-to-end training. The initial learning rate of the noise estimation subnetwork is set to e-4, and it is reduced to half of the original value after every 6 epochs until it reaches 1.25e-5, and the mini-batch size is fixed to 64; the learning rate of the denoising subnetwork is also set to e-4, and it is decayed by half every 100,000 iterations. It ends when the learning rate is less than e-5, and the mini-batch size is also fixed to 64.

[0079] After the network model is trained, the test data set is input into the noise estimation subnetwork to obtain the noise level estimation and Gaussian curvature, and then the original image is fused as a mixed image and input into the denoising network to obtain the final denoised image. The scheme of the present invention is compared with the existing method based on the PSNR evaluation index, which verifies that the scheme has a higher signal-to-noise ratio and subjective effect.

[0080] Embodiment 3

[0081] In one embodiment, a learnable curvature map-based image enhancement system is proposed, which is used to implement an image enhancement method based on a learnable curvature map. The system includes the following modules: a model construction module, a first data analysis module, a second data analysis module, and a data output module. The model construction module is used to construct a two-stage denoising model according to image enhancement requirements; the first data analysis module is used to perform feature extraction to obtain noise level estimation, noise distribution estimation, and Gaussian curvature; the second data analysis module is used to perform image denoising; and the data output module is used to output image data after image denoising and enhancement.

[0082] Specifically, in the process of performing image enhancement, first, according to the image requirements in the actual application process, a two-stage denoising model is constructed using a model construction module; then, the first data analysis module is used to perform feature extraction on the image data to be analyzed, and the noise level estimation and noise distribution estimation are obtained, and Gaussian learning curvature is introduced to obtain Gaussian curvature; then, based on the output result of the first data analysis module, the second data analysis module is used to combine local information and global information to obtain denoised image data; finally, the denoised and enhanced image data is output through the data output module.

[0083] In a further embodiment, in order to improve the image enhancement performance of the denoising model, a performance optimization module is also included, which optimizes and adjusts the parameters of the denoising model by constructing a training set and optimizing parameter feedback.

[0084] In a preferred embodiment, in a drone-based intelligent inspection task for power grids, the photos / videos taken by the drone during the inspection process are often limited by weather conditions in the actual environment, such as fog, rainy season, dim light, etc., which can easily cause problems such as blurred image details and poor content visibility, posing severe challenges to subsequent image processing. To this end, the image enhancement method and system based on a learnable curvature map proposed in the present invention re-balances and enhances local features through attention, and combines a multi-scale upsampling series structure to achieve information exchange between different stages, thereby obtaining local and global features of the image, and obtaining rich feature map information, effectively overcoming the problem of blurred object contours and low overall image quality, meeting the richness of details of the drone-collected images, and providing a solid foundation for subsequent image recognition and processing.

[0085] In summary, the image enhancement method and system proposed in the present invention maintains the contrast and texture edge of the image by introducing curvature learning, effectively obtaining more meaningful image features; at the same time, the curvature supervised attention module is used for effective denoising prior modeling, and finally the residual connection is introduced to obtain a high-quality denoised image. In addition, the present invention can not only effectively remove Gaussian noise from the image, but also improve the denoising effect of different noise types through model training, and has strong robustness.

[0086] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. An image enhancement method based on a learnable curvature map, characterized in that: The following steps are involved: Step 1: Build a two-stage denoising model including a noise estimation subnetwork and a denoising subnetwork, and apply it to the image data process after optimizing the model performance. The noise estimation subnetwork is used to perform feature extraction to obtain noise level estimation and noise distribution estimation, and includes in order: 4 convolution layers with 3*3 convolution kernels, attention mechanism, convolution layer, noise output module, Gaussian curvature module. In the process of performing noise level estimation and noise distribution estimation, padding operation is used in the convolution layer to obtain network features of equal size in each layer, and attention mechanism is used before the last convolution layer to perform weighted extraction on the noise distribution estimation; the denoising subnetwork is used to perform denoising operation on the real image based on the output data of the noise estimation subnetwork, including The invention discloses a method for extracting noise from a mixed image by using a convolutional layer. The method comprises the following steps: a deep network, a shallow network, a residual module, and a multi-scale module. In the process of denoising, the received data is synchronously input into the deep network and the shallow network, and then sequentially input into two serial residual modules after passing through the shallow network, and then sequentially enter into two serial multi-scale modules after passing through the deep network, and then the two outputs are combined and sent to the CSAM module, and finally enter the convolutional layer for reconstruction, generate a residual image, connect the original image, and output the final denoised image; wherein the deep network is used to extract the noise residual features of the input mixed image; and the shallow network is used to extract multi-scale features; Step 2: The image data to be analyzed is transferred to the denoising model; Step 3: In the first stage, the denoising model is used to perform feature extraction to obtain noise level estimation and noise distribution estimation; wherein, the noise estimation subnetwork uses a 4-layer neural network model with a 3*3 convolution kernel as the convolution layer, and uses padding operation to obtain network features of equal size in each layer; then, the attention mechanism is used before the last convolution layer to perform weighted extraction on the noise distribution estimation; finally, the Gaussian curvature is used as a priori to calculate the Gaussian curvature; Step 4: Calculate Gaussian curvature based on the obtained noise level estimate and noise distribution estimate; Step 5. In the second stage, based on Gaussian curvature, a denoising model is used to combine local information and global information, and the obtained Gaussian curvature, noise level estimation, and noise image are input into the denoising subnetwork. At the same time, the noise at the specific position obtained in the first stage is processed to obtain the denoised image data; Step 6: Output denoised image data; In the process of performing performance optimization on the denoising model, a staged training method is adopted, and the training output data of the first stage is used for the performance training of the second stage, which correspondingly includes the following steps: Step 1.3.1, build an image test set; Step 1.3.2: Use the image test set and the loss function L1 to train the noise estimation subnetwork to obtain the optimized network parameters, noise level estimation, and Gaussian curvature; Step 1.3.3, input the noise level estimation and Gaussian curvature into the denoising sub-network for training; Step 1.3.4, negatively feed back the training result in step 1.3.3 to the noise estimation subnetwork; Step 1.3.5, the noise estimation subnetwork performs adjustment control on relevant parameters according to the received results; Step 1.3.6: Construct the loss function L2 and determine the degree of model performance optimization based on the calculation result of the loss function; Step 1.3.7: When the model performance optimization degree meets the training completion conditions, the final network model parameters are obtained and the performance optimization training is ended.

2. The image enhancement method based on a learnable curvature map according to claim 1, characterized in that: The loss function L1 used in the noise estimation subnetwork training includes: the noise distribution loss obtained by training and the noise estimation loss, that is: In the formula, y nl Represents the noise level of the real image; represents the estimated noise level; y nd Represents the noise level of the real image; Represents the estimated noise level.

3. The image enhancement method based on a learnable curvature map according to claim 2, characterized in that: The loss function L2 used in the denoising sub-network training includes: the difference between the predicted value output by the denoising sub-network and the clean image, that is: In the formula, represents the output predicted image; x represents the clean image.

4. An image enhancement system based on a learnable curvature map, used to implement the image enhancement method according to any one of claims 1 to 3, characterized in that: Includes the following modules: The model building module is set to build a two-stage denoising model according to actual application requirements; A first data analysis module is configured to perform feature extraction on the image data to be analyzed and obtain relevant parameters; A second data analysis module is configured to perform image denoising processing based on the output data of the first data analysis module; The data output module is configured to output the processing result of the second data analysis module.

Images