Image glare removal system and method based on glare degree estimation

Abstract

This invention provides an image glare removal system and method based on glare intensity estimation, belonging to the fields of computer vision and image processing technology. The method includes: extracting image features of different scales from an image containing glare as cascaded features; estimating low-frequency and high-frequency glare components from the glare-containing image using gradient separation and template matching to obtain glare intensity estimates; extracting glare intensity estimation features at different scales; concatenating the glare intensity estimation features with downsampled glare estimates at the corresponding scales, and inputting the concatenation kernel into a dynamic convolution kernel generator to generate a dynamic convolution kernel; modulating the cascaded features using the dynamic convolution kernel through dynamic convolutional layers; and reconstructing the image features to obtain a glare-free image. This method compensates for the bias of synthetic training datasets, improves the generalization ability of deep neural networks, and can solve image glare removal problems that better reflect real-world scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision technology and image processing technology, and more specifically, relates to an image glare removal system and method based on glare degree estimation. Background Technology

[0002] When a lens captures an image, if there is a strong light source in the scene, flare often appears in the resulting photograph due to light scattering and internal reflection within the lens. Flare reduces image detail, obscures image content, and affects image quality. Therefore, flare removal is an important problem in low-level vision tasks in order to eliminate flare in images and clearly obtain scene details.

[0003] Ideally, once the camera is focused, light rays from all directions emitted by a point light source will converge onto a single point on the image sensor after refraction through the camera lens. However, in reality, due to factors such as lens optical quality, manufacturing defects, and scratches and dust accumulated during daily use, light does not always travel along the ideal path. When a scene contains light sources that are much brighter than other areas, some of this light will undergo non-ideal reflection or refraction, resulting in glare in other areas of the image. This glare can manifest as halos, streaks, bright lines, spots, color casts, and blurring.

[0004] Traditional glare removal methods primarily target single types of glare, with a focus on green ghosting. Green ghosting typically consists of one or more highly saturated green circular bright spots. Existing green ghosting removal methods first detect green ghosting based on unique features such as shape, brightness, saturation, and position, then remove the detected areas, and finally repair the removed areas based on surrounding scene information. The main limitation of these methods is the need for feature extraction for each type of glare. Because glare is highly diverse in type and style and difficult to distinguish from the background, extracting the features corresponding to various types of glare is extremely challenging.

[0005] Deep convolutional neural networks (CNNs) have achieved state-of-the-art performance in glare removal. Compared to traditional methods, CNNs can leverage their powerful feature representation capabilities to learn the characteristics of all types of glare, removing various types of glare and restoring scene information. However, CNNs require a large number of paired training instances. In the real world, it is difficult to collect images of the same scene containing and without glare. Therefore, most existing methods use synthetic data pairs to provide sufficient training data for CNNs. Specifically, a pure glare image with a completely black background and no scene information is first obtained through real-world photography or simulation. This image is then added to a glare-free scene image, creating a data pair of an image containing glare and its corresponding image without glare. However, because the synthesis of glare and scene data in the real world is very complex and difficult to obtain, the data obtained through synthetic methods differs from real data. Based on these biased data pairs, CNNs struggle to remove glare from real scenes. Therefore, current technical solutions are limited by the bias in the data pairs, necessitating more effective solutions to handle glare in real scenes. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide an image glare removal system and method based on glare level estimation. This system and method are designed to solve the problems that existing methods for synthesizing glare and scenes are very complex and difficult to obtain, the data obtained through synthesis methods differs from real data, and deep convolutional neural networks are unable to remove glare from real scenes.

[0007] To achieve the above objectives, the present invention provides an image glare removal system based on glare level estimation, comprising: a glare sensing module and a glare level estimation modulation module;

[0008] The glare sensing module includes an image encoder, an image decoder, and an intermediate sensing unit;

[0009] The image encoder and image decoder include the same number of glare sensing units of different scales placed symmetrically about the intermediate sensing unit; the glare sensing units in the image encoder are arranged from largest to smallest scale, and the glare sensing units in the image decoder are arranged from smallest to largest scale; the intermediate sensing unit contains several glare sensing units of the same scale.

[0010] The image encoder is used to extract image features of different scales from the glare-containing image and input them as cascaded features to the dynamic convolution modulation unit; the intermediate sensing unit is used to further extract features from the image features transmitted by the glare sensing unit; the image decoder is used to reconstruct the image features to obtain a glare-free image by adding the image features of each scale in the modulated cascaded features to the scale image features output by the corresponding glare sensing unit as the input of the next glare sensing unit.

[0011] The glare level estimation modulation module includes a glare level estimation unit, a glare level estimation encoder, a dynamic convolution kernel generator, and a dynamic convolution modulation unit connected in sequence.

[0012] The glare level estimation unit is used to estimate the low-frequency glare component and the high-frequency glare component from the glare-containing image using gradient separation and template matching. The low-frequency glare component estimates and the high-frequency glare component estimates are then concatenated along the channel dimension to obtain the glare level estimate. The glare level estimation encoder is used to extract glare level estimation features at different scales from the glare-containing image with the concatenated glare level estimates. The glare level estimation features are then concatenated with the downsampled glare estimates at the corresponding scale and input into the dynamic convolution kernel generator to generate a dynamic convolution kernel. The dynamic convolution modulation unit is used to modulate the cascaded features through a dynamic convolution layer using the dynamic convolution kernel.

[0013] More preferably, the glare sensing module adopts a U-Net structure network; the glare sensing unit includes, in the following order according to the information flow: a convolutional layer, a deep convolutional layer, a simple gate mechanism, a three-dimensional weighted attention mechanism, a convolutional layer, an adder, a convolutional layer, a simple gate mechanism, a convolutional layer, and an adder.

[0014] More preferably, the method for estimating the high-frequency glare component is as follows: multiple Gabor filters and anisotropic filters are used to process the image containing glare, and the kernel of the filter is used to match the specific shape of the high-frequency glare component to obtain the estimation result of the high-frequency glare component.

[0015] More preferably, the low-frequency glare component is estimated as follows:

[0016]

[0017] Among them, I F and F L These represent images containing glare and estimation results of low-frequency glare components, respectively. and These represent the two-dimensional discrete cosine transform and its inverse transform, respectively. `div(·)` represents the Laplacian operation and the divergence operation, respectively; K is an H×W matrix with the same scale as the input glare-containing image, and the value of each element is... ε is the weight of the regularization term.

[0018] On the other hand, the present invention provides an image glare removal method based on glare level estimation, comprising the following steps:

[0019] An image encoder is used to extract image features of different scales from images containing glare, and these features are then input into a dynamic convolutional modulation unit as cascaded features.

[0020] Low-frequency glare components and high-frequency glare components were estimated from glare-containing images using gradient separation and template matching methods.

[0021] The low-frequency glare component estimates and high-frequency glare component estimates are concatenated along the channel dimension to obtain the glare intensity estimate;

[0022] Glare estimation features at different scales are extracted from stitched images containing glare estimation.

[0023] The glare level estimation features are concatenated with the glare estimation at the corresponding downsampled scale and then input into the dynamic convolution kernel generator to generate dynamic convolution kernels.

[0024] Dynamic convolutional kernels are used to modulate cascaded features through dynamic convolutional layers;

[0025] A glare-free image is obtained by reconstructing the image features by adding the image features at each scale in the modulated cascaded features to the scale image features output by the corresponding glare sensing unit as the input of the next glare sensing unit.

[0026] More preferably, the glare sensing unit includes, in the following order according to the information flow: a convolutional layer, a deep convolutional layer, a simple gate mechanism, a three-dimensional weighted attention mechanism, a convolutional layer, an adder, a convolutional layer, a simple gate mechanism, a convolutional layer, and an adder.

[0027] More preferably, the method for estimating the high-frequency glare component is as follows: multiple Gabor filters and anisotropic filters are used to process the image containing glare, and the kernel of the filter is used to match the specific shape of the high-frequency glare component to obtain the high glare component estimation result.

[0028] More preferably, the low-frequency glare component is estimated as follows:

[0029]

[0030] Among them, I F and F L These represent images containing glare and estimation results of low-frequency glare components, respectively. and These represent the two-dimensional discrete cosine transform and its inverse transform, respectively. `div(·)` represents the Laplacian operation and the divergence operation, respectively; K is an H×W matrix with the same scale as the input glare-containing image, and the value of each element is... ε is the weight of the regularization term.

[0031] In summary, compared with the prior art, the above-described technical solutions conceived by this invention have the following advantages:

[0032] Beneficial effects:

[0033] This invention provides an image glare removal system and method based on glare level estimation. The glare level estimation modulation module includes a glare level estimation unit. This unit estimates low-frequency and high-frequency glare components from the glare-containing image using gradient separation and template matching. The low-frequency and high-frequency glare component estimates are concatenated along the channel dimension to obtain a glare level estimate. This estimate is then combined with the glare-containing image to generate a dynamic convolution kernel, which modulates the cascaded features to ultimately obtain a glare-free image. This approach compensates for biases in the synthetic training dataset, improves the generalization ability of deep neural networks, and can solve image glare removal problems that better reflect real-world scenarios.

[0034] The image glare removal system and method based on glare degree estimation provided by this invention estimates low-frequency glare components and high-frequency glare components from glare-containing images by gradient separation and template matching. By analyzing the general characteristics of glare, the low-frequency glare components and high-frequency glare components are estimated separately, which is more consistent with the characteristics of glare in real scenes.

[0035] The image glare removal system and method based on glare level estimation provided by this invention sets up a glare level estimation modulation module in order to enable the deep neural network to better learn the characteristics of real glare. By estimating the glare level of the image, the module predicts the dynamic convolution kernel, so as to use the modulated cascaded features of the dynamic convolution layer to guide the deep neural network to learn the characteristics of real glare.

[0036] The image glare removal system and method based on glare level estimation provided by this invention proposes a glare perception unit to enhance the ability of deep neural networks to capture complex glare. By utilizing a three-dimensional weighted attention mechanism, it can perceive the potential information between multiple glare level estimates and effectively distinguish between glare and scene information. Attached Figure Description

[0037] Figure 1 This is a flowchart of an image glare removal method based on glare level estimation provided in an embodiment of the present invention;

[0038] Figure 2(a) is a schematic diagram of the overall architecture of the glare sensing unit provided in an embodiment of the present invention;

[0039] Figure 2(b) is a schematic diagram of the three-dimensional weighted attention mechanism provided in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the glare estimation modulation module structure provided in an embodiment of the present invention;

[0041] Figure 4(a) is an image containing glare provided in an embodiment of the present invention;

[0042] Figure 4(b) is a schematic diagram of low-frequency glare component estimation provided in an embodiment of the present invention;

[0043] Figure 4(c) is a schematic diagram of high-frequency glare component estimation provided in an embodiment of the present invention;

[0044] Figure 5(a) is an image containing glare provided in an embodiment of the present invention;

[0045] Figure 5(b) is a glare-free image after glare removal provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] This invention provides an image glare removal method and system based on glare level estimation. The purpose is to guide a deep learning network to learn more accurate features by estimating the glare level of the image, so that it can compensate for the bias in the synthetic data used for training and remove more realistic glare.

[0048] This invention provides an image glare removal method based on glare level estimation, comprising the following steps:

[0049] A deep learning network architecture for image glare removal is constructed. The basic features of glare (such as shape, size and smoothness) are used to estimate the degree of glare damage to the image. The estimation results are used to guide the deep neural network to learn more realistic glare features.

[0050] More specifically, a. the basic characteristics of glare are as follows:

[0051] Glare manifests in various forms, including starburst glare and ghost glare. Glare has some common characteristics, such as low-frequency glare components (e.g., fog glare) typically appearing as large areas with small gradients, while high-frequency glare components (e.g., starburst glare and ghost glare) have specific shapes. These characteristics of glare can more accurately describe the glare in real scenes, thus providing more realistic glare features for deep neural network learning.

[0052] b. Glare intensity estimation: Based on the general characteristics of glare, gradient separation and template matching methods are used to estimate the low-frequency glare components and the high-frequency glare components respectively, so as to obtain the true glare intensity estimate;

[0053] Based on their manifestation, different types of glare can be divided into two components: low-frequency glare and high-frequency glare. Low-frequency glare mainly includes hazy glare, while high-frequency glare includes various special forms of glare, such as starburst glare and ghost glare. Since the gradient of low-frequency glare is small and high-frequency glare has a specific shape, gradient separation and template matching methods are used to estimate the low-frequency glare and high-frequency glare components, respectively, so as to obtain a more realistic estimate of the degree of glare.

[0054] c. To enable the deep neural network to learn more realistic glare features, a glare level estimation modulation module is constructed to guide the deep neural network using the estimated glare level. The glare level estimation modulation module first estimates the glare level of the input image, then uses the glare level estimation encoder to extract features at different scales, generates dynamic convolutional kernels at different scales, and uses the dynamic convolutional layers to modulate the cascaded features of the backbone network, thereby guiding the deep neural network to learn more realistic glare features.

[0055] d. To enhance the ability of deep neural networks to capture complex glare, this invention provides a glare sensing unit. The glare sensing unit is based on a three-dimensional weighted attention mechanism, which can better learn glare features. Through the three-dimensional weighted attention mechanism, the glare sensing unit can perceive the potential information between various types of glare and can effectively distinguish between glare and scene information.

[0056] e. In the network structure, a U-Net structure network is used as the backbone network. Each scale of the backbone network consists of several glare sensing units. In addition, a glare level estimation modulation module is added to modulate the cascaded features.

[0057] The specific process is as follows: First, the encoder extracts features of different scales of the scene image containing glare. After the features are modulated by the glare degree estimation modulation module, they are input into the glare perception unit of the corresponding scale of the decoder. After being added to the features of the corresponding scale of the decoder, they are used as the input part of the glare perception unit of the next scale. Finally, the scene image without glare is reconstructed, thus achieving the purpose of glare removal.

[0058] The overall architecture of the system is described below, such as... Figure 1As shown, the entire architecture consists of two parts: a backbone network based on glare sensing units and a glare level estimation modulation module. The backbone network based on glare sensing units is the main network that recovers a glare-free image from an image containing glare. It adopts a U-shaped architecture and is divided into an image encoder, an intermediate connection part, and an image decoder. The image encoder and the image decoder each contain four scale units, and each unit and the intermediate connection part are composed of several glare sensing units. The image encoder extracts features at different scales from the input image, and the image decoder receives these features and reconstructs the output image. The glare level estimation modulation module estimates the glare level of the image by using gradient separation and template matching methods. Based on the glare level estimation results, it predicts dynamic convolution kernels and uses dynamic convolution layers to modulate the cascaded features of each scale output by the backbone network image encoder.

[0059] The glare sensing unit is described in detail below. As shown in Figure 2(a), the two key structures of the glare sensing unit are introduced first: the three-dimensional weighted attention mechanism and the simple gate mechanism. The three-dimensional weighted attention structure is shown in Figure 2(b). Based on the input feature M, the three-dimensional weighted attention Ω is calculated by formula (1).

[0060]

[0061] Where μ and σ are the element mean and element variance of the input feature M, respectively; λ is a constant parameter set to 0.1; the attention weights are multiplied element-wise with the input features to obtain the output features; the simple gate structure first divides the input features into two parts evenly along the channel dimension to obtain two feature maps X and Y, and multiplies X and Y element-wise to obtain the output feature values;

[0062] The structure of the glare sensing unit is shown in Figure 2(a). The input features are passed through a convolutional layer, a deep convolutional layer, a simple gate mechanism, a three-dimensional weighted attention mechanism, and a convolutional layer, respectively. After being added to the input features, intermediate features are obtained. Then, the intermediate features are passed through a convolutional layer, a simple gate mechanism, and a convolutional layer, respectively, and added to the intermediate features to obtain the output features.

[0063] The glare estimation modulation module is described in detail below, and its structure is as follows: Figure 3 As shown, it comprises four parts: a glare estimation unit, a glare estimation encoder, a dynamic convolution kernel generator, and a dynamic convolution modulation unit;

[0064] The glare level estimation unit estimates the glare level from the image containing glare. For low-frequency glare components, gradient separation is used to estimate them using the gradient cutoff function δ shown in formula (2). τ (·) Obtain the portion of the input image with larger gradient values;

[0065]

[0066] Among them, I F Represents an image containing glare. The gradient operation is represented by τ, which is the gradient cutoff threshold and is set to 0.043. Then, the final low-frequency glare component estimation result is obtained by using formula (3).

[0067]

[0068] Among them, I F and F L These represent images containing glare and estimation results of low-frequency glare components, respectively. and These represent the two-dimensional discrete cosine transform and its inverse transform, respectively. `div(·)` represents the Laplacian operation and the divergence operation, respectively; K is an H×W matrix with the same scale as the input graph, and the value of each element is... ε is the weight of the regularization term, set to 10. -10 ;

[0069] The glare estimation unit uses a template matching method to estimate the high-frequency glare components. It uses multiple Gabor filters and anisotropic filters to process the image containing glare. The kernel of the filter is used to match the specific shape of the high-frequency glare components to obtain the estimation results of the high-frequency glare components.

[0070] The low-frequency glare component estimates and high-frequency glare component estimates are concatenated along the channel dimension to obtain the final glare level estimate, as shown in Figures 4(a), 4(b), and 4(c).

[0071] After the glare estimation is concatenated with the input image, feature information at different scales is extracted by the glare intensity estimation encoder. After these feature information is concatenated with the glare estimation downsampled to the corresponding scale, the corresponding dynamic convolution kernel is generated by the dynamic convolution kernel generator.

[0072] The dynamic convolution kernel is then used as the kernel parameter of the dynamic convolution layer to process the cascaded features of the backbone network, so as to estimate the modulation cascaded features using the degree of glare.

[0073] To further verify the effectiveness and robustness of this invention, the glare removal model was trained using a publicly available synthetic dataset. The trained glare removal model was then used to remove glare from real glare-containing images, yielding the following results: Figure 5(a) and 5(b) The resulting image is shown.

[0074] In summary, this invention proposes an effective deep neural network structure for glare removal, which can solve the image glare removal problem that better reflects real-world scenarios. Compared with existing glare removal technologies, the technical solution conceived in this invention has the following technical advantages:

[0075] Compared with existing technologies, the glare estimation provided by this invention compensates for the bias of synthetic training datasets, improves the generalization ability of deep neural networks, and can solve the problem of image glare removal that is more in line with real-world scenarios.

[0076] To address the difficulty in describing the characteristics of different types of glare, the glare estimation method provided in this invention analyzes the general characteristics of glare and estimates low-frequency and high-frequency glare separately, which is more consistent with the characteristics of glare in real-world scenarios.

[0077] To enable deep neural networks to better learn the characteristics of real glare, this invention provides a glare level estimation modulation module. By estimating the glare level of an image, it predicts dynamic convolution kernels, thereby utilizing the modulated cascaded features of the dynamic convolutional layers to guide the deep neural network in learning the characteristics of real glare.

[0078] To enhance the ability of deep neural networks to capture complex glare, this invention provides a glare sensing unit for forming the backbone network. By utilizing an attention mechanism with three-dimensional weights, it can perceive potential information between multiple glare level estimates and effectively distinguish between glare and scene information.

[0079] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image glare removal system based on glare intensity estimation, characterized in that, include: Glare sensing module and glare intensity estimation modulation module; The glare sensing module includes an image encoder, an image decoder, and an intermediate sensing unit; The image encoder and image decoder include the same number of glare sensing units of different scales symmetrically placed about the intermediate sensing unit; the glare sensing units in the image encoder are arranged from largest to smallest scale, and the glare sensing units in the image decoder are arranged from smallest to largest scale; the intermediate sensing unit contains several glare sensing units of the same scale. The image encoder is used to extract image features of different scales from the image containing glare, and input them as cascaded features to the dynamic convolution modulation unit; the intermediate sensing unit is used to further extract features from the image features transmitted by the glare sensing unit; the image decoder is used to reconstruct the image features to obtain a glare-free image by adding the image features of each scale in the modulated cascaded features to the scale image features output by the corresponding glare sensing unit as the input of the next glare sensing unit. The glare level estimation modulation module includes a glare level estimation unit, a glare level estimation encoder, a dynamic convolution kernel generator, and a dynamic convolution modulation unit connected in sequence. The glare estimation unit is used to estimate the low-frequency glare component and the high-frequency glare component from the glare-containing image using gradient separation and template matching, and then concatenates the low-frequency glare component estimates and high-frequency glare component estimates along the channel dimension to obtain the glare level estimate; the glare level estimation encoder is used to extract glare level estimation features at different scales from the glare-containing image with the concatenated glare level estimates; after concatenating the glare level estimation features with the glare estimates downsampled to the corresponding scale, the results are input into the dynamic convolution kernel generator to generate a dynamic convolution kernel; the dynamic convolution modulation unit is used to modulate the cascaded features through the dynamic convolution layer using the dynamic convolution kernel.

2. The image glare removal system according to claim 1, characterized in that, The glare sensing module adopts a U-Net structure network; the glare sensing unit includes, in the following order according to the information flow: a convolutional layer, a deep convolutional layer, a simple gate mechanism, a three-dimensional weighted attention mechanism, a convolutional layer, an adder, a convolutional layer, a simple gate mechanism, a convolutional layer, and an adder.

3. The image glare removal system according to claim 1 or 2, characterized in that, The method for estimating high-frequency glare components is as follows: multiple Gabor filters and anisotropic filters are used to process the image containing glare, and the kernel of the filter is used to match the specific shape of the high-frequency glare components to obtain the estimation result of the high glare components.

4. The image glare removal system according to claim 1 or 2, characterized in that, The low-frequency glare component is estimated to be: Among them, I F and F L These represent images containing glare and estimation results of low-frequency glare components, respectively. and These represent the two-dimensional discrete cosine transform and its inverse transform, respectively. `div(·)` represents the Laplacian operation and the divergence operation, respectively; K is an H×W matrix with the same scale as the input glare-containing image, and the value of each element is... ε is the weight of the regularization term.

5. An image glare removal method based on the image glare removal system of claim 1, characterized in that, Includes the following steps: An image encoder is used to extract image features of different scales from images containing glare, and these features are then input into a dynamic convolutional modulation unit as cascaded features. Low-frequency glare components and high-frequency glare components were estimated from glare-containing images using gradient separation and template matching methods. The low-frequency glare component estimates and high-frequency glare component estimates are concatenated along the channel dimension to obtain the glare intensity estimate; Glare estimation features at different scales are extracted from stitched images containing glare estimation. The glare level estimation features are concatenated with the glare estimation at the corresponding downsampled scale and then input into the dynamic convolution kernel generator to generate dynamic convolution kernels. Dynamic convolution kernels are used to modulate cascaded features through dynamic convolutional layers; A method is adopted to reconstruct the image features to obtain a glare-free image by adding the image features at each scale in the modulated cascaded features to the scale image features output by the corresponding glare sensing unit as the input of the next glare sensing unit.

6. The image glare removal method according to claim 5, characterized in that, The glare sensing unit, in the order of information flow, includes: a convolutional layer, a deep convolutional layer, a simple gate mechanism, a three-dimensional weighted attention mechanism, a convolutional layer, an adder, a convolutional layer, a simple gate mechanism, a convolutional layer, and an adder.

7. The image glare removal method according to claim 5 or 6, characterized in that, The method for estimating high-frequency glare components is as follows: multiple Gabor filters and anisotropic filters are used to process the image containing glare, and the kernel of the filter is used to match the specific shape of the high-frequency glare components to obtain the estimation result of the high glare components.

8. The image glare removal method according to claim 5, characterized in that, The low-frequency glare component is estimated to be: Among them, I F and F L These represent images containing glare and estimation results of low-frequency glare components, respectively. and These represent the two-dimensional discrete cosine transform and its inverse transform, respectively. `div(·)` represents the Laplacian operation and the divergence operation, respectively; K is an H×W matrix with the same scale as the input glare-containing image, and the value of each element is... ε is the weight of the regularization term.

Images