A night flare removal method based on illumination law and dynamic mask

Abstract

This invention relates to the field of image processing technology, providing a method for removing nighttime flares based on the law of illumination and dynamic masking. The method includes: using a pre-trained depth estimation model to process a background image and a flare image based on the law of illumination to obtain a baseline image for the deep learning model; processing the baseline image of the deep learning model through a dynamic masking module to obtain an input image for the deep learning model; inputting the input image of the deep learning model into a neural network model; concatenating the output of the neural network model with the baseline image of the deep learning model; and finally, passing the convolutional layer to obtain an image with the flares removed. This invention constrains the strength relationships between multiple flares through the law of illumination and depth estimation, enabling the synthesis of a flare dataset that better conforms to the physical laws of the real world. Furthermore, the dynamic masking module allows the model to focus more on processing the flare region, achieving excellent image restoration results.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for removing nighttime flare based on the law of illumination and dynamic masking. Background Technology

[0002] Nighttime flare removal aims to address lens flare issues caused by dirt or defects, which scatter light across the lens surface or reflect it between lens elements, resulting in incomplete focus. Lens defects include design flaws and physical damage. The final image may exhibit streaks or halos, degrading image quality, affecting visual appeal, and hindering its application in other downstream tasks.

[0003] Neural network-based algorithms often rely on good datasets, and existing methods mainly rely on semi-synthetic datasets. However, the industry practice of using semi-synthetic datasets generally involves randomly transforming a single flare using affine transformations and then adding that flare to the background image. Since real-world scenes typically contain more than one flare, datasets obtained through this synthesis method result in poor model performance in scenes with multiple flares, leading to insufficient model generalization.

[0004] While some works have attempted to synthesize images by randomly transforming multiple flares using affine transformations and then adding them to a background image, this still results in a significant deviation between the synthesized dataset and the real-world scene. This is because, in the real world, the intensity of flares produced by light sources at different distances and angles varies due to the first law of illumination. If multiple light sources are subjected to random affine transformations, this could lead to the synthesis of images that contradict physical laws, negatively impacting the performance of neural network models.

[0005] Furthermore, nighttime flares often only occupy a local area of ​​an image, and existing flare removal methods feed the entire image into the neural network. However, this approach can cause the model to misprocess non-flare areas, such as treating objects that resemble flares as flares and removing them, and some irrelevant areas can also affect the processing results for flare areas. Summary of the Invention

[0006] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a method for removing nighttime flare based on the law of illumination and dynamic masking.

[0007] This invention provides a method for removing nighttime flares based on the law of illumination and dynamic masking, comprising the following steps:

[0008] S1: Use a pre-trained depth estimation model to estimate the depth of the background image to obtain a depth image. Process the depth image and the flare image based on the illuminance law to obtain a brightness-adjusted flare image. By stitching the brightness-adjusted flare image and the background image, obtain the benchmark image of the deep learning model.

[0009] S2: The brightness of the baseline image of the deep learning model is processed by the dynamic masking module to obtain an image containing only the flare area. The input image of the deep learning model is obtained by stitching the image containing only the flare area with the background image.

[0010] S3: Input the input image of the deep learning model into the deep learning neural network model, and stitch the output of the deep learning neural network model with the reference image of the deep learning model to obtain the deep learning stitched image.

[0011] S4: Deep learning stitches images and performs flare removal through convolutional layers to obtain an image without flares.

[0012] According to the nighttime flare removal method based on illuminance law and dynamic masking provided by the present invention, step S1 further includes:

[0013] S11: Use a pre-trained depth estimation model to estimate the depth of the background image in the original dataset to obtain a depth image;

[0014] S12: Randomly select flare images from the original dataset, obtain multiple affine flare images through random affine transformation, and add the multiple affine flare images element by element.

[0015] S13: Randomly select a background image from the original dataset, apply S11 to obtain a depth image, apply S12 to obtain an affine transformation flare image, perform spatial location estimation on the obtained depth image and affine transformation flare image to obtain the position and offset angle of the light source that generated the flare image.

[0016] S14: Substitute the light source position and offset angle into the illuminance law formula to obtain the brightness ratio to be adjusted for each flare image;

[0017] S15: Multiply the brightness adjustment ratio of each flare image by the multiple affine transformation flare images obtained in step S12 to obtain multiple brightness-adjusted flare images.

[0018] S16: Add the brightness-adjusted flare image to the background image selected in step S13 to obtain the baseline image for the deep learning model.

[0019] The nighttime flare removal method based on illuminance law and dynamic masking provided by the present invention further includes step S2, which further includes:

[0020] S21: The brightness matrix is ​​obtained by processing the reference image of the deep learning model through the brightness calculation module, and the brightness distribution map is obtained by transforming the brightness matrix;

[0021] S22: The brightness distribution map is processed by a module consisting of a fully connected layer and a Softmax layer to obtain a brightness threshold of 0 to 1;

[0022] S23: Generate a mask with the same dimensions as the baseline image of the deep learning model using a brightness threshold.

[0023] S24: Multiply the baseline image of the deep learning model with the mask obtained in step S23 to obtain an image containing only the flare region;

[0024] S25: Add the image containing only the flare region to the background image in step S13 to obtain the input image for the deep learning model.

[0025] The nighttime flare removal method based on the illuminance law and dynamic masking provided by the present invention further includes, in step S11, using a pre-trained depth estimation model to estimate the depth of the background image in the original dataset, thereby obtaining a depth image. The expression for the depth image is:

[0026]

[0027] in, For depth images, Background image, This is an operation for performing depth estimation using a depth estimation model.

[0028] According to the present invention, a nighttime flare removal method based on the illuminance law and dynamic masking further includes, in step S12, obtaining multiple affine transformed flare images from the flare image through random affine transformation, wherein the expression for the affine transformed flare image is:

[0029]

[0030] in, For the number of flares, This is the affine flare image obtained after affine transformation. For random affine transformation, These are the sequence numbers corresponding to multiple flares. Image of a solar flare.

[0031] The nighttime flare removal method based on the illuminance law and dynamic masking provided by the present invention further includes, in step S13, using multiple affine transformed flare images and depth images to perform spatial location estimation, obtaining the light source position and offset angle of each flare. The expressions for the light source position and offset angle are as follows:

[0032]

[0033] in, The position of the light source. For offset angle, For spatial location estimation, For depth images, This is the affine flare image obtained after affine transformation.

[0034] The present invention provides a method for removing nighttime flares based on the illuminance law and dynamic masking, further comprising step S14, which involves substituting the position and offset angle of the light source into the illuminance law formula to obtain the illuminance ratio between two flares. The expression for the illuminance ratio is:

[0035]

[0036] in, The illuminance of the first flare. The illuminance of the second flare; The location of the source of the first flare. This indicates the location of the source of the second flare. The offset angle of the first flare. This represents the offset angle of the second flare;

[0037] The brightness ratio to be adjusted in the flare image is obtained based on the illuminance ratio, and then the flare brightness is adjusted.

[0038] The nighttime flare removal method based on the illuminance law and dynamic mask provided by the present invention further includes obtaining a luminance distribution map by performing a luminance calculation module on the image in step S21, wherein the luminance matrix expression is:

[0039]

[0040] Where Y is the luminance matrix, R is the red matrix, G is the green matrix, and B is the blue matrix. This is the brightness calculation module.

[0041] The nighttime flare removal method based on the illuminance law and dynamic masking provided by the present invention further includes, in step S23, using a luminance threshold to generate a mask matrix with the same dimensions as the original image. The expression for the mask matrix is:

[0042]

[0043] in, For the mask matrix, Here is the baseline image for the deep learning model, where i is the row number of the pixel and j is the column number of the pixel. This is the brightness threshold.

[0044] According to the present invention, a nighttime flare removal method based on illuminance law and dynamic masking further includes, in step S23, setting the mask to 1 for the locations where the pixels of the reference image of the deep learning model are greater than the brightness threshold, and setting the mask to 0 for the other locations.

[0045] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0046] This invention designs a method for synthesizing nighttime flare datasets based on the law of illumination. By using the law of illumination and depth estimation, the method constrains the strength relationships between multiple flares, enabling the synthesis of a flare dataset that more closely reflects real-world physical laws. The invention also includes a dynamic masking module. This module dynamically learns a brightness threshold based on the scene's brightness distribution and uses this threshold to extract flare regions, removing unnecessary image areas. This allows the model to focus more on processing flare regions and avoids erroneous processing of non-flare areas, achieving excellent image restoration results. This data synthesis method and the dynamic masking module demonstrate good performance in removing nighttime flares in various scenarios.

[0047] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a nighttime flare removal method based on the law of illumination and dynamic masking provided by the present invention.

[0050] Figure 2 The image shows the effect of nighttime flare removal inside a factory, based on the law of illumination and dynamic masking, which is provided by this invention.

[0051] Figure 3The image shows the effect of removing nighttime flare in a ground parking lot, based on the illuminance law and dynamic masking method provided by this invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0053] like Figure 1 As shown, a nighttime flare removal method based on the law of illumination and dynamic masking is presented. It obtains the distance between the light source and the lens in the image through a depth estimation model. Using this distance and the incident angle of the light, it calculates the illumination of the lens from different light sources. Based on this principle, multiple flare images with relative intensity relationships are obtained. Furthermore, the invention includes a dynamic masking module that dynamically learns a brightness threshold based on the scene's brightness distribution and uses this threshold to extract flare regions and remove unnecessary image areas. The nighttime flare removal method based on the law of illumination and dynamic masking includes the following steps:

[0054] S1: A pre-trained depth estimation model is used to estimate the depth of the background image, obtaining a depth image. Based on the illuminance law, the depth image and the flare image are processed to obtain a brightness-adjusted flare image. The brightness-adjusted flare image and the background image are then stitched together to obtain the baseline image for the deep learning model. Specifically, this includes:

[0055] S11: Given a background image Using a pre-trained depth estimation model for background images Perform depth estimation to obtain a depth image. Depth image The expression is:

[0056]

[0057] in, For depth images, Background image, This describes the operation of depth estimation using a depth estimation model (Dense Prediction Transformer).

[0058] S12: Randomly select flare images from the dataset Multiple affine flare images are obtained through random affine transformation. The expression for the affine flare image is:

[0059]

[0060] in, For the number of flares, This is the affine flare image obtained after affine transformation. This is a random affine transformation. These are the sequence numbers corresponding to multiple flares. For flare images, random affine transformations include translation, scaling, cropping, and rotation operations;

[0061] The obtained affine transformation flare images are added element by element.

[0062] S13: Randomly select a background image from the original dataset, and perform spatial position estimation on the depth image corresponding to the background image obtained in step S11 and the affine flare image obtained in step S12 after affine transformation, to obtain the location of the light source that produced different flares. and offset angle Light source position and offset angle The expression is:

[0063]

[0064] in, The position of the light source. For offset angle, For spatial location estimation, For depth images, This is the affine flare image obtained after affine transformation.

[0065] Furthermore, the position of the light source is the distance between the light source and the lens, and the offset angle is the angle between the ray and the lens normal.

[0066] S14: According to the first illuminance law, since the same flare image is used to generate multiple flares, the ratio between the two light sources can be used as the ratio for adjusting the flare brightness.

[0067] Specifically, substituting the light source position and offset angle obtained in step S13 into the illuminance law formula, and using the conclusions of the illuminance law, the illuminance ratio between the two flares is obtained. The expression for the illuminance ratio is:

[0068]

[0069] in, The illuminance of the first flare. The illuminance of the second flare; The location of the source of the first flare. This indicates the location of the source of the second flare. The offset angle of the first flare. This represents the offset angle of the second flare;

[0070] The illuminance ratio is used to determine the appropriate brightness adjustment for each flare, and this ratio is then used to adjust the flare brightness.

[0071] Furthermore, the first law of illuminance used in this invention is: illuminance is inversely proportional to the square of the distance and directly proportional to the cosine of the offset angle.

[0072] S15: Multiply the brightness adjustment ratio of each flare by the multiple affine transformation flares obtained in step S12 to obtain multiple brightness-adjusted flare images.

[0073] S16: Add the light source corresponding to the flare image obtained by random affine transformation in step S12 to the background image selected in step S13 to obtain the benchmark image of the deep learning model.

[0074] S2: The baseline image for the deep learning model is processed using a dynamic masking module to obtain an image containing only the flare regions. This image is then stitched together with the background image to obtain the input image for the deep learning model. Specifically, this includes:

[0075] S21: The baseline image of the deep learning model is used to obtain a brightness matrix through the brightness calculation module. The brightness matrix expression is:

[0076]

[0077] Where Y is the luminance matrix, R is the red matrix, G is the green matrix, and B is the blue matrix. This is a brightness calculation module;

[0078] A brightness distribution map can be obtained by transforming the brightness matrix.

[0079] S22: Input the brightness distribution map obtained in step S21 into a module consisting of a fully connected layer and a Softmax layer, and after processing, obtain a brightness threshold of 0 to 1;

[0080] Specifically, the brightness distribution map is flattened using a flattening operation. The flattened brightness distribution map vector is then fed into a linear layer to obtain a brightness threshold. The obtained brightness threshold is then normalized to between 0 and 1 using the sigmoid activation function.

[0081] S23: Using the brightness threshold obtained in step S22, generate a mask with the same length and width as the reference image of the deep learning model; where pixels in the reference image of the deep learning model are greater than the brightness threshold, set the corresponding position in the mask to 1, and set all other positions in the mask to 0, thus obtaining the mask matrix. The expression for the mask matrix is:

[0082]

[0083] in, For the mask matrix, Here is the baseline image for the deep learning model, where i is the row number of the pixel and j is the column number of the pixel. This is the brightness threshold.

[0084] S24: Multiply the baseline image of the deep learning model with the mask obtained in step S23 to obtain an image containing only the flare region.

[0085] S25: Add the image containing only the flare region to the background image in step S13 to obtain the input image for the deep learning model.

[0086] S3: Input the input image of the deep learning model into the deep learning neural network model, and stitch the output of the deep learning neural network model with the reference image of the deep learning model to obtain the deep learning stitched image.

[0087] S4: Deep learning stitches images and performs flare removal through convolutional layers to obtain an image without flares.

[0088] The beneficial effects of this invention are as follows: This invention designs a method for synthesizing nighttime flare datasets based on the law of illumination. By using the law of illumination and depth estimation, the strength relationship between multiple flares is constrained, enabling the synthesis of flare datasets that better conform to the physical laws of the real world. This invention also designs a dynamic masking module. By dynamically learning a brightness threshold based on the scene's brightness distribution, this threshold is used to extract flare regions and remove unnecessary image areas. This allows the model to focus more on processing flare regions and avoids erroneous processing of non-flare regions, achieving excellent image restoration results. The data synthesis method and dynamic masking module of this invention have good effects on nighttime flare removal in various scenarios.

[0089] like Figure 2The image shows the effect of applying the data synthesis method and dynamic masking module of this invention to remove flare spots inside a factory at night. Figure 3 The image shows the effect of applying the data synthesis method and dynamic masking module of this invention on flare removal in a nighttime parking lot. It can be seen that this nighttime flare removal method based on the illuminance law and dynamic masking has good results in removing nighttime flares under different scenarios.

[0090] Furthermore, in the publicly available evaluation dataset, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics are used to evaluate the restoration performance, with higher PSNR and SSIM indicating better image restoration results.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for removing nighttime flare based on the law of illumination and dynamic masking, characterized in that, Includes the following steps: S1: Use a pre-trained depth estimation model to estimate the depth of the background image to obtain a depth image. Process the depth image and the flare image based on the illuminance law to obtain a brightness-adjusted flare image. By stitching the brightness-adjusted flare image and the background image, obtain the benchmark image of the deep learning model. S11: Use a pre-trained depth estimation model to estimate the depth of the background image in the original dataset to obtain a depth image; S12: Randomly select flare images from the original dataset, obtain multiple affine flare images through random affine transformation, and add the multiple affine flare images element by element. S13: Randomly select a background image from the original dataset, apply S11 to obtain a depth image, apply S12 to obtain an affine transformation flare image, perform spatial location estimation on the obtained depth image and affine transformation flare image to obtain the position and offset angle of the light source that generated the flare image. S14: Substitute the light source position and offset angle into the illuminance law formula to obtain the brightness ratio to be adjusted for each flare image; By substituting the position and offset angle of the light source into the illuminance law formula, the illuminance ratio between the two flares is obtained. The expression for the illuminance ratio is: in, The illuminance of the first flare. The illuminance of the second flare; The location of the source of the first flare. This indicates the location of the source of the second flare. The offset angle of the first flare. This is the offset angle of the second flare; The brightness ratio to be adjusted in the flare image is obtained based on the illuminance ratio, and then the flare brightness is adjusted accordingly. S15: Multiply the brightness adjustment ratio of each flare image by the multiple affine transformation flare images obtained in step S12 to obtain multiple brightness-adjusted flare images. S16: Add the brightness-adjusted flare image to the background image selected in step S13 to obtain the baseline image for the deep learning model; S2: The brightness of the baseline image of the deep learning model is processed by the dynamic masking module to obtain an image containing only the flare area. The input image of the deep learning model is obtained by stitching the image containing only the flare area with the background image. S3: Input the input image of the deep learning model into the deep learning neural network model, and stitch the output of the deep learning neural network model with the reference image of the deep learning model to obtain the deep learning stitched image. S4: Deep learning stitches images and performs flare removal through convolutional layers to obtain an image without flares.

2. The method for removing nighttime flares based on the law of illumination and dynamic masking according to claim 1, characterized in that... Step S2 further includes: S21: The brightness matrix is ​​obtained by processing the reference image of the deep learning model through the brightness calculation module, and the brightness distribution map is obtained by transforming the brightness matrix; S22: The brightness distribution map is processed by a module consisting of a fully connected layer and a Softmax layer to obtain a brightness threshold of 0 to 1; S23: Generate a mask with the same dimensions as the baseline image of the deep learning model using a brightness threshold. S24: Multiply the baseline image of the deep learning model with the mask obtained in step S23 to obtain an image containing only the flare region; S25: Add the image containing only the flare region to the background image in step S13 to obtain the input image for the deep learning model.

3. The method for removing nighttime flares based on the law of illumination and dynamic masking according to claim 1, characterized in that... In step S11, a pre-trained depth estimation model is used to estimate the depth of the background image in the original dataset, obtaining a depth image. The expression for the depth image is: in, For depth images, Background image, This is an operation for performing depth estimation using a depth estimation model.

4. The method for removing nighttime flares based on the law of illumination and dynamic masking according to claim 1, characterized in that... In step S12, the flare image is transformed into multiple affine flare images through random affine transformation. The expression for the affine flare image is: in, For the number of flares, This is the affine flare image obtained after affine transformation. For random affine transformation, These are the sequence numbers corresponding to multiple flares. Image of a solar flare.

5. A method for removing nighttime flare based on the law of illumination and dynamic masking according to claim 1, characterized in that... In step S13, multiple affine transformed flare images and depth images are used for spatial location estimation to obtain the source position and offset angle of each flare. The expressions for the source position and offset angle are: in, The position of the light source. For offset angle, For spatial location estimation, For depth images, This is the affine flare image obtained after affine transformation.

6. A method for removing nighttime flare based on the law of illumination and dynamic masking according to claim 2, characterized in that... In step S21, the luminance matrix is ​​obtained by processing the reference image of the deep learning model through the luminance calculation module. The expression of the luminance matrix is ​​as follows: Where Y is the luminance matrix, R is the red matrix, G is the green matrix, and B is the blue matrix. This is the brightness calculation module.

7. A method for removing nighttime flares based on the law of illumination and dynamic masking according to claim 2, characterized in that... In step S23, a mask matrix with the same dimensions as the original image is generated using a brightness threshold. The expression for the mask matrix is: in, For the mask matrix, Here is the baseline image for the deep learning model, where i is the row number of the pixel and j is the column number of the pixel. This is the brightness threshold.

8. The method for removing nighttime flares based on the law of illumination and dynamic masking according to claim 2, characterized in that, In step S23, for pixels in the reference image of the deep learning model that are greater than the brightness threshold, the corresponding position of the mask is set to 1, and all other positions of the mask are set to 0.

Images