Fog-containing image generation method based on monocular vision depth estimation

Abstract

The invention discloses a monocular vision depth estimation-based foggy image generation method, which is oriented to foggy day image simulation generation application, adopts an improved physical model and an improved depth map estimation model to process a fogless image, and simulates to generate a vivid foggy image. And the application requirements of image processing research such as image defogging and foggy day image target detection are met. The method comprises the following steps: firstly, generating an initial scene relative depth map by using a monocular vision depth estimation network; then, in combination with the characteristics of the fog-containing image, performing optimization processing on the initial depth estimation image by adopting processing technologies such as image super-resolution and edge smoothing to obtain a scene optimization depth image; and finally, based on the optimized depth map, generating a transmissivity map of the scene by adopting an improved transmissivity generation model. And finally carrying out fusion processing on the improved transmissivity image and the scene image to obtain a vivid fog-containing image. The method can be applied to the fields of automatic driving simulation, forest fire prevention, remote sensing positioning, virtual reality and the like.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and more specifically, relates to a method for generating foggy images based on monocular vision depth estimation. Background Art

[0002] The generation of foggy images is of great significance in multiple fields. First, in computer vision and artificial intelligence, foggy images can be used to train and test deep learning models, improving the robustness of algorithms for tasks such as target detection, object recognition, and semantic segmentation in adverse weather conditions. Second, in autonomous driving and intelligent transportation, foggy weather is a significant factor affecting driving safety. Synthesizing foggy images can simulate different visibility conditions and be used to test and optimize perception systems, ensuring that autonomous vehicles can still accurately identify road information in low-visibility environments. Furthermore, in remote sensing and satellite imagery analysis, simulating foggy environments can help develop defogging enhancement technologies and improve the accuracy of tasks such as aerial monitoring and environmental assessment. The generation of foggy images also plays a role in film and television production, game development, and augmented reality (AR), providing support for realistic rendering of virtual scenes and making foggy environments more natural and vivid.

[0003] Traditional fogging methods are often based on simple image filtering techniques or fixed physical models. The resulting haze effect lacks realism and fails to accurately reflect the optical properties of natural scenes. In recent years, with the development of deep learning technology, monocular depth estimation networks have made significant progress, capable of estimating high-precision depth information from monocular images, providing new solutions for dehazing and fogging tasks. However, existing methods still have shortcomings in terms of realism, adaptability, and computational efficiency. Summary of the Invention

[0004] In response to the technical problems in the existing technology that traditional monocular visual depth estimation networks still have shortcomings in realism, adaptability, computational efficiency, etc., the present invention proposes a foggy image generation method based on monocular visual depth estimation. Its purpose is to achieve efficient, realistic and flexible haze effect generation by combining deep learning with physical models, so as to solve the defects of traditional methods in realism, hardware dependence and computational complexity.

[0005] To address the problem that existing datasets for specific scenes are limited and the generated foggy images are not realistic, this paper provides a method for generating foggy images based on monocular depth estimation. This method combines a monocular visual depth estimation network to obtain and optimize a monocular depth estimation map, and then uses the optimized depth map to obtain the required fog. The method includes the following steps:

[0006] Scene depth estimation: A monocular visual depth estimation network is used to process the real-world fog-free scene image J(x) to obtain the initial depth estimation map of the scene. The initial depth estimation map is then processed using a depth map optimization method to obtain an optimized depth estimation map. The depth value is normalized to the range [0, 1] and recorded as D(x), where x is the image pixel index value.

[0007] Scene brightness adjustment: Based on the fact that the ambient light intensity in foggy scenes is lower than that in normal scenes, an adaptive image brightness adjustment method is used to adjust the brightness of the real-world fog-free image J(x) to match the image brightness of foggy scenes. The adjusted image is denoted as J1(x).

[0008] Scene transmittance estimation: The scene transmittance estimation model is used to process the depth estimation map D(x) to obtain the scene transmittance estimation map t(x);

[0009] Scene fog map generation: Based on the scene transmittance estimation map t(x), the fog intensity generation model is used to generate the scene fog intensity map Fog(x);

[0010] Foggy image synthesis: The fog intensity map Fog(x) and the brightness adjustment map J1(x) are fused using the image fusion method, and the fused image is sharpened and filtered to obtain the final foggy image, which is recorded as I(x).

[0011] Furthermore, when using a monocular vision depth estimation model to estimate the scene depth during scene depth estimation, the depth estimation at the depth edge is easily affected by noise, resulting in inaccurate depth estimation at the edge, which in turn causes a halo effect at the depth edge when generating foggy images. To address this issue, the present invention designs a depth map optimization method to optimize the initial depth estimation map, eliminating the boundary halo effect by smoothing the depth edges.

[0012] Considering that edge smoothing may cause image blur, the initial depth estimation image is first super-resolution processed before edge smoothing. The depth image after super-resolution processing is recorded as D L (x), and then the depth map D after super resolution processing L (x) Perform edge extraction and smoothing. The basic steps of edge smoothing are as follows:

[0013] Step 1: Extract the depth map D after super-resolution processing L (x) and invert it to get the grayscale inverted depth map D I (x), and then use the edge detection algorithm to process the depth map D after super resolution. L (x) is processed to obtain the depth edge map D e (x);

[0014] Step 2: D e (x) Perform image morphological expansion operation to expand the depth edge, and record the edge expansion map as D d (x);

[0015] Step 3: Use Gaussian blur kernel to expand the edge of image D d (x) is processed to obtain the edge weight graph, namely:

[0016] W1=GaussianBlur1(D d (x)),

[0017] W1 is the edge weight map, GaussianBlur1(·) is the Gaussian filter kernel;

[0018] Step 4: Fusion is performed according to the following formula to obtain the final optimized depth estimation map, namely:

[0019] D(x)=D I (x)×(1-W1)+GaussianBlur2(D I (x))×W1,

[0020] GaussianBlur1(·) and GaussianBlur2(·) are Gaussian filter kernels with different window sizes and standard deviations. The window sizes are 15 and 25, and the standard deviations are 10 and 15, respectively.

[0021] Furthermore, in the scene brightness adjustment step, an image brightness adaptive adjustment method is designed to simulate the ambient brightness of the foggy scene. The brightness of the fog-free image is adjusted to obtain the scene brightness degradation map J1(x). The brightness adjustment adopts the following formula:

[0022]

[0023] Among them, L mean is the average brightness value of J(x), N(x) is Gaussian noise, k is an adjustment factor for adjusting the brightness change rate, which is set according to the brightness requirements. The adjustment factor k of the present invention can be set to 2.5, and α is an adjustment factor for adjusting the relative value of the signal and noise.

[0024] Furthermore, in the scene transmittance estimation step, a scene transmittance estimation model is designed. The scene transmittance estimation model obtains the scene transmittance estimation map t(x) according to the scene depth estimation value D(x):

[0025]

[0026] The design of the present invention is different from the commonly used transmittance calculation method. It adopts a polynomial fitting method. The second term on the right side of the above formula β×D(x) 2 The transmittance increases faster at smaller depths, and the third term This shows a sublinear growth trend, indicating that light attenuation is more gradual at long distances. This combination means that t(x) can be larger than the previous exponential decay, resulting in clearer images of nearby objects. In other words, the improved transmittance formula generates a greater contrast between distant fog concentration and nearby objects than previous atmospheric scattering models. The fog concentration is determined by the factor β, which ranges from (0 to 1).

[0027] In the step of generating the scene fog map, the calculation formula of the fog intensity generation model is as follows:

[0028] Fog(x)=A(x)·(1-t(x))+Poisson(x),

[0029] Here, A(x) is the atmospheric light value (fog color value). Since foggy scenes are generally overcast, the sky is also darker. Using white fog would make the light appear too bright, and the restoration of the foggy scene would not be realistic. Therefore, choosing an appropriate atmospheric light value is the key to adding fog. Here, it is defined as a constant value vector. In the real world, fog is almost always non-uniform. Therefore, this paper takes into account the non-uniformity of fog and uses a Poisson distribution to generate random perturbation noise to simulate the randomness of the non-uniform variation of fog intensity.

[0030] The foggy image synthesis step uses the image fusion method to fuse the fog intensity map Fog(x) and the brightness adjustment map J1(x), and then performs sharpening filtering on the fused image to obtain the final foggy image. The specific implementation is as follows:

[0031] I(x)=Sharpe[J1(x)×t(x)+Fog(x)],

[0032] Considering that slight sharpening can highlight the details of objects behind the fog, the foggy scene of this method can be made more attractive.

[0033] Among them, Shape[·] is the sharpening filter operator, that is:

[0034] I sharpe (x) = I(x) - L(I(x)),

[0035] Where L is a low-intensity Laplace sharpening kernel with a size of:

[0036]

[0037] γ is the factor that determines the degree of sharpness.

[0038] In general, the above technical solutions constructed by the present invention can achieve the following beneficial effects:

[0039] 1. To address the problem that the existing monocular vision depth estimation network is easily interfered by noise at the depth edge, the present invention adopts a depth estimation optimization method. Through image super-resolution processing and depth edge smoothing, it retains the depth details and overcomes the problem of inaccurate depth estimation caused by noise interference at the depth edge, thereby improving the generation effect of fog-free images.

[0040] 2. The present invention proposes an improved scene transmittance estimation method, which enhances the contrast between distant transmittance and near transmittance, making the scene transmittance estimation smoother and more consistent with real and complex foggy scenes.

[0041] 3. This invention combines the characteristics of non-uniform fog in the real world and uses Poisson distribution to randomly perturb the fog intensity distribution in the scene, making the fog intensity distribution closer to the real scene, avoiding excessive smoothness and distortion of the fog, and enhancing the layering of the fog map. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is the overall flow chart for generating foggy images proposed in this invention.

[0043] Figure 2 Flowchart for scene depth estimation and optimization processing.

[0044] Figure 3 and Figure 4 This is an example diagram of the processing effect of the algorithm proposed in this invention. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0046] refer to Figure 1 The method for generating a foggy image based on monocular depth estimation in this embodiment is implemented by the following steps.

[0047] S1: Scene Depth Estimation: This paper uses Depth-Anything-Net as a monocular depth estimation network to process a real-world fog-free scene image J(x) to obtain an initial scene depth estimate. Compared to other monocular depth estimation algorithms, Depth-Anything-Net is faster and more accurate, making it a well-suited monocular depth estimation network for generating foggy images.

[0048] refer to Figure 2 , considering that edge smoothing may cause image blur, before edge smoothing, the initial depth estimation map is first super-resolution processed, and the depth map after super-resolution processing is recorded as D L (x). Super-resolution processing uses the Lanczos interpolation method, a high-quality interpolation method based on the sinc function. Lanczos interpolation can better preserve details and sharpen images when scaling. Since the quality of the generated depth map is generally average, using Lanczos interpolation can prevent severe jagged effects and make edges smoother without causing severe loss of detail. The present invention uses a = 4, that is, the calculation of each interpolation point considers the pixels within an 8×8 neighborhood.

[0049] Then the depth map D after super-resolution processing L (x) Perform edge extraction and smoothing. The basic steps of edge smoothing are as follows:

[0050] The first step is to convert the three-channel pseudo-color depth estimation map after super-resolution processing and interpolation into a grayscale inverted depth map, namely:

[0051] D I (x)=1-(0.299×R+0.587×G+0.114×B),

[0052] R, G, and B represent the values of red (Red), green (Green), and blue (Blue), respectively.

[0053] Then, in the edge extraction step, the present invention uses the Canny edge detection algorithm to extract the depth map D after super-resolution processing. L (x) is processed to obtain the depth edge map D e (x). Specifically, the image is first denoised using a Gaussian filter with a window size of 5×5 and a standard deviation of 1.4. The Sobel detection operator is then used to extract edges in the horizontal and vertical directions of the image and calculate their gradient magnitude and direction. Next, in the non-maximum suppression algorithm, if the current pixel is smaller than the adjacent pixel in the same direction, it is set to 0, and only the local maximum is considered as a candidate edge point. Double-threshold detection is then performed, with the low threshold set to 50 and the high threshold set to 150.

[0054] In the second step, after Canny edge detection, the present invention uses a graphic morphological expansion algorithm to process the depth edge map D e (x), expand the depth edge, and record the edge expansion map as D d The dilation structure element is set to an elliptical structure with a window size of 5×5, which should be consistent with the filter kernel size of the Gaussian Canny edge detection algorithm.

[0055] The third step is to obtain the edge weight graph W1. The present invention first uses Gaussian filtering to check the edge expansion graph D. d (x) is blurred, the filter kernel window size is 15×15, and the standard deviation is 10.

[0056] The fourth step is to use a filter kernel with a window size of 25×25 and a standard deviation of 15 to compare D I (x) is processed to obtain GaussianBlur2(D I (x)), the edge weight graph W1 obtained in the third step is combined with D I (x), and GaussianBlur2(D I (x)) are fused and the optimized depth estimation map is finally obtained, namely:

[0057] D(x)=D I (x)×(1-W1)+GaussianBlur2(D I (x))×W1.

[0058] S2: Scene brightness adjustment: The ambient light intensity of foggy scenes is lower than that of normal scenes. The image brightness adaptive adjustment method is used to adjust the brightness of the real-world fog-free image J(x) to make it consistent with the image brightness of foggy scenes. The adjusted image is recorded as J1(x). The present invention uses the average brightness adaptive Gamma transform to adjust the brightness and contrast of the image to obtain the image brightness degradation map.

[0059]

[0060] L mean is the average brightness of the image. In this embodiment, the adjustment factor k is set to 2.5. α is an adjustment factor used to adjust the relative value of the signal and noise. In this example, the value is 0.85. The mean of Gaussian noise is 0 and the standard deviation is 15.

[0061] S3: Scene transmittance estimation: The scene transmittance estimation model is used to process the depth estimation map D(x) to obtain the scene transmittance estimation map t(x). The present invention designs a scene transmittance estimation model. The scene transmittance estimation model obtains the scene transmittance estimation map t(x) based on the scene depth estimation value D(x) using the following formula:

[0062]

[0063] The design of the present invention is different from the commonly used transmittance calculation method. It adopts a polynomial fitting method. The second term on the right side of the above formula β×D(x) 2 The transmittance increases faster at smaller depths, and the third term This shows a sublinear growth trend, indicating that light attenuation is more gradual at long distances. This combination means that t(x) can be larger than the previous exponential decay, resulting in clearer images of nearby objects. In other words, the improved transmittance formula generates a greater contrast between distant fog concentration and nearby objects than previous atmospheric scattering models. The fog concentration is determined by the factor β, which ranges from (0 to 1).

[0064] The present invention defines the β value as an adjustable value, which is related to the concentration of the fog to be generated. If the β value is increased, the fog in the distance will be thicker, while the fog in the near distance will be reduced, and the contrast between the near and far scenes will increase, for example, α = 0.8. Conversely, if β is reduced, there will also be fog in the near distance, and the fog in the whole picture will look smoother, for example, β = 0.2. Figure 3 and Figure 4 In this embodiment, the value of β is 0.5, and a moderate haze image is generated.

[0065] S4: Scene fog map generation: Based on the scene transmittance estimation map t(x), the fog intensity generation model is used to generate the scene fog intensity map Fog(x). The calculation formula of the fog intensity generation model is as follows:

[0066] Fog(x)=A(x)·(1-t(x))+Poisson(x),

[0067] Here, A(x) is the atmospheric light value (fog color value). Since foggy scenes are generally overcast, the sky is also darker. Using white fog would make the light appear too bright, and the restoration of the foggy scene would not be realistic. Therefore, choosing an appropriate atmospheric light value is the key to adding fog. Here, it is defined as a constant value vector. In the real world, fog is almost always non-uniform. Therefore, this paper takes into account the non-uniformity of fog and uses a Poisson distribution to generate random perturbation noise to simulate the randomness of the non-uniform variation of fog intensity.

[0068] The present invention will obtain the fog intensity map to be generated, and the color of the fog depends on the value of A(x). The fog color obtained by the previous algorithm is brighter and does not conform to the real cloudy and foggy scene. Therefore, this embodiment takes the atmospheric light value as It is a light gray fog, which is suitable for cloudy scenes. Poisson noise is added to generate non-uniform fog.

[0069] S5: Fog Image Generation: The present invention fuses the image brightness degradation map generated in S2, the scene transmittance estimation map generated in S3, and the fog intensity map generated in S4 to obtain a foggy image. This is specifically achieved using the following formula:

[0070] I(x)=Sharpe[J1(x)×t(x)+Fog(x)],

[0071] Considering that slight sharpening can highlight the details of objects behind the fog, the foggy scene of this method can be made more attractive.

[0072] Among them, Shape[·] is the sharpening filter operator, that is:

[0073] I sharpe (x) = I(x) - L(I(x)),

[0074] Where L is a low-intensity Laplace sharpening kernel with a size of:

[0075]

[0076] γ is the factor that determines the degree of sharpness.

[0077] The present invention then performs a sharpening filter on the foggy image to enhance the subtle details of objects behind the fog, thereby producing a visually appealing cloudy foggy image I(x). Because high-intensity sharpening can produce unnecessary noise, only low-intensity sharpening filtering is required to generate the foggy image. In this embodiment, the value of γ should not be too large; in this embodiment, the value of γ is 0.2.

[0078] Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for generating foggy images based on monocular visual depth estimation, characterized in that: The steps include: Scene depth estimation: A monocular visual depth estimation network is used to process the real-world fog-free scene image J(x) to obtain the initial depth estimation map of the scene. The initial depth estimation map is then processed using a depth map optimization method to obtain an optimized depth estimation map. The depth value is normalized to the range [0, 1] and recorded as D(x), where x is the image pixel index value. Scene brightness adjustment: Based on the fact that the ambient light intensity in foggy scenes is lower than that in normal scenes, an adaptive image brightness adjustment method is used to adjust the brightness of the real-world fog-free image J(x) to match the image brightness of foggy scenes. The adjusted image is denoted as J1(x). Scene transmittance estimation: The scene transmittance estimation model is used to process the depth estimation map D(x) to obtain the scene transmittance estimation map t(x); Scene fog map generation: Based on the scene transmittance estimation map t(x), the fog intensity generation model is used to generate the scene fog intensity map Fog(x); Foggy image synthesis: The fog intensity map Fog(x) and the brightness adjustment map J1(x) are fused using the image fusion method, and the fused image is sharpened and filtered to obtain the final foggy image, which is recorded as I(x).

2. The method for generating a foggy image according to claim 1, wherein: In the scene depth estimation step, the depth map optimization processing method is used to process the initial depth estimation map, specifically comprising the following steps: First, the initial depth estimation map is super-resolution processed, and the depth map after super-resolution processing is recorded as D L (x), and then the depth map D after super resolution processing L (x) Perform edge extraction and smoothing. The basic steps of edge smoothing are as follows: Step 1: Extract the depth map D after super-resolution processing L The brightness component of (x) is inverted to obtain the grayscale inverted depth map D I (x), and then use the edge detection algorithm to process the depth map D after super resolution. L (x) is processed to obtain the depth edge map D e (x); Step 2: D e (x) Perform image morphological expansion operation to expand the depth edge, and record the edge expansion map as D d (x); Step 3: Use Gaussian blur kernel to expand the edge of image D d (x) is processed to obtain the edge weight graph, namely: W1=GaussianBlur1(D d (x)), W1 is the edge weight map, GaussianBlur1(·) is the Gaussian filter kernel; Step 4: Fusion is performed according to the following formula to obtain the final optimized depth estimation map, namely: D(x)=D I (x)×(1-W1)+GaussianBlur2(D Y (x))×W1, where GaussianBlur1(·) and GaussianBlur2(·) are Gaussian filter kernels with different window sizes and standard deviations.

3. The method for generating a foggy image according to claim 1, wherein: In the scene brightness adjustment step, the image brightness adaptive adjustment method is used to adjust the brightness of the fog-free image to obtain the scene brightness degradation map J1(x). The brightness adjustment adopts the following formula: Among them, L mean is the average brightness value of J(x), N(x) is Gaussian noise, k is the adjustment factor used to adjust the brightness change rate, which is set according to the brightness requirements, and α is the adjustment factor used to adjust the relative value of signal and noise.

4. The method for generating a foggy image according to claim 1, wherein: In the scene transmittance estimation step, the scene transmittance estimation model obtains the scene transmittance estimation map t(x) according to the scene depth estimation value D(x): β is a factor that determines the concentration of fog and ranges from (0,1).

5. The method for generating a foggy image according to claim 1, wherein: In the step of generating the scene fog map, the calculation formula of the fog intensity generation model is as follows: Fog(x)=A(x)·(1-t(x))+Poisson(x), Where A(x) is the atmospheric light value.

6. The method for generating a foggy image according to claim 1, wherein: The foggy image synthesis step uses an image fusion method to fuse the fog intensity map Fog(x) and the brightness adjustment map J1(x), and performs a sharpening filter on the fused image to obtain the final foggy image. The specific implementation is as follows: I(x)=Sharpe[J1(x)×t(x)+Fog(x)], Among them, Shape[·] is the sharpening filter operator, that is: I sharpe (x)=I(x)-L(I(x)), Where L is a low-intensity Laplace sharpening kernel with a size of: γ is the factor that determines the degree of sharpness.

7. The method for generating a foggy image according to claim 1, wherein: The monocular visual depth estimation network includes Depth-Anything-Net.

8. The method for generating a foggy image according to claim 2, wherein: The super-resolution processing is specifically performed using the Lanczos interpolation method.

Images