A method for generating a fog-containing image based on monocular vision depth estimation

Abstract

This invention discloses a method for generating foggy images based on monocular vision depth estimation. Aimed at applications in foggy image simulation, this method employs an improved physical model and an improved depth map estimation model to process fog-free images and simulate realistic foggy images, meeting the application needs of image processing research such as image dehazing and target detection in foggy images. First, this invention uses a monocular vision depth estimation network to generate an initial relative depth map of the scene. Then, considering the characteristics of foggy images, it uses image super-resolution and edge smoothing techniques to optimize the initial depth estimation map, obtaining an optimized depth map of the scene. Finally, based on the optimized depth map, an improved transmittance generation model is used to generate a transmittance map of the scene. Finally, the improved transmittance map is fused with the scene image to obtain a realistic foggy image. This invention can be applied to fields such as autonomous driving simulation, forest fire prevention, remote sensing positioning, and virtual reality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and image processing, and more particularly relates to a fog image generation method based on monocular vision depth estimation. BACKGROUND

[0002] The generation of fog images is of great significance in many fields. First, in the field of computer vision and artificial intelligence, fog images can be used to train and test deep learning models, improving the robustness of algorithms in tasks such as target detection, object recognition, and semantic segmentation under adverse weather conditions. Second, in the field of autonomous driving and intelligent transportation, fog is an important factor affecting driving safety. Synthesizing fog images can simulate different visibility conditions, which can be used to test and optimize perception systems to ensure that autonomous vehicles can accurately recognize road information in low-visibility environments. In addition, in the field of remote sensing and satellite image analysis, simulating foggy environments helps to develop de-fogging enhancement techniques, improving the accuracy of tasks such as aerial monitoring and environmental assessment. The generation of fog images also plays a role in the fields of film and television production, game development, and augmented reality (AR), providing support for realistic rendering of virtual scenes, making foggy environments more natural and lively.

[0003] Traditional fogging methods are mostly based on simple image filtering techniques or fixed physical models, and the generated fog effect lacks realism, making it difficult to accurately reflect the optical properties of natural scenes. In recent years, with the development of deep learning technology, monocular vision depth estimation networks have made significant progress, enabling high-precision depth information estimation from monocular images, providing a new solution for de-fogging and fogging tasks. However, existing methods still have deficiencies in terms of realism, adaptability, and computational efficiency. SUMMARY

[0004] To address the technical problem of the deficiencies of traditional monocular vision depth estimation networks in terms of realism, adaptability, and computational efficiency, the present application proposes a fog image generation method based on monocular vision depth estimation, which aims to combine deep learning and physical models to achieve efficient, realistic, and flexible fog effect generation, addressing the deficiencies of traditional methods in terms of realism, hardware dependence, and computational complexity.

[0005] To address the problem of the small number of existing scene-specific datasets and the unrealistic fog images generated, the present application provides a fog image generation method based on monocular depth estimation. The present application combines a monocular vision depth estimation network to obtain a monocular depth estimation map and optimize it, and then uses the optimized depth map to obtain the required fog, including the following steps:

[0006] Scene depth estimation: a monocular vision depth estimation network is used to process a real-world haze-free scene image J(x) to obtain an initial depth estimation map of the scene, and then a depth map optimization processing method is used to process the initial depth estimation map to obtain an optimized depth estimation map, the depth value is normalized to [0, 1], and is denoted as D(x), x is an image pixel index value;

[0007] Scene brightness adjustment: based on the fact that the ambient light intensity of a haze scene is lower than that of a normal scene, an image brightness self-adaptive adjustment method is used to adjust the brightness of a real-world haze-free image J(x) to make it conform to the image brightness of a haze scene, and the adjusted image is denoted as J1(x);

[0008] Scene transmittance estimation: a scene transmittance estimation model is used to process the depth estimation map D(x) to obtain a scene transmittance estimation map t(x);

[0009] Scene fog map generation: based on the scene transmittance estimation map t(x), a fog intensity generation model is used to generate a scene fog intensity map Fog(x);

[0010] Haze image synthesis: an image fusion method is used to fuse the fog intensity map Fog(x) and the brightness adjustment map J1(x), and a sharpening filter is used to process the fused image to obtain a final haze image, denoted as I(x).

[0011] Further, in the scene depth estimation step, when a monocular vision depth estimation model is used to estimate the depth of the scene, the depth estimation at the depth edge is easily disturbed by noise, resulting in inaccurate depth estimation at the edge and affecting the generation of the halo effect at the depth edge in the haze image. Therefore, the application designs a depth map optimization method to optimize the initial depth estimation map, which eliminates the boundary halo effect by smoothing the depth edge.

[0012] Considering that edge smoothing may cause image blurring, before edge smoothing, the initial depth estimation map is first subjected to super-resolution processing, and the super-resolution processed depth map is denoted as D L (x), then the super-resolution processed depth map D L (x) is subjected to edge extraction and smoothing processing, and the basic steps of edge smoothing processing are as follows:

[0013] First step: extract the brightness component of the super-resolution processed depth map D L (x) and obtain the gray inverted depth map D I (x) by inversion, then use an edge detection algorithm to process the super-resolution processed depth map D L (x) to obtain a depth edge map D e (x);

[0014] Second step: to D e (x) is expanded, and the edge expansion graph is denoted as D d (x);

[0015] Third step: the edge expansion graph D d (x) is processed by using a Gaussian blur kernel to obtain an edge weight graph, that is:

[0016] W1=GaussianBlur1(D d (x)),

[0017] W1 is the edge weight graph, and GaussianBlur1(·) is a Gaussian filter kernel.

[0018] Fourth step: fusion is performed according to the following formula to obtain a final optimized depth estimation graph, that is:

[0019] D(x)=D I (x) x (1-W1)+GaussianBlur2(D I (x)) x W1,

[0020] wherein GaussianBlur1(·) and GaussianBlur2(·) are Gaussian filter kernels with different window sizes and different standard deviations, the window sizes are 15 and 25 respectively, and the standard deviations are 10 and 15 respectively.

[0021] Further, in the scene brightness adjustment step, an image brightness adaptive adjustment method is designed to simulate the ambient brightness of a foggy scene, to adjust the brightness of a non-fog image to obtain a scene brightness degradation graph J1(x). The brightness adjustment adopts the following formula:

[0022]

[0023] wherein L mean is the average brightness value of J(x), N(x) is Gaussian noise, k is an adjustment factor for adjusting the brightness change rate, and can be set to 2.5 according to the brightness requirement, and a is an adjustment factor for adjusting the relative value of the signal and the noise.

[0024] Further, in the scene transmittance estimation step, a scene transmittance estimation model is designed, which obtains a scene transmittance estimation graph t(x) according to the scene depth estimation value D(x) by using the following formula:

[0025]

[0026] The design of the present application is different from the commonly used transmittance calculation method, a polynomial fitting method is adopted, the second term β×D(x) on the right side of the above formula 2 The transmittance increases faster at the position with smaller depth, the third term Embodies a sub-linear growth trend, indicating that the attenuation of light is relatively flat at a long distance, and the combination of the two means that t(x) can be larger than the previous exponential decay, which means that the object in the near place will be clearer, in other words, the contrast of the fog concentration generated in the distance and the object in the near place after using the improved transmittance formula can be larger than the previous atmospheric scattering model. The concentration of the fog depends on the size of the fog concentration factor β, where the range of β is between (0, 1).

[0027] In the scene fog map generation step, the fog intensity generation model calculation formula is as follows:

[0028] Fog(x)=A(x)·(1-t(x))+Poisson(x),

[0029] Where A(x) is the atmospheric light value (fog color value), since the fog scene is generally in a dark day, the color of the sky is also more gray, at this time if the white fog is too bright, the restoration of the fog scene is not real, so choosing a suitable atmospheric light value is the key to fogging, here it is defined as a constant value vector. In the real world, the fog is almost non-uniform, so the present application combines the non-uniform characteristics of the fog and uses Poisson distribution to generate random disturbance noise to simulate the randomness of the non-uniform change of the fog intensity.

[0030] In the fog-containing image synthesis step, the image fusion method is adopted to fuse the fog intensity map Fog(x) and the brightness adjustment map J1(x), and the fused image is sharpened and filtered to obtain the final fog-containing image, which is realized by the following formula:

[0031] I(x)=Sharpe[J1(x)×t(x)+Fog(x)],

[0032] Considering that slight sharpening can highlight the details of the objects behind the fog, the fog scene of this method can be observed.

[0033] Where Shape[·] is a sharpening filter operator, that is:

[0034] I sharpe (x)=I(x)-L(I(x)),

[0035] Where L is a low-intensity Laplace sharpening kernel with a size of:

[0036]

[0037] Gamma is a sharpening degree determining factor.

[0038] Overall, the above technical solutions constructed by the present application can achieve the following beneficial effects:

[0039] 1. The present application aims at the problem that the existing monocular vision depth estimation network is easily disturbed by noise at the depth edge, adopts a depth estimation optimization method, and through image super-resolution processing and depth edge smoothing processing, the depth details are retained, the problem of inaccurate depth estimation caused by noise disturbance of the depth edge is overcome, and the generation effect of the haze-free image is improved.

[0040] 2. The present application proposes an improved scene transmittance estimation method, enhances the contrast of far transmittance and near transmittance, and can make the scene transmittance estimation more smooth and more consistent with the real complex foggy scene.

[0041] 3. The present application combines the characteristics of non-uniformity of real-world fog, adopts Poisson distribution to randomly disturb the scene fog intensity distribution, makes the fog intensity distribution more close to the real scene, avoids excessive smoothness and distortion of the fog, and enhances the level of the fog map. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The present application proposes a whole flow chart of haze-containing image generation.

[0043] Figure 2 The scene depth estimation and optimization processing flow chart.

[0044] Figure 3 And Figure 4 The present application proposes an algorithm processing effect example diagram. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0046] With reference to Figure 1 The monocular depth estimation-based haze-containing image generation method of the present embodiment is implemented by the following steps.

[0047] S1: Scene depth estimation: The present application uses Depth-Anything-Net as monocular vision depth estimation network to process real world haze-free scene image J(x), so as to obtain initial scene depth estimation map. Depth-Anything-Net is faster and more accurate than other monocular vision depth estimation algorithms, and is a very suitable monocular depth estimation network for haze-containing image generation.

[0048] Reference Figure 2 Considering that edge smoothing may cause blurring of the image, before edge smoothing processing, the initial depth estimation map is first subjected to super-resolution processing, and the depth map after super-resolution processing is denoted as D L (x). The super-resolution processing uses Lanczos interpolation method, which is a high-quality interpolation method based on sinc function, and can better preserve details and sharpen effects when scaling images. Since the quality of the generated depth map is generally good, Lanczos interpolation can prevent severe jagged effects and make the edges smoother, without causing severe loss of details. The present application uses a=4, that is, the calculation of each interpolation point considers the pixels in an 8x8 neighborhood.

[0049] Then the depth map D L (x) after super-resolution processing is subjected to edge extraction and smoothing processing, and the basic steps of edge smoothing processing are as follows:

[0050] The first step is to convert the three-channel pseudo-color depth estimation map after super-resolution processing interpolation into a gray-reversed depth map, that is:

[0051] D I (x) = 1-(0.299xR+0.587xG+0.114xB),

[0052] R, G and B represent the values of red (Red), green (Green) and blue (Blue), respectively.

[0053] Subsequently, the present application uses Canny edge detection algorithm to process the depth map D L (x) after super-resolution processing in the edge extraction step, to obtain a depth edge map D e (x). Specifically, first, Gaussian filtering is used to denoise the image, with a window size of 5x5 and a standard deviation of 1.4. Then, Sobel detection operator is used to extract edges in the horizontal and vertical directions of the image and calculate the gradient amplitude and gradient direction. Next, in the non-maximum suppression algorithm, if the current pixel is smaller than the adjacent pixels in the same direction, it is set to 0, and only the local maximum is considered as a candidate edge point. Subsequently, double-threshold detection is performed, with the low threshold set to 50 and the high threshold set to 150.

[0054] Second step, after Canny edge detection, the invention uses the graph morphological dilation algorithm to process the depth edge map D e (x), and expand the depth edge, and record the edge expansion map as D d (x). The dilation structure element is set to an elliptical structure with a window size of 5x5, and the size is consistent with the filter kernel size of the Gaussian Canny edge detection algorithm.

[0055] Third step, in order to obtain the edge weight map W1, the invention first uses a Gaussian filter kernel to blur the edge expansion map D d (x), and the filter kernel window size is 15x15 and the standard deviation is 10.

[0056] Fourth step, the filter kernel window size is 25x25 and the standard deviation is 15, and D I (x) is processed to obtain GaussianBlur2(D I (x)), the edge weight map W1 obtained in the third step is fused with D I (x), and GaussianBlur2(D I (x)), and finally the optimized depth estimation map is obtained, that is:

[0057] D(x) = D I (x) x (1-W1) + GaussianBlur2(D I (x)) x W1.

[0058] S2: scene brightness adjustment: the ambient light intensity of the foggy scene is lower than that of the normal scene, an image brightness self-adaptive adjustment method is used to adjust the brightness of the real world non-fog image J(x) to make it conform to the image brightness of the foggy scene, and the adjusted image is recorded as J1(x). The invention uses an average brightness self-adaptive Gamma transformation to adjust the brightness and contrast of the image to obtain an image brightness degradation map,

[0059]

[0060] L mean is the average brightness of the image, in this embodiment, the value of the adjustment factor k is 2.5, and a is an adjustment factor for adjusting the relative value of the signal and the noise, and in this example, the value is 0.85, the mean of the Gaussian noise is 0, and the standard deviation is 15.

[0061] S3: Scene transmittance estimation: a scene transmittance estimation model is used to process the depth estimation map D(x) to obtain a scene transmittance estimation map t(x). The present application designs a scene transmittance estimation model, which obtains the transmittance estimation map t(x) of the scene according to the scene depth estimation value D(x) by using the following formula:

[0062]

[0063] The design of the present application is different from the commonly used transmittance calculation method, and a polynomial fitting method is used, and the second term β×D(x) on the right side of the above formula 2 makes the transmittance increase faster at positions with smaller depth, and the third term embodies a sub-linear growth trend, indicating that the attenuation of light is relatively flat at a long distance, and the combination of the two means that t(x) can be larger than the previous exponential decay, meaning that the objects near the camera will be clearer. In other words, the contrast between the fog density in the distance and the objects near the camera generated by the improved transmittance formula can be larger than that of the previous atmospheric scattering model. The concentration of fog depends on the size of the fog concentration factor β, where β is in the range (0, 1).

[0064] The present application defines the value of β as an adjustable value, which is related to the concentration of the fog to be generated. If the value of β is increased, the fog in the distance will be thicker, and the fog near the camera will be reduced, and the contrast between the near and far scenes will be increased, for example, α=0.8. Conversely, if β is reduced, there will be fog near the camera, and the overall picture will look smoother, for example, β=0.2. In Figure 3 and Figure 4 , the value of β of the present embodiment is 0.5, and a moderate haze image is generated.

[0065] S4: Scene fog map generation: based on the scene transmittance estimation map t(x), a fog intensity generation model is used to generate a scene fog intensity map Fog(x). The calculation formula of the fog intensity generation model is as follows:

[0066] Fog(x)=A(x)·(1-t(x))+Poisson(x),

[0067] where A(x) is the atmospheric light value (fog color value). Since the fog scene is generally in a dark day, the color of the sky is also more gray, at this time, if white fog is used, the light will be too bright, and the restoration of the fog scene is not true, so selecting a suitable atmospheric light value is the key to fogging. Here it is defined as a constant value vector. In the real world, almost all fogs are non-uniform, so the present application combines the non-uniform characteristics of fog and uses Poisson distribution to generate random disturbance noise to simulate the randomness of the non-uniform change of fog intensity.

[0068] The present application obtains the fog intensity map to be generated, and the color of the fog depends on the value of A(x). The color of the fog obtained by the previous algorithm is too bright, and does not conform to the real overcast and foggy scene. Therefore, the present embodiment takes the atmospheric light value as The fog is light gray, which conforms to the overcast scene, and Poisson noise is added to generate non-uniform fog.

[0069] S5: fog image generation: the present application fuses the image brightness degradation map generated in S2, the scene transmittance estimation map generated in S3 and the fog intensity map generated in S4 to obtain the fog image. The following formula is used to realize the fusion:

[0070] I(x)=Sharpe[J1(x)×t(x)+Fog(x)],

[0071] Considering that slight sharpening can highlight the details of the objects behind the fog, the overcast and foggy scene of the method can be more observable.

[0072] wherein Shape[·] is a sharpening filter operator, i.e.:

[0073] I sharpe (x)=I(x)-L(I(x)),

[0074] wherein L is a low-intensity Laplace sharpening kernel, and the size is:

[0075]

[0076] γ is a sharpening degree determining factor.

[0077] The present application then performs sharpening filtering on the fog image to strengthen the small details of the objects behind the fog, thereby obtaining the observable overcast and foggy image I(x). Since high-intensity sharpening can produce unnecessary noise, only low-intensity sharpening filtering is needed in the generation of the fog image. In the present embodiment, the value of γ should not be too large, and the value of γ in the present embodiment is 0.2.

[0078] The above is easily understood by those skilled in the art, and the above is only a preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a fog image based on monocular vision depth estimation, characterized in that, Comprising the following steps: Scene depth estimation: a monocular vision depth estimation network is used to process the real-world haze-free scene image J(x) to obtain an initial depth estimation map of the scene, and then a depth map optimization processing method is used to process the initial depth estimation map to obtain an optimized depth estimation map, the depth value is normalized to [0, 1], and is denoted as D(x), x is the index value of the image pixel point; Scene brightness adjustment: based on the fact that the ambient light intensity of the haze scene is lower than that of the normal scene, an image brightness self-adaptive adjustment method is used to adjust the brightness of the real-world haze-free image J(x) to make it conform to the image brightness of the haze scene, and the adjusted image is denoted as J1(x); Scene transmittance estimation: a scene transmittance estimation model is used to process the depth estimation map D(x) to obtain a scene transmittance estimation map t(x); Scene fog map generation: based on the scene transmittance estimation map t(x), a fog intensity generation model is used to generate a scene fog intensity map Fog(x); Scene depth estimation step, the depth map optimization processing method for processing the initial depth estimation map specifically includes the following steps: W1 is an edge weight map, and GaussianBlur1(·) is a Gaussian filter kernel; First, the initial depth estimation map is super-resolution processed, and the super-resolution processed depth map is denoted as D L (x). Then, the super-resolution processed depth map D L (x) is edge extracted and smoothed, and the steps of the edge smoothing processing are as follows: First step: extract the super-resolution processed depth map D L (x) and perform inversion to obtain the gray-scale inverted depth map D I (x), and then use an edge detection algorithm to process the super-resolution processed depth map D L (x) to obtain the depth edge map D e (x); Second step: D e (x) is dilated to expand the depth edges, and the edge expansion map is denoted as D d (x). Step 3: using a Gaussian blur kernel to expand the edge map D d (x) performing processing to obtain an edge weight map, specifically: W1 = GaussianBlur1(D d (x)) Fourth step: fusion is carried out according to the following formula to obtain the final optimized depth estimation map, specifically: Where GaussianBlur1(·) and GaussianBlur2(·) are Gaussian filter kernels with different window sizes and different standard deviations; D(x) = D I (x) x (1 - W1) + GaussianBlur2(D I (x)) x W1, In the scene transmittance estimation step, the scene transmittance estimation model obtains the transmittance estimation map t(x) of the scene according to the scene depth estimation value D(x) by the following formula: Beta is the concentration-dependent factor of fog, ranging from 0 to 1. ， 2. The haze image generation method according to claim 1, wherein In the scene brightness adjustment step, an image brightness self-adaptive adjustment method is used to adjust the brightness of the haze-free image to obtain a scene brightness degradation map J1(x), and the brightness adjustment uses the following formula:

3. The haze image generation method according to claim 1, wherein ， where L mean is the average luminance value of J(x), N(x) is Gaussian noise, k is an adjustment factor for adjusting the luminance change rate, which is set according to the luminance requirement, and a is an adjustment factor for adjusting the relative value of the signal and the noise. In the scene fog map generation step, the fog intensity generation model calculation formula is as follows: Fog(x)=A(x)·(1-t(x))+Poisson(x), Where A(x) is the atmospheric light value.

4. The haze image generation method according to claim 1, wherein In the haze image synthesis step, the image fusion method is used to fuse the fog intensity map Fog(x) and the brightness adjustment map J1(x), and sharpening filtering is performed on the fused image to obtain the final haze image, which is realized by the following formula: I(x)=Sharpe[J1(x)×t(x)+Fog(x)], Where Shape[·] is a sharpening filter operator, ​ I sharpe (x) = I(x) - L(I(x)), Wherein, L is a low intensity Laplacian sharp kernel, and the size is: ， Gamma is a sharpness degree determining factor.

5. The fog image containing generation method according to claim 1, characterized by, The monocular vision depth estimation network comprises a Depth-Anything-Net.

6. The fog image containing generation method according to claim 1, characterized by, The super-resolution processing specifically adopts a Lanczos interpolation method.

Images