A method for image dehazing without prior knowledge based on multi-frame reconstruction and rendering

Abstract

The present invention discloses a method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering. First, a three-dimensional point cloud space is obtained by reconstructing the original image array, and then voxels with low transparency are removed from the three-dimensional point cloud space and re-rendered. A suitable transparency threshold is determined according to the changes in different rendering results, and the optimal rendered defogging image is output. Finally, the optimal rendered defogging image and the low-resolution rendered foggy image are residually fused, and the original foggy image is used as a reference for super-resolution to obtain a high-resolution rendered defogging image. The present invention no longer requires various prior knowledge of image defogging, and does not require a large number of training data sets. The generated data has higher credibility and natural effects, and the credibility and accuracy of the image defogging problem are greatly improved. It can be applied in the fields of remote sensing defogging, underwater defogging, night defogging, non-uniform defogging, etc. And it can be extended to image dereflection, image ghost removal and other problems.

Description

Technical Field

[0001] The present invention relates to an image defogging method in the fields of image processing technology, computer vision and deep learning, and specifically relates to a method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering. Background Art

[0002] Image dehazing tasks have a wide range of applications in the field of computer vision, such as target recognition, medical image processing, aerospace imaging, etc. The image dehazing task aims to restore a normal image from an image with medium scattering during the imaging process. This problem is pathological because a single foggy image has serious information loss compared to a fog-free image of the same scene.

[0003] In recent years, with the development of deep learning, various algorithms for processing complex haze scenes have been continuously improved, including those based on complex network structures, those based on optical atmospheric priors, and those based on depth estimation. However, none of them can accurately perceive scene information. Therefore, there are still problems such as insufficient algorithm robustness, poor scene adaptation, and the need for data set drive.

[0004] Since lighting and scenes are prone to change, it has always been difficult to collect dehazing training datasets. Compared with indoors, real outdoor datasets during the day and at night are difficult to obtain. Existing real datasets are all based on near-field non-uniform haze generated by fog machines, which is different from real haze scenes. Simulation methods have also been rapidly iterated, gradually transitioning from superimposing haze masks to rendering and other methods. However, there is still a huge gap in diversity and authenticity compared to real datasets.

[0005] As long as we can train network algorithms without relying on datasets, it will be possible to generate a large number of real-life datasets. Some methods have proposed multi-frame disparity estimation to remove haze, but this method can only use weak information from disparity and cannot overcome the impact of haze on disparity calculation. Relying on prior information, the estimation of haze is still not accurate and reliable enough. How to remove haze accurately and robustly has become an important research topic. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art, solve the problems existing in the background technology, and provide a method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering, which can defog a foggy image and generate a defogging image with high credibility, high precision, and no dependence on the scene.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] S1: In a foggy scene, multiple original foggy images with displacement relationships are collected and formed into an original image array; then the original image array is reconstructed by three-dimensional matching to obtain the camera extrinsic parameters corresponding to the original image array and the three-dimensional point cloud space of the foggy scene;

[0009] S2: training the neural radiation field according to the camera extrinsics corresponding to the original image array, the three-dimensional point cloud space of the foggy scene and the original image array to obtain a trained neural radiation field, setting an initial transparency threshold of the trained neural radiation field, taking the shooting angle of an original foggy image in the original image array as the target shooting angle, and rendering the trained neural radiation field to the target shooting angle to obtain a low-resolution rendered foggy image;

[0010] S3: deleting voxels smaller than the current transparency threshold in the trained neural radiation field to obtain the voxel threshold deleted neural radiation field, rendering the target shooting angle with the voxel threshold deleted neural radiation field to obtain a rendered defogging image;

[0011] S4: adding a sampling threshold to the current transparency threshold and updating the transparency threshold, repeating S3, obtaining a rendered dehazed image after the current sampling threshold is increased, calculating the image structure similarity SSIM between the two rendered dehazed images before and after the current sampling threshold is increased, then adding the same sampling threshold to the current transparency threshold for multiple times in sequence and updating the transparency threshold, repeating S3, obtaining the rendered dehazed image after each same sampling threshold, thereby obtaining the image structure similarity SSIM between the two rendered dehazed images before and after each increase in the sampling threshold, taking one of the two rendered dehazed images before and after with the maximum image structure similarity SSIM as the optimal rendered dehazed image, and the transparency threshold corresponding to the optimal rendered dehazed image as the optimal transparency threshold;

[0012] S5: Based on the optimal transparency threshold, the low-resolution rendered foggy image, the optimal rendered defogging image, and the original foggy image corresponding to the target shooting angle in the original image array are super-fused to obtain a high-resolution rendered defogging image.

[0013] In S1, the image acquisition method of the original image array is an array camera method, a video method or a rendering method.

[0014] The S5 is specifically:

[0015] S5.1: Using an adaptive residual algorithm, the low-resolution rendered foggy image and the optimal rendered defogging image are subtracted to obtain a defogging residual image;

[0016] S5.2: When the ratio of voxels with a transparency of 0 to voxels with a transparency less than the optimal transparency threshold in the trained neural radiation field is less than 50%, the pixel values ​​of each channel image of the dehazing residual map are doubled and used as the illumination-corrected dehazing residual map; otherwise, the dehazing residual map is subjected to illumination correction according to the low-resolution rendered foggy image to obtain the illumination-corrected dehazing residual map;

[0017] S5.3: Use a super-resolution algorithm to perform super-resolution fusion of the original foggy image corresponding to the target shooting angle in the original image array and the illumination correction defogging residual image with a reference image to obtain a high-resolution rendered defogging image.

[0018] The super-resolution algorithm is a reference super-resolution RefSR algorithm.

[0019] Beneficial effects of the present invention:

[0020] The present invention does not require a large amount of data training or prior knowledge. For haze images, haze can be accurately removed by array multi-frame method. The equivalent properties of voxel integral rendering in neural radiation field and light medium scattering in haze imaging are utilized to accurately and reliably calculate the haze space and remove the influence of transparent medium (haze) on imaging.

[0021] With appropriate real-shot haze array data, the present invention can generate a large number of credible haze removal training data sets to provide support for other single-frame defogging algorithms.

[0022] The present invention no longer requires a large amount of haze training data sets for image defogging, and no longer requires any physical priors such as atmospheric scattering. Therefore, it can be widely used in remote sensing defogging, underwater defogging, night defogging, ordinary defogging, non-uniform defogging and other scenes. It can also be extended to image dereflection, image ghost removal and other problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the present invention;

[0024] Figure 2 Schematic diagram of the relationship between voxel deletion and image defogging in the method of the present invention.

[0025] Figure 3 It is a schematic diagram of an intermediate result image of the present invention;

[0026] Figure 4 It is a schematic diagram of the voxel deletion change principle of the present invention.

[0027] Figure 5 This is a diagram showing the experimental results of voxel deletion changes in the present invention.

[0028] Figure 6 These are the defogging results of the simulation and real-shot data of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0030] The embodiments and implementation processes fully implemented according to the content of the present invention are as follows:

[0031] like Figure 1 As shown, the present invention comprises the following steps:

[0032] S1: In a foggy scene, multiple original foggy images with displacement relationships are collected and formed into an original image array; then the original image array is reconstructed by three-dimensional matching to obtain the camera extrinsics corresponding to the original image array and the three-dimensional point cloud space of the foggy scene; the original foggy image described in the present invention refers to the image after scattering caused by the presence of a translucent medium in the optical path from the object to the camera. Compared with fog-free images, foggy images have lower saturation and clarity. In specific implementations, three-dimensional matching reconstruction mainly relies on the colmap tool of the open source traditional method. Input the array image obtained in the image acquisition, and output the camera parameter information of camera.bin, the array image information of images.bin, the three-dimensional point cloud file of point3D.bin, the reconstruction project file of project.ini colmap, and the reconstruction basic data file of database.db.

[0033] In S1, the image acquisition method of the original image array is an array camera method, a video method or a rendering method.

[0034] The array camera method is not affected by the displacement of haze over time. The lens focus, white balance, exposure, tone mapping and other parameters of all cameras in the array camera are set to be consistent (or similar). Then all cameras are used to shoot the foggy scene at the same time to obtain their respective original foggy images and form an original image array.

[0035] The video method uses a single camera to move up, down, left, and right, directly capture multiple frames of images or capture a video and extract frames, and the multiple frames or images obtained by extracting frames form the original image array. In this method, parameters such as white balance, focus, exposure, and tone mapping change slightly, but the stability requirements for haze are high.

[0036] The rendering method uses a computer vision rendering engine (such as Unreal Engine) to obtain data, creates exponential height fog and volume fog in the scene project file to generate haze, and obtains the foggy image rendered by the engine by changing the virtual camera to obtain the original image array. This method does not consider haze movement and camera parameters, but there may be differences between the created scene and the actual scene.

[0037] According to the physical model of the haze imaging process, the depth relationship is as follows:

[0038] t(x)=e -βd(x)

[0039] I(x)=R(x)*t(x)+L(x)*(1-t(x))

[0040] Where x is the current pixel image signal, t(x) represents the amount of transmission attenuation, e indicates exponential attenuation, also known as exponential fog, β is the attenuation exponent, representing the unit attenuation intensity, I(x) is the image signal received by the camera, R(x) represents the signal reflected by the object, d(x) represents the depth of light propagation in the atmosphere, and L(x) represents the global atmospheric background light.

[0041] However, the common dehazing model is two-dimensional and cannot support three-dimensional radiation fields.

[0042] Based on the optical scattering model, the two-dimensional defogging model is no longer used, and the haze formula can also be expressed as:

[0043] dL(x,ω) / dx=-σ a L(x,ω)

[0044] σ a is the attenuation absorption coefficient, ω and ω′ are the directions of incidence and emission for the same medium in light scattering, and L(x, ω) represents the light scattering propagation function in the ω direction at the x position.

[0045] Based on the optical scattering model, there is absorption, internal scattering and external scattering in haze. The formula can be specifically as follows:

[0046]

[0047] σ t =σ0+σ s

[0048] Among them, σ t represents the sum of the light attenuation absorption coefficient and the light outscattering coefficient, σ s is the light outward scattering coefficient, p is the current position of the scattering medium, and f p () is the isotropic scattering coefficient at that position, Le(x, ω) is the stimulated light scattering propagation function, and s represents the spatial scattering range.

[0049] Although this formula can handle all optical scattering problems, it is very complicated due to a large number of three-dimensional integrals. In order to facilitate calculation, the high-dimensional part is simplified using the idea of ​​computer rendering. The formula is:

[0050]

[0051]

[0052] Among them, M is the initial position, L i () is the scattering propagation function affected by external light, T(x) represents the transmission attenuation of the sum of the light attenuation absorption coefficient and the light external scattering coefficient from position p to position x, L(M, ω) represents the integration result of L(p, ω) in the ω direction of position M in the previous stage, and p represents the position of the integration space.

[0053] Light scattering is a dynamic process. After a scattering, the light will continue to scatter until it reaches a steady state. The formula is:

[0054] G n+1 =G n *f ms

[0055] Among them, G n is the state after the nth scattering, f ms This is the effect of a single scattering.

[0056] Rendering and scattering equations often calculate the approximate results of the first few scatterings, but the real haze scene is the result of an infinite series of scatterings. We believe that although the intensity of each scattering is different, the scattering law is consistent, so the infinite scattering can be approximated as a first-order scattering of another intensity. The formula is as follows:

[0057]

[0058] Among them, F ms This is an infinite scattering effect, which means that a three-dimensional scattering haze scene can also be approximated by a first-level rendering formula.

[0059] The integral relationship formula for the three-channel (r, g, b) color C of the three-dimensional neural radiation field that finally enters the camera is as follows:

[0060] r(t)=o+td

[0061]

[0062]

[0063] Where o is the three-dimensional position of the camera pixel, d is the three-dimensional vector of the camera's outgoing light, t represents the distance the camera emits, and t n ,t f is the distance at which the integration starts and ends, r(t) is the spatial position of the voxel at distance t, σ() is the transparency of that position, c is the voxel color at that position, and T(t) is the accumulated transparency of the ray.

[0064] Comparing the formula of the three-dimensional neural radiance field with the formula for scattering and rendering approximation derived above, it can be considered that for a hazy scene, the three-dimensional neural radiance field can be used for fitting.

[0065] S2: Train the neural radiance field according to the external camera parameters corresponding to the original image array, the three-dimensional point cloud space of the hazy scene, and the original image array to obtain a trained neural radiance field. Set the initial transparency threshold of the trained neural radiance field. Take the shooting angle of an original hazy image in the original image array as the target shooting angle, and render the target shooting angle with the trained neural radiance field to obtain a low-resolution rendered hazy image.

[0066] S3: Delete the voxels in the trained neural radiance field that are less than the current transparency threshold to obtain a voxel threshold-deleted neural radiance field. Render the target shooting angle with the voxel threshold-deleted neural radiance field to obtain a rendered dehazed image. The purpose of voxel deletion is to limit the alpha transparency parameter of the result of the neural network field. Image calculation is performed on the seven parameters (r, g, b)(x, y, z)(a < t) to obtain I(t). The spatial voxels less than t in space become transparent, which is equivalent to being deleted and will not enter the subsequent rendering process. The deleted voxels are continuously multiplied in a specific direction to obtain weights. Multiplying the weights and the color matrix (r, g, b) can obtain the value of a certain pixel in the image. Continuously repeat until the specific values of all pixel values in the image are calculated. Calculate the structural similarity index measure (SSIM) of the images for calculating the thresholds I(t) and I(t + Δt). When its change is the smallest, obtain the threshold t' and the most suitable dehazed image I(t'). As Figure 2 shown, Figure 2 (a1) and (a2) in are the comparison diagrams before and after voxel deletion, Figure 2 (b1) and (b2) in are the original hazy image and the rendered dehazed image in a non-uniform haze scene, Figure 2 (c1) and (c2) in are the original hazy image and the rendered dehazed image in a uniform haze scene. It can be seen that the present invention can maintain the normal color of the image, avoid the obvious over-saturation problem in image dehazing, and has a very good effect on suppressing non-uniform haze. The experimental results also prove that this method is applicable to remote sensing dehazing, night dehazing, non-uniform dehazing, and underwater dehazing.

[0067] S4: adding a sampling threshold to the current transparency threshold and updating the transparency threshold, repeating S3, obtaining a rendered dehazed image after the current sampling threshold is increased, calculating the image structure similarity SSIM between the two rendered dehazed images before and after the current sampling threshold is increased, then adding the same sampling threshold to the current transparency threshold for multiple times in sequence and updating the transparency threshold, repeating S3, obtaining the rendered dehazed image after each same sampling threshold, thereby obtaining the image structure similarity SSIM between the two rendered dehazed images before and after each increase in the sampling threshold, taking one of the two rendered dehazed images before and after with the maximum image structure similarity SSIM as the optimal rendered dehazed image, and the transparency threshold corresponding to the optimal rendered dehazed image as the optimal transparency threshold;

[0068] The schematic diagram of voxel deletion change principle is as follows Figure 4 As shown in the figure, multi-image 3D reconstruction MPI is often discrete in depth, so the sampling is also approximate, and it is impossible to distinguish the real gray and black boundaries. After that, the sampling results are voxel deleted. As the threshold t changes, the voxels in the sampling space continue to fall off. Compared with the black and white junction, the gray haze and black opaque objects have a larger area. Therefore, when the voxels are first deleted, the changes are relatively fast, and the voxels that fall off in the middle are all at the junction of the haze and the opaque objects, and finally the voxels of the objects begin to fall off completely. Therefore, under the same △t, the voxels select the interval with the smallest change in the middle to obtain the threshold t, which can remove the haze to the greatest extent and obtain image information.

[0069] The experimental results of voxel deletion change are shown in the figure below. Figure 5 As shown, Figure 5 (a) is the peak signal-to-noise ratio (PSNR) of the rendered dehazed image corresponding to different voxel thresholds t under different haze concentrations compared to the rendered dehazed image corresponding to the nearest neighboring voxel threshold of t, (b) is the peak signal-to-noise ratio (PSNR) of the rendered dehazed image corresponding to different voxel thresholds t under different haze concentrations compared to the real haze-free image, (c) is the image structure similarity (SSIM) of the rendered dehazed image corresponding to different voxel thresholds t under different haze concentrations compared to the rendered dehazed image corresponding to the nearest neighboring voxel threshold of t, (d) is the image similarity (SSIM) of the rendered dehazed image corresponding to different voxel thresholds t under different haze concentrations compared to the real haze-free image. Interval PSNR and interval SSIM are comparisons of images from thresholds t and t+Δt. PSNR and SSIM by GT are comparisons of images under different t thresholds and haze-free images. The experiment also proves the feasibility of the threshold method. The indoor real-shot images also show the same distribution relationship. The lines of different colors from 0 to 0.2 represent different haze concentrations.

[0070] S5: Based on the optimal transparency threshold, the low-resolution rendered foggy image, the optimal rendered defogging image, and the original foggy image corresponding to the target shooting angle in the original image array are super-fused to obtain a high-resolution rendered defogging image. Figure 3 As shown, Figure 3 (a) is the optimal rendered dehazed image. Figure 3 (b) is a low-resolution rendered foggy image. Figure 3 (c) is the dehazing residual image. Figure 3 (d) is a high-resolution rendered dehazed image. Figure 3 (e) is the original foggy image.

[0071] S5 is as follows:

[0072] S5.1: The haze creates a double scattering effect when the light hits the object and reflects into the camera, which needs to be compensated by the adaptive residual algorithm. The low-resolution rendered foggy image I' is converted into haze And the optimal rendered dehazed image I' dehaze After the difference, we get the dehazing residual image I mask ;

[0073] S5.2: When the ratio of voxels with a transparency of 0 to voxels with a transparency less than the optimal transparency threshold in the trained neural radiation field is less than 50%, the current foggy scene is a global haze scene, and the defogging residual image I mask The pixel values ​​of each channel are doubled and used as the residual image of illumination correction and defogging I' mask Otherwise, the current foggy scene is a local non-uniform haze scene, and the foggy image I' is rendered according to the low resolution haze For the dehazing residual image I mask Perform lighting correction to obtain the lighting correction defogging residual image I' mask ;

[0074] S5.3: Use super-resolution algorithm to calculate the original foggy image I corresponding to the target shooting angle in the original image array haze And the illumination correction dehazing residual map I' mask Perform super-resolution fusion with reference images to obtain high-resolution rendered dehazed images.

[0075] The super-resolution algorithm is any existing reference super-resolution RefSR algorithm.

[0076] In the specific implementation, the simulation and real-shot data defogging results of the present invention are shown in the figure Figure 6 As shown, it can be seen that the present invention can retain scene details and remove complex haze at night.

[0077] Table 1 is a schematic diagram of the relationship between the actual frame number and the dehazing quality of the present invention. The calculation accuracy of the entire voxel (haze particle) is limited by the number of sampled images and the discrete sampling accuracy of the network. The higher the network sampling accuracy, the higher the calculation cost. As shown in Table 1, as the number of images increases, the haze reconstruction effect of 4 to 10 images increases rapidly. The effect of 10 to 20 images is slightly higher, and sometimes interference occurs. However, the haze removal effect of 4 to 10 images also increases rapidly, and the growth rate is more obvious than that of haze reconstruction. The effect of 10 to 20 images slows down, but it is also more obvious than that of haze reconstruction.

[0078] Table 1: Schematic diagram of the relationship between the number of real-time shooting frames and haze reconstruction and removal

[0079]

[0080] This invention is the first to use neural radiation field three-dimensional reconstruction defogging in the field of multi-frame non-prior image defogging. It realizes the refined processing under complex haze conditions and can produce reliable, robust and high-confidence defogging results with natural effects and good visual quality. Compared with other mentioned image defogging methods, this invention can achieve the best results on both real-shot data and simulated data, and is advanced in the field of image defogging.

Claims

1. A method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering, characterized in that: The following steps are involved: S1: In a foggy scene, multiple original foggy images with displacement relationships are collected and formed into an original image array; then the original image array is reconstructed by three-dimensional matching to obtain the camera extrinsic parameters corresponding to the original image array and the three-dimensional point cloud space of the foggy scene; S2: training the neural radiation field according to the camera extrinsics corresponding to the original image array, the three-dimensional point cloud space of the foggy scene and the original image array to obtain a trained neural radiation field, setting an initial transparency threshold of the trained neural radiation field, taking the shooting angle of an original foggy image in the original image array as the target shooting angle, and rendering the trained neural radiation field to the target shooting angle to obtain a low-resolution rendered foggy image; S3: deleting voxels smaller than the current transparency threshold in the trained neural radiation field to obtain the voxel threshold deleted neural radiation field, rendering the target shooting angle with the voxel threshold deleted neural radiation field to obtain a rendered defogging image; S4: adding a sampling threshold to the current transparency threshold and updating the transparency threshold, repeating S3, obtaining a rendered dehazed image after the current sampling threshold is increased, calculating the image structure similarity SSIM between the two rendered dehazed images before and after the current sampling threshold is increased, then adding the same sampling threshold to the current transparency threshold for multiple times in sequence and updating the transparency threshold, repeating S3, obtaining the rendered dehazed image after each same sampling threshold, thereby obtaining the image structure similarity SSIM between the two rendered dehazed images before and after each increase in the sampling threshold, taking one of the two rendered dehazed images before and after with the maximum image structure similarity SSIM as the optimal rendered dehazed image, and the transparency threshold corresponding to the optimal rendered dehazed image as the optimal transparency threshold; S5: Based on the optimal transparency threshold, the low-resolution rendered foggy image, the optimal rendered defogging image, and the original foggy image corresponding to the target shooting angle in the original image array are super-resolution fused to obtain a high-resolution rendered defogging image.

2. The method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering according to claim 1, characterized in that: In S1, the image acquisition method of the original image array is an array camera method, a video method or a rendering method.

3. The method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering according to claim 1, characterized in that: The S5 is specifically: S5.1: Using an adaptive residual algorithm, the low-resolution rendered foggy image and the optimal rendered defogging image are subtracted to obtain a defogging residual image; S5.2: When the ratio of voxels with a transparency of 0 to voxels with a transparency less than the optimal transparency threshold in the trained neural radiation field is less than 50%, the pixel values ​​of each channel image of the dehazing residual map are doubled and used as the illumination-corrected dehazing residual map; otherwise, the dehazing residual map is subjected to illumination correction according to the low-resolution rendered foggy image to obtain the illumination-corrected dehazing residual map; S5.3: Use a super-resolution algorithm to perform super-resolution fusion of the original foggy image corresponding to the target shooting angle in the original image array and the illumination correction defogging residual image with a reference image to obtain a high-resolution rendered defogging image.

4. The method for defogging an image without prior knowledge based on multi-frame reconstruction and rendering according to claim 3, characterized in that: The super-resolution algorithm is a reference super-resolution RefSR algorithm.

Images