Depth information prediction method of captured images based on atmospheric light scattering

Abstract

The present invention discloses a method for predicting depth information of captured images based on atmospheric light scattering. The main design concept of the present invention is to use a global atmospheric light scattering model to make a preliminary depth prediction of the captured RGB image, and then perform local atmospheric light simulation based on the initial depth prediction result, redefine the local atmospheric light scattering model, and restore the global depth information and local depth information of the RGB image by modeling the global and local atmospheric light scattering, and represent them with a global depth map and a local depth map, and then fuse the two, and finally optimize the edges of the fused depth map, and output the final target depth map, that is, an image containing the depth information prediction result. Compared with the prior art, the depth detection result of the present invention has stable performance and low cost, and while ensuring high-quality output of the depth image, it can also greatly improve the operating rate, so it has a more universal application scenario.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a method for predicting depth information of a captured image based on atmospheric light scattering. Background Art

[0002] Although image depth information processing tasks can be accomplished through dedicated hardware devices that can perceive depth information (such as laser rangefinders and Kinect cameras), these dedicated hardware devices are usually expensive, consume a lot of power, or have very limited usage scenarios (for example, they can only be used indoors). Therefore, most images are still monocular images taken by ordinary cameras. However, the acquired images can only record the color information of the shooting scene, and it is difficult to detect the depth information of the image, making it impossible to accurately and efficiently predict the scene depth to obtain high-quality image processing results. Summary of the invention

[0003] In view of the above, the present invention aims to provide a method for predicting depth information of captured images based on atmospheric light scattering, thereby solving the problem that it is difficult to detect depth information from a monocular image.

[0004] The technical solution adopted by the present invention is as follows:

[0005] The present invention provides a method for predicting depth information of a captured image based on atmospheric light scattering, which includes:

[0006] Use the global atmospheric light scattering model to make a preliminary depth prediction on the captured image and obtain a global depth map;

[0007] Performing local atmospheric light simulation on the global depth map to obtain a local depth map;

[0008] Fusing the global depth map and the local depth map to obtain a fused depth map;

[0009] After edge optimization is performed on the fused depth map, a target depth map including a depth information prediction result is output.

[0010] In at least one possible implementation manner, the preliminary depth prediction includes: restoring depth information of the shooting scene according to a medium perspective view and atmospheric light information of the shooting scene.

[0011] In at least one possible implementation manner, the medium perspective view is acquired based on a dark channel prior principle.

[0012] In at least one possible implementation, the local atmospheric light simulation includes:

[0013] Performing scene segmentation on the global depth map based on the number of pixels;

[0014] Based on the scene segmentation results, local atmospheric light estimation is performed in combination with the bright channel principle.

[0015] In at least one possible implementation, the local atmospheric light estimation based on the bright channel principle includes: calculating the brightness of the corresponding pixel of each scene in the bright channel, and taking the average of the top n brightness values ​​as the local atmospheric light estimation of a single scene.

[0016] In at least one possible implementation manner, the scene segmentation includes: dividing the global depth map into several layers.

[0017] In at least one possible implementation manner, the edge optimization of the fused depth map includes: transferring the structure of the guidance image to the fused depth map through a structure-preserving image guidance filter, and maintaining and smoothing the edge of the image.

[0018] In at least one possible implementation manner, the structure preserving image guided filter includes a first filter and a second filter, the first filter is used to transmit the structure of the input image, and the second filter is used to smooth the transmitted image.

[0019] The main design concept of the present invention is to use the global atmospheric light scattering model to make a preliminary depth prediction of the captured RGB image, and then use the initial depth prediction result to simulate the local atmospheric light, redefine the local atmospheric light scattering model, and restore the global depth information and local depth information of the RGB image by modeling the global and local atmospheric light scattering, and represent them with a global depth map and a local depth map, and then fuse the two, and finally optimize the edges of the fused depth map, and output the final target depth map, that is, the image containing the depth information prediction result. Compared with the prior art, the depth detection result of the present invention has stable performance and low cost, and while ensuring the high-quality output of the depth image, it can also greatly improve the operating rate, so it has a more universal application scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a method for predicting depth information of a captured image based on atmospheric light scattering provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0023] The present invention proposes an embodiment of a method for predicting depth information of a captured image based on atmospheric light scattering. Specifically, Figure 1 shown, including:

[0024] Step S1: Use a global atmospheric light scattering model to perform preliminary depth prediction on the captured image to obtain a global depth map;

[0025] In actual operation, the global atmospheric light model can be used for initial depth prediction. By modeling the atmospheric light scattering model, the depth information of the scene captured in the monocular image (which can be understood as an ordinary RGB image) can be preliminarily obtained, which can be understood as obtaining the initial global depth map. The global atmospheric light scattering model mentioned here can be defined as follows:

[0026] I(x,y)=A·ρ(x,y)·t(x,y)+A(1-t(x,y)) (1)

[0027] In the above formula, I, ρ represent the scene image and scene reflectivity captured by the camera, respectively, A represents the atmospheric light value, t is the medium perspective, which represents the ability of reflected light to penetrate the medium, A·ρ(x,y) constitutes the direct attenuation term of atmospheric light, and A·(1-t(x,y)) constitutes the ambient light model. Specifically, according to the global atmospheric light scattering model, the total light energy captured can be divided into two parts: the radiated light of the scene and the ambient light. By obtaining the medium perspective of the scene, the depth information of different positions in the scene can be calculated. Among them, the medium perspective can be expressed as:

[0028] t(x,y)=e -βd(x,y) (2)

[0029] In the above formula, β represents the scattering coefficient, and d(x,y) is the scene depth. It should be noted that one of the concepts of the present invention is to restore the depth information of the shooting scene based on the medium perspective view and atmospheric light information of the shooting scene, and those skilled in the art can understand that based on the dark channel prior principle, in a haze-free image, the intensity value of at least one color channel in most pixel areas is equal to or close to 0, and the image dark channel can be expressed as the following formula:

[0030] I dark (x,y)=J dark (x,y)t(x,y)+A(1-t(x,y)) (3)

[0031] In the above formula, I dark and J dark are the dark channel of the image obtained by the camera and the dark channel of the ideal fog-free scene, respectively. The medium perspective view t(x, y) can be obtained by calculating the following formula:

[0032]

[0033] Step S2: performing local atmospheric light simulation on the global depth map to obtain a local depth map.

[0034] In actual operation, the initial global depth map obtained by the global atmospheric light scattering model is segmented based on the number of pixels. This is because pixels in the same shooting scene may have similar depths. In order to avoid pixel aggregation in certain areas after segmentation, a scene depth layer segmentation method based on the number of pixels is proposed. Furthermore, in the process of scene segmentation, the global depth map generated by the global atmospheric light scattering model can be divided into several layers to maintain the edge shape of the original scene. The specific scene segmentation strategy can be expressed as the following formula:

[0035]

[0036] In the above formula, i indicates that the pixel belongs to the i-th scene. According to the above formula, the global depth map generated by the global atmospheric light model is divided into different depth layers.

[0037] Next, according to the above scene segmentation strategy, the local atmospheric light scattering model is redefined for local depth prediction, that is, formula (1) is redefined and expressed as follows:

[0038] I(x,y)=L(i)·ρ(x,y)T(i)+L(i)·(1-T(i))(x,y) (6)

[0039] In the above formula, I, L(i) and T(i) represent the atmospheric light value and medium perspective of the i-th scene, respectively. The redefined atmospheric light scattering model can effectively avoid the atmospheric light scattering model error caused by uneven radiation of different scenes.

[0040] Next, a bright channel is introduced into the local atmospheric light scattering model after scene segmentation to obtain the local atmospheric light. After scene segmentation, the local atmospheric light may be too dark. If the traditional global atmospheric light acquisition method is used to obtain the local atmospheric light, its value may be lower than the actual value due to the existence of local chromatic aberration. This is inconsistent with the atmospheric light of the real scene. Therefore, the present invention considers introducing a bright channel into the local atmospheric light scattering model to obtain the local atmospheric light. Based on the bright channel prior principle: each region has one of the three RGB color channels with a channel value close to or equal to 1. Based on this premise, the bright channel map of the image can be defined as:

[0041]

[0042] Based on the results of scene segmentation processing, the bright channel principle is used to perform a new local atmospheric light estimation. By estimating the local atmospheric light based on the bright channel information of the original input captured image, the local atmospheric light error caused by chromatic aberration can be effectively reduced. In actual operation, the first 10% of the corresponding pixel brightness in the bright channel of each scene can be calculated, and then the average value can be taken as the atmospheric light value of a single scene.

[0043] Step S3: Fusing the global depth map and the local depth map to obtain a fused depth map.

[0044] It can be understood that the idea of ​​this step is to fuse the global depth map and the local depth map restored after modeling the global atmospheric light scattering model and the local atmospheric light scattering model respectively. Specifically, it can be based on the Bayesian cross model. The Bayesian formula is used to perform depth fusion through the posterior probability. The formula is as follows:

[0045]

[0046] In the above formula, p(F) is the prior probability and H(z) is the feature vector of pixel z.

[0047] The Bayesian theory is used to fuse the global and local prediction results (global depth map and local depth map) to obtain a fused depth map. For details, please refer to the following. Given the binary images of the above two depth maps, D1 and D2, the likelihood probability is calculated as follows:

[0048]

[0049] In the above formula, N F Represents the number of foreground pixels, N B is the number of background pixels, is the number of pixels whose depth feature falls into the foreground bin: b(D(z)), and the depth distribution histogram of the background is also given by N B and N bB(D(z)) represents.

[0050] The calculation method of the aforementioned posterior probability can be referred to as follows:

[0051]

[0052] In the above formula, p' represents p(D j (z)|B i ).

[0053] Similarly, D can be calculated j As the priori posterior probability. Specifically, the fusion depth is calculated using two posterior probabilities through the Bayesian fusion model:

[0054] D B (D1(z),D2(z))=p(F1|D2(z))+p(F2|D1(z)) (11)

[0055] Step S4: After edge optimization is performed on the fused depth map, a target depth map is output.

[0056] In actual operation, the structure of the guided image can be transferred to the fused depth map through a structure-preserving image guided filter (edge-preserving filter) to maintain the edge of the image and smooth it. The structure-preserving image guided filter includes a first filter and a second filter, the first filter is used to transmit the structure of the input image, and the second filter is used to smooth the transmitted image. In the structure-preserving image guided filter, the objective function of the first filter can be defined as:

[0057]

[0058] In the above formula, i represents the pixel index, λ is a parameter used to control the trade-off between the two terms, and V = (V x ,V y ) is a guidance vector field used to guide the calculation:

[0059] V x (h,w)=I(h,w+1)-I(h,w),V y (h,w)=I(h+1,w)-I(h,w) (13)

[0060] According to the above formula, the following linear equation can be solved to obtain the output edge optimization result O:

[0061]

[0062] In the above formula, D x and D y represents the discrete differential operator, and A represents the identity matrix.

[0063] In summary, the main design concept of the present invention is to use the global atmospheric light scattering model to make a preliminary depth prediction of the captured RGB image, and then use the initial depth prediction result to simulate the local atmospheric light, redefine the local atmospheric light scattering model, and restore the global depth information and local depth information of the RGB image by modeling the global and local atmospheric light scattering, and represent them with a global depth map and a local depth map, and then fuse the two, and finally optimize the edges of the fused depth map, and output the final target depth map, that is, the image containing the depth information prediction result. Compared with the prior art, the depth detection result of the present invention has stable performance and low cost, and while ensuring high-quality output of the depth image, it can also greatly improve the operating rate, so it has a more universal application scenario.

[0064] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c may represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c may be single or multiple.

[0065] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the present invention is not limited to the scope of implementation shown in the drawings, and all changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the specification and drawings, should be within the protection scope of the present invention.

Claims

1. A method for predicting depth information of captured images based on atmospheric light scattering, characterized in that: include: Use the global atmospheric light scattering model to make a preliminary depth prediction on the captured image and obtain a global depth map; Performing local atmospheric light simulation on the global depth map to obtain a local depth map, wherein the local atmospheric light simulation comprises: performing scene segmentation on the global depth map based on the number of pixels; and performing local atmospheric light estimation based on the scene segmentation result and in combination with the bright channel principle; The global depth map and the local depth map are fused to obtain a fused depth map, specifically comprising: using Bayesian theory to fuse the global depth map and the local depth map to obtain a fused depth map; After edge optimization is performed on the fused depth map, a target depth map including a depth information prediction result is output.

2. The method for predicting depth information of captured images based on atmospheric light scattering according to claim 1, characterized in that: The preliminary depth prediction includes: restoring the depth information of the shooting scene according to the medium perspective view and atmospheric light information of the shooting scene.

3. The method for predicting depth information of captured images based on atmospheric light scattering according to claim 2, characterized in that: The medium perspective image is obtained based on the dark channel prior principle.

4. The method for predicting depth information of captured images based on atmospheric light scattering according to claim 1, characterized in that: The local atmospheric light estimation based on the bright channel principle includes: calculating the brightness of the corresponding pixel of each scene in the bright channel, and taking the average of the top n brightness values ​​as the local atmospheric light estimation of a single scene.

5. The method for predicting depth information of captured images based on atmospheric light scattering according to claim 1, characterized in that: The scene segmentation includes dividing the global depth map into several layers.

6. The method for predicting depth information of captured images based on atmospheric light scattering according to any one of claims 1 to 5, characterized in that: The edge optimization of the fused depth map includes: transferring the structure of the guide image to the fused depth map through a structure-preserving image guide filter, and maintaining and smoothing the edge of the image.

7. The method for predicting depth information of captured images based on atmospheric light scattering according to claim 6, characterized in that: The structure preserving image guided filter comprises a first filter and a second filter, wherein the first filter is used to transmit the structure of an input image, and the second filter is used to smooth the transmitted image.

Images