A depth information estimation method based on all-focus image cost aggregation

Abstract

This invention discloses a method for estimating depth information based on cost aggregation of all-focus images. The method includes the following steps: first, capturing an image sequence of different focal lengths of the scene to be estimated and creating a disc-like focus metric function; then, convolving the image sequence with the disc-like focus metric function to obtain the image sequence's focus level, and then calculating a guide image based on the image sequence's focus level; second, determining the focus level of the cost-aggregated image sequence based on the guide image and the focus level; and finally, mapping the focus level after cost aggregation to a depth result and an all-focus image, and performing guided filtering to obtain the final depth estimation result of the scene. This method can effectively improve the accuracy of scene depth estimation in consumer electronic devices.

Description

Technical Field

[0001] The present invention belongs to the field of consumer electronic equipment applications, and in particular relates to a depth information estimation method based on all-focus image cost aggregation. Background Art

[0002] With the significant increase in computing power of consumer electronics, many complex applications that previously required server-side execution can now be run on consumer devices. This includes virtual reality technologies like AR and VR. The widespread adoption of these technologies depends crucially on the accuracy of scene depth estimation. Therefore, rapidly calculating scene depth information through mobile device imaging is a key challenge facing the current field of depth estimation. Currently, there are many methods for acquiring scene depth information. The mainstream approach uses active sensing devices to project a preset pattern onto the scene, then collects deformed scene images to recover the scene's depth information. However, these methods often require expensive or specialized equipment, and the accuracy of depth information depends on the precision of the device itself. Furthermore, these methods are susceptible to lighting effects, which can degrade depth estimation performance. Another depth information acquisition method is stereo vision. This method utilizes multiple cameras to establish a multi-view stereo vision system. These cameras capture images of the same scene at different locations, then estimate the scene's depth information by finding pairs of image pixels with a certain baseline. While there is currently a large body of work on multi-view stereo vision, its complex matching process can lead to poor generalization. Furthermore, the introduction of the matching process increases the computational overhead of depth estimation, which is not conducive to time-sensitive applications. When it comes to calculating depth information for consumer electronic devices such as mobile phone cameras, obtaining depth information by calculating the focal position in the image is more convenient and efficient than the above two methods. This method only requires adjusting the focal length between the camera and the scene to obtain the scene's depth information based on the focal point information in the image.

[0003] Generally, a focus metric function can evaluate the focus level of an image. The sparse initial depth information of the image can be obtained through the focus metric function. How to obtain more accurate depth estimation from the sparse initial depth information is an urgent problem that needs to be solved for estimating scene depth information of current consumer electronic devices. This patent proposes a depth information estimation method based on full-focus image cost aggregation. This method obtains the initial depth result of the scene through a typical focus evaluation operator, and then uses a cost aggregation method based on the full-focus image to obtain a more accurate depth information estimation of the scene. Summary of the Invention

[0004] In order to overcome the problems existing in the above-mentioned technologies, the purpose of the present invention is to provide a depth information estimation method based on all-focus image cost aggregation, which can effectively improve the accuracy of scene depth estimation in consumer electronic devices.

[0005] The technical solution adopted by the present invention is: a depth information estimation method based on all-focus image cost aggregation, comprising the following steps:

[0006] Step 1: Adjust the focal length of the mobile phone camera and collect image sequences of different focal lengths of the scene to be estimated. Where p represents the number of images, and its value range is 1≤p≤P, P is the total number of images, i represents the coordinate position of the image sequence, and its range is 1≤i≤M×M, M is the width and height of a single image in the image sequence;

[0007] Step 2: Create the dish-type focusing metric function FM according to formula (1) D , where x, y are pixel positions in the image, R1, R2, R3 are FM D The three radius parameters are: π is the circumference of a circle;

[0008]

[0009] Step 3: Convert the image sequence in step 1 and the dish focus metric function FM in step 2 D According to formula (2), the convolution operation is performed to obtain the focus level of the image sequence

[0010]

[0011] Where * represents the convolution operator, C p (i) represents the focus level at position i in the p-th image;

[0012] Step 4: The focus level C of the image sequence obtained in step 3 is p (i), 1≤i≤M×M According to formula (3), the initial depth result D is obtained Init (i), 1≤i≤M×M;

[0013]

[0014] in Represents a function for solving the focus level index p of the image sequence, and then the initial depth result D Init (i), 1≤i≤M×M is mapped to the guidance image I according to formula (4) Guid ;

[0015]

[0016] Step 5: Focus the image sequence obtained in step 3 at level C p (i), 1≤i≤M×M and the guidance image I obtained in step 4 Guid According to formula (5), the focus level C′ after cost aggregation is obtained p (i), 1≤i≤M×M;

[0017]

[0018] in W x,y (I Guid ) is the guide image I Guid The weight at the pixel x,y position, μ k and σ k Respectively represent the pixel k as the center and the window ω k The mean and variance of |ω| is the window ω k The number of pixels, ε is the smoothing factor;

[0019] Step 6: Aggregate the focus level C′ obtained in step 5 p (i) According to formula (6), the depth result D′(i) after cost aggregation is obtained, 1≤i≤M×M;

[0020]

[0021] Step 7: Based on the depth result D′(i), 1≤i≤M×M after cost aggregation obtained in step 6, the all-focus image AIF(i), 1≤i≤M×M is obtained according to formula (7);

[0022] AIF(i)=I D′(i) (i),1≤i≤M×M (7)

[0023] Step 8: Based on the cost-aggregated depth result D′(i), 1≤i≤M×M obtained in step 6 and the all-focus image AIF(i), 1≤i≤M×M obtained in step 7, the final depth estimation result D(i), 1≤i≤M×M is obtained according to formula (8);

[0024] D(i)=GF(D′(i),AIF(i)),1≤i≤M×M (8)

[0025] where GF(·) represents the guided filter function.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) The all-focus image-guided cost aggregation method proposed in this invention for the scene depth estimation problem in consumer electronic devices can correctly guide the sparse initial depth estimation results and effectively improve the accuracy of scene depth estimation;

[0028] (2) The depth information estimation method proposed in the present invention has the advantage of parallel computing. Its high computational efficiency and low system resource occupancy rate are conducive to its extension to other similar scenarios and fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a depth information estimation method based on all-focus image cost aggregation;

[0030] Figure 2 It is a schematic diagram of the framework of a depth information estimation method based on all-focus image cost aggregation. DETAILED DESCRIPTION

[0031] like Figure 1 、 Figure 2 As shown, a depth information estimation method based on all-focus image cost aggregation includes the following steps:

[0032] Step 1: Adjust the focal length of the mobile phone camera and collect image sequences of different focal lengths of the scene to be estimated. Where p represents the number of images, and its value range is 1≤p≤P, P is the total number of images, i represents the coordinate position of the image sequence, and its range is 1≤i≤M×M, M is the width and height of a single image in the image sequence;

[0033] Step 2: Create the dish-type focusing metric function FM according to formula (1) D , where x, y are pixel positions in the image, R1, R2, R3 are FM D The three radius parameters are: π is the circumference of a circle;

[0034]

[0035] Step 3: Convert the image sequence in step 1 and the dish focus metric function FM in step 2 D According to formula (2), the convolution operation is performed to obtain the focus level of the image sequence

[0036]

[0037] Where * represents the convolution operator, C p (i) represents the focus level at position i in the p-th image;

[0038] Step 4: The focus level C of the image sequence obtained in step 3 is p(i), 1≤i≤M×M According to formula (3), the initial depth result D is obtained Init (i), 1≤i≤M×M;

[0039]

[0040] in Represents a function for solving the focus level index p of the image sequence, and then the initial depth result D Init (i), 1≤i≤M×M is mapped to the guidance image I according to formula (4) Guid ;

[0041]

[0042] Step 5: Focus the image sequence obtained in step 3 at level C p (i), 1≤i≤M×M and the guidance image I obtained in step 4 Guid According to formula (5), the focus level C′ after cost aggregation is obtained p (i), 1≤i≤M×M;

[0043]

[0044] in W x,y (I Guid ) is the guide image I Guid The weight at the pixel x,y position, μ k and σ k Respectively represent the pixel k as the center and the window ω k The mean and variance of |ω| is the window ω k The number of pixels, ε is the smoothing factor;

[0045] Step 6: Aggregate the focus level C′ obtained in step 5 p (i) According to formula (6), the depth result D′(i) after cost aggregation is obtained, 1≤i≤M×M;

[0046]

[0047] Step 7: Based on the depth result D′(i), 1≤i≤M×M after cost aggregation obtained in step 6, the all-focus image AIF(i), 1≤i≤M×M is obtained according to formula (7);

[0048] AIF(i)=I D′(i) (i),1≤i≤M×M (7)

[0049] Step 8: Based on the cost-aggregated depth result D′(i), 1≤i≤M×M obtained in step 6 and the all-focus image AIF(i), 1≤i≤M×M obtained in step 7, the final depth estimation result D(i), 1≤i≤M×M is obtained according to formula (8);

[0050] D(i)=GF(D′(i),AIF(i)),1≤i≤M×M (8)

[0051] where GF(·) represents the guided filter function.

Claims

1. A depth information estimation method based on all-focus image cost aggregation, comprising the following steps: Step 1: Adjust the focal length of the mobile phone camera and collect image sequences of different focal lengths of the scene to be estimated. Where p represents the number of images, and its value range is 1≤p≤P, P is the total number of images, i represents the coordinate position of the image sequence, and its range is 1≤i≤M×M, M is the width and height of a single image in the image sequence; Step 2: Create the dish-type focusing metric function FM according to formula (1) D , where x, y are pixel positions in the image, R1, R2, R3 are FM D The three radius parameters are: π is the circumference of a circle; Step 3: Convert the image sequence in step 1 and the dish focus metric function FM in step 2 D According to formula (2), the convolution operation is performed to obtain the focus level of the image sequence Where * represents the convolution operator, C p (i) represents the focus level at position i in the p-th image; Step 4: The focus level C of the image sequence obtained in step 3 is p (i), 1≤i≤M×M According to formula (3), the initial depth result D is obtained Init (i), 1≤i≤M×M; in Represents a function for solving the focus level index p of the image sequence, and then the initial depth result D Init (i), 1≤i≤M×M is mapped to the guidance image I according to formula (4) Guid ; Step 5: Focus the image sequence obtained in step 3 at level C p (i), 1≤i≤M×M and the guidance image I obtained in step 4 Guid According to formula (5), the focus level C′ after cost aggregation is obtained p (i), 1≤i≤M×M; in W x,y (I Guid ) is the guide image I Guid The weight at the pixel x,y position, μ k and σ k Respectively represent the pixel k as the center and the window ω k The mean and variance of |ω| is the window ω k The number of pixels, ε is the smoothing factor; Step 6: Aggregate the focus level C′ obtained in step 5 p (i) According to formula (6), the depth result D′(i) after cost aggregation is obtained, 1≤i≤M×M; Step 7: Based on the depth result D′(i), 1≤i≤M×M after cost aggregation obtained in step 6, the all-focus image AIF(i), 1≤i≤M×M is obtained according to formula (7); AIF(i)=I D′(i) (i),1≤i≤M×M (7) Step 8: Based on the cost-aggregated depth result D′(i), 1≤i≤M×M obtained in step 6 and the all-focus image AIF(i), 1≤i≤M×M obtained in step 7, the final depth estimation result D(i), 1≤i≤M×M is obtained according to formula (8); D(i)=GF(D′(i),AIF(i)),1≤i≤M×M (8) where GF(·) represents the guided filter function.

Images