A fast light field depth estimation method based on multi-parallax scale

Abstract

The application discloses a kind of fast light field depth estimation methods based on multiple parallax scales.The specific steps are as follows: (1) constructing database;(2) input light field image into neural network, neural network extracts feature vector, interacts with edge enhancement module to enhance edge characteristics, and constructs matching cost of different parallax scales according to the obtained feature vector, carries out matching cost regularization through cost aggregation module, and finally obtains light field parallax map through parallax regression;(3) construct the loss function of neural network, the loss function includes absolute value loss function and weighted binary classification cross-entropy loss function;(4) according to the light field camera parameter, the camera focal length and baseline length corresponding to adjacent light field image are obtained, and then according to the light field parallax map, it is converted into light field depth map;(5) input the light field image of test set into the trained neural network, and obtain the light field depth map.The method of the application can directly obtain the accurate depth map of the light field image.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning technology, and in particular to a fast depth estimation method for light field images based on multiple disparity scales. Background Technology

[0002] Traditional cameras capture two-dimensional images, failing to reflect the three-dimensional spatial structure of a scene. Light field imaging, however, possesses a unique imaging principle. A light field image records the direction and position of light propagation in space, simultaneously containing both spatial and angular information about the light rays, describing a four-dimensional light field. Light field cameras primarily acquire the light information of the scene through microlens arrays, thereby reconstructing the scene's three-dimensional information. Image depth refers to the distance from the photographed object to the camera. Image depth estimation is a popular research area in computer vision. If the three-dimensional information, i.e., depth, of a scene can be recovered from a captured two-dimensional image, it will benefit many computer vision applications. Therefore, depth estimation from light field images has significant theoretical research and practical application value.

[0003] Several light field depth estimation methods have been proposed and achieved good results, but some problems also exist. These light field image depth estimation methods are mainly divided into two categories: (1) obtaining depth information using epipolar plane images (EPI); and (2) calculating depth based on stereo matching methods. The advantage of using EPI methods is that the theory is simple and the optimization speed using neural networks is fast, but it is not robust to noise, and for light field images with large baselines, the estimated depth information has low accuracy and large error. As for stereo matching methods, when using neural network optimization, a 4D matching cost volume is often constructed and processed using 3D convolution. Although a very accurate depth image can be obtained, it requires a high computational cost, resulting in long running time and high GPU memory usage. Therefore, it is urgent to solve the problem of providing a depth estimation method with high robustness, low computational cost, and high accuracy for different types of light field images. Summary of the Invention

[0004] This invention provides a fast light field image depth estimation method. Taking into account the special physical structure of the light field, this invention designs a neural network model for light field image depth estimation. While ensuring the model is lightweight, it can perform accurate depth estimation of light field images, achieving fast and accurate light field depth estimation.

[0005] The technical solution adopted in this invention is as follows:

[0006] A fast optical field depth estimation method based on multiple disparity scales includes the following steps:

[0007] Step 1: Construct a database, including a training set and a test set; the database includes virtual synthetic and real-world light field images, and simultaneously has two types of light field images: small baseline and large baseline.

[0008] Step 2: Input the light field images in the training set into the neural network. The neural network includes a feature extraction module, a matching cost construction module, a cost aggregation module, a disparity regression module, and an edge enhancement module. The feature extraction module, matching cost construction module, cost aggregation module, and disparity regression module constitute the main network. The neural network extracts feature vectors through the feature extraction module, interacts with the edge enhancement module to enhance edge characteristics, constructs matching costs at different disparity scales based on the obtained feature vectors, performs matching cost regularization through the cost aggregation module, and finally obtains the light field disparity map through the disparity regression module.

[0009] Step 3: Construct the loss function of the neural network, which includes an absolute value loss function and a weighted binary cross-entropy loss function; train the neural network.

[0010] Step 4: Based on the light field camera parameters, obtain the camera focal length and baseline length corresponding to adjacent light field images, and then convert the light field disparity map obtained in Step 2 into a light field depth map.

[0011] Step 5: In the testing phase, the light field images of the test set are input into the trained neural network. Through the steps of feature extraction, edge enhancement, matching cost construction, cost aggregation and disparity regression, a light field disparity map is obtained, and finally a light field depth map is obtained through transformation.

[0012] Furthermore, in step 1, when constructing the database, it is also necessary to construct a binary disparity edge map extracted by the Sobel operator.

[0013] Furthermore, in step 2, the feature extraction module uses basic residual blocks to extract reliable features, sets the stride to 2 for downsampling during depthwise convolution, uses spatial pyramid pooling to fuse features at different scales, and finally enhances the attention of important features through a bottleneck attention enhancement module to obtain an effective feature representation for constructing the matching cost.

[0014] Furthermore, in step 2, when the matching cost construction module constructs matching costs at different disparity scales, it uses different feature vectors to calculate the correlation, thereby reducing the original dimension of the matching cost and enabling the cost aggregation network to use 2D convolution processing.

[0015] Furthermore, in step 2, the cost aggregation module adopts a multi-disparity scale aggregation method. It utilizes matching costs of different scales obtained from the special physical structure of the light field, and performs upsampling-fusion operations layer by layer from coarse scale to fine scale to aggregate matching costs containing information of different scales into one, and performs regularization through 4-layer bottleneck residual blocks to improve the accuracy of the obtained light field disparity map.

[0016] Furthermore, in step 2, the cost aggregation module also includes an edge feature fusion module. The edge feature fusion module uses an edge attention guidance mechanism to guide the matching of each pixel in the cost based on the intermediate edge features provided by the edge enhancement module, and performs feature enhancement of the edge region.

[0017] Furthermore, in step 2, the edge enhancement module shares features with the main network and provides edge features to the main network for use, thereby improving the accuracy and effectiveness of the estimated light field disparity map in the edge and occluded areas.

[0018] Furthermore, in step 3, a three-step training strategy is adopted when training the neural network. First, the main network is trained to obtain an initial light field disparity map. Then, the edge enhancement module is trained so that it can output a light field disparity edge map. Finally, the two parts are trained together to obtain the final optimized light field disparity map.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] (1) The image processing method of the present invention can directly obtain the accurate depth map of the light field image, thereby recovering the three-dimensional information of the scene and increasing the diversity of scene three-dimensional reconstruction.

[0021] (2) When constructing the matching cost for light field image matching, the dimension of the matching cost is reduced by the operation of feature correlation, and the 3D convolution in the cost aggregation network is replaced with 2D, which greatly reduces the amount of computation, making the present invention lightweight and enabling fast light field depth estimation.

[0022] (3) The edge enhancement module and multi-disparity scale cost aggregation module proposed in this invention improve the accuracy of the light field disparity map in the edge and occluded areas as well as the accuracy of the overall estimation results, and ensure the high accuracy and low error of the final light field depth map.

[0023] (4) Training on light field datasets with different baseline sizes increases the universality of this invention for different types of light field images. During the test, this method achieves accurate light field depth estimation on both virtual synthesis and real-world light field images while ensuring speed and lightweightness, further verifying the generalization and effectiveness of the method of this invention. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the method of the present invention;

[0025] Figure 2 This is a diagram of the overall network structure of the method of this invention;

[0026] Figure 3 This is a structural diagram of the edge enhancement module in the network model of this invention;

[0027] Figure 4 This is a structural diagram of the multi-disparity scale cost aggregation module in the network model of this invention;

[0028] Figure 5 This is an experimental result of the method of the present invention for disparity estimation on a small baseline virtual synthetic dataset 4D Light Field;

[0029] Figure 6 This is an experimental result of the disparity estimation method of the present invention on the large baseline virtual synthetic dataset Sparse Light Field;

[0030] Figure 7 This is a diagram showing the experimental results of disparity estimation using the method of this invention on a real-world dataset. Detailed Implementation

[0031] This embodiment provides a fast optical field depth estimation method based on multiple disparity scales, such as... Figure 1 As shown, it includes the following steps:

[0032] (1) Construct a database, including a training set and a test set. The database contains virtual synthetic and real-world light field images, with two types of light field images: small baseline and large baseline. It also has an edge map of the real light field disparity map.

[0033] (2) Input the light field images in the training set into the neural network model, extract the feature vector through the feature extraction module, enhance the edge characteristics of the feature vector through the edge enhancement module, construct the matching cost of different disparity scales based on the obtained feature vector, perform matching cost regularization through the cost aggregation module, and finally obtain the light field disparity map through the disparity regression module.

[0034] (3) Construct the loss function of the neural network model, which includes the absolute value loss function and the weighted binary cross-entropy loss function; wherein, the absolute value loss function is constructed to reduce the absolute error of the output disparity map, and the weighted binary cross-entropy loss function is constructed to make the edge enhancement module output the disparity edge map; then the neural network is trained.

[0035] (4) Using the camera focal length and baseline length corresponding to adjacent light field images, the estimated light field disparity map is converted into a light field depth map.

[0036] (5) During the testing phase, light field images of various types of test sets are input into the trained neural network, and light field disparity maps are output. Finally, light field depth maps are obtained through transformation.

[0037] 1. A detailed description of the database construction involved in step (1):

[0038] This embodiment primarily utilizes three datasets: the small baseline virtual synthesis dataset 4D Light Field, the large baseline virtual synthesis dataset Sparse Light Field, and a real-world dataset. Simultaneously, a light field disparity edge map is constructed according to the needs of the edge enhancement module in the neural network model. The dataset contains a real light field disparity map. This method uses the Sobel operator to process the real light field disparity map to solve for the initial disparity edge map. Different gray values ​​in the edge map represent the gradient of the object edge. Furthermore, a manually set threshold method is used: for each light field image, the portion of the initial disparity edge map with gray values ​​greater than or equal to the threshold is set to 1, while the portion with gray values ​​less than the threshold is set to 0, resulting in a binary disparity edge map of the light field image.

[0039] 2. The overall network structure of this embodiment is as follows: Figure 2 As shown below, the feature extraction, edge enhancement, matching cost construction, cost aggregation and disparity regression modules involved in step (2) will be described in detail below. The feature extraction module, matching cost construction module, cost aggregation module and disparity regression module constitute the main network.

[0040] 2.1 Feature extraction module, used to extract feature vectors from the input light field image to obtain effective feature representation.

[0041] Table 1. Detailed structure of the feature extraction module

[0042]

[0043] This method uses basic residual blocks to extract reliable features and sets a stride of 2 during depthwise convolution for downsampling. A spatial pyramid pooling mechanism is employed to obtain four feature vectors at different scales after downsampling. These vectors are then upsampled to the same scale using bilinear interpolation. The feature vectors output from these different convolutional layers are concatenated and merged, and then fused through convolution. Finally, a Bottleneck Attention Module (BAM) is used to enhance the attention of important features, resulting in an effective feature representation for constructing the matching cost.

[0044] 2.2 Edge enhancement module, such as Figure 3As shown, the input is the light field edge map processed by the Sobel operator, and the output is the light field parallax edge map.

[0045] Table 2. Detailed Structure of the Edge Enhancement Module

[0046]

[0047] This module interacts with the feature extraction module in the main network and uses the intermediate feature vectors from the feature extraction module for feature fusion. The two modules are trained together and influence each other, thereby enhancing the edge characteristics of the feature vectors. At the same time, this module can also provide intermediate edge features to the edge feature fusion module, further enhancing the edge feature characteristics and improving the accuracy of the light field disparity map in edge and occluded areas.

[0048] 2.3 Matching Cost Construction Module: Based on the special physical structure of the light field, this module constructs matching costs at different disparity scales for light field images with different angular resolutions. This method utilizes the effective feature representations obtained after feature extraction and employs correlation operations on feature vectors to construct 3D matching costs.

[0049]

[0050] Where f warp[.,.] This indicates that given a disparity d, the surrounding features F s Warp to the center feature F c The expression `<.,.>` represents a dot product operation (relevance calculation) performed on different features along the feature dimension to measure the similarity between two features. `(h,w)` and `c` represent the spatial and channel dimensions of the extracted features, respectively, and `N` represents the number of channel dimensions. This method yields a 3D matching cost C(d,h,w), which can then be processed using 2D convolution. Compared to 3D convolution methods, this significantly reduces computation time and GPU memory usage.

[0051] 2.4 Multi-parallax scale cost aggregation module, network structure as follows Figure 4 As shown, this is used to aggregate and regularize the constructed pyramid-shaped matching costs. This invention proposes a layer-by-layer multi-disparity scale cost aggregation network structure. From coarse to fine scales, it performs upsampling-fusion operations layer by layer, aggregating matching costs containing information from different scales into a unified whole. Regularization is then performed through four bottleneck residual blocks to improve the accuracy of the overall disparity estimation results. Simultaneously, this module also includes an edge feature fusion module. Utilizing an edge attention guidance mechanism, combined with intermediate edge features provided in the edge enhancement module, it guides each pixel in the matching cost to perform feature enhancement in edge regions, further improving the disparity estimation performance of this method in edge regions.

[0052] The 2.5 disparity regression module performs disparity regression on the aggregated and regularized matching costs, outputting an estimated light field disparity map. This method employs a soft argmin operation for disparity regression, estimating a continuous and accurate disparity map. This operation is performed on the matching cost C under disparity d. d A softmax operation σ(.) is performed to calculate a probability voxel, and its probability within the disparity range (-D) is then evaluated. max To D max Perform a disparity-weighted summation to obtain the final estimated disparity.

[0053]

[0054] 3. A detailed description of the training neural network part involved in step (3):

[0055] To enable the model to perform edge enhancement and disparity estimation simultaneously, this method proposes a three-step training strategy.

[0056] First, without using an edge enhancement module, the neural network model is trained and brought to convergence, enabling the network to estimate an initial coarse light field disparity map. In this step, the method uses an absolute value loss function to supervise the neural network and reduce the error of the output disparity map.

[0057] The second step is to train the edge enhancement module. At this stage, the rest of the neural network is frozen, and the network parameters remain unchanged. An input light field image is used, and the Sobel operator is applied to extract edges. The edge enhancement module outputs the edge map of the light field disparity map. A weighted binary cross-entropy loss function is used to supervise the output, enabling the neural network to accurately estimate the light field disparity edge map. The class balance weight formula for the weighted binary cross-entropy loss function is:

[0058]

[0059] Where α and β represent the weights for negative and positive samples, respectively, |Y + | and | Y - | represent the number of positive and negative samples respectively, while λ controls the weight of positive samples relative to negative samples.

[0060] The third step involves training the entire neural network model together. In addition to the absolute value loss supervision applied to the light field disparity map in the first step, and the weighted binary cross-entropy loss supervision applied to the estimated light field disparity edge map in the second step, absolute value loss supervision is also applied to the edges of the final model's light field disparity map. Specifically, the Sobel operator is used to calculate the edge maps of both the estimated and true disparity maps, and then the absolute value error between the two edge maps is used for absolute value loss supervision to optimize the entire neural network model.

[0061] The neural network model of this invention uses the Adam optimizer, with a learning rate initialized to 0.001. Each training step involves 40,000 iterations, with the first and third steps using a batch size of 8, and the second step using a batch size of 16. During training, the learning rate is divided by 2 every 10,000 iterations, and the parameter λ of the weighted binary cross-entropy loss function is set to 1.1.

[0062] 4. A detailed description of the conversion between the light field parallax map and the depth map involved in step (4):

[0063] A light field image is a four-dimensional dataset that records the direction and position of incident light rays. It can be represented by a 4D coordinate system (u,v,x,y). Here, (u,v) represents the angular resolution of the light field image, similar to how a regular camera captures the same scene from different viewpoints; (x,y) represents the spatial resolution of the light field image, similar to the image resolution of a two-dimensional plane image. Observing the image formed by (x,y) at different angular resolutions (u,v) will result in pixel differences at the viewpoint, which is the light field parallax estimated by this method.

[0064] The light field parallax estimated by this method is for views at adjacent angular resolutions. Based on the camera parameters of the light field camera, the relative positions Δu or Δv of views at adjacent angular resolutions can be obtained, i.e., the baseline B (in mm) of the light field camera. Simultaneously, the distance f (in mm) between the microlens array and the imaging plane in the light field camera can also be obtained, i.e., the physical focal length. This is based on the physical size d of each pixel of the camera sensor. x (Unit: mm / pixel) can be used to obtain the pixel focal length f of the light field camera. x (unit: pixel): f x =f / d x Furthermore, based on the imaging geometry, a conversion formula from the disparity map to the depth map is derived, and the light field depth map of the scene is calculated:

[0065]

[0066] 5. A detailed description of the neural network model testing part involved in step (5):

[0067] During the testing phase, the neural network model is first loaded with pre-trained network parameters. Then, the test light field map is input into the neural network. Through the network module described in step (2), a precise light field disparity map is finally output. This involves obtaining the baseline B and pixel focal length f of the light field camera. x Then, using the conversion formula from disparity map to depth map derived in step (4), the accurate light field depth map of the scene is obtained.

[0068] like Figure 5 As shown, this invention trains and tests a model on a small baseline virtual synthetic dataset, 4D Light Field, and outputs a light field disparity map. From Figure 5 As can be seen, the disparity map calculated by the method of the present invention is very accurate. In particular, the edge enhancement module and the multi-disparity scale cost aggregation module proposed in the present invention improve the overall estimation effect of the light field disparity map, especially some edge regions and small object structures in the map, with very high accuracy, which can enhance the clarity and depth of the obtained stereo image.

[0069] like Figure 6 As shown, this invention trains and tests the model on the large baseline virtual synthetic dataset Sparse Light Field, outputting a light field disparity map. From Figure 6 As can be seen, the method of this invention can also achieve very accurate results for large baseline light field data. The parallax estimation of the edges and occluded areas is very accurate, and the result of light field parallax estimation is very accurate.

[0070] like Figure 7 As shown, this invention is tested on a real-world dataset, outputting a light field disparity map. This real-world dataset contains small baseline light field data; therefore, when loading the network model parameters, a pre-trained model on a 4D light field is used for testing on real-world light field data. Figure 7 As can be seen, the method of this invention can also estimate a very accurate light field disparity map for real-world light field data. Using the estimated real-world image and light field disparity map, the 3D information of the scene can be recovered, and thus, the 3D scene of the real world can be perceived and rendered.

Claims

1. A method for fast light field depth estimation based on multi-parallax scale, characterized in that, The method comprises the following steps: Step 1, constructing a database, including a training set and a test set; the database includes virtual synthetic and real-world light field images, and has both small-baseline and large-baseline light field image types; Step 2, inputting the light field images in the training set into a neural network, the neural network comprising a feature extraction module, a matching cost construction module, a cost aggregation module, a disparity regression module and an edge enhancement module, wherein the feature extraction module, the matching cost construction module, the cost aggregation module and the disparity regression module constitute a main network; the neural network extracts a feature vector through the feature extraction module, interacts with the edge enhancement module to enhance edge characteristics, constructs matching costs of different disparity scales according to the obtained feature vector, performs matching cost regularization through the cost aggregation module, and finally obtains a light field disparity map through the disparity regression module; the edge enhancement module shares features with the main network and delivers edge features to the main network for use, thereby improving the accuracy and effect of the estimated light field disparity map in edge and occlusion areas; when the matching cost construction module constructs matching costs of different disparity scales, different feature vectors are used to perform correlation operations, thereby reducing the original matching cost dimension and enabling the cost aggregation network to use 2D convolution processing; Step 3, constructing a loss function of the neural network, the loss function comprising an absolute value loss function and a weighted binary cross-entropy loss function; training the neural network; Step 4, obtaining the camera focal length and baseline length corresponding to adjacent light field images according to the light field camera parameters, and then converting the light field disparity map obtained in step 2 into a light field depth map; Step 5, in the test phase, inputting the light field images of the test set into the trained neural network, obtaining a light field disparity map through the steps of feature extraction, edge enhancement, matching cost construction, cost aggregation and disparity regression, and finally obtaining a light field depth map through conversion. 2.The method of claim 1, wherein, In step 1, when constructing the database, a binary disparity edge map extracted by a Sobel operator also needs to be constructed. 3.The method of claim 1, wherein, In step 2, the feature extraction module uses a basic residual block to extract reliable features, sets the stride to 2 for downsampling during deep convolution, uses a spatial pyramid pooling mechanism to fuse features of different scales, and finally enhances the attention of important features through a bottleneck attention enhancement module to obtain effective feature representation for constructing matching costs.

4. The method of claim 1, wherein, In step 2, the cost aggregation module uses a multi-disparity scale aggregation method to aggregate matching costs containing different scale information into one through upsampling-fusion operations from coarse scale to fine scale, and improves the accuracy of the obtained light field disparity map through 4-layer bottleneck residual blocks for regularization.

5. The method of claim 4, wherein, In step 2, the cost aggregation module also comprises an edge feature fusion module, which uses an edge attention guidance mechanism to guide each pixel in the matching cost to enhance the features in the edge region according to the intermediate edge features provided by the edge enhancement module.

6. The method of claim 1, wherein, In step 3, a three-step training strategy is adopted when training the neural network, that is, first, the main network is trained to obtain an initial light field disparity map, then the edge enhancement module is trained to enable it to output a light field disparity edge map, and finally, the two parts are jointly trained to obtain the final optimized light field disparity map.

Images