CVP-MVSNet adaptive depth updating method based on histogram segmentation

Abstract

The invention discloses a CVP-MVSNet adaptive depth updating method based on histogram segmentation. The method comprises the following steps: step a, generating an initial depth map of an image based on a CVP-MVSNet network; b, on the basis of the initial depth map, extracting the depth map and constructing a corresponding depth histogram DL; c, the horizontal coordinate of the histogram is set as depth, the vertical coordinate of the histogram is set as frequency or frequency number, the area of the histogram is counted, the horizontal coordinate is traversed, the depth histogram is divided into M-1 subintervals with the same area, the two ends of each subinterval are scaled to dmin to dmax, and each horizontal coordinate division point represents the updated assumed depth; d, iterating the initial depth map in the step a by using the updated assumed depth to generate an updated initial depth map; and e, performing up-sampling by using the updated assumed depth and the updated initial depth map, constructing a local cost body of the CVP-MVSNet network step by step, and performing depth refinement to generate a final high-resolution depth map. The method has the characteristic of high depth estimation accuracy in different scenes.

Description

Technical Field

[0001] The present invention relates to the field of depth estimation technology in three-dimensional reconstruction tasks, and in particular to a CVP-MVSNet adaptive depth update method based on histogram segmentation for three-dimensional reconstruction of objects with uneven depth distribution, especially objects with front and back spatial faults. Background Art

[0002] In actual 3D reconstruction scenes, when the spatial scale of the reconstructed objects is not concentrated, there are front and back spatial faults, or there are multiple discontinuous reconstructed objects in the same scene, the quality of the 3D reconstruction will be affected. Therefore, the 3D reconstruction processing of such scenes is particularly important.

[0003] With the rapid development of computer vision and 3D reconstruction, Multi-view Stereo Network (MVSNet) has become an important tool for depth estimation and 3D reconstruction. As an advanced neural network structure, CVP-MVSNet achieves high-precision depth estimation by fusing multi-view information. However, the existing CVP-MVSNet has limitations in its depth estimation strategy at fixed depth intervals when dealing with different scenes and complex texture areas. Especially in areas with drastic depth changes, the fixed depth interval setting can easily lead to insufficient information expression in depth-dense areas, while in areas with gentle depth changes or open areas, it may cause a waste of computing resources.

[0004] In summary, traditional depth estimation methods often neglect the detailed analysis of local depth distribution characteristics, especially the weak ability of adaptive adjustment of depth values ​​in the global range. Summary of the invention

[0005] The object of the present invention is to provide a CVP-MVSNet adaptive depth update method based on histogram segmentation. The present invention has the characteristics of high accuracy of depth estimation in different scenarios.

[0006] Technical solution: A CVP-MVSNet adaptive depth update method based on histogram segmentation, comprising the following steps:

[0007] Step a. Generate an initial depth map of the image based on the CVP-MVSNet network;

[0008] Step b. Based on the initial depth map, extract the depth map and construct the corresponding depth histogram DL;

[0009] Step c. Let the horizontal axis of the histogram be depth and the vertical axis be frequency or count, count the area of ​​the histogram, traverse the horizontal axis, divide the depth histogram into M-1 sub-intervals of equal area, and scale the two ends of the sub-interval to dmin to dmax. Each horizontal axis division point represents the updated hypothetical depth;

[0010] Step d. Iterate the initial depth map in step a using the updated assumed depth to generate an updated initial depth map;

[0011] Step e. Use the updated hypothesized depth and the updated initial depth map to upsample and gradually construct the local cost volume of the CVP-MVSNet network for depth refinement to generate the final high-resolution depth map.

[0012] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, in step a, the method for generating the initial depth map is as follows:

[0013] Step a1: Construct the image pyramid structure of the CVP-MVSNet network;

[0014] Step a2: Use a feature extraction network to extract features layer by layer on the image pyramid structure;

[0015] Step a3: Project all the image features extracted in step 2 to M uniformly distributed hypothetical depths of the parallel plane in front of the reference view through homography transformation, and then calculate the variance of the same pixel in different views at the same depth as the cost volume;

[0016] Step a4: Convert the cost volume in step a3 into a probability distribution, determine the relative likelihood of each depth hypothesis relative to other hypotheses, and thus generate a probability volume;

[0017] Step a5: Combine the cost volume and probability volume in steps a3 and a4 to extract the best depth estimate for each pixel in the reference view and generate an initial depth map.

[0018] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, in step d, the iterative process is as follows: replace the assumed depth in step a3 with the updated assumed depth obtained in step c, re-execute steps a3 to a5, and generate an updated initial depth map.

[0019] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, the construction method of step a1 is as follows: the input multi-view image is downsampled L times, and the resolution of each layer is 1 / 2 of the previous layer, forming an L+1 layer image pyramid, where the L layer has the lowest resolution and the 0 layer has the highest resolution.

[0020] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, the length and width resolution of all images in the L layer are not less than 70 pixels.

[0021] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, the feature extraction method of step a2 is as follows: the feature extraction network has 9 layers in total, the number of output channels of the first 3 layers is 64, the number of output channels of the middle 3 layers is 32, and the number of output channels of the last 3 layers is 16. The size of each layer is 3×3, the step size is 1, and a Leaky ReLU layer is assigned after each layer.

[0022] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, in step a3, the depth range is from dmin to dmax.

[0023] In the aforementioned CVP-MVSNet adaptive depth update method based on histogram segmentation, in step a4, the probability distribution conversion method is: perform a Softmax operation on each cost body depth layer in step a3 to convert the cost body into a probability distribution.

[0024] Beneficial effects: The present invention focuses on how to utilize the local characteristics of the image, especially by means of the effective means of histogram segmentation, to improve and optimize the depth estimation mechanism of CVP-MVSNet; the histogram segmentation of the present invention can reflect the grayscale level of image pixels or the distribution of feature space, and can divide the depth distribution into multiple representative intervals according to the statistical characteristics of the image data; this method effectively improves the refinement and adaptability of depth estimation.

[0025] The CVP-MVSNet adaptive depth update method based on histogram segmentation proposed in the present invention is universal in deep learning algorithms using depth assumptions, overcomes the problems of insufficient information in depth dense areas and unreasonable allocation of computing power in sparse areas in the depth estimation process of the existing CVP-MVSNet, realizes the refinement and adaptive control of depth distribution, and improves the accuracy and computational efficiency of depth estimation.

[0026] The present invention improves the CVP-MVSNet architecture and uses histogram segmentation technology to achieve adaptive updating of depth assumptions. For reconstruction scenarios with front and back faults in depth distribution, it not only improves the reconstruction effect but also effectively saves computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagrams of several typical space fault scenarios.

[0028] Figure 2 For: Image pyramid construction process.

[0029] Figure 3 For: CVP-MVSNet feature extraction process.

[0030] Figure 4For: initial depth hypothesis and homography transformation.

[0031] Figure 5 : Schematic diagram of the improved CVP-MVSNet framework.

[0032] Figure 6 For: Depth map before and after improvement.

[0033] Figure 7 : 3D reconstruction effect before and after improvement.

[0034] Figure 8 Figure 3: Comparison of the three-dimensional reconstruction effects of the CVP-MVSNet method before and after improvement. DETAILED DESCRIPTION

[0035] Example 1. A CVP-MVSNet adaptive depth update method based on histogram segmentation, see Figure 1-Figure 7 ,

[0036] Step 1: This step is the image pyramid structure of the CVP-MVSNet network. The input multi-view image is downsampled L times, and the resolution of each layer is 1 / 2 of the previous layer, forming an L+1 layer image pyramid, where the L layer has the lowest resolution and the 0 layer has the highest resolution. The length and width resolution of all images in the L layer are not less than 70 pixels.

[0037] Step 2: This step is the feature pyramid structure of the CVP-MVSNet network. The feature extraction network is used to extract features layer by layer on the image pyramid. The feature extraction network has 9 layers, the number of output channels of the first 3 layers is 64, the number of output channels of the middle 3 layers is 32, and the number of output channels of the last 3 layers is 16. The size of each layer is 3×3, the step size is 1, and a Leaky ReLU layer is assigned after each layer.

[0038] Step 3: Project all image features in step 2 to M uniformly distributed hypothetical depths of the parallel plane in front of the reference view through homography transformation, with a depth range of dmin to dmax, and then calculate the variance of the same pixel in different views at the same depth as the cost volume.

[0039] Step 4: Perform a Softmax operation on each cost volume depth layer in step 3 to convert the cost volume into a probability distribution and determine the relative likelihood of each deep hypothesis relative to other hypotheses, thereby generating a probability volume.

[0040] Step 5: Combine the cost volume and probability volume in steps 3 and 4 to extract the best depth estimate for each pixel in the reference view and generate an initial depth map.

[0041] Step 6: This step is the core content of the present invention. Based on the depth map generated in step 5, the depth map is extracted and the corresponding depth histogram DL is constructed.

[0042] Step 7: This step is the core content of the present invention. Let the horizontal axis of the histogram be depth and the vertical axis be frequency or number, count the area of ​​the histogram, traverse the horizontal axis, divide the depth histogram into M-1 sub-intervals of equal area, and scale the two ends of the sub-interval to dmin to dmax. Each horizontal axis division point represents the updated hypothetical depth.

[0043] Step 8: This step is the core content of the present invention. The updated assumed depth obtained in step 7 is used to replace the assumed depth in step 3, and steps 3 to 5 are re-executed to generate an updated initial depth map.

[0044] Step 9: This step is the CVP-MVSNet network step-by-step depth refinement, using the updated hypothesized depth and the updated minimum resolution depth map for upsampling and step-by-step construction of the local cost volume for depth refinement until the initial highest resolution depth map is obtained.

[0045] Example 2. A CVP-MVSNet adaptive depth update method based on histogram segmentation, see Figure 1-Figure 7 :

[0046] First, the process of building an image pyramid is as follows Figure 2 As shown, the multi-view images to be processed are downsampled to form a multi-resolution hierarchy, where each resolution of a lower layer is reduced to half of the previous layer, and the image resolution of the bottom layer is ensured to be at least 70×70 pixels.

[0047] Then, the feature pyramid structure of CVP-MVSNet is established to extract features for each layer of the image pyramid. The feature extraction network used is as follows: Figure 3 As shown in the figure, the network contains 9 layers of convolution operations, which are set as follows: the number of output channels of the first three layers is set to 64, the middle three layers is 32, and the last three layers is 16; the size of the convolution kernel of each layer is 3×3, the step size is 1, and a Leaky ReLU activation function is connected after each layer of convolution operation to increase the nonlinear expression ability.

[0048] Afterwards Figure 4 As shown in the figure, all image features extracted from the feature pyramid are mapped to M preset uniformly distributed depth hypothesis levels on the parallel plane in front of the reference view through homography transformation, and the depth range is from the minimum value dmin to the maximum value dmax. Then, the difference between the corresponding pixels of different views under the same depth hypothesis is calculated to construct the cost volume.

[0049] Furthermore, a Softmax function is applied to each generated depth layer cost volume to convert it into a probability distribution, thereby obtaining the likelihood of each depth hypothesis compared to other hypotheses, and then forming a probability volume. Combined with the previous cost volume, the best depth estimate is selected for each pixel in the reference view to generate a preliminary depth map.

[0050] However, this depth map is limited by the fixed depth interval of CVP-MVSNet. Especially in areas with drastic depth changes, the fixed depth interval setting may lead to insufficient information expression in the depth-dense areas, while in areas with gentle depth changes or open areas, it may cause a waste of computing resources. Therefore, the present invention improves the CVP-MVSNet model by adding an initial assumption depth update branch. Figure 5 As shown in the figure and described in steps 6-8, the initial depth map generated by the original network is updated through an assumed depth update branch. Instead of relying solely on the initial depth map for step-by-step refinement, the depth histogram of the initial depth map is used to construct a new adaptive assumed depth. The new assumed depth is then used as the depth slice of steps 3-5 to generate an updated initial depth map. Finally, the step-by-step refinement strategy of CVP-MVSNet is executed to generate the final high-resolution depth map.

[0051] Next, the specific method of the present invention is described in detail in conjunction with the accompanying drawings:

[0052] (1) First, process the initial depth map DL generated in step 5. The depth map contains the depth information corresponding to each pixel, which is the result of preliminary estimation through matching and optimization algorithms. In order to further refine and optimize these estimates, the depth values ​​of all pixels are extracted from the depth map DL, and a depth histogram is constructed based on these values. The horizontal axis represents the depth value, and the vertical axis represents the number or frequency of pixels falling at that depth.

[0053] (2) After constructing the depth histogram, the horizontal axis (i.e., depth axis) of the histogram is traversed, and the M discrete depth values ​​are regarded as continuous depth intervals, and the histogram is divided into M-1 sub-intervals of equal area. This ensures that the number of pixels in each depth interval is relatively balanced, which is beneficial for subsequent depth optimization processing. At the same time, in order to maintain the effective range of the depth value, the depth boundary of the sub-interval needs to be scaled to between the actual minimum depth dmin and the maximum depth dmax. In this process, each segmentation point not only represents a depth dividing line, but also serves as an updated depth hypothesis value, which can better reflect the depth distribution characteristics of the real scene than the original uniformly distributed depth hypothesis.

[0054] (3) Finally, the new depth hypothesis value is used to replace the initial depth hypothesis that was simply set in step 3. This means that in the next round of iterations, the network will re-execute the process from step 3 to step 5 based on the updated depth hypothesis to generate an updated depth map DL.

[0055] There are two benefits to doing this. On the one hand, for areas where the actual depth is dense, the new hypothesized depth generated based on histogram segmentation will be more concentrated in the depth-dense area, so that the updated depth map can reflect more relevant depth information. On the other hand, for areas where the actual depth is sparse, the updated hypothesized depth also becomes sparse accordingly, reducing the waste of computing resources in irrelevant areas. The final generated initial depth map D0 will more accurately reflect the three-dimensional structure of the scene.

[0056] Finally, on the updated initial depth map, the step-by-step refinement strategy of CVP-MVSNet is continued, the low-resolution depth map is upsampled step by step and the local cost volume is constructed until the final high-resolution depth map is obtained. The 3D point cloud model is generated through depth map filtering and fusion.

[0057] Figure 6 The depth map generated by the original CVP-MVSNet network is shown. The depth of the rear building model area is completely chaotic. Figure 7 The depth map generated by the improved CVP-MVSNet network of the present invention is shown, and the depth of the corresponding area is more uniform, indicating that the present invention has improved the depth estimation of such reconstructed scenes.

[0058] Figure 8 The effects of 3D reconstruction using the CVP-MVSNet method before and after the improvement are compared. The left side shows the reconstruction effect of the original CVP-MVSNet network, and a large area of ​​the building model in the middle and rear is missing. The right side shows the reconstruction effect of the improved CVP-MVSNet network of the present invention. The point cloud near the target area is denser, and the building model in the rear can also be well reconstructed, indicating that the present invention can better handle scenes with discontinuous depth.

[0059] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all methods or steps in the process, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A CVP-MVSNet adaptive depth update method based on histogram segmentation, characterized in that: The steps include: Step a. Generate an initial depth map of the image based on the CVP-MVSNet network; Step b. Based on the initial depth map, extract the depth map and construct the corresponding depth histogram DL; Step c. Let the horizontal axis of the histogram be depth and the vertical axis be frequency or count, count the area of ​​the histogram, traverse the horizontal axis, divide the depth histogram into M-1 sub-intervals of equal area, and scale the two ends of the sub-interval to dmin to dmax. Each horizontal axis division point represents the updated hypothetical depth; Step d. Iterate the initial depth map in step a using the updated assumed depth to generate an updated initial depth map; Step e. Use the updated hypothesized depth and the updated initial depth map to upsample and gradually construct the local cost volume of the CVP-MVSNet network for depth refinement to generate the final high-resolution depth map.

2. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 1 is characterized in that: In step a, the initial depth map is generated as follows: Step a1: Construct the image pyramid structure of the CVP-MVSNet network; Step a2: Use a feature extraction network to extract features layer by layer on the image pyramid structure; Step a3: Project all the image features extracted in step 2 to M uniformly distributed hypothetical depths of the parallel plane in front of the reference view through homography transformation, and then calculate the variance of the same pixel in different views at the same depth as the cost volume; Step a4: Convert the cost volume in step a3 into a probability distribution, determine the relative likelihood of each depth hypothesis relative to other hypotheses, and thus generate a probability volume; Step a5: Combine the cost volume and probability volume in steps a3 and a4 to extract the best depth estimate for each pixel in the reference view and generate an initial depth map.

3. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 2 is characterized in that: In step d, the iteration process is as follows: the assumed depth in step a3 is replaced with the updated assumed depth obtained in step c, and steps a3 to a5 are re-executed to generate an updated initial depth map.

4. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 2 is characterized in that: The construction method of step a1 is as follows: downsample the input multi-view image L times, and the resolution of each layer is 1 / 2 of the previous layer, forming an L+1 layer image pyramid, where the L layer has the lowest resolution and the 0 layer has the highest resolution.

5. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 4 is characterized in that: The length and width resolution of all images in layer L shall not be less than 70 pixels.

6. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 2, characterized in that: The feature extraction method of step a2 is as follows: the feature extraction network has 9 layers in total, the number of output channels of the first 3 layers is 64, the number of output channels of the middle 3 layers is 32, and the number of output channels of the last 3 layers is 16. The size of each layer is 3×3, the step size is 1, and a Leaky ReLU layer is assigned after each layer.

7. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 2, characterized in that: In step a3, the depth range is from dmin to dmax.

8. The CVP-MVSNet adaptive depth update method based on histogram segmentation according to claim 2, characterized in that: In step a4, the probability distribution is converted by performing a Softmax operation on each cost volume depth layer in step a3 to convert the cost volume into a probability distribution.

Images