Depth map correction method and device based on clustering and neighborhood expansion

Abstract

The present invention proposes a depth map correction method and device based on clustering and neighborhood expansion. The absolute depth map is used to restore the scale of the monocular depth map to the real scale through histogram matching; the monocular depth is further corrected based on clustering and neighborhood expansion; the absolute depth map and the corrected monocular depth map are fused using bilateral filtering to achieve the generation of a complete depth map. The present invention can solve the problem of holes in the depth map generated by traditional dense matching. The processing process is simple and the calculation efficiency is high. It is suitable for scenes such as depth completion and three-dimensional reconstruction of low-overlapping images.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction and relates to a depth map correction technology based on clustering and neighborhood expansion, and specifically relates to combining a monocular depth map and an absolute depth map obtained by traditional dense matching to achieve complete depth map generation, wherein depth correction based on clustering and neighborhood expansion is the key technology of this solution. Background Art

[0002] Accurately reconstructing an image into a three-dimensional scene usually involves depth map recovery, that is, inferring the depth information of the three-dimensional scene from a two-dimensional image. It has important application value in virtual reality, augmented reality, robot navigation, unmanned driving and other fields.

[0003] Traditional methods of obtaining depth maps mainly include binocular stereo matching and multi-view dense matching. These methods rely on extracting dense feature points from images of multiple perspectives and matching them to calculate the depth information of the scene. However, in areas with weak texture or complex texture in the image, the matching process may fail, resulting in the inability to recover the depth of these areas and holes in the depth map. In addition, when the overlap between images is low, there will also be a problem of missing depth in non-overlapping areas, which ultimately affects the integrity and accuracy of 3D reconstruction. In recent years, monocular depth estimation has developed rapidly in the field of computer vision and has attracted widespread attention. Compared with traditional stereo vision-based methods, monocular depth estimation technology can directly predict the depth of each pixel from a single image by using a large number of data sets to learn the depth relationship between pixels in the image, and has broad application prospects. However, most of the current monocular depth estimation methods can only restore the relative depth relationship of the image, and there are problems of global scale ambiguity and inaccurate depth. Some monocular depth estimation methods that can directly predict absolute depth have poor generalization, are difficult to adapt to changes in different scenes, have poor accuracy, and are difficult to directly apply to 3D reconstruction. Since neither traditional dense matching nor monocular depth estimation can obtain complete and correct image depth maps, considering the combined use of these two technologies can effectively make up for their respective shortcomings. However, there are still certain challenges in how to use the absolute depth map to restore the scale of the monocular depth map and how to fuse the two types of depth maps. Summary of the invention

[0004] Aiming at the problem of holes in depth maps generated by traditional dense matching, the present invention proposes a depth map correction technical solution based on clustering and neighborhood expansion in combination with a monocular depth map.

[0005] The technical solution proposed by the present invention for the above technical problem is a depth map correction method based on clustering and neighborhood expansion, which includes the following process:

[0006] Using the absolute depth map, the scale of the monocular depth map is restored to the real scale through histogram matching;

[0007] Further correction of monocular depth based on clustering and neighborhood expansion;

[0008] The absolute depth map and the corrected monocular depth map are fused using bilateral filtering to generate a complete depth map.

[0009] Furthermore, the correction implementation process includes the following steps,

[0010] Step 1: Obtain the absolute depth map of the image using dense matching , obtain the monocular depth map of the image based on deep learning ; The monocular depth map Resample to absolute depth map Same size, make sure both are pixel aligned;

[0011] Step 2: Transform the monocular depth map With absolute depth map Perform histogram matching to obtain a new monocular depth map ;

[0012] Step 3: For the monocular depth map A monocular depth value in , record the absolute depth map Absolute depth at the same pixel position , classified into the collection , calculate the set The average of the absolute depth values and standard deviation ;

[0013] Step 4: Based on the standard deviation To determine the mapping strategy, if the standard deviation is less than the preset threshold , then the mean As The mapping result, otherwise the set Perform K-means clustering on the elements in k subsets, and accordingly k Cluster Centers , ;

[0014] Step 5: For The elements in the subset are found in the monocular depth map, and the neighborhood is expanded outward, and the monocular depth value expanded to is updated as the cluster center of this class. ;

[0015] Step 6: Repeat steps 3 to 5 for all the monocular depth values ​​in the monocular depth map to generate the corrected monocular depth map. ;

[0016] Step 7: Use bilateral filtering to transform the absolute depth map And the corrected monocular depth map The fusion is performed to obtain the final depth map.

[0017] Moreover, in step 1, the monocular depth map is resampled using a nearest neighbor sampling method.

[0018] Moreover, in step 2, a new monocular depth map is obtained by using histogram matching. The implementation method includes the following process:

[0019] Get the absolute depth map Valid depth value area and generate a mask;

[0020] Count the absolute depth map separately With monocular depth map Depth histogram in the masked area;

[0021] Calculate the cumulative histogram of the two depth histograms;

[0022] For monocular depth map Each depth value in is calculated based on its position in the corresponding cumulative histogram in the absolute depth map Find the corresponding depth value in the cumulative histogram to replace the original depth value and get a new monocular depth map .

[0023] Moreover, the neighborhood expansion in step 5 is implemented by taking a certain monocular depth as the center and continuously searching the neighborhood for a position equal to the depth.

[0024] Moreover, the deep fusion using bilateral filtering in step 7 is implemented in the following manner:

[0025] Preserve absolute depth map The area with effective depth in the image is obtained by using the corrected monocular depth map. Fill Invalid depth area;

[0026] Create a width of 0 at the junction of the two depth maps. w Buffer zone;

[0027] For a point in the buffer , calculated by the following formula Depth value of the point after bilateral filtering :

[0028]

[0029] in, So is the set of all depths within the window centered at For collection Pixels in

[0030] and They are Point and The depth value of the point, is the normalization factor; is a weight function based on the depth value difference, It is a weight function based on spatial distance.

[0031] Moreover, it is used for fast 3D reconstruction of low-overlap images.

[0032] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the depth map correction method based on clustering and neighborhood expansion as described above is implemented.

[0033] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the depth map correction method based on clustering and neighborhood expansion as described above.

[0034] On the other hand, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the depth map correction method based on clustering and neighborhood expansion as described above.

[0035] The present invention proposes a depth map correction scheme based on clustering and neighborhood expansion. First, based on the absolute depth map, the absolute scale of the monocular depth map is restored by histogram matching; then the monocular depth is further corrected by clustering analysis and neighborhood expansion; finally, the monocular depth and the absolute depth are fused by bilateral filtering. In this way, a complete depth map can be obtained, which can be applied to depth completion, three-dimensional reconstruction of weak texture and complex texture areas. The invention uses the absolute depth map to restore the scale of the monocular depth map, and fuses it to obtain a complete and correct depth map. Compared with the prior art, the present invention has the following advantages: the present invention can complete the depth map, the calculation method is simple, the operation efficiency is high, and it is suitable for scenes such as rapid three-dimensional reconstruction of low overlapping images. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of correcting a monocular depth map and fusing it with an absolute depth map according to an embodiment of the present invention.

[0037] Figure 2The figure is a schematic diagram of the principle of correcting a monocular depth map by using histogram matching according to an absolute depth map according to an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of monocular depth correction based on clustering and neighborhood expansion according to an embodiment of the present invention, wherein part (a) is a schematic diagram of pixel-by-pixel mapping between a monocular and an absolute depth map; part (b) is a schematic diagram of clustering analysis; part (c) is a schematic diagram of neighborhood expansion; and part (d) is a schematic diagram of updating a monocular depth map.

[0039] Figure 4 This is a schematic diagram of the bilateral filtering fusion of absolute depth and corrected monocular depth in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of the effect of using the depth map correction method proposed in the present invention on low-altitude UAV images.

[0041] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0043] The embodiment of the present invention provides a depth map correction method based on clustering and neighborhood extension, which includes three processes: first, using the absolute depth map, the scale of the monocular depth map is restored to the real scale through histogram matching; then the monocular depth is further corrected based on clustering and neighborhood extension; finally, the absolute depth map and the corrected monocular depth map are fused by bilateral filtering to achieve the generation of a complete depth map.

[0044] like Figure 1 As shown, the depth map correction method based on clustering and neighborhood expansion provided in the embodiment of the present invention includes three processes, which can be specifically divided into the following steps:

[0045] 1) First, the absolute depth map is used to restore the scale of the monocular depth map to the actual scale through histogram matching.

[0046] Furthermore, process 1) includes step 1 and step 2.

[0047] Step 1: Use binocular dense matching or multi-view dense matching technology to obtain the absolute depth map of the image , using the monocular depth estimation method based on deep learning to obtain the monocular depth map of the image ; The monocular depth map Resample to absolute depth map Same size, making sure both are pixel aligned.

[0048] The depth map resampling method is preferably nearest neighbor sampling. The specific implementation of binocular dense matching or multi-view dense matching technology can adopt existing technology, which will not be described in detail in the present invention.

[0049] In specific implementation, the monocular depth estimation method based on deep learning preferably recommends the use of the Depth Anything network, which is a highly practical monocular depth estimation solution in the prior art. At the same time, images with real depth labels and massive images without real depth labels are used for training to obtain better monocular depth estimation results. In specific implementation, a teacher model T can be first trained using the labeled image Dl, which can accurately predict depth information from the image. Then the unlabeled image Du is predicted using the trained teacher model T to generate a pseudo-depth label. The purpose of this step is to expand the data set that can be used for training. Finally, the labeled data set is combined with the generated pseudo-depth label data set to train the student model S, thereby obtaining a depth estimation model that can perform well even on unlabeled data.

[0050] Step 2, such as Figure 2 As shown, the monocular depth map With absolute depth map Perform histogram matching to obtain a new monocular depth map .

[0051] The implementation method further preferably adopted in the embodiment includes the following sub-steps:

[0052] Step 2.1, get the absolute depth map Effective depth value (ie depth value 0) area and generate a mask;

[0053] Step 2.2: Count the absolute depth maps separately With monocular depth map Depth histogram in the masked area;

[0054] Step 2.2, based on each depth value in the absolute depth map or monocular depth map The probability of appearing in the image , the cumulative histograms of the corresponding depth histograms are obtained using the cumulative distribution function CDF;

[0055]

[0056] in, Represents the depth value of each pixel in the absolute depth map or monocular depth map, =0,1,…, , is the corresponding cumulative distribution function value.

[0057] Step 2.3, for the monocular depth map Each depth value in the absolute depth map is calculated based on its position in the cumulative histogram. Find the corresponding depth value in the cumulative histogram to replace the original depth value, so as to obtain a new monocular depth map This step establishes the mapping relationship between the monocular depth value and the absolute depth value by substitution, and updates the monocular depth map more finely to make it as close to the real depth as possible.

[0058] 2) Further correction of monocular depth based on clustering and neighborhood expansion.

[0059] Furthermore, process 2) includes steps 3-6.

[0060] Step 3, such as Figure 3 As shown in part (a) of the figure, for a monocular depth map A monocular depth value in , record the absolute depth map Absolute depth at the same pixel position , put it into the collection ( Figure 3 In part (a) are two different monocular depths), for example, the green pixel depth values ​​are , the blue pixel depth values ​​are , the corresponding depth values ​​of these positions in the absolute depth map D are , .

[0061] Calculating Collections The average of the absolute depth values and standard deviation Mean and standard deviation The calculation formulas are:

[0062]

[0063] In the formula, For collection The number of elements in .

[0064] Step 4, such as Figure 3 As shown in part (b) of To determine the mapping strategy, if the standard deviation Less than a certain threshold , then the mean As The mapping result is = ; If the standard deviation is greater than or equal to , then for the set The elements in are clustered by K-means. Cluster to k In the subset, we get k Cluster Centers , Label the subset.

[0065] In specific implementation, the threshold According to the empirical value obtained by experiment. Take 1m.

[0066] For ease of implementation reference, the following provides a collection of The specific implementation of K-means clustering of the elements in includes the following sub-steps:

[0067] Step 4.1, in the collection N Random selection k The absolute depth value is used as the initial cluster center;

[0068] Step 4.2, calculate the set N The Euclidean distance from each absolute depth value in to each cluster center is calculated and assigned to the class where the nearest cluster center is located;

[0069] Step 4.3, for each class, recalculate the mean of all absolute depth values ​​within it and use the mean as the new cluster center;

[0070] Step 4.4, repeat steps 4.2 and 4.3 until there is no significant change in the cluster center between two consecutive iterations or the maximum number of iterations is reached.

[0071] When clustering, the optimal number of clusters k It can be determined by methods such as the elbow rule or silhouette coefficient.

[0072] Step 5, such as Figure 3 As shown in parts (c) and (d) of The elements in the cluster subset are found in the monocular depth map, and the neighborhood is expanded outward, and the expanded monocular depth value is updated as the cluster center of the class. Among them, the neighborhood expansion is specifically to take a certain monocular depth as the center and continuously search the neighborhoods outwards for positions equal to its depth. When implemented specifically, the neighborhoods can use four neighborhoods or eight neighborhoods.

[0073] Step 6: Repeat steps 3 to 5 for all the monocular depth values ​​in the monocular depth map to generate the corrected monocular depth map. .

[0074] 3) Use bilateral filtering to fuse the absolute depth map and the corrected monocular depth map to generate a complete depth map.

[0075] Furthermore, process 3) includes step 7.

[0076] Step 7, such as Figure 4 As shown, the absolute depth map is transformed into And the corrected monocular depth map The fusion is performed to obtain the final depth map.

[0077] The implementation method further preferably adopted in the embodiment includes the following sub-steps:

[0078] Step 7.1: Keep the absolute depth map has a valid depth (depth value 0), using the corrected monocular depth map Fill Invalid depth (depth value 0) area.

[0079] Step 7.2: Create a width of w The buffer zone, Figure 4 middle w =4, the specific implementation can be determined by actual production w Get value;

[0080] Step 7.3, for a point in the buffer zone , calculated by Equation 4 Depth value of the point after bilateral filtering :

[0081]

[0082] in, So centered The collection of all depths in the window. r It can be used as an adaptive window size, and its value is determined based on whether the extended boundary is broken. For collection Pixels in and They are Point and The depth value of the point, is a normalization factor that ensures that the weights sum to 1 and is defined as , Indicates the modulus length;

[0083] It is a weighting function based on the depth value difference, which is used to reduce the impact of the depth with large depth difference on the target depth. It is a weight function based on spatial distance, which is used to reduce the influence of pixels far from the target pixel on it. Both use Gaussian function, and the expression is:

[0084]

[0085] in, is the weight that controls the difference in depth values; is the weight controlling the spatial distance, is a mathematical constant, and Represent two pixels respectively. and is an image I Location and The depth value at Indicates the modulus length.

[0086] In specific implementation, the above process can be automatically run using computer software technology, which is particularly suitable for scenarios such as rapid three-dimensional reconstruction of low-overlap images, such as drone images with adjacent overlap <30%.

[0087] In order to facilitate understanding of the technical effect of the present invention, as a comparison, Figure 5 The effect of using the depth map correction method proposed in the present invention on low-altitude drone images in a certain area is given. By comparing the absolute depth map with the monocular depth map corrected by histogram matching, it can be clearly seen that the depth values ​​of the overlapping areas between them are already relatively close. The monocular depth map after depth correction can also restore the depth information of the invalid depth positions (i.e., black areas) in the absolute depth map after the monocular depth map after histogram matching.

[0088] Based on the three indicators of mean error, median error and root mean square error in Table 1, it can be seen that the method based on clustering and neighborhood extension shows a lower error level than the histogram matching method in a certain regional image.

[0089] Table 1

[0090]

[0091] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute the depth map correction method based on clustering and neighborhood expansion.

[0092] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the depth map correction method based on clustering and neighborhood expansion provided by the above-mentioned methods.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the depth map correction method based on clustering and neighborhood expansion provided by the above methods.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A depth map correction method based on clustering and neighborhood expansion, characterized in that: Including using the absolute depth map to restore the scale of the monocular depth map to the real scale through histogram matching; The monocular depth is further corrected based on clustering and neighborhood expansion; the absolute depth map and the corrected monocular depth map are fused using bilateral filtering to generate a complete depth map; The correction implementation process includes the following steps: Step 1: Obtain the absolute depth map of the image using dense matching , obtain the monocular depth map of the image based on deep learning ; The monocular depth map Resample to absolute depth map Same size, make sure both are pixel aligned; Step 2: Transform the monocular depth map With absolute depth map Perform histogram matching to obtain a new monocular depth map ; Step 3: For the monocular depth map A monocular depth value in , record the absolute depth map Absolute depth at the same pixel position , classified into the collection , calculate the set The average of the absolute depth values and standard deviation ; Step 4: Based on the standard deviation To determine the mapping strategy, if the standard deviation is less than the preset threshold , then the mean As The mapping result, otherwise the set Perform K-means clustering on the elements in k subsets, and accordingly k Cluster Centers , ; Step 5: For The elements in the subset are found in the monocular depth map, and the neighborhood is expanded outward, and the monocular depth value expanded to is updated as the cluster center of this class. ; Step 6: Repeat steps 3 to 5 for all the monocular depth values ​​in the monocular depth map to generate the corrected monocular depth map. ; Step 7: Use bilateral filtering to transform the absolute depth map And the corrected monocular depth map The fusion is performed to obtain the final depth map.

2. A depth map correction method based on clustering and neighborhood expansion as claimed in claim 1, characterized in that: In step 1, the monocular depth map is resampled using the nearest neighbor sampling method.

3. The depth map correction method based on clustering and neighborhood expansion according to claim 1, characterized in that: In step 2, a new monocular depth map is obtained by using histogram matching. The implementation method includes the following process: Get the absolute depth map Valid depth value area and generate a mask; Count the absolute depth map separately With monocular depth map Depth histogram in the masked area; Calculate the cumulative histogram of the two depth histograms; For monocular depth map Each depth value in is calculated based on its position in the corresponding cumulative histogram in the absolute depth map Find the corresponding depth value in the cumulative histogram to replace the original depth value and get a new monocular depth map .

4. The depth map correction method based on clustering and neighborhood expansion according to claim 1, characterized in that: The neighborhood expansion in step 5 is implemented by taking a certain monocular depth as the center and continuously searching the neighborhood for positions equal to the depth.

5. The depth map correction method based on clustering and neighborhood expansion according to claim 1, characterized in that: In step 7, deep fusion using bilateral filtering is implemented in the following way: Preserve absolute depth map The area with effective depth in the image is obtained by using the corrected monocular depth map. Fill Invalid depth area; Create a width of 0 at the junction of the two depth maps. w Buffer zone; For a point in the buffer , calculated by the following formula Depth value of the point after bilateral filtering : in, So is the set of all depths within the window centered at For collection Pixels in and They are Dot and The depth value of the point, is the normalization factor; is a weight function based on the depth value difference, It is a weight function based on spatial distance.

6. A depth map correction method based on clustering and neighborhood expansion as claimed in claim 1 or 2 or 3 or 4 or 5, characterized in that: Used for fast 3D reconstruction of low-overlap images.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the depth map correction method based on clustering and neighborhood expansion is implemented as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the depth map correction method based on clustering and neighborhood expansion is implemented as claimed in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the depth map correction method based on clustering and neighborhood expansion is implemented as claimed in any one of claims 1 to 6.

Images