Depth map processing method, device and electronic device

Abstract

The present invention provides a depth map processing method, device and electronic device. The method includes: performing stereo matching on an input main image and a secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map; determining the area of ​​a low confidence zone in the first confidence map, and obtaining a low confidence zone and a high confidence zone; when the area of ​​the low confidence zone in the first confidence map is greater than or equal to a predetermined threshold, using a monocular depth estimation model to perform monocular depth estimation on the main image to obtain a second depth map, and fitting the first depth map and the second depth map in the high confidence zone to obtain a corresponding relationship, using the corresponding relationship and the second depth map to obtain a third depth map, and performing subsequent processing and application of the depth map on the third depth map; when the area of ​​the low confidence zone in the first confidence map is less than a predetermined threshold, performing subsequent processing and application of the depth map on the first depth map. The present invention improves the quality of the depth map.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a depth map processing method, device and electronic equipment. Background Art

[0002] With the development of computer technology and multimedia technology, people's perception of the three-dimensional world continues to increase. More and more applications need to obtain the distance (i.e. depth) of the three-dimensional scene relative to the camera, such as bokeh, three-dimensional reconstruction, human-computer interaction, pattern recognition, etc., all of which use depth maps to represent the third-dimensional information of objects. However, general camera technology can only record three-dimensional space in a two-dimensional way. Therefore, how to obtain high-quality depth information has become a crucial technology for computer vision.

[0003] Common methods for generating depth maps include stereo matching algorithms, such as local stereo matching, global stereo matching, and semi-global stereo matching formed by combining the two. Most local stereo matching algorithms are based on window implementation, which has low complexity and is easy to implement in real time. However, the depth maps generated by local stereo matching algorithms are often noisy and prone to false matches in low-texture areas, repeated texture areas, and occluded areas. Global algorithms generally add smoothing terms and calculate by solving optimization problems, which are often complex and computationally expensive. Semi-global stereo matching, as a pixel-by-pixel matching method, uses mutual information to evaluate the matching cost and uses dynamic programming to search for the optimal path within a one-dimensional smooth constraint. The above stereo matching-based algorithms are prone to false matches in low-texture areas, repeated texture areas, and occluded areas, thereby obtaining incorrect depth values. Summary of the invention

[0004] The depth map processing method, device and electronic device provided by the present invention can improve the quality of the depth map by combining monocular depth estimation based on deep learning on the basis of binocular stereo matching.

[0005] In a first aspect, the present invention provides a depth map processing method, the method comprising:

[0006] Performing stereo matching on the input main image and the secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map;

[0007] Determine the area of ​​the low confidence region in the first confidence map, and obtain the low confidence region and the high confidence region;

[0008] When the area of ​​the low confidence region in the first confidence map is greater than or equal to a predetermined threshold, a monocular depth estimation model is used to perform monocular depth estimation on the main image to obtain a second depth map, and the first depth map and the second depth map in the high confidence region are fitted to obtain a corresponding relationship, and a third depth map is obtained by using the corresponding relationship and the second depth map, and the third depth map is subsequently processed and applied;

[0009] When the area of ​​the low confidence region in the first confidence map is smaller than the predetermined threshold, subsequent processing and application of the depth map are performed on the first depth map.

[0010] Optionally, before performing stereo matching on the input main image and the secondary image based on the binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map, the method further includes:

[0011] A monocular depth estimation model is trained based on deep learning to obtain a monocular depth estimation model.

[0012] Optionally, the obtaining a third depth map by using the corresponding relationship and the second depth map includes:

[0013] A third depth map is obtained by calculating the corresponding relationship for the second depth map, and the third depth map conforms to the distribution of the first depth map.

[0014] Optionally, the obtaining a third depth map by using the corresponding relationship and the second depth map includes:

[0015] Calculating the second depth map inside the low confidence area through the corresponding relationship to obtain a fourth depth map inside the low confidence area;

[0016] The pixels in the first depth map inside the low confidence area are replaced with corresponding pixels in the fourth depth map, that is, the first depth map is taken from the high confidence area, the fourth depth map is taken from the low confidence area, and the first depth map and the fourth depth map are merged to obtain a third depth map.

[0017] In a second aspect, the present invention provides a depth map processing device, the device comprising:

[0018] A binocular stereo matching unit, configured to perform stereo matching on the input main image and the secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map;

[0019] A determination unit, used to determine the area of ​​the low confidence region in the first confidence map, and obtain the low confidence region and the high confidence region;

[0020] a monocular depth estimation unit, configured to perform monocular depth estimation on the main image using a monocular depth estimation model to obtain a second depth map when an area of ​​a low confidence region in the first confidence map is greater than or equal to a predetermined threshold;

[0021] A fitting unit, used for fitting the first depth map and the second depth map in the high confidence region to obtain a corresponding relationship;

[0022] A first processing unit, configured to obtain a third depth map by using the corresponding relationship and the second depth map, and perform subsequent processing and application of the third depth map;

[0023] The second processing unit is configured to perform subsequent processing and application of the first depth map when the area of ​​the low confidence region in the first confidence map is smaller than the predetermined threshold.

[0024] Optionally, the device further comprises:

[0025] The model building unit is used to perform monocular depth estimation model training based on deep learning to obtain a monocular depth estimation model before the binocular stereo matching unit performs stereo matching on the input main image and the secondary image based on the binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map.

[0026] Optionally, the first processing unit is further configured to calculate the second depth map through the corresponding relationship to obtain a third depth map, where the third depth map conforms to the distribution of the first depth map.

[0027] Optionally, the first processing unit is further used to calculate the second depth map inside the low confidence area through the corresponding relationship to obtain a fourth depth map inside the low confidence area; replace the pixels in the first depth map inside the low confidence area with corresponding pixels in the fourth depth map, that is: take the first depth map from the high confidence area, take the fourth depth map from the low confidence area, and merge the first depth map and the fourth depth map to obtain a third depth map.

[0028] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned depth map processing method.

[0032] In a fourth aspect, the present invention provides a chip, comprising:

[0033] at least one processor; and

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned depth map processing method.

[0036] In a fifth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the above-mentioned depth map processing method when executed by a processor.

[0037] The depth map processing method, device and electronic device provided by the embodiments of the present invention combine the monocular depth estimation based on deep learning and the depth estimation based on binocular stereo matching, and complement each other's advantages, so as to obtain a more accurate depth map and improve the quality of the depth map, especially in areas prone to matching errors such as dense repetitive textures, low textures, and occluded areas; in addition, for the relationship between the monocular depth establishment and the binocular stereo matching depth of dimensionless or different image processing domains (depth map domain or disparity map domain), the monocular depth can be converted to the binocular depth distribution, and the subsequent processing and application can be seamlessly connected. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a depth map processing method according to an embodiment of the present invention;

[0039] Figure 2 A flowchart of a depth map processing method according to another embodiment of the present invention;

[0040] Figure 3 It is a structural schematic diagram of a depth map processing device according to an embodiment of the present invention;

[0041] Figure 4 FIG. 4 is a schematic diagram of the structure of a depth map processing device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] An embodiment of the present invention provides a depth map processing method, the method is applied to electronic devices, such as Figure 1 As shown, the method includes:

[0044] S11. Perform stereo matching on the input main image and the secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map.

[0045] S12. Determine the area of ​​the low confidence region in the first confidence map, and obtain a low confidence region and a high confidence region.

[0046] S13. When the area of ​​the low confidence zone in the first confidence map is greater than or equal to a predetermined threshold, a monocular depth estimation model is used to perform monocular depth estimation on the main image to obtain a second depth map, and the first depth map and the second depth map in the high confidence zone are fitted to obtain a corresponding relationship, and a third depth map is obtained by using the corresponding relationship and the second depth map, and the third depth map is subsequently processed and applied.

[0047] S14: When the area of ​​the low confidence region in the first confidence map is smaller than the predetermined threshold, perform subsequent processing and application of the depth map on the first depth map.

[0048] The depth map processing method provided by the embodiment of the present invention combines the monocular depth estimation based on deep learning and the depth estimation based on binocular stereo matching, and complements the advantages of the two, so as to obtain a more accurate depth map and improve the quality of the depth map, especially in areas prone to matching errors such as dense repetitive textures, low textures, and occluded areas; in addition, for the relationship between the monocular depth establishment and the binocular stereo matching depth of dimensionless or different image processing domains (depth map domain or disparity map domain), the monocular depth can be converted to the binocular depth distribution, which can be seamlessly connected to subsequent processing and application.

[0049] The depth map processing method of the present invention is described in detail below with reference to specific embodiments.

[0050] This embodiment obtains a depth map based on binocular stereo matching. When the matching confidence is low, a depth map is obtained by monocular depth estimation, and the depth map of the monocular depth estimation is converted, or the two are fused so that their distribution is consistent with the binocular stereo matching, which is convenient for subsequent applications.

[0051] like Figure 2 As shown, the depth map processing method provided in this embodiment includes:

[0052] S21. Perform monocular depth estimation model training based on deep learning to obtain a monocular depth estimation model M1.

[0053] S22. Perform stereo matching on the main image and the secondary image based on a stereo matching algorithm to obtain a depth map Dmap1 and a confidence map Cmap1 of the depth map.

[0054] S23. According to the confidence map Cmap1, determine whether there is a low confidence area S1 greater than or equal to a predetermined threshold in the confidence map Cmap1, and obtain the low confidence area mask1 and the high confidence area mask2. When S1 is greater than or equal to the predetermined threshold, execute steps S24-S26; otherwise, execute step S27.

[0055] S24, it is considered that the depth map has many unreliable areas and poor quality. At this time, the monocular depth estimation model M1 is used to perform monocular depth estimation on the main image to obtain a depth map Dmap2;

[0056] S25. Since the image domains (or depth map domains or disparity map domains) or dimensions of monocular and binocular processing are different, in order to facilitate the use of the same subsequent processing and application, it is necessary to establish a relationship between the depth maps of monocular and binocular, and perform a unified distribution process. At this time, Dmap1 and Dmap2 in the high confidence area mask2 are fitted to obtain the corresponding relationship R1;

[0057] S26, obtaining Dmap3 by using the corresponding relationship and the monocular depth map Dmap2, and performing subsequent processing and application of the depth map on Dmap3.

[0058] Specifically, you can choose to calculate the monocular depth map Dmap2 through the relationship R1 to obtain Dmap3, so that Dmap3 conforms to the distribution of the stereo depth map Dmap1; or, you can choose to calculate the Dmap2 inside the low confidence area mask1 through the relationship R1 to obtain the Dmap3' inside the low confidence area mask1, and replace the pixels in Dmap1 inside the low confidence area mask1 with the corresponding pixels in Dmap3', that is: take the stereo depth Dmap1 from the high confidence area mask2, take Dmap3' from the low confidence area mask1, merge Dmap1 and Dmap3', complement each other's advantages, and obtain Dmap3.

[0059] S27: The depth map Dmap1 is considered to be credible, and subsequent processing and application of the depth map are performed on Dmap1.

[0060] Optionally, when S1 is greater than or equal to the predetermined threshold, steps S25-S26 may not be executed. After executing step S24, subsequent processing and application of the depth map are performed directly on the distribution of Dmap2. However, compared with the subsequent processing and application of the depth map of Dmap1 in step S27, different parameters need to be used for processing, and the system needs to save two sets of subsequent processing processes.

[0061] The depth map processing method provided by the embodiment of the present invention combines the monocular depth estimation based on deep learning and the depth estimation based on binocular stereo matching, and complements the advantages of the two, so as to obtain a more accurate depth map and improve the quality of the depth map, especially in areas prone to matching errors such as dense repetitive textures, low textures, and occluded areas; in addition, for the relationship between the monocular depth establishment and the binocular stereo matching depth of dimensionless or different image processing domains (depth map domain or disparity map domain), the monocular depth can be converted to the binocular depth distribution, and the subsequent processing and application can be seamlessly connected; in addition, the computational complexity of the lightweight monocular depth estimation is low, the efficiency is high, and it can be processed in real time.

[0062] The embodiment of the present invention further provides a depth map processing device, the device is located in an electronic device, such as Figure 3 As shown, the device comprises:

[0063] A binocular stereo matching unit 11 is used to perform stereo matching on the input main image and the secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map;

[0064] A determination unit 12, configured to determine the area of ​​the low confidence region in the first confidence map, and obtain the low confidence region and the high confidence region;

[0065] A monocular depth estimation unit 13, configured to perform monocular depth estimation on the main image using a monocular depth estimation model to obtain a second depth map when the area of ​​the low confidence region in the first confidence map is greater than or equal to a predetermined threshold;

[0066] A fitting unit 14, configured to fit the first depth map and the second depth map in the high confidence region to obtain a corresponding relationship;

[0067] A first processing unit 15 is configured to obtain a third depth map by using the corresponding relationship and the second depth map, and perform subsequent processing and application of the third depth map;

[0068] The second processing unit 16 is configured to perform subsequent processing and application of the first depth map when the area of ​​the low confidence region in the first confidence map is smaller than the predetermined threshold.

[0069] The depth map processing device provided by the embodiment of the present invention combines the monocular depth estimation based on deep learning and the depth estimation based on binocular stereo matching, and complements the advantages of the two, so as to obtain a more accurate depth map and improve the quality of the depth map, especially in areas prone to matching errors such as dense repetitive textures, low textures, and occluded areas; in addition, for the relationship between the monocular depth establishment and the binocular stereo matching depth of dimensionless or different image processing domains (depth map domain or disparity map domain), the monocular depth can be converted to the binocular depth distribution, which can be seamlessly connected to subsequent processing and applications.

[0070] Furthermore, if Figure 4 As shown, the device also includes:

[0071] The model building unit 17 is used to perform monocular depth estimation model training based on deep learning to obtain a monocular depth estimation model before the binocular stereo matching unit 11 performs stereo matching on the input main image and the secondary image based on the binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map.

[0072] Optionally, the first processing unit 15 is further configured to calculate the second depth map through the corresponding relationship to obtain a third depth map, where the third depth map conforms to the distribution of the first depth map.

[0073] Optionally, the first processing unit 15 is further used to calculate the second depth map inside the low confidence area through the corresponding relationship to obtain a fourth depth map inside the low confidence area; replace the pixels in the first depth map inside the low confidence area with corresponding pixels in the fourth depth map, that is: take the first depth map from the high confidence area, take the fourth depth map from the low confidence area, and merge the first depth map and the fourth depth map to obtain a third depth map.

[0074] The device of this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0075] An embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0076] at least one processor; and

[0077] a memory communicatively connected to the at least one processor; wherein,

[0078] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned depth map processing method.

[0079] An embodiment of the present invention further provides a chip, comprising:

[0080] at least one processor; and

[0081] a memory communicatively connected to the at least one processor; wherein,

[0082] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned depth map processing method.

[0083] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the above-mentioned depth map processing method is implemented.

[0084] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0085] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A depth map processing method, characterized in that: The method comprises: Performing stereo matching on the input main image and the secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map; Determine the area of ​​the low confidence region in the first confidence map, and obtain the low confidence region and the high confidence region; When the area of ​​the low confidence region in the first confidence map is greater than or equal to a predetermined threshold, a monocular depth estimation model is used to perform monocular depth estimation on the main image to obtain a second depth map, and the first depth map and the second depth map in the high confidence region are fitted to obtain a corresponding relationship, and a third depth map is obtained by using the corresponding relationship and the second depth map, and the third depth map is subsequently processed and applied; When the area of ​​the low confidence region in the first confidence map is smaller than the predetermined threshold, performing subsequent processing and application of the depth map on the first depth map; The obtaining of the third depth map by using the corresponding relationship and the second depth map includes: calculating the second depth map inside the low confidence area by using the corresponding relationship to obtain a fourth depth map inside the low confidence area; replacing pixels in the first depth map inside the low confidence area with corresponding pixels in the fourth depth map, that is, taking the first depth map from the high confidence area, taking the fourth depth map from the low confidence area, and fusing the first depth map and the fourth depth map to obtain the third depth map.

2. The method according to claim 1, characterized in that Before performing stereo matching on the input main image and the secondary image based on the binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map, the method further includes: A monocular depth estimation model is trained based on deep learning to obtain a monocular depth estimation model.

3. The method according to claim 1 or 2, characterized in that: The obtaining a third depth map by using the corresponding relationship and the second depth map comprises: A third depth map is obtained by calculating the corresponding relationship for the second depth map, and the third depth map conforms to the distribution of the first depth map.

4. A depth map processing device, characterized in that: The device comprises: A binocular stereo matching unit, configured to perform stereo matching on the input main image and the secondary image based on a binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map; A determination unit, used to determine the area of ​​the low confidence region in the first confidence map, and obtain the low confidence region and the high confidence region; a monocular depth estimation unit, configured to perform monocular depth estimation on the main image using a monocular depth estimation model to obtain a second depth map when an area of ​​a low confidence region in the first confidence map is greater than or equal to a predetermined threshold; A fitting unit, used for fitting the first depth map and the second depth map in the high confidence region to obtain a corresponding relationship; A first processing unit, configured to obtain a third depth map by using the corresponding relationship and the second depth map, and perform subsequent processing and application of the third depth map; A second processing unit, configured to perform subsequent processing and application of the first depth map on the first depth map when the area of ​​the low confidence region in the first confidence map is smaller than the predetermined threshold; The first processing unit is further used to calculate the second depth map inside the low confidence area through the corresponding relationship to obtain a fourth depth map inside the low confidence area; replace the pixels in the first depth map inside the low confidence area with corresponding pixels in the fourth depth map, that is, take the first depth map from the high confidence area, take the fourth depth map from the low confidence area, and merge the first depth map and the fourth depth map to obtain a third depth map.

5. The device according to claim 4, characterized in that The device also includes: The model building unit is used to perform monocular depth estimation model training based on deep learning to obtain a monocular depth estimation model before the binocular stereo matching unit performs stereo matching on the input main image and the secondary image based on the binocular stereo matching algorithm to obtain a first depth map and a first confidence map of the first depth map.

6. The device according to claim 4 or 5, characterized in that The first processing unit is further configured to calculate the third depth map through the corresponding relationship for the second depth map, where the third depth map conforms to the distribution of the first depth map.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A chip, characterized in that: The chip comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

9. A chip module, characterized in that: The chip module comprises the chip according to claim 8.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images