Image depth estimation method and device, equipment, computer readable storage medium

Abstract

The application discloses an image depth estimation method and device, equipment and a computer readable storage medium. The image depth estimation method comprises the following steps: acquiring a reference image and a plurality of source images corresponding to the reference image, wherein the source images have the same view angle as the reference image; performing depth map initialization on the reference image to determine an initial depth image of the reference image; and performing depth propagation on the initial depth image of the reference image based on the plurality of source images to obtain depth information of the reference image. The above scheme can obtain the depth information of the image, and the accuracy of the depth information is high.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image depth estimation method, apparatus, device, and computer-readable storage medium. Background Art

[0002] Multi-view depth estimation aims to recover frame-by-frame depth from multiple correlated images and then obtain the 3D geometric structure of the scene through depth fusion methods.

[0003] After decades of development, traditional multi-view reconstruction technology has become relatively mature, but there are still some unresolved issues, such as the reconstruction of weak textures and finer structures. The current common solution to the weak texture problem is to construct an image pyramid, which essentially reduces the weak texture of the image to a resolvable level, but it still cannot solve the problem of large areas of weak texture. The reconstruction of finer structures is more difficult, mainly because existing depth recovery technology cannot handle details well, because detailed structures often appear in a very small area in the image and are easily obliterated by the background, posing a great challenge to the reconstruction of details. Summary of the Invention

[0004] The main technical problem solved by this application is to provide an image depth estimation method and device, equipment, and computer-readable storage medium.

[0005] In a first aspect, the present application provides an image depth estimation method, which includes: obtaining a reference image and corresponding multiple source images, wherein the source image and the reference image have overlapping perspectives; initializing a depth map of the reference image to determine an initial depth image of the reference image; and performing depth propagation on the initial depth image of the reference image based on the multiple source images to obtain depth information of the reference image.

[0006] Therefore, by obtaining a reference image with overlapping perspectives and corresponding multiple source images, after initializing the depth map of the reference image, the initial depth image of the reference image can be determined, and then the initial depth image of the reference image can be depth propagated using multiple source images to obtain the depth information of the reference image. By performing depth propagation on the initial depth image of the reference image, the depth information of the reference image obtained can better process the detail structure, which is conducive to improving the reconstruction effect of the detail structure.

[0007] Among them, the obtaining of the reference image and the corresponding multiple source images includes: obtaining multiple associated images with overlapping perspectives with the reference image; obtaining the image overlap and image baseline distance between each of the associated images and the reference image; and selecting a preset number of associated images whose image overlap and image baseline distance meet preset conditions as the multiple source images.

[0008] Therefore, among multiple associated images that have overlapping perspectives with the reference image, a preset number of associated images whose image overlap and image baseline distance meet preset conditions are selected as multiple source images, so that the initial depth image of the reference image can be determined using the multiple source images, and when the initial depth image of the reference image is depth propagated, the depth information of the reference image in a larger area can be obtained, and it is conducive to estimating the accurate depth information of the reference image.

[0009] The step of initializing the depth map of the reference image to determine the initial depth image of the reference image includes randomly assigning depth and normal to all pixels in the reference image whose depth information is unknown to form the initial depth image of the reference image.

[0010] Therefore, by randomly assigning depth and normal to all pixels in the reference image, an initial depth image of the reference image can be formed, so as to facilitate depth propagation of the initial depth image of the reference image based on multiple source images, thereby achieving better restoration of the detailed structure of the scene and maintaining the integrity of the weak texture area. When there are pixels with known depth information in the reference image, the convergence speed during propagation optimization can be improved.

[0011] Among them, the depth propagation of the initial depth image of the reference image based on the multiple source images to obtain the depth information of the reference image includes: for each pixel point on the reference image, using the depth information of each sampling point in the local image block on the reference image corresponding to the current pixel point in the initial depth image to perform depth propagation to obtain the preliminary depth information of the current pixel point in the reference image; optimizing the preliminary depth information of the current pixel point in the reference image to obtain the target depth information of the current pixel point in the reference image; and obtaining the depth information of the reference image based on the target depth information of each pixel point in the reference image.

[0012] Therefore, for each pixel on the reference image, the depth information of each sampling point in the local image block corresponding to the current pixel on the reference image in the initial depth image of the reference image is used to perform depth propagation and depth information optimization, so as to obtain the target depth information of each pixel in the reference image, and then obtain the depth information of the reference image, which can better restore the detailed structure of the scene and better maintain the integrity of the weak texture area.

[0013] Among them, the depth propagation of depth information of each sampling point in the local image block on the reference image corresponding to the current pixel point in the initial depth image to obtain preliminary depth information of the current pixel point in the reference image includes: obtaining several sampling points in the local image block corresponding to the current pixel point; comparing the normalized mutual correlation values ​​of the several sampling points and the current pixel point between the reference image and the source image, and taking the depth information of the sampling point with the largest corresponding normalized mutual correlation value in the initial depth image as the preliminary depth information of the current pixel point in the reference image.

[0014] Therefore, by obtaining several sampling points in the local image block corresponding to the current pixel, and then comparing the normalized cross-correlation values ​​of several sampling points and the current pixel between the reference image and the source image, the depth information of the sampling point with the largest normalized cross-correlation value is propagated to the current pixel, so as to quickly propagate the depth information of the sampling point to the current pixel, thereby improving the convergence speed of the propagation process.

[0015] The step of obtaining a plurality of sampling points in the local image block corresponding to the current pixel point includes: dividing a plurality of sampling areas in the local image block corresponding to the current pixel point; and selecting at least one pixel point with the largest normalized mutual correlation value in each sampling area as the sampling point.

[0016] Therefore, by dividing the local image block corresponding to the current pixel point into several sampling areas, and then selecting the pixel point with the largest normalized mutual correlation value as the sampling point in each sampling area, the depth information of the corresponding sampling point with the largest normalized mutual correlation value can be propagated to the current pixel point, which can reduce the computational complexity and improve the convergence speed of the propagation process.

[0017] The plurality of sampling areas include a plurality of strip areas radially extending from the current pixel point to the periphery of the current pixel point, and / or a ring area centered on the current pixel point.

[0018] Therefore, by dividing the local image block corresponding to the current pixel point into several strip areas that radiate outward from the current pixel point as the center, the depth information of the sampling point with the largest normalized mutual correlation value in each strip area is propagated to the current pixel point, and high-confidence points in the distance can be quickly propagated to the current position, which not only improves the convergence speed but also improves the reconstruction effect of flat weak texture areas; and by dividing the local image block corresponding to the current pixel point into annular areas centered on the current pixel point, considering the principle of local spatial consistency, the depth information of at least one sampling point with the largest normalized mutual correlation value in the annular area is propagated to the current pixel point, which can greatly improve the probability of foreground propagation, so that the depth of the detail structure can be better estimated.

[0019] Among them, the optimizing the preliminary depth information of the current pixel point in the reference image to obtain the target depth information of the current pixel point in the reference image includes: perturbing the depth and normal of the current pixel point in the reference image; comparing the normalized mutual correlation value before the perturbation with the normalized mutual correlation value after the perturbation; in response to the normalized mutual correlation value after the perturbation being better than the normalized mutual correlation value before the perturbation, replacing the depth and normal of the current pixel point in the reference image with the depth and normal after the perturbation to obtain the target depth information of the current pixel point in the reference image.

[0020] Therefore, through depth propagation, the depth information of the sampling point can be propagated to the current pixel roughly and correctly, so as to obtain the preliminary depth information of the current pixel in the reference image. Further, by perturbing the depth and normal of the current pixel in the reference image after propagation, the normalized cross-correlation value before perturbation and the normalized cross-correlation value after perturbation are compared. If the normalized cross-correlation value after perturbation is better, the depth and normal of the current pixel in the reference image are replaced with the depth and normal after perturbation to improve the accuracy of depth estimation.

[0021] Wherein, after performing depth propagation on the initial depth image of the reference image based on the multiple source images to obtain the depth information of the reference image, the method further includes: performing depth consistency verification on the depth information of the reference image based on the reference image and the multiple source images to obtain the depth estimation result of the reference image.

[0022] Therefore, after using multiple source images to perform depth propagation on the initial depth image of the reference image to obtain the depth information of the reference image, a depth consistency check is performed on the depth information of the reference image based on the reference image and multiple source images to obtain the depth estimation result of the reference image. The initial depth image of the reference image is determined using multiple source images, and by performing depth propagation on the initial depth image of the reference image, the depth information of the reference image obtained can better process the detail structure, which is conducive to improving the reconstruction effect of the detail structure.

[0023] Among them, the depth consistency check is performed on the depth information of the reference image based on the reference image and the multiple source images to obtain the depth estimation result of the reference image, including: based on the depth information of the reference image and the depth information of the multiple source images, obtaining the corresponding reprojection error and projection depth error; for each pixel point, determining whether the corresponding reprojection error and projection depth error meet the preset conditions; in response to the reprojection error and projection depth error corresponding to the current pixel point meeting the preset conditions, determining that the depth information of the current pixel point is valid; in response to the reprojection error and projection depth error corresponding to the current pixel point not meeting the preset conditions, determining that the depth information of the current pixel point is invalid.

[0024] Therefore, after obtaining the depth information of the reference image, the corresponding reprojection error and projection depth error can be obtained based on the depth information of the reference image and the depth information of multiple source images. Then, for each pixel point, it can be determined whether the corresponding reprojection error and projection depth error meet the preset conditions, and when the reprojection error and projection depth error corresponding to the current pixel point meet the preset conditions, it is determined that the depth information of the current pixel point is valid. When the reprojection error and projection depth error corresponding to the current pixel point do not meet the preset conditions, it is determined that the depth information of the current pixel point is invalid, so that the noise on the depth map of the reference image can be eliminated to obtain the depth estimation result of the reference image.

[0025] In order to solve the above problems, the second aspect of the present application provides an image depth estimation device, including: an acquisition module, the acquisition module is used to acquire a reference image and corresponding multiple source images, wherein the source image and the reference image have overlapping perspectives; an initialization module, the initialization module is used to initialize the depth map of the reference image and determine the initial depth image of the reference image; and a solution module, the solution module is used to perform depth propagation on the initial depth image of the reference image based on the multiple source images to obtain the depth information of the reference image.

[0026] In order to solve the above problem, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the image depth estimation method in the above first aspect.

[0027] In order to solve the above problem, the fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the image depth estimation method in the above first aspect when executed by a processor.

[0028] The above scheme obtains a reference image with overlapping perspectives and corresponding multiple source images. After initializing the depth map of the reference image, the initial depth image of the reference image can be determined, and then the initial depth image of the reference image is depth-propagated using multiple source images to obtain the depth information of the reference image. By performing depth propagation on the initial depth image of the reference image, the depth information of the reference image obtained can better process the detail structure, which is beneficial to improving the reconstruction effect of the detail structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of an embodiment of the image depth estimation method of the present application;

[0030] Figure 2 yes Figure 1 A flow chart of an embodiment of step S11;

[0031] Figure 3 yes Figure 1 A flow chart of an embodiment of step S13;

[0032] Figure 4 yes Figure 3 A flow chart of an embodiment of step S131;

[0033] Figure 5 yes Figure 4 A flow chart of an embodiment of step S1311;

[0034] Figure 6 This is a structural diagram of an embodiment of a method for obtaining sampling points in an image depth estimation method of the present application;

[0035] Figure 7 yes Figure 3 A flow chart of an embodiment of step S132;

[0036] Figure 8 yes Figure 1 A flow chart of an embodiment of step S14;

[0037] Figure 9This is a schematic diagram of the framework of an embodiment of an image depth estimation device of the present application;

[0038] Figure 10 This is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0039] Figure 11 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0041] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0042] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" is simply a description of an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " generally indicates that the related objects are in an "or" relationship. Furthermore, "multiple" in this document means two or more than two.

[0043] The image depth estimation method of the present application may be executed by an image depth estimation device. For example, the image depth estimation method may be executed by a terminal device, a server, or other electronic device, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the image depth estimation method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0044] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the image depth estimation method of the present application. Specifically, it may include the following steps:

[0045] Step S11: Acquire a reference image and a plurality of corresponding source images, wherein the source image and the reference image have overlapping viewing angles.

[0046] In this embodiment, the execution subject takes the image depth estimation device as an example for explanation. When the image depth estimation device performs depth estimation on the current frame, it is necessary to first obtain the reference image and multiple source images corresponding to the current frame. The reference image is the image that needs to be depth estimated, and the source image is the image used for matching when performing depth estimation on the current frame. The source image and the reference image have overlapping perspectives. For example, the reference image is the standard image that needs to generate a stereoscopic vision model, and the source image is the reference image for generating the stereoscopic vision model. The reference image and the source image can be images of the same object taken at different angles; the number of source images can be multiple. Considering the balance between the speed and robustness of depth estimation, for each frame of reference image, it is more appropriate to select about 8 frames of source images according to the correlation between images. The specific number of source images is not limited in this embodiment.

[0047] Step S12: Initializing the depth map of the reference image to determine an initial depth map of the reference image.

[0048] In one embodiment, the above step S12 may specifically include: randomly assigning depths and normals to all pixels in the reference image whose depth information is unknown, to form an initial depth image of the reference image.

[0049] The reference image can be initialized with a depth map. Depth map initialization is to assign a random depth and normal to each pixel in the reference image, which is used to calculate the normalized cross-correlation of local image blocks between the reference image and the source image. By randomly assigning depth and normal to all pixels in the reference image, the depth information of each sampling point in the local image block corresponding to the current pixel in the initial depth image of the reference image can be used for depth propagation, thereby achieving better restoration of the detailed structure of the scene and maintaining the integrity of weak texture areas. When there is structural prior information, such as when there are sparse map points, since the depth and normal of the map points are known, they can be initialized with sparse map points. Therefore, the depth information of these points with known depth and normal can be quickly and accurately propagated to other pixels, which can improve the convergence speed during propagation optimization.

[0050] Step S13: performing depth propagation on the initial depth image of the reference image based on the multiple source images to obtain depth information of the reference image.

[0051] The reference image and the source image can be images of the same object taken at different angles. After determining the initial depth image of the reference image, depth propagation is performed on the initial depth image of the reference image according to a preset propagation direction, so that the depth information of the pixel points of the reference image has a higher reliability, thereby obtaining the depth information of the reference image and ensuring that the reference image has a better depth map quality.

[0052] The above scheme obtains a reference image with overlapping perspectives and corresponding multiple source images. After initializing the depth map of the reference image, the initial depth image of the reference image can be determined, and then the initial depth image of the reference image is depth-propagated using multiple source images to obtain the depth information of the reference image. By performing depth propagation on the initial depth image of the reference image, the depth information of the reference image obtained can better process the detail structure, which is beneficial to improving the reconstruction effect of the detail structure.

[0053] In one embodiment, after the above step S13, the image depth estimation method may further include:

[0054] Step S14: Based on the reference image and the multiple source images, perform a depth consistency check on the depth information of the reference image to obtain a depth estimation result of the reference image.

[0055] After performing depth propagation on the initial depth image of the reference image to obtain the depth information of the reference image, the consistency of the depth information of each pixel of the reference image and the depth information of each pixel on the multiple source images can be measured based on the reference image and the multiple source images. By performing a depth consistency check on the depth information of the reference image, it can be verified whether each pixel of the reference image has depth consistency with the multiple source images, and the noise points on the depth map of the reference image can be eliminated to obtain the depth estimation result of the reference image.

[0056] The above scheme uses multiple source images to perform depth propagation on the initial depth image of the reference image to obtain the depth information of the reference image, and then performs depth consistency verification on the depth information of the reference image based on the reference image and multiple source images, so as to obtain the depth estimation result of the reference image, and uses multiple source images to determine the initial depth image of the reference image, and performs depth propagation on the initial depth image of the reference image. The depth information of the reference image obtained can better process the detail structure, which is conducive to improving the reconstruction effect of the detail structure.

[0057] See also Figure 2 , Figure 2 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S11 in FIG. In this embodiment, the above step S11 may specifically include the following steps:

[0058] Step S111: Acquire a plurality of associated images having overlapping viewing angles with the reference image.

[0059] Step S112: Obtaining the image overlap and image baseline distance between each of the associated images and the reference image.

[0060] Step S113: selecting a preset number of associated images whose image overlap and image baseline distance satisfy preset conditions as the multiple source images.

[0061] It is understandable that the selection of multiple source images corresponding to the reference image can follow two criteria: image overlap and image baseline distance. Image overlap refers to the proportion of the same area observed in the reference image and the source image in the reference image. Under normal circumstances, the higher the image overlap, the larger the area that can be used to estimate the effective depth. However, in a multi-view scenario, the higher the image overlap, the smaller the image baseline distance between the reference image and the source image may be, resulting in excessive triangulation error, which is not conducive to estimating accurate depth. Therefore, when selecting source images, it is necessary to consider both image coverage and image baseline distance, as shown in formula (1):

[0062]

[0063] in, Represents the collection of filtered source images, T ij represents the set of common view map points between image i and image j, sort↓ represents sorting by the number of common view map points from most to least, θ Tij represents the angle and the set of angles between the common view map points and the optical centers of images i and j, Median represents the median of the angles, and K represents the maximum number of source images screened.

[0064] Therefore, among all associated images that have overlapping perspectives with the reference image, multiple associated images whose image overlap and image baseline distance meet preset conditions are selected as multiple source images, so that the initial depth image of the reference image can be determined using multiple source images, and when the initial depth image of the reference image is depth propagated, the depth information of the reference image in a larger area can be obtained, and it is conducive to estimating the accurate depth information of the reference image.

[0065] See also Figure 3 , Figure 3 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S13 in FIG. In this embodiment, the above step S13 may specifically include the following steps:

[0066] Step S131: For each pixel point on the reference image, depth propagation is performed using the depth information of each sampling point in the local image block on the reference image corresponding to the current pixel point in the initial depth image to obtain preliminary depth information of the current pixel point in the reference image.

[0067] Step S132: Optimizing preliminary depth information of the current pixel in the reference image to obtain target depth information of the current pixel in the reference image.

[0068] Step S133: obtaining depth information of the reference image based on target depth information of each pixel in the reference image.

[0069] For each pixel in the reference image, the sampling points in the local image block corresponding to the current pixel in the reference image are determined. Then, using the depth information of each sampling point in the local image block corresponding to the current pixel in the initial depth image of the reference image, the depth information of each sampling point in the initial depth image of the reference image is propagated to the current pixel, obtaining the preliminary depth information of the current pixel in the reference image. Since depth propagation can propagate approximately correct depth information of the sampling point to the current pixel, if there is an error in the depth information of the sampling point, the error in the depth information propagated to the current pixel will be amplified. Therefore, it is necessary to optimize the propagated depth information to obtain the target depth information of each pixel in the reference image, so as to better restore the detailed structure of the scene and better maintain the integrity of weak texture areas. After obtaining the target depth information of each pixel in the reference image, the depth information of the reference image can be obtained.

[0070] See also Figure 4 , Figure 4 yes Figure 3 Schematic diagram of a flow chart of an embodiment of step S131 in FIG. In this embodiment, the above step S131 may specifically include the following steps:

[0071] Step S1311: Acquire a number of sampling points in the local image block corresponding to the current pixel point.

[0072] Step S1312: Compare the normalized cross-correlation values ​​of the multiple sampling points and the current pixel between the reference image and the source image, and use the depth information of the sampling point with the largest normalized cross-correlation value in the initial depth image as the preliminary depth information of the current pixel in the reference image.

[0073] In a single iteration, in addition to considering the normalized cross-correlation value corresponding to its own depth and normal for each pixel, the normalized cross-correlation value corresponding to the depth and normal of the sampling point must also be considered. By obtaining several sampling points in the local image block corresponding to the current pixel, and then comparing the normalized cross-correlation values ​​of several sampling points and the current pixel between the reference image and the source image, the depth information of the sampling point with the largest corresponding normalized cross-correlation value in the initial depth image is propagated to the current pixel, so that the depth information of the sampling point can be quickly propagated to the current pixel, which can improve the convergence speed of the propagation process.

[0074] See also Figure 5 , Figure 5 yes Figure 4 Schematic diagram of a flow chart of an embodiment of step S1311 in FIG. In this embodiment, the above step S1311 may specifically include the following steps:

[0075] Step S13111: Divide the local image block corresponding to the current pixel into several sampling areas.

[0076] Step S13112: In each sampling area, select at least one pixel point with the largest normalized mutual correlation value as the sampling point.

[0077] By dividing the local image block corresponding to the current pixel into several sampling areas, and then selecting the pixel with the largest normalized mutual correlation value in each sampling area as the sampling point, the depth information of the sampling point with the largest normalized mutual correlation value in each sampling area can be propagated to the current pixel, which can reduce the computational complexity and improve the convergence speed of the propagation process.

[0078] Please combine Figure 6 , Figure 6 It is a structural diagram of an embodiment of a method for obtaining sampling points in the image depth estimation method of the present application. In one embodiment, the plurality of sampling areas include a plurality of strip areas S1 radially extending from the current pixel point O as the center to the periphery of the current pixel point O. The depth propagation scheme in this embodiment is improved from the basic checkerboard propagation scheme; specifically, for the current pixel point O, since there are several sampling candidate points D in each direction, in order to reduce the computational complexity, a sampling candidate point D with the largest normalized cross-correlation value can be selected from the strip area S1 formed in each direction as the sampling point D1 in that direction. In a single iteration, each pixel point will consider not only its own depth and the normalized cross-correlation value of the normal, but also the depth of the pixel point. Figure 6The normalized cross-correlation values ​​of the depth and normal of the 8 sampling points D1 in the image are obtained. It can be understood that by dividing the local image block corresponding to the current pixel point O into a number of strip areas S1 radially extending from the current pixel point O as the center to the periphery of the current pixel point O, the depth information of the sampling point D1 with the largest normalized cross-correlation value in each strip area S1 is propagated to the current pixel point O, and high-confidence points in the distance can be quickly propagated to the current position, which not only improves the convergence speed, but also improves the reconstruction effect of flat and weak-textured areas.

[0079] In other embodiments, the plurality of sampling areas include an annular area S2 centered at the current pixel point O. For finer structures, since the local image block corresponding to the current pixel point O may include most of the background area, that is, Figure 6 In the window shown, the sampling points D1 in the eight directions corresponding to the current pixel O are mostly concentrated in the background area, so it may not be possible to effectively restore the depth of the foreground. Therefore, considering the principle of local spatial consistency, this embodiment can also increase Figure 6 The four neighborhoods of the annular region S2 are sampled, and the two sampling points D1 with the largest normalized cross-correlation values ​​are selected. By dividing the local image block corresponding to the current pixel O into an annular region S2 centered on the current pixel O, and taking into account the principle of local spatial consistency, the depth information of at least one sampling point D1 with the largest normalized cross-correlation value in the annular region S2 is propagated to the current pixel O. This can greatly improve the probability of foreground propagation, thereby enabling better depth estimation of detailed structures.

[0080] See also Figure 7 , Figure 7 yes Figure 3 FIG. 1 is a flow chart of an embodiment of step S132 in FIG. In this embodiment, the above step S132 may specifically include the following steps:

[0081] Step S1321: perturb the depth and normal of the current pixel in the reference image.

[0082] Step S1322: Compare the normalized cross-correlation value before the disturbance with the normalized cross-correlation value after the disturbance.

[0083] Step S1323: In response to the normalized cross-correlation value after the disturbance being better than the normalized cross-correlation value before the disturbance, the depth and normal of the current pixel point in the reference image are replaced with the depth and normal after the disturbance to obtain the target depth information of the current pixel point in the reference image.

[0084] Through depth propagation, the depth information of the sampling point can be propagated roughly and correctly to the current pixel, thereby obtaining the preliminary depth information of the current pixel in the reference image; if there is an error in the sampling point, the error in the depth information propagated to the current pixel will be amplified, so it is necessary to optimize the depth information of the current pixel after propagation. Specifically, by perturbing the depth and normal of the current pixel in the reference image after propagation, such as making a random small local perturbation, and then comparing the normalized cross-correlation value before the perturbation with the normalized cross-correlation value after the perturbation, if the normalized cross-correlation value after the perturbation is better, the depth and normal of the current pixel in the reference image are replaced with the depth and normal after the perturbation to improve the accuracy of the depth estimation. After multiple such perturbations, the target depth information of the current pixel in the reference image with higher accuracy can be obtained.

[0085] See also Figure 8 , Figure 8 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S14 in FIG. In this embodiment, the above step S14 may specifically include the following steps:

[0086] Step S141: obtaining corresponding reprojection error and projection depth error based on the depth information of the reference image and the depth information of the multiple source images.

[0087] Step S142: For each pixel point, determine whether the corresponding reprojection error and projection depth error meet preset conditions.

[0088] Step S143: In response to the reprojection error and the projection depth error corresponding to the current pixel point satisfying a preset condition, determining that the depth information of the current pixel point is valid.

[0089] Step S144: In response to the reprojection error and the projection depth error corresponding to the current pixel point not satisfying a preset condition, determining that the depth information of the current pixel point is invalid.

[0090] Therefore, after obtaining the depth information of the reference image, the corresponding reprojection error and projection depth error can be obtained based on the depth information of the reference image and the depth information of multiple source images. Then, for each pixel point, it can be determined whether the corresponding reprojection error and projection depth error meet the preset conditions, and when the reprojection error and projection depth error corresponding to the current pixel point meet the preset conditions, it is determined that the depth information of the current pixel point is valid. When the reprojection error and projection depth error corresponding to the current pixel point do not meet the preset conditions, it is determined that the depth information of the current pixel point is invalid, so that the noise on the depth map of the reference image can be eliminated and the depth estimation result of the reference image can be obtained. The above process is expressed by the following formula:

[0091]

[0092] X′ j =P j P i -1 x i d(x i ) (3)

[0093]

[0094] Among them, P i and P j are the projection matrices of image I and image J, respectively, x i and x j They are the points on the image I and their depth d(x j ) projected onto the source image J, X j ′ is the three-dimensional vector corresponding to the projection point on image J, where X j ′(0),X j ′(1) and X j ′(2) are respectively X j ' components on the x-axis, y-axis, and z-axis, τ is the preset threshold of the reprojection error, and β is the preset threshold of the projected depth error. In one embodiment, τ can be set to 4 and β can be set to 0.01. Therefore, the preset conditions are that the reprojection error corresponding to the current pixel is less than or equal to τ, and the projected depth error is less than or equal to β. For each pixel, when the reprojection error and the projected depth error meet the preset conditions, it can be determined that the depth information of the current pixel is valid.

[0095] The image depth estimation method of the present application can implement depth consistency verification on the GPU (graphics processing unit) to make the process more efficient. Since the video memory of a single graphics card is limited, it is impossible to load all images into the video memory at once, so a block loading strategy can be adopted. Specifically, since all images have corresponding camera poses, the center point of all camera poses is first calculated, the image farthest from the center point is selected, the source image and reference image associated with it are loaded into the video memory, and then the correlation score s is calculated for all remaining images, where, express The GPU-located subset is selected; the image with the highest correlation score s and the source image calculated using it as a reference image are then loaded into the video memory until no more can be loaded. This approach fully utilizes the machine's memory and video memory in the processing flow, combined with multi-threaded parallel technology, fully utilizing the GPU's multi-core advantages and computing power, greatly improving runtime efficiency.

[0096] See also Figure 9 , Figure 9Schematic diagram of a framework of an embodiment of a scene display device of the present application. In this embodiment, an image depth estimation device 90 includes: an acquisition module 900 for acquiring a reference image and a plurality of corresponding source images, wherein the source images and the reference image have overlapping viewing angles; an initialization module 901 for initializing a depth map of the reference image to determine an initial depth map of the reference image; and a solution module 902 for performing depth propagation on the initial depth map of the reference image based on the plurality of source images to obtain depth information of the reference image.

[0097] In the above scheme, the acquisition module 900 acquires a reference image with overlapping perspectives and corresponding multiple source images. After the initialization module 901 initializes the depth map of the reference image, the initialization image of the reference image can be determined. The solution module 902 can use multiple source images to perform depth propagation on the initial depth image of the reference image to obtain the depth information of the reference image. By performing depth propagation on the initial depth image of the reference image, the depth information of the reference image obtained can better process the detail structure, which is beneficial to improving the reconstruction effect of the detail structure.

[0098] In some embodiments, the image depth estimation apparatus 90 further includes: a verification module 904 for performing a depth consistency check on the depth information of the reference image based on the reference image and the multiple source images to obtain a depth estimation result of the reference image.

[0099] In some embodiments, the acquisition module 900 performs the steps of acquiring a reference image and corresponding multiple source images, specifically including: acquiring multiple associated images that have overlapping perspectives with the reference image; acquiring the image overlap and image baseline distance between each of the associated images and the reference image; and selecting a preset number of associated images whose image overlap and image baseline distance meet preset conditions as the multiple source images.

[0100] In some embodiments, the initialization module 901 performs the step of initializing the depth map of the reference image and determining the initial depth image of the reference image, specifically including: randomly assigning depth and normal to all pixel points in the reference image whose depth information is unknown to form the initial depth image of the reference image.

[0101] In some embodiments, the solution module 902 performs a step of performing depth propagation on the initial depth image of the reference image based on the multiple source images to obtain the depth information of the reference image, specifically including: for each pixel point on the reference image, using the depth information of each sampling point in the local image block on the reference image corresponding to the current pixel point to perform depth propagation in the initial depth image to obtain the preliminary depth information of the current pixel point in the reference image; optimizing the preliminary depth information of the current pixel point in the reference image to obtain the target depth information of the current pixel point in the reference image; and obtaining the depth information of the reference image based on the target depth information of each pixel point in the reference image.

[0102] In some embodiments, the solution module 902 performs a step of performing depth propagation using the depth information of each sampling point in the local image block on the reference image corresponding to the current pixel point to obtain preliminary depth information of the current pixel point in the reference image. Specifically, the step may include: obtaining a number of sampling points in the local image block corresponding to the current pixel point; comparing the normalized mutual correlation values ​​of the number of sampling points and the current pixel point between the reference image and the source image, and taking the depth information of the sampling point with the largest normalized mutual correlation value in the initial depth image as the preliminary depth information of the current pixel point in the reference image.

[0103] In some embodiments, the solving module 902 executes the step of obtaining a plurality of sampling points in the local image block corresponding to the current pixel point, which may specifically include: dividing a plurality of sampling areas in the local image block corresponding to the current pixel point; and selecting at least one pixel point with the largest normalized mutual correlation value in each sampling area as the sampling point.

[0104] In some embodiments, the plurality of sampling areas include a plurality of strip areas radially extending from the current pixel point to the periphery of the current pixel point, and / or a ring area centered on the current pixel point.

[0105] In some embodiments, the solution module 902 performs the step of optimizing the preliminary depth information of the current pixel point in the reference image to obtain the target depth information of the current pixel point in the reference image, which may specifically include: perturbing the depth and normal of the current pixel point in the reference image; comparing the normalized mutual correlation value before the perturbation with the normalized mutual correlation value after the perturbation; in response to the normalized mutual correlation value after the perturbation being better than the normalized mutual correlation value before the perturbation, replacing the depth and normal of the current pixel point in the reference image with the depth and normal after the perturbation to obtain the target depth information of the current pixel point in the reference image.

[0106] In some embodiments, the verification module 904 performs a step of performing a depth consistency check on the depth information of the reference image based on the reference image and the multiple source images to obtain a depth estimation result of the reference image, which may specifically include: obtaining a corresponding reprojection error and a projection depth error based on the depth information of the reference image and the depth information of the multiple source images; for each pixel point, determining whether the corresponding reprojection error and projection depth error meet preset conditions; in response to the reprojection error and projection depth error corresponding to the current pixel point meeting the preset conditions, determining that the depth information of the current pixel point is valid; in response to the reprojection error and projection depth error corresponding to the current pixel point not meeting the preset conditions, determining that the depth information of the current pixel point is invalid.

[0107] See also Figure 10 , Figure 10 This is a schematic diagram of the framework of an embodiment of an electronic device of the present application. Electronic device 100 includes a memory 101 and a processor 102 coupled to each other. Processor 102 is configured to execute program instructions stored in memory 101 to implement the steps of any of the above-described image depth estimation method embodiments. In a specific implementation scenario, electronic device 100 may include, but is not limited to, a microcomputer and a server.

[0108] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned image depth estimation method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0109] In the above scheme, the processor 102 obtains a reference image with overlapping perspectives and corresponding multiple source images. After initializing the depth map of the reference image, it can use the multiple source images to perform depth propagation on the initial depth image of the reference image to obtain the depth information of the reference image. By performing depth propagation on the initial depth image of the reference image, the depth information of the reference image obtained can better process the detail structure, which is conducive to improving the reconstruction effect of the detail structure.

[0110] See also Figure 11 , Figure 11 The computer-readable storage medium 110 stores program instructions 1100 that can be executed by a processor, and the program instructions 1100 are used to implement the steps of any of the above-mentioned image depth estimation method embodiments.

[0111] The present disclosure relates to the field of augmented reality, which obtains image information of target objects in a real environment, and then uses various vision-related algorithms to detect or identify the relevant features, states and attributes of the target objects, thereby obtaining an AR effect that combines virtuality and reality and matches the specific application. For example, the target object may involve faces, limbs, gestures, movements, etc. related to the human body, or markers, landmarks related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and object posture or depth detection. Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstruction, virtual effect overlay displays, etc. related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effects display, virtual model display and other interactive scenes.

[0112] Convolutional neural networks can be used to detect or identify the relevant features, states, and attributes of target objects. The above-mentioned convolutional neural networks are network models obtained by model training based on deep learning frameworks.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0114] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A method for estimating image depth, characterized in that: The method comprises: Acquire a reference image and a plurality of corresponding source images, wherein the source images and the reference images have overlapping viewing angles; Initializing a depth map of the reference image to determine an initial depth map of the reference image; Performing depth propagation on the initial depth image of the reference image based on the multiple source images to obtain depth information of the reference image specifically includes: For each pixel on the reference image, depth propagation is performed using depth information of each sampling point in the local image block on the reference image corresponding to the current pixel in the initial depth image to obtain preliminary depth information of the current pixel in the reference image, wherein the sampling point is determined in the following manner: a plurality of sampling regions are divided in the local image block corresponding to the current pixel; normalized cross-correlation values ​​between the plurality of sampling points and the current pixel in the reference image and the source image are compared; and within each sampling region, at least one pixel having the largest normalized cross-correlation value is selected as the sampling point; Optimizing preliminary depth information of the current pixel in the reference image to obtain target depth information of the current pixel in the reference image; Depth information of the reference image is obtained based on target depth information of each pixel in the reference image.

2. The image depth estimation method according to claim 1, wherein: The obtaining of a reference image and a plurality of corresponding source images includes: Acquire a plurality of associated images having overlapping viewing angles with the reference image; Obtaining an image overlap and an image baseline distance between each of the associated images and the reference image; A preset number of associated images whose image overlap and image baseline distance meet preset conditions are selected as the multiple source images.

3. The image depth estimation method according to claim 1, wherein: Initializing the depth map of the reference image to determine the initial depth image of the reference image includes: Depths and normals are randomly assigned to all pixels in the reference image whose depth information is unknown, to form an initial depth image of the reference image.

4. The image depth estimation method according to claim 1, wherein: The performing depth propagation using the depth information of each sampling point in the local image block on the reference image corresponding to the current pixel in the initial depth image to obtain preliminary depth information of the current pixel in the reference image includes: The depth information of the sampling point with the largest corresponding normalized mutual correlation value in the initial depth image is used as the preliminary depth information of the current pixel in the reference image.

5. The image depth estimation method according to claim 4, characterized in that The plurality of sampling areas include a plurality of strip areas radially extending from the current pixel point to the periphery of the current pixel point, and / or a ring area centered on the current pixel point.

6. The image depth estimation method according to claim 1, wherein: The optimizing the preliminary depth information of the current pixel in the reference image to obtain the target depth information of the current pixel in the reference image includes: Perturbing the depth and normal of the current pixel in the reference image; Compare the normalized cross-correlation value before the perturbation with the normalized cross-correlation value after the perturbation; In response to the normalized cross-correlation value after the disturbance being better than the normalized cross-correlation value before the disturbance, the depth and normal of the current pixel point in the reference image are replaced with the depth and normal after the disturbance to obtain the target depth information of the current pixel point in the reference image.

7. The image depth estimation method according to claim 1, wherein: After performing depth propagation on the initial depth image of the reference image based on the multiple source images to obtain depth information of the reference image, the method further includes: Based on the reference image and the multiple source images, a depth consistency check is performed on the depth information of the reference image to obtain a depth estimation result of the reference image.

8. The image depth estimation method according to claim 7, characterized in that: The performing a depth consistency check on the depth information of the reference image based on the reference image and the multiple source images to obtain a depth estimation result of the reference image includes: Obtaining corresponding reprojection error and projection depth error based on the depth information of the reference image and the depth information of the plurality of source images; For each pixel point, determine whether the corresponding reprojection error and projection depth error meet the preset conditions; In response to a reprojection error and a projection depth error corresponding to the current pixel point satisfying a preset condition, determining that the depth information of the current pixel point is valid; In response to the reprojection error and the projection depth error corresponding to the current pixel point not satisfying a preset condition, it is determined that the depth information of the current pixel point is invalid.

9. An image depth estimation device, characterized in that: include: an acquisition module, the acquisition module being configured to acquire a reference image and a plurality of corresponding source images, wherein the source images and the reference images have overlapping viewing angles; An initialization module, configured to initialize a depth map of the reference image and determine an initial depth map of the reference image; A solution module, configured to perform depth propagation on the initial depth image of the reference image based on the multiple source images to obtain depth information of the reference image, specifically comprising: For each pixel on the reference image, depth propagation is performed using depth information of each sampling point in the local image block on the reference image corresponding to the current pixel in the initial depth image to obtain preliminary depth information of the current pixel in the reference image, wherein the sampling point is determined in the following manner: a plurality of sampling regions are divided in the local image block corresponding to the current pixel; normalized cross-correlation values ​​between the plurality of sampling points and the current pixel in the reference image and the source image are compared; and within each sampling region, at least one pixel having the largest normalized cross-correlation value is selected as the sampling point; Optimizing preliminary depth information of the current pixel in the reference image to obtain target depth information of the current pixel in the reference image; Depth information of the reference image is obtained based on target depth information of each pixel in the reference image.

10. An electronic device, characterized in that: The method comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the image depth estimation method according to any one of claims 1 to 8.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the image depth estimation method according to any one of claims 1 to 8 is implemented.

Images