Image depth prediction method and electronic device

Abstract

The application relates to the technical field of image processing, in particular to an image depth prediction method and an electronic device. The method comprises the following steps: acquiring a main view and a first auxiliary view, the main view comprising a first pixel, the first pixel corresponding to a second pixel under a first assumed depth, and the second pixel being located on the first auxiliary view; updating feature information of the second pixel according to the feature information of the second pixel and feature information of at least one third pixel; a fourth pixel is a pixel corresponding to the first pixel on the first auxiliary view under a second assumed depth; and obtaining a probability of the first assumed depth according to the feature information of the first pixel and the updated feature information of the second pixel. In the method, the features of the pixels between adjacent sampling points can be converged into the features of the nearby sampling points, and the possibility that the corresponding assumed depth is the real depth of the main view pixel is determined according to the features, so that the accuracy of depth prediction is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image depth prediction method and electronic device. Background Technology

[0002] Three-dimensional (3D) image reconstruction technology has wide applications, such as geological surveying, mapping, augmented reality (AR), and virtual reality (VR). In particular, the development of AR technology, with mobile devices like smartphones now possessing AR capabilities, has led to the emergence of AR-based applications (apps) for environmental interaction, information delivery, and virtual entertainment. This, in turn, has created even greater demand for 3D reconstruction technology.

[0003] Currently, 3D reconstruction technology can be mainly divided into the following three types.

[0004] a) Laser acquisition has the best reconstruction effect and can directly output point cloud results, but the acquisition equipment is expensive and the acquisition process is difficult, making it unsuitable for ordinary consumers.

[0005] b. Depth camera acquisition, which can directly acquire depth information of the scene being photographed, and is relatively cheaper than laser acquisition.

[0006] However, the 3D reconstruction results are generally poor.

[0007] c. Combining RGB images with dense reconstruction (multi-view stereo matching, MVS) algorithms is cost-effective, and its reconstruction results are second only to laser acquisition. Therefore, it has comprehensive advantages over laser acquisition and depth camera acquisition. In particular, dense reconstruction algorithms based on deep learning have high accuracy and computational speed, and have been widely adopted.

[0008] Dense reconstruction algorithms consist of two parts: depth prediction and point cloud fusion. The image depth predicted by depth prediction directly determines the accuracy of the point cloud. The principle of depth prediction is to set multiple hypothetical depths and, through univariate variation, find the sampling points of the main view pixels in the auxiliary view at each hypothetical depth. Then, the similarity between the main view pixels and the sampling points at each hypothetical depth is calculated, thereby determining the probability that each hypothetical depth is the true depth of the main view pixels. The hypothetical depth with the highest probability is taken as the prediction result. Therefore, how to involve the pixels corresponding to the main view pixels at the true depth in the auxiliary view in depth prediction is of great significance for improving the accuracy of depth prediction. Summary of the Invention

[0009] This application provides an image depth prediction method and electronic device to improve the effect of depth prediction.

[0010] In a first aspect, embodiments of this application provide an image depth prediction method, comprising: acquiring a main view and a first auxiliary view, wherein the main view and the first auxiliary view are images captured by a camera facing a first object at different angles, the main view includes a first pixel, the first pixel corresponds to a second pixel at a first assumed depth, and the second pixel is located on the first auxiliary view; updating the feature information of the second pixel according to feature information of the second pixel and feature information of at least one third pixel; at least one third pixel is located on a line connecting the second pixel and a fourth pixel, the fourth pixel is the pixel corresponding to the first pixel at the second assumed depth on the first auxiliary view, the second assumed depth and the first assumed depth are adjacent in a preset D assumed depths, where D is a positive integer greater than 1; obtaining the probability of the first assumed depth according to the feature information of the first pixel and the updated feature information of the second pixel, the probability of the first assumed depth being the true depth of the first pixel.

[0011] In this method, the features of pixels between adjacent sampling points can be aggregated into the features of nearby sampling points, and the probability that the corresponding assumed depth is the true depth of the main view pixel can be determined accordingly. This allows the pixel features corresponding to the true depth to participate in depth prediction with a higher probability, thereby improving the accuracy of depth prediction.

[0012] Homography can be used to determine the second pixel corresponding to the first pixel at a first assumed depth. In homography, the coordinates of the first pixel in the main view are converted to the coordinates of the object point in the camera coordinate system at the assumed depth. Then, the coordinates of the object point in the camera coordinate system are converted to pixel coordinates in the auxiliary view, and the pixel corresponding to these coordinates is the second pixel. The specific implementation process of homography can be found in existing technologies and will not be elaborated here.

[0013] Assumed depth refers to the assumed depth for pixels during depth prediction. It can be a depth set manually based on camera parameters from the main view and auxiliary views.

[0014] True depth refers to the actual depth of a pixel, which is specifically the distance between the object point corresponding to that pixel and the plane recording the image.

[0015] The line connecting the second and fourth pixels refers to a line segment with the second pixel as one endpoint and the fourth pixel as the other endpoint. In other words, the line segment has the second pixel and the fourth pixel as endpoints.

[0016] In one possible implementation, the method further includes: updating the feature information of the fourth pixel based on the feature information of the fourth pixel and the feature information of at least one fifth pixel; the at least one fifth pixel is located on the line connecting the second pixel and the fourth pixel; and obtaining the probability of a second hypothetical depth based on the feature information of the first pixel and the updated feature information of the fourth pixel, the probability of the second hypothetical depth being used to represent the possibility that the second hypothetical depth is the true depth of the first pixel.

[0017] In one possible implementation, the at least fifth pixel and the at least third pixel are not the same, and the at least fifth pixel is closer to the fourth pixel than the at least third pixel; or, a pixel contained in the at least fifth pixel but not contained in the at least third pixel is closer to the fourth pixel than a pixel contained in the at least third pixel but not contained in the at least fifth pixel.

[0018] In other words, in this implementation, the features of pixels outside the epipolar sampling point can be aggregated into the features of the nearest sampling point to participate in depth prediction, thereby avoiding or reducing misjudgments in image depth prediction.

[0019] In one possible implementation, the union of the at least fifth pixel and the at least third pixel includes all pixels between the second pixel and the third pixel.

[0020] In other words, all pixels between adjacent sampling points can participate in depth prediction, thus ensuring that the pixel features corresponding to the true depth can participate in depth prediction and improving the accuracy of depth prediction.

[0021] In one possible implementation, the method further includes: reducing the size of the value interval corresponding to the D hypothetical depths to obtain the size of a first interval, which is used as the value interval of the hypothetical depth when the first pixel depth is predicted again. Here, the value interval corresponding to the D hypothetical depths refers to an interval including the depth values ​​of the D hypothetical depths; taking D depth values ​​within this value interval yields the D hypothetical depths.

[0022] The predicted depth, determined by the probabilities of the D hypothetical depths, can be used as the median of the first interval. This allows for the determination of an interval with specific upper and lower limits, enabling further prediction of the depth of the first pixel. In one example, the hypothetical depth with the highest probability among the D hypothetical depths can be used as the predicted depth. In another example, each of the D hypothetical depths can be multiplied by its probability to obtain a depth value. The D hypothetical depths yield D depth values, which are then summed to obtain the predicted depth.

[0023] In other words, in this implementation, the range of values ​​for the assumed depth can be narrowed, and the predicted depth can be used as the median of the narrowed range. This can improve the accuracy of depth prediction while reducing computational complexity.

[0024] In one possible implementation, the magnitude of the narrowing of the value range corresponding to the D hypothetical depths is negatively correlated with the degree of dispersion between the probabilities of the D hypothetical depths.

[0025] It is understandable that the higher the dispersion of multiple probabilities, the higher the uncertainty of whether the assumed depth is or is not the true depth of the pixel. In this case, slightly narrowing the range of values ​​for the assumed depth can make the narrowed range include the true depth with a high probability.

[0026] In one possible implementation, the magnitude of the narrowing of the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths. In informatics, entropy (which can be simply called entropy) is used to represent the degree of dispersion among multiple probabilities (probabilities can also be called probability values), reflecting the uncertainty of a certain indicator. The larger the entropy value, the greater the uncertainty, and the greater the amount of information used to describe this uncertainty.

[0027] In this implementation, the entropy value reflects the uncertainty of the D hypothetical depths being the true depths of pixels, and also reflects the encoding length required for all possible states at the D hypothetical depths. The distribution of the D first probabilities can be represented by the entropy value; that is, the entropy value can relatively accurately reflect the concentration of the distribution of the D first probabilities. Therefore, by using the entropy value to represent the dispersion of the D first probabilities, the extent of interval narrowing can be determined, further avoiding the exclusion of the true depth from the value range corresponding to the hypothetical depth.

[0028] In one possible implementation, the method further includes: reducing the size of the value intervals corresponding to the D′ hypothetical depths to obtain the size of the second interval, the second interval being used as the value interval of the hypothetical depth when the sixth pixel depth is predicted again; the reduction of the value intervals corresponding to the D′ hypothetical depths is equal to the reduction of the value intervals corresponding to the D hypothetical depths; the sixth pixel is a pixel on the main view and is different from the first pixel; D′ is a positive integer greater than 1.

[0029] The range of values ​​corresponding to the D' hypothetical depths refers to the range of depth values ​​that include the D' hypothetical depths. By taking D' depth values ​​within this range, the D' hypothetical depths are obtained.

[0030] The predicted depth, determined by the probabilities of the D′ hypothetical depths, can be used as the median of the second interval. This allows for the determination of an interval with specific upper and lower limits, enabling the prediction of the first pixel's depth again. In one example, the hypothetical depth with the highest probability among the D′ hypothetical depths can be used as the predicted depth. In another example, each of the D′ hypothetical depths can be multiplied by its probability to obtain a depth value. D′ hypothetical depths yield D′ depth values, which are then summed to obtain the predicted depth.

[0031] It is understandable that the difference between the actual depths of different pixels on the main view is fixed. By reducing the size of the assumed depth intervals corresponding to different pixels by the same reduction factor, the accuracy of pixel depth prediction on the main view can be improved.

[0032] In one possible implementation, the magnitude of the narrowing of the value intervals corresponding to the D hypothetical depths is negatively correlated with the degree of dispersion of the probabilities of the D hypothetical depths, and the degree of dispersion of the probabilities of the D' hypothetical depths is also negatively correlated.

[0033] In this implementation, the reduction range is constrained by the degree of dispersion of different pixels on the main view. That is, when setting the reduction range, the degree of dispersion of different pixels on the main view can be taken into account, which can improve the rationality of the reduction range.

[0034] In one possible implementation, the magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths and negatively correlated with the entropy value of the sixth pixel at the D' hypothetical depths.

[0035] Secondly, embodiments of this application provide an image depth prediction device, comprising: an acquisition unit, configured to acquire a main view and a first auxiliary view, wherein the main view and the first auxiliary view are images captured by a camera facing a first object at different angles, the main view including a first pixel, the first pixel corresponding to a second pixel at a first assumed depth, and the second pixel located on the first auxiliary view; an update unit, configured to update the feature information of the second pixel according to feature information of the second pixel and feature information of at least one third pixel; the at least one third pixel is located on a line connecting the second pixel and a fourth pixel, the fourth pixel being the pixel corresponding to the first pixel on the first auxiliary view at the second assumed depth, the second assumed depth and the first assumed depth being adjacent in a preset D assumed depths, where D is a positive integer greater than 1; and a obtaining unit, configured to obtain the probability of the first assumed depth according to the feature information of the first pixel and the updated feature information of the second pixel, the probability of the first assumed depth being the true depth of the first pixel.

[0036] In one possible implementation, the updating unit is further configured to update the feature information of the fourth pixel based on the feature information of the fourth pixel and the feature information of at least one fifth pixel; the at least one fifth pixel is located on the line connecting the second pixel and the fourth pixel; the obtaining unit is further configured to obtain the probability of the second assumed depth based on the feature information of the first pixel and the updated feature information of the fourth pixel, the probability of the second assumed depth being used to represent the possibility that the second assumed depth is the true depth of the first pixel.

[0037] In one possible implementation, the at least fifth pixel and the at least third pixel are not the same, and the at least fifth pixel is closer to the fourth pixel than the at least third pixel; or, a pixel contained in the at least fifth pixel but not contained in the at least third pixel is closer to the fourth pixel than a pixel contained in the at least third pixel but not contained in the at least fifth pixel.

[0038] In one possible implementation, the union of the at least fifth pixel and the at least third pixel includes all pixels on the line connecting the second pixel and the fourth pixel.

[0039] In one possible implementation, the apparatus further includes a reduction unit for reducing the size of the value intervals corresponding to the D assumed depths to obtain the size of a first interval, which is used as the value interval of the assumed depth when the first pixel depth is predicted again.

[0040] In one possible implementation, the magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths.

[0041] In one possible implementation, the reduction unit is further used to reduce the size of the value intervals corresponding to the D′ hypothetical depths to obtain the size of the second interval, which is used as the value interval of the hypothetical depth when the sixth pixel depth is predicted again; the reduction magnitude of the value intervals corresponding to the D′ hypothetical depths is equal to the reduction magnitude of the value intervals corresponding to the D hypothetical depths; the sixth pixel is a pixel on the main view and is different from the first pixel; D′ is a positive integer greater than 1.

[0042] In one possible implementation, the magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths and negatively correlated with the entropy value of the sixth pixel at the D′ hypothetical depths.

[0043] Thirdly, embodiments of this application also provide an electronic device, including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the method provided in the first aspect.

[0044] Fourthly, according to an embodiment of this application, a computer storage medium stores a computer program that, when executed by a processor, implements the method provided in the first aspect.

[0045] In a sixth aspect, embodiments of this application provide a computer program product, including a program for implementing the method as provided in the first aspect.

[0046] The image depth prediction method and electronic device provided in this application can aggregate the features of pixels that are not sampling points on the epipolar line into the features of nearby sampling points. This allows for depth prediction using the features of more pixels on the epipolar line without setting a larger number of assumed depths, thereby improving the accuracy of depth prediction. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a three-dimensional reconstruction process;

[0048] Figure 2 A diagram illustrating an image depth prediction architecture provided in this application embodiment;

[0049] Figure 3 A schematic diagram illustrating the mapping relationship between the main view pixels and the auxiliary view under different assumed depths, provided for embodiments of this application;

[0050] Figure 4A This is a schematic diagram of the spacing between adjacent sampling points provided in an embodiment of this application;

[0051] Figure 4B This is a schematic diagram of the spacing between adjacent sampling points provided in an embodiment of this application;

[0052] Figure 4C This is a schematic diagram of the spacing between adjacent sampling points provided in an embodiment of this application;

[0053] Figure 5 A flowchart for determining the reduction range is provided in an embodiment of this application;

[0054] Figure 6 A schematic diagram of a neural network structure provided in an embodiment of this application;

[0055] Figure 7 A flowchart of an image depth prediction method provided in an embodiment of this application;

[0056] Figure 8A schematic diagram of a neural network structure provided in an embodiment of this application;

[0057] Figure 9 A flowchart of an image depth prediction method provided in an embodiment of this application;

[0058] Figure 10 A schematic block diagram of an image depth prediction device provided in an embodiment of this application;

[0059] Figure 11 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the embodiments described in this specification are only some embodiments of this application, and not all embodiments.

[0061] Dense reconstruction, also known as multi-view matching, belongs to the field of multi-view solid geometry. It is used to calculate, pixel by pixel, the object points corresponding to or represented by each pixel in an image, given the camera pose (its orientation and position in 3D space), thus obtaining a 3D point cloud of the object's surface. For more details, please refer to [link to relevant documentation]. Figure 1 In dense reconstruction, depth prediction can be performed based on RGB images to obtain a depth map, and each pixel on the depth map can be converted into a point cloud. The point cloud can be converted into a mesh to further refine the image's detailed information, resulting in a white model, thereby completing 3D reconstruction. The white model is a type of 3D model that can be used for surveying, AR, VR, image semantic recognition, etc.

[0062] In this context, the object point corresponding to a pixel refers to the 3D point in the real world that corresponds to that pixel. It can be understood that an image is a representation of objects in the real world, and a pixel is the representation of a point (i.e., an object point) on that object. In dense reconstruction techniques, depth prediction refers to predicting the depth of an image using camera parameters and information from the RGB image itself. A depth map is an image with pixel depth. Pixel depth is used to represent the vertical distance between the plane recording the image and the object represented by that pixel. For example, the plane recording the image can be the focal plane of the camera. The focal plane is the plane passing through the camera's focal point and perpendicular to the camera's optical axis. A point cloud, or 3D point cloud, refers to a dataset of points in a certain coordinate system, where each point contains information such as 3D coordinates (X, Y, Z), color, classification value, intensity value, and time. A white model refers to a 3D model obtained by further simplifying the image details after converting the point cloud into a mesh.

[0063] For dense reconstruction, depth prediction can be done in two ways: single-stage depth prediction and multi-stage depth prediction.

[0064] The basic process of single-stage depth prediction is as follows: Based on camera parameters (camera extrinsic and intrinsic matrices, etc.), the camera geometric relationship between the main view and the auxiliary view is determined. Then, using this geometric positional relationship, the pixel mapping relationship between the main view and the auxiliary view is determined, and a matching cost volume is constructed accordingly to complete the depth prediction. The specific process may include the following steps.

[0065] 1) Feature extraction:

[0066] Two-dimensional convolutional neural networks (CNNs) can be used to extract features from multiple acquired images to obtain high-dimensional semantic information. Specifically, feature information of individual pixels in the image can also be obtained.

[0067] 2) Pixel matching:

[0068] The images from the multiple images for which depth prediction is required are designated as the primary view (reference view), and the images from the multiple images with the highest correlation to the primary view are designated as secondary views (source views). The primary view and secondary views are captured from different camera angles. In other words, the primary view and secondary views are images taken with the camera facing the object from different angles.

[0069] Set D hypothetical depths within the depth value range R provided by the main view camera parameters. Hypothetical depths refer to the depths assumed for pixels, which can be manually set, guessed, or estimated.

[0070] By utilizing the camera geometry between the main view and the auxiliary views, all pixels in the main view are projected onto the auxiliary views through D assumed depths, resulting in sampling points (i.e., the pixels in the auxiliary view corresponding to a pixel in the main view at the assumed depth). Thus, each pixel in the main view will have D sampling points in each auxiliary view. Based on the geometric relationship, the D sampling points corresponding to the same pixel in the main view will be distributed along the same epipolar line. The epipolar line is the intersection of the epipolar plane formed by the lines connecting the object point to the camera center (focus) of the main view and the camera center (focus) of the auxiliary view, and the auxiliary view itself.

[0071] Next, for a single pixel in the main view, its feature matching relationship (cost) or similarity with D sampling points can be calculated, resulting in a sequence of D relationships or D similarities. The D similarities of a single pixel constitute its matching cost volume, and the set of matching cost volumes for all pixels in the main view is called the main view's matching cost volume. When there are multiple auxiliary views, the main view and each auxiliary view form a main view matching cost volume. After aggregating the main view matching cost volumes corresponding to different auxiliary views using methods such as weighted summation, the final main view matching cost volume is obtained. This matching cost volume represents the depth matching relationship between the main view and all auxiliary views.

[0072] 3) Regularization:

[0073] In step 2), the numerical distribution of matching costs among different pixels in the main view matching cost values ​​shows significant differences, and it only represents matching relationships between pixels. Typically, a 3D convolutional neural network (3D CNN) can be used to extract information and smooth the main view matching cost values; this process is called regularization.

[0074] 4) Deep prediction:

[0075] Step 3) The matching cost of the main view after regularization can be calculated using the softmax activation function to obtain the predicted probability values ​​of the main view pixels at the preset D hypothetical depths. The set of predicted probability values ​​at the D hypothetical depths can be called the probability volume, which represents the normalized matching probability value of each pixel in the main view at different hypothetical depths. The predicted depth of each pixel in the main view is obtained by taking the depth corresponding to the maximum probability value or by calculating the expected depth based on the probability distribution, thus obtaining the depth image or depth prediction map of the main view.

[0076] In step 2) of the single-stage depth prediction, the number of assumed depths, D, directly affects the depth prediction accuracy. A larger value for D results in smaller intervals between adjacent depth assumption planes and finer granularity of the depth preset, thus leading to higher accuracy. Conversely, a larger interval between adjacent depth assumption planes results in coarser granularity of the depth preset, thus reducing accuracy. However, a larger number of assumed depths, D, directly leads to a linear increase in the shape and size of the matching cost. Furthermore, since the regularization uses a 3DCNN, its computational cost and time consumption also increase significantly.

[0077] Therefore, a multi-stage depth prediction method is proposed, which introduces a coarse-to-fine mechanism based on single-stage depth prediction. It uses an image pyramid feature extraction network to obtain feature maps of different resolutions. First, a small number of depth hypotheses D1 are preset within the hypothetical depth value range provided by the camera parameters on the low-resolution feature map for coarse-grained depth prediction. Then, steps 2, 3, and 4 of the single-stage depth prediction are repeated. Next, using the depth predicted in the previous stage as the center, an even smaller number of hypothetical depths D_2 are selected, supplemented by a smaller depth hypothesis interval, to construct the hypothetical depth for each pixel for depth prediction. This process is repeated. Since the depth prediction in subsequent stages is based on the depth predicted in the previous stage, reducing the number of hypothetical depths D while narrowing the hypothetical depth value range and the hypothetical depth interval reduces the prediction difficulty in subsequent stages and continuously improves the depth prediction accuracy, achieving better depth prediction results. The hypothetical depth value range, also called the depth hypothesis interval, is a length range within which multiple hypothetical depths with different depth values ​​can be selected.

[0078] As discussed above, the similarity between pixels in the main view and the sampling points mapped to those pixels in the auxiliary view is calculated, and the probability that the assumed depth corresponds to the true depth of the main view pixel is determined based on this similarity. However, since the assumed depths corresponding to each sampling point are hypothetical, these hypothetical depths may not be the true depths of the main view pixels, meaning that each sampling point may not actually correspond to a pixel in the main view. Therefore, using the similarity between these sampling points and the main view pixels to predict depth may not accurately predict the true depth of the main view pixels, or the prediction effect may be poor.

[0079] Furthermore, for multi-stage depth prediction, the assumed depth range used in subsequent stages depends on the prediction results of the previous stage. That is, the assumed depth range is adjusted based on the previous stage's prediction results before being used in subsequent stages. Currently, a fixed depth reduction rate is used to adjust the assumed depth range. This can lead to situations where the range reduction is too drastic, resulting in a range that doesn't contain the true depth, or too conservative, leading to excessive computational time. Therefore, for multi-stage depth prediction, it's necessary to use different reduction rates to narrow the assumed depth range depending on the specific circumstances.

[0080] To address the above situation, this application embodiment improves step 2) by providing an image depth prediction method. In this method, a pixel A1 in the main view A can be set to correspond to pixel P1 in the auxiliary view B under an assumed depth D1 (i.e., the distance between the object point a1 corresponding to pixel A1 and the image recording plane of the main view is assumed depth D1). The image recording plane can be the focal plane of the camera. Pixel P1 can be considered the sampling point of pixel A1 in the auxiliary view B under the constraint of assumed depth D1. Features of one or more pixels close to pixel P1 on the epipolar line can be aggregated into the features of pixel P1 to obtain new features of pixel P1. In other words, pooling operations can be performed around pixel P1 to gather more pixel information on the epipolar line and obtain new features of pixel P1. Then, based on the new features of pixel P1 and the features of pixel A1, the similarity or matching relationship between pixel P1 and pixel A1 is determined, and the probability that the assumed depth D1 is the true depth of pixel A1 is determined accordingly, which can improve the probability of depth prediction.

[0081] Next, in different embodiments, the image depth prediction method provided in the embodiments of this application will be illustrated by examples.

[0082] The image depth prediction method provided in this application embodiment can be executed by an electronic device. This electronic device can be any apparatus, device, platform, or cluster of devices with computing and processing capabilities. For example, the electronic device can be user equipment (UE), which can be mobile or fixed. Examples include mobile phones with image processing capabilities, tablet personal computers (TPCs), media players, smart TVs, laptop computers (LCs), personal digital assistants (PDAs), personal computers (PCs), cameras, camcorders, smartwatches, wearable devices (WDs), etc. This application embodiment does not limit the scope of the method.

[0083] Figure 2This application illustrates an image depth prediction architecture provided by an embodiment of the present application. The architecture may include a feature extraction module, which can acquire multiple images and camera parameters for each of the multiple images. The different images are captured by cameras facing scene objects from different angles. The cameras used to capture the different images may be the same or different. The multiple images include a preset main view, i.e., the image whose depth needs to be predicted. For example, the main view may be image A. When image A is the main view, it can be referred to as main view A. Images other than the main view are used as auxiliary views. A main view may have one or more auxiliary views.

[0084] The feature extraction module can extract features from each pixel in an image. For example, the feature extraction module may include one or more two-dimensional convolutional layers. By using the convolution kernels in these one or more two-dimensional convolutional layers to perform convolution operations on the pixel features in the image, high-dimensional semantic information of the pixels can be obtained. The high-dimensional semantic information of the pixels can also refer to the features extracted through the feature extraction operation; these can also be called pixel features.

[0085] The sampling point determination module can obtain the maximum and minimum possible depths of pixel A1 on the main view A. The maximum possible depth refers to the maximum possible depth value of pixel A1. That is, the actual depth of pixel A1 will not exceed the maximum possible depth. The minimum possible depth refers to the minimum possible depth value of pixel A1. That is, the actual depth of pixel A1 will not be less than the minimum possible depth. The process of determining the maximum and minimum possible depths of a pixel can be found in existing technology descriptions and will not be repeated here. Therefore, the sampling point determination module can obtain an initial hypothetical depth value range for pixel A1, where the upper limit of this initial hypothetical depth value range is the maximum possible depth value of pixel A1, and the lower limit is the minimum possible depth value of pixel A1. Within this initial hypothetical depth value range, D hypothetical depths can be set. The depth values ​​of the different hypothetical depths among these D hypothetical depths are different.

[0086] The sampling point determination module can also obtain the camera parameters of the main view A and its auxiliary view. The camera parameters of the image can include the camera's extrinsic and intrinsic parameters. The extrinsic parameter matrix represents the camera's position, orientation, etc. The intrinsic parameter matrix represents the relationship between the camera coordinate system and the image coordinate system, etc. Therefore, the sampling point determination module can determine a pixel P1 in the auxiliary view B based on the hypothetical depth d1 of the D hypothetical depths of pixel A1, the coordinates of pixel A1 in the main view, the camera parameters of the main view, and the camera parameters of the auxiliary view B, using a homography transformation algorithm. Pixel P1 can be called sampling point P1, or the sampling point P1 of pixel A1 at the hypothetical depth d1. D hypothetical depths can each determine D sampling points. In the homography transformation, the coordinates of pixel A1 in the main view can be converted to the coordinates of the object point in the camera coordinate system at the hypothetical depth, and then the coordinates of the object point in the camera coordinate system can be converted to pixel coordinates in the auxiliary view. The pixel corresponding to these pixel coordinates is the sampling point or pixel of pixel A1 in the auxiliary view at that hypothetical depth. The specific implementation process of homography transformation can be found in the introduction of existing technologies, and will not be elaborated here.

[0087] In some embodiments, see Figure 3 We can set D to 3, and the three hypothetical depths are d1, d2, and d3. At these three hypothetical depths, three sampling points, as shown in Figure 4, can be determined: sampling point P1, sampling point P2, and sampling point P3. According to computational graphics or epipolar geometry, these three sampling points are all located on the epipolar line. This epipolar line is the intersection of the epipolar plane determined by pixel A1, the camera center of the main view A, and the camera center of the auxiliary view, with the auxiliary view B1.

[0088] The sampling point determination module can pass the coordinates of the determined D sampling points to the epipolar pooling module. The epipolar pooling module can then determine the epipolar line (based on epipolar geometry, the line connecting any two of the D sampling points is sufficient to determine the epipolar line) and the spacing between adjacent sampling points based on these coordinates. The spacing between sampling points refers to the spacing along the epipolar line direction. Specifically, it can determine the number of pixels located between two adjacent sampling points and on the epipolar line. This number of pixels represents the size of the spacing.

[0089] The epipolar pooling module determines the size of the pooling kernel for the current sampling point (i.e., the sampling point currently being analyzed or of interest) based on the distance between the current sampling point and its two adjacent sampling points. The pooling kernel can be understood as a pooling window, which performs pooling operations on the features of pixels located on the epipolar line and covered by the pooling kernel of the current sampling point, resulting in new features for the current sampling point. Pooling can be understood as an information aggregation or convergence operation. The pixel features extracted by the feature extraction module are typically in the form of feature values ​​or feature matrices. Specifically, pooling operations can involve averaging multiple feature values ​​or feature matrices, finding the maximum value, or weighted summation, etc., which will not be listed here. The specific implementation process of pooling operations can be found in existing technology descriptions and will not be elaborated upon here.

[0090] Next, combined Figure 3 , Figure 4A , Figure 4B , Figure 4C The example illustrates the process of determining the pooling kernel size for the current sampling point.

[0091] Can be set Figure 3 The sampling point P2 shown is the current sampling point, and the sampling points on both sides of it are sampling point P1 and sampling point P2, respectively. The interval between sampling point P2 and sampling point P1 is represented by ΔP1, and the interval between sampling point P2 and sampling point P3 is represented by ΔP2.

[0092] In one example, such as Figure 4A As shown, the size of ΔP1 can be set to be less than one pixel, and the size of ΔP2 can also be less than one pixel. That is, sampling points P1, P2, and P3 are three consecutively adjacent pixels on auxiliary view B. In this case, pooling is not performed on sampling point P2.

[0093] In one example, such as Figure 4B As shown, the size of ΔP1 can be set to two pixels, meaning that sampling point P1 and sampling point P2 are two pixels apart on the epipolar line. The size of ΔP2 can also be set to two pixels, meaning that sampling point P3 and sampling point P2 are two pixels apart on the epipolar line. It can be set as follows... Figure 4BThe pooling kernel shown is configured to cover the pixel in ΔP1 closest to sampling point P2, and also the pixel in ΔP2 closest to sampling point P2. Pooling is then performed on the features of the pixels covered by the pooling kernel (the pixels in ΔP1 closest to sampling point P2, the pixels in ΔP2 closest to sampling point P2, and sampling point P2 itself). This pools the features of these two pixels onto sampling point P2, resulting in new features for sampling point P2. It can be understood that the pixels in ΔP1 closest to sampling point P2 and the pixels in ΔP2 closest to sampling point P2 could both be the pixels or feature points corresponding to pixel A1 at the true depth. Pooling their feature information into the new features of sampling point P2 improves the accuracy of depth prediction.

[0094] Similarly, the pixel in ΔP1 that is closer to sampling point P1 can be pooled onto sampling point P1 during the pooling process of sampling point P1. The pixel in ΔP2 that is closer to sampling point P3 can be pooled onto sampling point P3 during the pooling process of sampling point P3, thus improving the accuracy of depth prediction.

[0095] In one example, such as Figure 4C As shown, the size of ΔP1 can be set to 4 pixels, meaning that sampling point P1 and sampling point P2 are 4 pixels apart on the epipolar line. The size of ΔP2 can also be set to 4 pixels, meaning that sampling point P3 and sampling point P2 are 4 pixels apart on the epipolar line. It can be set as follows... Figure 4C The pooling kernel shown covers the two pixels in ΔP1 closest to sampling point P2, and also covers the two pixels in ΔP2 closest to sampling point P2. Pooling is performed on the features of the pixels covered by the pooling kernel (the two pixels in ΔP1 closest to sampling point P2, the two pixels in ΔP2 closest to sampling point P2, and sampling point P2 itself). This pools the features of the two pixels in ΔP1 closest to sampling point P2 and the two pixels in ΔP2 closest to sampling point P2 onto sampling point P2, resulting in new features for sampling point P2. It can be understood that the two pixels in ΔP1 closest to sampling point P2 and the two pixels in ΔP2 closest to sampling point P2 could be the pixels or feature points corresponding to pixel A1 at the true depth. Pooling their feature information into the new features of sampling point P2 improves the accuracy of depth prediction.

[0096] Similarly, the two pixels in ΔP1 closest to sampling point P1 can be pooled onto sampling point P1 during the pooling process of sampling point P1. Likewise, the two pixels in ΔP2 closest to sampling point P3 can be pooled onto sampling point P3 during the pooling process of sampling point P3, thus improving the accuracy of depth prediction.

[0097] The above text combined Figures 4A-4C The example provided illustrates how to determine the size of the pooling kernel for the current sampling point, but it is not a limitation. In short, features of pixels closer to the current sampling point from the pixels spaced between the current sampling point and its neighboring sampling points can be pooled into the features of the current sampling point to form a new feature for the current sampling point. In some embodiments, pixels closer to the current sampling point can refer to a subset of pixels from the plurality of pixels spaced between the current sampling point and its neighboring sampling points, which are closer to the current sampling point than other pixels in that plurality of pixels. In some embodiments, pixels closer to the current sampling point can refer to all pixels from the plurality of pixels spaced between the current sampling point and its neighboring sampling points.

[0098] In some embodiments, the assumed depth d3 corresponding to sampling point P3 can be set as the maximum possible depth. Then, the features of the pixels closest to sampling point P3 in the pixels between sampling point P3 and sampling point P2 can be pooled with the features of sampling point P3 to obtain new features of sampling point P3.

[0099] In some embodiments, the assumed depth d1 corresponding to sampling point P1 can be set as the minimum possible depth. Then, the features of the pixels closest to sampling point P1 in the pixels between sampling point P1 and sampling point P2 can be pooled with the features of sampling point P1 to obtain new features of sampling point P1.

[0100] The above method allows for the convergence of features from pixels near the sampling point and located on the epipolar line with the features of the sampling point itself, resulting in new features for the sampling point. The similarity between the sampling point and pixel A1 can then be calculated based on these new features and the features of pixel A1.

[0101] When there are multiple auxiliary views for the main view A, for auxiliary views other than auxiliary view B1, homography transformation can be performed using the aforementioned D hypothetical depths to determine D sampling points on each auxiliary view. For each of the D sampling points on an auxiliary view, the aforementioned dynamic pooling operation is applied to obtain new features for each sampling point. Then, based on the new features of each sampling point and the features of pixel A1, the pixel degree of each sampling point and pixel A1 is calculated. Thus, for an auxiliary view, D similarities can be obtained. Different similarities among the D similarities correspond to different hypothetical depths among the D hypothetical depths.

[0102] The similarity scores of all auxiliary views corresponding to the same hypothetical depth can be aggregated, for example, by taking the average, taking the maximum value, or using a weighted summation. In this way, the similarity scores of each auxiliary view at the same hypothetical depth can be combined to obtain a comprehensive similarity score for each auxiliary view at that same hypothetical depth. For D hypothetical depths, D comprehensive similarity scores can be obtained. Each of the D comprehensive similarity scores corresponds to a different hypothetical depth within those D hypothetical depths. These D comprehensive similarity scores can constitute the matching cost volume of pixel A1 in this prediction stage.

[0103] By referring to the method for determining the matching cost of pixel A1, the matching cost of other pixels on the main view A can be determined.

[0104] Back Figure 2 In some embodiments, the epipolar pooling module can pass the matching cost of each pixel on the main view A to the regularization module. In one example, the regularization module can be a 3D convolutional neural network, which can extract information and smooth the matching cost of each pixel on the main view. For details, please refer to the prior art description, which will not be repeated here.

[0105] For a pixel, such as pixel A1, in the main view A, its matching cost, or the matching cost after regularization, can be converted into a probability volume in the depth prediction module. As mentioned above, the matching cost of pixel A1 consists of D similarities (if there is only one auxiliary view in the main view) or D combined similarities (if there are multiple auxiliary views in the main view). A normalization function (such as the softmax function) can be used to map the D similarities or D combined similarities into D probability values. These D probability values ​​constitute the probability volume of pixel A. Each probability value represents the probability that the assumed depth corresponding to that probability value is the true depth of pixel A1. That is, the larger each probability value is, the greater the probability that the assumed depth corresponding to that probability value is the true depth of pixel A1, or the greater the likelihood that the true depth of pixel A1 is the assumed depth corresponding to that probability value.

[0106] The probability volume of other pixels on the main view can be determined by referring to the method used to determine the probability volume of pixel A1, which will not be elaborated here.

[0107] Continue reading Figure 2The depth prediction module can generate a depth map of the main view based on the probability volume of each pixel. In this depth map, the depth of each pixel can be the depth corresponding to the highest probability value in that pixel's probability volume.

[0108] In some embodiments, Figure 2 The architecture shown also includes a range adjustment module. This module can include range size adjustment units. Specifically, the depth prediction module can pass the probability volume of the main view pixels to the range size adjustment units.

[0109] In some embodiments, the interval size adjustment unit can determine the reduction magnitude based on the dispersion of probability values ​​in the pixel's probability volume. The reduction magnitude is used to narrow the hypothetical depth value interval to obtain a new hypothetical depth value interval. The reduction magnitude refers to the difference between the size of the hypothetical depth interval before and after reduction. A larger reduction magnitude results in a greater difference between the sizes of the hypothetical depth interval before and after reduction. Conversely, a larger reduction magnitude results in a smaller difference between the sizes of the hypothetical depth interval before and after reduction.

[0110] In this embodiment, the dispersion of probability values ​​in the pixel's probability volume is negatively correlated with the reduction magnitude. It can be understood that the higher the dispersion of multiple probability values, the smaller the differences between them, the more diffuse the distribution, or the shorter and wider the distribution curve. This indicates that the maximum probability value among the multiple probability values ​​is smaller. In other words, when the dispersion of multiple probability values ​​is high, if the assumed depth corresponding to the maximum probability value is used as the predicted depth, the prediction result is unreliable or has a high degree of nonconfidence. Nonconfidence is the opposite of confidence; the higher the nonconfidence, the lower the confidence. Therefore, when the dispersion of probability values ​​in the pixel's probability volume is high, the range of assumed depth values ​​sampled in the current prediction stage can be slightly reduced to avoid excluding the true value (the actual depth of the main view pixel) from the reduced range of assumed depth values. In other words, the higher the dispersion of probability values ​​in the pixel's probability volume, the smaller the reduction magnitude.

[0111] In one illustrative example of this embodiment, the entropy value E of pixel p can be calculated based on the degree of dispersion of probability values ​​in the probability volume of pixel p. pixelThe entropy value is negatively correlated with the dispersion of probability values ​​in the probability volume. In one example, the entropy value E of pixel p can be calculated using formula (4). pixel .

[0112]

[0113] Where D is the number of assumed depths in the prediction of this stage, prob d (p) represents the probability that the true depth of pixel p is the assumed depth d. Here, the entropy value E... pixel This represents the encoding length required to describe all possible states of pixel p at D hypothetical depths.

[0114] In some embodiments, the scaling factor of pixel p can be determined based on its entropy value. The scaling factor is negatively correlated with the entropy value.

[0115] It is understood that the depths of different pixels in the main view may differ, but the difference between the depths of different pixels is fixed. For example, the depth of pixel A1 can be set to d1, and the depth of pixel A2 can be set to d1+1. If the assumed depth range of pixel A1 is [d1-Δd, d1+Δd], then the assumed depth range of pixel A2 should also be [d1+1-Δd, d1+1+Δd]. That is, the size of the assumed depth range of pixel A1 and the size of the assumed depth range of pixel A2 should be consistent. Only in this way can the overall accuracy of the depth prediction results of each pixel in the main view be guaranteed or improved. Therefore, in this embodiment, the same reduction factor can be used to reduce the assumed depth range of different pixels in the main view.

[0116] In some embodiments, the assumed depth range of pixel p and other pixels in the main view can be adjusted according to the reduction magnitude corresponding to pixel p, so that the assumed depth range of different pixels in the main view can be reduced in the same proportion.

[0117] In some embodiments, see Figure 5 It can calculate the entropy value of each pixel in multiple pixels on the main view (e.g., Figure 5 The entropy values ​​e1, e2, e3, and e4 are shown, and the average entropy value of multiple pixels is calculated. These multiple pixels can be some pixels on the main view, or they can be all pixels on the main view.

[0118] It can be based on the entropy mean. Calculate the nonconfidence. In one example, the nonconfidence can be calculated using formula (5).

[0119]

[0120] Where D represents the number of assumed depths in the prediction of this stage.

[0121] The reduction range can be determined based on the level of unreliability. For example, the reduction range can be represented by a ratio. The new hypothetical depth range, obtained by multiplying the current depth hypothesis range by this ratio, is used as the size of the hypothetical depth range for the next stage of depth prediction. It can be understood that the value of the ratio is inversely proportional to the reduction range. The smaller the ratio, the greater the reduction. The larger the ratio, the smaller the reduction.

[0122] In one example, the ratio can be calculated using formula (6).

[0123]

[0124] Using the above method, the reduction range of the assumed depth value interval can be determined, and then the size of the new assumed depth value interval can be determined based on the size and reduction range of the assumed depth value interval in this stage.

[0125] For a pixel in the main view, such as pixel A1, after determining the size of the new depth assumption interval, the predicted depth of pixel A1 can be used as the median of the new assumption depth value interval, thereby determining an assumption depth value interval with specific upper and lower limits. This assumption depth value interval can be used to preset the depth of pixel A1 again.

[0126] In some embodiments, the assumed depth corresponding to the maximum probability value in the probability volume is used as the predicted depth. In some embodiments, the expected depth can be calculated based on the probability distribution to calculate the expected depth, and this expected depth can be used as the predicted depth. Specifically, the probability distributions of each probability value in the probability volume of pixel A1 can be multiplied by the corresponding assumed depth, and then summed. The sum obtained is used as the expected depth.

[0127] The image depth prediction method provided in this application can aggregate the features of pixels between sampling points under different assumed depths into the features of the sampling points, and calculate the similarity between the sampling points and the pixels of the main view, thereby improving the accuracy of depth prediction. Furthermore, the scheme provided in this application can determine the reduction range based on the dispersion between the probability values ​​of each assumed depth. This allows for a slight reduction of the depth assumption range when the prediction result is unreliable, and a significant reduction of the depth assumption range when the prediction result is reliable, so as to perform depth prediction again. This achieves the goal of reducing computational complexity while avoiding the exclusion of the true value.

[0128] See Figure 6 In some embodiments, this application provides a neural network that, when executed by an electronic device, can implement the image depth prediction method provided in this application. This method can employ a multi-stage depth prediction approach for image depth prediction. For example... Figure 6 As shown, the neural network may include a feature extraction module, a sampling point determination module, an epipolar pooling module, a regularization module, and a depth prediction module. The modules in this neural network can also be called layers; for example, the feature extraction module can be called a feature extraction layer, the sampling point determination module can be called a sampling point determination layer, and so on.

[0129] Next, we will introduce an example based on the training process of a neural network. Figure 6 The functions of each module in the neural network shown.

[0130] The neural network is trained using a supervised training method. During training, the ground truth depth map of the main view (containing the true depth of pixels) is used to supervise the depth prediction output of the neural network. The network parameters of the modules in the neural network (e.g., feature extraction module, regularization module, etc.) are iteratively updated using a gradient backpropagation mechanism until the loss converges (reaches a preset value). The network parameters of the trained neural network can be saved to the hard drive of an electronic device and used for image depth prediction.

[0131] When preparing training data, or the training set, we can group an image and the n (n is an integer greater than or equal to 1) images with the highest correlation to it, based on the given matching relationships between images. The depth prediction object, or subject, is used as the main view, and the other images are used as auxiliary views. We obtain the camera parameters of the images in each view group, which serve as the corresponding camera parameter set; we also obtain the ground truth depth map corresponding to the main view in each view group. Each training dataset consists of one view group, one corresponding camera parameter set, and one ground truth depth map. All training data is stored on the server's hard drive.

[0132] Next, combined Figure 7 , Figure 8 , Figure 9 The flowchart shown provides an example of the function of each module in the neural network.

[0133] Training data, including view groups and camera parameters, can be input into the feature extraction module. The feature extraction module can perform step 701, extracting pixel features from each image in the view group to obtain the features, or feature information, of pixels in the main view and auxiliary views. For example, the feature extraction module may include one or more concatenated two-dimensional convolutional layers for extracting two-dimensional pixel features. For example, in one training iteration, N training data sets (N being a pre-defined training batch size) can be read from the server hard drive into a computing chip (e.g., CPU, GPU, or Ascend chip). It is understood that for multi-stage depth prediction, different scale image features are used for depth prediction at different stages. Therefore, the image feature extraction module can extract the image features required for each stage of the multi-stage depth prediction. The multi-stage depth prediction can be configured to have P stages (i.e., depth prediction is performed once in each of the P stages), and the feature extraction module can extract P sets of image features, where each set of image features includes the image features of N main views and the corresponding auxiliary views of each of the N main views. The k-th image feature group out of P groups can be used in the k-th stage of the P-stage depth prediction process. Here, P is a positive integer greater than or equal to 2. K is a positive integer greater than or equal to 1, representing the stage number of the current prediction stage, i.e., the k-th stage in the multi-stage prediction. As the stage number increases, the image feature data also increases sequentially. The current stage, k, represents the current depth prediction stage in the multi-stage depth prediction process. Image features can refer to the features of pixels in the image.

[0134] The feature extraction module can input its extracted pixel features and camera parameters into the sampling point determination module. The sampling point determination module can execute step 702 to determine whether the stage number k of the current stage is equal to 1. Here, for a training iteration, a stage with a stage number of 1 refers to the first stage in the multi-stage depth prediction during the iterative training of multi-stage depth prediction.

[0135] If k equals 1, the sampling point determination module can execute step 703a, setting Dk hypothetical depths based on the initial hypothetical depth value range and the current stage's hypothetical depth quantity Dk. Here, Dk can be a preset value. The execution steps of step 703a can be referred to the above description. Figure 2 The functions of the sampling point determination module shown are not described in detail here.

[0136] If k is not equal to 1, it indicates that the current stage is not the first stage in multi-stage depth prediction. In this case, the sampling point determination module can execute step 703b, setting the hypothetical depth based on the adjusted hypothetical depth value range from the previous stage and the number of hypothetical depths Dk in the current stage. The adjusted hypothetical depth value range from the previous stage is the new hypothetical depth value range determined in the previous stage. The process for determining the new hypothetical depth value range can be referred to the above section. Figure 2 The functions of the interval adjustment module shown are not described in detail here.

[0137] After setting the assumed depth, the sampling point determination module can execute step 704 to determine the sampling points. Homography transformation can be used to map pixels from the main view to the auxiliary view to obtain the sampling points. Specifically, if the current prediction stage is stage 1, the initial assumed depth value interval R = [Depth...]. min Dept max ], divided into D1 depth hypotheses {depth i |i=1,2,...,D1}. Where, Depth min Denotes the minimum possible depth, Depth max This represents the maximum possible depth. It can be represented by the coordinates p of all pixels on the main view. ref The pixel coordinate set {p} is obtained by mapping the homography transformation onto all auxiliary views. src1 P src2 , ..., p srcn}. The set of coordinates that each pixel coordinate in the main view maps to on an auxiliary view is denoted by {p src|i , i = 1, 2, ..., D1} represent depths according to different assumptions. i The coordinates obtained through mapping. If the current prediction stage is stage k not equal to 1, then the new hypothetical depth range Area is determined by the prediction of the previous stage. k And the assumed depth D of stage k k Set the assumed depth. All pixel coordinates p on the main view. ref The pixel coordinate set {p} is obtained by mapping the homography transformation onto all auxiliary views. src1 p src2 , ..., p srcn}. The set of coordinates that each pixel coordinate in the main view maps to on an auxiliary view is denoted by {p src|i i = 1, 2, ..., D k} represents the depth based on different assumptions. i |i = 1, 2, ..., D k The coordinates obtained by mapping. The implementation process of step 704 can also be referred to the above text. Figure 2 The description of the sampling point determination module shown will not be repeated here.

[0138] The sampling point determination module can input the information of the determined sampling points, such as the coordinates of the sampling points, into the epipolar pooling module. The epipolar pooling module can execute step 705 to determine the spacing between adjacent sampling points. This can be done based on the pixel coordinate set {p} obtained in step 704. src|i i = 1, 2, ..., D k}, calculate the distance between the coordinates of adjacent sampling points, the distance can be expressed as {Δp} src|i i = 1, 2, ..., D k -1} represents.

[0139] The epipolar pooling module can also perform step 706, which involves performing pooling operations centered on the sampling point coordinates and according to the spacing between adjacent sampling points. Specifically, this can be done on the epipolar line of each auxiliary view, using the sampling point coordinates {p src|i i = 1, 2, ..., D k Centered on}, the sampling point spacing {Δp} calculated in step 705 is used as the reference point. src|i i = 1, 2, ..., D k -1} sets the size of the dynamic pooling kernel, performing dynamic pooling on the features of pixels surrounding the sampling point to aggregate the features of the sampling point, thus achieving information aggregation. For details, please refer to the above text. Figure 2 The diagram illustrates the functionality of the polar pooling module.

[0140] Step 707 can also be performed on the epipolar pooling module to construct a cost volume based on the pooling operation result and the features of the main view pixels. Specifically, after processing in step 706, all auxiliary views can obtain the feature set of sampling points {feat} of the auxiliary views. src1 feat src2 , ...,feat srcn}, where the sampling point features of each auxiliary view can be represented by a shape H k *W k *D k The three-dimensional tensor representation. The pixel features of the main view are copied to D... k After that, we can also obtain a shape that is also H. k *W k *D k The three-dimensional feature tensor. The pixel features of the main view are multiplied by the sampled point features of each auxiliary view to obtain the corresponding cost volume.

[0141] Step 708 can also be performed on the epipolar pooling module to aggregate the costvolume generated by each auxiliary view with the main view (e.g., taking the mean, taking the maximum, or weighted sum) to obtain the final cost volume, whose shape can be H. k *W k *D k We obtain a three-dimensional tensor.

[0142] Step 709 can also be executed for the epipolar pooling module to output the aggregated cost volume.

[0143] For more information on the functionality of the polar pooling module, please refer to the section above. Figure 2 The diagram illustrates the implementation of the polar pooling module's functionality.

[0144] The epipolar pooling module can output the aggregated cost volume to the regularization module. The regularization module can then execute step 710, extracting and smoothing information from the cost volume of each main view. For example, a 3D convolutional neural network can be used to extract and smooth information from the cost volume of each main view. For details, please refer to the existing technology descriptions; further elaboration is omitted here.

[0145] Continue reading Figure 7 The depth prediction module can obtain the cost volume of each main view and execute step 711 to determine the probability volume of a pixel based on its cost volume. Specifically, the cost volumes of the N main views output by the regularization module are in D. k Dimension (D) k Perform softmax activation function operation on each hypothetical depth to obtain D. k A hypothetical depth {depth i |i = 1, 2, ..., D k The normalized probability volume at} has a shape of H. k *W k *D k .

[0146] The depth prediction module can also perform step 712, determining the predicted depth based on the assumed depth and the probability volume. Specifically, based on D... k A hypothetical depth {depth i |i = 1, 2, ..., D k} and probability volume, calculate the expected depth (see above for details) to obtain the predicted depth of each pixel. The predicted depth shape of a pixel in the main view is H. k *W k .

[0147] The depth prediction module can also execute step 713 to determine whether k is equal to P. P is a preset value that represents the number of stages in the multi-stage depth prediction, meaning that the Pth stage is the last stage of the multi-stage depth prediction.

[0148] If k is not equal to P (i.e. k is less than P), the depth prediction module 713 can input the probability volume and the predicted depth into the interval adjustment module.

[0149] Continue reading Figure 7 The interval adjustment module can execute step 714, which calculates the entropy value corresponding to the pixel based on the probability volume of the previous pixel in the main view. Specifically, it can adjust the probability volume (determined by D) determined in step 711 according to the formula (4) shown above. k The probability value constitutes D. k Each probability value and D k (Assuming a one-to-one correspondence between the depths), the entropy of dimension Dk is calculated to obtain the entropy value E corresponding to the pixel. pixel Or, in other words, the entropy value E at the pixel level. pixel The entropy values ​​of each pixel in a front view form a shape of H. k *W k The matrix. Please refer to the above text for details. Figure 2 The function of the interval size adjustment unit shown will not be described again here.

[0150] The interval adjustment module can also execute step 715 to determine the unreliability of the prediction result based on the pixel entropy value. Specifically, the pixel granularity entropy value E, determined in step 714, can be calculated using formula (5) shown above. pixel After taking the average, this value is used as the exponent of the number of depth hypotheses Dk in the current stage. The resulting calculation yields the unreliability of the depth result for the current stage, which can be expressed as nonconfidence. k express.

[0151] The interval adjustment module can also execute step 716 to determine the reduction range based on the degree of unreliability. The reduction range can be calculated using formula (6) shown above, with a ratio. k express.

[0152] The interval adjustment module can also execute step 717 to determine a new range of hypothesis depth values ​​based on the reduction margin. The size of the current hypothesis depth range can be multiplied by the reduction margin (Ratio). k This allows us to obtain the size of the new hypothetical depth range, Area. k+1 After determining the size of the new hypothetical depth range, Area k+1 The prediction result, or prediction depth, of the current stage can be used as the median of the new hypothetical depth value range. This new hypothetical depth value range can also be called the hypothetical depth value range for the next stage, and it is used for image depth prediction in the next stage. In other words, the hypothetical depth used for image depth prediction in the next stage is taken from the new hypothetical depth value range.

[0153] In one example, the range of values ​​for the hypothesis depth in the current stage can be the maximum hypothesis depth in the current stage. Subtract the minimum assumed depth The difference obtained can be used to calculate the size of the new hypothetical depth range, Area, using formula (7). k+1 .

[0154]

[0155] Steps 714-717 can be executed by the interval size adjustment unit in the interval adjustment module.

[0156] The interval adjustment module can output the determined new hypothesis depth value interval to the sampling point determination module, so that the sampling point determination module can set the hypothesis depth from the new hypothesis depth value interval in the next stage.

[0157] Continue reading Figure 7 If k = P, meaning the current stage is the last stage of multi-stage depth prediction, then the parameters of the modules in the neural network can be optimized in step 718.

[0158] Specifically, when k is less than P, repeat steps 702-717 until all prediction stages output the predicted depth map {Depth} for each main view. pred|k Let k = 1, 2, ..., P. If there are N main views, then there are N sets of predicted depth maps. For each of the N main views, the difference between the predicted depth map and the ground truth depth map is calculated as the loss. The network parameters of the modules (e.g., feature extraction module, regularization module) in this neural network are updated through gradient backpropagation.

[0159] Repeat steps 702-717 until the loss converges. Save the parameters of the neural module (e.g., feature extraction module and regularization module) to the server hard disk for use in image depth prediction.

[0160] Figure 8 An architecture for image depth prediction using neural networks is presented. The functions of each module in this architecture can be found in the description above. Figure 6 The description of the illustrated embodiment will not be repeated here. Unlike training a neural network, when using a neural network for image depth prediction, pre-trained parameters (such as those from the feature extraction and regularization modules) can be directly used for image depth prediction. Furthermore, the input data does not include the ground truth depth map; the input data may include a view group and its corresponding summing parameter group. When k = P, i.e., the predicted depth map determined in the last stage of multi-stage depth prediction, can be output as the final prediction result.

[0161] In summary, the image depth prediction method provided in this application can converge the feature information around the sampling points by judging the spacing between sampling points on the epipolar line of the auxiliary view and using an appropriate dynamic pooling kernel. Without changing the shape of the subsequent cost volume, it fully utilizes the pixel features on the epipolar line of the auxiliary view, ensuring high performance while providing high accuracy. This solves the problem that when calculating the pixel matching between the main view and the auxiliary view, it is impossible to sample the features of each pixel on the epipolar line of the auxiliary view, resulting in insufficient sampling of areas with drastic texture changes in the auxiliary view, potentially missing the true value and leading to incorrect depth prediction. It also solves the problem that when the sampling interval is small, the cost volume is large (using a small interval indicates assuming a large number of depths, resulting in a large cost volume), leading to high accuracy but significantly increased computation time; while when the sampling interval is large, the cost volume is small, resulting in shorter computation time but severely compromised accuracy.

[0162] Furthermore, the image depth prediction method provided in this application can quantify the reliability of the depth prediction result by calculating the entropy of the depth prediction probability in the current stage. This allows for adaptive adjustment of the size of the depth assumption region in the next stage, thereby minimizing the assumed depth value range without affecting the probability that the actual depth falls within the adjusted assumed depth value range, thus improving the efficiency of depth prediction. It also solves the problem that if the prediction result in the previous stage is inaccurate when the reduction range is manually set, the actual depth may be excluded from the reduced assumed depth value range.

[0163] In summary, this application provides an image depth prediction method that can be executed by an electronic device. Figure 9As shown, the method includes the following steps.

[0164] Step 901: Obtain a main view and a first auxiliary view. The main view and the first auxiliary view are images taken by the camera when facing the first object from different angles. The main view includes a first pixel, the first pixel corresponds to a second pixel under a first assumed depth, and the second pixel is located on the first auxiliary view.

[0165] Step 902: Update the feature information of the second pixel based on the feature information of the second pixel and the feature information of at least one third pixel; the at least one third pixel is located on the line connecting the second pixel and the fourth pixel, the fourth pixel is the pixel corresponding to the first pixel on the first auxiliary view under the second assumed depth, the second assumed depth and the first assumed depth are adjacent in D preset assumed depths, where D is a positive integer greater than 1.

[0166] Step 903: Based on the feature information of the first pixel and the updated feature information of the second pixel, obtain the probability of the first assumed depth. The probability of the first assumed depth is used to represent the possibility that the first assumed depth is the true depth of the first pixel.

[0167] In some embodiments, the method further includes: updating the feature information of the fourth pixel based on the feature information of the fourth pixel and the feature information of at least one fifth pixel; the at least one fifth pixel is located on the line connecting the second pixel and the fourth pixel; obtaining the probability of the second assumed depth based on the feature information of the first pixel and the updated feature information of the fourth pixel, the probability of the second assumed depth being used to represent the possibility that the second assumed depth is the true depth of the first pixel.

[0168] In one example of these embodiments, the at least fifth pixel and the at least third pixel are not the same, and the at least fifth pixel is closer to the fourth pixel than the at least third pixel; or, a pixel contained in the at least fifth pixel but not contained in the at least third pixel is closer to the fourth pixel than a pixel contained in the at least third pixel but not contained in the at least fifth pixel.

[0169] In one example of these embodiments, the union of the at least fifth pixel and the at least third pixel includes all pixels on the line connecting the second pixel and the fourth pixel.

[0170] In some embodiments, the method further includes: reducing the size of the value intervals corresponding to the D assumed depths to obtain the size of a first interval, wherein the first interval is used as the value interval of the assumed depth when the first pixel depth is predicted again.

[0171] In one example of these embodiments, the magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths.

[0172] In some embodiments, the method further includes: reducing the size of the value intervals corresponding to the D′ hypothetical depths to obtain the size of a second interval, the second interval being used as the value interval of the hypothetical depth when the sixth pixel depth is predicted again; the reduction magnitude of the value intervals corresponding to the D′ hypothetical depths is equal to the reduction magnitude of the value intervals corresponding to the D hypothetical depths; the sixth pixel is a pixel on the main view and is different from the first pixel; D′ is a positive integer greater than 1.

[0173] In one example of these embodiments, the magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths and negatively correlated with the entropy value of the sixth pixel at the D′ hypothetical depths.

[0174] The image depth prediction method provided in this application embodiment can aggregate the features of pixels that are not sampling points on the epipolar line into the features of nearby sampling points. This allows for depth prediction using the features of more pixels on the epipolar line without setting a larger number of assumed depths, thereby improving the accuracy of depth prediction.

[0175] See Figure 10 This application provides an image depth prediction device 1000. The device 1000 includes:

[0176] The acquisition unit 1010 is used to acquire a main view and a first auxiliary view. The main view and the first auxiliary view are images taken by a camera when facing a first object from different angles. The main view includes a first pixel, the first pixel corresponds to a second pixel under a first assumed depth, and the second pixel is located on the first auxiliary view.

[0177] The update unit 1020 is used to update the feature information of the second pixel according to the feature information of the second pixel and the feature information of at least one third pixel; the at least one third pixel is located on the line connecting the second pixel and the fourth pixel, the fourth pixel is the pixel corresponding to the first pixel on the first auxiliary view under the second assumed depth, the second assumed depth and the first assumed depth are adjacent in D preset assumed depths, where D is a positive integer greater than 1.

[0178] The unit 1030 is used to obtain the probability of the first assumed depth based on the feature information of the first pixel and the updated feature information of the second pixel. The probability of the first assumed depth is used to represent the possibility that the first assumed depth is the true depth of the first pixel.

[0179] The functional units of device 1000 can be referenced. Figures 2-9 The method implementation shown is as follows, for example Figure 9 The method embodiments shown will not be described again here.

[0180] The foregoing mainly described the apparatus provided in the embodiments of this application from the perspective of method and flow. It is understood that each terminal, in order to achieve the above functions, includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] See Figure 11 This application provides an electronic device 1100. The electronic device 1100 can perform... Figures 2-9 The operations performed by the electronic devices in the illustrated embodiments, for example Figure 9 As shown. The electronic device 1100 may include a processor 1110 and a memory 1120. The memory 1120 stores instructions that can be executed by the processor 1110. When the instructions are executed by the processor 1110, the electronic device 1000 can perform the operations performed by the electronic devices in the above-described method embodiments, such as... Figure 9 The operations performed by the electronic device in the illustrated embodiment.

[0182] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0183] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0184] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0185] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. An image depth prediction method, characterized in that, include: Obtain a main view and a first auxiliary view. The main view and the first auxiliary view are images taken by a camera when facing a first object from different angles. The main view includes a first pixel, the first pixel corresponds to a second pixel under a first assumed depth, and the second pixel is located on the first auxiliary view. Update the feature information of the second pixel based on the feature information of the second pixel and the feature information of at least one third pixel; The at least one third pixel is located on the line connecting the second pixel and the fourth pixel. The fourth pixel is the pixel corresponding to the first pixel in the first auxiliary view at the second assumed depth. The second assumed depth and the first assumed depth are adjacent in a preset D assumed depths, where D is a positive integer greater than 1. Based on the feature information of the first pixel and the updated feature information of the second pixel, the probability of the first assumed depth is obtained. The probability of the first assumed depth is used to represent the possibility that the first assumed depth is the true depth of the first pixel.

2. The method according to claim 1, characterized in that, The method further includes: updating the feature information of the fourth pixel based on the feature information of the fourth pixel and the feature information of at least one fifth pixel; wherein the at least one fifth pixel is located on the line connecting the second pixel and the fourth pixel; Based on the feature information of the first pixel and the updated feature information of the fourth pixel, the probability of the second assumed depth is obtained. The probability of the second assumed depth is used to represent the possibility that the second assumed depth is the true depth of the first pixel.

3. The method according to claim 2, characterized in that, The at least one fifth pixel and the at least one third pixel are different from each other, and the at least one fifth pixel is closer to the fourth pixel than the at least one third pixel; or, A pixel contained in at least one fifth pixel but not contained in at least one third pixel is closer to the fourth pixel than a pixel contained in at least one third pixel but not contained in at least one fifth pixel.

4. The method according to claim 2 or 3, characterized in that, The union of the at least fifth pixel and the at least third pixel includes all pixels on the line connecting the second pixel and the fourth pixel.

5. The method according to claim 1, characterized in that, The method further includes: By narrowing the value intervals corresponding to the D hypothetical depths, the size of the first interval is obtained. The first interval is used as the value interval of the hypothetical depth when the first pixel depth is predicted again.

6. The method according to claim 5, characterized in that, The magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths.

7. The method according to claim 4, characterized in that, The method further includes: The size of the value interval corresponding to the D' assumed depths is reduced to obtain the size of the second interval, which is used as the value interval of the assumed depth when the sixth pixel depth is predicted again; the reduction of the value interval corresponding to the D' assumed depths is equal to the reduction of the value interval corresponding to the D assumed depths; the sixth pixel is a pixel on the main view and is different from the first pixel; D' is a positive integer greater than 1.

8. The method according to claim 7, characterized in that, The magnitude of the reduction in the value range corresponding to the D hypothetical depths is negatively correlated with the entropy value of the first pixel at the D hypothetical depths and negatively correlated with the entropy value of the sixth pixel at the D' hypothetical depths.

9. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images