Scene depth estimation method and device, image processing equipment and storage medium

Abstract

The invention discloses a scene depth estimation method and device, image processing equipment and a storage medium, and belongs to the technical field of computer vision, and the scene depth estimation method comprises the steps: selecting a single image with shielding, reflection or perspective from a multi-view image set for depth fusion; the improved depth information is spread to other views in a re-projection mode, so that the time overhead is reduced while the reconstruction effect is improved; performing semantic division on the index image to obtain the weight of each semantic factor; the problem that feature points are difficult to match due to a special area is solved by introducing single-view depth estimation; fusing a single-view estimation result and a multi-view depth estimation result under the constraint of semantic information, and spreading the fused result to other view angles in a re-projection manner to obtain a consistent depth estimation result corresponding to the current scene; consistency estimation can be accurately carried out on the depth of each area in the scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, relates to a scene depth estimation method, device, image processing equipment and storage medium. Background Art

[0002] With the rapid development of computer vision technology, 3D reconstruction technology has received more and more attention and has been widely used in many fields such as virtual reality, autonomous driving, and cultural heritage protection. Multi-view stereo vision reconstruction is one of the most commonly used methods in the field of 3D reconstruction. Figure 1 The consistency hypothesis is an important cornerstone of multi-view geometry theory. It states that when observing the same scene or object from different perspectives, the basic geometric structure and appearance properties of the observed scene or object, such as texture and color, should be consistent despite the different observation angles. However, in the real world, objects often occlude each other or have mirror reflections, transmissions, etc., resulting in the inability to effectively and correctly restore feature points under certain perspectives. This situation will cause information loss or errors during the reconstruction process, especially in those details that can only be observed from specific angles, making the reconstruction result unable to accurately reflect the complete form of the object.

[0003] In multi-view stereo reconstruction, the occlusion, specular reflection and transmission areas in the image violate the multi-view Figure 1 Therefore, it is difficult for existing methods to accurately estimate the depth of each area in the scene. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a scene depth estimation method, apparatus, image processing device and storage medium, the purpose of which is to solve the technical problem that the prior art is difficult to accurately and consistently estimate the depth of each area in the scene.

[0005] To achieve the above object, according to one aspect of the present invention, a scene depth estimation method is provided, comprising:

[0006] Obtaining a multi-view image of the current scene, and calculating a multi-view depth estimation result corresponding to the multi-view image;

[0007] Selecting a single-view image with visual deviation from the multi-view images as an index image;

[0008] Calculating a single-view depth estimation result corresponding to the index image;

[0009] Performing semantic division on the index image to obtain weights of various semantic factors within a semantic range; the semantic factors include distance factors, gradient factors and coplanar factors;

[0010] Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result by using the weights of the semantic factors to obtain a depth fusion image;

[0011] The depth fusion image is projected to each other perspective except the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0012] Further, the calculating the single-view depth estimation result corresponding to the index image includes:

[0013] Inputting the index image into a neural window fully connected conditional random field improved model to obtain a preliminary depth estimation result of the index image;

[0014] Denoising is performed on the preliminary depth estimation result of the index image to obtain a single-view depth estimation result corresponding to the index image.

[0015] Furthermore, the neural window fully connected conditional random field improved model includes: a convolutional neural network and a fully connected conditional random field model; the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image.

[0016] Furthermore, the weight of the distance factor is (i,j) and (u,v) represent different pixels;

[0017] The weights of the gradient factors are: s ij and uv The single view depth estimation results of pixels (i, j) and (u, v) are respectively, and Respectively represent the depth gradient coefficients in the x and y directions at the pixel point;

[0018] The coplanar factor weights include the vertical factor weights and level factor weights Parameters σ1, σ2, and σ3 are used to balance the distance factor, gradient factor, and coplanar factor to ensure the continuity of the fusion result.

[0019] Furthermore, the calculation process of the depth fusion map includes:

[0020] S51: Normalize the distance factor weight, the gradient factor weight, and the coplanar factor weight to get the interpolation weight

[0021] S52: Using formula Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result to obtain the depth fusion image;

[0022] Wherein, Ω is the pixel set of the semantically divided region to which the pixel (i, j) in the index image belongs, (u, v) represents the other pixels in the pixel set Ω except (i, j), and m uv and uv are the multi-view depth estimation results and single-view depth estimation results of pixels (u, v), s ij is the single view depth estimation result of pixel (i, j), f ij is the fusion depth corresponding to (i, j).

[0023] Further, the S51 includes: using the formula The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; wherein, Pixel (i, j), pixel (u, v), pixel (p, k) and pixel (g, h) are different points in the pixel set Ω, n s ∈[0, 1], is the product of three weights corresponding to (i, j) and (u, v), is the product of the three weights corresponding to (i, j) and (g, h), It is the product of three weights corresponding to (i, j) and (p, k).

[0024] Furthermore, projecting the depth fusion image to each other perspective except the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene includes:

[0025] The depth information of the depth fusion map is projected to other perspectives by combining perspective projection and bundle adjustment.

[0026] A depth consistency check is performed on the reprojected depth information, and depth adjustment is performed on pixels with deviations to ensure consistency between depth information of the current scene at multiple viewing angles.

[0027] According to another aspect of the present invention, there is provided a scene depth estimation device, comprising:

[0028] An acquisition module, used to acquire multi-view images of the current scene;

[0029] A determination module, configured to select a single-view image with visual deviation from the multi-view images as an index image;

[0030] A calculation module, used to calculate a single-view depth estimation result corresponding to the index image;

[0031] A division module, used for performing semantic division on the index image to obtain weights of various semantic factors within a semantic range; the semantic factors include distance factors, gradient factors and coplanar factors;

[0032] A fusion module, used to perform image fusion on the single-view depth estimation result and the multi-view depth estimation result by using the weights of the semantic factors to obtain a depth fusion image;

[0033] A projection module is used to project the depth fusion image to each other perspective except the perspective corresponding to the index image, so as to obtain a consistent depth estimation result corresponding to the current scene.

[0034] According to another aspect of the present invention, there is provided an image processing device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0036] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0037] (1) The present invention provides a scene depth estimation method by selecting a single-view image with visual deviation from the multi-view images as an index image; specifically, selecting a single image with visual deviations such as occlusion, reflection or perspective from the multi-view image set for depth fusion, and propagating the improved depth information to other views by reprojection, thereby improving the reconstruction effect while reducing time overhead; performing semantic division on the index image to obtain the weights of each semantic factor within the semantic range; introducing single-view depth estimation to deal with the problem of difficult matching of feature points in special areas; under the constraint of semantic information, fusing the single-view estimation result and the multi-view depth estimation result, and propagating the depth fusion to other view angles by reprojection to obtain a consistent depth estimation result corresponding to the current scene. That is, the present application avoids using multi-view images of areas such as occlusion, specular reflection and transmission in the image. Figure 1 The consistency assumption can achieve accurate and consistent estimation of the depth of each area in the scene.

[0038] (2) In this scheme, the index image is input into the improved model of the neural window fully connected conditional random field to obtain a preliminary depth estimation result of the index image; this process introduces a single-view depth estimation algorithm based on the neural window fully connected conditional random field, uses the conditional random field to construct an energy function, and then obtains depth prediction by optimizing the energy function, thereby improving the accuracy of single-view depth estimation.

[0039] (3) In this solution, the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image. The network structure is improved by cascading the feature map with the original image and introducing jump connections to improve the accuracy of single-view depth estimation.

[0040] (4) In this scheme, expressions of four semantic factors are set up. Under the semantic constraints, the distance, gradient and plane factors are comprehensively considered to fuse the single-view depth estimation results of a single image containing special areas and the multi-view depth estimation results, thereby improving the accuracy and consistency of the depth map as a whole.

[0041] (5) The formula used in this scheme is The single-view depth estimation result and the multi-view depth estimation result are image-fused; the above formula converts the depth gradient (s ij -s uv ) is added to each multi-view depth m uv In the example, the structure of s is used for each m uv Create a depth map and then weight it using pixel-based weights; low computational complexity and good fusion effect.

[0042] (6) The formula used in this scheme is The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; the normalized weight expands the local influence to the entire image semantic division range, and the fused depth estimation of each pixel (i, j) is the weighted fusion of the single view depth and the multi-view depth within the semantic range to which it belongs. ij Through the above strategies, the depth of special areas can be better estimated.

[0043] (7) In this scheme, perspective projection and bundle adjustment are combined to project the depth information of the depth fusion image to other perspectives; the improved depth fusion image is projected to other perspectives using the internal and external parameters of the camera to ensure the global consistency of the depth data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a scene depth estimation method provided in Embodiment 1 of the present invention;

[0045] Figure 2 A schematic diagram of a scene depth estimation method provided in Embodiment 1 of the present invention;

[0046] Figure 3 A schematic diagram of fully connected CRFs and windowed fully connected CRFs provided in Example 1 of the present invention;

[0047] Figure 4 A schematic diagram of the network structure of a single-view depth estimation model based on neural window fully connected conditional random field optimization provided in Example 1 of the present invention;

[0048] Figure 5 A schematic diagram of image semantic segmentation provided by Embodiment 1 of the present invention;

[0049] Figure 6 This is a schematic diagram of polar line geometry provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] Example 1

[0052] like Figure 1 and Figure 2 As shown, this embodiment provides a scene depth estimation method, including: steps S1-S6. Among them, S1: obtain a multi-view image of the current scene, and calculate the multi-view depth estimation result corresponding to the multi-view image; S2: select a single-view image with visual deviation from the multi-view image as the index image; S3: calculate the single-view depth estimation result corresponding to the index image; S4: perform semantic division on the index image to obtain the weight of each semantic factor within the semantic range; the semantic factors include distance factors, gradient factors and coplanar factors; S5: use the weights of each semantic factor to fuse the single-view depth estimation result and the multi-view depth estimation result to obtain a depth fusion map;

[0053] S6: Project the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0054] Operation S1: Acquire multi-view image data of a scene, and calculate multi-view depth estimation results corresponding to the multi-view images. The goal of this step is to acquire multi-view images of a scene by shooting from multiple angles to ensure comprehensive information coverage.

[0055] In operation S2, the quality of the acquired multi-view images can be evaluated, taking into account factors including viewing angle, clarity, lighting conditions and surface features of the object, and images containing special areas such as occlusion, specular reflection, perspective, etc. are preferentially selected as index images; then, the most representative single-view image is selected from the multi-view image data as the index image to ensure that the image has sufficient details and recognizability for subsequent depth estimation.

[0056] It should be noted that there is no sequential requirement for steps S3 and S4. S3-S4 can be performed sequentially. Alternatively, the index image can be semantically divided to determine the weights of each semantic factor, and then the single-view depth estimation result corresponding to the index image can be calculated. The calculation of the multi-view depth estimation result corresponding to the multi-view image in S1 only needs to be performed before S4.

[0057] Operation S4: semantically segmenting the index image according to the scene structure information. Semantically segmenting the index image using the depth information and the image features to ensure that the semantic information of each region is fully expressed.

[0058] Operation S5: performing image fusion on the single-view depth estimation result and the multi-view depth estimation result based on distance, gradient and coplanarity factors within each semantic range to obtain a depth fusion image.

[0059] Operation S6: Propagate the depth information of the depth fusion map to other perspectives by reprojection to achieve global consistency of depth data in special areas. In this step, the reprojection technology is used to map the information of the depth fusion map to other perspectives to ensure the consistency of depth data from different perspectives.

[0060] Furthermore, S3 includes: inputting the index image into the neural window fully connected conditional random field improved model to obtain a preliminary depth estimation result of the index image; denoising the preliminary depth estimation result of the index image to obtain a single view depth estimation result corresponding to the index image.

[0061] In operation S3, a neural window fully connected CRF model is pre-built, such as Figure 3 and Figure 4 As shown in the figure, the structure of the model is designed to adapt to the input index image features. The CRF model can consider the relationship between pixels, construct an energy function through the texture and position information of the image and the prediction results, and guide the depth estimation by optimizing the energy function. The index image is input into the model for depth estimation, and a preliminary depth map is output. Furthermore, the output depth map is post-processed, including smoothing and noise removal, to optimize the quality of the depth map, and techniques such as bilateral filtering are used to reduce the impact of noise.

[0062] Furthermore, the neural window fully connected conditional random field improved model includes: a convolutional neural network and a fully connected conditional random field model; the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image.

[0063] Furthermore, the basic idea of ​​image semantic segmentation is: first, identify the entire image, identify the main objects in the index image and their meaning and contextual relationship in the index image. At the same time, define semantic categories for index image features and analysis targets, such as walls, windows, etc. Then, refine and distinguish each object in the image and accurately classify them into corresponding semantic categories.

[0064] Considering that the depth similarity of objects in the same semantic category is more relevant, the proximity of object positions is fully considered in the segmentation process to achieve better constraints in the depth fusion process. At the same time, distinguishing the foreground from the background and identifying the boundaries of different objects are also critical in the image semantic segmentation process. In addition, attention to image details such as texture, color, shape, and consideration of the overall context are also essential to ensure the accuracy and consistency of semantic classification. Figure 5 A schematic diagram of image semantic division provided in Example 1 of the present invention.

[0065] The first factor is the distance factor, which is calculated as follows: The main consideration is the distance between pixels (i, j) and (u, v), and the distance is used to represent the degree of proximity. Because in most cases, the closer the distance between pixels, the closer their depth values ​​are. The parameter σ1 is used to control the neighborhood radius of each point.

[0066] The second factor It is a gradient factor, which is mainly used to measure the similarity of depth gradients and assign greater weights to similar gradients: in and Represent the depth gradients in the x and y directions at pixel (i, j), and Represents the depth gradient in the x and y directions at the pixel (u, v). The parameter σ2 is used to avoid very similar or identical gradients from having extremely high weights.

[0067] The third factor and the fourth factor are plane factors, representing the coplanarity in the x and y directions, respectively, and are used to strengthen the influence between points in the same plane. Pixels belonging to the same plane are given a higher weight:

[0068]

[0069] Among them, σ3 sets the minimum weight to avoid and The weights disappear when combined.

[0070] It should be noted that the parameters σ1, σ2, and σ3 are used to balance the three factors of distance, gradient, and coplanarity to ensure the continuity of the fusion results.

[0071] Further, S5 includes: S51: using the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor to normalize to obtain the interpolation weight S52: Using formula The single-view depth estimation result and the multi-view depth estimation result are image-fused to obtain the depth fusion image; wherein Ω is the pixel set of the semantically divided area to which the pixel (i, j) in the index image belongs, (u, b) represents the other pixels in the pixel set Ω except (i, j), and m uv and uv are the multi-view depth estimation results and single-view depth estimation results of pixels (u, v), s ij is the single view depth estimation result of pixel (i, j), f ij is the fusion depth corresponding to (i, j).

[0072] Let s represent the single-view depth estimation result of image I, and m represent the single depth map calculated by the multi-view depth estimation algorithm for the current image I. For each pixel (i, j) of the semantically divided image I, the fused depth estimation result f ij Calculation:

[0073]

[0074] Where Ω is the pixel set of the semantic partition area to which pixel (i, j) belongs, (u, v) represents the other pixels in Ω except (i, j), and m uv and uv are the multi-view depth estimation results and single-view depth estimation results of pixels (u, v), s ij is the single view depth estimation result of pixel (i, j), is the interpolation weight; the above formula converts the depth gradient (s ij -s uv ) is added to each multi-view depth m uv In the example, the structure of s is used for each m uv Create a depth map and then weight it using pixel-based weights.

[0075] The single-view-multi-view depth fusion algorithm based on semantic partitioning, under semantic constraints, comprehensively considers distance, gradient and plane factors to fuse the single-view depth estimation results of a single image containing special areas with the multi-view depth estimation results, which can improve the accuracy and consistency of the depth map as a whole.

[0076] Further, S51 includes: using the formula The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; among them, Pixel (i, j), pixel (u, v), pixel (p, k) and pixel (g, h) are different points in the pixel set Ω, n s ∈[0, 1], is the product of three weights corresponding to (i, j) and (u, v), is the product of the three weights corresponding to (i, j) and (g, h), It is the product of three weights corresponding to (i, j) and (p, k).

[0077] Since the depth values ​​in the same semantic region usually change smoothly, the interpolation weight For each pixel (u,v)∈Ω, that is, the pixel with the same local structure as the pixel (i,j), the weight is calculated according to the local image structure. Each weight is calculated according to the product of four different factors: distance factor, gradient factor and two plane factors. The product of these four factors generates a non-normalized weight for each pixel in the semantic partition area Ω of the image.

[0078] After calculating all the factors, since each pixel (i, j) is affected by all pixels in the image semantic partition area Ω, the pixel weights are normalized so that the sum of all weights on the pixel (i, j) is 1: The normalized weights expand the local influence to the entire image semantic division range. The fused depth estimate of each pixel (i, j) is the weighted fusion of the single-view depth and multi-view depth within its semantic range. ij Through the above strategies, the depth of special areas can be better estimated.

[0079] Operation S6: Propagate the depth information of the depth fusion map to other perspectives by reprojection to achieve global consistency of depth data in special areas. In this step, the reprojection technology is used to map the information of the depth fusion map to other perspectives to ensure the consistency of the depth data.

[0080] Furthermore, S6 includes: projecting the depth information of the depth fusion map to other perspectives by combining perspective projection and bundle adjustment; performing a depth consistency check on the re-projected depth information, and performing depth adjustments on pixels with deviations to ensure the consistency of the depth information of the current scene between multiple perspectives.

[0081] Among them, reprojection propagates the information of the three-dimensional points observed from one perspective to other perspectives, ensuring the accuracy and consistency of the depth information between different perspectives. The reprojection depth propagation process is: using the improved depth fusion image as the source of depth information, projecting each pixel in the depth map under this perspective into the world coordinate system and converting it into three-dimensional space coordinates according to the internal and external parameters of the camera, and then reprojecting it to the target perspective according to the camera parameters of the target perspective to complete the depth propagation.

[0082] In this embodiment, operation S6 includes sub-operation S61 to sub-operation S63.

[0083] Step S61, selecting the viewing angle to be reprojected, and preparing the camera parameters and depth information required for the reprojection process.

[0084] Step S62, using a reprojection algorithm to propagate the depth information in the depth fusion image to the selected viewing angle, and calculating the depth value after reprojection.

[0085] Specifically, the coordinates (x, y) of the pixels in image I are converted from the image coordinate system to the camera coordinate system:

[0086]

[0087] Where (x c ,y c ) is the coordinate value of the pixel (x, y) in the camera coordinate system, z c and D(x,y) is the depth value of the pixel, (f x ,f y ,v0,v0) is the camera intrinsic parameter of image I.

[0088] Convert the coordinates in the camera coordinate system to the world coordinate system, and assume that the camera external parameters corresponding to the image I are the rotation matrix R and the translation vector T:

[0089]

[0090] Where (x w ,y w ,z w ) represents the coordinate value of the pixel in the world coordinate system.

[0091] Reproject image I from the world coordinate system to the camera coordinate system of image I′. Use P wRepresents the coordinates of the point in the world coordinate system (x w ,y w ,z w ), P′ c Represents the point coordinates (x′) projected to the camera coordinate system of image I′ c ,y′ c ), z′ c The corresponding depth is:

[0092] P c ′=R′*(P w -T′);

[0093] Where R′ and T′ correspond to the rotation matrix and translation vector of image I′ respectively. Finally, the coordinates are converted to the image coordinate system:

[0094]

[0095] Where (x′, y′) is the coordinate of the pixel (x, y) on image I after being projected to the I′ viewing angle, and D′(x′, y′) is the depth value after projection.

[0096] Step S63 , performing a depth consistency check on the re-projected depth information to optimize and correct the depth value to ensure its accuracy and consistency under different viewing angles.

[0097] Epipolar geometry provides geometric constraints between two camera perspectives. Given the position of a point in one view, the epipolar line constrains the possible position of this point in another view, providing accurate geometric constraints for the propagation of depth information. Figure 6 As shown, O1 and O2 represent the optical centers of the left and right cameras respectively, the line between the optical centers O1O2 is called the baseline, point P represents the object space point, p and p' represent the projection points of point P on the left and right imaging planes respectively, and the plane formed by point P and O1 and O2 is the epipolar plane. The intersection of the epipolar plane and the imaging plane is called the epipolar line, that is Figure 6 If the projection point of the known spatial point P on the left imaging plane is p, according to the polar geometry constraint, the projection point of the point P on the right imaging plane is located on the intersection of the polar plane and the right imaging plane, that is, the polar line l'.

[0098] Based on this, the accuracy of the projection can be determined by verifying whether the reprojected coordinates (x′, y′) obtained by the above solution are on the epipolar line. For pixels with deviations, their positions are adjusted by minimizing the distance from the point to the epipolar line.

[0099] To ensure accurate alignment and consistency of depth information from different perspectives, depth consistency check and fusion are also required. Depth consistency check can identify conflicts caused by error estimation, perspective differences or scene complexity, ensure consistency between depth information from multiple perspectives, and generate smoother and continuous surfaces.

[0100] Example 2

[0101] The present embodiment provides a scene depth estimation device, including: an acquisition module, a determination module, a calculation module, a division module, a fusion module and a projection module; wherein the acquisition module is used to acquire a multi-view image of the current scene and calculate a multi-view depth estimation result corresponding to the multi-view image; the determination module is used to select a single-view image with visual deviation from the multi-view image as an index image; the calculation module is used to calculate the single-view depth estimation result corresponding to the index image; the division module is used to perform semantic division on the index image to obtain the weights of each semantic factor within the semantic range; the semantic factors include distance factors, gradient factors and coplanar factors; the fusion module is used to use the weights of each semantic factor to perform image fusion on the single-view depth estimation result and the multi-view depth estimation result to obtain a depth fusion map; the projection module is used to project the depth fusion map to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0102] Example 3

[0103] This embodiment provides an image processing device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0104] Example 4

[0105] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0106] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A scene depth estimation method, characterized in that: include: Obtaining a multi-view image of the current scene, and calculating a multi-view depth estimation result corresponding to the multi-view image; Selecting a single-view image with visual deviation from the multi-view images as an index image; Calculating a single-view depth estimation result corresponding to the index image; Performing semantic division on the index image to obtain weights of various semantic factors within a semantic range; The semantic factors include distance factor, gradient factor and coplanar factor; Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result by using the weights of the semantic factors to obtain a depth fusion image; The depth fusion image is projected to each other perspective except the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

2. The scene depth estimation method according to claim 1, characterized in that: The calculating the single-view depth estimation result corresponding to the index image includes: Inputting the index image into a neural window fully connected conditional random field improved model to obtain a preliminary depth estimation result of the index image; Denoising is performed on the preliminary depth estimation result of the index image to obtain a single-view depth estimation result corresponding to the index image.

3. The scene depth estimation method according to claim 2, characterized in that: The neural window fully connected conditional random field improved model includes: a convolutional neural network and a fully connected conditional random field model; the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image.

4. The scene depth estimation method according to claim 1, characterized in that: The weight of the distance factor is (i,j) and (u,v) represent different pixels; The weights of the gradient factors are: s ij and uv The single view depth estimation results of pixels (i, j) and (u, v) are respectively, and Respectively represent the depth gradient coefficients in the x and y directions at the pixel point; The coplanar factor weights include the vertical factor weights and level factor weights The parameters v1, σ2, and σ3 are used to balance the distance factor, gradient factor, and coplanar factor to ensure the continuity of the fusion result.

5. The scene depth estimation method according to claim 4, characterized in that: The calculation process of the depth fusion map includes: S51: Normalize the product of the distance factor weight, the gradient factor weight, and the coplanar factor weight to obtain the interpolation weight S52: Using formula Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result to obtain the depth fusion image; Wherein, Ω is the pixel set of the semantically divided region to which the pixel (i, j) in the index image belongs, (u, v) represents the other pixels in the pixel set Ω except (i, j), and m uv and uv are the multi-view depth estimation results and single-view depth estimation results of pixel (u, v), s ij is the single view depth estimation result of pixel (i, j), f ij is the fusion depth corresponding to (i, j).

6. The scene depth estimation method according to claim 5, characterized in that: The S51 comprises: using the formula The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; wherein, Pixel (i, j), pixel (u, v), pixel (p, k) and pixel (g, h) are different points in the pixel set Ω, n s ∈[0, 1], is the product of three weights corresponding to (i, j) and (u, v), is the product of the three weights corresponding to (i, j) and (g, h), It is the product of three weights corresponding to (i, j) and (p, k).

7. The scene depth estimation method according to any one of claims 1 to 6, characterized in that: The projecting the depth fusion image to each other perspective except the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene includes: The depth information of the depth fusion map is projected to other perspectives by combining perspective projection and bundle adjustment. A depth consistency check is performed on the reprojected depth information, and depth adjustment is performed on pixels with deviations to ensure consistency between depth information of the current scene at multiple viewing angles.

8. A scene depth estimation device, characterized in that: include: An acquisition module, used to acquire a multi-view image of the current scene and calculate a multi-view depth estimation result corresponding to the multi-view image; A determination module, configured to select a single-view image with visual deviation from the multi-view images as an index image; A calculation module, used to calculate a single-view depth estimation result corresponding to the index image; A division module, used for performing semantic division on the index image to obtain the weight of each semantic factor within a semantic range; The semantic factors include distance factor, gradient factor and coplanar factor; A fusion module, used to perform image fusion on the single-view depth estimation result and the multi-view depth estimation result by using the weights of the semantic factors to obtain a depth fusion image; A projection module is used to project the depth fusion image to each other perspective except the perspective corresponding to the index image, so as to obtain a consistent depth estimation result corresponding to the current scene.

9. An image processing device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images