A scene depth estimation method, device, image processing device and storage medium

Abstract

The present invention discloses a scene depth estimation method, device, image processing equipment and storage medium, belonging to the field of computer vision technology. The scene depth estimation method comprises: selecting a single image with occlusion, reflection or perspective in a multi-view image set for depth fusion, and propagating the improved depth information to other views by reprojection, thereby improving the reconstruction effect while reducing time overhead; performing semantic division on the index image to obtain the weight of each semantic factor; introducing single-view depth estimation to deal with the problem that feature points in special areas are difficult to match; under the constraint of semantic information, the single-view estimation result and the multi-view depth estimation result are fused, and propagated to other view angles by reprojection to obtain a consistent depth estimation result corresponding to the current scene; thus, accurate and consistent estimation of the depth of each area in the scene can be achieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, relates to a scene depth estimation method, apparatus, image processing device and storage medium. Background Art

[0002] With the rapid development of computer vision technology, 3D reconstruction technology has received more and more attention and has been widely used in many fields such as virtual reality, autonomous driving, and cultural heritage protection. Multi-view stereo vision reconstruction is one of the most commonly used methods in the field of 3D reconstruction. Figure 1 The consistency assumption is a key cornerstone of multi-view geometry theory. It states that when observing the same scene or object from different perspectives, the basic geometric structure and appearance attributes, such as texture and color, should be consistent, despite the different viewing angles. However, in the real world, objects often occlude each other or experience specular reflections and transmission, making it impossible to effectively and correctly recover feature points from certain perspectives. This can lead to missing or inaccurate information during reconstruction, especially for details that can only be observed from certain angles, making the reconstruction result inaccurately reflect the object's complete form.

[0003] In multi-view stereo vision reconstruction, the occlusion, specular reflection and transmission areas in the image violate the multi-view Figure 1 Therefore, existing methods find it difficult to accurately estimate the depth of various regions in the scene. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a scene depth estimation method, apparatus, image processing device and storage medium, the purpose of which is to solve the technical problem that the prior art is difficult to accurately and consistently estimate the depth of various areas in the scene.

[0005] To achieve the above object, according to one aspect of the present invention, a scene depth estimation method is provided, comprising:

[0006] Obtaining a multi-view image of the current scene, and calculating a multi-view depth estimation result corresponding to the multi-view image;

[0007] Selecting a single-view image with visual deviation from the multi-view images as an index image;

[0008] Calculating a single-view depth estimation result corresponding to the index image;

[0009] Performing semantic division on the index image to obtain weights of various semantic factors within a semantic range; the semantic factors include distance factors, gradient factors, and coplanar factors;

[0010] Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result using the weights of the semantic factors to obtain a depth fusion image;

[0011] The depth fusion image is projected to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0012] Furthermore, calculating the single-view depth estimation result corresponding to the index image includes:

[0013] Inputting the index image into a neural window fully connected conditional random field improved model to obtain a preliminary depth estimation result of the index image;

[0014] Denoising is performed on the preliminary depth estimation result of the index image to obtain a single-view depth estimation result corresponding to the index image.

[0015] Furthermore, the neural window fully connected conditional random field improved model includes: a convolutional neural network and a fully connected conditional random field model; the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image.

[0016] Furthermore, the weight of the distance factor is (i,j) and (u,v) represent different pixels;

[0017] The weights of the gradient factors are: s ij and s uv The single-view depth estimation results for pixels (i, j) and (u, v) are respectively, and Represent the depth gradient coefficients in the x and y directions at the pixel point respectively;

[0018] The coplanar factor weights include the vertical factor weights and level factor weights Parameters σ1, σ2, and σ3 are used to balance the distance factor, gradient factor, and coplanarity factor to ensure the continuity of the fusion results.

[0019] Furthermore, the calculation process of the depth fusion map includes:

[0020] S51: Normalize the weight of the distance factor, the weight of the gradient factor, and the weight of the coplanar factor to obtain the interpolation weight

[0021] S52: Using formula Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result to obtain the depth fusion image;

[0022] Where Ω is the pixel set of the semantic partition area to which the pixel (i, j) in the index image belongs, (u, v) represents the other pixels in the pixel set Ω except (i, j), and m uv and s uv They are respectively the pixel (u, v) multi-view depth estimation results and single-view depth estimation results, s ij is the single view depth estimation result of pixel (i, j), f ij is the fusion depth corresponding to (i, j).

[0023] Furthermore, the S51 includes: using the formula The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; wherein, Pixel (i, j), pixel (u, v), pixel (p, k) and pixel (g, h) are different points in the pixel set Ω, n s ∈[0, 1], is the product of three weights corresponding to (i, j) and (u, v), is the product of three weights corresponding to (i, j) and (g, h), It is the product of three weights corresponding to (i, j) and (p, k).

[0024] Furthermore, projecting the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene includes:

[0025] The depth information of the depth fusion map is projected to other perspectives by combining perspective projection and bundle adjustment.

[0026] A depth consistency check is performed on the re-projected depth information, and depth adjustment is performed on pixels with deviations to ensure consistency between the depth information of the current scene at multiple perspectives.

[0027] According to another aspect of the present invention, there is provided a scene depth estimation apparatus, comprising:

[0028] An acquisition module, used to acquire multi-view images of the current scene;

[0029] a determination module, configured to select a single-view image with visual deviation from the multi-view images as an index image;

[0030] A calculation module, configured to calculate a single-view depth estimation result corresponding to the index image;

[0031] A segmentation module, configured to perform semantic segmentation on the index image to obtain weights of various semantic factors within a semantic range; the semantic factors include distance factors, gradient factors, and coplanar factors;

[0032] a fusion module, configured to perform image fusion on the single-view depth estimation result and the multi-view depth estimation result using the weights of the semantic factors to obtain a depth fusion image;

[0033] A projection module is used to project the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0034] According to another aspect of the present invention, an image processing device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0036] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0037] (1) The present invention provides a scene depth estimation method, which selects a single-view image with visual deviation from the multi-view image as an index image; specifically, a single image with visual deviation such as occlusion, reflection or perspective is selected from the multi-view image set for depth fusion, and the improved depth information is propagated to other views by reprojection, thereby improving the reconstruction effect while reducing time overhead; the index image is semantically divided to obtain the weights of each semantic factor within the semantic range; the problem of difficult matching of feature points in special areas is handled by introducing single-view depth estimation; under the constraint of semantic information, the single-view estimation result and the multi-view depth estimation result are fused, and the depth fusion is propagated to other view angles by reprojection to obtain a consistent depth estimation result corresponding to the current scene. That is, the present application avoids using multi-view images of areas such as occlusion, mirror reflection and transmission in the image. Figure 1 The consistency assumption can achieve accurate and consistent estimation of the depth of each area in the scene.

[0038] (2) In this scheme, the index image is input into the improved model of the neural window fully connected conditional random field to obtain the preliminary depth estimation result of the index image; this process introduces a single-view depth estimation algorithm based on the neural window fully connected conditional random field, uses the conditional random field to construct an energy function, and then obtains depth prediction by optimizing the energy function, thereby improving the accuracy of single-view depth estimation.

[0039] (3) In this solution, the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image. The network structure is improved by concatenating the feature map with the original image and introducing skip connections to enhance the accuracy of single-view depth estimation.

[0040] (4) In this scheme, expressions of four semantic factors are set up. Under the semantic constraints, the distance, gradient and plane factors are comprehensively considered to fuse the single-view depth estimation results of a single image containing special areas and the multi-view depth estimation results, thereby improving the accuracy and consistency of the depth map as a whole.

[0041] (5) This scheme uses the formula The single-view depth estimation result and the multi-view depth estimation result are image fused; the above formula converts the depth gradient (s ij -s uv ) is added to each multi-view depth m uv In the example, the structure of s is used for each m uv Create a depth map and then weight it using pixel-based weights; low computational complexity and good fusion effect.

[0042] (6) This scheme uses the formula The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; the normalized weight expands the local influence to the entire image semantic division range, and the fused depth estimation of each pixel (i, j) is the weighted fusion of the single view depth and the multi-view depth within the semantic range to which it belongs. ij Through the above strategies, the depth of special areas can be better estimated.

[0043] (7) In this scheme, perspective projection and bundle adjustment are combined to project the depth information of the depth fusion map to other perspectives; the improved depth fusion map is projected to other perspectives using the internal and external parameters of the camera to ensure the global consistency of the depth data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of the scene depth estimation method provided in Example 1 of the present invention;

[0045] Figure 2 A schematic diagram of a scene depth estimation method provided in Example 1 of the present invention;

[0046] Figure 3 Schematic diagram of fully connected CRFs and windowed fully connected CRFs provided in Example 1 of the present invention;

[0047] Figure 4 A schematic diagram of the network structure of a single-view depth estimation model based on neural window fully connected conditional random field optimization provided in Example 1 of the present invention;

[0048] Figure 5 A schematic diagram of image semantic segmentation provided in Example 1 of the present invention;

[0049] Figure 6 This is a schematic diagram of epipolar geometry provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0051] Example 1

[0052] like Figure 1 and Figure 2 As shown, this embodiment provides a scene depth estimation method, including: steps S1-S6. Among them, S1: obtaining a multi-view image of the current scene, and calculating the multi-view depth estimation result corresponding to the multi-view image; S2: selecting a single-view image with visual deviation from the multi-view image as an index image; S3: calculating the single-view depth estimation result corresponding to the index image; S4: performing semantic division on the index image to obtain the weight of each semantic factor within the semantic range; the semantic factors include distance factor, gradient factor and coplanar factor; S5: using the weight of each semantic factor to fuse the single-view depth estimation result and the multi-view depth estimation result to obtain a depth fusion map;

[0053] S6: Project the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0054] Operation S1: Acquire multi-view image data of a scene and calculate multi-view depth estimation results corresponding to the multi-view images. The goal of this step is to acquire multi-view images of a scene by shooting from multiple angles to ensure comprehensive information coverage.

[0055] In operation S2, the quality of the acquired multi-view images can be evaluated, taking into account factors including viewing angle, clarity, lighting conditions and surface features of the object, and images containing special areas such as occlusion, mirror reflection, perspective, etc. are preferentially selected as index images; then, the most representative single-view image is selected from the multi-view image data as the index image to ensure that the image has sufficient details and recognizability for subsequent depth estimation.

[0056] It should be noted that there is no sequential requirement for steps S3 and S4; S3-S4 can be performed sequentially. Alternatively, semantic segmentation can be performed on the index image first to determine the weights of each semantic factor, and then the single-view depth estimation result corresponding to the index image can be calculated. The calculation of the multi-view depth estimation result corresponding to the multi-view image in S1 only needs to be performed before S4.

[0057] Operation S4: Semantically segment the index image based on the scene structure information. Semantically segment the index image using depth information and image features to ensure that the semantic information of each region is fully expressed.

[0058] Operation S5: Based on distance, gradient, and coplanarity factors within each semantic range, the single-view depth estimation result and the multi-view depth estimation result are fused to obtain a depth fusion image.

[0059] Operation S6: Propagate the depth information of the depth fusion image to other viewpoints through reprojection to achieve global consistency of depth data in specific areas. In this step, reprojection technology is used to map the information of the depth fusion image to other viewpoints to ensure the consistency of depth data from different viewpoints.

[0060] Furthermore, S3 includes: inputting the index image into the neural window fully connected conditional random field improved model to obtain a preliminary depth estimation result of the index image; denoising the preliminary depth estimation result of the index image to obtain a single-view depth estimation result corresponding to the index image.

[0061] In operation S3, a neural window fully connected CRF model is pre-built, such as Figure 3 and Figure 4 As shown in the figure, the model structure is designed to adapt to the input index image features. The CRF model considers the relationship between pixels and constructs an energy function using the image's texture and position information and the prediction results. This energy function is optimized to guide depth estimation. The index image is input into the model for depth estimation, outputting a preliminary depth map. The output depth map undergoes post-processing, including smoothing and noise removal, to optimize the depth map quality. Techniques such as bilateral filtering are used to reduce the impact of noise.

[0062] Furthermore, the improved neural window fully connected conditional random field model includes: a convolutional neural network and a fully connected conditional random field model; the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image.

[0063] Furthermore, the basic idea behind image semantic segmentation is to first identify the entire image, identifying the primary objects in the index image and their meaning and context within the index image. Semantic categories, such as walls and windows, are then defined for the index image features and analysis targets. Each object in the image is then refined and differentiated, accurately categorizing it into its corresponding semantic category.

[0064] Considering that the depth similarity of objects in close proximity within the same semantic category is more relevant, the proximity of objects is fully considered during segmentation to achieve better constraints during depth fusion. Distinguishing foreground from background and identifying the boundaries of different objects are also critical in the image semantic segmentation process. Furthermore, attention to image details such as texture, color, and shape, as well as consideration of the overall context, are essential to ensuring accurate and consistent semantic classification. Figure 5 This is a schematic diagram of image semantic segmentation provided by Example 1 of the present invention.

[0065] The first factor is the distance factor, which is calculated as follows: The main consideration is the distance between pixels (i, j) and (u, v), using distance to represent proximity. In most cases, the closer the distance between pixels, the closer their depth values. The parameter σ1 controls the neighborhood radius of each point.

[0066] The second factor It is a gradient factor, which is mainly used to measure the similarity of depth gradients and assign greater weights to similar gradients: in and Represent the depth gradients in the x and y directions at pixel (i, j), and Represents the depth gradient in the x and y directions at the pixel (u, v). The parameter σ2 is used to avoid very similar or identical gradients from having extremely high weights.

[0067] The third factor and the fourth factor is a plane factor representing the coplanarity in the x and y directions, and is used to strengthen the influence between points in the same plane. Pixels belonging to the same plane are given a higher weight:

[0068]

[0069] Among them, σ3 sets the minimum weight to avoid and The weights vanish when combined.

[0070] It should be noted that the parameters σ1, σ2, and σ3 are used to balance the three factors of distance, gradient, and coplanarity to ensure the continuity of the fusion results.

[0071] Furthermore, S5 includes: S51: using the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor to perform normalization to obtain the interpolation weight S52: Using formula The single-view depth estimation result and the multi-view depth estimation result are image-fused to obtain the depth fusion image; wherein Ω is the pixel set of the semantic partition area to which the pixel (i, j) in the index image belongs, (u, b) represents the other pixels in the pixel set Ω excluding (i, j), and m uv and s uv They are respectively the pixel (u, v) multi-view depth estimation results and single-view depth estimation results, s ij is the single view depth estimation result of pixel (i, j), f ij is the fusion depth corresponding to (i, j).

[0072] Let s represent the single-view depth estimation result of image I, and m represent the single depth map calculated by the multi-view depth estimation algorithm for the current image I. For each pixel (i, j) of the semantically segmented image I, the fused depth estimation result f ij Calculation:

[0073]

[0074] Among them, Ω is the pixel set of the semantic partition area to which pixel (i, j) belongs, (u, v) represents the other pixels in Ω except (i, j), and m uv and s uv They are respectively the pixel (u, v) multi-view depth estimation results and single-view depth estimation results, s ij is the single view depth estimation result of pixel (i, j), is the interpolation weight; the above formula converts the depth gradient (s ij -s uv ) is added to each multi-view depth m uv In the example, the structure of s is used for each m uv Create a depth map and then weight it using pixel-based weights.

[0075] The single-view-multi-view depth fusion algorithm based on semantic partitioning, under semantic constraints, comprehensively considers distance, gradient and plane factors to fuse the single-view depth estimation results of a single image containing special areas with the multi-view depth estimation results, which can improve the accuracy and consistency of the depth map as a whole.

[0076] Furthermore, S51 includes: using the formula Normalize the product of the weight of the distance factor, the weight of the gradient factor, and the weight of the coplanar factor; Pixel (i, j), pixel (u, v), pixel (p, k) and pixel (g, h) are different points in the pixel set Ω, n s ∈[0, 1], is the product of three weights corresponding to (i, j) and (u, v), is the product of three weights corresponding to (i, j) and (g, h), It is the product of three weights corresponding to (i, j) and (p, k).

[0077] Since the depth values ​​within the same semantic region usually change smoothly, the interpolation weight For each pixel (u,v)∈Ω, that is, the pixel with the same local structure as the pixel (i,j), the weight is calculated based on the local image structure. Each weight is calculated based on the product of four different factors: distance factor, gradient factor and two plane factors. The product of these four factors generates a non-normalized weight for each pixel in the semantic partition area Ω of the image.

[0078] After calculating all the factors, since each pixel (i, j) is affected by all pixels in the image semantic partition area Ω, the pixel weights are normalized so that the sum of all weights on the pixel (i, j) is 1: The normalized weights expand the local influence to the entire image semantic division range. The fused depth estimation of each pixel (i, j) is the weighted fusion of the single view depth and multi-view depth within the semantic range to which it belongs. ij Through the above strategies, the depth of special areas can be better estimated.

[0079] Operation S6: Propagate the depth information of the depth fusion image to other viewpoints through reprojection to achieve global consistency of depth data in special areas. In this step, reprojection technology is used to map the information of the depth fusion image to other viewpoints to ensure the consistency of depth data.

[0080] Furthermore, S6 includes: projecting the depth information of the depth fusion map to other perspectives using a combination of perspective projection and bundle adjustment; performing a depth consistency check on the re-projected depth information, and performing depth adjustments on pixels with deviations to ensure the consistency of the depth information of the current scene between multiple perspectives.

[0081] Reprojection propagates the information of 3D points observed from one perspective to other perspectives, ensuring the accuracy and consistency of depth information between different perspectives. The reprojection depth propagation process is as follows: using the improved depth fusion image as the source of depth information, each pixel in the depth map at that perspective is projected into the world coordinate system and converted into 3D space coordinates based on the intrinsic and extrinsic parameters of the camera. The depth propagation is then completed by reprojecting it to the target perspective based on the camera parameters of the target perspective.

[0082] In this embodiment, operation S6 includes sub-operations S61 to S63.

[0083] Step S61: Select the viewing angle to be reprojected and prepare the camera parameters and depth information required for the reprojection process.

[0084] Step S62: Use a reprojection algorithm to propagate the depth information in the depth fusion image to the selected viewing angle, and calculate the reprojected depth value.

[0085] Specifically, the coordinates (x, y) of the pixels in image I are converted from the image coordinate system to the camera coordinate system:

[0086]

[0087] Where (x c ,y c ) is the coordinate value of the pixel (x, y) in the camera coordinate system, z c and D(x,y) is the depth value of the pixel, (f x ,f y ,v0,v0) is the camera intrinsic parameter of image I.

[0088] Convert the coordinates in the camera coordinate system to the world coordinate system, and let the camera external parameters corresponding to the image I be the rotation matrix R and translation vector T:

[0089]

[0090] Where (x w ,y w ,z w ) represents the coordinate value of the pixel in the world coordinate system.

[0091] Reproject image I from the world coordinate system to the camera coordinate system of image I′. Use P wRepresents the coordinates of the point in the world coordinate system (x w ,y w ,z w ), P′ c Represents the point coordinates (x′) projected onto the camera coordinate system of image I′ c ,y′ c ), z′ c The corresponding depth is:

[0092] P c ′=R′*(P w -T′);

[0093] Where R′ and T′ correspond to the rotation matrix and translation vector of image I′ respectively. Finally, the coordinates are converted to the image coordinate system:

[0094]

[0095] Where (x′, y′) is the coordinate of the pixel (x, y) on image I after being projected to the perspective of I′, and D′(x′, y′) is the depth value after projection.

[0096] Step S63 : performing a depth consistency check on the re-projected depth information to optimize and correct the depth value to ensure its accuracy and consistency under different viewing angles.

[0097] Epipolar geometry provides geometric constraints between two camera perspectives. Given the position of a point in one view, the epipolar line constrains the possible position of this point in the other view, providing accurate geometric constraints for the propagation of depth information. Figure 6 As shown, O1 and O2 represent the optical centers of the left and right cameras respectively, the line between the optical centers O1O2 is called the baseline, point P represents the object space point, p and p' represent the projection points of point P on the left and right imaging planes respectively, and the plane formed by point P and O1 and O2 is the epipolar plane. The intersection of the epipolar plane and the imaging plane is called the epipolar line, that is, Figure 6 If the projection point of a known spatial point P on the left imaging plane is p, according to the epipolar geometry constraint, the projection point of point P on the right imaging plane lies on the intersection of the epipolar plane and the right imaging plane, i.e., the epipolar line l′.

[0098] Based on this, the accuracy of the projection can be determined by verifying whether the reprojected coordinates (x′, y′) obtained from the above solution are on the epipolar line. For pixels with deviations, their positions are adjusted by minimizing the distance from the point to the epipolar line.

[0099] To ensure accurate alignment and consistency of depth information from different viewpoints, depth consistency checking and fusion are required. Depth consistency checking can identify conflicts caused by misestimation, viewpoint differences, or scene complexity, ensuring consistency between depth information from multiple viewpoints and generating smoother, more continuous surfaces.

[0100] Example 2

[0101] This embodiment provides a scene depth estimation device, including: an acquisition module, a determination module, a calculation module, a division module, a fusion module and a projection module; wherein the acquisition module is used to acquire a multi-view image of the current scene and calculate the multi-view depth estimation result corresponding to the multi-view image; the determination module is used to select a single-view image with visual deviation from the multi-view image as an index image; the calculation module is used to calculate the single-view depth estimation result corresponding to the index image; the division module is used to perform semantic division on the index image to obtain the weights of each semantic factor within the semantic range; the semantic factors include distance factors, gradient factors and coplanar factors; the fusion module is used to use the weights of each semantic factor to fuse the single-view depth estimation result and the multi-view depth estimation result to obtain a depth fusion map; the projection module is used to project the depth fusion map to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

[0102] Example 3

[0103] This embodiment provides an image processing device, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0104] Example 4

[0105] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above method are implemented.

[0106] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A scene depth estimation method, characterized in that: include: Obtaining a multi-view image of the current scene, and calculating a multi-view depth estimation result corresponding to the multi-view image; Selecting a single-view image with visual deviation from the multi-view images as an index image; Calculating a single-view depth estimation result corresponding to the index image; Performing semantic division on the index image to obtain weights of various semantic factors within a semantic range; The semantic factors include distance factor, gradient factor and coplanar factor; Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result using the weights of the semantic factors to obtain a depth fusion image; Projecting the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene; The weight of the distance factor is ; and Indicates different pixels; The weights of the gradient factors are: ; and Pixels Single view depth estimation results, and Represents pixels The depth gradient coefficients in the x and y directions at The coplanar factor weights include the vertical factor weights and level factor weights ;parameter 、 and Used to balance distance factors, gradient factors, and coplanar factors to ensure the continuity of fusion results; S51: Normalize the product of the distance factor weight, the gradient factor weight, and the coplanar factor weight to obtain the interpolation weight ; S52: Using formula Performing image fusion on the single-view depth estimation result and the multi-view depth estimation result to obtain the depth fusion image; in, is the pixel set of the semantic partition area to which the pixel (i, j) in the index image belongs, and (u, v) represents the pixel set Excluding the other pixels except (i, j), and are the multi-view depth estimation results and single-view depth estimation results of pixel (u, v), is the single view depth estimation result of pixel (i, j), is the fusion depth corresponding to (i, j).

2. The scene depth estimation method according to claim 1, wherein: Calculating a single-view depth estimation result corresponding to the index image includes: Inputting the index image into a neural window fully connected conditional random field improved model to obtain a preliminary depth estimation result of the index image; Denoising is performed on the preliminary depth estimation result of the index image to obtain a single-view depth estimation result corresponding to the index image.

3. The scene depth estimation method according to claim 2, wherein: The neural window fully connected conditional random field improved model includes: a convolutional neural network and a fully connected conditional random field model; the convolutional neural network is used to extract the depth features of the index image; the fully connected conditional random field model is used to optimize the depth features to obtain a preliminary depth estimation result of the index image.

4. The scene depth estimation method according to claim 1, wherein: The S51 includes: using the formula The product of the weight of the distance factor, the weight of the gradient factor and the weight of the coplanar factor is normalized; wherein, ; Pixel (i, j), pixel (u, v), pixel (p, k) and pixel (g, h) are pixel sets Differences in n s ∈[0, 1], is the product of three weights corresponding to (i, j) and (u, v), is the product of three weights corresponding to (i, j) and (g, h), is the product of three weights corresponding to (i, j) and (p, k).

5. The scene depth estimation method according to any one of claims 1 to 4, characterized in that: The projecting the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene includes: The depth information of the depth fusion map is projected to other perspectives by combining perspective projection and bundle adjustment. A depth consistency check is performed on the re-projected depth information, and depth adjustment is performed on pixels with deviations to ensure consistency between the depth information of the current scene at multiple perspectives.

6. A scene depth estimation device, characterized in that: The method for estimating scene depth according to any one of claims 1 to 5 comprises: An acquisition module, configured to acquire a multi-view image of the current scene and calculate a multi-view depth estimation result corresponding to the multi-view image; a determination module, configured to select a single-view image with visual deviation from the multi-view images as an index image; A calculation module, configured to calculate a single-view depth estimation result corresponding to the index image; A segmentation module, configured to perform semantic segmentation on the index image to obtain weights of various semantic factors within a semantic range; the semantic factors include distance factors, gradient factors, and coplanar factors; a fusion module, configured to perform image fusion on the single-view depth estimation result and the multi-view depth estimation result using the weights of the semantic factors to obtain a depth fusion image; A projection module is used to project the depth fusion image to each perspective other than the perspective corresponding to the index image to obtain a consistent depth estimation result corresponding to the current scene.

7. An image processing device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images