A depth estimation method based on multi-view self-supervised learning

Abstract

The present application relates to the technical field of 3D computer vision, in particular to a depth estimation method based on multi-view self-supervised learning. A lightweight end-to-end self-supervised framework LS-MVSNet is adopted, a depth feature extraction module effectively reduces the number of model parameters and uses its adaptive ability to extract effective information from input images to predict depth maps, an edge feature extraction module obtains object edge information and combines it into the depth estimation process to improve the accuracy of depth estimation at the object edge; an end-to-end multi-view self-supervised depth estimation neural network model framework MVSDepth combines monocular depth estimation and multi-view stereo matching, uses the constraint of geometric prior to improve the scene depth estimation performance of the network, and uses a depth speed prior guide strategy to improve the efficiency of the model; in addition, a pose estimation module can predict the rotation matrix and translation matrix of the camera during operation, thereby making up for the deficiency of LS-MVSNet that needs to calibrate the camera external parameter in advance, and greatly improving the practicability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D computer vision, in particular to a depth estimation method based on multi-view self-supervised learning. BACKGROUND

[0002] With the rise of deep learning technology, depth estimation methods based on convolutional neural networks (CNN) have gradually become mainstream. Early work mainly focused on depth estimation using a single image, such as single-frame depth estimation based on monocular images and visual SLAM. With the development of hardware and datasets, depth estimation methods based on binocular and multi-view images have made significant progress, such as stereo matching and multi-view stereo reconstruction. In recent years, with the continuous evolution of deep learning models and the optimization of neural network architectures, depth estimation methods based on deep learning have made great breakthroughs in accuracy and efficiency.

[0003] Depth estimation can be divided into two categories from the training data, depth estimation based on supervised learning and depth estimation based on self-supervised learning. Depth estimation methods based on supervised learning require a large dataset containing depth ground truth images as training support. It is very difficult to obtain depth ground truth images, usually with the help of precise depth measurement equipment, which also causes high cost problems. At the same time, depth estimation methods based on supervised learning have poor generalization performance and are difficult to implement. Self-supervised learning of three-dimensional perception can fuse information from multiple views to predict scene depth information in the common area of the picture. Due to the strong feature extraction ability of neural networks, multi-view geometric depth estimation based on self-supervised learning has developed rapidly. In recent years, exciting and inspiring results have been achieved.

[0004] Multi-view stereo is an important and fundamental problem in 3D computer vision, whose goal is to reconstruct the 3D point cloud of a scene from a set of calibrated multi-view images. Traditional multi-view stereo mainly uses hand-crafted similarity metrics or rules to compute the matching degree between different view images, and recovers the three-dimensional point cloud according to the projection relationship. In recent years, with the development of deep learning technology, multi-view stereo methods based on deep learning have gradually become a research hotspot. Learning-based methods usually use convolutional neural networks to realize feature extraction, and construct a cost volume on the reference camera frustum, then predict the depth map under each view, and further reconstruct the 3D point cloud model according to the obtained depth map. They use large-scale 3D ground-truth data for training, and outperform traditional geometry-based methods in performance, occupying the leading position in the ranking list. However, learning-based methods are strongly dependent on the availability of 3D ground-truth data during training, and such data is not easy to obtain. Therefore, it has prompted the community to focus on self-supervised MVS methods. Recently, the number of self-supervised MVS methods has surged, transforming the depth estimation problem into an image reconstruction problem. The predicted depth map and the input image are used to synthesize an image on another view, so the self-supervised loss is constructed to measure the difference between the synthesized and real images on that view.

[0005] At present, some self-supervised methods have achieved similar or even comparable results to supervised methods. These self-supervised MVS methods are based on the photometric consistency assumption, which assumes that pixels belonging to the same 3D point have the same color properties under different viewing angles. However, in real-world environments, occlusion, reflection, non-Lambertian surfaces, and other variables can invalidate such an assumption. Some works have been done to eliminate the impact of non-Lambertian surfaces and other factors on the photometric consistency assumption, introducing additional information or constraints to assist the depth estimation process in addition to the photometric loss, such as using a pre-trained semantic feature extraction backbone network or neural rendering technology to enhance the model's ability to perceive scene structure and texture information, or generating more realistic and coherent synthetic images to reduce the impact of lighting changes. These means have indeed achieved significant results, but also brought high computational overhead and memory consumption.

[0006] A non-Lambertian surface refers to a surface that does not conform to the characteristics of a Lambertian surface, and is usually characterized by irregular behavior of absorbing, reflecting and transmitting light. A non-Lambertian surface can have the following characteristics: partially reflecting incident light rather than completely absorbing it; light incident in the same direction can also be scattered to different degrees; in some cases, light can be emitted spontaneously. It is precisely because of these characteristics of non-Lambertian surfaces that it is easy to cause matching errors in view matching at different viewing angles, resulting in inaccurate predicted depth information. Many researchers have proven in their work that monocular self-supervised depth estimation methods can effectively learn the depth information of weakly textured, reflective and other non-Lambertian surface regions, which is very challenging for multi-view stereo matching. However, the self-supervised monocular depth estimation model also has an inherent problem - the scale ambiguity problem. In the field of monocular self-supervised depth estimation, scale ambiguity is a common problem, which is mainly manifested in the lack of accuracy and consistency of depth estimation results, which is caused by the lack of external reference or calibration information. The scale ambiguity may be caused by changes in the size of objects in the scene, changes in viewing angle, or limitations of depth perception technology itself and other factors. This ambiguity will cause the depth estimation result to be unstable at different scenes or viewing angles, and cannot accurately reflect the size and distance relationship of objects in the real world. SUMMARY

[0007] In view of the above problems, the purpose of the present application is to provide a depth estimation method based on multi-view self-supervised learning, which adopts a lightweight end-to-end self-supervised framework LS-MVSNet and an end-to-end multi-view self-supervised depth estimation neural network model framework MVSDepth to construct an efficient and compact structure depth feature extraction module, effectively reducing the number of model parameters and using its adaptive ability to extract effective information from the input image to predict the depth map. The technical scheme is as follows:

[0008] A depth estimation method based on multi-view self-supervised learning, characterized in that it comprises the following steps:

[0009] Step 1: Construct a self-supervised depth estimation neural network model LS-MVSNet: the LS-MVSNet network model includes a depth feature extraction module, an edge feature extraction module and a data enhancement module, and takes one reference view and multiple source views as input;

[0010] Step 1.1: The depth feature extraction module adaptively extracts effective information from the input image to predict the depth map; specifically: based on the cost volume construction method of the depth feature extraction module, the constructed feature cost volume, and the depth convolutional neural network adaptively extracts the features suitable for stereo matching from the input image; the mechanism of the adaptive feature extraction window is adopted, so that the depth convolutional neural network can dynamically adjust the size and position of the feature extraction window according to the image content;

[0011] Step 1.2: The edge extraction module uses an edge detection algorithm to obtain object edge information and combines it into the depth estimation process to extract object contour features from the input image; specifically: first, use the Sobel operator to perform edge extraction on the input image to obtain a binary object edge image; then perform edge feature extraction on the obtained object edge image; finally, construct a feature cost volume based on the edge features;

[0012] Step 1.3: Add the feature cost volume constructed in the edge extraction module and the feature cost volume obtained by the depth feature extraction module directly, and then obtain the initial depth map through regression and interpolation, and obtain the final depth map through the refinement step;

[0013] Step 1.4: The data augmentation module simulates the occlusion situation existing in reality by generating a black mask under the reference view, and generates the corresponding occlusion mask under the source view through projection transformation combined with the predicted depth map;

[0014] Step 2: Depth estimation according to the depth estimation process

[0015] Determine the depth range of the pixel point under the condition that the camera motion speed and the rough depth value of the pixel point are known.

[0016] Further, in step 2, the depth estimation process specifically includes:

[0017] Step 2.1: Given a set of input images, including a reference view labeled as Ref and N source views labeled as Src i , where i ranges from 1 to N; the camera intrinsic parameters, rotation matrix and translation vector are represented by {K i ,R i ,t i}, where {K0,R0,t0} represents the camera parameters, rotation matrix and translation vector under the reference view Ref; after extracting the depth features from the input image, the reference view feature map F ref and the source view feature map F Srci are obtained;

[0018] Step 2.2: Add the feature map F SrciThe feature map under the perspective of the reference view Ref is converted into the reference view Ref through the following differentiable homographic projection formula

[0019]

[0020] In the formula, H i (d) is the homographic transformation factor of the i-th source view with a depth of d; t i represents the translation vector under the i-th source view, is a unit vector (0, 0, 1) T ;

[0021] For any pixel point x in the source view, the projection formula is used to convert it into the corresponding pixel point in the reference view ~ represents the projection equation;

[0022] Step 2.3: Reconstruct the feature cost volume on the reference camera frustum Using multiple feature cost volumes, a variance-based 3D cost volume C is constructed; the 3D cost volume C is regularized by a 3D U-Net, and the estimated depth D under the reference view is obtained by a Soft argmin operation; the formula is as follows:

[0023]

[0024] In the formula, represents the average value P(d) of all feature cost volumes, which is the probability of the depth being d; d max and d min represent the upper and lower limits of the scene depth;

[0025] Step 2.4: Calculate the sampling depth d:

[0026]

[0027] In the formula, M is the total number of samplings, and m ∈ * 0, 1, …, M+.

[0028] In the initial stage, the scene depth is uniformly sampled M times from d min to d max ; in the subsequent depth prediction refinement stage, the depth search range is further narrowed according to the coarse depth map; the initial step is uniformly sampled R times to obtain a fine step Δd;

[0029] Step 2.5: For any pixel point, its depth exploration is based on its coarse depth value to achieve accurate depth value prediction; it is represented as:

[0030]

[0031] In the formula, R is the sampling number of the initial step length, and the step length coefficient r is in the range of

[0032] Step 2.6: Given the corresponding intrinsic matrix K, the relative transformation matrix T from the reference view to the source view c , and the estimated depth D predicted in the reference view, any pixel point p i in the source view is converted to the pixel point p i ′ in the reference view by inverse transformation:

[0033] p i ′=KT c (D(p i )·K -1 p i )

[0034] In the formula, D(p i ) represents the depth value of the pixel point p i in the reference view;

[0035] Step 2.7: The inverse-transformed source view I i ′ is generated by bilinear sampling at the inverse-transformed pixel point p i ′, and a binary mask M i is randomly generated for each reference view to mask the invalid pixels projected outside the source image boundary; in the self-supervised multi-view stereo matching system, the N source views are all transformed to the perspective of the reference view to calculate the photometric consistency loss L PC :

[0036]

[0037] In the formula, represents the pixel-wise gradient, and represents the pixel-wise product; I r is the reference view, and ||·||1 and ||·||2 represent the l1 norm and the l2 norm.

[0038] Further, the loss function L of the self-supervised depth estimation neural network model framework LS-MVSNet in step 1 comprehensively considers photometric consistency, data augmentation consistency, structural similarity, and smoothness, as follows:

[0039] L=λ1L PC +λ2L DA +λ3L Smooth +λ4L SSIM

[0040]

[0041]

[0042] wherein L PC is a photometric consistency loss function, L DA is a data augmentation consistency loss function, L Smooth is a smoothness loss function, L SSIM is a structural similarity loss function; λ1, λ2, λ3 and λ4 are weights of respective loss functions; is a binary mask; D is a depth map predicted by a deep feature extraction module; is a depth map obtained by a data augmentation branch; μ x and μ y are luminance of images x and y, respectively; and are variance of images x and y, respectively, σ xy is a covariance of images x and y; C1 and C2 are constants added for stability; C1 = (k1L) 2 , C2 = (k2L) 2 , L is a dynamic range of pixel value, k1 and k2 are parameter adjustment variables; N is a number of pixels in a depth image, is a neighborhood set of pixel i; D i and D j are depth values at pixels i and j, respectively.

[0043] Further, on the basis of depth estimation by the self-supervised depth estimation neural network model LS-MVSNet, step 3 is further included:

[0044] a monocular moving camera-based depth estimation neural network model MVSDepth is constructed;

[0045] The MVSDepth network model includes a pose estimation module, a monocular depth estimation module and a multi-view stereo matching depth estimation module;

[0046] The pose estimation module estimates the relative extrinsic parameters of the camera between image frames, i.e. rotation matrix and translation matrix, through time-series images under free viewpoint;

[0047] The monocular depth estimation module uses time-continuous image frames as training signals for depth prediction and estimates the camera poses between consecutive image frames, simultaneously solving the depth values in the image scene and the camera poses between consecutive image frames;

[0048] The pose estimation network is trained using time-series image frames, and the current image frame and the predicted depth map are combined to couple the training, thereby realizing the depth estimation training of the self-supervised neural network;

[0049] The multi-view stereo matching depth estimation module adopts the framework of a self-supervised depth estimation neural network model LS-MVSNet, and after receiving prior information from the pose estimation module and the monocular depth estimation module, a more accurate depth map Dmvs is predicted.

[0050] Further, the pose estimation module in step 2 includes a pose estimation encoder and a pose estimation decoder.

[0051] The pose estimation encoder adopts a convolutional neural network, accepts multiple image frames as input, and the number of image frames is InputFrames; the image frames input into the encoder module first pass through a feature extraction module composed of 7 layers of convolutional layers and activation functions to complete camera pose feature extraction, and the extracted camera pose feature information is sent to the Poseconv convolutional layer; finally, the output is adjusted to extract the rotation and translation information of the camera.

[0052] The pose estimation decoder is composed of a series of convolutional layers, used to decode the camera pose features extracted from the encoder module, and finally obtain the 6-degree-of-freedom relative pose corresponding to the input image frames, i.e. the rotation matrix and the movement matrix.

[0053] Further, the monocular depth estimation module in step 2 includes a depth estimation encoder and a depth estimation decoder.

[0054] The depth estimation encoder includes multiple sub-modules, each of which includes a convolutional layer, a BatchNorma layer, a ReLU activation function, and a pooling layer; the convolutional layer learns multiple types of features from low to high levels; the BatchNorma layer integrates the learned feature distribution to speed up the convergence of the network; the pooling layer reduces the size of the feature map while increasing the receptive field of the convolutional layer;

[0055] The depth estimation decoder adopts a combination of upsampling layers and convolutional layers, gradually decodes low-resolution feature maps into depth maps with the same size and resolution as the input image through deconvolution and upsampling operations; it includes 4 identical up-convolution structures, each of which includes a feature map fusion, a deconvolution, and an upsampling process; the input of each up-convolution structure is the output of the previous layer network and the feature map of the same size in the depth estimation encoder network; using a skip connection method, the feature maps from different levels of the depth estimation encoder are fused with the feature maps of the corresponding levels of the depth estimation decoder, and information is fused from feature maps of different scales.

[0056] Further, the specific process of step 3 includes:

[0057] Step 3.1: input the continuous image frames at time T and time T-1 into the pose estimation network to obtain the relative extrinsic parameter matrix between the two images, i.e. the rotation matrix R m and the translation matrix T m between T-1→T;

[0058] Step 3.2: input the image frame at time T into the depth estimation network to obtain the corresponding depth map D mono , and then use the projection formula to project the image frame at time T to time T-1 to obtain the corresponding image:

[0059] I T→T-1 = I T <proj(D mono ,[R m ,T m ] T→T-1 ,K)>

[0060] In the above formula, K is the camera intrinsic matrix; proj(·) represents the re-projection function of the projected depth map D mono two-dimensional pixel coordinates; <·> is the sampling operator; I T→T-1 and I T represent the image obtained by projecting the image frame at time T to time T-1 and the image at time T itself, respectively; D mono is the monocular depth map predicted by the monocular depth estimation module;

[0061] Step 3.3: the monocular depth estimation network uses continuous image frames in the training process, and the reconstruction loss function for each pixel point is calculated. The minimum value of the pixel-by-pixel photometric loss function is ultimately a loss function, as follows:

[0062]

[0063] In the formula, pe(·) is the reconstruction loss function; I a and I b are any images, L SSIM is a structural similarity loss function; α is a weight coefficient set by experience; ||·||1 is the l1 norm; Loss p is the minimum reconstruction loss function; I T and I T+n→T are the image at time T and the image at time T+n, respectively; n is -1 or 1;

[0064] Step 3.4: the rotation matrix R m and the translation matrix T m obtained by the pose estimation module are combined with the monocular depth map D monoAs a known condition, more accurate and efficient depth prediction is realized in combination with a multi-view stereo matching process:

[0065] p t-1 =K·(R m ·(K -1 ·p t ·d pt )+T m )

[0066] In the formula, K is the intrinsic matrix of the camera; p t and p t-1 are a pixel point on the image frame at time t and a pixel point corresponding thereto on the image frame at time t-1 respectively; d pt represents the cross-sectional depth of the pixel point p in the image frame at time T in the frustum;

[0067] Step 3.5: A depth velocity prior guiding strategy is adopted to determine the sampling depth range of the pixel point under the condition that the camera motion velocity and the rough depth value of the pixel point are known.

[0068] Further, the step 3.5 specifically includes:

[0069] The triangulation is associated with the camera motion velocity, so that the video frame collected when the camera moves at high speed has triangulation prior, the depth range is expanded on the basis of the rough depth value, and then the multi-view stereo matching is used for depth regression, and the depth sampling range is as follows:

[0070]

[0071] In the formula, d is the sampling depth; d′ max and d′ min are the upper and lower limits of the sampling depth, and satisfy the following conditions:

[0072]

[0073] In the formula, D mono is the monocular depth estimation module predicted monocular depth map;

[0074] The specific values of d′ min and d′ max are calculated by the following formula:

[0075]

[0076]

[0077] In the formula, v is the camera motion velocity, v=α||T m ||2, T mis the camera translation matrix estimated by the pose estimation network, and is the camera frame rate; is a hyperparameter; is a scale function that converts the camera motion velocity v to the actual world scale.

[0078] Further, the depth sampling value is calculated by using the calculation formula of inverse depth sampling:

[0079]

[0080] In the formula, d j is the depth sampling value, and the subscript j=0, 1, 2,..., D0-1, and D0 is the depth sampling range.

[0081] The beneficial effects of the present application are:

[0082] 1) The present application proposes LS-MVSNet, a lightweight end-to-end self-supervised framework; a high-efficiency and compact structure depth feature extraction module is designed, which can effectively reduce the number of model parameters and utilize its adaptive ability to extract effective information from the input image to predict the depth map; it also includes an edge feature extraction module, which uses edge detection algorithm to obtain object edge information and combines it into the depth estimation process to improve the accuracy of depth estimation at the object edge. Meanwhile, data enhancement is used to improve the robustness and generalization of the model.

[0083] 2) The present application proposes an end-to-end multi-view depth estimation network model based on self-supervised learning, namely MVSDepth; the design innovation of the network model lies in the combination of the efficiency of monocular depth estimation and the geometric constraint of multi-view stereo matching; by effectively combining monocular depth estimation with multi-view stereo matching, the network's scene depth estimation performance is improved by using the geometric prior constraint, and the model's efficiency is improved by using the depth speed prior guidance strategy; in addition, the model also includes a pose estimation module, which can predict the rotation matrix and translation matrix of the camera during operation, thereby making up for the deficiency of LS-MVSNet that needs to calibrate the camera external parameter in advance, greatly improving the practicability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 It is a flow chart of the self-supervised depth estimation neural network model framework LS-MVSNet.

[0085] Figure 2 It is a depth feature extraction schematic diagram.

[0086] Figure 3 It is a depth estimation neural network model framework MVSDepth structure schematic diagram based on monocular moving camera.

[0087] Figure 4A structure diagram of a camera pose estimation network.

[0088] Figure 5 A structure diagram of an overall structure of a monocular depth estimation model. DETAILED DESCRIPTION

[0089] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0090] 1. A self-supervised depth estimation network model LS-MVSNet

[0091] The lightweight self-supervised multi-view stereo framework LS-MVSNet of the application is shown in Figure 1 The input of the network includes a reference view and multiple source views, which are directly sent to the main branch (i.e., the depth feature extraction module). After a series of operations such as feature extraction, homography transformation, cost volume construction, and cost volume regression, depth prediction can be achieved. After the input images are extracted, the edge profiles are sent to the edge feature extraction branch. The edge feature extraction branch and the main branch in the network model have approximately the same structure. The feature cost volume constructed in this branch and the feature cost volume obtained by the main branch are vectors with the same dimension. After direct addition, the initial depth map can be obtained through regression and interpolation. After the refinement step, the final depth map can be obtained. The data augmentation module simulates the occlusion that may exist in reality by randomly generating a black mask under the reference view. Combined with the depth map predicted by the main branch, the corresponding occlusion mask can be generated under the source view through projection transformation. This model combines the depth feature extraction module, the edge feature extraction module, and the data augmentation technology, which significantly reduces the computational overhead and memory consumption while ensuring the accuracy and robustness of depth map prediction. In this embodiment, the depth feature extraction module is used to adaptively extract effective information from the input image for depth prediction. The edge extraction module is used to extract the object contour features of the input image to improve the accuracy of depth estimation at the edge.

[0092] (1) Depth feature extraction

[0093] In order to obtain high-quality stereo matching results, this embodiment proposes a cost volume construction method based on the depth feature extraction module. This method uses a depth convolutional neural network to adaptively extract features suitable for stereo matching from the input image, making the final predicted depth map more accurate and robust. Since the image contains a wealth of information, only part of the information is useful for stereo matching. Therefore, this embodiment designs a mechanism for adaptively extracting feature windows, so that the network can dynamically adjust the size and position of the feature extraction window according to the image content, as shown in Figure 2As shown, there may be differences in the selection of associated pixel regions in different texture feature regions. These squares represent pixels, with the green pixel indicating the selected pixel. The features extracted from this pixel may be obtained by combining the surrounding colorless pixels. The depth feature extraction module can effectively combine the contextual information of pixels to enhance the feature extraction capability of depth information. The depth feature extraction module in this embodiment consists of a four-layer 2D convolutional neural network. All layers have a stride of 1 and an output channel of 4. Except for the last layer, each convolutional layer is followed by batch normalization and rectified linear units. More detailed parameter information is shown in Table 1. H and W represent the height and width of the input image.

[0094] Table 1. Detailed parameters of the edge feature extraction layer.

[0095]

[0096] After the input image passes through the feature extraction module, the output is a feature map containing all information about depth prediction. The depth feature extraction module proposed in this invention can extract enough information to estimate the depth map. At the same time, this method effectively reduces the number of parameters in the network, which not only improves inference speed but also saves computational resources.

[0097] (2) Edge feature extraction

[0098] To improve the quality of 3D object reconstruction, accurate reconstruction of object edges is crucial. In this regard, this invention proposes a novel contour feature extraction module that first performs an edge detection algorithm on the original image, and then extracts information to generate a feature map. Notably, the edge feature extraction module operates independently of the depth feature extraction module; there is no weight sharing between the two during training. Figure 1 As shown, the network model framework proposed in this embodiment has two feature extraction modules. The first feature extraction module consists of yellow and blue components. This module is used in both the main branch and the data augmentation branch, indicating that these two branches share the same feature extraction module and can share weights. This saves a significant amount of model parameters and computational resources, and reduces the complexity of the model. On the other hand, the module used for edge feature extraction consists of green and orange components. The depth map generated from the edge feature extraction module enables the construction of a cost volume based solely on object edges, thereby ensuring more accurate matching at these edges.

[0099] Specifically, first, edge extraction is performed on the input image using the Sobel operator to obtain a binary object edge image. Then, feature extraction is performed on the obtained object edge image. Feature extraction using the object edge image can effectively reduce matching errors caused by weak texture regions. It helps the network to predict relatively accurate depth values on the object contour, thereby helping the overall depth estimation. Subsequent operations are the same as the main branch, and an edge feature cost volume is constructed in the same way based on edge features. The edge feature cost volume is combined with the regular cost volume. As shown in FIG. 3B, the cost volume in the main branch and the cost volume in the edge feature extraction branch are composed of tensors with the same dimensions, and they are added with a weight ratio of 1:1. Figure 1

[0100] (3) Data augmentation

[0101] The objective of the present application is to strengthen the self-supervised data augmentation consistency by comparing the output of the original data and the augmented sample as a kind of regularization. Given the epipolar constraint between different views in the LS-MVSNet framework, the comprehensive data augmentation method should not change the spatial position of the pixels. This means that during data augmentation, the pixel relationship should be maintained to ensure consistency with the geometric constraints between views. In order to simulate the occlusion scene in a multi-view context, a binary clipping mask is randomly generated to occlude certain areas on the reference view. In the data augmentation module, the depth map predicted in the depth feature extraction module is used as the ground truth. Then, this occlusion mask is projected onto other source views through epipolar transformation, thereby occluding the corresponding areas in the source views.

[0102] (4) Depth estimation process

[0103] Given a set of input images, including one reference view labeled as Ref and N source views labeled as Src i , where i ranges from 1 to N. The intrinsic parameters, rotation matrices, and translation vectors are represented by {K i ,R i ,t i}, where {K0,R0,t0} represents the camera parameters, rotation matrices, and translation vectors under the reference view Ref. After extracting the depth features from the input images, the reference view feature map F ref and the source view feature map F Srci are obtained.

[0104] The feature map F Srci of the source view can be converted to the feature map of the reference view Ref perspective by the following differentiable homographic projection formula

[0105]

[0106] ​where H i (d) is the homography transformation factor of the i-th source view with depth d; t i denotes the translation vector in the i-th source view, is the unit vector (0, 0, 1) T .

[0107] Specifically, for any pixel point x in the source view, it can be converted to the corresponding pixel point in the reference view using the projection formula denotes the projection equation. Then the feature cost volume is reconstructed on the reference camera frustum Using multiple feature cost volumes, a variance-based 3D cost volume C can be constructed, and then regularized by a 3D U-Net. Finally, the estimated depth D in the reference view is obtained by the Soft argmin operation.

[0108]

[0109]

[0110] where P(d) represents the probability of the depth d, which is the average value of all feature cost volumes; d max and d min denote the upper and lower bounds of the scene depth.

[0111] The experimental results prove that the variance-based cost metric performs better than the traditional multi-view stereo matching method when aggregating the pairwise cost between the reference view and all source views. In a heuristic way, the variance-based cost metric reduces the impact of illumination reflections by averaging, which can cause the RGB values of the same region to change in different views. In addition, non-Lambertian surfaces can cause occlusion effects, i.e., some regions in the reference view are visible in other source views, which affects the accuracy and efficiency of stereo matching. The homography transformation formula also has an advantage for depth estimation. Compared with the traditional method of first estimating the disparity and then converting it to a depth value, the homography transformation formula can effectively improve the accuracy of depth estimation, especially in scenes with significant depth changes. This is because there is a non-linear inverse relationship between disparity and depth value, where smaller disparities correspond to larger depths, and larger disparities correspond to smaller depths. If only disparity estimation is considered without considering camera parameters, accurate depth can be obtained for close-range scenes, but larger errors can occur for long-range scenes. In other words, under the same disparity error, close-range and long-range scenes will produce large deviations in depth.

[0112] The sampling depth d is represented as:

[0113] ​

[0114] where M is the total number of uniform samplings.

[0115] This embodiment adopts a coarse-to-fine approach to predict the scene depth map. In the initial stage, the scene depth is uniformly sampled M times from d min to d max .

[0116] In the subsequent depth prediction refinement stage, the depth search range is further narrowed down according to the coarse depth map. The initial step is uniformly sampled R times to obtain the fine step Δd. For any pixel p, its depth exploration is based on its coarse depth value, thus achieving a more accurate depth value prediction.

[0117]

[0118] where R is the number of initial step samplings, and the step coefficient r is in the range of

[0119] To supervise the prediction of the depth map without a real depth map, the prior art utilizes the photometric consistency between the reference view and other source views. The main concept of these methods is to increase the similarity between the reference view and any source view back-projected to the reference view. Given the corresponding intrinsic matrix K, the relative transformation matrix T c from the reference view to the source view, and the estimated depth D predicted in the reference view, any pixel p i in the source view can be converted to the reference view by inverse transformation.

[0120] p i ′=KT c (D(p i )·K -1 p i )

[0121] where D(p i ) represents the depth value of pixel p i in the reference view.

[0122] The transformed source view I′ i is generated by bilinear sampling at the transformed pixel p′ i . In addition, a binary mask M i is randomly generated for each reference view in the experiment to mask the invalid pixels projected outside the source view boundary. In the self-supervised multi-view stereo matching system, all N source views are transformed to the perspective of the reference view to calculate the photometric consistency loss L PC .

[0123]

[0124] In the formula, Represents pixel-wise gradient, ⊙ represents pixel-wise product; I r For reference view, ||·||1 and ||·||2 represent l1 normal form and l2 normal form, respectively.

[0125] (5) Loss function design

[0126] like Figure 1 The end-to-end self-supervised depth estimation neural network model framework LS-MVSNet, as shown, has a depth estimation branch responsible for extracting feature information from the input image using a depth feature extraction module, and then predicting the depth map. An edge feature extraction branch is responsible for extracting feature information at object edges using an edge feature extraction module to enhance the reconstruction quality of object edges. Furthermore, a data augmentation branch is responsible for performing random transformations on the input image using a data augmentation module, while ensuring that the transformed image remains geometrically and photometrically consistent with the original image, thereby improving the model's generalization ability and robustness. This invention designs a loss function that comprehensively considers photometric consistency, data augmentation consistency, structural similarity, and smoothness to optimize network parameters.

[0127] L=λ1L PC +λ2L DA +λ3L Smooth +λ4L SSIM

[0128]

[0129] In the formula, L PC Let L be the photometric consistency loss function. DA For the data augmentation consistency loss function, L Smooth For the smoothness loss function, L SSIM Let λ1, λ2, λ3, and λ4 be the structural similarity loss function; λ1, λ2, λ3, and λ4 are the weights of each loss function. D is the binary mask; D is the depth map predicted by the depth feature extraction module. The depth map obtained for the data augmentation branch; μ x and μ y These represent the brightness of the image at x and y, respectively; and The variances of the images x and y are σ and y, respectively. xy Let C1 be the covariance of the images x and y; C2 and C1 are constants added for stability; C1 = (k1L) 2 C2 = (k2L) 2 L represents the dynamic range of pixel values, k1 and k2 are parameter adjustment variables; N is the number of pixels in the depth image. is the set of neighbors of pixel i; D i and D j are the depth values at pixels i and j, respectively.

[0130] data augmentation consistency loss L DA The data augmentation consistency is ensured by minimizing the difference between the depth map predicted by the main branch and the depth map predicted by the data augmentation branch in the effective area. Smooth The smoothness loss L SSIM is used to ensure the smoothness of the predicted depth map, which aims to minimize the discontinuity in the depth map, so as to obtain a more continuous and smooth depth map. The structural similarity loss L SSIM is used to ensure that the image obtained by back projection is similar in structure to the original image, which not only pays attention to the similarity at the pixel level. Specifically, it calculates the similarity between the three factors: brightness, contrast and structure, generates a similarity score, and the higher the score indicates that the projected view is closer to the original view. In this embodiment, the weights of each item in the loss function are set as λ1=0.8, λ2=0.1, λ3=0.0067, and λ4=0.0933, respectively.

[0131] The new multi-view stereo matching framework based on self-supervised learning LS-MVSNet proposed in the present application aims to realize the prediction of depth map using a small amount of computing resources. Among them, the depth feature extraction module uses a deep convolutional neural network to adaptively extract features useful for depth prediction from the input image, effectively reducing the number of parameters in the model; the edge feature extraction module is used to extract feature information of object edges, and the feature information of object edges is fused with global feature information to improve the reconstruction quality at the object edge; the data augmentation technology improves the robustness and generalization of the model.

[0132] 2. Monocular moving camera-based depth estimation network model MVSDepth

[0133] Because the LS-MVSNet network model needs to meet some relatively harsh conditions when completing the task. For example, the camera pose during shooting needs to be calibrated in advance, and the approximate depth range of the scene needs to be known in advance; at the same time, the LS-MVSNet model has the defect that the depth prediction on non-Lambertian surfaces is not accurate enough.

[0134] To solve some problems encountered by the LS-MVSNet and better realize depth estimation of a scene contained in an image, the application further provides an end-to-end neural network model: a monocular moving camera depth estimation network model MVSDepth, which combines the advantages of monocular depth estimation and the geometric interpretability of multi-view stereo matching to better realize depth estimation. Because in actual application, camera pose calibration in advance is not very realistic, and the existing computing devices have the ability to store continuous image frames, multi-view stereo matching can be realized. Self-supervised monocular depth estimation can analyze the corresponding camera relative pose and depth map through image encoding and decoding, and the predicted camera pose matrix is an important parameter indispensable in the multi-view stereo matching process. In addition, although the depth value predicted by monocular depth estimation is not very accurate, it can be provided as depth prior to the multi-view stereo matching process, which not only speeds up the depth value prediction speed of the multi-view stereo matching process, but also effectively improves the depth prediction ability of the network.

[0135] The overall structure of the monocular moving camera depth estimation network model MVSDepth is shown in Figure 3 , and the input is the image frame at T time and the image frame at T-1 time. The whole network is composed of three main branches, namely the pose estimation module, the monocular depth estimation module and the multi-view stereo matching depth estimation module. The three modules are closely related, and the pose estimation module and the monocular depth estimation module are coupled with each other and realize self-supervised training through projection transformation. The rotation matrix and translation matrix output by the pose estimation network are simply referred to as R m and T m , and the prediction result of the monocular depth estimation module, namely the monocular depth map Dmono, are all sent to the multi-view stereo matching depth estimation module as known parameters. The multi-view stereo matching network structure adopted in this embodiment is the LS-MVSNet framework in the foregoing, and after receiving the prior information of other branches, a more accurate depth map Dmvs can be predicted.

[0136] It can be understood that the MVSDepth adds a pose estimation module and a monocular depth estimation model on the basis of the LS-MVSNet. The pose estimation network can predict the camera external parameter according to the movement between the time sequence image frames. The depth value predicted by the monocular depth estimation network can be used as prior knowledge to tell the LS-MVSNet to perform depth estimation on this basis.

[0137] (1) Pose estimation module

[0138] The present application relates to the problem of predicting camera pose, mainly estimating the relative extrinsic parameters of camera between image frames, i.e. rotation matrix and translation matrix, through time series images under free viewpoint. The camera pose estimation network PoseNet of the present application realizes self-supervised estimation of camera pose by using the constraint relationship of three-dimensional geometry. The camera pose estimation network is composed of an encoder-decoder composed of a convolutional neural network, as shown in Figure 4

[0139] The encoder module, hereinafter referred to as PoseCNN, is a convolutional neural network that can accept multiple image frames as input, and the number of image frames is InputFrames. The image frames input into PoseCNN are first subjected to a feature extraction module composed of 7 layers of convolutional layers (nn.Conv2d) and activation functions (nn.ReLU) to complete camera pose feature extraction. The extracted camera pose feature information is sent to the Poseconv convolutional layer. The output size of the Poseconv convolutional layer is (6*InputFrames-1), and 6 is because the motion of the camera can be represented by six parameters of the rotation matrix, and because it is the relative pose between two adjacent image frames, so it is multiplied by (InputFrames-1). Finally, the output is adjusted, and the rotation and translation information of the camera can be extracted through PoseCNN.

[0140] The decoder module, hereinafter referred to as PoseDecoder, is also composed of a series of convolutional layers, which are used to decode the camera pose feature representation extracted from PoseCNN. The output size of PoseDecoder is the same as that of PoseCNN, which is (6*InputFrames-1). After decoding by the decoder, the 6-degree-of-freedom relative pose corresponding to the input image frames, i.e. the rotation matrix and the movement matrix, is finally obtained. The structure parameter details of the pose estimation network of the present application are shown in Table 2.

[0141] Table 2 Configuration parameter details of machine pose estimation network structure

[0142]

[0143] (2) Monocular depth estimation module

[0144] The network structure of the monocular depth estimation model (DepthNet module) is as shown in Figure 5 It uses time-continuous image frames as training signals, and in addition to predicting depth, it also estimates the camera pose between consecutive image frames, and simultaneously solves the depth value in the image scene and the camera pose between consecutive image frames.

[0145] ​The purpose of estimating the camera pose is to reconstruct the image using the predicted depth and camera pose by the network during training, so as to construct a loss function to constrain the depth estimation network, and change the depth prediction learning problem of the network into a view synthesis problem, and the target of the model during training is to achieve the optimal solution of the photometric reprojection error. Figure 5 As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map.

[0146] The encoder is composed of multiple similar sub-modules, and each sub-module includes a convolution layer, a BatchNorma layer, a ReLU activation function and a pooling layer. The convolution layer can learn various types of features from low to high; the BN layer can effectively integrate the learned feature distribution and accelerate the convergence of the network; and the pooling layer can reduce the size of the feature map while improving the receptive field of the convolution layer. The decoder adopts the mode of combining upsampling layers and convolution layers, and gradually decodes the low-resolution feature map into a depth map with the same size and resolution as the input image through deconvolution and upsampling operations.

[0147] The decoder is composed of four identical up-convolution structures, wherein the up-convolution structure includes a feature map fusion, a deconvolution and an upsampling process; the input of each up-convolution structure is the output of the previous layer network and the feature map of the same size in the encoder network. The decoder also uses the technology of jump connection to fuse the feature maps from different levels of the encoder with the feature maps of the corresponding levels of the decoder, so as to fuse information from feature maps of different scales, and the purpose is to improve the accuracy of depth estimation.

[0148] As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map. Figure 5 As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map. m As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map. m As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map. mono As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map.

[0149] I T→T-1 As shown in the figure, the DepthNet module adopts the U-Net architecture, and uses the encoder and decoder mode to encode the single-frame image X of the input module into a depth map Y. The encoder is responsible for feature extraction of the input image, and the decoder is responsible for mapping the extracted features into a depth map. T mono m , R​​m ] T→T-1 ,K)>

[0150] In the above formula, K is the camera intrinsic parameter matrix, which can be obtained through camera calibration; proj(·) represents the depth map D that can be returned after projection. mono The reprojection function for two-dimensional pixel coordinates; <·> is the sampling operator. T→T-1 and I T These represent the image obtained by projecting the image frame at time T onto time T-1, and the image at time T itself, respectively. D mono This is the monocular depth map for the monocular depth estimation module.

[0151] Monocular depth estimation networks use consecutive image frames during training, with some regions occluded or invisible between them. Due to occlusion, even when depth prediction is accurate in certain areas, the reconstructed image may differ from the previous frame, resulting in a large image reconstruction loss function. In such cases, using average reprojection leads to blurred depth discontinuities. Therefore, during training, this invention calculates the reconstruction loss for each pixel using the minimum value of the reconstruction loss function at that point. Thus, the final pixel-wise photometric loss is a loss function as follows:

[0152]

[0153] In the formula, pe(·) is the reconstruction loss function; I a and I b For any image, L SSIM Here, is the structural similarity loss function; α is a weight coefficient set empirically, taken as 0.85 in this embodiment; ||·||1 represents the L1 norm; Loss p The minimum reconstruction loss function; I T and I T+n→T These are the images at time T and time T+n, respectively; n is -1 or 1.

[0154] (3) Detailed design of MVSDepth network structure

[0155] The rotation matrix R has been obtained through the attitude estimation module and the monocular depth estimation module. m Translation matrix T m And the predicted monocular depth map D mono This can be used as a known condition, and the multi-view stereo matching process can be combined with the above-mentioned known conditions to achieve more accurate and efficient depth prediction.

[0156] p t-1 =K·(R) m ·(K -1 ·p t·d pt )+T m )

[0157] In the above formula, K is the camera's intrinsic parameter matrix, which can be obtained through camera calibration. m and T m These are the rotation and translation matrices predicted by the pose estimation network, p t and p t-1 These represent a pixel in the image frame at time t and its corresponding pixel in the image frame at time t-1, respectively. pt This represents the depth of pixel p in the cross section of the cone view in image frame T. In multi-view stereo matching rules, this depth is obtained by dividing the image into equal parts within a certain range according to a specific step size.

[0158] This means that during the matching process, if the step size is fixed, the larger the depth range, the greater the d. pt The more values ​​a pixel has, the more calculations are needed for prediction, which significantly reduces the prediction efficiency of the LS-MVSNet network model. If the depth range is divided into a fixed number of equal parts during the matching process, the number of calculations required for different pixels is the same, but this can lead to inaccurate depth predictions because the depth ranges in different regions are actually different. The larger the depth range, the larger the step size, and the coarser the predicted depth value. Therefore, if the depth range for different pixels can be given more precisely, a more accurate depth map can be predicted faster and more accurately.

[0159] Typically, learning-based MVS methods sample depth values ​​within a fixed range, calculated by COLMAP or learned by the network. However, this depth range describes the depth of the entire scene. When applied to a specific region within the scene, densely searching and computing over such a large area is computationally expensive and yields inaccurate depth prediction. As mentioned above, using the entire scene as the depth range for each pixel leads to inefficient and inaccurate depth prediction.

[0160] Therefore, the design concept of MVSDepth in this invention is to first use monocular depth estimation to predict a coarse depth map of the scene, and then, based on this, perform multi-view stereo matching to predict the depth for each pixel. The aim is to fully utilize the results of monocular depth estimation to improve the efficiency and accuracy of multi-view stereo matching.

[0161] After obtaining the coarse depth map predicted by monocular depth estimation, the next step is to determine the depth range of each pixel. To address this issue, this invention proposes a depth-velocity prior-guided strategy. The main idea of ​​this strategy is to determine the depth range of a pixel given the camera's motion velocity and the pixel's coarse depth value.

[0162] Triangulation is linked to camera motion speed. When the camera moves fast, the view point changes significantly (which is equivalent to the baseline distance between two cameras, and the baseline can be used to estimate depth by multi-view stereo matching geometry if it is appropriate), which provides sufficient triangulation prior for multi-view geometry. On the contrary, slow or static video frames share similar view points (the baseline is too small to meet the principle of triangulation, and it cannot be constrained by multi-view geometry), so the triangulation prior is limited. For those video image frames that do not meet the triangulation prior, the depth map of MVSDepth aligns the prediction results given by the monocular depth estimation network.

[0163] For video frames with sufficient triangulation prior, the present application expands the depth range on the basis of rough depth, and then uses multi-view stereo matching to regress depth, so as to obtain more accurate depth. The depth sampling range is as follows:

[0164]

[0165] wherein, d is the sampling depth; d' max and d' min are the upper and lower limits of the sampling depth, which are calculated according to the depth value predicted by the monocular depth estimation network and the translation vector T obtained by the pose estimation network. The maximum and minimum values corresponding to each pixel point are not the same, which actually dynamically reduces the depth range. In this way, the depth prediction can be accelerated, and the accuracy of the depth prediction can be improved.

[0166] d' max and d' min satisfy the following conditions:

[0167]

[0168] d' min and d' max The specific values of d' and d' can be calculated by the following formula:

[0169]

[0170]

[0171] wherein, v represents the camera motion speed, which can be calculated by the output of the pose estimation network. v = a ||T m ||2, T m is the camera translation matrix estimated by the pose estimation network, and a is the frame rate of the camera. β is a hyperparameter, is a scale function that can convert v into real-world scales, usually calculated by median scaling or camera height scaling. Usually limited in the range (0, 1).

[0172]

[0173] In the process of calculating the depth sampling value, the above formula is adopted, the calculation formula of inverse depth sampling, wherein j=0, 1, 2..., D0-1, D0 is the depth sampling range. It is proved that inverse depth sampling can obtain uniformly distributed depth values at the pixel level, which is beneficial for large-scale multi-frame matching. After obtaining the depth sampling value, the remaining calculation process is the same as LS-MVSNet, which will not be described here.

[0174] In summary, the depth estimation method based on multi-view self-supervised learning provided by the application adopts a lightweight end-to-end self-supervised framework LS-MVSNet and an end-to-end multi-view self-supervised depth estimation neural network model framework MVSDepth, constructs an efficient and compact structure depth feature extraction module, effectively reduces the number of model parameters and uses the adaptive ability to extract effective information from the input image to predict the depth map.

Claims

1. A method for depth estimation based on multi-view self-supervised learning, characterized in that, Comprising the following steps: Step 1: constructing a self-supervised depth estimation neural network model LS-MVSNet: the LS-MVSNet network model comprises a depth feature extraction module, an edge feature extraction module and a data enhancement module, and takes one reference view and multiple source views as input; Step 1.1: the depth feature extraction module adaptively extracts effective information from the input image to predict the depth map; specifically: based on the cost volume construction method of the depth feature extraction module, the constructed feature cost volume, the depth convolutional neural network is used to adaptively extract the features suitable for stereo matching from the input image; the mechanism of adaptive feature extraction window is adopted, so that the depth convolutional neural network can dynamically adjust the size and position of the feature extraction window according to the image content; Step 1.2: the edge extraction module uses an edge detection algorithm to obtain object edge information and combines it into the depth estimation process to extract object contour features from the input image; specifically: first, use the Sobel operator to perform edge extraction on the input image to obtain a binary object edge image; then perform edge feature extraction on the obtained object edge image; finally, construct a feature cost volume based on the edge features; Step 1.3: add the feature cost volume constructed in the edge extraction module and the feature cost volume obtained by the depth feature extraction module directly, then obtain the initial depth map through regression and interpolation, and obtain the final depth map through the refinement step; Step 1.4: the data enhancement module simulates the occlusion existing in reality through a randomly generated black mask under the reference view, and generates the corresponding occlusion mask under the source view through projection transformation combined with the predicted depth map; Step 2: depth estimation according to the depth estimation process Determine the depth range of the pixel point under the condition that the camera motion speed and the rough depth value of the pixel point are known.

2. The method of claim 1, wherein the multi-view self-supervised learning based depth estimation method is characterized by, In step 2, the depth estimation process specifically includes: Step 2.1: Given a set of input images, including one reference view labeled as Ref and N source views labeled as Src i , where i ranges from 1 to N; let {K i ,R i ,t i} denote the camera intrinsic parameters, rotation matrix and translation vector, where {K0,R0,t0} denotes the camera parameters, rotation matrix and translation vector under the reference view Ref; after extracting the depth features from the input images, obtain the reference view feature map F ref and source view feature maps Step 2.2: Transform the feature maps of the source view to the feature maps of the reference view Ref in the reference view's perspective by the differentiable homographic projection formula where H i (d) is the homography transformation factor of the i-th source view with depth d; t i denotes the translation vector under the i-th source view, is the unit vector (0, 0, 1) T ; For any pixel point x in the source view, use the projection formula convert it to the corresponding pixel point in the reference view ~ denotes the projection equation; Step 2.3: Reconstruct feature cost volume on reference camera view Using multiple feature cost volumes, construct a variance-based 3D cost volume C; regularize the 3D cost volume C through a 3D U-Net, and obtain the estimated depth D under the reference view through the Soft argmin operation; the formula is as follows: wherein P(d) represents the probability of depth d, where d is the average value of all feature cost volumes; d max and d min represents the upper and lower bounds of the scene depth; Step 2.4: calculate the sampling depth d: In the formula, M is the total number of samplings, m∈{0,1,…,M}; In the initial stage, the scene depth is uniformly sampled M times from d min to d max ; in the subsequent depth prediction refinement stage, the depth search range is further narrowed according to the coarse depth map; the initial step is uniformly sampled R times to obtain a fine step Δd; Step 2.5: for any pixel point, its depth exploration is based on its rough depth value to achieve accurate depth value prediction; represented as: In the formula, R is the sampling number of the initial step length, and the step length coefficient r is in the range of Step 2.6: Given the corresponding intrinsic matrix K, the relative transformation matrix T from the reference view to the source view c and the estimated depth D predicted in the reference view, any pixel point p in the source view is transformed to the pixel point p' in the reference view by the inverse transformation i i :​ p' i = KT c (D(p i )·K -1 p i ) where D(p i ) represents the depth value of the pixel point p i Depth value under the reference view; Step 2.7: Generate inverse-transformed source view I' by bilinear sampling at inverse-transformed pixel point p' i i Step 2.8: Generate a binary mask M for each reference view randomly i to mask out invalid pixels projected outside the source image boundary; in the self-supervised multi-view stereo matching system, transform N source views to the view angle of the reference view to calculate the photometric consistency loss L PC :​ wherein represents a pixel-wise gradient, and represents a pixel-wise product; I r is a reference view, and ||·||1 and ||·||2 represent the l1-norm and the l2-norm.

3. The method of claim 2, wherein, The loss function L of the self-supervised depth estimation neural network model framework LS-MVSNet in step 1 considers photometric consistency, data enhancement consistency, structural similarity and smoothness, as shown below: L = λ1L PC + λ2L DA + λ3L Smooth + λ4L SSIM wherein L PC is the photometric consistency loss function, L DA is the data augmentation consistency loss function, L Smooth is the smoothness loss function, L SSIM is the structural similarity loss function; λ1, λ2, λ3 and λ4 are the weights of each loss function; is the binary mask; D is the depth map predicted by the deep feature extraction module; is the depth map obtained by the data augmentation branch; μ x and μ y are the brightness of images x and y, respectively; and are the variance of images x and y, respectively, σ xy is the covariance of images x and y; C1 and C2 are constants added for stability; C1 = (k1L) 2 , C2 = (k2L) 2 , L is the dynamic range of pixel value, k1 and k2 are parameter adjustment variables; N is the number of pixels in the depth image, is the neighborhood set of pixel i; D i and D j are the depth values at pixels i and j, respectively.

4. The method of claim 1, wherein the multi-view self-supervised learning based depth estimation method is characterized by, Based on the depth estimation using the self-supervised depth estimation neural network model LS-MVSNet, step 3 is further included: Construct a monocular moving camera-based depth estimation neural network model MVSDepth; The MVSDepth network model comprises a pose estimation module, a monocular depth estimation module and a multi-view stereo matching depth estimation module; The pose estimation module estimates the relative extrinsic parameters of the camera between image frames, i.e. rotation matrix and translation matrix, through time-series images under free viewpoint; The monocular depth estimation module uses time-continuous image frames as training signals to perform depth prediction and estimate the camera pose between consecutive image frames, while solving the depth value in the image scene and the camera pose between consecutive image frames; The pose estimation network is trained using time sequence image frames, and the current image frame and the predicted depth map are combined to train the two, thereby realizing the depth estimation training of the self-supervised neural network; The multi-view stereo matching depth estimation module adopts the framework of the self-supervised depth estimation neural network model LS-MVSNet, and after receiving the prior information of the pose estimation module and the monocular depth estimation module, a more accurate depth map Dmvs is predicted.

5. The method of claim 4, wherein, The pose estimation module comprises a pose estimation encoder and a pose estimation decoder; The pose estimation encoder adopts a convolutional neural network, accepts multiple image frames as input, and the number of image frames is InputFrames; the image frames input into the encoder module first pass through a feature extraction module composed of 7 convolutional layers and activation functions to complete camera pose feature extraction, and the extracted camera pose feature information is sent to the Poseconv convolutional layer; finally, the output is adjusted to extract the rotation and translation information of the camera; The pose estimation decoder is composed of a series of convolutional layers, and is used to decode the camera pose feature extracted from the encoder module, and finally obtains the relative pose of 6 degrees of freedom corresponding to the input image frame, that is, the rotation matrix and the movement matrix.

6. The method of claim 4, wherein the multi-view self-supervised learning based depth estimation method is characterized by, The monocular depth estimation module comprises a depth estimation encoder and a depth estimation decoder; The depth estimation encoder comprises a plurality of sub-modules, each of which comprises a convolutional layer, a BatchNorma layer, a ReLU activation function and a pooling layer; the convolutional layer learns various types of features from low to high; the BatchNorma layer integrates the learned feature distribution to accelerate the convergence of the network; the pooling layer reduces the size of the feature map while improving the receptive field of the convolutional layer; The depth estimation decoder adopts the combination of upsampling layers and convolutional layers, and gradually decodes the low-resolution feature map into a depth map with the same size and resolution as the input image through deconvolution and upsampling operations; It comprises four identical up-convolution structures, wherein the up-convolution structure comprises a feature map fusion, a deconvolution and an upsampling process; the input of each up-convolution structure is the output of the previous layer network and the feature map of the same size in the depth estimation encoder network; The features from different levels of the depth estimation encoder are fused with the features from the corresponding levels of the depth estimation decoder using the skip connection method, and the information is fused from the features of different scales.

7. The method of claim 4, wherein the multi-view self-supervised learning based depth estimation method is based on a neural network. The specific process of step 3 comprises: Step 3.1: input the continuous image frames at time T and T-1 into the pose estimation network to obtain the relative extrinsic parameter matrix between the two images, i.e. the rotation matrix R m and the translation matrix T of T-1→T m ; Step 3.2: send the current image frame at time T into the depth estimation network to get its corresponding depth map D mono , then use the projection formula to project the image frame at time T to time T-1 to get the corresponding image: I T→T-1 = I T <proj(D mono , [R m ,T m ] T→T-1 ,K) In the formula above, K is the camera intrinsic matrix; proj(·) represents the depth map D after projection mono The reprojection function of two-dimensional pixel coordinates; <·> is the sampling operator; I T→T-1 and I T respectively represent the image frame at time T projected onto the corresponding image at time T-1 and the image at time T itself; D mono is the monocular depth estimation module predicted monocular depth map; step 3.3: the monocular depth estimation network uses continuous image frames in the training process, and the reconstruction loss of each pixel point is calculated. The minimum value of the pixel point reconstruction loss function is the final per-pixel photometric loss, which is a loss function as follows: where pe(·) is a reconstruction loss function; I a and I b are any images, respectively; L SSIM is a structural similarity loss function; a is a weight coefficient set by experience; ||·||1 is an l1 norm; Loss p is a minimum reconstruction loss function; I T and I T+n→T are an image at time T and an image at time T+n, respectively; n is -1 or 1. Step 3.4: The pose estimation module obtains a rotation matrix R m and a translation matrix T m , as known conditions, together with the monocular depth map D mono obtained by the monocular depth estimation module, to achieve more accurate and efficient depth prediction in combination with the multi-view stereo matching process: where K is the intrinsic matrix of the camera; p t and p t-1 are a pixel point on the image frame at time t and the corresponding pixel point on the image frame at time t-1, respectively; d pt represents the section depth of the pixel point p in the image frame at time T in the frustum; step 3.5: a depth velocity prior guidance strategy is adopted to determine the sampling depth range of the pixel point under the condition that the camera motion velocity and the rough depth value of the pixel point are known.

8. The method of claim 7, wherein, Step 3.5 specifically comprises: The triangulation is associated with the camera motion speed, so that the video frames collected when the camera moves at high speed have triangulation prior, the depth range is expanded on the basis of the rough depth value, and then the multi-view stereo matching mode is used for depth regression as follows: where d is the sampling depth; d' max and d' min are the upper and lower limits of the sampling depth, satisfying the following conditions: In the formula, D mono monocular depth map predicted by the monocular depth estimation module; d′ min and d′ max The specific values of d' and d" are calculated by the following equations: where v is the camera motion speed, v = a||T m ||2, T m is the camera translation matrix estimated by the pose estimation network, a is the frame rate of the camera; β is a hyper-parameter; is a scale function that converts the camera motion speed v to the actual world scale.

9. The method of claim 8, wherein, The inverse depth sampling formula is used to calculate the depth sampling value: In the formula, d j is a depth sampling value, subscript j = 0, 1, 2,..., D0-1, and D0 is a depth sampling range.

Images