A method for constructing depth estimation optimization model based on geometric consistency

Abstract

The method for constructing a depth estimation optimization model based on geometric consistency of the present invention comprises the following steps: S1. constructing a spatial geometric consistency supervision model; S2. generating a correction model; S3. optimizing the generation process of the depth map based on the depth estimation optimization model. The method for constructing a depth estimation optimization model based on geometric consistency of the present invention is mainly divided into two parts: first, constructing a supervision model based on spatial geometric consistency and generating a correction model on this basis; then, correcting the defects existing in the common use loss based on the construction of the supervision signal, and the two work together to ensure the reliability of the depth estimation results and optimize and improve the overall accuracy of the estimated depth map.

Description

Technical Field

[0001] The present invention relates to the field of depth estimation based on deep learning, and more specifically to a method for constructing a depth estimation optimization model based on geometric consistency. Background Art

[0002] Estimating the depth of an object from an image captured by a motion camera is a key task in the field of computer vision, and can be widely used in many fields such as 3D reconstruction, AR, and autonomous driving. Depth estimation constructs the corresponding geometric relationship between pixels in an image based on the spatial consistency of image pixels, and then uses a depth-based image reconstruction method to construct a reconstruction loss to determine the accuracy of depth estimation. The main problem in the background technology is that almost all current methods are self-supervised networks based on the Encoder-Decoder architecture, because this can reduce the cost of capturing true values, but this will make the network more dependent on pixel consistency. However, as the camera moves, the pixel values ​​presented by the same object in different images are different, and the inherent problems of depth estimation such as occlusion and motion will further hinder the judgment of the network. Therefore, it is particularly important to use a geometric method that is not affected by the environment to optimize the training of the supervised network. However, in reality, it is difficult to solve the impact of pixel changes or irregular pixel changes on deep network training at the pixel level, that is, it is difficult to establish the consistency of the loss function and the true depth prediction result from image reconstruction as a method to guide deep network training. Based on this, this method proposes a mechanism for optimization from the spatial geometry level, geometrically establishes a loss that matches the depth prediction result, that is, a geometric supervision mechanism, and establishes an optimization mechanism to further optimize the loss constructed by image reconstruction, so that a more fitting and matching connection is established between the loss function and the performance of deep network prediction, which can provide new research ideas and methods for subsequent research. Summary of the invention

[0003] The present invention provides a method for constructing a depth estimation optimization model based on geometric consistency, which ensures the reliability of the depth estimation result and optimizes and improves the overall accuracy of the estimated depth map.

[0004] The technical solution of the present invention is as follows:

[0005] The method for constructing a depth estimation optimization model based on geometric consistency of the present invention comprises the following steps: S1. constructing a spatial geometric consistency supervision model; S2. generating a correction model; S3. optimizing the generation process of a depth map based on the depth estimation optimization model.

[0006] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, in step S1, a mathematical method for determining geometric consistency is used to establish a geometric relationship between the image pixels used, and the pixel differences and position differences caused by different depth values ​​are calculated using geometry to obtain a spatial geometric consistency supervision model.

[0007] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, in step S1, a two-projection strategy is adopted through a symmetric transformation method to establish a depth-related geometric connection between image pixels and the real body, and then the Euler distance is used to construct a corresponding geometrically consistent supervision signal.

[0008] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, first, a coordinate association between a pixel and itself is constructed; then, a geometric consistency loss is constructed based on the difference in coordinates of the same pixel obtained twice; finally, a geometric consistency supervision signal is constructed based on the difference in coordinates obtained twice.

[0009] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, in step S2, a correction model for erroneous pixel matching and original loss is established using geometric consistency relationships and pixel connections to correct defects in photometric loss.

[0010] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, in step S2, regional division is performed to construct a correction model according to the numerical relationship between the reconstruction loss and the supervisory signal of geometric consistency in different regions, so as to correct the deviations of the depth prediction network in areas prone to occlusion and motion problems.

[0011] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, in step S3, the neural network is optimized using the reconstruction loss, and the reconstruction loss and the depth prediction network are optimized using the optimization model.

[0012] Optionally, in the above-mentioned method for constructing a depth estimation optimization model based on geometric consistency, in step S3, the correction model generated by S2 is used to correct the reconstruction loss, and then the final supervision signal is improved using the geometric consistency supervision model, thereby obtaining the final network training loss function used.

[0013] According to the technical solution of the present invention, the beneficial effects produced are:

[0014] The method for constructing a depth estimation optimization model based on geometric consistency of the present invention is mainly divided into two parts: first, constructing a supervision model based on spatial geometric consistency and generating a correction model based on this; then correcting the defects of the commonly used loss based on the construction of the supervision signal. The two work together to ensure the reliability of the depth estimation results and optimize and improve the overall accuracy of the estimated depth map.

[0015] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0017] Figure 1 is a flow chart of a method for constructing a depth estimation optimization model based on geometric consistency of the present invention;

[0018] Figure 2 It is a specific operation flow chart of the method for constructing a depth estimation optimization model based on geometric consistency of the present invention;

[0019] Figure 3 It is a schematic diagram comparing the effects of geometric loss and image reconstruction loss constructed by the geometric consistency supervision model of the present invention;

[0020] Figure 4 It is the process of building a supervision mechanism for the construction method of the depth estimation optimization model based on geometric consistency;

[0021] Figure 5 It is a schematic diagram of the working principle of the correction mechanism of the method for constructing a depth estimation optimization model based on geometric consistency of the present invention;

[0022] Figure 6 It is a visualization of the method for constructing a depth estimation optimization model based on geometric consistency of the present invention on the KITTI dataset. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0024] like Figure 1 and Figure 2 As shown, the method for constructing a depth estimation optimization model based on geometric consistency of the present invention comprises the following steps:

[0025] Problem construction (self-supervised depth estimation): Similar to other classic methods, since the self-supervised method aims to remove the expensive cost of collecting training truth values, the self-supervised depth estimation combines the depth estimation problem with image reconstruction and uses the loss of reconstructed image construction. In this process, the first thing to do is to find points that match the current image pixels from the adjacent images based on the predicted depth map, and then establish a matching relationship between the current image and the adjacent reference image, and use the bilinear sampling method to reconstruct the current image and construct the image reconstruction loss. The specific process is as follows:

[0026] First, establish the matching relationship between the current image and the adjacent image, as shown in formula (1):

[0027] p s ~KT t→s D t (p t )K -1 p t (1)

[0028] where p t represents the pixel coordinates in the target image, K represents the parameter matrix of the camera known before the experiment, and D t represents the dense depth map of pixels predicted by the network, T t→s represents the rotation and translation transformation of the camera during the movement, that is, the movement of the camera carrier, p s Represents the pixel coordinates of the adjacent image. In this way, the matching relationship between the target image pixels and the adjacent image is established using the projection relationship.

[0029] Then bilinear sampling is used to reconstruct and obtain the image reconstruction loss, where s∈{I t-1 ,I t+1 That is, the two images adjacent to the current image are sampled and reconstructed by sampling the pixels of the corresponding points, and the reconstruction loss is finally obtained.

[0030]

[0031] I s→t = <proj(p t , p s ), I s >, (2)

[0032] Among them, proj() is the sampling and reconstruction operation, SSIM is a commonly used correlation function for judging the similarity between two images, |||| means using L1 loss, that is, the pixels are directly subtracted, and α represents the proportion of the two parts of the loss, which is generally set to 0.85. s→tRepresents the current image reconstructed by taking pixels from adjacent images using sampling operations. pe() is the reconstruction loss constructed for the two images.

[0033] S1. Construct a spatial geometric consistency supervision model.

[0034] In this step, a mathematical method of geometric consistency is determined to establish a geometric relationship between the pixels of the image used, and this relationship is used to calculate the pixel differences and position differences caused by different depth values, so as to obtain a new type of geometry-based supervision model, namely, the spatial geometric consistency supervision model. Specifically, through the symmetric transformation method, a two-projection strategy is used to establish the geometric relationship between the image pixels and the real body and related to the depth, and then the Euler distance is used to construct the corresponding supervision signal. Figure 4 As shown in Figure 2, during the construction of the supervised model, a two-projection strategy is used to establish the connection between the pixel coordinates themselves.

[0035] Specifically, we first need to build the coordinate association of the pixel from itself to itself; then we build the loss of geometric consistency based on the coordinate difference of the same pixel obtained twice, such as Figure 4 As shown, the coordinate transformation shown in formula (3) can be obtained for the entire image, and the geometric consistency relationship related to the depth map is established, where T t→s Etc. represents the target image I t (Current image) to source image (adjacent image) I s The change process of D is the movement of the camera when taking two images, and D t Represents the dense depth map of the pixel predicted by the network, which is generated by the depth prediction network prediction.

[0036] p' t ~KT s→t D s (p s )Norm <T t→s D t (p t )K -1 p t > (3)

[0037] Where Norm() represents the pixel normalization process, that is, dividing the third dimension value, T s→t Represents the source image (adjacent image) I s To the target image I t The change process of D s represents the depth map of the predicted source image (adjacent image), p' t Represents the pixel coordinates projected back onto itself.

[0038] Then, the geometric consistency supervision signal (i.e., geometric distance supervision signal) is constructed based on the difference in the coordinates obtained twice, as shown in formula (4):

[0039]

[0040] Where V represents the pixel space of the image, which is also the pixel set of the image, p t represents the pixel coordinates in the target image, p' t Represents the pixel coordinates projected back to itself. L dis It is the geometric distance supervision signal.

[0041] like Figure 3 As shown, it is a schematic diagram of the comparison of the geometric loss and image reconstruction loss effects of the geometric consistency supervision model constructed by the present invention. (a) represents the overall process of the geometric supervision mechanism based on geometric consistency established by the present method, projecting the spatial point to the adjacent image, establishing the first connection, and then projecting it back to the current image to establish the loop consistency connection of the pixels. (b) represents the target image in the later stage of training, the image reconstruction loss established on the target image, and the geometric consistency supervision mechanism established based on the present method from top to bottom.

[0042] S2. Generate a correction model.

[0043] In this step, the geometric consistency relationship and pixel connection are used to establish a correction mechanism for erroneous pixel matching and original loss, thereby correcting the defects of photometric loss.

[0044] Specifically, based on the geometric distance supervision signal constructed in step S1 and in combination with the image pixel loss, it is used to correct the deviations of the depth prediction network in areas prone to problems such as occlusion and motion, so that the depth prediction network has a more accurate judgment in these areas and eliminates the impact of unexpected deviations caused by the environment on network training.

[0045] By using the widely used image-based reconstruction loss (i.e. Figure 2 The correction model is constructed by combining the "image reconstruction loss" in step S1 with the supervisory signal of geometric consistency established in step S1. The correction model is constructed by dividing the regions according to the numerical relationship between the two (reconstruction loss and supervisory signal of geometric consistency) in different regions, such as Figure 5 As shown in (a), the reconstruction loss in the photometric variation region exceeds the tolerance range, while the geometric consistency supervision signal is reasonably correct, so the reconstruction loss needs to be weakened. Figure 5In (b) of the object boundary area, the unexpected deviation of the reconstruction loss caused by bilinear sampling can also be weakened and repaired. Therefore, the image is divided into three parts, and the correction model (correction mechanism) is used to process different areas respectively. Specifically, according to the relationship between the image reconstruction loss and the geometric consistency supervision signal (loss function), the image area is divided into three categories: correct area, reconstruction loss failure area, and invalid area. Among them, the invalid area represents Figure 5 (c) Object occlusion and object motion in Fig. 1, because reconstruction in these areas cannot be obtained from adjacent images.

[0046] Then, the correction model is used to process the regions separately according to the three classification methods, as shown in formula (5):

[0047]

[0048] Where μ and ε represent the tolerance of two supervisory signals: image reconstruction loss and geometric consistency supervisory signal. p represents the reconstruction loss.

[0049] The corrective model works like Figure 5 As shown in the figure, the correction model corrects the original image reconstruction loss in many areas such as light intensity change, boundary, motion and occlusion, and the mismatch between the reconstruction loss and the predicted depth accuracy. It combines the loss signal generated by the supervision model with the image reconstruction loss, and distinguishes and processes the regions according to the quantized size of the two.

[0050] S3. Optimize the depth map generation process based on the depth estimation optimization model.

[0051] The optimization mechanism combines S1 and S2 to reasonably and correctly optimize the depth map generation process. The depth map generated by the deep network prediction in 1 is optimized, mainly in the optimization of the depth prediction network. The network is optimized on the basis of S1 and S2, making the network more robust and improving the accuracy of network prediction.

[0052] This step includes two aspects: optimizing the neural network using the reconstruction loss and optimizing the reconstruction loss and the depth prediction network using the optimization model.

[0053] Use the optimization model to optimize the reconstruction loss and the depth prediction network, including: For the reconstruction loss and the depth prediction network, Figure 5 The many areas shown will cause the network to make wrong judgments and eventually reduce the overall accuracy of the network. Here, the correction model generated by S2 is used to correct the reconstruction loss, and then the geometric consistency supervision model is used to improve the final supervision signal to achieve the purpose of improving the final performance of the network, and then the final network training loss function used is obtained:

[0054]

[0055] L fin =M⊙L p +M'⊙λ1L dis +λ2L s (7)

[0056] Where L s is the commonly used smoothing loss, ⊙ represents bitwise multiplication. M represents the mask correction signal generated by S2 formula (5), L p represents the reconstruction loss. L dis is the geometric distance supervision signal. L s is a smooth function, It represents the partial differential of the x coordinate, dt represents the differential of the image coordinate, and the same is true for y and x.

[0057] The present invention finds that due to the lack of true value, almost all self-supervised depth estimation methods use reprojection error as the main supervisory signal. However, in some special areas, such as textureless areas, boundary areas, occluded areas and motion areas, it is impossible to train a network with pixel reprojection error. The reason is that the loss based on reprojection error produces unexpected deviations due to irregular illumination changes and wrong reprojection matching of pixels.

[0058] Therefore, the method for constructing a depth estimation optimization model based on geometric consistency of the present invention solves the difficulty of training the network in abnormal areas where the reprojection loss has unexpected deviations. The method of the present invention consists of two parts: 1) constructing a supervision model through the geometric consistency of image pixels. The supervision model can correctly guide the training of the deep network and correct the misleading of the reprojection loss to the network; and the correction model constructs a correction mask through the supervision signal to help the network make correction judgments, which is used to correct the error signal, realize the correction signal, and correct the deviation of the reprojection loss. By dividing the image into three parts and using the correction model to process different areas separately, the performance of the network is improved. The combination of the supervision model and the correction model reduces the probability of depth estimation errors, improves the overall accuracy, and provides new ideas for subsequent depth estimation research.

[0059] The method of the present invention outperforms other methods and also performs better when the model of the present invention is applied to other self-supervised depth estimation methods. Figure 6 As shown in the figure, compared with the previous method (baseline method), this method performs better and more natural overall, and the local boundaries in some areas are also clearer.

[0060] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for constructing a depth estimation optimization model based on geometric consistency, characterized in that: The following steps are involved: S1. Constructing a spatial geometric consistency supervision model, determining a mathematical method for geometric consistency, establishing a geometric relationship between pixels of the image used, and using the geometric calculation of pixel differences and position differences caused by different depth values ​​to obtain the spatial geometric consistency supervision model, Specifically, we first need to build the coordinate association from the pixel itself to itself; then, we build the geometric consistency loss based on the coordinate difference between the two pixels. Then, we can get the coordinate transformation shown in formula (3) for the whole image and establish the geometric consistency relationship related to the depth map, where Represents the target image , that is, the current image to the source image, that is, the adjacent image is the movement of the camera when taking the two images, and Represents the dense depth map of the pixel predicted by the network, which is generated by the depth prediction network prediction. （3） in Represents the pixel normalization process, that is, dividing the third dimension value. Represents the source image, i.e., the adjacent image To the target image the process of change, represents the predicted source image, i.e. the depth map of the adjacent image, Represents the pixel coordinates projected back to itself; Then, the geometric consistency supervision signal is constructed based on the difference between the two coordinates, as shown in formula (4): ， in Represents the pixel space of the image, that is, the pixel set of the image, represents the pixel coordinates in the target image, Represents the pixel coordinates projected back to itself, It is the geometric distance supervision signal; S2. Generate correction model; S3. Optimizing the depth map generation process based on the depth estimation optimization model.

2. The method for constructing a depth estimation optimization model based on geometric consistency according to claim 1, characterized in that: In step S1, a double projection strategy is used to establish a depth-related geometric connection between image pixels and the real body through a symmetric transformation method, and then the Euler distance is used to construct a corresponding geometric consistency supervision signal.

3. The method for constructing a depth estimation optimization model based on geometric consistency according to claim 2, characterized in that: First, a coordinate association between a pixel and itself is constructed; then, a loss of geometric consistency is constructed based on the difference in coordinates of the same pixel obtained twice; finally, a supervisory signal of the geometric consistency is constructed based on the difference in the coordinates obtained twice.

4. The method for constructing a depth estimation optimization model based on geometric consistency according to claim 1, characterized in that: In step S2, a correction model for erroneous pixel matching and original loss is established using the relationship of geometric consistency and pixel connection to correct the defects of photometric loss.

5. The method for constructing a depth estimation optimization model based on geometric consistency according to claim 1, characterized in that: In step S2, the correction model is constructed by dividing the regions according to the numerical relationship between the reconstruction loss and the supervisory signal of the geometric consistency in different regions, so as to correct the deviation of the depth prediction network in the regions where occlusion and motion problems are likely to occur.

6. The method for constructing a depth estimation optimization model based on geometric consistency according to claim 1, characterized in that: In step S3, the neural network is optimized using the reconstruction loss, and the reconstruction loss and the depth prediction network are optimized using the optimization model.

7. The method for constructing a depth estimation optimization model based on geometric consistency according to claim 1, characterized in that: In step S3, the correction model generated by S2 is used to correct the reconstruction loss, and then the geometric consistency supervision model is used to improve the final supervision signal, thereby obtaining the final network training loss function used.

Images