Self-supervised feed-forward cycle depth estimation method based on image rendering
Through the self-supervised feedforward loop depth estimation method based on image rendering, combined with parameter prediction network and Gaussian splashing algorithm, the accuracy and accuracy of depth estimation in complex environments is solved, and efficient and accurate depth information extraction is achieved. in multi-object and monocular scenarios.
Patent Information
- Application Number
- CN202411091619.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-08
AI Technical Summary
Existing depth estimation techniques, especially multi-mesh depth estimation, have problems such as high environmental dependence and high interference with the surface texture of objects, resulting in low accuracy and accuracy of depth information in complex environments.
Using self-supervised feedforward loop depth estimation method based on image rendering, predict rendering parameters through parameter prediction networks, combine image rendering and photometric information to construct depth estimation loss for self-supervised training, and use Gaussian splattering algorithm to render image to reduce rendering algorithm Error, improve the accuracy and speed of depth estimation.
It improves the accuracy and accuracy of depth estimation, is suitable for monocular and multi-object depth estimation scenarios, reduces the dependence on the environment, reduces the time-consuming of depth estimation, and does not require a large amount of training data, making it easy to implement in the industry.
Smart Images

Figure CN118918159A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision, and in particular to a self-supervised feedforward loop depth estimation method based on image rendering. Background Art
[0002] Depth estimation is an important research branch of computer vision. It collects images of the object under test and uses image processing methods to extract the depth information of the object under test. The extracted depth information of the object under test can reflect the distance between the object under test and the camera, which plays an important role in extracting the geometric information of the object under test and realizing three-dimensional scene reconstruction.
[0003] The current depth estimation technology can be mainly divided into two categories: monocular depth estimation and multi-camera depth estimation. Monocular depth estimation is based on the image obtained from a single viewpoint for depth estimation. It is easy to operate and industrially implement, but the accuracy and precision of the depth information of the object under test are not high. Multi-camera depth estimation obtains multiple images from multiple different viewpoints, and then uses the principles of feature matching and parallax calculation to calculate the position deviation between corresponding points in different images to jointly estimate the depth information of the object under test. Commonly used algorithms include SAD matching algorithm, BM algorithm, etc. Compared with monocular depth estimation, multi-camera depth estimation has higher accuracy and precision in extracting the depth information of the object under test, but it also has the problem of high dependence on the environment, and is easily disturbed by factors such as lighting and surface texture of the object. In complex environments such as industrial sites, there is also the problem of low accuracy and precision of the depth information of the object under test. Summary of the invention
[0004] In response to the above problems and technical requirements, this application proposes a self-supervised feedforward loop depth estimation method based on image rendering. The technical solution of this application is as follows:
[0005] A self-supervised feedforward loop depth estimation method based on image rendering, the self-supervised feedforward loop depth estimation method comprising:
[0006] Acquire an original image of the target object and initialize network parameters of a parameter prediction network, use the parameter prediction network to perform parameter prediction on the original image and output a spatial position prediction result and a first rendering parameter, wherein the spatial position prediction result includes a predicted spatial coordinate of each pixel point in the original image in a world coordinate system;
[0007] Rendering the image according to the first rendering parameters using a rendering algorithm to obtain a first reconstructed image, and calculating an image reconstruction error between the original image and the first reconstructed image;
[0008] Performing spatial position transformation on the predicted position parameters in the first rendering parameters according to a predetermined coordinate transformation method to obtain second rendering parameters, wherein the predicted position parameters in the first rendering parameters are predicted coordinates of each pixel point in the original image in the camera coordinate system;
[0009] Using a rendering algorithm to render an image according to second rendering parameters to obtain a second reconstructed image, using a parameter prediction network to predict parameters of the second reconstructed image and output third rendering parameters;
[0010] Calculate the spatial position error according to the predicted position parameter in the third rendering parameter and the predicted position parameter in the second rendering parameter;
[0011] The depth estimation loss of the original image is calculated based on the image reconstruction error and the spatial position error. The network parameters of the parameter prediction network are updated according to the depth estimation loss until the predetermined accuracy requirements are met. The depth estimation result of the target object is obtained based on the spatial position prediction result of the original image.
[0012] A further technical solution is that, when there is an image overlap area between the second reconstructed image and the original image, the spatial position error is calculated according to the predicted position parameter in the third rendering parameter and the predicted position parameter in the second rendering parameter, including:
[0013] The spatial position error is calculated according to the predicted position parameters of the pixel points in the image overlapping area in the third rendering parameter and the second rendering parameter.
[0014] A further technical solution is that the original image is a single viewpoint image of the target object obtained by a monocular vision system; and the spatial position error is calculated according to the predicted position parameters of the pixel points in the overlapping area of the image in the third rendering parameter and the second rendering parameter, including:
[0015] For each pixel point in the overlapping area of the images, the Euclidean distance between the predicted position parameter P1 of the pixel point included in the second rendering parameter in the camera coordinate system and the predicted position parameter P1' of the pixel point included in the third rendering parameter in the camera coordinate system is calculated as the scale error of the pixel point; the spatial position error is obtained according to the scale error of all pixels in the overlapping area of the images.
[0016] A further technical solution is that the original image includes single-viewpoint images at multiple different viewpoints acquired by a multi-eye vision system; and calculating the spatial position error according to predicted position parameters of pixel points in the third rendering parameter and the second rendering parameter in the image overlapping area includes:
[0017] For the single-view image in the original image and any single-view image in which the image overlaps with the second reconstructed image, for each pixel in the image overlap area between the single-view image and the second reconstructed image, calculate the Euclidean distance between the predicted position parameter P1 of the pixel in the camera coordinate system included in the second rendering parameters and the predicted position parameter P1' of the pixel in the camera coordinate system included in the third rendering parameters as the scale error of the pixel; obtain the position error between the single-view image and the second reconstructed image according to the scale errors of all pixels in the image overlap area;
[0018] A spatial position error is obtained according to the position errors between all single-viewpoint images and the second reconstructed image.
[0019] A further technical solution is to obtain the depth estimation result of the original image including:
[0020] When the original image is a single-viewpoint image of the target object obtained by a monocular vision system, the parameter prediction network after the network parameters are updated outputs the spatial position prediction result of the original image, and the depth estimation result of the target object is obtained according to the spatial position prediction result;
[0021] When the original image includes single-viewpoint images at multiple different viewpoints acquired through a multi-view vision system, the parameter prediction network after network parameter update outputs the spatial position prediction result of each single-viewpoint image in the original image, and the depth estimation result of the target object is obtained according to the spatial position prediction results of each single-viewpoint image and the pose calibration relationship between different single-viewpoint images.
[0022] Its further technical solution is that obtaining the depth estimation result of the original image includes: using the parameter prediction network after the network parameters are updated to output the spatial position prediction result of each single-viewpoint image in the original image, and obtaining the depth estimation result of the target object according to the spatial position prediction results of each single-viewpoint image and the pose calibration relationship between different single-viewpoint images.
[0023] A further technical solution is to calculate the depth estimation loss of the original image based on the image reconstruction error and the spatial position error, including:
[0024] When the original image is a single-view image of the target object obtained by a monocular vision system, the image reconstruction error of the single-view image and the spatial position error of the single-view image are weightedly calculated according to their respective weight coefficients to obtain the depth estimation loss of the original image;
[0025] When the original image includes single-view images at multiple different viewpoints acquired through a multi-view vision system, the image reconstruction error and the spatial position error of the single-view image are weightedly calculated according to their respective weight coefficients to obtain the depth estimation loss of the single-view image. The depth estimation losses of all single-view images in the original image are combined to obtain the depth estimation loss of the original image.
[0026] A further technical solution is that the rendering algorithm used is a Gaussian splash algorithm.
[0027] A further technical solution is that calculating the image reconstruction error between the original image and the first reconstructed image includes:
[0028] The L1 loss between the original image and the first reconstructed image is calculated as the image reconstruction error.
[0029] A further technical solution is to perform spatial position transformation on the predicted position parameters in the first rendering parameters according to a predetermined coordinate transformation method to obtain second rendering parameters, including:
[0030] The predicted position parameter P in the first rendering parameter is spatially transformed to P1=P*R+T to obtain the predicted position parameter P1 in the second rendering parameter, where R is a predetermined rotation matrix and T is a predetermined translation matrix.
[0031] A further technical solution is that the parameter prediction network is built based on a CNN network or a Transformer network.
[0032] The beneficial technical effects of this application are:
[0033] The present application discloses a self-supervised feedforward loop depth estimation method based on image rendering. The method introduces deep learning to use a parameter prediction network to predict rendering parameters, and uses the rendering parameters in combination with image rendering to perform image rendering. It also effectively combines the photometric information in the original image and the scale information contained in the photometric information, and constructs a depth estimation loss including image reconstruction error and spatial position error for self-supervised training of the parameter prediction network, thereby improving the accuracy and precision of depth estimation. The method of self-supervised feedforward loop training does not need to rely on a large amount of training data and is easy to implement in industry.
[0034] This depth estimation method uses the Gaussian splash algorithm for image rendering. Thanks to the high precision and rendering speed of Gaussian splash, the depth estimation method is prevented from being disturbed by the errors of the rendering algorithm itself, further ensuring the accuracy of the depth estimation, and making the depth estimation method faster, reducing the time consumption of depth estimation. In addition, this method can be applied to both monocular depth estimation scenes and multi-eye depth estimation scenes, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flowchart of a self-supervised feedforward loop depth estimation method according to an embodiment of the present application. DETAILED DESCRIPTION
[0036] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.
[0037] This application discloses a self-supervised feed-forward loop depth estimation method based on image rendering, please refer to Figure 1 The method flow chart of the self-supervised feedforward loop depth estimation method comprises the following steps:
[0038] Step 1: Obtain the original image of the target object and initialize the network parameters of the parameter prediction network. The parameter prediction network is built based on a CNN network or a Transformer network, and can adopt an existing deep neural network architecture. This application does not limit its network architecture.
[0039] Step 2: Use the parameter prediction network to predict the parameters of the original image and output the spatial position prediction result and the first rendering parameter.
[0040] The obtained spatial position prediction result includes the predicted spatial coordinates of each pixel in the original image in the world coordinate system. The obtained first rendering parameters include predicted position parameters and image effect parameters, wherein the predicted position parameters include the predicted coordinates of each pixel in the original image in the camera coordinate system. The image effect parameters include texture parameters, lighting and shadow parameters, color parameters, contrast parameters, and various other advanced effect parameters.
[0041] Step 3: Rendering the image using a rendering algorithm according to first rendering parameters to obtain a first reconstructed image, and calculating an image reconstruction error between the original image and the first reconstructed image.
[0042] The image reconstruction error represents the degree of deviation between the original image and the first reconstructed image. The image reconstruction error between the original image and the first reconstructed image can be calculated by the deviation between each corresponding pixel point. In one embodiment, the L1 loss of the original image and the first reconstructed image is calculated as the image reconstruction error.
[0043] In order to avoid additional errors introduced in the image rendering process, in one embodiment, the rendering algorithm used is a Gaussian splash algorithm, which has higher rendering accuracy and faster rendering speed.
[0044] Step 4: spatially transform the predicted position parameters in the first rendering parameters according to a predetermined coordinate transformation method to obtain second rendering parameters, and the image effect parameters in the second rendering parameters except the predicted position parameters are consistent with those in the first rendering parameters.
[0045] In actual implementation, the predicted position parameter P in the first rendering parameter is spatially transformed to P1=P*R+T to obtain the predicted position parameter P1 in the second rendering parameter, where R is a predetermined rotation matrix and T is a predetermined translation matrix, and these two matrices are set according to the actual spatial position transformation requirements.
[0046] Step 5: Use a rendering algorithm to render the image according to the second rendering parameters to obtain a second reconstructed image, use a parameter prediction network to predict parameters of the second reconstructed image and output third rendering parameters.
[0047] Step 6, calculate the spatial position error based on the predicted position parameters in the third rendering parameters and the predicted position parameters in the second rendering parameters. Since the second reconstructed image is rendered by the second rendering parameters, on the basis of using the Gaussian splash algorithm to render the image to ensure the rendering accuracy, theoretically, the predicted position parameters in the third rendering parameters output by using the parameter prediction network to perform parameter prediction on the second reconstructed image should be consistent with the predicted position parameters in the second rendering parameters. However, due to the existence of the estimation error, there will be a deviation between the predicted position parameters in the third rendering parameters and the predicted position parameters in the second rendering parameters, so this step further calculates the spatial position error.
[0048] In addition, the visible space of the parameter prediction network is the range covered by the original image. The prediction results of the parameter prediction network for the range beyond the visible space are very poor. Therefore, in order to ensure that the parameter prediction network can accurately output the predicted position parameters in the third rendering parameters, and to avoid the problems of gradient explosion and gradient disappearance, when performing spatial position transformation in step 4, it is necessary to ensure that at least a part of the spatial position after transformation falls within the visible space of the parameter prediction network, that is, to ensure that there is an image overlap area between the second reconstructed image and the original image. When calculating the spatial position error, it is not calculated using the pixels in the entire domain of the second reconstructed image and the original image, but the spatial position error is calculated based on the predicted position parameters of the pixels in the image overlap area in the third rendering parameters and the second rendering parameters. Including:
[0049] For each pixel point in the image overlap area between the second reconstructed image and the original image, the Euclidean distance between the predicted position parameter P1 of the pixel point in the camera coordinate system included in the second rendering parameter and the predicted position parameter P1' of the pixel point in the camera coordinate system included in the third rendering parameter is calculated as the scale error of the pixel point. Then, the spatial position error is obtained according to the scale error of all the pixels in the image overlap area. For example, the common method is to calculate the sum of the scale errors of all the pixels as the spatial position error. The correspondence between the pixel points and the predicted position parameters in the camera coordinate system can be obtained through the camera intrinsic parameter matrix, which will not be described in detail.
[0050] Step 7, the depth estimation loss of the original image is calculated based on the image reconstruction error and the spatial position error. The depth estimation loss obtained combines the two parts of the error, image reconstruction error and spatial position error, and can effectively combine the photometric information in the original image and the scale information contained in the photometric information to characterize the estimation error. In one embodiment, the image reconstruction error of the single-view image and the spatial position error of the single-view image are weighted according to their respective weight coefficients to obtain the depth estimation loss of the single-view image. The weight coefficients of the two parts of the error are determined based on experience and debugging.
[0051] Step 8, update the network parameters of the parameter prediction network according to the depth estimation loss, such as the common use of back propagation to update the network parameters, until the predetermined accuracy requirement is reached, and the depth estimation result of the target object is obtained according to the spatial position prediction result of the original image. In actual implementation, when the depth estimation loss of the original image is less than a predetermined threshold, it can be considered that the predetermined accuracy requirement is met, and the predetermined threshold is custom set according to the accuracy requirement.
[0052] The self-supervised feedforward loop depth estimation method of this application can be applied to both monocular depth estimation scenarios and multi-camera depth estimation scenarios:
[0053] (1) For monocular depth estimation scenarios
[0054] In the monocular depth estimation scenario, a monocular vision system including a camera is used to obtain a single-view image of the target object as the original image. At this time, the visible space of the parameter prediction network is the range covered by the single-view image.
[0055] Then, the spatial position error can be directly obtained in step 6 according to the scale error of all pixels in the image overlap area between the single-view image and the second reconstructed image. And the depth estimation loss of the original image can be obtained by weighting the image reconstruction error of the single-view image and the spatial position error of the single-view image according to their respective weight coefficients in step 7.
[0056] Similarly, in the above step 8, after the self-supervised feedforward training is performed through the method of the present application to achieve the predetermined accuracy requirement, the parameter prediction network output of the original image can be used to update the network parameters. At this time, the predicted spatial coordinates of each pixel point contained in the spatial position prediction result obtained have high accuracy and precision, and then the depth estimation result of the target object can be obtained based on the spatial position prediction result.
[0057] (2) For multi-view depth estimation scenarios
[0058] In the multi-view depth estimation scenario, a multi-view vision system containing multiple cameras is used to obtain the original image. The multiple cameras in the multi-view vision system are arranged at different viewpoints to obtain the single viewpoint images of the target object at the corresponding viewpoints. These cameras are pre-calibrated in pose. Therefore, the original image obtained in this scenario actually includes the single viewpoint images of the target object at multiple different viewpoints, and there is a pre-determined and known pose calibration relationship between the single viewpoint images obtained by these different cameras. At this time, the visible space of the parameter prediction network is the total range covered by all single viewpoint images.
[0059] Therefore, in this scenario, in the above step 6, for each pixel in the image overlap area between the single-view image and the second reconstructed image, the Euclidean distance between the predicted position parameter P1 of the pixel included in the second rendering parameter in the camera coordinate system and the predicted position parameter P1' of the pixel included in the third rendering parameter in the camera coordinate system is calculated as the scale error of the pixel; the position error of the single-view image and the second reconstructed image is obtained according to the scale error of all pixels in the image overlap area. In this case, it is possible that there is more than one single-view image in the original image that has an image overlap area with the second reconstructed image, then the spatial position error is obtained according to the position error of all single-view images and the second reconstructed image, for example, the final spatial position error is obtained by adding the position error of all single-view images and the second reconstructed image. In this case, the pose conversion can be achieved by having a predetermined known pose calibration relationship between different single-view images, thereby determining the corresponding relationship between the pixels, which will not be described in detail in this embodiment.
[0060] Correspondingly, in step 7 of this scenario, under a set of network parameters for each iteration, the corresponding image reconstruction error and spatial position error are calculated for each acquired single-view image according to the method of the present application, and then the depth estimation loss of the single-view image is calculated based on the image reconstruction error and spatial position error of each single-view image in the original image. Finally, the depth estimation loss of all single-view images in the original image is combined to obtain the depth estimation loss of the original image, and the network parameters are iterated and updated.
[0061] Then in the above step 8, when the self-supervised feedforward training reaches the predetermined accuracy requirement, the parameter prediction network outputs the spatial position prediction result of each single-view image in the original image using the updated network parameters. According to the spatial position prediction results of each single-view image and the pose calibration relationship between different single-view images, the depth estimation result of the target object can be obtained according to existing feature matching and disparity calculation methods.
Claims
1. A self-supervised feed-forward recurrent depth estimation method based on image rendering, It is characterized in that The self-supervised feedforward loop depth estimation method comprises: Acquire an original image of the target object and initialize network parameters of a parameter prediction network, use the parameter prediction network to perform parameter prediction on the original image and output a spatial position prediction result and a first rendering parameter, wherein the spatial position prediction result includes a predicted spatial coordinate of each pixel point in the original image in a world coordinate system; Performing image rendering according to the first rendering parameters using a rendering algorithm to obtain a first reconstructed image, and calculating an image reconstruction error between the original image and the first reconstructed image; Performing spatial position transformation on the predicted position parameters in the first rendering parameters according to a predetermined coordinate transformation method to obtain second rendering parameters, wherein the predicted position parameters in the first rendering parameters are predicted coordinates of each pixel point in the original image in the camera coordinate system; Using a rendering algorithm to render an image according to the second rendering parameters to obtain a second reconstructed image, using a parameter prediction network to predict parameters of the second reconstructed image and output third rendering parameters; Calculate a spatial position error according to the predicted position parameter in the third rendering parameter and the predicted position parameter in the second rendering parameter; The depth estimation loss of the original image is calculated based on the image reconstruction error and the spatial position error. The network parameters of the parameter prediction network are updated according to the depth estimation loss until a predetermined accuracy requirement is reached. The depth estimation result of the target object is obtained based on the spatial position prediction result of the original image.
2. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that The second reconstructed image and the original image have an image overlap area, and the calculating of the spatial position error according to the predicted position parameter in the third rendering parameter and the predicted position parameter in the second rendering parameter includes: The spatial position error is calculated according to the predicted position parameters of the pixel points in the image overlapping area in the third rendering parameter and the second rendering parameter.
3. The self-supervised feedforward loop depth estimation method according to claim 2, It is characterized in that The original image is a single-viewpoint image of the target object obtained by a monocular vision system; calculating the spatial position error according to the predicted position parameters of the pixel points in the image overlap area in the third rendering parameter and the second rendering parameter includes: For each pixel point in the image overlapping area between the second reconstructed image and the original image, calculate the Euclidean distance between the predicted position parameter P1 of the pixel point in the camera coordinate system included in the second rendering parameter and the predicted position parameter P1' of the pixel point in the camera coordinate system included in the third rendering parameter as the scale error of the pixel point; and obtain the spatial position error based on the scale error of all pixels in the image overlapping area.
4. The self-supervised feedforward loop depth estimation method according to claim 2, It is characterized in that The original image includes a single viewpoint image at multiple different viewpoints acquired by a multi-eye vision system; calculating the spatial position error according to the predicted position parameters of the pixel points in the image overlap area in the third rendering parameter and the second rendering parameter includes: For the single-view image in the original image and any one of the single-view images having an image overlap region with the second reconstructed image, for each pixel in the image overlap region between the single-view image and the second reconstructed image, calculate the Euclidean distance between the predicted position parameter P1 of the pixel in the camera coordinate system included in the second rendering parameters and the predicted position parameter P1' of the pixel in the camera coordinate system included in the third rendering parameters as the scale error of the pixel; and obtain the position error between the single-view image and the second reconstructed image according to the scale errors of all the pixels in the image overlap region; A spatial position error is obtained according to the position errors between all single-viewpoint images and the second reconstructed image.
5. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that Obtaining the depth estimation result of the original image includes: When the original image is a single-viewpoint image of the target object obtained by a monocular vision system, a parameter prediction network after network parameters are updated is used to output a spatial position prediction result of the original image, and a depth estimation result of the target object is obtained according to the spatial position prediction result; When the original image includes single-viewpoint images at multiple different viewpoints acquired through a multi-view vision system, the parameter prediction network after network parameter update outputs the spatial position prediction result of each single-viewpoint image in the original image, and the depth estimation result of the target object is obtained according to the spatial position prediction results of each single-viewpoint image and the pose calibration relationship between different single-viewpoint images.
6. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that The depth estimation loss of the original image calculated according to the image reconstruction error and the spatial position error comprises: When the original image is a single-view image of the target object obtained by a monocular vision system, an image reconstruction error of the single-view image and a spatial position error of the single-view image are weightedly calculated according to respective weight coefficients to obtain a depth estimation loss of the original image; When the original image includes single-viewpoint images at multiple different viewpoints acquired through a multi-view vision system, the image reconstruction error of the single-viewpoint image and the spatial position error of the single-viewpoint image are weightedly calculated according to their respective weight coefficients to obtain the depth estimation loss of the single-viewpoint image, and the depth estimation loss of all single-viewpoint images in the original image is combined to obtain the depth estimation loss of the original image.
7. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that The rendering algorithm used is the Gaussian splatter algorithm.
8. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that The calculating the image reconstruction error between the original image and the first reconstructed image comprises: An L1 loss between the original image and the first reconstructed image is calculated as an image reconstruction error.
9. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that Performing spatial position transformation on the predicted position parameters in the first rendering parameters according to a predetermined coordinate transformation method to obtain second rendering parameters includes: The predicted position parameter P in the first rendering parameter is spatially transformed to P1=P*R+T to obtain the predicted position parameter P1 in the second rendering parameter, where R is a predetermined rotation matrix and T is a predetermined translation matrix.
10. The self-supervised feedforward loop depth estimation method according to claim 1, It is characterized in that The parameter prediction network is built based on a CNN network or a Transformer network.
