Depth estimation method and device for picking scene

Abstract

The invention discloses a picking scene-oriented depth estimation method and device, and the method comprises the steps: carrying out the monocular depth estimation of an input left image of a picking scene through employing a small-size monocular depth estimation model, and obtaining a relative depth map; the small monocular depth estimation model is obtained by training a pseudo-labeled image set obtained by prediction based on a large monocular depth model; performing semi-global block matching based on the input left and right images of the picking scene to obtain an absolute depth map; performing RANSAC regression on a matching point obtained by matching the parallax values of the relative depth map and the absolute depth map through a linear model to obtain linear transformation of the parallax value of the relative depth map and the parallax value of the absolute depth map; and recovering the relative depth map based on the linear transformation to obtain a final absolute depth map. According to the depth estimation method, dense, continuous and accurate depth information with an absolute scale in a picking environment can be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a depth estimation method and device for a picking scene. Background Art

[0002] When agricultural robots perform automated agronomic operations such as strawberry picking or flower and fruit thinning, they need to estimate the depth of the operating environment in order to obtain the position information of the objects to be operated and the obstacles to be avoided in three-dimensional space.

[0003] Traditional vision-based depth estimation mostly uses binocular vision and stereo matching methods to simulate human stereo vision for depth estimation. A common method is to use two cameras on the left and right to shoot images of the same scene from different perspectives, extract feature points from the left and right images, such as (SIFT, SURF), and find matching feature points in the other image based on epipolar constraints. Calculate the horizontal coordinate difference of the matching feature points on the image, i.e., the disparity, and calculate the depth of each pixel based on the camera parameters and the disparity using the triangulation principle. In order to improve the robustness of matching, methods such as semi-global matching (SGM) and semi-global block matching SGBM (Semi-Global Block Matching) have been proposed.

[0004] However, these traditional binocular depth estimation methods based on matching are still subject to factors such as lighting changes, texture loss, and occlusion, which can lead to matching errors. Depth discontinuities are prone to occur at locations such as where the texture of the object surface changes dramatically or at the edge of the object. This problem is particularly serious in scenes with complex branches and fruits, such as strawberry picking, and often leads to missing or incorrect depth estimation of the area of ​​interest.

[0005] In recent years, deep learning has made significant progress in the field of stereo depth estimation. Deep learning-based methods can learn the mapping relationship from stereo images to depth maps end-to-end, with higher accuracy and robustness. Common deep learning networks include DispNet, GC-Net, PWC-Net, RAFT-Stereo, etc.

[0006] However, the training of the binocular depth estimation neural network requires a large amount of training data (including depth labels) in corresponding or similar scenes. These data are not easy to obtain in the picking scene and rarely appear in public datasets, resulting in the poor performance of the current binocular depth estimation network in the strawberry picking scene.

[0007] On the other hand, methods for building basic models for monocular depth estimation, such as the Depth Anything network, collect and automatically annotate large-scale unlabeled data, thereby significantly expanding the data coverage and enabling the network to have strong generalization capabilities. It can achieve zero-sample generalization in many scenarios (such as strawberry picking scenarios). However, monocular depth estimation naturally has the problem of scale missing. In zero-sample generalization scenarios, only scale-independent depth can be obtained, and the real depth value with units cannot be estimated. Such depth information cannot be fully utilized by robots. Summary of the invention

[0008] The present invention provides a depth estimation method for a picking scene, which can obtain dense, continuous, accurate and absolutely scaled depth information in a picking environment.

[0009] A specific embodiment of the present invention provides a depth estimation method for a picking scene, comprising:

[0010] A small monocular depth estimation model is used to perform monocular depth estimation on the left image of the input picking scene to obtain a relative depth map, wherein the small monocular depth estimation model is trained by a pseudo-annotated image set predicted by a large monocular depth model;

[0011] The absolute depth map is obtained by performing semi-global block matching based on the left and right images of the input picking scene;

[0012] The linear model is used to perform RANSAC regression on the matching points obtained after matching the disparity value pixels of the relative depth map and the absolute depth map, so as to obtain the linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map;

[0013] The relative depth map is restored based on the linear transformation to obtain a final absolute depth map.

[0014] Preferably, performing RANSAC regression on the matching points obtained after matching the disparity values ​​of the relative depth map and the absolute depth map through a linear model to obtain a linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map includes:

[0015] Matching the disparity value of the relative depth map with the disparity value of the absolute depth map to obtain a matching point, wherein the disparity value of the relative depth map is used as an independent variable of the matching point, and the disparity value of the absolute depth map is used as a dependent variable of the matching point;

[0016] Based on the matching points obtained by matching, the scale factor and offset of the linear transformation of the disparity value of the depth map and the disparity value of the absolute depth map are obtained by using the linear model and RANSAC regression.

[0017] Preferably, before performing pixel matching on the disparity values ​​of the relative depth map and the absolute depth map, abnormal disparity values ​​of the relative depth map and the absolute depth map are eliminated.

[0018] Preferably, the final absolute depth map for:

[0019]

[0020] Among them, scale is the scale factor, bias is the offset, is a relative depth map.

[0021] Preferably, the pseudo-annotated image set is obtained based on the prediction of a large monocular depth model, including:

[0022] An unlabeled image set of picking scenes is obtained, and a large monocular depth model is used to predict the depth of the unlabeled images of picking scenes to obtain a pseudo-labeled image set.

[0023] Preferably, obtaining a set of unlabeled images of picking scenes includes:

[0024] The picking environment video data is collected by a camera, and the picking environment video data is frame-extracted to obtain an unlabeled image set of the picking scene.

[0025] Preferably, a large monocular depth model is used to predict the depth of an unlabeled image of a picking scene, and the large monocular depth model uses DINOv2-G as a backbone network.

[0026] Preferably, a small monocular depth estimation model is used to perform monocular depth estimation on the left image of the input picking scene, and the small monocular depth estimation model uses DINOv2-S as the backbone network.

[0027] A specific embodiment of the present invention further provides a depth estimation device for a picking scene, comprising:

[0028] A processing unit is used to perform monocular depth estimation on the left image of the input picking scene using a small monocular depth estimation model to obtain a relative depth map, wherein the small monocular depth estimation model is trained by a pseudo-annotated image set predicted by a large monocular depth model; perform semi-global block matching based on the left and right images of the input picking scene to obtain an absolute depth map, and perform RANSAC regression on the matching points obtained after pixel matching of the disparity values ​​of the relative depth map and the absolute depth map using a linear model to obtain a linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map;

[0029] An output unit is used to restore the relative depth map to a final absolute depth map based on the linear transformation.

[0030] A specific embodiment of the present invention also provides a depth estimation device for picking scenes, comprising: a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the depth estimation method for picking scenes.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention obtains a linear transformation of the disparity value of the depth map and the disparity value of the absolute depth map through RANSAC regression. Using the linear transformation, the present invention converts a dense and accurate relative depth map into an absolute depth map, thereby obtaining dense, continuous, accurate and absolutely scaled depth information of the picking environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a depth estimation method for a picking scene provided by a specific embodiment of the present invention;

[0034] Figure 2 A scale-free relative depth map provided by a specific embodiment of the present invention;

[0035] Figure 3 A sparse and noisy absolute depth map provided by a specific embodiment of the present invention;

[0036] Figure 4 A depth map for restoring absolute scale is provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0038] In the face of the fact that monocular depth model prediction in the prior art suffers from size loss, and binocular depth estimation often leads to missing or erroneous depth estimation of the region of interest, the present invention utilizes RANSAC regression to obtain a linear transformation of the disparity values ​​of the relative depth map and the disparity values ​​of the absolute depth map, and restores the relative depth map based on the linear transformation to obtain the final absolute depth map, thereby fusing the dense, continuous, precise relative depth map with the absolute size of the absolute depth map to obtain dense, continuous, precise depth information with absolute scale.

[0039] A specific embodiment of the present invention provides a depth estimation method for picking scenes, such as Figure 1 As shown, including:

[0040] S1. A specific embodiment of the present invention obtains a small monocular depth estimation model by training a pseudo-annotated image set, including:

[0041] S11. Obtain an unlabeled image set: collect large-scale video data of the strawberry picking environment through cameras. The data distribution needs to at least cover the conditions of subsequent system applications (for example, distance, lighting, strawberry growth method, strawberry planting agronomy, strawberry growth stage, etc.). The video data is extracted frame by frame to form a large-scale unlabeled image set of real strawberry picking scenes.

[0042] S12. Use a large monocular depth model to predict the depth of the unlabeled image of the picking scene to obtain a pseudo-labeled image set. Since the predicted depth output by the large monocular depth model of the present invention is relatively accurate, the pseudo-labeled image set constructed based on the predicted depth can be used to train a small monocular depth estimation model of smaller size.

[0043] In a specific embodiment, the large-scale monocular depth model provided by the specific embodiment of the present invention is the largest model (DINOv2-G backbone network) for monocular depth estimation (egDepthAnything).

[0044] S13. The specific embodiment of the present invention trains a small monocular depth estimation model (DINOv2-S backbone network) that can be used in real time through a pseudo-annotated image set, and uses this as a kind of knowledge distillation to make the depth estimation model provided by the specific embodiment of the present invention more familiar with the data distribution of the real world, that is, the predicted depth output by the small monocular depth estimation model is closer to the predicted depth output by the large monocular depth model. Since the depth estimation method provided by the present invention is used in real time in a picking scene, and the large monocular depth estimation model has a long running time and requires a large storage space, in order to adapt to the picking scene, it is necessary to use a monocular depth estimation model with as little data as possible.

[0045] S2. In a specific embodiment of the present invention, the left image of the picking scene is input into a small monocular depth estimation model for monocular depth estimation to obtain a relative depth map. The relative depth map is a dense depth map without scale information. The relative depth map provided by the specific embodiment of the present invention for:

[0046]

[0047] Among them, x l is the left image of the picking scene, and M(·) is a small monocular depth estimation model.

[0048] The specific embodiment of the present invention performs semi-global block matching on the left and right images of the picking scene to obtain an absolute depth map with a global absolute scale but sparse and noisy. The absolute depth map provided by the specific embodiment of the present invention for:

[0049]

[0050] Among them, x r is the right image of the picking scene, and S(·) is the semi-global block matching (SGBM, Semi-Global BlockMatching).

[0051] S3, using RANSAC regression to obtain a linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map based on the disparity value of the relative depth map and the absolute depth map, including:

[0052] S31, the specific embodiment of the present invention will The disparity value of each pixel in and The disparity value of the same pixel in Pixel matching is performed to obtain matching points, each of which contains a set of depth values, where the independent variable is the disparity value of the depth map, the dependent variable is the disparity value of the absolute depth map, and i is the index of the matching point.

[0053] S32. Eliminate outliers: In order to improve the robustness of subsequent estimation, the specific embodiment of the present invention eliminates some obviously erroneous matching points, that is, for example, points with too large or too small depth.

[0054] S33. A specific embodiment of the present invention uses a linear model to perform RANSAC robust regression on the matching points after removing outliers to obtain a linear transformation, and the linear transformation includes a scale factor and an offset. A specific embodiment of the present invention uses RANSAC regression to find a set of scale factors and offsets so that most matching points are located near the linear model.

[0055] The disparity value D of the absolute depth map provided by the specific embodiment of the present invention is abs for:

[0056] D abs =scale*D rel +bias

[0057] Among them, D rel is the disparity value relative to the depth map, scale is the scale factor, and bias is the offset.

[0058] The error function is error i =|D absi -(scale*D rel +bias).

[0059] S4. The specific embodiment of the present invention uses the linear transformation of the scale factor and the offset obtained in step S33 to restore the scale or deviation of the relative depth map to obtain a dense depth map of restored scale, that is, the final absolute depth map. That is, the specific embodiment of the present invention obtains the final absolute depth map by linearly transforming each pixel of the relative depth map.

[0060] Absolute depth map provided by a specific embodiment of the present invention for:

[0061]

[0062] Among them, scale is the scale factor, bias is the offset, is a relative depth map.

[0063] In a specific embodiment, this embodiment estimates the depth of the strawberry picking environment by using a depth estimation method for a picking scene provided by a specific embodiment of the present invention, such as Figure 2-Figure 4 As shown, Figure 2-Figure 4 Detailed description is given below. Figure 2 Relative depth map inferred from a monocular color image for a small monocular depth estimation model It has the characteristics of relatively accurate relative depth, continuous depth estimation, smooth surface, few noise points and missing points, but it only has relative scale information and no depth value with units. Such depth information cannot be fully utilized by the robot. Figure 3 The depth map is obtained by processing a pair of calibrated binocular color images using the traditional binocular vision matching method. It has a depth value in real units and the depth information is accurate at the effective matching point. However, the depth map has large noise and many error points and missing points. Figure 4 The depth map obtained by the method of the present invention combines the advantages of monocular depth estimation and binocular traditional depth estimation to obtain dense, continuous, accurate and absolute scale depth information.

[0064] A specific embodiment of the present invention further provides a depth estimation device for a picking scene, including a processing unit and an output unit:

[0065] The processing unit provided in a specific embodiment of the present invention is used to use a small monocular depth estimation model to perform monocular depth estimation on the left image of the input picking scene to obtain a relative depth map, wherein the small monocular depth estimation model is trained based on a pseudo-annotated image set predicted by a large monocular depth model; semi-global block matching is performed on the left and right images of the input picking scene to obtain an absolute depth map, and RANSAC regression is performed on the matching points obtained after pixel matching of the disparity values ​​of the relative depth map and the absolute depth map through a linear model to obtain a linear transformation of the disparity values ​​of the relative depth map and the disparity values ​​of the absolute depth map.

[0066] The output unit provided in the specific embodiment of the present invention is used to restore the relative depth map to the final absolute depth map based on the linear transformation.

[0067] A specific embodiment of the present invention also provides a depth estimation device for picking scenes, comprising: a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the depth estimation method for picking scenes.

Claims

1. A depth estimation method for picking scenes, characterized in that: include: A small monocular depth estimation model is used to perform monocular depth estimation on the left image of the input picking scene to obtain a relative depth map, wherein the small monocular depth estimation model is trained by a pseudo-annotated image set predicted by a large monocular depth model; The absolute depth map is obtained by performing semi-global block matching based on the left and right images of the input picking scene; The linear model is used to perform RANSAC regression on the matching points obtained after matching the disparity value pixels of the relative depth map and the absolute depth map, so as to obtain the linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map; The relative depth map is restored based on the linear transformation to obtain a final absolute depth map.

2. The depth estimation method for picking scenes according to claim 1 is characterized in that: The linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map is obtained by performing RANSAC regression on the matching points obtained by matching the disparity values ​​of the relative depth map and the absolute depth map through a linear model, including: Matching the disparity value of the relative depth map with the disparity value of the absolute depth map to obtain a matching point, wherein the disparity value of the relative depth map is used as an independent variable of the matching point, and the disparity value of the absolute depth map is used as a dependent variable of the matching point; Based on the matching points obtained by matching, the scale factor and offset of the linear transformation of the disparity value of the depth map and the disparity value of the absolute depth map are obtained by using the linear model and RANSAC regression.

3. The depth estimation method for picking scenes according to claim 2 is characterized in that: Before performing pixel matching on the disparity values ​​of the relative depth map and the absolute depth map, abnormal disparity values ​​of the relative depth map and the absolute depth map are eliminated.

4. The depth estimation method for picking scenes according to claim 2 is characterized in that: The final absolute depth map for: Among them, scale is the scale factor, bias is the offset, is a relative depth map.

5. The depth estimation method for picking scenes according to claim 1, characterized in that: The pseudo-annotated image set is predicted based on a large monocular depth model, including: An unlabeled image set of picking scenes is obtained, and a large monocular depth model is used to predict the depth of the unlabeled images of picking scenes to obtain a pseudo-labeled image set.

6. The depth estimation method for picking scenes according to claim 5, characterized in that: Obtain unlabeled image sets of picking scenes, including: The picking environment video data is collected by a camera, and the picking environment video data is frame-extracted to obtain an unlabeled image set of the picking scene.

7. The depth estimation method for picking scenes according to claim 5, characterized in that: A large monocular depth model is used to predict the depth of unlabeled images of picking scenes, and the large monocular depth model uses DINOv2-G as the backbone network.

8. The depth estimation method for picking scenes according to claim 1, characterized in that: A small monocular depth estimation model is used to perform monocular depth estimation on the left image of the input picking scene, and the small monocular depth estimation model uses DINOv2-S as the backbone network.

9. A depth estimation device for picking scenes, characterized in that: include: A processing unit is used to perform monocular depth estimation on the left image of the input picking scene using a small monocular depth estimation model to obtain a relative depth map, wherein the small monocular depth estimation model is trained by a pseudo-annotated image set predicted by a large monocular depth model; perform semi-global block matching based on the left and right images of the input picking scene to obtain an absolute depth map, and perform RANSAC regression on the matching points obtained after pixel matching of the disparity values ​​of the relative depth map and the absolute depth map using a linear model to obtain a linear transformation of the disparity value of the relative depth map and the disparity value of the absolute depth map; An output unit is used to restore the relative depth map to a final absolute depth map based on the linear transformation.

10. A depth estimation device for picking scenes, characterized in that: include: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the depth estimation method for picking scenes described in any one of claims 1-8.

Images