Method for locating three-dimensional sphere center of spherical fruit based on RGB-D

Abstract

This invention discloses a method for 3D center localization of spherical fruits based on RGB-D. The method involves acquiring color and depth images of spherical fruits in an orchard environment. Target detection is performed on the fruits in the color images to obtain 2D bounding boxes, and corresponding depth regions of interest (ROIs) are extracted from the depth images. Consistency screening is performed on the depth data within the ROIs to remove outliers, yielding valid depth data. This valid depth data is then combined with backprojection of the center pixel of the bounding box and camera intrinsic parameters to obtain an initial estimated center position. The valid depth data is further filtered to construct a local 3D point cloud. Based on the surface normal and spherical geometric constraints of the point cloud, a spherical structure point cloud is selected. Then, based on radius priors, backprojection of the center is performed, and clustering is used to obtain the algorithm's center estimation result. Finally, the algorithm's center is fused with the initial center to obtain the final 3D center coordinates. This invention enables stable and accurate acquisition of the 3D position of spherical fruits in complex orchard environments and under conditions of foliage occlusion.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and agricultural automation technology, and in particular to a method for three-dimensional center localization of spherical fruits based on RGB-D. Background Technology

[0002] With the development of smart agriculture, automated fruit harvesting technology has become an important research direction in the field of agricultural robotics. Accurate acquisition of the three-dimensional spatial position of the fruit is a key prerequisite for achieving precise grasping and harvesting by robotic arms.

[0003] Currently, methods for fruit localization mainly include two-dimensional image detection methods and three-dimensional point cloud methods. Two-dimensional image-based methods typically use deep learning models to detect the fruit, which can accurately obtain the fruit's position in the image plane, but cannot directly obtain its three-dimensional spatial coordinates. On the other hand, three-dimensional point cloud-based methods usually rely on lidar or structured light devices to obtain spatial information. Although they can achieve three-dimensional localization, they suffer from problems such as high equipment costs and complex data processing.

[0004] In recent years, RGB-D cameras have become an important tool for 3D fruit localization due to their ability to simultaneously acquire color images and depth information. However, in actual orchard environments, fruits are often affected by factors such as shading from branches and leaves, changes in lighting, and background interference, resulting in a large number of noise points and outliers in the depth images. This leads to low accuracy and poor stability when directly using depth data for 3D reconstruction or sphere center estimation. Therefore, there is an urgent need for a method that can effectively suppress depth noise, improve point cloud quality, and combine the geometric properties of a sphere to achieve stable and high-precision 3D sphere center localization in complex orchard environments. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to overcome the shortcomings of the prior art and provide a three-dimensional center localization method for spherical fruits based on RGB-D. This method utilizes depth consistency constraints and spherical geometric characteristics to achieve accurate identification and three-dimensional localization of the fruits, thereby solving the aforementioned technical problems.

[0006] Technical Solution: This invention provides a method for 3D center localization of a spherical fruit based on RGB-D, comprising: acquiring an RGB-D image of the spherical fruit and determining the target region of the fruit based on the color image; extracting depth data corresponding to the target region and filtering the depth data to obtain effective depth information; determining the initial estimated position of the fruit center based on the effective depth information; back-projecting the filtered depth data into a 3D space to construct a point cloud, and determining the algorithm's center estimation result based on the spherical geometric features of the point cloud; and fusing the initial estimated position with the algorithm's center estimation result to obtain the final 3D center coordinates of the fruit.

[0007] Furthermore, the specific steps include: S1: Acquire RGB-D images of spherical fruits, including color and depth images; S2: Perform target detection on the color image, obtain the two-dimensional detection box of the fruit, and extract the depth region of interest (ROI) corresponding to the detection box in the depth image; S3: Perform global depth consistency filtering on the depth pixels Z(u,v) within the ROI, construct a depth consistency criterion based on the depth distribution, remove outliers whose depth deviates from the main distribution, and obtain the set of effective depth pixels after filtering. ; S4: The filtered set of valid depth pixels Medium-depth information statistics Combined with the center pixel of the two-dimensional detection box ( ), using camera intrinsic parameters, to determine the center pixel ( By back-projecting to three-dimensional space, the initial estimated position of the fruit's center can be obtained; S5: For the effective depth pixel set Perform local depth continuity filtering for each point. Calculate its two-dimensional neighborhood The depth variation ΔZ is calculated, and when ΔZ exceeds a preset threshold, the point is removed, resulting in a set of points with local depth continuity. ; S6: A set of depth-local continuity Mid-depth pixels are back-projected into 3D space based on the camera imaging model to construct a local 3D point cloud set. ; S7: Yes Perform local surface normal estimation Calculate the rough center of the sphere The angles between the normals of each point and the corresponding theoretical radial directions are compared, and points that satisfy the spherical geometric constraints are retained to obtain a point cloud that conforms to the spherical structure. ; S8: For each point Based on the radius prior, the center of the sphere is projected backward along the normal direction of its surface to obtain candidate center points. A clustering algorithm is then used to determine the main clusters for all candidate points, and the center of the main cluster is selected as the center of the sphere in the algorithm. ; S9: Constructing a comprehensive confidence index According to the confidence index The algorithm's sphere center estimation result is weighted and fused with the initial sphere center estimation result to obtain the final three-dimensional sphere center coordinates. .

[0008] Furthermore, in steps S1 and S2, a binocular camera is used to simultaneously acquire color images and depth images containing spherical fruits in a natural orchard environment, and target detection is performed on the color images to obtain two-dimensional detection boxes of the target fruits. Based on the two-dimensional detection boxes, the corresponding depth region of interest (ROI) is extracted in the depth image, and the depth region of interest and the detection box in the color image maintain a correspondence in pixel coordinates.

[0009] Furthermore, in step S3, the depth pixels Z(u,v) within the ROI are statistically analyzed to calculate the median depth. And calculate the median absolute deviation Construct a globally consistent criterion and retain those that satisfy the criteria. The points are used as global depth consistency points, where k is a scaling factor, and outliers that deviate from the main distribution in depth are removed.

[0010] Furthermore, in step S4, based on the filtered set of effective depth pixels... Median depth Combined with the center of the detection frame ( By back-projecting the center pixel into 3D space using camera intrinsic parameters, an initial sphere center estimate is obtained. =( , ) in, For camera focal length parameters, These are the coordinates of the camera's principal point.

[0011] Furthermore, in step S5, the set Perform local depth continuity filtering for any point Based on its corresponding two-dimensional pixel coordinates Determine the pixel neighborhood in a depth image. Calculate its depth change: When the depth change is greater than a preset threshold At that time, the corresponding points are removed to obtain a set of depth-local continuity. .

[0012] Furthermore, in step S6, for the depth pixel set For any pixel in the array, its pixel coordinates are ( ), corresponding to a depth value =Z( Combining the camera's intrinsic parameters, the coordinates are transformed to three-dimensional coordinates in the camera coordinate system, and calculated as follows: Through the above back projection process, the set Medium-depth pixels are converted into 3D spatial points, forming a local 3D point cloud set. This provides basic data for subsequent surface normal estimation and sphere center calculation.

[0013] Furthermore, in step S7, the three-dimensional point cloud set... Each of the three-dimensional points In three-dimensional space, select its neighborhood point set. Perform a mean-removal process to obtain the mean point of the neighborhood point set. : Construct the covariance matrix: The covariance matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the smallest eigenvalue is selected as the local surface normal direction of the three-dimensional point. Calculate the rough center of the sphere And compare the angles between the normal at each point and the corresponding theoretical radial direction. ,when When the angle is less than a preset threshold, the three-dimensional point is determined to satisfy the spherical geometric consistency constraint and is retained as part of the spherical consistency point cloud, thus obtaining a point cloud that conforms to the spherical structure. .

[0014] Furthermore, in step S8, for each point Based on the radius prior, y is along the surface normal corresponding to that point. By performing a back projection and obtaining the results from the fruit experiment calibration, candidate points for the sphere's center are generated, and the calculation relationship is as follows: Wherein, the radius prior R is an empirical value of the average radius of the spherical fruit to be detected, and different constants can be set according to the fruit variety; a set of candidate center points is formed for all candidate center points. Cluster analysis was performed using density-based clustering (DBSCAN) in three-dimensional space to obtain multiple candidate clusters; the cluster containing the most candidate sphere centers was selected as the principal cluster. The spatial mean of all candidate sphere centers within the main cluster is used as the sphere center estimation result of the algorithm. .

[0015] Furthermore, in step S9, based on the comprehensive confidence index, the algorithm's sphere center estimation results are evaluated respectively. Compared with the initial sphere center estimation results By performing weighted fusion, the final three-dimensional sphere center coordinates are obtained: in, For planar fusion weights, Weights for depth-oriented fusion.

[0016] Furthermore, the planar directional fusion weights Convergence confidence of candidate points at the center of the sphere and depth consistency confidence Composition, that is in, This is the weighting adjustment coefficient.

[0017] Furthermore, the depth-direction fusion weights for: Where β is the depth-direction weight adjustment coefficient.

[0018] Beneficial effects: Compared with the prior art, the present invention has at least the following beneficial effects: (1) This invention introduces a global depth consistency screening method based on median and median absolute deviation, which effectively removes outliers that deviate from the main distribution in depth images, significantly reduces the influence of depth noise caused by illumination changes, reflection and occlusion, and improves the reliability of depth data; (2) By performing local depth continuity constraint screening in two-dimensional pixel space and removing depth mutation points, this invention can effectively suppress the influence of branch and leaf occlusion and background interference on the fruit area, thereby obtaining an effective depth area with continuous structure and clear boundaries. (3) This invention estimates the local surface normal of the point cloud and combines the consistency constraint between the spherical normal and the radial direction to screen points that conform to the spherical structure, effectively eliminating non-fruit surface points and improving the accuracy of the sphere center estimation. (4) This invention uses the prior of fruit radius to generate candidate points of the sphere center by normal back projection, and combines density clustering method to extract the main cluster. It can still obtain reliable sphere center estimation results stably even in the presence of occlusion or incomplete point cloud. (5) This invention constructs a comprehensive confidence index to weight and fuse the initial sphere center estimation and the algorithm sphere center estimation results, effectively combining statistical and geometric information to further improve the stability and accuracy of sphere center positioning. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0021] As shown in the figure, a method for 3D center localization of a spherical fruit based on RGB-D is proposed. First, an RGB-D camera is used to acquire color and depth images of the fruit. Object detection is performed in the color image to obtain a 2D bounding box, and the corresponding depth region of interest (ROI) is extracted in the depth image. Statistical analysis is performed on the depth pixels within the ROI, and a global depth consistency criterion is constructed based on the median and median absolute deviation to remove abnormal depth points. Further, local depth continuity is filtered in the 2D pixel space to remove abrupt depth changes. The filtered depth pixels are back-projected into 3D space to construct a local point cloud, and surface normals are estimated. Points with spherical consistency are selected based on the angle between the normal and the radial direction of the sphere. Based on the prior fruit radius, the point cloud is back-projected to generate candidate center points, and clustering is used to obtain the algorithm's center estimation results. Finally, a comprehensive confidence index is constructed, and the algorithm's center and the initial center are weighted and fused to obtain the final 3D center coordinates.

[0022] In some embodiments, the following steps are specifically included: S1: Collect RGB-D images of fruits on fruit trees in the orchard. Specifically, use a binocular camera to simultaneously collect color and depth images of spherical fruits in a natural orchard environment.

[0023] S2: Perform target detection on the acquired fruit color image to obtain the two-dimensional detection box of the fruit, and extract the depth region of interest (ROI) corresponding to the detection box in the depth image. Specifically, perform target detection on the color image to obtain the two-dimensional detection box of the target fruit, and extract the corresponding depth region of interest (ROI) in the depth image based on the two-dimensional detection box. The depth region of interest and the detection box in the color image maintain a correspondence in pixel coordinates.

[0024] S3: Perform global depth consistency screening on depth pixels Z(u,v) within the ROI, construct a depth consistency criterion based on the depth distribution, remove outliers whose depth deviates from the main distribution, and obtain the set of effective depth pixels Z' after screening; Statistical analysis is performed on the depth pixels Z(u,v) within the ROI to calculate the median depth. And calculate the median absolute deviation Construct a globally consistent criterion and retain those that satisfy the criteria. The points are used as global depth consistency points, where k is a scaling factor, and outliers that deviate from the main distribution in depth are removed.

[0025] S4: Set of valid depth pixels after filtering Medium-depth information statistics Combined with the center pixel of the two-dimensional detection box ( ), using camera intrinsic parameters, to determine the center pixel ( By back-projecting to three-dimensional space, the initial estimated position of the fruit's center can be obtained; Based on the filtered set of valid depth pixels Median depth Combined with the center of the detection frame ( By back-projecting the center pixel into 3D space using camera intrinsic parameters, an initial sphere center estimate is obtained. =( , ): in, For camera focal length parameters, These are the coordinates of the camera's principal point.

[0026] S5: For the effective depth pixel set Perform local depth continuity filtering for each point. Calculate its two-dimensional neighborhood The depth variation ΔZ is calculated, and when ΔZ exceeds a preset threshold, the point is removed, resulting in a set of points with local depth continuity. ; For sets Perform local depth continuity filtering for any point Based on its corresponding two-dimensional pixel coordinates Determine the pixel neighborhood in a depth image. Calculate its depth change: When the depth change is greater than a preset threshold At that time, the corresponding points are removed to obtain a set of depth-local continuity. .

[0027] S6: A set of depth-local continuity Mid-depth pixels are back-projected into 3D space based on the camera imaging model to construct a local 3D point cloud set. ; For the depth pixel set For any pixel in the array, its pixel coordinates are ( ), corresponding to a depth value =Z( Combining the camera's intrinsic parameters, the coordinates are transformed to three-dimensional coordinates in the camera coordinate system, and calculated as follows: Through the above back projection process, the set Medium-depth pixels are converted into 3D spatial points, forming a local 3D point cloud set. This provides basic data for subsequent surface normal estimation and sphere center calculation.

[0028] S7: Yes Perform local surface normal estimation Calculate the rough center of the sphere The angles between the normals of each point and the corresponding theoretical radial directions are compared, and points that satisfy the spherical geometric constraints are retained to obtain a point cloud that conforms to the spherical structure. ; For 3D point cloud sets Each of the three-dimensional points In three-dimensional space, select its neighborhood point set. Perform a mean-removal process to obtain the mean point of the neighborhood point set. : Construct the covariance matrix: The covariance matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the smallest eigenvalue is selected as the local surface normal direction of the three-dimensional point. Calculate the rough center of the sphere And compare the angles between the normal at each point and the corresponding theoretical radial direction. ,when When the angle is less than a preset threshold, the three-dimensional point is determined to satisfy the spherical geometric consistency constraint and is retained as part of the spherical consistency point cloud, thus obtaining a point cloud that conforms to the spherical structure. .

[0029] S8: For each point Based on the radius prior, the center of the sphere is projected backward along the normal direction of its surface to obtain candidate center points. A clustering algorithm is then used to determine the main clusters for all candidate points, and the center of the main cluster is selected as the center of the sphere in the algorithm. ; For each point Based on the radius prior, y is along the surface normal corresponding to that point. By performing a back projection and obtaining the results from the fruit experiment calibration, candidate points for the sphere's center are generated, and the calculation relationship is as follows: Wherein, the radius prior R is an empirical value of the average radius of the spherical fruit to be detected, and different constants can be set according to the fruit variety; a set of candidate center points is formed for all candidate center points. Cluster analysis was performed using density-based clustering (DBSCAN) in three-dimensional space to obtain multiple candidate clusters; the cluster containing the most candidate sphere centers was selected as the principal cluster. The spatial mean of all candidate sphere centers within the main cluster is used as the sphere center estimation result of the algorithm. .

[0030] S9: Constructing a comprehensive confidence index The algorithm's center estimation result is weighted and fused with the initial center estimation result based on the confidence level to obtain the final three-dimensional center coordinates. ; Based on the comprehensive confidence index, the results of the algorithm's sphere center estimation were analyzed. Compared with the initial sphere center estimation results By performing weighted fusion, the final three-dimensional sphere center coordinates are obtained: Among them, the planar direction fusion weight Convergence confidence of candidate points at the center of the sphere and depth consistency confidence constitute, For depth-direction fusion weights: i.e. β is the weight adjustment coefficient, and β is the weight adjustment coefficient in the depth direction.

Claims

1. A method for three-dimensional center localization of a spherical fruit based on RGB-D, characterized in that, The process includes acquiring RGB-D images of spherical fruits and determining the target area of ​​the fruit based on the color images; extracting the depth data corresponding to the target area and filtering the depth data to obtain effective depth information; The initial estimated position of the fruit's center is determined based on the effective depth information; The filtered depth data is back-projected into 3D space to construct a point cloud, and the sphere center estimation result of the algorithm is determined based on the spherical geometric features of the point cloud. The initial estimated position is fused with the center-of-sphere estimation result of the algorithm to obtain the final three-dimensional center-of-sphere coordinates of the fruit.

2. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 1, characterized in that, Specifically, the following steps are included: S1: Acquire RGB-D images of spherical fruits, including color and depth images; S2: Perform target detection on the color image, obtain the two-dimensional detection box of the fruit, and extract the depth region of interest (ROI) corresponding to the detection box in the depth image; S3: Perform global depth consistency filtering on the depth pixels Z(u,v) within the ROI, construct a depth consistency criterion based on the depth distribution, remove outliers whose depth deviates from the main distribution, and obtain the set of effective depth pixels after filtering. ; S4: The filtered set of valid depth pixels Medium-depth information statistics Combined with the center pixel of the two-dimensional detection box ( ), using camera intrinsic parameters, to determine the center pixel ( By back-projecting to three-dimensional space, the initial estimated position of the fruit's center can be obtained; S5: For the effective depth pixel set Perform local depth continuity filtering for each point. Calculate its two-dimensional neighborhood The depth variation ΔZ is calculated, and when ΔZ exceeds a preset threshold, the point is removed, resulting in a set of points with local depth continuity. ; S6: A set of depth-local continuity Mid-depth pixels are back-projected into 3D space based on the camera imaging model to construct a local 3D point cloud set. ; S7: Yes Perform local surface normal estimation Calculate the rough center of the sphere The angles between the normals of each point and the corresponding theoretical radial directions are compared, and points that satisfy the spherical geometric constraints are retained to obtain a point cloud that conforms to the spherical structure. ; S8: For each point Based on the radius prior, the center of the sphere is projected backward along the normal direction of its surface to obtain candidate center points. A clustering algorithm is then used to determine the main clusters for all candidate points, and the center of the main cluster is selected as the center of the sphere in the algorithm. ; S9: Constructing a comprehensive confidence index According to the confidence index The algorithm's sphere center estimation result is weighted and fused with the initial sphere center estimation result to obtain the final three-dimensional sphere center coordinates. .

3. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 2, characterized in that: In step S3, the depth pixels Z(u,v) within the ROI are statistically analyzed to calculate the median depth. , And calculate the median absolute deviation , Construct a globally consistent criterion and retain those that satisfy the criteria. The points are used as global depth consistency points, where k is a scaling factor, and outliers that deviate from the main distribution in depth are removed.

4. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 2, characterized in that: In step S4, based on the filtered set of valid depth pixels... Median depth Combined with the center of the detection frame ( By back-projecting the center pixel into 3D space using camera intrinsic parameters, an initial sphere center estimate is obtained. =( , ) , in, For camera focal length parameters, These are the coordinates of the camera's principal point.

5. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 2, characterized in that: In step S5, the set Perform local depth continuity filtering for any point Based on its corresponding two-dimensional pixel coordinates Determine the pixel neighborhood in a depth image. Calculate its depth change: , When the depth change is greater than a preset threshold At that time, the corresponding points are removed to obtain a set of depth-local continuity. .

6. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 2, characterized in that: In step S7, the three-dimensional point cloud set Each of the three-dimensional points In three-dimensional space, select its neighborhood point set. Perform a mean-removal process to obtain the mean point of the neighborhood point set. : , Construct the covariance matrix: , The covariance matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the smallest eigenvalue is selected as the local surface normal direction of the three-dimensional point. Calculate the rough center of the sphere And compare the angles between the normal at each point and the corresponding theoretical radial direction. ,when When the angle is less than a preset threshold, the three-dimensional point is determined to satisfy the spherical geometric consistency constraint and is retained as part of the spherical consistency point cloud, thus obtaining a point cloud that conforms to the spherical structure. .

7. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 2, characterized in that: In step S8, for each point Based on the prior radius R along the surface normal corresponding to that point By performing a back projection and obtaining the results from the fruit experiment calibration, candidate points for the sphere's center are generated, and the calculation relationship is as follows: , Wherein, the radius prior R is an empirical value of the average radius of the spherical fruit to be detected, and different constants can be set according to the fruit variety; a set of candidate center points is formed for all candidate center points. Cluster analysis was performed using density-based clustering (DBSCAN) in three-dimensional space to obtain multiple candidate clusters; the cluster containing the most candidate sphere centers was selected as the principal cluster. The spatial mean of all candidate sphere centers within the main cluster is used as the sphere center estimation result of the algorithm. 。 8. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 2, characterized in that: In step S9, based on the comprehensive confidence index, the algorithm's sphere center estimation results are evaluated respectively. Compared with the initial sphere center estimation results By performing weighted fusion, the final three-dimensional sphere center coordinates are obtained: , in, For planar fusion weights, Weights for depth-oriented fusion.

9. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 8, characterized in that: The planar fusion weights Convergence confidence of candidate points at the center of the sphere and depth consistency confidence Composition, that is ， , ， in, This is the weighting adjustment coefficient.

10. The method for three-dimensional center positioning of a spherical fruit based on RGB-D according to claim 8, characterized in that: The depth direction fusion weights for: ， Where β is the depth-direction weight adjustment coefficient.

Images