Apple orchard occluded apple spatial pose estimation system and method for picking robot

Abstract

The application discloses an orchard sheltered apple space posture estimation system and method for a picking robot, comprising an image acquisition module, a directional target detection module, a plane angle extraction module, a plane fruit axis decision module, a space posture estimation module and a coordinate system conversion module. The YOLO11-OBB model is used for apple target detection, and the center coordinates, width, height and long axis direction angle of the directional bounding box are output. The positioning and direction angle extraction of the apple target can be realized in the sheltered environment. The plane fruit axis angle decision is performed by fusing the OBB short axis direction angle and the ROI depth gradient direction angle. The plane fruit axis angle is mapped to the spatial component with the ROI normal vector by using the fruit shape index and the angle difference value judgment mechanism. The derivation from the two-dimensional plane angle to the three-dimensional space posture angle is realized. The problem that the single information source angle prediction is unreliable when the apple contour is close to a circle or the contour is missing due to shelter is avoided. The accuracy and robustness of the plane fruit axis angle estimation are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for estimating the spatial pose of occluded apples in orchards for harvesting robots. Background Technology

[0002] Currently, fruit pose estimation is a key technology for achieving precise and non-destructive harvesting by picking robots. Deep learning has been widely applied in fruit pose estimation. For example, deep learning network pipelines can simultaneously classify the maturity of tomato fruits and estimate their spatial pose; keypoint detection-based methods train fruit feature point labels to detect the position and pose of fruit stems, stalks, and other parts; multi-task learning networks are used to locate the navel point of the fruit and predict the rotation vector, or to detect and segment the fruit and sepals to estimate the pose; in addition, there are methods that estimate the pose of grape bunches by combining instance segmentation with cylinder fitting, and methods that utilize geometric perception networks, point cloud registration, and geometric understanding algorithms for grasping estimation and 6D pose estimation. These methods have achieved good pose estimation accuracy and harvesting success rate in specific scenarios.

[0003] However, accurate fruit posture recognition remains a challenge in complex orchard environments with shading and changing lighting. Taking apple harvesting as an example, fruits exhibit diverse growth postures and are often obstructed by branches, leaves, or other fruits. Existing methods largely rely on key point detection or segmentation of specific parts of the fruit, such as the calyx and stem. However, during actual harvesting, severe shading by branches and leaves makes it difficult for cameras to stably observe these parts. Point cloud-based methods suffer from insufficient real-time performance and difficulties in subsequent robot control due to non-specific angle values ​​output at the end-to-end. More critically, robotic arms typically employ fixed, static postures for harvesting, lacking the ability to dynamically adjust based on fruit shading information and actual posture. This makes them ill-suited to changing harvesting scenarios, potentially leading to weak gripping by the end effector and resulting in apples falling or being damaged.

[0004] Therefore, how to estimate the spatial orientation angle of apples in real time and accurately without relying on the identification of small target features such as calyxes and fruit stalks in a shaded orchard environment, so as to provide a dynamic grasping posture basis for the picking robot arm that can be directly executed, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a spatial pose estimation system and method for orchard shaded apples for harvesting robots to overcome or at least partially solve the above problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a spatial pose estimation system for shaded apples in an orchard for a harvesting robot, comprising: An image acquisition module is used to acquire an RGB image and a depth image containing an apple target, and to register the RGB image and the depth image; The directional target detection module uses the YOLO11-OBB model to detect apple targets in the RGB image and outputs the center coordinates, width, height, and major axis angle of the directional bounding box. The planar angle extraction module determines the minor axis angle based on the major axis angle, constructs a region of interest in the depth image with the center of the oriented bounding box as the sampling center, and calculates the depth gradient direction angle of the region of interest. The planar fruit axis decision module is used to determine the planar fruit axis angle of the apple from the minor axis angle and the depth gradient direction angle based on the fruit shape index corresponding to the directional bounding box and the difference between the minor axis direction angle and the depth gradient direction angle. The spatial attitude estimation module is used to perform spatial component mapping between the plane fruit axis angle and the normal vector of the region of interest to obtain the spatial attitude vector of the apple in the camera coordinate system, and calculate the pitch angle and roll angle of the apple based on the spatial attitude vector. The coordinate system transformation module is used to transform the pitch angle and the roll angle to the robot base coordinate system to obtain the robot arm pitch angle and robot arm roll angle, which serve as the basis for the robot arm's end-effector posture control.

[0008] Furthermore, the YOLO11-OBB model adds a rotation angle prediction branch to the standard object detection branch, and represents the oriented bounding box in the form of five parameters. The five parameters include: center coordinates, width, height, and the angle between the major axis and the horizontal axis of the image. The value range of the horizontal axis angle is [-π / 2, π / 2).

[0009] Furthermore, the minor axis direction angle is obtained by rotating the major axis direction angle by 90° and then constraining it to the range of [-π / 2, π / 2) through segmentation. In the width and height of the directional bounding box, the larger value is the major axis length and the smaller value is the minor axis length.

[0010] Furthermore, the region of interest is a square region constructed with the center of the oriented bounding box as the sampling center, which is constructed in the depth map. The radius of the side of the region of interest is based on one-third of the length of the minor axis and is determined by upper and lower threshold constraints.

[0011] Furthermore, the depth gradient direction angle is defined as the direction angle of the gradient vector in the image plane, that is, the angle between the gradient vector and the horizontal axis of the image, and its expression is:

[0012] Where arctan(·) is the arctangent function. g v The vertical depth gradient g u This represents the horizontal depth gradient.

[0013] Furthermore, the vertical and horizontal depth gradients are calculated using the edge-average depth difference method. The depth values ​​of the neighboring areas on both sides of the edge of the region of interest are averaged and then differencing to calculate the average depth of the three edge columns of the region of interest. The expression for the average depth of the three edge columns is as follows:

[0014] in, , , , For the three-column edge region, , , , The average depth of the three edge columns. The depth value of a pixel. , , , These represent the total number of pixels within the top, bottom, left, and right edge sampling areas, respectively. The horizontal depth gradient and the vertical depth gradient are calculated based on the average depth of the three edge columns of the region of interest, and their expressions are as follows:

[0015] Where 'a' is the radius of the region of interest.

[0016] Furthermore, the process of determining the planar fruit axis angle includes: When the fruit shape index of the directional bounding box meets the preset conditions, the mapped depth gradient direction angle is used as the apple plane fruit axis angle. When the fruit shape index deviates from the preset condition, the difference between the minor axis direction angle and the depth gradient direction angle is compared. If the difference is greater than the first threshold, the minor axis direction angle is flipped by 90° and used as the planar fruit axis angle. If the difference is less than the second threshold, the minor axis direction angle is directly output as the planar fruit axis angle.

[0017] Furthermore, the planar fruit axis angle is decomposed into unit direction components of the X-axis and Y-axis in the camera coordinate system; Using the principle of directional derivative, the depth change rate is obtained from the depth gradient of the region of interest along the fruit axis direction, and the normal component is obtained by multiplying the depth change rate with the unit vector of the Z-axis of the camera coordinate system. The unit direction component is combined with the normal component to obtain the spatial attitude vector; The spatial attitude vector is aligned with the horizontal plane of the camera coordinate system. The angle between the two points is taken as the pitch angle, and the projection of the spatial attitude vector onto the horizontal plane is taken as the angle between the two points. The included angle of the planes is referred to as the roll angle.

[0018] Furthermore, the coordinate system transformation module is specifically used for: The rotation matrix from the camera coordinate system to the robot base coordinate system is obtained through hand-eye calibration; the spatial attitude vector is transformed to the robot base coordinate system using the rotation matrix; and the pitch angle and roll angle of the robot arm are determined based on the attitude vector in the base coordinate system.

[0019] Secondly, embodiments of the present invention provide a method for estimating the spatial pose of shaded apples in an orchard for use with a harvesting robot, specifically including the following steps: Acquire an RGB image and a depth image containing the apple target, and register the RGB image and the depth image; The YOLO11-OBB model is used to detect apple targets in the RGB image, and the center coordinates, width, height, and major axis angle of the oriented bounding box are output. The minor axis direction angle is determined based on the major axis direction angle, and a region of interest is constructed in the depth image with the center of the oriented bounding box as the sampling center. The depth gradient direction angle of the region of interest is calculated. Based on the fruit shape index corresponding to the directional bounding box and the difference between the minor axis direction angle and the depth gradient direction angle, the planar fruit axis angle of the apple is determined from the minor axis direction angle and the depth gradient direction angle. The planar fruit axis angle is mapped to the normal vector of the region of interest using spatial components to obtain the spatial attitude vector of the apple in the camera coordinate system, and the pitch and roll angles of the apple are calculated based on the spatial attitude vector. The pitch and roll angles are converted into the robot arm's pitch and roll angles in the robot's base coordinate system to control the posture of the harvesting end.

[0020] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a spatial pose estimation system for shaded apples in orchards for harvesting robots, which has the following beneficial effects: 1. This invention makes a planar fruit axis angle decision by fusing the OBB short axis direction angle and the ROI depth gradient direction angle. By using the fruit shape index and angle difference judgment mechanism, it effectively avoids the problem of unreliable angle prediction from a single information source when the apple outline is missing due to its near-circular shape or occlusion. This improves the accuracy and robustness of planar fruit axis angle estimation.

[0021] 2. This invention performs spatial component mapping between the planar fruit axis angle and the ROI normal vector, realizing the complete derivation from two-dimensional planar angle to three-dimensional spatial attitude angle. It can estimate the spatial attitude angle of the apple in real time based solely on the apple fruit shape edge contour and surface depth gradient information without relying on calyx point detection or fruit segmentation. It has good adaptability to complex orchard environments such as branch and leaf shading. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a structural diagram of the spatial attitude estimation system provided in an embodiment of the present invention; Figure 2 This is a flowchart of the apple attitude angle calculation using YOLO11-OBB provided in this embodiment of the invention; Figure 3 This is a schematic diagram of apple posture in a modern dwarf rootstock densely planted apple orchard provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of apple longitudinal and transverse diameter measurement provided in an embodiment of the present invention; Figures 5(a)-(c) show different varieties of apples provided in the embodiments of the present invention; Figures 6(a)-(c) are statistical charts of fruit shape index of different varieties of apples provided in the embodiments of the present invention; Figures 7(a)-(f) are diagrams of the six dataset types of apples provided in the embodiments of the present invention; Figure 8 This is a schematic diagram illustrating the number of six types of data tags provided in this embodiment of the invention; Figure 9 This is a structural diagram of the YOLO11-OBB model provided in this embodiment of the invention; Figure 10 This is a schematic diagram of OBB major and minor axis conversion provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of solving the depth gradient direction angle provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of apple planar fruit axis angle decision provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the spatial fruit axis component mapping of an apple provided in an embodiment of the present invention; Figures 14(a)-(b) are schematic diagrams of the attitude angle verification system provided in the embodiments of the present invention; Figures 15(a)-(c) are schematic diagrams of pitch angle and roll angle provided in the embodiments of the present invention; Figures 16(a)-(d) are schematic diagrams of the pitch angle and roll angle of the end effector of the robotic arm provided in the embodiments of the present invention; Figures 17(a)-(b) are schematic diagrams of the visualization of matrix label box information provided in the embodiments of the present invention; Figure 18 This is a schematic diagram of the standardized confusion matrix provided in an embodiment of the present invention; Figures 19(a)-(b) are schematic diagrams of large errors in the planar fruit axis angle provided in the embodiments of the present invention; Figures 20(a)-(f) are schematic diagrams of large errors in the spatial attitude angle of apples provided in the embodiments of the present invention; Figure 21 This is a schematic diagram of the apple-picking robot structure provided in an embodiment of the present invention; Figure 22 This is a flowchart of the spatial attitude estimation method provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention discloses a spatial pose estimation system for shaded apples in orchards for harvesting robots. system, such as Figure 1 As shown, it includes: An image acquisition module is used to acquire an RGB image and a depth image containing an apple target, and to register the RGB image and the depth image; The directional target detection module uses the YOLO11-OBB model to detect apple targets in the RGB image and outputs the center coordinates, width, height, and major axis angle of the directional bounding box. The planar angle extraction module determines the minor axis angle based on the major axis angle, constructs a region of interest in the depth image with the center of the oriented bounding box as the sampling center, and calculates the depth gradient direction angle of the region of interest. The planar fruit axis decision module is used to determine the planar fruit axis angle of the apple from the minor axis angle and the depth gradient direction angle based on the fruit shape index corresponding to the directional bounding box and the difference between the minor axis direction angle and the depth gradient direction angle. The spatial attitude estimation module is used to perform spatial component mapping between the plane fruit axis angle and the normal vector of the region of interest to obtain the spatial attitude vector of the apple in the camera coordinate system, and calculate the pitch angle and roll angle of the apple based on the spatial attitude vector. The coordinate system transformation module is used to transform the pitch angle and the roll angle to the robot base coordinate system to obtain the robot arm pitch angle and robot arm roll angle, which serve as the basis for the robot arm's end-effector posture control.

[0026] This invention makes a planar fruit axis angle decision by fusing the OBB short axis direction angle and the ROI depth gradient direction angle. By using the fruit shape index and angle difference judgment mechanism, it effectively avoids the problem of unreliable angle prediction from a single information source when the apple outline is missing due to its near-circular shape or occlusion. This improves the accuracy and robustness of planar fruit axis angle estimation.

[0027] The overall workflow of the system is as follows Figure 2As shown, the image acquisition module first acquires RGB and depth images containing apple targets and registers them. Then, the directional target detection module uses the YOLO11-OBB model to detect apple targets and perform directional bounding box regression on the RGB images, outputting the center coordinates, width, height, and major axis angle of the directional bounding box corresponding to each apple target. The planar angle extraction module converts the major axis angle to obtain the minor axis angle, which serves as a candidate for the planar fruit axis direction. The larger of the width and height values ​​of the directional bounding box represents the major axis length, and the smaller value represents the minor axis length. Simultaneously, a region of interest (ROI) is constructed in the depth image with the center of the directional bounding box as the sampling center, and the depth gradient of the surface of the ROI is calculated to obtain the gradient direction angle. Next, the planar fruit axis decision module makes a decision based on the fruit shape index corresponding to the OBB box and the difference between the minor axis angle and the depth gradient direction angle to determine the final planar fruit axis angle of the apple. Next, the spatial attitude estimation module performs spatial component mapping between the planar fruit axis angle and the ROI normal vector, expanding the two-dimensional planar fruit axis vector into a three-dimensional spatial attitude vector in the camera coordinate system, and calculates the apple's pitch and roll angles based on this vector. Finally, the coordinate system transformation module transforms the above attitude angles to the robot's base coordinate system using the rotation matrix obtained from hand-eye calibration, and finally calculates and outputs the pitch and roll angles required for the robotic arm to pick the apple, providing a basis for attitude control of the picking end effector.

[0028] In modern, densely planted apple orchards using dwarf rootstocks, apple trees exhibit diverse growth postures, with drooping, lateral, and upright postures being the three most common. Specific examples can be found in [link to relevant documentation]. Figure 3 As shown; drooping fruits hang naturally with the stem pointing upwards and the calyx downwards. Robots can approach the fruit directly and easily, greatly reducing the difficulty of grasping it, making it the easiest posture type for robots to grasp. Lateral-growing fruits grow horizontally with a small angle between the stem and the branch, requiring the robot to precisely adjust the grasping angle to avoid collisions with the branch. Upright fruits grow upwards with the calyx upwards and the stem downwards, mostly on dense foliage or short branches. Compared to the previous two postures, upright apples are relatively rare in orchards, and their unique growth posture and complex obstructions make them more difficult for robots to grasp.

[0029] The Fruit Shape Index (SI) is an important indicator for measuring apple shape. This invention uses apple shape as a key basis to estimate apple posture. The Fruit Shape Index is the ratio of the fruit's longitudinal diameter to its transverse diameter. This ratio provides a direct indication of whether the apple is round or elongated. It is calculated by the ratio of the fruit's longitudinal diameter (L) to its transverse diameter (D). Figure 4 As shown in equation (1). Wherein, the longitudinal diameter is defined as the maximum axial length of the fruit from top to bottom, and the transverse diameter is the radial diameter at the widest point of the fruit.

[0030] (1) (2) As shown in equation (2), based on 1, apples with a fruit shape index less than 0.99 are classified as oblate, apples with a fruit shape index deviation less than 0.01 are classified as round, and apples with a fruit shape index greater than 1.01 are classified as oblong.

[0031] Among the morphological characteristics of apples with different shapes and colors, the differences in fruit shape among different varieties are significant. Fuji apples are typically oval with a flat and wide shoulder, as shown in Figure 5(a), and generally have a relatively low fruit shape index. In contrast, Huaniu apples are typically conical in shape, with a relatively tall and long appearance, as shown in Figure 5(b), and have a relatively high fruit shape index. Golden Delicious apples are also conical in shape, and although their fruit shape index is higher than that of Fuji, it is lower than that of the Huaniu variety, as shown in Figure 5(c).

[0032] The three types of apples mentioned above exhibit significant differences in color characterization and morphological structure. In this embodiment, through on-site surveys and measurements in orchards, 50 sets of fruit shape index data were randomly collected and statistically analyzed for each of the Fuji, Huaniu, and Golden Delicious apples. The average fruit shape indices of the three types of apples were 0.88, 1.21, and 1.08, respectively, as shown in Figure 6. This data difference clearly indicates that they have significant differences in fruit shape. Given the differences in posture angle estimation among apples of different shapes, the apple posture angle estimation system and method proposed in this invention need to have broad applicability and feasibility. Based on this, for the convenience of subsequent in-depth research and result analysis, the most common Fuji apple will be used as the main research object in the following text.

[0033] In this embodiment, relying on two platforms—a fully mechanized apple research base and a fully mechanized apple testing station—apple datasets were sampled in a major apple-producing area. An Intel RealSense D435 depth camera and multiple smartphones were used to capture images at different distances and angles, ultimately constructing a multimodal dataset of 2968 high-quality Fuji apple images. To facilitate subsequent model training, the resolution of the smartphone-captured dataset was cropped to match the 1280 resolution of the depth camera. The resolution is consistent at 720. In the dataset partitioning, the apple cases are defined based on the image features as follows: if an apple is only penetrated by a single branch in the image, it is determined to be branch occlusion (shelter_branch), as shown in Figure 7(a); when the main visible area of ​​an apple is occluded by other apples, it is classified as apple occlusion (shelter_leaf), as shown in Figure 7(b); if the occlusion source is mainly a leaf and the coverage area exceeds one-fifth of the fruit surface area, it is defined as leaf occlusion (shelter_apple), as shown in Figure 7(c); for apples with no obvious occlusion or only slight occlusion, they are considered to meet the direct picking conditions (pick), as shown in Figure 7(d); when an apple is only partially in the field of view, or when it is occluded by the main trunk causing more than one-fifth of the fruit to be invisible, it is marked as an incomplete apple (incomplete), as shown in Figure 7(e); if there are complex cases such as multiple occlusion types superimposed, they are uniformly classified as mixed occlusion (ignore), as shown in Figure 7(f). The dataset uses AnyLabeling for OBB annotation, with a total of 12,710 apple targets annotated across six datasets. The specific number of labels for each dataset is as follows: Figure 8 As shown.

[0034] like Figure 9 As shown in the YOLO11-OBB model structure diagram, YOLO11 employs an improved feature extraction backbone network and an enhanced feature fusion mechanism, improving multi-scale feature representation capabilities through optimized convolutional module design. In the decoder section, the detection head adds a rotation angle prediction branch to the standard object detection branch. mAP It is a comprehensive performance reflection of the model across all category labels, as shown in equation (3), where C For a set of categories, APc For the first c Average precision of the class.

[0035] (3) This model introduces an Oriented Bounding Box (OBB) with five parameters. It represents the target's position, size, and orientation, enabling tight wrapping and precise localization of tilted objects. Specifically, for each detected apple target, the model outputs the center of the oriented bounding box. Width (w), height (h), and the angle between the major axis and the horizontal axis of the image. Its angle range is Based on the morphological characteristics of Fuji apples, the fruit axis typically passes through the center of the fruit along its minor axis. Therefore, the minor axis direction angle is defined as the apple fruit axis angle. The angle between the minor axis of the OBB bounding box and the horizontal axis of the image is denoted as θ. Therefore, the minor axis direction angle needs to be derived from the major axis direction angle. Since the major and minor axes of the OBB box are perpendicular to each other, It is possible Obtained by rotating 90 degrees. Figure 10 All possible scenarios were demonstrated to ensure the minor axis orientation angle. The result is Within the range, segmented processing is adopted, and the calculation method is shown in equation (4).

[0036] (4) Because apple poses are highly variable, the OBB angle extracted solely from the fruit's edge contour is sometimes insufficient to accurately represent its planar axis direction. Therefore, this invention further incorporates ROI depth gradient information from the apple surface for subsequent joint determination of the axis angle. Considering the surface geometry of Fuji apples, the apple surface exhibits greater curvature along the axis direction, resulting in significant depth variations; while the curvature along the equator is less pronounced, leading to relatively gentler depth changes. Therefore, the direction of the fastest depth decrease, reflected by the gradient direction angle, geometrically points towards the axis direction. Thus, the gradient direction angle (… The angle can be approximated as the fruit axis angle, and the solution process is as follows: Figure 11 As shown. Centered on the OBB frame. Using the sampling center, construct a square region of interest (ROI) in the depth map, with a side length and radius of... a Using one-third of the minor axis length as a benchmark, and constrained by a double threshold of 15 pixels (upper limit 15 pixels, lower limit 10 pixels) to avoid the gradient angle approaching 0 due to excessively small ROI size. To effectively suppress random noise and outlier interference in the original depth map and improve the tolerance of invalid depth value data, this invention employs the edge-averaged depth difference method for gradient calculation. The edge-averaged depth difference method mainly involves averaging the depth values ​​of the neighboring areas on both sides of the ROI edge and then differencing them. Specifically, it calculates the gradient values ​​of the three columns of the ROI edge region (…). , , , The average depth of ) , , , As shown in formula (5). Wherein, The depth value of a pixel. , , , These represent the total number of pixels within the top, bottom, left, and right edge sampling areas, respectively.

[0037] (5) Based on the average edge depth mentioned above, the depth gradient is defined as the depth change per unit pixel. The horizontal depth gradient of the ROI is calculated accordingly. and vertical depth gradient As shown in Equation (6), this reflects the rate of depth change along the horizontal and vertical directions of the image. Further combining the gradient components from the two orthogonal directions yields the gradient descent vector. The direction of this vector points towards the direction of the fastest decrease in depth value within the ROI region, that is, from the surface region farther from the camera to the surface region closer to the camera, and its magnitude quantifies the drasticness of this depth change. Based on this, the gradient direction angle... Defined as the orientation angle of the gradient vector in the image plane, that is, the angle between the gradient vector and the horizontal axis of the image, with an angle range of... The gradient needs to be solved piecewise according to the sign of the gradient components, and the calculation method is shown in equation (7). To facilitate the subsequent direct subtraction with the minor axis angle of OBB to obtain the difference, the gradient direction angle is... Mapped to Interval As shown in equation (8): (6) (7) (8) Complete the OBB minor axis direction angle and gradient direction angle After solving for each of the two angles, the final apple plane axis angle needs to be determined from these two perspectives. In real orchard environments, deviations can occur in both directions due to factors such as fruit morphology variations, foliage shading, light variations, and depth sensor noise. OBB detection suffers from reduced angle prediction reliability when the target outline is nearly circular, and angle shifts also occur when shading causes severe outline loss. Depth gradient calculation relies entirely on the depth map and apple surface quality, which also leads to reduced gradient direction accuracy in open orchards.

[0038] To effectively integrate OBB geometric information and depth gradient information, and to coordinate potential orientation conflicts between them, a decision-making scheme for the apple plane fruit axis angle was designed. The decision-making principle is as follows: Figure 12 As shown. When the fruit shape index of the OBB box satisfies At this time, the apple's outline is close to a perfect circle, and the OBB shape information cannot reliably distinguish the fruit axis direction. Therefore, the gradient direction angle is directly used. As the apple's flat fruit axis angle When the fruit shape index deviates from this range, the minor axis direction angle is further compared. Angle with gradient direction Difference: If the difference is greater than 45°, it indicates that there is a misjudgment of the major and minor axes between the OBB frame and the actual fruit shape. The angle of the minor axis direction needs to be adjusted. After being flipped 90°, it becomes the flat fruit axis angle of the apple. If the difference is less than 45°, then the angle in the minor axis direction is... Reliable, directly outputs the angle of the apple's planar fruit axis. Mean absolute error (MAE) and root mean square error (RMSE) are commonly used error assessment metrics in regression tasks, and their calculation formulas are Equations (9) and (10), respectively. n The total number of samples, y i Let i be the true value of the i-th sample. Let be the predicted value for the i-th sample.

[0039] (9) (10) After determining the apple's planar fruit axis orientation, it needs to be combined with the ROI normal vector to construct the apple's pose vector in space. The basic principle of this mapping is that the fruit coordinate system and the camera coordinate system are parallel, which directly generates similar right-angled triangles in space. The fruit axis spatial vector is then the longer side of these similar triangles, as shown below. Figure 13 As shown. The image coordinate system is located within the camera coordinate system. On the plane, the coordinate axes of the two coordinate systems are aligned. Therefore, the orientation angle corresponding to the two-dimensional fruit axis angle in the image plane is... It can be directly used as the fruit coordinate system. and Component in unit direction and To characterize the tilt feature of the fruit axis in the depth dimension, perpendicular to... normal components of a plane This needs to be solved using depth gradient. Based on the principle of directional derivative, the rate of change of depth along the fruit axis can be calculated, and then compared with... Multiplying the unit vectors of the axes yields The calculation method is shown in equation (11). Therefore, the planar fruit axis vector can be extended to a three-dimensional vector in the camera coordinate system. As shown in equation (12). Wherein, i , j , k These are mutually orthogonal unit vectors in the fruit coordinate system.

[0040] (11) (12) Transform the vector v in the fruit coordinate system to the robot base coordinate system using the rotation matrix R between the camera coordinate system and the robot base coordinate system. As shown in equation (13), the unit vector of the apple's spatial pose in the robot's base coordinate system has been obtained. As shown in equation (14), the pitch angle of the apple can be solved. and roll angle Two attitude parameters. To eliminate directional ambiguity and facilitate subsequent angle transformation, both angles are mapped to [ ] interval. Wherein, pitch angle Describe the fruit axis relative to the horizontal plane The pitch angle, the solution and linear mapping process are shown in equation (15). Roll angle Describes the lateral deflection angle of the fruit axis in the horizontal plane, i.e., the vector and... The projection angle of the plane can be calculated and mapped using equation (16), where, They are respectively The components projected onto the X, Y, and Z axes.

[0041] (13) (14) (15) (16) To verify the estimation accuracy and system stability of the apple pose estimation algorithm in a real orchard setting, and thus support the robot in completing continuous and non-destructive harvesting operations, this invention constructs an apple pose angle verification system. This system includes an apple pose estimation platform and a self-developed apple pose projection angle measurement device, as shown in Figures 14(a) and 14(b), respectively. After the system starts the depth camera and acquires images, it uses a deep learning model to estimate the pose information of all apples in the image in real time. When the user selects any target apple in the image, the right panel dynamically displays the spatial position and pose angle of that apple. During the experiment, the operator aligned the projection angle measuring device with the smallest axis of the apple and performed the projection angle measurement, i.e., part (c) of Figure 14(b). Then, according to the projection angle conversion formula, the spatial vector formed by the apple core and calyx points was calculated to be perpendicular to the horizontal plane. and vertical plane The angle between them. Therefore, the system uses the measured data as the true value. The pose estimation results output by the algorithm Verification is required.

[0042] The calibration of the vision system is a crucial step in establishing the mapping relationship between the image coordinate system and the robot's base coordinate system, enabling the accurate conversion of planar orientation information in the image into a spatial pose representation in the base coordinate system. In the imaging and pose estimation system, the robot's base coordinate system is involved. Camera coordinate system Fruit coordinate system and image coordinate system The camera model transformation is based on the pinhole imaging principle, using camera intrinsic parameters to map image pixel coordinates to the camera coordinate system. The transformation relationship is shown in equation (17). For the target depth value, For pixel coordinates, , For camera focal length, These are the spatial coordinates in the camera coordinate system.

[0043] (17) Solve for the fruit's attitude angle in the fruit coordinate system. By combining the rotation matrix between the camera coordinate system and the robot base coordinate system, the pitch angle of the robotic arm in the robot base coordinate system can be further obtained. ) and roll angle ( ).

[0044] Figure 15(a) shows the relative orientation of the robot arm's base coordinate system and gripper coordinate system to help understand the robot arm's pitch angle. ) and roll angle ( The Euler angles defined here are not generated in the standard coordinate system around the fixture, but in an absolute coordinate system (base coordinate system). This is to clearly demonstrate the Euler angles of the robotic arm, as shown in Figures 15(a) and 15(b). The fruit coordinate system is transformed to align the apple's pose angles with the robot's base coordinate system using a transformation matrix.

[0045] To achieve precise apple grasping with ease of control by the robotic arm, this invention constructs a mathematical mapping model between the apple's posture angle and the end effector's posture. The robotic arm's joints rapidly rotate to their respective postures to reduce unnecessary movements. The initial pose of the end effector is defined as the grasping posture corresponding to an apple pitch angle and roll angle of 90°, as shown in Figures 16(a)-(d). In the specific posture calculation process, the pitch angle of the robotic arm's end effector (…) The angle of pitch relative to the apple The relationship satisfies equation (18). Similarly, the roll angle at the end of the robotic arm ( ) and the roll angle of the apple The relationship is given by equation (19). Through the above conversion model, the system can convert the apple posture parameters obtained by the vision sensor into motion control commands at the end of the robotic arm in real time, thereby ensuring that the grasping mechanism can always approach the target with the optimal posture.

[0046] (18) (19) In this embodiment, the dataset used in this invention includes 2968 apple images, which are divided into training and validation sets in a 9:1 ratio. The training set contains 2672 images used for learning the YOLO11-OBB model; the validation set contains 296 images, primarily used for performance evaluation and hyperparameter tuning during model training. To accelerate the training process of the deep learning model, this invention employs a rectangular training strategy, effectively reducing the additional computational burden caused by image padding. During model training, resolution settings are crucial to model performance. To ensure the model can learn rich apple features, the training resolution is set to match the maximum resolution of the depth camera, i.e., 1280. 1280. In addition, the model was trained using the YOLO11X-OBB weight file and a GPU RTX 4070 SUPER (12G).

[0047] After training the model on the apple dataset in this invention, the visualization results of each label are shown in Figures 17(a)-(b). The vast majority of labels in the figures exhibit clear rectangular outlines, fully demonstrating that the Fuji apples used in this study have regular shapes and are suitable for labeling and model training using oriented bounding boxes (OBB).

[0048]

[0049] Table 1. Detection Performance of YOLO11-OBB Model Wherein, mAP50 is the average accuracy with a threshold of 50%, and mAP50-95 is the average accuracy with a threshold of 50%-95%.

[0050] As shown in Table 1, the precision of various labels is generally lower than the recall. This is because, in the object detection task, the number of samples in categories such as "shelter_apple" (fruit occlusion) is highly unbalanced. Figure 8 As shown, the model's learning of these categories during training may not be sufficient. For categories with a small sample size, in order to detect as many samples as possible from these categories, the model lowers the threshold required to classify them. This can lead to some samples that do not belong to the category being incorrectly predicted, thus reducing precision. Consequently, precision is generally lower than recall. Figure 18As shown in the figure, the Confusion Matrix Normalized represents the normalized confusion matrix, Predicted represents the predicted value, True represents the actual value, incomplete represents the incomplete value, and background represents the background. Under the mAP50 and mAP50-95 indices, the directly picked category shows the best prediction performance, reaching 0.946 and 0.928 respectively. This is due to the sufficient sample size and simplicity of this category. The overall mAP50 and mAP50-95 indices for all labels are 0.834 and 0.775, respectively.

[0051]

[0052] Table 2. Planar fruit axis angle error at different test distances For Fuji apples in the orchard, error tests were conducted at different distances when the camera was approximately 0.5m, 1.0m, and 1.5m away from the apple target, as shown in Table 2. The MAE of the model at the three test distances were 2.94°, 6.36°, and 13.24°, respectively, and the RMSE were 4.03°, 10.85°, and 27.29°, respectively. It can be seen that as the distance between the camera and the apple target increases, the MAE and RMSE of the model in the plane angle prediction both show an increasing trend. Among them, the error values ​​of groups (39), (45), and (49) are relatively large. The former is mainly due to the fact that the shape index approaches 1, and the shape of the apple is very close to a circle in the image. Finally, the depth gradient direction is used as the plane angle, as shown in Figure 19(a). The latter two issues primarily arise because when calculating apple depth information at a long distance using a camera, the depth accuracy of the region of interest (ROI) decreases significantly, and the number of available pixels in the region also decreases accordingly. This leads to a large deviation between the depth gradient direction angle and the directed bounding box (OBB) angle, resulting in errors in the decision of the planar angle, as shown in Figure 19(b). Experiments show that the algorithm still maintains high accuracy at a working distance of approximately 1.0m. Furthermore, the MAE and RMSE of the planar apple axis angle of the algorithm are 5.9° and 15.9°, respectively, at different test distances.

[0053]

[0054] Table 3. Apple Spatial Attitude Angle Error Test Table To test the robustness of the algorithm used in the system of this invention under various orchard conditions, a total of 30 sets of experiments were conducted, covering five apple tag categories: pick (direct picking), shelter_leaf (leaf occlusion), shelter_branch (branch occlusion), shelter_apple (fruit occlusion), and ignore (mixed occlusion). Each set of data recorded the specific tag category of the apple, the distance between the fruit and the camera, and the corresponding pitch and roll angle errors of the apple within the camera's field of view, as shown in Table 3. The experimental results showed that the mean absolute error of the pitch angle and the mean absolute error of the roll angle were 18.7° and 8.3°, respectively; the root mean square error of the pitch angle and the root mean square error of the roll angle were 26.2° and 11.8°, respectively. This indicates that the pitch angle error is much larger than the roll angle error. Because Fuji apples have a greater longitudinal curvature than lateral curvature when ripe, the greater the curvature in the image, the greater the gradient change of the pixels in that direction. Therefore, the gradient change of the pitch angle in the vertical direction is more sensitive to angle estimation than the gradient change of the roll angle in the horizontal direction.

[0055] The experimental group with an average absolute error of pitch and roll angle greater than 20° was selected for focused analysis to explore the effects of various factors on the apple's attitude angle. α The effects of β) are specifically observed in groups (4), (9), (12), (17), (24), (27), (28), and (29). Illumination conditions are crucial for the accurate acquisition of image depth information. Under different lighting scenarios, sunlight can interfere with the near-infrared light used by the depth camera for distance measurement, thereby altering the distribution of the depth gradient on the apple surface, as shown in Figure 20(a). When the apple shape index approaches 1, the apple image outline is approximately circular, and the apple plane angle will ultimately be selected by the depth gradient algorithm. At a greater test distance, the obtained error value is usually larger, as shown in Figure 20(b). The algorithm proposed in this study needs to calculate the edge gradient of the apple's region of interest (ROI). Therefore, this region must be the surface of the apple fruit, and cannot include the calyx region or the stem region, and must ensure that there are no obstructions such as branches and leaves. Different interference scenarios are shown in Figures 20(c), 20(d), 20(e), and 20(f), respectively. These interference factors will affect the gradient calculation of the ROI. Experiments on apple spatial angle errors demonstrate that the OBB-based apple pose estimation algorithm has a certain degree of tolerance to surrounding obstructions such as tree branches, and exhibits good practicality and applicability.

[0056] In this embodiment, an apple-picking robot posture experiment and verification were conducted in an orchard. In this verification, the apple-picking robot system used was based on an Intel RealSense D435 depth camera to achieve real-time apple pose estimation, coupled with an auboi5 robotic arm to perform dynamic picking posture adjustment, and a self-developed two-finger pneumatic suction end effector to achieve firm fruit gripping. Figure 21 As shown. Through 30 multi-pose picking experiments with the robot, the overall picking success rate of the system reached 80%, an improvement of 10% compared to traditional methods. Current picking robots employ a conservative end effector design, reserving a large margin to ensure successful grasping. Applying the posture algorithm proposed in this paper to a compact end effector will further significantly improve the rate of secure and damage-free grasping.

[0057] Based on the same inventive concept, embodiments of the present invention also provide a method for estimating the spatial pose of shaded apples in orchards for use with harvesting robots, such as... Figure 22 As shown, it includes the following steps: S1. Obtain an RGB image and a depth image containing the apple target, and register the RGB image and the depth image; S2. Use the YOLO11-OBB model to detect the apple target in the RGB image and output the center coordinates, width, height and major axis angle of the oriented bounding box; S3. Determine the minor axis direction angle based on the major axis direction angle, and construct a region of interest in the depth image with the center of the oriented bounding box as the sampling center, and calculate the depth gradient direction angle of the region of interest. S4. Based on the fruit shape index corresponding to the directional bounding box and the difference between the minor axis direction angle and the depth gradient direction angle, determine the planar fruit axis angle of the apple from the minor axis direction angle and the depth gradient direction angle; S5. Map the plane fruit axis angle to the normal vector of the region of interest using spatial components to obtain the spatial attitude vector of the apple in the camera coordinate system, and calculate the pitch angle and roll angle of the apple based on the spatial attitude vector. S6. Convert the pitch angle and roll angle into the robot arm pitch angle and roll angle in the robot base coordinate system to control the posture of the picking end.

[0058] Since the principle behind this method is similar to that of the aforementioned spatial pose estimation system for orchard occluded apples for harvesting robots, the implementation of this method can be found in the implementation of the aforementioned system, and the repetitions will not be repeated.

[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A spatial pose estimation system for shaded apples in orchards for harvesting robots, characterized in that, include: An image acquisition module is used to acquire an RGB image and a depth image containing an apple target, and to register the RGB image and the depth image; The directional target detection module uses the YOLO11-OBB model to detect apple targets in the RGB image and outputs the center coordinates, width, height, and major axis angle of the directional bounding box. The planar angle extraction module determines the minor axis angle based on the major axis angle, constructs a region of interest in the depth image with the center of the oriented bounding box as the sampling center, and calculates the depth gradient direction angle of the region of interest. The planar fruit axis decision module is used to determine the planar fruit axis angle of the apple from the minor axis angle and the depth gradient direction angle based on the fruit shape index corresponding to the directional bounding box and the difference between the minor axis direction angle and the depth gradient direction angle. The depth gradient direction angle is defined as the angle between the gradient vector and the horizontal axis of the image, and its expression is: Where arctan(·) is the arctangent function. g v The vertical depth gradient g u The horizontal depth gradient; Depth gradient direction angle Mapping to interval get , The expression is: The vertical and horizontal depth gradients are calculated using the edge-average depth difference method. The depth values ​​of the neighboring areas on both sides of the edge of the region of interest are averaged and then differencing to calculate the average depth of the three edge columns of the region of interest. The expression for the average depth of the three edge columns is as follows: in, , , , For the three-column edge region, , , , The average depth of the three edge columns. The depth value of a pixel. , , , These represent the total number of pixels within the sampling areas of the left, right, top, and bottom edges, respectively. The horizontal depth gradient and the vertical depth gradient are calculated based on the average depth of the three edge columns of the region of interest, and their expressions are as follows: Where a is the radius of the region of interest; The spatial attitude estimation module is used to perform spatial component mapping between the plane fruit axis angle and the normal vector of the region of interest to obtain the spatial attitude vector of the apple in the camera coordinate system, and calculate the pitch angle and roll angle of the apple based on the spatial attitude vector. The coordinate system transformation module is used to transform the pitch angle and the roll angle to the robot base coordinate system to obtain the robot arm pitch angle and robot arm roll angle, which serve as the basis for the robot arm's end-effector posture control.

2. The orchard shading apple spatial pose estimation system for harvesting robots as described in claim 1, characterized in that, The YOLO11-OBB model adds a rotation angle prediction branch to the standard object detection branch, and represents the oriented bounding box in the form of five parameters. The five parameters include: center coordinates, width, height, and the angle between the major axis and the horizontal axis of the image. The value range of the horizontal axis angle is [-π / 2, π / 2].

3. The orchard shading apple spatial pose estimation system for harvesting robots as described in claim 1, characterized in that, The minor axis angle is obtained by rotating the major axis angle by 90° and then constraining it to the range of [-π / 2, π / 2) through segmentation. In the width and height of the directional bounding box, the larger value is the major axis length and the smaller value is the minor axis length.

4. The orchard shading apple spatial pose estimation system for harvesting robots as described in claim 1, characterized in that, The region of interest is a square region constructed with the center of the oriented bounding box as the sampling center. It is constructed in the depth map. The radius of the side of the region of interest is based on one-third of the length of the minor axis and is determined by upper and lower threshold constraints.

5. The orchard shading apple spatial pose estimation system for harvesting robots as described in claim 1, characterized in that, The process of determining the angle of the planar fruit axis includes: When the fruit shape index of the directional bounding box meets the preset conditions, the mapped depth gradient direction angle is used as the apple plane fruit axis angle. When the fruit shape index deviates from the preset condition, the difference between the minor axis direction angle and the depth gradient direction angle is compared. If the difference is greater than the first threshold, the minor axis direction angle is flipped by 90° and used as the planar fruit axis angle. If the difference is less than the second threshold, the minor axis direction angle is directly output as the planar fruit axis angle.

6. The orchard shading apple spatial pose estimation system for harvesting robots as described in claim 1, characterized in that, The planar fruit axis angle is decomposed into unit direction components of the X and Y axes in the camera coordinate system; Using the principle of directional derivative, the depth change rate is obtained from the depth gradient of the region of interest along the fruit axis direction, and the normal component is obtained by multiplying the depth change rate with the unit vector of the Z-axis of the camera coordinate system. The unit direction component is combined with the normal component to obtain the spatial attitude vector; The pitch angle is defined as the angle between the spatial attitude vector and the horizontal plane of the camera coordinate system, and the projection of the spatial attitude vector onto the horizontal plane of the camera coordinate system and the fruit coordinate system. The included angle of the planes is referred to as the roll angle.

7. The orchard shading apple spatial pose estimation system for harvesting robots as described in claim 1, characterized in that, The coordinate system transformation module is specifically used for: The rotation matrix from the camera coordinate system to the robot base coordinate system is obtained through hand-eye calibration; the spatial attitude vector is transformed to the robot base coordinate system using the rotation matrix; and the pitch angle and roll angle of the robot arm are determined based on the attitude vector in the base coordinate system.

8. A method for estimating the spatial pose of shaded apples in orchards for harvesting robots, characterized in that, Includes the following steps: S1. Obtain an RGB image and a depth image containing the apple target, and register the RGB image and the depth image; S2. Use the YOLO11-OBB model to detect the apple target in the RGB image and output the center coordinates, width, height and major axis angle of the oriented bounding box; S3. Determine the minor axis direction angle based on the major axis direction angle, and construct a region of interest in the depth image with the center of the oriented bounding box as the sampling center, and calculate the depth gradient direction angle of the region of interest. S4. Based on the fruit shape index corresponding to the directional bounding box and the difference between the minor axis direction angle and the depth gradient direction angle, determine the planar fruit axis angle of the apple from the minor axis direction angle and the depth gradient direction angle; The depth gradient direction angle is defined as the angle between the gradient vector and the horizontal axis of the image, and its expression is: Where arctan(·) is the arctangent function. g v The vertical depth gradient g u The horizontal depth gradient; Depth gradient direction angle Mapping to interval get , The expression is: The vertical and horizontal depth gradients are calculated using the edge-average depth difference method. The depth values ​​of the neighboring areas on both sides of the edge of the region of interest are averaged and then differencing to calculate the average depth of the three edge columns of the region of interest. The expression for the average depth of the three edge columns is as follows: in, , , , For the three-column edge region, , , , The average depth of the three edge columns. The depth value of a pixel. , , , These represent the total number of pixels within the sampling areas of the left, right, top, and bottom edges, respectively. The horizontal depth gradient and the vertical depth gradient are calculated based on the average depth of the three edge columns of the region of interest, and their expressions are as follows: Where a is the radius of the region of interest; S5. Map the plane fruit axis angle to the normal vector of the region of interest using spatial components to obtain the spatial attitude vector of the apple in the camera coordinate system, and calculate the pitch angle and roll angle of the apple based on the spatial attitude vector. S6. Convert the pitch angle and roll angle into the robot arm pitch angle and roll angle in the robot base coordinate system to control the posture of the picking end.

Images