Real-time apple space attitude angle estimation method and system in orchard environment

Abstract

The invention belongs to the technical field of agriculture and computer vision, and discloses an apple real-time space attitude angle estimation method and system in an orchard environment, and the method comprises the following steps: obtaining a segmentation mask image of an apple fruit image when a YOLOv8 model does not recognize the calyx of the apple fruit, and combining with a rotational inertia algorithm to obtain a real-time space attitude angle of the apple; estimating the maximum transverse diameter and the first attitude angle of the apple which cannot identify the calyx; when the YOLOv8 model recognizes the calyx of the apple fruit, obtaining the surface center coordinate of the apple fruit, and calculating the position of the fruit core based on the maximum transverse diameter; and constructing a calyx-core vector based on the core position, decomposing the calyx-core vector into an X ''O'' Y ''plane and a Y'' O ''Z'' plane through space geometry, and further calculating a second attitude angle.The apple attitude estimation method provided by the invention can assist a robot in accurately estimating the attitude of the apple, reduce damage and improve the picking success rate.

Description

Technical Field

[0001] The present invention belongs to the fields of agriculture and computer vision technology, and particularly relates to a method and system for estimating the real-time spatial posture angle of apples in an orchard environment. Background Art

[0002] The apple industry has become a major fruit sector, not only contributing significantly to farmers' income growth and economic development but also playing a crucial role in modern agricultural development. With the booming apple industry, global apple production and cultivated area have increased rapidly. However, the degree of mechanization in apple harvesting remains low, with manual picking still the primary method. This significantly reduces picking efficiency and increases labor costs, making the development of harvesting robots particularly important. Vision systems are a key enabler for the development of harvesting robots, and posture estimation is a crucial foundation for developing vision systems for harvesting robots.

[0003] Fruit pose information, a crucial vector parameter for fruit characterization, includes complex features such as the fruit's tilt angle. Pose estimation algorithms obtain the fruit's angles, providing information for the robot to subsequently adjust the appropriate grasping posture (pitch, yaw, and roll angles). Under specific laboratory conditions, the pose estimation algorithm of (Giefer et al., 2019) employed a cascaded structure consisting of two independent convolutional neural networks, achieving an angular error of less than 15° for apples. Accurately identifying apples obscured by branches and leaves is a key challenge for harvesting robots. (Niu et al., 2017) used the convex hull theory and the Shape Context algorithm to accurately detect the symmetry axis of partially occluded individual apples. Testing on 90 partially occluded individual apple images revealed an average angular error of 7.72° for the Shape Context algorithm. To meet the needs of different harvesting end-effectors and improve the success rate of automated fruit picking, (Jang and Hwang, 2024) trained a YOLOv8 model to detect and segment tomato fruit and sepals. Point clouds of the two components were obtained from RGB-D data, and the center point of each object was calculated using spherical fitting and point cloud statistics. The tomato's pose was then estimated using the center points of the fruit and sepals as vectors. (Sun et al., 2023) proposed a multi-task model, Point Fractal Network (FPENeT), to locate the navel of citrus fruits and predict their rotation vectors. (Kok and Chen, 2024) studied a method for estimating apple orientation based on keypoint detection and occluded apple circle detection. The extracted information was used to calculate the unit vector of the apple's orientation. The median error angle in orchard experiments was 17.6°, compared to 14.6° in the laboratory. Numerous researchers have studied fruit posture using various methods. Algorithms for posture estimation using vectors have clear posture angles, allowing for a direct description of the fruit's rotational state in three-dimensional space and transforming abstract posture information into a mathematical model. These vector parameters can be directly input into a robot controller to guide the robotic arm in adjusting the posture of the end effector, opening up the possibility for subsequent engineering applications.

[0004] In the complex, unstructured environments of real orchards, apple-picking robots lack the ability to accurately grasp the specific posture angles of the fruit. This makes it difficult to construct a mathematical model that can be directly input into the robot controller, effectively preventing them from accurately guiding the robotic arm and the end-effector to the appropriate posture. Inaccurate posture recognition can cause the end-effector to accidentally contact branches, breaking them or damaging the apple skin, potentially compromising the robot's harvesting success rate. Summary of the Invention

[0005] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:

[0006] A method for estimating the real-time spatial attitude angle of apples in an orchard environment comprises the following steps:

[0007] When the YOLOv8 model fails to recognize the calyx of the apple fruit, the segmentation mask of the apple fruit image is obtained and combined with the moment of inertia algorithm to estimate the maximum horizontal diameter and the first posture angle of the apple that failed to recognize the calyx;

[0008] When the YOLOv8 model recognizes the calyx of the apple fruit, the surface center coordinates of the apple fruit are obtained, and the fruit center position is calculated based on the maximum transverse diameter;

[0009] A calyx-fruit center vector is constructed based on the fruit center position, and the calyx-fruit center vector is decomposed into Plane and plane, and then calculate the second attitude angle.

[0010] Preferably, the method for obtaining the maximum transverse diameter includes:

[0011] Obtain the segmentation mask of the apple fruit image and calculate the minimum and maximum moments of inertia of the apple using the moment of inertia algorithm.

[0012] Obtaining a minimum moment of inertia axis based on the minimum moment of inertia, and obtaining a maximum moment of inertia axis based on the maximum moment of inertia;

[0013] Calculating four pixel intersections of the minimum moment of inertia axis and the maximum moment of inertia axis with the edge of the segmentation mask image respectively;

[0014] The four pixel intersections are converted into spatial coordinates in the robot base coordinate system, and the maximum axis length is determined by calculating the Euclidean distance to obtain the maximum transverse diameter.

[0015] Preferably, the method for obtaining the first attitude angle includes:

[0016] Calculating a roll angle θ based on the minimum moment of inertia;

[0017] Let the pitch angle be 0 to obtain the first attitude angle (0, θ).

[0018] Preferably, the method for calculating the position of the fruit center includes:

[0019] Get the surface center coordinates of the apple fruit , set the robot base coordinate system to ;

[0020] Then the vector from the base coordinate to the apple is , the vector from the base coordinate to the robot camera is , and then the vector from the robot camera to the surface center coordinate of the apple fruit is obtained as ;

[0021] Vector-based , the surface center coordinates and the maximum transverse diameter, and obtain the position of the fruit center:

[0022] ,

[0023] ,

[0024] in, represents a unit vector, Indicates the center of the fruit, D indicates the maximum transverse diameter, and R indicates the maximum radius. .

[0025] Preferably, the method for calculating the second posture angle includes:

[0026] Get the calyx coordinates , based on the surface center coordinates and the fruit center position , calculate the calyx-fruit center vector 、 Normal vector of the plane and Normal vector of the plane :

[0027] ,

[0028] ,

[0029] ;

[0030] Based on the calyx-fruit center vector , the normal vector and the normal vector , calculate the second posture angle (α, β):

[0031] ,

[0032] .

[0033] The present invention also provides a real-time spatial attitude angle estimation system for apples in an orchard environment, wherein the system applies the above method and comprises: a first attitude angle calculation module, a fruit center position calculation module, and a second attitude angle calculation module;

[0034] The first attitude angle calculation module is used to obtain a segmentation mask of the apple fruit image when the YOLOv8 model fails to recognize the calyx of the apple fruit, and estimate the maximum transverse diameter and the first attitude angle of the apple that failed to recognize the calyx in combination with the moment of inertia algorithm;

[0035] The fruit center position calculation module is used to obtain the surface center coordinates of the apple fruit when the YOLOv8 model recognizes the calyx of the apple fruit, and calculate the fruit center position based on the maximum horizontal diameter;

[0036] The second attitude angle calculation module constructs a calyx-fruit center vector based on the fruit center position, and decomposes the calyx-fruit center vector into Plane and plane, and then calculate the second attitude angle.

[0037] Preferably, in the first attitude angle calculation module, the process of calculating the maximum transverse diameter includes:

[0038] Obtain the segmentation mask of the apple fruit image and calculate the minimum and maximum moments of inertia of the apple using the moment of inertia algorithm.

[0039] Obtaining a minimum moment of inertia axis based on the minimum moment of inertia, and obtaining a maximum moment of inertia axis based on the maximum moment of inertia;

[0040] Calculating four pixel intersections of the minimum moment of inertia axis and the maximum moment of inertia axis with the edge of the segmentation mask image respectively;

[0041] The four pixel intersections are converted into spatial coordinates in the robot base coordinate system, and the maximum axis length is determined by calculating the Euclidean distance to obtain the maximum transverse diameter.

[0042] Preferably, in the first attitude angle calculation module, the process of calculating the first attitude angle includes:

[0043] Calculating a roll angle θ based on the minimum moment of inertia;

[0044] Let the pitch angle be 0 to obtain the first attitude angle (0, θ).

[0045] Preferably, the workflow of the fruit center position calculation module includes:

[0046] Get the surface center coordinates of the apple fruit , set the robot base coordinate system to ;

[0047] Then the vector from the base coordinate to the apple is , the vector from the base coordinate to the robot camera is , and then the vector from the robot camera to the surface center coordinate of the apple fruit is obtained as ;

[0048] Vector-based , the surface center coordinates and the maximum transverse diameter, and obtain the position of the fruit center:

[0049] ,

[0050] ,

[0051] in, represents a unit vector, Indicates the center of the fruit, D indicates the maximum transverse diameter, and R indicates the maximum radius. .

[0052] Preferably, in the second posture angle calculation module, the process of calculating the second posture angle includes:

[0053] Get the calyx coordinates , based on the surface center coordinates and the fruit center position , calculate the calyx-fruit center vector 、 Normal vector of the plane and Normal vector of the plane :

[0054] ,

[0055] ,

[0056] ;

[0057] Based on the calyx-fruit center vector , the normal vector and the normal vector , calculate the second posture angle (α, β):

[0058] ,

[0059] .

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The accuracy of apple and calyx detection by the model in the present invention is 0.953 and 0.703 respectively, and the accuracy of fruit segmentation is 0.953. The proposed apple posture estimation method can assist the robot to accurately estimate the apple posture, reduce damage and improve the picking success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0064] Figure 2 A straight line modeling diagram of an embodiment of the present invention;

[0065] Figure 3 Schematic diagram of a first attitude angle estimation according to an embodiment of the present invention;

[0066] Figure 4 Schematic diagram of a second posture angle estimation according to an embodiment of the present invention;

[0067] Figure 5 Schematic diagram of a measuring device according to an embodiment of the present invention, wherein (a) is a structural diagram of the device, (b) is a schematic diagram of the device coordinate system, (c) is a schematic diagram of the attitude angle direction, (d) is a schematic diagram of the projection angle θ1, and (e) is a schematic diagram of the projection angle θ2. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] The angle between the apple stalk and the fruit has biomechanical characteristics. Fruit farmers adjust their hand posture to match the natural posture of the apple, achieving more efficient and lossless apple picking. When picking a single fruit, fruit farmers only need to adjust the pitch and roll angles of their hands. Therefore, in order to facilitate the establishment of the corresponding mathematical model and the convenient transmission of parameters to the robot controller, the apple posture angle is defined, that is, the calyx-fruit center vector is respectively The plane angle α and Plane β.

[0071] Example 1:

[0072] In this embodiment, if Figure 1 As shown, a method for estimating the real-time spatial posture angle of apples in an orchard environment includes the following steps:

[0073] S1. When the YOLOv8 model fails to recognize the calyx of an apple fruit, obtain a segmentation mask of the apple fruit image and, combined with the moment of inertia algorithm, estimate the maximum transverse diameter and first attitude angle of the apple that failed to recognize the calyx.

[0074] The method for obtaining the maximum horizontal diameter includes: obtaining a segmentation mask image of an apple fruit image, and calculating the minimum and maximum moments of inertia of the apple in combination with a moment of inertia algorithm; obtaining a minimum moment of inertia axis based on the minimum moment of inertia, and obtaining a maximum moment of inertia axis based on the maximum moment of inertia; respectively calculating four pixel intersections of the minimum moment of inertia axis and the maximum moment of inertia axis with the edge of the segmentation mask image; converting the four pixel intersections into spatial coordinates in a robot base coordinate system, and determining the maximum axis length by calculating the Euclidean distance to obtain the maximum horizontal diameter.

[0075] In this embodiment, apples usually grow along the fruit axis (i.e., the direction from the fruit stalk to the calyx), so the direction of the axis of minimum moment of inertia is approximately the direction of the fruit axis; the moment of inertia can be calculated by using the fruit mask image and binarizing it.

[0076] The minimum moment of inertia of a binary image can be determined by finding the axis that minimizes the moment of inertia of the object. The direction of this axis represents the orientation of the binary image. In the two-dimensional case, the moment of inertia is obtained by calculating the square of the distance from each point on the object to the specified straight line and integrating it according to the density. By calculating the moment of inertia of the binary image, we can find this axis and then determine the orientation of the binary image. In formula (1), b(x,y) is the characteristic function of the binary image, and S is the area of ​​the binary image represented by the characteristic function. In formula (2), is the centroid coordinate of the weighted average of the binary image density, which is the static moment of the binary image.

[0077] , (1)

[0078] , (2)

[0079] Where I represents the integration range. Assuming a straight line minimizes the moment of inertia of the binary image, the moment of inertia is solved as shown in Equation (3), where r is the distance from any point in the binary image to the straight line.

[0080] , (3)

[0081] The distance ρ from the origin to the straight line and the counterclockwise angle between the straight line and the x-axis (i.e., the roll angle) θ are used to model the straight line, as shown in Equation (4). Figure 2 As shown, this modeling method is more convenient. When the coordinate system is translated or rotated, the two parameters can change continuously. Compared with the method of representing a straight line with an intercept and a slope, this method does not cause any problems with the two parameters when the straight line is parallel to the coordinate axis.

[0082] , (4)

[0083] , (5)

[0084] Let the point on the line closest to the origin be , moving any distance s along θ through this point, the point after movement is still on the line, and any point on the line (x0, y0) can be expressed as Equation (5). The distance between any point (x, y) in the binary image and the nearest point (x0, y0) on the given line can be expressed as Equation (6), and substituting Equation (5) into Equation (6) yields Equation (7). When x, y, ρ, and θ are known, solve for s that minimizes the distance r between the two points. Therefore, taking the derivative of s in Equation (8) yields Equation (9). The relationship between any point (x, y) on the binary image and the nearest point (x0, y0) on the hypothetical line can be obtained by substituting Equation (8) into Equation (5) to yield Equation (9).

[0085] , (6)

[0086] , (7)

[0087] , (8)

[0088] , (9)

[0089] , (10)

[0090] In summary, we know how to calculate the distance from any point on a binary image to the hypothetical line. Therefore, substituting Equation (10) into Equation (3) yields Equation (11). Taking the derivative of ρ and setting the reciprocal to 0 yields Equation (12). Therefore, the axis corresponding to the minimum second-order moment passes through the centroid of the binary image.

[0091] , (11)

[0092] , (12)

[0093] To determine the angle of the line, the center of mass of the binary image can be translated to coincide with the origin of the coordinate system, eliminating the distance ρ from the origin to the line. Only the inclination of the line needs to be solved, which facilitates the calculation of the formula. The moment of inertia is calculated as shown in formula (13). The coordinates of any point on the translated binary image are (x′, y′). At the same time, the moment of inertia can also be expressed as formula (14), where , , , after simplification, we can obtain Equation (15). First, take the derivative of Equation (15) with respect to θ, then set the derivative equal to zero. Finally, assuming a ≠ c, we can obtain Equation (16). If b = 0 and a = c, we can obtain Equation (17). At this point, the two axes of minimum and maximum moments of inertia of the binary image have been solved.

[0094] , (13)

[0095] , (14)

[0096] , (15)

[0097] , (16)

[0098] , (17)

[0099] The apple image is calculated using the YOLOv8 model combined with the moment of inertia algorithm to solve the two axes of the apple's minimum and maximum moment of inertia.

[0100] After calculating the moment of inertia of the apple image, the intersection of the two axes of maximum and minimum inertia with the four pixels at the edge of the fruit segmentation image (the endpoints of the inertia axes) is calculated. Image edges often contain noise. To prevent invalid depth values ​​at the four endpoints, which would prevent the correct calculation of the endpoint's spatial coordinates, the algorithm repeatedly randomly takes depth values ​​around the endpoints. These depth values ​​are then bubble sorted and median filtered. The final mean of the depth values ​​is determined as the valid depth value for the endpoint. The pseudo code for this implementation is shown in Table 1.

[0101] Table 1

[0102] .

[0103] The calibration of the visual system is a key step in determining the relationship between the pixel coordinate system and the world coordinate system. It enables the two-dimensional pixel coordinates to be accurately converted into the three-dimensional coordinates of the robot. The accuracy of the calibration directly determines the accuracy of the robot positioning. The relationship between the world coordinate system and the pixel coordinate system is shown in formula (18). Where u and v are pixel points, X is the value of the pixel coordinate system. W 、YW 、Z W is the corresponding world coordinate point, f x 、f y is the focal length of the camera, u0 and v0 are the main points, Z c is the target depth, R' is a 3×3 rotation orthogonal matrix, and T is a 3×1 translation matrix.

[0104] ; (18)

[0105] Hand-eye calibration is performed based on affine transformation combined with the least squares method, and the four intersection points are converted into spatial coordinates in the robot's base coordinate system. Finally, the maximum axis length is determined by Euclidean distance, which is the maximum horizontal diameter of the apple.

[0106] The method for obtaining the first attitude angle includes: calculating the roll angle θ based on the minimum moment of inertia in formula (17); since the fluctuation range of the pitch angle is narrow, the pitch angle is approximated to 0 to obtain the first attitude angle (0, θ), as shown in FIG. Figure 3 shown.

[0107] S2. When the YOLOv8 model identifies the calyx of an apple fruit, it obtains the surface center coordinates of the apple fruit and calculates the center position of the fruit based on the maximum horizontal diameter.

[0108] Methods for calculating the fruit center position include:

[0109] Get the surface center coordinates of the apple fruit , set the robot base coordinate system to ;

[0110] Then the vector from the base coordinate to the apple is , the vector from the base coordinate to the robot camera is , and then the vector from the robot camera to the surface center coordinate of the apple fruit is obtained as ;

[0111] Vector-based , surface center coordinates And the maximum transverse diameter, get the center position of the fruit:

[0112] , (19)

[0113] , (20)

[0114] in, represents a unit vector, Indicates the center of the fruit, D indicates the maximum transverse diameter, and R indicates the maximum radius. .

[0115] S3. Construct the calyx-fruit center vector based on the fruit center position, and decompose the calyx-fruit center vector into Plane and plane, and then calculate the second attitude angle.

[0116] In this embodiment, to facilitate the subsequent calculation of the attitude angle, a fruit coordinate system is defined based on the principle of similar triangles and with the camera coordinate system as a reference, and finally the apple attitude angle is defined.

[0117] The method for calculating the second posture angle includes: obtaining the calyx coordinates , based on the surface center coordinates and fruit center position , calculate the calyx-fruit center vector 、 Normal vector of the plane and Normal vector of the plane :

[0118] , (twenty one)

[0119] , (twenty two)

[0120] , (twenty three)

[0121] Based on calyx-fruit center vector , normal vector and normal vector , calculate the second attitude angle (α, β), such as Figure 4 As shown:

[0122] , (twenty four)

[0123] . (25)

[0124] Example 2:

[0125] In this embodiment, a real-time spatial attitude angle estimation system for apples in an orchard environment includes: a first attitude angle calculation module, a fruit center position calculation module, and a second attitude angle calculation module.

[0126] The first attitude angle calculation module is used to obtain the segmentation mask map of the apple fruit image when the YOLOv8 model fails to recognize the calyx of the apple fruit, and combine it with the moment of inertia algorithm to estimate the maximum transverse diameter and first attitude angle of the apple that failed to recognize the calyx.

[0127] In the first attitude angle calculation module, the process of calculating the maximum horizontal diameter includes: obtaining the segmentation mask map of the apple fruit image, and combining the moment of inertia algorithm to calculate the minimum moment of inertia and maximum moment of inertia of the apple; obtaining the minimum moment of inertia axis based on the minimum moment of inertia, and obtaining the maximum moment of inertia axis based on the maximum moment of inertia; respectively calculating the four pixel intersections of the minimum moment of inertia axis and the maximum moment of inertia axis with the edge of the segmentation mask map; converting the four pixel intersections into spatial coordinates in the robot base coordinate system, and determining the maximum axis length by calculating the Euclidean distance to obtain the maximum horizontal diameter.

[0128] In the first attitude angle calculation module, the process of calculating the first attitude angle includes: calculating the roll angle θ based on the minimum moment of inertia; since the fluctuation range of the pitch angle is relatively narrow, the pitch angle is approximated to 0 to obtain the first attitude angle (0, θ).

[0129] The fruit center position calculation module is used to obtain the surface center coordinates of the apple fruit when the YOLOv8 model recognizes the calyx of the apple fruit, and calculate the fruit center position based on the maximum horizontal diameter.

[0130] The workflow of the fruit center position calculation module includes: obtaining the surface center coordinates of the apple fruit , set the robot base coordinate system to ; Then the vector from the base coordinate to the apple is , the vector from the base coordinate to the robot camera is , and then the vector from the robot camera to the surface center coordinate of the apple fruit is obtained as ; Vector-based , surface center coordinates And the maximum transverse diameter, get the center position of the fruit:

[0131] , (26)

[0132] , (27)

[0133] in, represents a unit vector, Indicates the center of the fruit, D indicates the maximum transverse diameter, and R indicates the maximum radius. .

[0134] The second attitude angle calculation module constructs the calyx-fruit center vector based on the fruit center position, and decomposes the calyx-fruit center vector into Plane and plane, and then calculate the second attitude angle.

[0135] In the second posture angle calculation module, the process of calculating the second posture angle includes: obtaining the calyx coordinates , based on the surface center coordinates and fruit center position , calculate the calyx-fruit center vector 、 Normal vector of the plane and Normal vector of the plane :

[0136] , (28)

[0137] , (29)

[0138] , (30)

[0139] Based on calyx-fruit center vector , normal vector and normal vector , calculate the second attitude angle (α, β):

[0140] , (31)

[0141] . (32)

[0142] Example 3

[0143] In this embodiment, in order to verify the accuracy and robustness of the apple pose estimation algorithm in the actual orchard to ensure the continuous and non-destructive picking of the robot, an apple pose angle verification system is also provided. The system includes a self-developed apple pose projection angle measurement device and an apple pose estimation algorithm platform. By turning on the algorithm platform, the positions and spatial pose angles (α, β) of all apples within the camera's field of view can be viewed in real time. The projection angle measurement device is moved to measure the projection angle of the apple's minimum fruit axis. Finally, the projection angle conversion formula is used to calculate the spatial vector of the apple core-calyx point and the angle of the apple core-calyx point. Plane and The angle (α, β) between the two planes.

[0144] After completing the hand-eye calibration and determining the depth camera's position in the robot's coordinate system, the robot is no longer required to measure the apple's pose. Because the eye-outside-the-hand calibration method maintains the relative position of the camera and the robot arm's base, as the tester moves the camera around the orchard, the virtual base's coordinate system also moves with it. The measured pose angles of the apple are then converted to the virtual base's coordinate system for calculation.

[0145] The angles between the apple calyx-fruit center space vector and the two planes cannot be measured directly, so this embodiment provides a set of measurement devices, such as Figure 5As shown, it includes: a base, a measuring pointer, a level, a compass, and a fastening nut. The working process of the device is as follows: first obtain the projection angle between the vector and the two planes, and indirectly measure α and β. Use the compass and level to calibrate the coordinate system of the device and the camera so that the coordinate system of the device is relatively parallel to the coordinate system of the camera. Use two measuring pointers to measure the calyx-fruit center vector in the direction of the calyx-fruit center. Plane and The projection angles are θ1 and θ2 respectively.

[0146] By measuring the projection angle of the apple calyx-fruit center space vector on two orthogonal planes (such as the horizontal plane and the vertical plane), the actual angle between the vector and the plane is calculated using trigonometric function relationships, such as formulas (31)-(34), and it is translated to the origin of the fruit coordinate system for convenience of calculation.

[0147] , (33)

[0148] , (34)

[0149] , (35)

[0150] Among them, θ1 and θ2 are the projection angles measured by the device, l1, l2, and l3 are the three components of the vector in the coordinate system, α1 and β1 are the angles between the vector converted by indirect measurement of the projection angle and the plane, and their values ​​are also the required attitude angles α and β.

[0151] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for estimating the real-time spatial attitude angle of apples in an orchard environment, characterized in that: The following steps are involved: When the YOLOv8 model fails to recognize the calyx of the apple fruit, the segmentation mask of the apple fruit image is obtained and combined with the moment of inertia algorithm to estimate the maximum horizontal diameter and the first attitude angle of the apple that failed to recognize the calyx; When the YOLOv8 model recognizes the calyx of the apple fruit, the surface center coordinates of the apple fruit are obtained, and the fruit center position is calculated based on the maximum transverse diameter; A calyx-fruit center vector is constructed based on the fruit center position, and the calyx-fruit center vector is decomposed into Plane and plane, and then calculate the second attitude angle.

2. The method for estimating the real-time spatial posture angle of apples in an orchard environment according to claim 1, characterized in that: The method for obtaining the maximum transverse diameter includes: Obtain the segmentation mask of the apple fruit image and calculate the minimum and maximum moments of inertia of the apple using the moment of inertia algorithm. Obtaining a minimum moment of inertia axis based on the minimum moment of inertia, and obtaining a maximum moment of inertia axis based on the maximum moment of inertia; Calculating four pixel intersections of the minimum moment of inertia axis and the maximum moment of inertia axis with the edge of the segmentation mask image respectively; The four pixel intersections are converted into spatial coordinates in the robot base coordinate system, and the maximum axis length is determined by calculating the Euclidean distance to obtain the maximum transverse diameter.

3. The method for estimating the real-time spatial posture angle of apples in an orchard environment according to claim 2, characterized in that: The method for obtaining the first attitude angle includes: Calculating a roll angle θ based on the minimum moment of inertia; Let the pitch angle be 0 to obtain the first attitude angle (0, θ).

4. The method for estimating the real-time spatial posture angle of apples in an orchard environment according to claim 1, characterized in that: The method for calculating the position of the fruit center includes: Get the surface center coordinates of the apple fruit , set the robot base coordinate system to ; Then the vector from the base coordinate to the apple is , the vector from the base coordinate to the robot camera is , and then the vector from the robot camera to the surface center coordinate of the apple fruit is obtained as ; Vector-based , the surface center coordinates and the maximum transverse diameter, and obtain the position of the fruit center: ， ， in, represents a unit vector, Indicates the center of the fruit, D indicates the maximum transverse diameter, and R indicates the maximum radius. .

5. The method for estimating the real-time spatial posture angle of apples in an orchard environment according to claim 4, characterized in that: The method for calculating the second posture angle includes: Get the calyx coordinates , based on the surface center coordinates and the fruit center position , calculate the calyx-fruit center vector 、 Normal vector of the plane and Normal vector of the plane : ， ， ； Based on the calyx-fruit center vector , the normal vector and the normal vector , calculate the second posture angle (α, β): ， 。 6. A system for estimating the real-time spatial attitude angle of apples in an orchard environment, the system applying the method according to any one of claims 1 to 5, characterized in that: include: a first attitude angle calculation module, a fruit center position calculation module, and a second attitude angle calculation module; The first attitude angle calculation module is used to obtain a segmentation mask of the apple fruit image when the YOLOv8 model fails to recognize the calyx of the apple fruit, and estimate the maximum transverse diameter and the first attitude angle of the apple that failed to recognize the calyx in combination with the moment of inertia algorithm; The fruit center position calculation module is used to obtain the surface center coordinates of the apple fruit when the YOLOv8 model recognizes the calyx of the apple fruit, and calculate the fruit center position based on the maximum horizontal diameter; The second attitude angle calculation module constructs a calyx-fruit center vector based on the fruit center position, and decomposes the calyx-fruit center vector into Plane and plane, and then calculate the second attitude angle.

Images