A method for target recognition, segmentation, precise positioning and harvesting sequence planning of spherical fruits obscured under dense canopies

Abstract

The present invention discloses a method for target recognition and segmentation, precise positioning, and picking sequence planning for spherical fruits obscured under dense canopies. Through fruit-branch segmentation and contour extraction, the two-dimensional features of the fruit sphere and branches are obtained. Furthermore, the method accurately positions the spherical fruits under canopy obstruction and corrects the positioning errors of the picking points. The method also determines the spatial interference relationship between the target fruit and the branches, eliminating fruits that cannot be picked due to interference between the branches and fruits. Simultaneously, the collision relationship between multiple fruits is determined and the collision coefficient is calculated. This method guides a robotic arm to complete the picking operation in the order of picking independent fruit queues and picking collision fruit queues. The present invention effectively avoids the problems of fruit misidentification, mispositioning, and excessive positioning deviations that lead to picking failures. It also prevents the robotic arm from forcibly picking fruits that are improperly positioned or severely interfered with by branches and leaves, which can cause fruit damage or damage to the robotic arm, thereby ensuring efficient harvesting.

Description

Technical Field

[0001] The present invention relates to the field of agricultural robots, and in particular to a method for target recognition and segmentation, precise positioning, and harvesting sequence planning of spherical fruits obscured under dense canopies. Background Art

[0002] Amidst a labor shortage in agriculture, robotic harvesting as an alternative to traditional manual harvesting has become a research hotspot. Numerous harvesting robots, such as apple, grape, and kiwifruit harvesting robots, have already appeared and are being deployed. During the harvesting process, intelligent harvesting robots face a critical challenge: identifying and locating the target fruit. This not only depends on the robot's accuracy in identifying and locating the fruit, but also determines its efficiency.

[0003] In complex environments like orchards, such as semi-structured, unstructured, and dense canopy environments, fruit identification and positioning using deep learning and traditional machine vision technologies only capture the fruit's location without simultaneously acquiring information about surrounding obstacles. This makes it impossible to pinpoint the correct picking point for fruit that is damaged due to occlusion, and it also fails to address the issues of positioning point occlusion and fruit overlap and collision during orchard operations, which are major causes of picking failure. Furthermore, improper picking positions or forced picking of fruit that is severely obscured by branches, trellises, and other factors can damage the fruit and even the robotic arm. Without effective feasibility assessment and order planning, the system can be cumbersome and severely impact operational efficiency. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a target recognition and segmentation, precise positioning and picking sequence planning method for spherical fruits obscured under dense canopy conditions, aiming to solve the problems in the existing technology of difficult precise positioning of spherical fruits under canopy obscuration conditions, and multi-fruit-branch collision and interference judgment, and to achieve high-precision and high-efficiency robot spherical fruit picking operations.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] A method for target recognition, segmentation, precise positioning, and harvesting sequence planning of spherical fruits obscured under dense canopies:

[0007] The depth camera acquires an RGB image containing the target fruit and the branches of the fruit tree canopy. The RGB image is input into the trained target instance segmentation model, and target recognition and instance segmentation are performed on the spherical fruits and branches in the RGB image. The mask image of each fruit target and the mask image of each branch target are output in sequence. The edge point set of each fruit in the image coordinate system is extracted to form the fruit contour point set in sequence. The edge point set of each branch in the image coordinate system is extracted to form the branch contour point set in sequence.

[0008] The fruit contour point set is used to complete the two-dimensional plane of the fruit that is touched and blocked to obtain a completed two-dimensional plane area. The credibility of the completed two-dimensional plane area is calculated. According to a set threshold, the fruit targets that cannot be correctly restored are eliminated. Then, based on the depth information in the completed two-dimensional plane area, the depth point set of the spherical fruit is extracted and corrected, and the picking point is estimated to determine the three-dimensional coordinates of the correct picking point of the spherical fruit;

[0009] Determine the spatial relationship between the fruit to be picked and the branches, use the branch contour point set to perform two-dimensional completion of the branches of the fruit tree canopy, and restore a continuous and complete two-dimensional branch plane area. If there is an intersection with the completed two-dimensional plane area, then combine the effective depth value of the branch and the depth value of the three-dimensional sphere center picking point of the fruit to eliminate all target fruits with spatial interference between branches and fruits; next, if it is determined that the completed two-dimensional plane areas of multiple fruits have an intersection, then combine the depth value of the fruit picking point to determine whether there is a spatial contact relationship between the fruits. If so, calculate the contact coefficient of the fruit to complete the calculation of the contact fruit picking sequence score;

[0010] Prioritize picking independent fruit queues. When the independent fruit queue is finished, pick the adjacent fruit queues and complete the picking operation in sequence.

[0011] Furthermore, the two-dimensional plane completion of the fruit that is touched and blocked is performed, specifically:

[0012] For each spherical fruit, three convex hulls are randomly selected from the fruit contour point set, and the minimum circumscribed circle of the three convex hulls is determined. It is checked whether the minimum circumscribed circle contains all the points of the fruit contour point set. If not, the radius of the minimum circumscribed circle is increased, and three more convex hulls are continuously extracted. The recursive process is continued and repeated iterations are performed until the minimum circumscribed circle can contain all the points of the fruit contour point set.

[0013] The minimum circumscribed circle radius and center at the end of the iteration are used as the estimated radius and center of the two-dimensional area of ​​the fruit, respectively. The two-dimensional plane area formed is the completed two-dimensional plane area.

[0014] Furthermore, the credibility of the completed two-dimensional plane region satisfies:

[0015]

[0016] Among them, D i To complete the credibility of the two-dimensional plane area, |E i | represents the pixel area of ​​the surface enclosed by the contour points of this type of spherical fruit, r i Indicates the estimated radius of the two-dimensional area of ​​the fruit that is in contact with and obscured by the fruit.

[0017] Furthermore, the depth point set extraction and deviation correction of the spherical fruit are specifically as follows:

[0018] Get all the depth points in the completed two-dimensional plane area {D i}, the {D i} is divided into three clusters according to the size of the depth value: foreground obstacles, fruit body and fruit back background. Only the fruit body depth value set {C2} of the blocked fruit is retained, and the effective depth value d of {C2} must satisfy: μ i -2σ i ≤d≤μ i +2σ i , eliminating invalid depth values ​​that do not meet the above conditions, the effective depth point set of the blocked fruit after filtering and correction is {D i '}, where μ i is the mean value of {C2} depth, σ i Indicates the standard deviation of the depth value of the cluster where {C2} is located.

[0019] Furthermore, the picking points of the spherical fruits are speculated to be:

[0020] The effective depth point cloud of the occluded fruit {D i The points contained in '} are used for geometric sphere feature fitting and 3D prediction of the sphere center. The coordinates of the sphere center of the spherical fruit are O i (X i ,Y i ,Z i ) of the minimized error function (E i )satisfy: Respectively (X i ,Y i ,Z i ,R i ) calculate the partial derivative, set the partial derivative to zero, and fit the correct three-dimensional spherical center position of each fruit O i (X i ,Y i ,Z i ), which is the picking point; where (X, Y, Z) is any point inside or on the boundary of the spherical fruit, R i is the three-dimensional estimated radius of the sphere of this type of spherical fruit.

[0021] Furthermore, the two-dimensional completion of the fruit tree canopy branches is specifically as follows:

[0022] For each discrete branch contour point set {m j}, its center point c j (x j ,y j ) as the feature point of this point set, and search for the surrounding area with a radius of 2d j Is there a discrete point set of other branches within the range of {m k}, if it exists, it is considered that {m j} and {m k} belong to the same branch, calculate {m k The center point c k (x k ,y k ), where d j is the discrete branch contour point set {m j}'s maximum pixel diameter;

[0023] If the number of feature points is n, all feature points c0, c1, c2…c belonging to the same branch n The high-order polynomial fitting is performed, and the obtained polynomial is the two-dimensional completed branch curve of the fruit tree canopy. This curve constitutes a continuous and complete two-dimensional plane area of ​​the branch β = {(u b ,v b )}|b=1,2,...,n, where (u b ,v b ) is an arbitrary point on the branch plane.

[0024] Furthermore, the effective depth value of the branch and the depth value of the three-dimensional spherical center picking point of the fruit are combined to eliminate all target fruits with spatial interference between the branch and the fruit, specifically:

[0025] The following conditions are met if and only if at least one point in the two-dimensional plane area of ​​the branch is located in the two-dimensional plane area of ​​the fruit:

[0026] So that (u b -x) 2 +(v b -y) 2 <r i 2

[0027] It is considered that the branches and fruits overlap in the plane; among them, r i The estimated radius of the two-dimensional area of ​​the fruit that is in contact with and blocked by the fruit;

[0028] By comparing the depth value d of the picking point of the fruit's three-dimensional sphere centerf and the effective depth of the branches d b , determine the front-to-back spatial relationship between the fruit to be picked and the branches:

[0029] If d f -d b If ≤0, the fruit is considered to be in front of the branch and the branch has no spatial interference with the fruit;

[0030] If d f -d b > 0, the fruit is considered to be behind the branch. If |d f -d b |≤10mm, it is considered that the branches have spatial interference with the fruit, the fruit cannot be picked, and the fruit target is removed. On the contrary, if |d f -d b |>10mm, even if the fruit is located behind the branch, it is considered that the branch has no spatial interference with the fruit.

[0031] Furthermore, the determination of the contact relationship between multiple fruits is specifically as follows:

[0032] For the complementary two-dimensional plane region A of the fruit that is touching and obscured and the complementary two-dimensional plane region B of the fruit that is touching and obscured, if and only if the intersection of the two regions is not an empty set, then the two fruits are touching each other in the plane region. Continue to judge the touching relationship between the two fruits in space. Otherwise, the two fruits are not touching each other in space.

[0033] If the difference in the depth values ​​of the picking points of the two fruits is greater than the sum of the three-dimensional radii of the two fruits, it is considered that the two fruits do not collide in space; conversely, if the difference in the depth values ​​of the picking points of the two fruits is less than the sum of the three-dimensional radii of the two fruits, it is considered that the two fruits do collide in space.

[0034] Furthermore, the fruit's contact coefficient is:

[0035]

[0036] Among them, u m is the fruit contact coefficient, S A∩B is the overlapping area of ​​the two fruits’ complementary two-dimensional planes, S A∪B is the sum of the areas of the two fruits' complementary two-dimensional planes.

[0037] Further:

[0038] For picking of independent fruit queues, first score the picking sequence of each fruit in the independent fruit queue f n :

[0039]

[0040] Where ω1 is the confidence index weight of the fruit in the independent fruit queue, P n is the confidence index of the fruit in the independent fruit queue, is the canopy depth weight of the independent fruit queue, l n is the depth within the canopy at which fruits within individual fruit cohorts are located;

[0041] Then according to the ranking score f n Sort from high to low and pick all the independent fruit queues in turn;

[0042] For picking of fruit in the fruit queue, firstly score the picking order of each fruit in the fruit queue f. m :

[0043]

[0044] Where ω2 is the confidence index weight of the fruit in the fruit queue, P m is the confidence index of the fruit in the fruit queue, is the depth weight of the canopy where the fruit in the fruit-touching queue is located, l m is the depth of the fruit in the canopy where the fruit in the fruit-collision queue is located, σ is the collision coefficient weight, μ m is the collision coefficient;

[0045] Then according to the ranking score f m Sort from high to low and pick all the fruits in the queue one by one.

[0046] The beneficial effects of the present invention are:

[0047] (1) The present invention realizes the recognition, segmentation, precise positioning and sequence planning of spherical fruits obscured under dense canopies through the coordinated cooperation among the fruit-branch target recognition and contour extraction sub-method, the obscured spherical fruit completion and precise positioning sub-method, the multi-fruit-branch collision coefficient and spatial interference judgment and elimination sub-method, and the spherical fruit robot optimal operation sequence planning sub-method under canopy occlusion conditions, thereby effectively avoiding the problems of misidentification, mispositioning and excessive positioning deviation of fruits. The method is simple, reliable and highly practical.

[0048] (2) The present invention provides a method for target recognition, segmentation, precise positioning and picking sequence planning of spherical fruits obscured under dense canopies. Aiming at the complex dense canopy growth environment of unstructured orchards, the method obtains the two-dimensional features of the fruit sphere and branches through fruit-branch segmentation and contour extraction, and completes the precise positioning of spherical fruits under canopy obstruction conditions, the correction of picking point positioning errors, the judgment of the spatial interference relationship between the target fruit and the branches, and the elimination of fruits that cannot be picked due to interference between branches and fruits. At the same time, the collision relationship between multiple fruits is judged and the collision coefficient is calculated, guiding the robotic arm to plan a reasonable picking sequence and harvest the fruits one by one according to agronomic requirements, so as to avoid the robotic arm forcibly picking the target fruit in an inappropriate position or the fruit seriously interfered by branches and leaves, causing the fruit to be damaged and the robotic arm to be damaged, thereby ensuring efficient harvesting, realizing intelligent and mechanized harvesting by robots, improving the economic benefits of fruit farmers, and effectively saving labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of each sub-method of the present invention and corresponding functions;

[0050] Figure 2 It is a specific workflow diagram of the present invention;

[0051] Figure 3 Schematic diagram of the fruit-branch target recognition and contour extraction sub-method of the present invention;

[0052] Figure 4 The present invention is a two-dimensional plane completion flow chart for pasting and obscured fruits;

[0053] Figure 5 Schematic diagram of two-dimensional plane completion of the fruit being touched and blocked according to the present invention;

[0054] Figure 6 This is a flow chart of the two-dimensional completion method for fruit tree canopy branches of the present invention;

[0055] Figure 7 This is a schematic diagram of the two-dimensional completion of the branches and trunks of the fruit tree canopy according to the present invention;

[0056] Figure 8 This is a flow chart of eliminating branch-fruit interference in the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0058] The present invention discloses a target recognition, segmentation, precise positioning and order planning method for spherical fruits obscured under dense canopies, comprising a fruit-branch target recognition and contour extraction sub-method, a spherical fruit completion and precise positioning sub-method, a multi-fruit-branch collision coefficient and spatial interference judgment and elimination sub-method, and a spherical fruit robot optimal operation sequence planning sub-method under canopy occlusion conditions.

[0059] like Figure 1 As shown in the figure, the fruit-branch target recognition and contour extraction sub-method can perform target recognition and instance segmentation of spherical fruits and branches in RGB images, forming multi-fruit recognition and segmentation results (contour two-dimensional point set) and multi-branch recognition and segmentation results (contour two-dimensional point set); the occluded spherical fruit completion and precise positioning sub-method can realize the two-dimensional plane completion of the occluded fruit and calculate the three-dimensional sphere center position and radius of the occluded fruit; the multi-fruit-branch collision coefficient and spatial interference judgment and elimination sub-method can complete fruit-branch spatial interference elimination, fruit-fruit collision judgment and collision coefficient calculation; the optimal operation order sub-method of the spherical fruit robot under canopy occlusion conditions can obtain the operation order of fruits without collision and the operation order of fruits with collision.

[0060] like Figure 2 As shown, the specific implementation steps of the present invention are as follows:

[0061] (1) Using the fruit-branch target recognition and contour extraction sub-method, a depth camera is used to obtain an RGB image containing the target fruit and the branches of the fruit tree canopy, and a depth point set aligned with the pixel coordinate system of the RGB image. The RGB image is input into the trained Mask-RCNN-R50-FPN3x target instance segmentation model (stored in the robot's main controller), and the spherical fruits and branches in the RGB image are subjected to target recognition and instance segmentation. The mask image of each fruit target and the mask image of each branch target are output in turn, and each fruit target is labeled: fruit0, fruit1, ..., fruit i , label each branch target: branch0, branch1, ..., branch j For each obtained mask image of the fruit target, the Canny edge detection algorithm is used to extract the edge point set of each fruit in the image coordinate system, which in turn constitutes the fruit contour point set; for each obtained mask image of the branch target, the Canny edge detection algorithm is used to extract the edge point set of each branch in the image coordinate system, which in turn constitutes the branch contour point set;

[0062] (2) Using the sub-method of completing and accurately positioning the occluded spherical fruit, the fruit contour point set described in (1) is completed with the two-dimensional plane of the fruit that is stuck and occluded, and the completed two-dimensional plane area of ​​the occluded fruit is obtained. At the same time, the credibility of the completed two-dimensional plane area of ​​the fruit that is stuck and occluded is calculated. Then, according to the set credibility threshold of the completed two-dimensional plane area of ​​the fruit that is stuck and occluded, the fruit targets that cannot be correctly restored due to serious sticking and occlusion are eliminated. Then, based on the depth information in the completed two-dimensional plane area of ​​the fruit that is stuck and occluded, the depth point set of the spherical fruit is extracted and corrected, and the picking point is estimated, and finally the three-dimensional coordinates of the correct picking point of the spherical fruit are determined;

[0063] (3) The spatial relationship between the fruit to be picked and the branches is judged by using the multi-fruit-branch collision coefficient and the spatial interference judgment and elimination sub-method. The two-dimensional point set of the branch outline is used to complete the two-dimensional branches of the fruit tree canopy, and a continuous and complete two-dimensional plane area of ​​the branches is restored. If there is an intersection with the two-dimensional plane area of ​​the fruit, then the effective depth value of the branch and the depth value of the three-dimensional spherical center of the fruit are combined to eliminate all target fruits with spatial interference between the branches and fruits; next, the collision relationship between multiple fruits is judged and the collision coefficient is calculated. If there is an intersection between multiple fruits in the two-dimensional plane area, then the specific picking point depth values ​​are combined to judge whether there is a collision relationship between the fruits in space. If so, their collision coefficients are calculated to complete the calculation of the collision fruit picking sequence score;

[0064] (4) Using the optimal operation sequence sub-method of the spherical fruit robot under the condition of canopy occlusion, the independent fruit queue is picked first. When the independent fruit queue is completed, the adjacent fruit queue is picked to complete the picking operation in sequence. During the picking operation, the principle of picking from shallow canopy to deep canopy and giving priority to prominent fruits is followed.

[0065] like Figure 3 As shown in the figure, the fruit-branch target recognition and contour extraction sub-method is specifically as follows: by pre-collecting spherical fruit and branch images, the fruits and branches are respectively labeled to construct a spherical fruit-branch detection image dataset; the preset Mask-RCNN-R50-FPN3x model is used to train the labeled image dataset; and a fully trained Mask-RCNN-R50-FPN3x target instance segmentation model is obtained;

[0066] During the operation, the RGB image containing the target fruit and the branches of the fruit tree canopy is input into the well-trained Mask-RCNN-R50-FPN3x target instance segmentation model, which outputs the mask image of each fruit target and the mask image of each branch target in turn, and uses the Canny edge detection algorithm to extract the edge point set of each fruit in the image coordinate system, which in turn constitutes the fruit contour point set {k1}{k2}{k3}...{ki}...{k n}; For each branch target mask image obtained, the Canny edge detection algorithm is used to extract the edge point set of each branch in the image coordinate system, and the branch contour point set {m1}{m2}{m3}...{m j}...{m t}.

[0067] like Figure 4 、 Figure 5 As shown, the two-dimensional plane completion of the touched and obscured fruit is specifically as follows: for each spherical fruit, three convex hulls (outer edge points) are randomly selected from the fruit contour point set, and the minimum circumscribed circle of the three convex hulls is determined. It is checked whether the minimum circumscribed circle contains all the points of the fruit contour point set. If not, the radius of the minimum circumscribed circle is increased, and three more convex hulls (outer edge points) are continuously extracted. The recursive process is continued and iterated repeatedly until the minimum circumscribed circle can contain all the points of the fruit contour point set.

[0068] The constraints are satisfied:

[0069]

[0070] In the formula, |p i -C i (x,y)| represents point p i To the center C of the smallest circumcircle i The Euclidean distance of (x,y), Represents the fruit contour point set {k i}, i=all points p contained in (1,2,3...,n) i This condition is met, r i Indicates the minimum circumscribed circle radius at the end of the iteration;

[0071] Here the radius r i 、Center of circle C i (x, y) is used as the estimated radius and center of the two-dimensional area of ​​the fruit, and the two-dimensional plane area formed is the completed two-dimensional plane area of ​​the fruit that is touched and blocked.

[0072] In this embodiment, the credibility of the completed two-dimensional plane area of ​​the fruit that is stuck and blocked is calculated, and then the fruit targets that cannot be correctly restored due to severe sticking or blocking are eliminated according to the set credibility threshold of the completed two-dimensional plane area of ​​the fruit that is stuck and blocked. Specifically,

[0073] For i spherical fruits, the credibility D of the completed two-dimensional plane area of ​​their touching and occluded fruits is i for:

[0074]

[0075] Among them, |E i | represents the contour point set of this type of spherical fruit {k i}, i = the pixel area of ​​the curved surface enclosed by (1, 2, 3..., n);

[0076] The greater the difference between the pixel area of ​​the surface enclosed by the fruit contour point set and the area of ​​the completed two-dimensional plane of the fruit that is touched and occluded, the higher the credibility D of the completed two-dimensional plane of the fruit that is touched and occluded. i The lower the value;

[0077] When the credibility is lower than the set threshold D i ≤ξ, where ξ=60%, it is considered that the fruit is severely occluded and the credibility of the completed result is insufficient. The fruit targets that cannot be correctly restored due to severe contact and occlusion are eliminated.

[0078] Furthermore, the depth point set extraction and deviation correction of the spherical fruit are specifically as follows:

[0079] By extracting all the depth point sets in the completed two-dimensional plane area of ​​the fruit that is stuck and blocked, the depth point set of the blocked fruit is expressed as:

[0080]

[0081] in, Depth point set of the occluded fruit {D i} corresponds to the depth value of the position (u1, v1), (x, y) represents the center of the circle C i The horizontal and vertical coordinates of (x,y), r i The estimated radius of the two-dimensional area of ​​the fruit that is in contact with and blocked by the fruit;

[0082] The depth point cloud of the occluded fruit is clustered using a clustering algorithm {D i} are divided into 3 clusters, which correspond to the foreground obstacles, fruit body, and fruit back background from small to large according to the depth value. The depth value sets of each cluster are {C1}, {C2}, and {C3} respectively;

[0083] Only the fruit body depth value set {C2} of the obscured fruit is retained, and individual invalid depth values ​​in the depth point set caused by noise interference during the acquisition process are eliminated;

[0084] Calculate the mean depth value μ of the fruit body depth value set {C2} of the occluded fruit i :

[0085]

[0086] Calculate the standard deviation of the depth values ​​of the cluster where the fruit body depth value set {C2} of the occluded fruit belongs:

[0087]

[0088] in, Represents the depth value of the corresponding position (u2, v2) in the fruit body depth value set {C2} of the obscured fruit, The number of points in the fruit body depth value set {C2} representing the obscured fruit;

[0089] The effective depth value d of the fruit body depth value set {C2} of the obscured fruit must satisfy:

[0090] μ i -2σ i ≤d≤μ i +2σ i

[0091] Eliminating invalid depth values ​​that do not meet the above conditions, the effective depth point set of the occluded fruit after filtering and correction is {D i '}.

[0092] In this embodiment, the picking points of the spherical fruits are estimated to be:

[0093] The effective depth point cloud of the occluded fruit {D i Point P contained in '} (X,Y,Z) Perform geometric sphere feature fitting and 3D inference of the sphere center, and the coordinates of the sphere center of the spherical fruit are O i (X i ,Y i ,Z i ) of the minimized error function (E i )satisfy:

[0094]

[0095] Where (X, Y, Z) is any point inside or on the boundary of the spherical fruit, R i is the three-dimensional inferred radius of the sphere of such spherical fruit;

[0096] Respectively (X i ,Y i ,Z i ,R i ) to find the partial derivatives:

[0097]

[0098] And let the partial derivative be zero, then the correct three-dimensional spherical center position O of each fruit can be fitted. i (X i,Y i ,Z i ), as the three-dimensional sphere center, i.e., the picking point.

[0099] like Figure 6 、 Figure 7 As shown, in this embodiment, the two-dimensional completion of the fruit tree canopy branches is specifically as follows:

[0100] t discrete branch contour point sets {m1}{m2}{m3}...{m j}...{m t} for splicing and fusion;

[0101] For each discrete branch contour point set {m j}, its center point c j (x j ,y j ) as the feature points of this point set:

[0102]

[0103] Among them, (x i ,y i ) is {m j}, n j is {m j The number of embedded points in};

[0104] Then calculate the discrete branch contour point set {m j The maximum pixel diameter d j :

[0105]

[0106] Among them, (x p ,y p ) and (x q ,y q ) is {m j Any two points in};

[0107] For each discrete branch contour point set {m j}, search the area around it with a radius of 2d j Is there a discrete point set of other branches within the range of {m k}, if it exists, it is considered that {m j} and {m k} belong to the same branch, calculate {m k The center point c k (x k ,y k );

[0108] If the number of feature points is n, all feature points c0, c1, c2…c belonging to the same branch n Perform high-order polynomial fitting to complete the two-dimensional information of branches:

[0109]

[0110] This polynomial is the two-dimensional completed branch curve of the fruit tree canopy. This curve constitutes a continuous and complete two-dimensional plane area of ​​branches β = {(u b ,v b )}|b=1,2,...,n, where (u b ,v b ) is an arbitrary point on the branch plane.

[0111] like Figure 8 As shown, in this embodiment, the depth point set around the overlapping area of ​​branches and fruits is combined to eliminate all target fruits with spatial interference between branches and fruits, specifically:

[0112] The following conditions are met if and only if at least one point in the two-dimensional plane area of ​​the branch is located in the two-dimensional plane area of ​​the fruit:

[0113] So that (u b -x) 2 +(v b -y) 2 <r i 2

[0114] It is considered that the branches and fruits overlap in the plane;

[0115] The two-dimensional curve of the fruit tree trunk β={(u b ,v b )}|b=1,2,...,n, in the depth point set acquired by the depth camera, extract the corresponding fruit tree branch depth point set {D b};

[0116] Among them, (u b ,v b ) is any point on the two-dimensional curve of the fruit tree branch completed in two dimensions, and in the corresponding fruit tree branch depth point set {D b}, the depth value of this point is

[0117] The average depth of the fruit tree branches is calculated as the effective depth of the branches d. b :

[0118]

[0119] Among them, n b To form the depth point set of the fruit tree branches {D b}Number of midpoints;

[0120] The depth value d of the fruit's three-dimensional sphere center picking point is recorded as f , determine the front-to-back spatial relationship between the fruit to be picked and the branches:

[0121] If d f -d b If ≤0, the fruit is considered to be in front of the branch and the branch has no spatial interference with the fruit;

[0122] If d f -d b > 0, the fruit is considered to be behind the branch. If |d f -d b |≤10mm, the branches are considered to have spatial interference with the fruit, and the fruit is considered unpickable and removed. On the contrary, if |d f -d b |>10mm, even if the fruit is located behind the branch, it is considered that the branch has no spatial interference with the fruit.

[0123] In this embodiment, the determination of the contact relationship between multiple fruits and the calculation of the contact coefficient are specifically as follows:

[0124] For a complementary two-dimensional plane region A of a fruit that is touching and obscured, and another complementary two-dimensional plane region B of a fruit that is touching and obscured, if and only if the intersection of the two regions is not an empty set, that is:

[0125]

[0126] Then the two fruits are touching in the plane area, and we continue to judge the touching relationship between the two fruits in space. Otherwise, the two fruits are not touching in space.

[0127] The depth of the picking point of the two fruits is Z A 、Z B The three-dimensional radii of the two fruits are R A 、R B ;

[0128] If the difference between the two depth values ​​is greater than the sum of the three-dimensional radii of the two fruits:

[0129] |Z A -Z B |≥R A +R B

[0130] It is believed that there is no spatial contact between the two fruits;

[0131] On the contrary, if the difference between the two depth values ​​is less than the sum of the three-dimensional radii of the two fruits:

[0132] |Z A -Z B |<R A +R B

[0133] It is believed that the two fruits are touching in space;

[0134] Their collision coefficient is u m :

[0135]

[0136] Among them, S A∩B is the overlapping area of ​​the two fruits’ complementary two-dimensional planes, S A∪B is the sum of the areas of the two fruits' complementary two-dimensional planes.

[0137] In this embodiment, the optimal operation sequence sub-method of the spherical fruit robot under the canopy occlusion condition is specifically as follows:

[0138] For the spherical fruit targets, they are divided into independent fruits and collision fruits according to whether there is collision between the fruits. The queues are F n With F m , prioritize picking independent fruit queue F n When the independent fruit queue is finished, the fruit queue F is picked and touched. m ;

[0139] Independent Fruit Queue F n The sorting score of each fruit in n :

[0140]

[0141] Where ω1 is the confidence index weight of the fruit in the independent fruit queue, P n is the confidence index of the fruit, is the canopy depth weight of the independent fruit queue, l n is the depth at which the fruit is located within the canopy;

[0142] According to the ranking score f n Sort from high to low, that is, the next picking target for the current nth fruit is labeled F n+1 =max(f n ), pick up all the independent fruit queues in turn;

[0143] Fruit Queue F m The sorting score of each fruit in m :

[0144]

[0145] Where ω2 is the confidence index weight of the fruit placed in the fruit queue, P m is the confidence index of the fruit, is the canopy depth weight of the fruit-touching queue, l m is the depth of the fruit in the canopy, σ is the weight of the collision coefficient, μ m is the collision coefficient;

[0146] According to its ranking score f m Sort from high to low, that is, the next picking target for the current mth fruit is labeled F m+1 =max(f m ), pick all the fruits in the queue one by one.

[0147] The embodiments described are preferred implementations of the present invention, but the present invention is not limited to the above implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention are within the scope of protection of the present invention.

Claims

1. A method for target recognition, segmentation, precise positioning, and harvesting planning of spherical fruits obscured under dense canopies, characterized by: The depth camera acquires an RGB image containing the target fruit and the branches of the fruit tree canopy. The RGB image is input into the trained target instance segmentation model, and target recognition and instance segmentation are performed on the spherical fruits and branches in the RGB image. The mask image of each fruit target and the mask image of each branch target are output in sequence. The edge point set of each fruit in the image coordinate system is extracted to form the fruit contour point set in sequence. The edge point set of each branch in the image coordinate system is extracted to form the branch contour point set in sequence. The fruit contour point set is used to complete the two-dimensional plane of the fruit that is touched and blocked to obtain a completed two-dimensional plane area. The credibility of the completed two-dimensional plane area is calculated. According to a set threshold, the fruit targets that cannot be correctly restored are eliminated. Then, based on the depth information in the completed two-dimensional plane area, the depth point set of the spherical fruit is extracted and corrected, and the picking point is estimated to determine the three-dimensional coordinates of the correct picking point of the spherical fruit; Determine the spatial relationship between the fruit to be picked and the branches, use the branch contour point set to perform two-dimensional completion of the branches of the fruit tree canopy, and restore a continuous and complete two-dimensional branch plane area. If there is an intersection with the completed two-dimensional plane area, then combine the effective depth value of the branch and the depth value of the three-dimensional sphere center picking point of the fruit to eliminate all target fruits with spatial interference between branches and fruits; next, if it is determined that the completed two-dimensional plane areas of multiple fruits have an intersection, then combine the depth value of the fruit picking point to determine whether there is a spatial contact relationship between the fruits. If so, calculate the contact coefficient of the fruit to complete the calculation of the contact fruit picking sequence score; Prioritize picking independent fruit queues. When independent fruit queues are finished, pick adjacent fruit queues and complete the picking operation in sequence. The method combines the effective depth value of the branches and the depth value of the three-dimensional spherical center picking point of the fruit to eliminate all target fruits with spatial interference between the branches and fruits, specifically: The following conditions are met if and only if at least one point in the two-dimensional plane area of ​​the branch is located in the two-dimensional plane area of ​​the fruit: So that (u b -x) 2 +(v b -y) 2 <r i 2 It is considered that the branches and fruits overlap in the plane; among them, r i The estimated radius of the two-dimensional area of ​​the fruit that is in contact with and blocked by the fruit; By comparing the depth value d of the picking point of the fruit's three-dimensional sphere center f and the effective depth of the branches d b , determine the front-to-back spatial relationship between the fruit to be picked and the branches: If d f -d b If ≤0, the fruit is considered to be in front of the branch and the branch has no spatial interference with the fruit; If d f -d b > 0, the fruit is considered to be behind the branch. If |d f -d b |≤10mm, it is considered that the branches have spatial interference with the fruit, the fruit cannot be picked, and the fruit target is removed. On the contrary, if |d f -d b |>10mm, even if the fruit is located behind the branch, it is considered that the branch has no spatial interference with the fruit; Among them, (u b ,v b ) is an arbitrary point on the branch plane, and β is the two-dimensional plane area of ​​the branch.

2. The target recognition, segmentation, precise positioning and sequence planning method according to claim 1, characterized in that: Perform two-dimensional plane completion of the fruit that is touched and blocked, specifically: For each spherical fruit, three convex hulls are randomly selected from the fruit contour point set, and the minimum circumscribed circle of the three convex hulls is determined. It is checked whether the minimum circumscribed circle contains all the points of the fruit contour point set. If not, the radius of the minimum circumscribed circle is increased, and three more convex hulls are continuously extracted. The recursive process is continued and repeated iterations are performed until the minimum circumscribed circle can contain all the points of the fruit contour point set. The minimum circumscribed circle radius and center at the end of the iteration are used as the estimated radius and center of the two-dimensional area of ​​the fruit, respectively. The two-dimensional plane area formed is the completed two-dimensional plane area.

3. The target recognition, segmentation, precise positioning and sequence planning method according to claim 2, characterized in that: The credibility of the completed two-dimensional plane region satisfies: Among them, D i To complete the credibility of the two-dimensional plane area, |E i | represents the pixel area of ​​the surface enclosed by the contour points of this type of spherical fruit, r i Indicates the estimated radius of the two-dimensional area of ​​the fruit that is in contact with and obscured by the fruit.

4. The target recognition, segmentation, precise positioning and sampling planning method according to claim 3, characterized in that: The depth point set extraction and deviation correction of the spherical fruit are specifically as follows: Get all the depth points in the completed two-dimensional plane area {D i }, the {D i } is divided into three clusters according to the size of the depth value: foreground obstacles, fruit body and fruit back background. Only the fruit body depth value set {C2} of the blocked fruit is retained, and the effective depth value d of {C2} must satisfy: μ i -2σ i ≤d≤μ i +2σ i , eliminating invalid depth values ​​that do not meet the above conditions, the effective depth point set of the blocked fruit after filtering and correction is {D i '}, where μ i is the mean value of {C2} depth, σ i Indicates the standard deviation of the depth value of the cluster where {C2} is located.

5. The target recognition, segmentation, precise positioning and sequence planning method according to claim 4, characterized in that: The picking points of the spherical fruit are speculated to be: The effective depth point cloud of the occluded fruit {D i The points contained in '} are used for geometric sphere feature fitting and 3D prediction of the sphere center. The coordinates of the sphere center of the spherical fruit are O i (X i ,Y i ,Z i ) to minimize the error function E i satisfy: Respectively (X i ,Y i ,Z i ,R i ) calculate the partial derivative, set the partial derivative to zero, and fit the correct three-dimensional spherical center position of each fruit O i (X i ,Y i ,Z i ), which is the picking point; where (X, Y, Z) is any point inside or on the boundary of the spherical fruit, R i is the three-dimensional estimated radius of the sphere of this type of spherical fruit.

6. The target recognition, segmentation, precise positioning and sampling sequence planning method according to claim 1, characterized in that: The two-dimensional completion of the branches and trunks of the fruit tree canopy is specifically as follows: For each discrete branch contour point set {m j }, its center point c j (x j ,y j ) as the feature point of this point set, and search for the surrounding area with a radius of 2d j Is there a discrete point set of other branches within the range of {m k }, if it exists, it is considered that {m j } and {m k } belong to the same branch, calculate {m k The center point c k (x k ,y k ), where d j is the discrete branch contour point set {m j }'s maximum pixel diameter; If the number of feature points is n, all feature points c0, c1, c2…c belonging to the same branch n The high-order polynomial fitting is performed, and the obtained polynomial is the two-dimensional completed branch curve of the fruit tree canopy. This curve constitutes a continuous and complete two-dimensional plane area of ​​the branch β = {(u b ,v b )}|b=1,2,...,n, where (u b ,v b ) is an arbitrary point on the branch plane.

7. The target recognition, segmentation, precise positioning and sequence planning method according to claim 1, characterized in that: The determination of the contact relationship between multiple fruits is specifically as follows: For the complementary two-dimensional plane region A of the fruit that is touching and obscured and the complementary two-dimensional plane region B of the fruit that is touching and obscured, if and only if the intersection of the two regions is not an empty set, then the two fruits are touching each other in the plane region. Continue to judge the touching relationship between the two fruits in space. Otherwise, the two fruits are not touching each other in space. If the difference in the depth values ​​of the picking points of the two fruits is greater than the sum of the three-dimensional radii of the two fruits, it is considered that the two fruits do not collide in space; conversely, if the difference in the depth values ​​of the picking points of the two fruits is less than the sum of the three-dimensional radii of the two fruits, it is considered that the two fruits do collide in space.

8. The target recognition, segmentation, precise positioning and sampling sequence planning method according to claim 7, characterized in that: The sticking coefficient of the fruit is: Among them, u m is the fruit contact coefficient, S A∩B is the overlapping area of ​​the two fruits’ complementary two-dimensional planes, S A∪B is the sum of the areas of the two fruits' complementary two-dimensional planes.

9. The target recognition, segmentation, precise positioning and sampling sequence planning method according to claim 8, characterized in that: For picking of independent fruit queues, first score the picking sequence of each fruit in the independent fruit queue f n : Where ω1 is the confidence index weight of the fruit in the independent fruit queue, P n is the confidence index of the fruit in the independent fruit queue, is the canopy depth weight of the independent fruit queue, l n is the depth within the canopy at which fruits within individual fruit cohorts are located; Then according to the ranking score f n Sort from high to low and pick all the independent fruit queues in turn; For picking of fruit in the fruit queue, firstly score the picking order of each fruit in the fruit queue f. m : Where ω2 is the confidence index weight of the fruit in the fruit queue, P m is the confidence index of the fruit in the fruit queue, is the depth weight of the canopy where the fruit in the fruit-touching queue is located, l m is the depth of the fruit in the canopy where the fruit in the fruit-collision queue is located, σ is the collision coefficient weight, μ m is the collision coefficient; Then according to the ranking score f m Sort from high to low and pick all the fruits in the queue one by one.

Images