An intelligent and precise fruit picking method based on visual perception

Abstract

The present invention provides a method for intelligent and precise fruit picking based on visual perception, which relates to the field of fruit picking and adopts a picking robot comprising a mobile chassis (1), a walking mechanism (2), an industrial computer (3), a collection frame (4), a picking mechanical arm (5), an end effector (6), a monocular vision camera (7), a binocular stereo camera (8) and an air blowing component (9); the picking robot obtains the relative motion state between the picking point and the blocked fruit after the blowing through recognition by the binocular stereo camera (8), blowing away the fruit by the air blowing component (9) and obtaining the relative motion state between the picking point and the blocked fruit after the blowing away through a timestamp, thereby realizing perception of the picking point in motion and precise clipping when the fruit is blocked and presents complex motion after the blocking is removed, thereby improving the picking efficiency, reducing the probability of equipment damage and achieving control of the harvesting cost.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit picking, and in particular to an intelligent and precise fruit picking method based on visual perception. Background Art

[0002] Bunch fruits refer to a type of fruit that grows in bunches, that is, a bunch of fruits are grown on one mother branch. Currently, common bunch fruits include grapes, lychees, longans, black currants, etc.; due to the clustered structure of bunch fruits, they have the characteristics of large volume and weight and small fruit stalks, therefore, picking bunch fruits is more cumbersome, and problems such as damage to fruits or other branches may easily occur during the picking process.

[0003] The complex operating environment for bunch fruit picking severely restricts the development of automated picking in this area. In some regions, bunch fruit is still picked manually using semi-automated picking positions, typically using handheld, semi-automated tools. This method not only reduces picking efficiency but also increases labor costs, reducing the profitability of bunch fruit cultivation. Currently, with the increasing development of intelligent and digital machinery, some automated picking robots are also being used for bunch fruit picking. However, most current bunch fruit picking robots are only capable of unobstructed fruit picking. When fruit is obstructed by adjacent clusters of fruit, branches, or leaves (due to factors such as wind and gravity, which can easily lead to mutual obstruction between fruit, branches, and leaves), the picking robots may lose their target, be unable to avoid obstacles, or even fail to harvest. This can affect the overall picking process, reduce picking efficiency, and even lead to damage to the fruit and branches, impacting the overall profitability of the fruit trees. Therefore, in order to achieve precise picking during the picking process, it is necessary to blow away the obstructions of the target fruit. After the obstructions are blown away, they are affected by the combined effects of gravity, wind resistance and blowing force, and are prone to uncontrollable complex movements, thereby obstructing the target fruit twice or even multiple times. The target fruit is also easily offset by the blowing force, that is, the picking point is prone to offset, further increasing the difficulty of picking the target fruit, and at the same time increasing the probability of damage to the fruit and the picking robot, increasing the picking time and increasing the picking cost. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide an intelligent and precise fruit picking method based on visual perception. This picking method can realize the perception of picking points in motion and precise clipping when the fruit is obscured and exhibits complex movements after being removed, thereby improving the efficiency of picking, reducing the probability of equipment damage, and controlling the harvesting cost.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for intelligent and precise fruit picking based on visual perception uses a picking robot comprising a mobile chassis, a walking mechanism, an industrial computer, a collection frame, a picking robot arm, an end effector, a monocular vision camera, a binocular stereo camera, and an air blowing component. The picking steps are as follows:

[0007] Step A: The picking robot moves to the picking area, and the binocular stereo camera captures the location of the ripe bunch fruit, which is recorded as the target fruit. The picking robot arm controls the end effector to move toward the target fruit and separates the target fruit from the obstructed fruit using the air blowing component. The binocular stereo camera captures the dynamic separation degree of the target fruit and the obstructed fruit in real time.

[0008] During this process, the monocular vision camera records a large field of view image including the target fruit picking point, the end effector, and the picking robot arm in real time;

[0009] Step B: Obtain the relative motion relationship between the picking point and the obstructed fruit, use the size of the end effector as a threshold, find the moment when the dynamic separation is maximized, and obtain the optimal picking point based on this moment;

[0010] Step C, when the picking point of the target fruit falls into the optimal picking point, pinching and shearing are performed;

[0011] Step D: After the picking is completed, the picking robot resets and proceeds to the next picking.

[0012] Based on further optimization of the above scheme, the walking mechanism is arranged at the bottom of the mobile chassis and is used for the motion walking control of the entire picking robot; the industrial computer, collection frame, picking robot arm and monocular vision camera are respectively arranged on the mobile chassis; the end effector is arranged at the end of the picking robot arm away from the mobile chassis, and the binocular stereo camera is arranged at the end of the picking robot arm away from the mobile chassis and is located on the upper side of the end effector; the blowing assembly is arranged on the end face of the mobile chassis through a lifting bracket and the blowing assembly includes a steering motor, an air compressor and a nozzle; the drivers of the walking mechanism, picking robot arm, end effector, monocular vision camera, binocular stereo camera, blowing assembly and lifting bracket are respectively electrically connected to the industrial computer.

[0013] Based on the further optimization of the above scheme, the specific method of step A is:

[0014] Step A1: After the binocular stereo camera identifies the partially obscured target fruit using the Yolov5s algorithm, a frame of image is taken for positioning and identification; the obscured target fruit is then color segmented using the HSV color gamut value to separate the image containing the ripe fruit identification frame and the obscured fruit identification frame; then, the overlap of the ripe fruit identification frame and the obscured fruit identification frame is determined: if the overlap condition is met, the obscured fruit identification frame in front of the ripe fruit identification frame is considered to be a subordinate fruit of the corresponding target fruit; if the overlap condition is not met, the fruit is not successfully paired and the corresponding fruit image is removed;

[0015] Step A2: Using an air blowing component to blow away the obstructing fruit, thereby separating the target fruit from the obstructing fruit, and simultaneously performing color segmentation on the identified target fruit cluster using the HSV color gamut value to separate the image containing the fruit identification frame and the branch identification frame; then, determining the degree of overlap between the separated fruit identification frame and the branch identification frame to obtain the target fruit and its corresponding branch;

[0016] Step A3: Divide the single-valued connected domain of the branch corresponding to the target fruit into ten equal parts in the vertical direction, determine the width of the connected domain in the horizontal direction of each dividing line, and extract the branch image of the adjacent branches in the horizontal direction that have completed the fruit relationship. Compare the width differences of the single-valued connected domains to obtain the picking point corresponding to the target fruit;

[0017] Step A4: Obtain the target fruit picking points using a binocular stereo camera μ(x 1 ,y 1 ,z 1 ） With the center of the fruit obscured v(x 2 ,y 2 ,z 2 ） The coordinates in the world coordinate system are synchronized using the timestamp method to match the spatial coordinates of the two motion trajectories of the picking point and the center of the occluded fruit at the same time, and the distance between the two at the same time is obtained. D :

[0018]

[0019] And the picking point during the blowing process is predicted in real time through Wasserstein distance μ With the center of the fruit obscured v Distance between:

[0020]

[0021]

[0022] Where, expressR x R The upper edge distributions are μ and v The set of probability measures of ; 、 Don't say μ and v The corresponding cumulative distribution function is:

[0023]

[0024] Where: t represents the probability threshold of the cumulative distribution function;

[0025] The distance between the obstructed fruit and the picking point is obtained and tracked and recorded in real time, thereby obtaining the dynamic separation between the target fruit and the obstructed fruit in real time.

[0026] Based on further optimization of the above solution, the specific method of segmenting by HSV color gamut value in step A1 and step A2 is:

[0027] First, the pre-trained Yolov5s is used to identify the target fruit image and capture the image within the recognition frame. Then, for the captured image, the image is segmented using the statistical H (hue), S (saturation), and V (brightness) value ranges. The segmented image is binarized and convex hull filled to output the minimum rectangular area of the connected domain. Finally, the camera pixel coordinates of the target fruit, the obstructing fruit or branch, the center point of the recognition frame, and the length and width of the recognition frame are obtained through the obtained minimum rectangular area.

[0028] Based on the further optimization of the above scheme, the specific method of comparing the width differences of the single-valued connected domains in step A3 to obtain the picking points corresponding to the target fruits is as follows: first, the width values in the adjacent horizontal directions are subtracted from each other to obtain the width difference. △X i ( i=0,1,2,…,9 ), compare the width differences with each other: △X i The value at the beginning of the decrease is used as the starting point for judgment. When three consecutive width differences (such as: ) is greater than three pixels, it is determined △X i The corresponding connected domain is the picking point; if no point that meets the standard deviation condition is found, the final △X The connected domain corresponding to 9 is the picking point.

[0029] Based on the further optimization of the above scheme, the specific method of step B is:

[0030] Step B1: Under the influence of wind, the obstructed fruit will shift, causing the picking point to shift downward. To accurately clip the picking point, the motion trajectories of the two are recorded in real time and analyzed and predicted. During the actual picking process, the obstructed fruit first shifts significantly, then moves back and forth very slightly, while the picking point shifts slightly. A binocular stereo camera is used to obtain the world coordinates of the picking point and the obstructed fruit. The image is captured in real time to record the motion trajectories of the two, and the motion direction and motion vector of the two are obtained:

[0031]

[0032] Where: Indicates the motion trajectory of the occluded fruit; Represents the motion trajectory of the picking point; Represents the relative motion relationship between the obstructed fruit and the picking point; i, j, k Represent the motion components of the picking point and the obstructed fruit on the x-axis, y-axis, and z-axis respectively;

[0033] Step B2: The probability distribution input in random computer experiments usually has finite support and a certain degree of continuity. Therefore, the size of the end effector is used as the threshold and the Wasserstein distance is assigned to the space:

[0034]

[0035] Where: is a constant and ;

[0036] After the Wasserstein distance is assigned to this space, it can be similar to the distance-based space filling design in Euclidean space. The corresponding space filling criterion is defined. For any support distribution function F on [0,1], a natural number m (the value of m is relatively large) is taken and piecewise linear function approximation is adopted:

[0037]

[0038] Where: I is the characteristic function;

[0039] therefore, The function can be approximated by (m-1) dimensional space, namely:

[0040]

[0041] Where: a n represents the distribution function value of the 𝑛th segment;

[0042] Right now The maximum Wasserstein distance is:

[0043]

[0044] The maximum distance between the blocked fruit and the picking point is obtained to prepare for the robot arm motion trajectory planning. When the distance between the two reaches the maximum value, the picking point is predicted.

[0045] Based on the further optimization of the above scheme, the specific method of step C is:

[0046] Step C1: When the distance between the obstructed fruit and the picking point in step B reaches the peak value, the monocular vision camera records the motion vector between the picking point and the obstructed fruit, and controls the picking robot arm and uses the DH ( Denavit-Hartenberg ) Matrix transformation algorithm plans the motion trajectory of the end effector;

[0047] Step C2: The trajectory planning of the picking robot arm to the predicted picking point is performed using the fifth-order polynomial interpolation method. The end effector is adjusted from the initial position to the predicted picking point position by adjusting the function of each joint angle changing with time:

[0048]

[0049] in, T Time 0 is the initial pose, T f The moment is the set target posture, and T The turning angle at time 0 is 、 T f The turning point of time is ,but:

[0050]

[0051] at the same time:

[0052]

[0053] The end effector of the picking robot moves from the initial position to the picking point, and the end effector clamps and shears the picking point to complete the picking.

[0054] Based on the further optimization of the above solution, the specific method of planning the motion trajectory of the end effector by using the DH matrix transformation algorithm in the robot kinematics principle in step C1 is as follows:

[0055] The picking robot arm consists of four joints, and the motion of each joint is described by an independent DH parameter, including the joint angle , joint offset d i, connecting rod length u i Torsion angle of connecting rod :

[0056]

[0057] Calculate the transformation matrix of each joint in turn and multiply them to get the overall transformation matrix of the end effector:

[0058]

[0059] Where: T i Indicates the i The transformation matrix of each joint, i=1,2,3,4;

[0060] Transformation matrix of the end effector relative to the base coordinate system T 0 is:

[0061]

[0062] Set the transformation matrix of the end effector T target , solve the corresponding joint angles through inverse kinematics .

[0063] The following are the technical effects of the solution of the present invention:

[0064] The present application uses a binocular stereo camera in conjunction with a monocular vision camera, and utilizes a binocular stereo camera provided at the end of a picking robot arm to realize the recognition of obscured ripe bunch fruits (i.e., target fruits), obstructed fruits, picking points, etc., and blows away the obstructed fruits through an air blowing component provided on a mobile chassis, thereby realizing the separation of target fruits from obstructed fruits, avoiding problems such as wrong picking, missed picking, and erroneous picking caused by obstruction of obstructed fruits during the picking process, and realizing precise picking; the distance between the picking point and the obstructed fruits is obtained through a binocular stereo camera, and based on the acquisition of timestamps and continuous image frames, the motion vector between the picking point and the obstructed fruits is analyzed to obtain the binocular stereo camera. The relative motion relationship between the two is calculated, so that the picking time is obtained by the maximum distance between the two, effectively avoiding the influence of the obstructed fruit on the picking robot arm and the end effector during the picking of the target fruit, realizing precise picking in the complex motion process of the obstruction being blown away, and avoiding the problems of difficulty in picking or failure in picking caused by the target fruit, obstructed fruit, etc. caused by the blowing process; afterwards, the optimal motion trajectory of the end effector is planned according to the DH matrix transformation algorithm in the robot kinematics principle, and the picking of the target fruit is completed, avoiding damage to the fruit during the picking process of the end effector and problems such as wrong picking and missed picking due to fruit obstruction. In addition, the present application can effectively control the movement of the picking robot arm through the independent operation of the binocular stereo camera and the monocular vision camera, and the cooperation between the large field of view picture and the small field of view picture, to ensure that the end effector moves according to the optimal path, improve the picking accuracy, and improve the picking efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the structure of the picking robot in an embodiment of the present invention.

[0066] Figure 2 This is a schematic diagram of a binocular stereo camera acquiring picking points in an embodiment of the present invention.

[0067] Figure 3 Schematic diagram of the blowing of the blowing assembly in an embodiment of the present invention.

[0068] Figure 4 Schematic diagram of the motion of each joint in an embodiment of the present invention; wherein, Figure 4 (a) is a schematic diagram of joint positions. Figure 4 (b) is a schematic diagram of joint velocity. Figure 4 (c) is a schematic diagram of joint acceleration.

[0069] Figure 5 This is a flow chart of picking and shearing in an embodiment of the present invention.

[0070] Among them, 1. Mobile chassis; 2. Walking mechanism; 3. Industrial computer; 4. Collection frame; 5. Picking robot arm; 6. End effector; 7. Monocular vision camera; 8. Binocular stereo camera; 9. Blowing component. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0072] Example 1:

[0073] A method for intelligent and precise fruit picking based on visual perception is provided, which comprises a picking robot including a mobile chassis 1, a walking mechanism 2, an industrial computer 3, a collection frame 4, a picking robot arm 5, an end effector 6, a monocular vision camera 7, a binocular stereo camera 8 and an air blowing component 9. The walking mechanism 2 is arranged at the bottom of the mobile chassis 1 and is used for the motion and walking control of the entire picking robot (such as Figure 1 As shown, the walking mechanism 2 can adopt the existing crawler walking mechanism or other structures); the industrial computer 3, the collection frame 4, the picking robot arm 5 and the monocular vision camera 7 are respectively arranged on the mobile chassis 1 (as shown in FIG. Figure 1 As shown, the picking robot arm 5 adopts a multi-axis robot arm, and the specific number of axes is set according to actual conditions); the end effector 6 is set at the end of the picking robot arm 5 away from the mobile chassis 1, and the binocular stereo camera 8 is set at the end of the picking robot arm 5 away from the mobile chassis 1 and is located on the upper side of the end effector 6 (as shown in FIG. Figure 1 As shown); the blowing assembly 9 is arranged on the end surface of the mobile chassis 1 through a lifting bracket and the blowing assembly 9 includes a steering motor, an air compressor and a nozzle; the walking mechanism 2, the picking robot arm 5, the end effector 6, the monocular vision camera 7, the binocular stereo camera 8, the blowing assembly 9 and the drivers of the lifting bracket are respectively electrically connected to the industrial computer 3.

[0074] Taking litchi as the picking target, the specific steps are as follows:

[0075] Step A: The picking robot moves to the picking area via the walking mechanism 2 and starts the binocular stereo camera 8:

[0076] Step A1: The binocular stereo camera 8 cooperates with the industrial computer 3 to obtain the location of a partially obscured ripe bunch of fruit, which is recorded as the target fruit. After the binocular stereo camera 8 identifies the partially obscured target fruit using the Yolov5s algorithm, it takes a frame of image for positioning and identification. The obscured target fruit is then color-segmented using HSV color gamut values to separate images containing a ripe fruit identification frame and an obscured fruit identification frame. Specifically, the target fruit image is first identified using the pre-trained Yolov5s algorithm, and the image within the identification frame is captured. The captured image is then color-segmented using the statistical H (hue), S (saturation), and V (brightness) value ranges (the value range is set according to actual conditions). The segmented image is then binarized and convex hull filled (both using conventional methods in the art), and the minimum rectangular area of the connected domain is output. Finally, the obtained minimum rectangular area is used to obtain the camera pixel coordinates of the target fruit and the obscured fruit, the center point of the identification frame, and the length and width of the identification frame.

[0077] Then the overlap between the image of the ripe fruit recognition frame and the image of the occluded fruit recognition frame is judged, namely:

[0078]

[0079] Where: C1 represents the image overlap between the mature fruit recognition frame and the occluded fruit recognition frame; X 1 、Y 1 、h 1 、l 1 represents the horizontal coordinate, vertical coordinate, height and width of the starting pixel of the mature fruit recognition frame respectively; X 2 、Y 2 、h 2 、l 2 represents the horizontal coordinate, vertical coordinate, height and width of the starting pixel of the occluded fruit recognition frame respectively;

[0080] If the overlap condition is met, the occluded fruit recognition frame in front of the mature fruit recognition frame is considered to be a subordinate fruit of the corresponding target fruit. If the overlap condition is not met, the fruit is not successfully paired and the corresponding fruit image is removed.

[0081] Step A2: The picking robot arm 5 controls the end effector 6 to move toward the target fruit, and uses the blowing component 9 to blow away the obstructing fruit to separate the target fruit from the obstructing fruit (e.g., Figure 3As shown in the figure, the color segmentation of the identified target fruit bunch is performed synchronously through the HSV color gamut value, and the image containing the fruit identification frame and the branch identification frame is separated. Specifically, the target fruit image is first identified through the pre-trained Yolov5s, and the image within the identification frame is intercepted; then, for the intercepted image, the image is segmented by color gamut using the statistical H (hue), S (saturation), and V (brightness) value ranges (the value range is set according to the actual situation); and the image after color gamut segmentation is binarized and convex hull filling is performed (both conventional methods in the field can be used), and the minimum rectangular area of the connected domain is output; finally, the camera pixel coordinates of the target fruit and branch, the center point of the identification frame, and the length and width of the identification frame are obtained respectively through the obtained minimum rectangular area.

[0082] In this embodiment, image data annotation is performed for mature fruits according to the minimum standards for picking of "one branch, one fruit" and "one branch, multiple fruits". First, the RGB image samples of mature fruits taken under natural conditions in the field are screened, and the sample images with poor quality such as blur are removed. The screened images are then annotated with a labeling tool for mature fruit detection and fruit stalk segmentation data (the labeling tool can use Labelme software). After the labeling is completed, the annotated mature fruit detection and fruit stalk segmentation data are randomly divided into a training set and a test set at a ratio of 8:2. The loss function of the mature fruit detection and fruit stalk segmentation data includes two parts, the mature fruit branch detL and the fruit stalk segmentation branch segL. Among them, the fruit detection branch detL uses the original detection head loss function setting of Yolov5s, and its total detection loss consists of three parts: classification loss classL, confidence loss confL and bounding box regression loss boxL, namely:

[0083]

[0084] Where: Represent the loss weights for classification loss, confidence loss, and bounding box regression loss, respectively. In this example, their values are 0.5, 1.0, and 0.05, respectively. Classification loss and confidence loss are calculated using the binary cross entropy loss function (BCEWithLogitsLoss).

[0085] Afterwards, the overlap between the separated fruit identification frame and the branch identification frame is judged to obtain the target fruit and its corresponding branch, namely:

[0086]

[0087] Where: C2 represents the image overlap between the fruit recognition frame and the branch recognition frame; X 3 、Y 3 、h 3 、l3 represents the horizontal coordinate, vertical coordinate, height and width of the fruit recognition frame’s starting pixel respectively; X 4 、Y 4 、h 4 、l 4 respectively represent the horizontal coordinate, vertical coordinate, height and width of the branch identification frame’s starting point pixel;

[0088] If the coincidence condition is met, the fruit identification frame below the branch identification frame is considered to be the subordinate fruit of the corresponding branch. If the coincidence condition is not met, the branches are not successfully paired and the corresponding branch image is removed.

[0089] Step A3: Divide the single-value connected domain of the branch corresponding to the target fruit into ten equal parts in the height direction, determine the width of the connected domain in the horizontal direction of each dividing line, and extract the branch image of the adjacent branches that complete the fruit relationship pairing in the horizontal direction, compare the width difference of the single-value connected domain, and obtain the picking point of the corresponding target fruit. Figure 2 As shown, the specific method is:

[0090] First, subtract the width values in the adjacent horizontal directions from each other to obtain the width difference. △X i ( i=0,1,2,…, 9 ), compare the width differences with each other: △X i The value at the beginning of the decrease is used as the starting point for judgment. When three consecutive width differences (such as: ) is greater than three pixels, it is determined △X i The corresponding connected domain is the picking point; if no point that meets the standard deviation condition is found, the final △X The connected domain corresponding to 9 is the picking point.

[0091] Step A4: The binocular stereo camera 8 obtains the dynamic separation degree of the target fruit and the occluded fruit in real time:

[0092] The target fruit picking points are obtained by using a binocular stereo camera 8 μ(x 1 ,y 1 ,z 1 ） With the center of the fruit obscured v(x 2 , y 2 ,z 2 ）The coordinates in the world coordinate system (the two-dimensional pixel coordinates in the image are converted into three-dimensional coordinates in the world coordinate system by binocular parallax calculation, and the conventional binocular parallax conversion method in this field can be used). The timestamp synchronization method is used to match the spatial coordinate points of the two motion trajectories of the picking point and the center of the occluded fruit at the same time to obtain the distance between the two at the same time. D :

[0093]

[0094] And the picking point during the blowing process is predicted in real time through Wasserstein distance μ With the center of the fruit obscured v Distance between:

[0095]

[0096]

[0097] Where, express R x R The upper edge distributions are μ and v The set of probability measures of ; 、 Don't say μ and v The corresponding cumulative distribution function is:

[0098]

[0099] Where: t The probability threshold representing the cumulative distribution function is obtained through a large amount of empirical data;

[0100] The distance between the obstructed fruit and the picking point is obtained and tracked and recorded in real time, thereby obtaining the dynamic separation between the target fruit and the obstructed fruit in real time.

[0101] During this process, the monocular vision camera 7 records a large field of view image including the target fruit picking point, the end effector 6 and the picking robot arm 5 in real time.

[0102] Step B: Obtain the relative motion relationship between the picking point and the obstructed fruit, use the size of the end effector 6 as a threshold, find the moment when the dynamic separation is maximized, and obtain the optimal picking point based on this moment; specifically:

[0103] Step B1: Under the action of wind, the obstructed fruit will shift, accompanied by a downward shift of the picking point. To accurately clip the picking point, the motion trajectories of the two are recorded in real time and analyzed and predicted. In the actual picking process, the obstructed fruit moves first with a large shift, then moves back and forth with a very small shift, and the picking point is slightly shifted. The world coordinates of the picking point and the obstructed fruit are obtained using a binocular stereo camera 8. The image is captured in real time to record the motion trajectories of the two, and the motion direction and motion vector of the two are obtained:

[0104]

[0105] Where: Indicates the motion trajectory of the occluded fruit; Represents the motion trajectory of the picking point; Represents the relative motion relationship between the obstructed fruit and the picking point; i, j, k They represent the motion components of the picking point and the blocked fruit on the x-axis, y-axis, and z-axis respectively.

[0106] Step B2: The probability distribution input in random computer experiments usually has finite support and a certain degree of continuity. Therefore, the size of the end effector 6 is used as a threshold and the Wasserstein distance is assigned to the space:

[0107]

[0108] Where: is a constant and ;

[0109] After the Wasserstein distance is assigned to this space, it can be similar to the distance-based space filling design in Euclidean space. The corresponding space filling criterion is defined. For any support distribution function F on [0,1], a natural number m (the value of m is relatively large) is taken and piecewise linear function approximation is adopted:

[0110]

[0111] Where: I is the characteristic function;

[0112] therefore, The function can be approximated by (m-1) dimensional space, namely:

[0113]

[0114] Where: a n represents the distribution function value of the 𝑛th segment;

[0115] Right now The maximum Wasserstein distance is:

[0116]

[0117] The maximum distance between the blocked fruit and the picking point is obtained to prepare for the robot arm motion trajectory planning. When the distance between the two reaches the maximum value, the picking point is predicted.

[0118] Step C: When the picking point of the target fruit falls into the optimal picking point, perform clipping. The specific method is as follows:

[0119] Step C1: When the distance between the obstructed fruit and the picking point in step B reaches the peak value, the monocular vision camera 7 records the motion vector between the picking point and the obstructed fruit, and plans the motion trajectory of the end effector 6 by controlling the picking robot 5 and using the DH (Denavit-Hartenberg) matrix transformation algorithm in the robot kinematics principle. Specifically,

[0120] The picking robot arm consists of four joints, and the motion of each joint is described by an independent DH parameter, including the joint angle , joint offset d i , connecting rod length u i Torsion angle of connecting rod :

[0121]

[0122] Calculate the transformation matrix of each joint in turn and multiply them to get the overall transformation matrix of the end effector:

[0123]

[0124] Where: T i Indicates the i The transformation matrix of each joint, i=1,2,3,4;

[0125] The transformation matrix T0 of the end effector relative to the base coordinate system is:

[0126]

[0127] Set the transformation matrix of the end effector T target , solve the corresponding joint angles through inverse kinematics .

[0128] Step C2: The trajectory planning of the picking robot 5 to the predicted picking point is performed using a fifth-order polynomial interpolation method. The end effector 6 is transformed from its initial position to the position of the predicted picking point by adjusting the function of each joint angle changing with time:

[0129]

[0130] in, T Time 0 is the initial pose, T f The moment is the set target posture, and T The turning angle at time 0 is 、 T f The turning point of time is ,but:

[0131]

[0132] at the same time:

[0133]

[0134] The end effector 6 of the picking robot moves from the initial position to the picking point position, and the end effector 6 pinches and shears the picking point to complete the picking.

[0135] Step D: Picking is complete. The picking robot 5 moves the end effector 6 containing the target fruit to the collection frame 4. The end effector 6 releases the target fruit, completing the placement of the fruit bunch after picking. The entire mechanism then resets and begins the next picking cycle. If there are still fruits waiting to be picked in the camera image, the above process repeats. When there are no more fruits to be picked in the camera image frame, picking is complete, and the picking robot 5 adjusts its posture to the ready-to-pick mode.

[0136] Example 2:

[0137] As another preferred embodiment of the present invention, on the basis of the scheme of Example 1, the lifting bracket includes a bracket body, a screw, a lifting motor and a sliding seat, the bracket body is fixedly arranged on the end face of the movable base and a vertical slide groove is provided in the bracket body, a rotating screw is arranged in the vertical slide groove, a lifting motor is arranged at the bottom end of the bracket body and the output shaft of the lifting motor is fixedly connected to the bottom end of the screw, the sliding seat is slidably arranged in the vertical slide groove and the sliding seat is spirally penetrated by the screw (that is, the screw penetrates the sliding seat and is threadedly connected); the sliding seat is located on one side of the bracket body and a support seat is fixedly arranged, and the support seat is slidably connected to the corresponding side wall of the bracket body, a steering motor is arranged on the end face of the support seat and a rotatable nozzle is arranged on the bottom face of the support seat, and the end of the nozzle close to the support seat is fixedly sleeved on the outer wall of the output shaft of the steering motor, so that the nozzle is controlled to rotate around the central axis of the output shaft of the steering motor by the steering motor; the air compressor is fixedly arranged on the end face of the movable base and the air compressor is connected to the end of the nozzle away from the support seat through a conduit.

[0138] Example 3:

[0139] As another preferred embodiment of the present invention, based on the solution of Example 1 or Example 2, the collecting frame 4 is evenly paved with a sponge layer to prevent the fruit from being damaged when placed.

Claims

1. A method for intelligent and precise fruit picking based on visual perception, characterized by: The picking robot includes a mobile chassis, a walking mechanism, an industrial computer, a collection frame, a picking robot arm, an end effector, a monocular vision camera, a binocular stereo camera and an air blowing component. The picking steps are as follows: Step A: The picking robot moves to the picking area, and the binocular stereo camera captures the location of the ripe bunch fruit, which is recorded as the target fruit. The picking robot arm controls the end effector to move toward the target fruit and separates the target fruit from the obstructed fruit through the air blowing component. The binocular stereo camera obtains the dynamic separation between the target fruit and the occluded fruit in real time, specifically: Obtain the target fruit picking points through binocular stereo cameras μ(x 1 ,y 1 ,z 1 ） With the center of the fruit obscured v(x 2 ,y 2 ,z 2 ） The coordinates in the world coordinate system are synchronized using the timestamp method to match the spatial coordinates of the two motion trajectories of the picking point and the center of the occluded fruit at the same time, and the distance between the two at the same time is obtained. D : And the picking point during the blowing process is predicted in real time through Wasserstein distance μ With the center of the fruit obscured v Distance between: Where, express R x R The upper edge distributions are μ and v The set of probability measures of ; 、 Don't say μ and v The corresponding cumulative distribution function is: Where: t represents the probability threshold of the cumulative distribution function; Obtain the distance between the obstructed fruit and the picking point, and track and record it in real time, thereby obtaining the dynamic separation between the target fruit and the obstructed fruit in real time; During this process, the monocular vision camera records a large field of view image including the target fruit picking point, the end effector, and the picking robot arm in real time; Step B, obtaining the relative motion relationship between the picking point and the obstructed fruit; The size of the end effector is used as a threshold to find the moment when the dynamic separation is maximized, and the optimal picking point is obtained based on this moment. Specifically, the probability distribution input in random computer experiments usually has finite support and a certain degree of continuity. Therefore, the size of the end effector is used as a threshold and the Wasserstein distance is assigned to the space: Where: is a constant and ; After the space is endowed with Wasserstein distance, it can be similar to the distance-based space filling design in Euclidean space, defining the corresponding space filling criterion, and taking the natural number m for the distribution function F on any support in [0,1], using piecewise linear function approximation: Where: I is the indicator function; therefore, The function can be approximated by (m-1) dimensional space, namely: Where: a n represents the distribution function value of the 𝑛th segment; Right now The maximum Wasserstein distance is: Obtain the maximum distance between the obscured fruit and the picking point to prepare for the robot arm's motion trajectory planning. When the distance between the two reaches the maximum value, the picking point is predicted. Step C, when the picking point of the target fruit falls into the optimal picking point, pinching and shearing are performed; Step D: After the picking is completed, the picking robot resets and proceeds to the next picking.

2. The method for intelligent and precise fruit picking based on visual perception according to claim 1, characterized in that: The walking mechanism is arranged at the bottom of the mobile chassis and is used for the motion and walking control of the entire picking robot; the industrial computer, collection frame, picking robot arm and monocular vision camera are respectively arranged on the mobile chassis; the end effector is arranged at the end of the picking robot arm away from the mobile chassis, and the binocular stereo camera is arranged at the end of the picking robot arm away from the mobile chassis and is located on the upper side of the end effector; the blowing assembly is arranged on the end face of the mobile chassis through a lifting bracket and the blowing assembly includes a steering motor, an air compressor and a nozzle; the drivers of the walking mechanism, picking robot arm, end effector, monocular vision camera, binocular stereo camera, blowing assembly and lifting bracket are respectively electrically connected to the industrial computer.

3. The method for intelligent and precise fruit picking based on visual perception according to claim 2, characterized in that: The specific steps of step A also include: Step A1: After the binocular stereo camera identifies the partially obscured target fruit using the Yolov5s algorithm, a frame of image is taken for positioning and identification; the obscured target fruit is then color segmented using the HSV color gamut value to separate the image containing the ripe fruit identification frame and the obscured fruit identification frame; then, the overlap of the ripe fruit identification frame and the obscured fruit identification frame is determined: if the overlap condition is met, the obscured fruit identification frame in front of the ripe fruit identification frame is considered to be a subordinate fruit of the corresponding target fruit; if the overlap condition is not met, the fruit is not successfully paired and the corresponding fruit image is removed; Step A2: Using an air blowing component to blow away the obstructing fruit, thereby separating the target fruit from the obstructing fruit, and simultaneously performing color segmentation on the identified target fruit cluster using the HSV color gamut value to separate the image containing the fruit identification frame and the branch identification frame; then, determining the degree of overlap between the separated fruit identification frame and the branch identification frame to obtain the target fruit and its corresponding branch; Step A3: Divide the single-value connected domain of the branch corresponding to the target fruit into ten equal parts in the height direction, determine the width of the connected domain in the horizontal direction of each dividing line, and extract the branch image of the branches that complete the fruit relationship pairing in the adjacent horizontal directions separately, compare the width difference of the single-value connected domain, and obtain the picking point corresponding to the target fruit.

4. The method for intelligent and precise fruit picking based on visual perception according to claim 3, characterized in that: The specific method of segmenting by HSV color gamut value in step A1 and step A2 is: First, the pre-trained Yolov5s is used to identify the target fruit image and capture the image within the recognition frame. Then, the image is segmented using the statistical H, S, and V value ranges. The segmented image is binarized and convex hull filled to output the minimum rectangular area of the connected domain. Finally, the camera pixel coordinates of the target fruit, the obstructing fruit or branch, the center point of the recognition frame, and the length and width of the recognition frame are obtained through the obtained minimum rectangular area.

5. The method for intelligent and precise fruit picking based on visual perception according to claim 3, characterized in that: The specific method of comparing the width differences of the single-valued connected domains in step A3 to obtain the picking points corresponding to the target fruits is as follows: first, the width values in the adjacent horizontal directions are subtracted from each other to obtain the width difference. △X i , i=0,1,2,…,9 , compare the width differences with each other: △X i The value at the beginning of the decrease is used as the starting point for judgment. When the standard deviation of the three consecutive width differences is greater than three pixels, it is judged that △X i The corresponding connected domain is the picking point; if no point that meets the standard deviation condition is found, the final △X The connected domain corresponding to 9 is the picking point.

6. The method for intelligent and precise fruit picking based on visual perception according to claim 3, characterized in that: The "obtaining the relative motion relationship between the picking point and the obstructed fruit" in step B is specifically as follows: under the action of wind, the obstructed fruit will shift, accompanied by the downward amplitude shift of the picking point. In order to accurately clip the picking point, the motion trajectories of the two are recorded in real time, and the motion trajectories of the two are analyzed and predicted; in the actual picking process, the obstructed fruit moves by first shifting significantly, then performing a very small reciprocating motion, and the picking point is slightly shifted; using a binocular stereo camera to obtain the world coordinates of the picking point and the obstructed fruit, taking a real-time screenshot of the image to record the motion trajectories of the two, and obtaining the motion direction and motion vector of the two: Where: Indicates the motion trajectory of the occluded fruit; Represents the motion trajectory of the picking point; Represents the relative motion relationship between the obstructed fruit and the picking point; i, j, k They represent the motion components of the picking point and the blocked fruit on the x-axis, y-axis, and z-axis respectively.

7. The method for intelligent and precise fruit picking based on visual perception according to claim 6, characterized in that: The specific method of step C is: Step C1: When the distance between the obstructed fruit and the picking point in step B reaches the peak value, the monocular vision camera records the motion vector between the picking point and the obstructed fruit, and plans the motion trajectory of the end effector by controlling the picking robot arm and using the DH matrix transformation algorithm in the robot kinematics principle; Step C2: The trajectory planning of the picking robot arm to the predicted picking point is performed using the fifth-order polynomial interpolation method. The end effector is adjusted from the initial position to the predicted picking point position by adjusting the function of each joint angle changing with time: in, T Time 0 is the initial pose, T f The moment is the set target posture, and T The turning angle at time 0 is 、 T f The turning point of time is ,but: at the same time: The end effector of the picking robot moves from the initial position to the picking point, and the end effector clamps and shears the picking point to complete the picking.

Images