Intelligent and accurate fruit picking method based on visual perception

Abstract

The invention provides an intelligent and accurate fruit picking method based on visual perception, relates to the field of fruit picking, and adopts a picking robot comprising a movable chassis (1), a walking mechanism (2), an industrial computer (3), a collecting frame (4), a picking mechanical arm (5), an end effector (6), a monocular vision camera (7), a binocular stereo camera (8) and a blowing assembly (9). According to the picking robot, through recognition of the binocular stereo camera (8), by means of blowing of the blowing assembly (9) and through a timestamp, the relative motion state between a picking point and sheltered fruits after blowing is obtained, and therefore sensing and precise clamping and shearing of the picking point in the motion state are achieved under the condition that the fruits are sheltered and subjected to complex motion after sheltering is removed; the picking efficiency is improved, the equipment damage probability is reduced, and the harvesting cost is controlled.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit picking, and in particular to an intelligent and precise fruit picking method based on visual perception. Background Art

[0002] Bunch fruits refer to a type of fruit that grows in bunches, that is, a mother branch of a bunch of fruits contains multiple fruits in bunches. Currently common bunch fruits include grapes, lychees, longans, black currants, etc.; due to the clustered structure of bunch fruits, they have the characteristics of large size and weight and small fruit stalks, therefore, picking bunch fruits is more cumbersome, and problems such as damage to fruits or other branches may easily occur during the picking process.

[0003] Due to the complex working environment of bunch fruit picking, the development of picking automation in this field is seriously restricted. In some areas, bunch fruits are still picked manually in semi-automatic picking positions, that is, manually holding semi-automatic tools to pick bunch fruits. This method not only affects the picking efficiency, but also increases the labor production cost and reduces the planting income of bunch fruits. At present, with the increasing development of intelligent and digital machinery, some automatic picking robots are also used in the picking process of bunch fruits. However, most of the current bunch fruit picking robots can only complete the unobstructed picking of fruits. When the fruit is blocked by other adjacent clusters of fruits, branches, leaves, etc. (due to factors such as wind and gravity, it is easy to cause mutual obstruction of fruits, branches, and leaves), the picking robot will be unable to lose the target, unable to avoid obstacles, and pick failures, which will affect the overall picking rhythm and reduce the picking efficiency. In severe cases, it may even cause damage to the fruit and branches, thereby affecting the overall income of the fruit trees. Therefore, in the picking process, if accurate picking is to be achieved, it is necessary to blow away the obstructions of the target fruit. After the obstructions are blown away, they are affected by the combined effects of gravity, wind resistance and blowing force, and are prone to uncontrollable complex movement, thereby blocking the target fruit twice or even multiple times. The target fruit is also easily offset by the blowing force, that is, the picking point is prone to offset, further increasing the difficulty of picking the target fruit, while increasing the probability of damage to the fruit and the picking robot, increasing the time required for picking, and increasing the cost of picking. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, the object of the present invention is to provide an intelligent and precise fruit picking method based on visual perception. This picking method can realize the perception and precise clipping of picking points in motion when the fruit is obscured and presents complex movements after being removed, thereby improving the picking efficiency, reducing the probability of equipment damage, and controlling the harvesting cost.

[0005] The purpose of the present invention is achieved through the following technical solutions: A method for intelligent and precise fruit picking based on visual perception, using a picking robot including a mobile chassis, a walking mechanism, an industrial computer, a collection frame, a picking mechanical arm, an end effector, a monocular vision camera, a binocular stereo camera and a blowing component, and the picking steps are as follows: Step A: The picking robot moves to the picking area, and the binocular stereo camera obtains the position of the ripe bunch fruit and records it as the target fruit. The picking robot arm controls the end effector to move toward the target fruit, and separates the target fruit from the blocked fruit through the blowing component. The binocular stereo camera obtains the dynamic separation degree of the target fruit and the blocked fruit in real time. During this process, the monocular vision camera records the large field of view image including the target fruit picking point, the end effector and the picking robot arm in real time; Step B, obtaining the relative motion relationship between the picking point and the blocked fruit, taking the size of the end effector as the threshold, finding the moment when the dynamic separation is maximized, and obtaining the optimal picking point based on the moment; Step C, when the picking point of the target fruit falls into the optimal picking point, clamping and shearing are performed; Step D: After picking is completed, the picking robot resets and proceeds to the next picking.

[0006] Based on further optimization of the above scheme, the walking mechanism is arranged at the bottom of the mobile chassis and is used for the motion walking control of the entire picking robot; the industrial computer, the collection frame, the picking robot arm and the monocular vision camera are respectively arranged on the mobile chassis; the end effector is arranged at the end of the picking robot arm away from the mobile chassis, and the binocular stereo camera is arranged at the end of the picking robot arm away from the mobile chassis and is located on the upper side of the end effector; the blowing assembly is arranged on the end face of the mobile chassis through a lifting bracket and the blowing assembly includes a steering motor, an air compressor and a nozzle; the drivers of the walking mechanism, the picking robot arm, the end effector, the monocular vision camera, the binocular stereo camera, the blowing assembly and the lifting bracket are respectively electrically connected to the industrial computer.

[0007] Based on the further optimization of the above scheme, the specific method of step A is: Step A1, after the binocular stereo camera identifies the partially obscured target fruit through the Yolov5s algorithm, a frame of image is taken for positioning and identification; the obscured target fruit is then color segmented using the HSV color gamut value to separate the image containing the ripe fruit identification frame and the obscured fruit identification frame; then the coincidence of the image of the ripe fruit identification frame and the obscured fruit identification frame is judged: if the coincidence condition is met, the obscured fruit identification frame in front of the ripe fruit identification frame is considered to be a subordinate fruit of the corresponding target fruit; if the coincidence condition is not met, the fruit is not successfully paired and the corresponding fruit image is removed; Step A2, using an air blowing component to blow away the obstructed fruit, separating the target fruit from the obstructed fruit, and simultaneously performing color segmentation on the identified target fruit bunch through the HSV color gamut value, separating the image containing the fruit identification frame and the branch identification frame; then, determining the overlap between the separated fruit identification frame and the branch identification frame, and obtaining the target fruit and its corresponding branch; Step A3, by dividing the single-valued connected domain of the branch corresponding to the target fruit into ten equal parts in the height direction, determining the width of the connected domain in the horizontal direction of each equal-dividing line, and extracting the branch image of the branches that complete the fruit relationship pairing in the adjacent horizontal directions, and comparing the width difference of the single-valued connected domain, the picking point of the corresponding target fruit is obtained; Step A4: Obtain the target fruit picking points through binocular stereo cameras μ(x 1 ,y 1 ,z 1 ） With the center of the fruit blocked v(x 2 ,y 2 ,z 2 ） The coordinates in the world coordinate system are synchronized by timestamp, and the spatial coordinate points of the two motion trajectories of the picking point and the center of the blocked fruit are matched at the same time to obtain the distance between the two at the same time. D :

[0008] And the picking point during the blowing process is predicted in real time through Wasserstein distance μ With the center of the fruit blocked v Distance between:

[0009]

[0010] In the formula, express R x R The upper edge distribution is μ and v The set of probability measures of ; , Don't say μ and v The corresponding cumulative distribution function is:

[0011] Where: t represents the probability threshold of the cumulative distribution function; The distance between the obstructed fruit and the picking point is obtained, and is tracked and recorded in real time, thereby obtaining the dynamic separation between the target fruit and the obstructed fruit in real time.

[0012] Based on further optimization of the above solution, the specific method of segmenting by HSV color gamut value in step A1 and step A2 is: First, the target fruit image is recognized through the pre-trained Yolov5s, and the image in the recognition frame is captured; then, for the captured image, the image is segmented by using the statistical H (hue), S (saturation), and V (brightness) value ranges; the image after color gamut segmentation is binarized and convex hull filling is performed to output the minimum rectangular area of ​​the connected domain; finally, the camera pixel coordinates of the target fruit, the occluded fruit or branch, the center point of the recognition frame, and the length and width of the recognition frame are obtained through the obtained minimum rectangular area.

[0013] Based on the further optimization of the above scheme, the specific method of comparing the width difference of the single-value connected domain in step A3 to obtain the picking point corresponding to the target fruit is: first subtract the width values ​​in the adjacent horizontal directions from each other to obtain the width difference △X i ( i=0,1,2,…,9 ), the width differences are compared with each other: △X i The value at the beginning of the decrease is used as the starting point for judgment. When three consecutive width differences (such as: ) is greater than three pixels, then it is judged △X i The corresponding connected domain is the picking point; if no point that meets the standard deviation condition is found, the final △X The connected domain corresponding to 9 is the picking point.

[0014] Based on the further optimization of the above scheme, the specific method of step B is: Step B1: Under the action of wind, the blocked fruit will shift, and the picking point will shift downward. In order to accurately clip the picking point, the movement trajectories of the two are recorded in real time, and the movement trajectories of the two are analyzed and predicted; in the actual picking process, the movement of the blocked fruit is first greatly offset, and then a very small reciprocating motion is performed, and the picking point is slightly offset; the world coordinates of the picking point and the blocked fruit are obtained by using a binocular stereo camera, and the image is captured in real time to record the movement trajectories of the two, and the movement direction and motion vector of the two are obtained:

[0015] Where: Indicates the motion trajectory of the occluded fruit; represents the motion trajectory of the picking point; Indicates the relative motion relationship between the obstructed fruit and the picking point; i, j, k They represent the motion components of the picking point and the obstructed fruit on the x-axis, y-axis, and z-axis respectively; Step B2: The probability distribution input in random computer experiments usually has finite support and a certain degree of continuity. Therefore, the size of the end effector is used as the threshold and the Wasserstein distance is assigned to the space:

[0016] Where: is a constant and ; After the space is endowed with Wasserstein distance, it can be similar to the distance-based space filling design in Euclidean space, define the corresponding space filling criterion, and take the natural number m (the value of m is relatively large) for any support distribution function F on [0,1], and use piecewise linear function approximation:

[0017] Where: I is the indicative function; therefore, The function can be approximated by (m-1) dimensional space, namely:

[0018] Where: a n represents the distribution function value of the 𝑛th segment; Right now The maximum Wasserstein distance is:

[0019] The maximum distance between the blocked fruit and the picking point is obtained to prepare for the robot arm motion trajectory planning. When the distance between the two reaches the maximum value, the picking point is predicted.

[0020] Based on the further optimization of the above scheme, the specific method of step C is: Step C1: When the distance between the blocked fruit and the picking point in step B reaches the peak value, the monocular vision camera records the motion vector between the picking point and the blocked fruit, and controls the picking robot arm and uses the DH ( Denavit-Hartenberg ) Matrix transformation algorithm plans the motion trajectory of the end effector; Step C2: The trajectory planning of the picking robot arm to the predicted picking point uses the fifth-order polynomial interpolation method to plan the trajectory, and adjusts the end effector from the initial posture to the predicted picking point posture through the function of each joint angle changing with time:

[0021] in, T Time 0 is the initial pose, T f The moment is the set target posture, and T The turning angle at time 0 is , T f The turning angle of time is ,but:

[0022] at the same time:

[0023] The end effector of the picking robot moves from the initial position to the picking point, and the end effector clamps and shears the picking point to complete the picking.

[0024] Based on the further optimization of the above scheme, the specific method of planning the motion trajectory of the end effector by using the DH matrix transformation algorithm in the robot kinematics principle in step C1 is: The picking robot arm consists of four joints, and the motion of each joint is described by an independent DH parameter, including the joint angle , Joint offset d i , connecting rod length u i Torsion angle of connecting rod :

[0025] Calculate the transformation matrix of each joint in turn and multiply them to get the overall transformation matrix of the end effector:

[0026] Where: T i Indicates i The transformation matrix of each joint, i=1,2,3,4; Transformation matrix of the end effector relative to the base coordinate system T 0 is:

[0027] Set the transformation matrix of the end effector T target , solve the corresponding joint angles through inverse kinematics .

[0028] The following are the technical effects of the solution of the present invention: The present application uses a binocular stereo camera in conjunction with a monocular vision camera, and utilizes a binocular stereo camera disposed at the end of a picking robot arm to realize identification of obscured ripe bunch fruits (i.e., target fruits), obstructed fruits, picking points, etc., and uses an air blowing component disposed on a mobile chassis to blow away the obstructed fruits, thereby achieving separation of the target fruits from the obstructed fruits, avoiding problems such as mis-picking, missed picking, and erroneous picking caused by obstruction of the obstructed fruits during the picking process, and realizing precise picking; the distance between the picking point and the obstructed fruits is obtained by a binocular stereo camera, and based on the acquisition of timestamps and continuous image frames, the motion vector between the picking point and the obstructed fruits is analyzed to obtain the binocular stereo camera. The relative motion relationship between the two is determined, so that the picking time is obtained through the maximum distance between the two, effectively avoiding the picking robot arm and the end effector being affected by the obstructed fruit due to the reciprocating motion during the picking of the target fruit, realizing precise picking in the complex motion process of the obstruction being blown away, and avoiding the problems of difficulty in picking or picking failure caused by the movement of the target fruit, the obstructed fruit, etc. due to the blowing process; afterwards, the optimal motion trajectory of the end effector is planned according to the DH matrix transformation algorithm in the robot kinematics principle to complete the picking of the target fruit, avoid damage to the fruit during the picking process of the end effector, and avoid problems such as wrong picking and missed picking due to fruit occlusion. In addition, the present application can effectively control the movement of the picking robot arm through the independent operation of the binocular stereo camera and the monocular vision camera, and utilize the coordination between the large field of view picture and the small field of view picture, to ensure that the end effector moves along the optimal path, improve the picking accuracy, and improve the picking efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the structure of the picking robot in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of a binocular stereo camera acquiring picking points in an embodiment of the present invention.

[0031] Figure 3 Schematic diagram of the blowing of the blowing assembly in the embodiment of the present invention.

[0032] Figure 4 Schematic diagram of the motion of each joint in an embodiment of the present invention; wherein, Figure 4 (a) is a schematic diagram of joint positions. Figure 4 (b) is a schematic diagram of joint velocity. Figure 4 (c) is a schematic diagram of joint acceleration.

[0033] Figure 5 The present invention is a flowchart of the picking and pinching shears in an embodiment of the present invention.

[0034] Among them, 1. Mobile chassis; 2. Walking mechanism; 3. Industrial computer; 4. Collection frame; 5. Picking robot arm; 6. End effector; 7. Monocular vision camera; 8. Binocular stereo camera; 9. Blowing component. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0036] Embodiment 1: A method for intelligent and precise fruit picking based on visual perception, comprising a picking robot including a mobile chassis 1, a walking mechanism 2, an industrial computer 3, a collection frame 4, a picking mechanical arm 5, an end effector 6, a monocular vision camera 7, a binocular stereo camera 8 and a blowing component 9, wherein the walking mechanism 2 is arranged at the bottom of the mobile chassis 1 and is used for motion walking control of the entire picking robot (such as Figure 1 As shown, the walking mechanism 2 can adopt the existing crawler walking mechanism or other structures); the industrial computer 3, the collection frame 4, the picking robot arm 5 and the monocular vision camera 7 are respectively arranged on the mobile chassis 1 (as shown in Figure 1 As shown, the picking robot arm 5 adopts a multi-axis robot arm, and the specific number of axes is set according to actual conditions); the end effector 6 is arranged at the end of the picking robot arm 5 away from the mobile chassis 1, and the binocular stereo camera 8 is arranged at the end of the picking robot arm 5 away from the mobile chassis 1 and is located on the upper side of the end effector 6 (as shown in FIG. Figure 1 As shown); the blowing component 9 is arranged on the end surface of the mobile chassis 1 through a lifting bracket, and the blowing component 9 includes a steering motor, an air compressor and a nozzle; the walking mechanism 2, the picking robot arm 5, the end effector 6, the monocular vision camera 7, the binocular stereo camera 8, the blowing component 9 and the driver of the lifting bracket are electrically connected to the industrial computer 3 respectively.

[0037] Taking litchi as the picking target, the specific steps of picking are: Step A: The picking robot moves to the picking area via the walking mechanism 2 and starts the binocular stereo camera 8: Step A1, the binocular stereo camera 8 cooperates with the industrial computer 3 to obtain the position of the partially obscured ripe string-shaped fruit and records it as the target fruit; after the binocular stereo camera 8 identifies the partially obscured target fruit through the Yolov5s algorithm, it takes a frame of image for positioning and identification; then the HSV color gamut value is used to perform color segmentation on the obscured target fruit, and separate the image containing the ripe fruit identification frame and the obscured fruit identification frame, specifically: firstly, the target fruit image is identified through the pre-trained Yolov5s, and the image in the identification frame is intercepted; then, for the intercepted image, the image is color gamut segmented using the statistical H (hue), S (saturation), and V (brightness) value range (the value range is set according to the actual situation); and the image after the color gamut segmentation is binarized and convex hull filling is performed (both conventional means in the field can be used), and the minimum rectangular area of ​​the connected domain is output; finally, the camera pixel coordinates of the target fruit and the obscured fruit, the center point of the identification frame, and the length and width of the identification frame are respectively obtained through the obtained minimum rectangular area.

[0038] Then the overlap between the image of the ripe fruit recognition frame and the image of the occluded fruit recognition frame is judged, namely:

[0039] Where: C1 represents the image overlap between the ripe fruit recognition frame and the occluded fruit recognition frame; X 1 、Y 1 、h 1 、l 1 represents the horizontal coordinate, vertical coordinate, height and width of the starting pixel of the mature fruit identification frame; X 2 、Y 2 、h 2 、l 2 respectively represent the horizontal coordinate, vertical coordinate, height and width of the starting pixel of the occluded fruit recognition frame; If the coincidence condition is met, the occluded fruit identification frame in front of the ripe fruit identification frame is considered to be a subordinate fruit of the corresponding target fruit. If the coincidence condition is not met, the fruit is not successfully paired and the corresponding fruit image is removed. Step A2: The picking robot arm 5 controls the end effector 6 to move toward the target fruit, and uses the blowing component 9 to blow away the obstructing fruit to separate the target fruit from the obstructing fruit (e.g. Figure 3As shown), and the fruit bunch of the identified target fruit is color segmented by HSV color gamut value, and the image containing the fruit identification frame and the branch identification frame is separated, specifically: firstly, the target fruit image is identified by the pre-trained Yolov5s, and the image in the identification frame is intercepted; then, for the intercepted image, the image is color gamut segmented by using the statistical H (hue), S (saturation), and V (brightness) value range (the value range is set according to the actual situation); and the image after color gamut segmentation is binarized and convex hull filling is performed (both conventional means in the art can be used), and the minimum rectangular area of ​​the connected domain is output; finally, the camera pixel coordinates of the target fruit and branch, the center point of the identification frame, and the length and width of the identification frame are obtained respectively through the obtained minimum rectangular area.

[0040] In this embodiment, image data annotation is performed for ripe fruits according to the minimum standards of "one branch, one fruit" and "one branch, multiple fruits" for picking; first, the RGB image samples of ripe fruits taken under natural conditions in the field are screened, and the sample images with poor quality such as blur are removed, and the screened images are annotated with ripe fruit detection and fruit stalk segmentation data using annotation tools (the annotation tool can use Labelme software). After the annotation is completed, the annotated ripe fruit detection and fruit stalk segmentation data are randomly divided into training sets and test sets at a ratio of 8:2. The loss function of ripe fruit detection and fruit stalk segmentation data includes two parts, the ripe fruit branch detL and the fruit stalk segmentation branch segL; among them, the fruit detection branch detL uses the setting of the original detection head loss function of Yolov5s, and its total detection loss consists of three parts: classification loss classL, confidence loss confL and bounding box regression loss boxL, namely:

[0041] Where: Respectively represent the loss weights of classification loss, confidence loss, and bounding box regression loss; in this embodiment, their values ​​are 0.5, 1.0, and 0.05, respectively. Classification loss and confidence loss are calculated using the binary cross entropy loss function (BCEWithLogitsLoss).

[0042] Afterwards, the overlap between the separated fruit identification frame and the branch identification frame is judged to obtain the target fruit and its corresponding branch, namely:

[0043] Where: C2 represents the image overlap between the fruit recognition frame and the branch recognition frame; X 3 、Y 3 、h 3 、l 3 respectively represent the horizontal coordinate, vertical coordinate, height and width of the fruit recognition frame’s starting pixel;X 4 、Y 4 、h 4 、l 4 respectively represent the starting pixel horizontal coordinate, vertical coordinate, identification frame height and identification frame width of the branch identification frame; If the coincidence condition is met, the fruit identification frame below the branch identification frame is considered to be the subordinate fruit of the corresponding branch. If the coincidence condition is not met, the branch pairing is not successful and the corresponding branch image is removed. Step A3: By dividing the single-value connected domain of the branch corresponding to the target fruit into ten equal parts in the height direction, determining the width of the connected domain in the horizontal direction of each equal-dividing line, and extracting the branch image separately for the branches that complete the fruit relationship pairing in the adjacent horizontal directions, and comparing the width difference of the single-value connected domain, the picking point of the corresponding target fruit is obtained, and the reference Figure 2 As shown, the specific method is: First, subtract the width values ​​in the adjacent horizontal directions from each other to obtain the width difference. △X i ( i=0,1,2,…, 9 ), the width differences are compared with each other: △X i The value at the beginning of the decrease is used as the starting point for judgment. When three consecutive width differences (such as: ) is greater than three pixels, then it is judged △X i The corresponding connected domain is the picking point; if no point that meets the standard deviation condition is found, the final △X The connected domain corresponding to 9 is the picking point.

[0044] Step A4: The binocular stereo camera 8 obtains the dynamic separation degree of the target fruit and the blocked fruit in real time: The target fruit picking points are obtained by using a binocular stereo camera 8 μ(x 1 ,y 1 ,z 1 ） With the center of the fruit blocked v(x 2 , y 2 ,z 2 ） The coordinates in the world coordinate system (the two-dimensional pixel coordinates in the image are converted into three-dimensional coordinates in the world coordinate system by binocular parallax calculation, and the conventional binocular parallax conversion method in the art can be used), and the timestamp synchronization method is used to match the spatial coordinate points of the two motion trajectories of the picking point and the center of the blocked fruit at the same time to obtain the distance between the two at the same time. D :

[0045] And the picking point during the blowing process is predicted in real time through Wasserstein distance μ With the center of the fruit blocked v Distance between:

[0046]

[0047] In the formula, express R x R The upper edge distribution is μ and v The set of probability measures of ; , Don't say μ and v The corresponding cumulative distribution function is:

[0048] Where: t The probability threshold representing the cumulative distribution function is obtained through a large amount of empirical data; The distance between the obstructed fruit and the picking point is obtained, and is tracked and recorded in real time, thereby obtaining the dynamic separation between the target fruit and the obstructed fruit in real time.

[0049] During this process, the monocular vision camera 7 records a large field of view image including the target fruit picking point, the end effector 6 and the picking robot arm 5 in real time.

[0050] Step B, obtaining the relative motion relationship between the picking point and the blocked fruit, taking the size of the end effector 6 as the threshold, finding the moment of maximum dynamic separation, and obtaining the optimal picking point based on the moment; specifically: Step B1, the blocked fruit will shift under the action of wind, and the picking point will shift downward. In order to accurately clip the picking point, the movement trajectories of the two are recorded in real time, and the movement trajectories of the two are analyzed and predicted; in the actual picking process, the movement of the blocked fruit is first greatly offset, and then a very small reciprocating motion is performed, and the picking point is slightly offset; the world coordinates of the picking point and the blocked fruit are obtained by using a binocular stereo camera 8, and the image is captured in real time to record the movement trajectories of the two, and the movement direction and motion vector of the two are obtained:

[0051] Where: Indicates the motion trajectory of the occluded fruit; represents the motion trajectory of the picking point; Indicates the relative motion relationship between the obstructed fruit and the picking point; i, j, k They represent the motion components of the picking point and the blocked fruit on the x-axis, y-axis, and z-axis respectively.

[0052] Step B2: The probability distribution input in the random computer experiment usually has finite support and a certain continuity. Therefore, the size of the end effector 6 is used as the threshold and the Wasserstein distance is assigned to the space:

[0053] Where: is a constant and ; After the space is endowed with Wasserstein distance, it can be similar to the distance-based space filling design in Euclidean space, define the corresponding space filling criterion, and take the natural number m (the value of m is relatively large) for any support distribution function F on [0,1], and use piecewise linear function approximation:

[0054] Where: I is the indicative function; therefore, The function can be approximated by (m-1) dimensional space, namely:

[0055] Where: a n represents the distribution function value of the 𝑛th segment; Right now The maximum Wasserstein distance is:

[0056] The maximum distance between the blocked fruit and the picking point is obtained to prepare for the robot arm motion trajectory planning. When the distance between the two reaches the maximum value, the picking point is predicted.

[0057] Step C: When the picking point of the target fruit falls into the optimal picking point, clamping and shearing are performed. The specific method is as follows: Step C1: When the distance between the blocked fruit and the picking point in step B reaches the operating peak, the monocular vision camera 7 records the motion vector between the picking point and the blocked fruit, and plans the motion trajectory of the end effector 6 by controlling the picking robot 5 and using the DH (Denavit-Hartenberg) matrix transformation algorithm in the robot kinematics principle, specifically: The picking robot arm consists of four joints, and the motion of each joint is described by an independent DH parameter, including the joint angle , Joint offset d i , connecting rod length u i Torsion angle of connecting rod :

[0058] Calculate the transformation matrix of each joint in turn and multiply them to get the overall transformation matrix of the end effector:

[0059] Where: T i Indicates i The transformation matrix of each joint, i=1,2,3,4; The transformation matrix T0 of the end effector relative to the base coordinate system is:

[0060] Set the transformation matrix of the end effector T target , solve the corresponding joint angles through inverse kinematics .

[0061] Step C2, the motion trajectory planning of the picking robot 5 to the predicted picking point adopts the fifth-order polynomial interpolation method to carry out trajectory planning, and the end effector 6 is transformed from the initial posture to the posture of the predicted picking point by adjusting the function of each joint angle changing with time:

[0062] in, T Time 0 is the initial pose, T f The moment is the set target posture, and T The turning angle at time 0 is , T f The turning angle of time is ,but:

[0063] at the same time:

[0064] The end effector 6 of the picking robot moves from the initial position to the picking point position, and the end effector 6 clamps and shears the picking point to complete the picking.

[0065] Step D, picking is completed, the picking robot arm 5 drives the end effector 6 containing the target fruit to move to the collection frame 4, the end effector 6 releases the target fruit, and the fruit bunch is placed after picking; then, the whole mechanism is reset and the next picking is carried out. At the same time, if there are still fruits to be picked in the camera image, the above operation is repeated. When there are no picking targets in the camera image frame, the picking is completed, and the picking robot arm 5 adjusts its posture to the waiting-for-picking mode.

[0066] Embodiment 2: As another preferred embodiment of the present invention, on the basis of the scheme of Example 1, the lifting bracket includes a bracket body, a screw, a lifting motor and a sliding seat, the bracket body is fixedly arranged on the end face of the moving base and a vertical slide groove is provided in the bracket body, a rotating screw is arranged in the vertical slide groove, a lifting motor is arranged at the bottom end of the bracket body and the output shaft of the lifting motor is fixedly connected to the bottom end of the screw, the sliding seat is slidably arranged in the vertical slide groove and the sliding seat is spirally penetrated by the screw (that is, the screw penetrates the sliding seat and is threadedly connected); the sliding seat is located on one side of the bracket body and a support seat is fixedly arranged and the support seat is slidably connected to the corresponding side wall of the bracket body, a steering motor is arranged on the end face of the support seat and a rotatable nozzle is arranged on the bottom face of the support seat, and an end of the nozzle close to the support seat is fixedly sleeved on the outer wall of the output shaft of the steering motor, so that the nozzle is controlled to rotate around the central axis of the output shaft of the steering motor by the steering motor; the air compressor is fixedly arranged on the end face of the moving base and the air compressor is connected to the end of the nozzle away from the support seat through a conduit.

[0067] Embodiment 3: As another preferred embodiment of the present invention, based on the solution of Example 1 or Example 2, a sponge layer is evenly laid on the collecting frame 4 to prevent the fruit from being damaged when placed.

Claims

1. A method for intelligent and precise fruit picking based on visual perception, characterized in that: A picking robot including a mobile chassis, a walking mechanism, an industrial computer, a collection frame, a picking robot arm, an end effector, a monocular vision camera, a binocular stereo camera and a blowing component is used. The picking steps are as follows: Step A: The picking robot moves to the picking area, and the binocular stereo camera obtains the position of the ripe bunch fruit and records it as the target fruit. The picking robot arm controls the end effector to move toward the target fruit, and separates the target fruit from the blocked fruit through the blowing component. The binocular stereo camera obtains the dynamic separation degree of the target fruit and the blocked fruit in real time. During this process, the monocular vision camera records the large field of view image including the target fruit picking point, the end effector and the picking robot arm in real time; Step B, obtaining the relative motion relationship between the picking point and the blocked fruit, taking the size of the end effector as the threshold, finding the moment when the dynamic separation is maximized, and obtaining the optimal picking point based on the moment; Step C, when the picking point of the target fruit falls into the optimal picking point, clamping and shearing are performed; Step D: After picking is completed, the picking robot resets and proceeds to the next picking.

2. The method for intelligent and precise fruit picking based on visual perception according to claim 1, characterized in that: The walking mechanism is arranged at the bottom of the mobile chassis and is used for the movement and walking control of the entire picking robot; the industrial computer, the collection frame, the picking robot arm and the monocular vision camera are respectively arranged on the mobile chassis; the end effector is arranged at the end of the picking robot arm away from the mobile chassis, and the binocular stereo camera is arranged at the end of the picking robot arm away from the mobile chassis and is located on the upper side of the end effector; the blowing assembly is arranged on the end face of the mobile chassis through a lifting bracket and the blowing assembly includes a steering motor, an air compressor and a nozzle; the drivers of the walking mechanism, the picking robot arm, the end effector, the monocular vision camera, the binocular stereo camera, the blowing assembly and the lifting bracket are respectively electrically connected to the industrial computer.

3. The method for intelligent and precise fruit picking based on visual perception according to claim 2, characterized in that: The specific method of step A is: Step A1, after the binocular stereo camera identifies the partially obscured target fruit through the Yolov5s algorithm, a frame of image is taken for positioning and identification; the obscured target fruit is then color segmented using the HSV color gamut value to separate the image containing the ripe fruit identification frame and the obscured fruit identification frame; then the coincidence of the image of the ripe fruit identification frame and the obscured fruit identification frame is judged: if the coincidence condition is met, the obscured fruit identification frame in front of the ripe fruit identification frame is considered to be a subordinate fruit of the corresponding target fruit; if the coincidence condition is not met, the fruit is not successfully paired and the corresponding fruit image is removed; Step A2, using an air blowing component to blow away the obstructed fruit, separating the target fruit from the obstructed fruit, and simultaneously performing color segmentation on the identified target fruit bunch through the HSV color gamut value, separating the image containing the fruit identification frame and the branch identification frame; then, determining the overlap between the separated fruit identification frame and the branch identification frame, and obtaining the target fruit and its corresponding branch; Step A3, by dividing the single-valued connected domain of the branch corresponding to the target fruit into ten equal parts in the height direction, determining the width of the connected domain in the horizontal direction of each equal-dividing line, and extracting the branch image of the branches that complete the fruit relationship pairing in the adjacent horizontal directions, and comparing the width difference of the single-valued connected domain, the picking point of the corresponding target fruit is obtained; Step A4: Obtain the target fruit picking points through binocular stereo cameras μ(x 1 ,y 1 ,z 1 ） With the center of the fruit blocked v(x 2 , y 2 ,z 2 ） The coordinates in the world coordinate system are synchronized by timestamp, and the spatial coordinate points of the two motion trajectories of the picking point and the center of the blocked fruit are matched at the same time to obtain the distance between the two at the same time. D : And the picking point during the blowing process is predicted in real time through Wasserstein distance μ With the center of the fruit blocked v Distance between: In the formula, express R x R The upper edge distribution is μ and v The set of probability measures of ; , Don't say μ and v The corresponding cumulative distribution function is: Where: t represents the probability threshold of the cumulative distribution function; The distance between the obstructed fruit and the picking point is obtained, and is tracked and recorded in real time, thereby obtaining the dynamic separation between the target fruit and the obstructed fruit in real time.

4. A method for intelligent and precise fruit picking based on visual perception according to claim 1 or 3, characterized in that: The specific method of segmenting by HSV color gamut value in step A1 and step A2 is: First, the target fruit image is recognized through the pre-trained Yolov5s, and the image in the recognition frame is captured; then, the image is segmented by color gamut using the statistical H, S, and V value ranges; the image after color gamut segmentation is binarized and convex hull filling is performed to output the minimum rectangular area of ​​the connected domain; finally, the camera pixel coordinates of the target fruit, the occluded fruit or branch, the center point of the recognition frame, and the length and width of the recognition frame are obtained through the obtained minimum rectangular area.

5. The method for intelligent and precise fruit picking based on visual perception according to claim 1 or 3, characterized in that: The specific method of comparing the width difference of the single-value connected domain in step A3 to obtain the picking point corresponding to the target fruit is: first, subtract the width values ​​in the adjacent horizontal directions from each other to obtain the width difference. △X i ( i=0,1,2,…,9 ), the width differences are compared with each other: △X i The value at the beginning of the decrease is used as the starting point for judgment. When the standard deviation of the three consecutive width differences is greater than three pixels, it is judged that △X i The corresponding connected domain is the picking point; if no point that meets the standard deviation condition is found, the final △X The connected domain corresponding to 9 is the picking point.

6. The method for intelligent and precise fruit picking based on visual perception according to claim 3 is characterized in that: The specific method of step B is: Step B1: Under the action of wind, the blocked fruit will shift, and the picking point will shift downward. In order to accurately clip the picking point, the movement trajectories of the two are recorded in real time, and the movement trajectories of the two are analyzed and predicted; in the actual picking process, the movement of the blocked fruit is first greatly offset, and then a very small reciprocating motion is performed, and the picking point is slightly offset; the world coordinates of the picking point and the blocked fruit are obtained by using a binocular stereo camera, and the image is captured in real time to record the movement trajectories of the two, and the movement direction and motion vector of the two are obtained: Where: Indicates the motion trajectory of the occluded fruit; represents the motion trajectory of the picking point; Indicates the relative motion relationship between the obstructed fruit and the picking point; i, j, k They represent the motion components of the picking point and the obstructed fruit on the x-axis, y-axis, and z-axis respectively; Step B2: The probability distribution input in random computer experiments usually has finite support and a certain degree of continuity. Therefore, the size of the end effector is used as the threshold and the Wasserstein distance is assigned to the space: Where: is a constant and ; After the space is endowed with Wasserstein distance, it can be similar to the distance-based space filling design in Euclidean space, define the corresponding space filling criterion, and use piecewise linear function approximation for any support distribution function F on [0,1], taking the natural number m: Where: I is the indicative function; therefore, The function can be approximated by (m-1) dimensional space, namely: Where: a n represents the distribution function value of the 𝑛th segment; Right now The maximum Wasserstein distance is: The maximum distance between the blocked fruit and the picking point is obtained to prepare for the robot arm motion trajectory planning. When the distance between the two reaches the maximum value, the picking point is predicted.

7. The method for intelligent and precise fruit picking based on visual perception according to claim 6, characterized in that: The specific method of step C is: Step C1, when the distance between the blocked fruit and the picking point in step B reaches the operating peak, the monocular vision camera records the motion vector between the picking point and the blocked fruit, and plans the motion trajectory of the end effector by controlling the picking robot arm and using the DH matrix transformation algorithm in the robot kinematics principle; Step C2: The trajectory planning of the picking robot arm to the predicted picking point uses the fifth-order polynomial interpolation method to plan the trajectory, and adjusts the end effector from the initial posture to the predicted picking point posture through the function of each joint angle changing with time: in, T Time 0 is the initial pose, T f The moment is the set target posture, and T The turning angle at time 0 is , T f The turning angle of time is ,but: at the same time: The end effector of the picking robot moves from the initial position to the picking point, and the end effector clamps and shears the picking point to complete the picking.

Images