Intelligent uncovering and picking method for fruits

Abstract

The invention provides an intelligent fruit sheltering and picking method, relates to the field of intelligent fruit picking, and adopts a picking robot comprising a chassis (10), a walking mechanism (20), an industrial computer (30), a mechanical arm (40), an end effector (50), a PID (Proportion Integration Differentiation) controller (60), a sheltering mechanism (70) and a binocular stereo camera (80). The picking robot can actively separate and shield fruits and analyze parameters of the motion characteristics from sensing the motion characteristics of the branch fruits so as to fit the motion trails of the branch fruits and characterize the motion trails of the branch fruits, so that the spatial information of the picking points of the mother branches is sensed, and the picking efficiency is improved. According to the method and the device, the picking points of the mother branches under the complicated movement after the shade is removed are recognized, and the precise picking of the string-shaped fruits is realized in cooperation with the control of the mechanical arm (40) and the end executor (50), so that the automatic picking efficiency of the string-shaped fruits is effectively improved, and the harvesting cost and the picking difficulty are reduced.

Description

Technical Field

[0001] The invention relates to the technical field of intelligent fruit picking, and in particular to an intelligent fruit picking method by removing obstructions. Background Art

[0002] When picking bunch fruits such as lychees, grapes, longans, and black currants, automated or semi-automated picking tools are usually used to clip the mother branches of the bunch fruits. This picking method not only ensures that the fruits are picked in bunches, improving the convenience of storage, transportation, and sales, but also avoids damage to the fruits during the picking process and improves picking efficiency. However, in the process of picking bunch fruits, especially in the process of picking bunch fruits with relatively scattered fruits such as lychee and longan, the mother branches of the fruits to be picked are easily blocked by the fruits of adjacent clusters, which increases the difficulty of picking bunch fruits. In addition, the automated picking process is prone to damage to the fruits of adjacent clusters and causes problems such as mis-picking, missed picking and wrong picking, which reduces the accuracy and efficiency of picking. Therefore, most bunch fruits are currently picked mainly by manual semi-automation, and the blocked fruit clusters are manually pushed aside to complete the precise picking of the target clusters. However, manually pushing aside the blocked fruit clusters is a great waste of manpower and material resources and increases the cost of fruit harvesting. Secondly, the picking efficiency is low and the picking time is long, which easily prolongs the entire harvesting cycle of the bunch fruits. Thirdly, it requires a high level of experience for the operator, who needs to distinguish between the mother branches of the blocked fruit and the target fruit clusters. It can be seen that the current field of bunch fruit picking lacks an automated picking robot for bunch fruits, which can actively separate the target fruit from the obstruction (for example, blown away by wind or removed by mechanical means, etc.) during the bunch fruit picking process, and after separating the obstruction, can effectively adapt to the complex trajectory movement of the target fruit and the obstruction due to environmental factors (such as gravity, external wind force, branch pulling force, etc.) and separation force (blowing away wind force or robot collision force, etc.) (such as the obstruction will appear to be greatly offset and then perform a small reciprocating motion), ensuring precise picking without damage to adjacent clusters of fruit, branches, and picking robots, thereby improving picking efficiency while reducing picking costs and losses. Summary of the invention

[0003] In view of the problems existing in the above-mentioned prior art, the object of the present invention is to provide an intelligent fruit removing and picking method, which can actively separate the obscured fruits while identifying the picking points of the mother branches under complex movements based on visual perception through an automated picking robot, thereby completing the precise clipping operation of bunch-type fruits, avoiding problems such as wrong picking, missed picking, and erroneous picking due to fruit obstruction and separation obstruction, as well as damage to the fruit or picking device, thereby improving picking efficiency and reducing harvesting costs.

[0004] The purpose of the present invention is achieved through the following technical solutions: An intelligent fruit picking method using a picking robot including a chassis, a walking mechanism, an industrial computer, a mechanical arm, an end effector, a PID controller, a fruit picking mechanism, and a binocular stereo camera. The specific picking method is as follows: Step A: The picking robot moves to the area to be picked, and the binocular stereo camera identifies the bunch-shaped fruits, performs three-dimensional reconstruction, and extracts various parameters of the bunch-shaped fruits; Step B, determining the de-shading pressure intensity for effectively separating the mother branch of the bunch-shaped fruit and the fruit that is blocked by it according to the parameter information and the spatial redundancy model; tracking and recording the motion characteristic data of the mother branch of the bunch-shaped fruit and the fruit that is blocked by it, and establishing motion characteristic mapping tables respectively; Step C, starting the shading removal mechanism to remove the shading, and monitoring the shading removal pressure in real time, using a PID controller in combination with a real-time pressure and motion characteristic mapping table to adjust the shading removal pressure, and fitting the spatial trajectory according to the spatial coordinates of the mother branch of the bunch-type fruit and the fruit it covers; Step D: sensing the picking point according to the spatial trajectory obtained by fitting, so as to realize the dynamic picking of bunch-type fruits.

[0005] Based on the further optimization of the above scheme, the walking mechanism is arranged on the lower side of the chassis and is used to control the movement of the entire picking robot; the industrial computer and the PID controller are arranged in the space of the chassis, the mechanical arm is arranged on the end face of the chassis, the end effector is arranged at the end of the mechanical arm away from the chassis, and the binocular stereo camera is installed on the upper side of the end effector; the de-obstruction mechanism is arranged on the end face of the chassis, including a vertical guide rail, a lead screw, a control motor, a slider, a connector, a rotating motor, an air spray gun and an electromagnetic control valve, the vertical guide rail is fixedly arranged on the end face of the base and a rotating lead screw is arranged in the vertical guide rail A control motor is arranged at the bottom of the vertical guide rail, and the output shaft of the control motor is fixedly connected to the bottom end of the lead screw, a slider is slidably arranged in the vertical guide rail, and the middle part of the slider is penetrated by the lead screw spiral, a connecting piece is fixedly arranged on one side of the slider, and the connecting piece is slidably connected to the corresponding side of the vertical guide rail, a rotatable air spray gun is arranged on the bottom surface of the connecting piece, and an electromagnetic control valve is arranged on the end of the air spray gun close to the connecting piece, a rotating motor is arranged on the end surface of the connecting piece, and the end of the air spray gun close to the connecting piece is fixedly sleeved on the outer wall of the output shaft of the rotating motor, so that the air spray gun is controlled to rotate around the central axis of the output shaft of the rotating motor by the rotating motor.

[0006] Based on the further optimization of the above scheme, the specific method of using the binocular stereo camera to identify bunch-shaped fruits in step A is as follows: Step A1, the binocular stereo camera recognizes the bunch of fruits through the pre-trained YOLOv8 model, and selects a frame of image for positioning; the image with overlapping parts of the fruit recognition frame is defined as a bunch of fruit with potential occlusion features; Step A2: Perform occlusion determination on the bunch of fruit with potential occlusion features:

[0007] Where: IoU represents the ratio of the intersection of images; S A , S B Respectively represent the area of ​​identification box A and identification box B; a x , c x Represents the horizontal coordinates of the two vertices of the recognition box A along the main diagonal; a y , c y Represents the ordinates of the two vertices of the identification box A along the main diagonal; b x , d x Represents the horizontal coordinates of the two vertices of the recognition box B along the main diagonal line; b y , d y Represents the ordinates of the two vertices of the identification box B along the main diagonal line; Preset occlusion threshold IoU d ,like IoU ≥ IoU d , then it is determined that the bunch of fruit has an occlusion feature, otherwise, it is not occluded.

[0008] Based on further optimization of the above scheme, the IoU d is 0.6.

[0009] Based on the further optimization of the above scheme, the specific method of performing three-dimensional reconstruction and extracting various parameters of string-shaped fruits in step A is as follows: Step A3: Use Canny edge detection to extract edge feature points of the bunch fruit image: First get the gradient strength of different pixels:

[0010] Where: G(x, y) represents the gradient amplitude; Indicates the gradient direction; G x , G y Represent the gradient of the image in the x and y directions respectively; Then, the non-maximum suppression and double threshold processing methods are used to retain the local maximum in the gradient direction and obtain the accurate detection frame edge: Preset high thresholdG max With low threshold G min ,like G(x, y) ≥ G max , then it is a strong edge, if G min ≤ G(x, y) < G max , then it is a weak edge, if G(x, y) < G max , then it is non-edge; Step A4: using the nearest neighbor matching method to match the edge features of the extracted bunches of fruit: First measure the distance between different feature points:

[0011] Where: express x i , x j The distance between two feature points; Representing images x No. r attribute value; Then, feature points with close distances are taken out, and these feature points are counted to complete the matching of edge features of bunch-shaped fruit clusters; Step A5: Convert the matched feature points into three-dimensional space coordinates:

[0012] Where: Q Represents the reprojection matrix of the binocular stereo vision camera; T Represents the baseline distance of the binocular stereo vision camera; f Indicates focal length; x’, x Respectively represent the horizontal coordinates of the left and right camera views of the binocular stereo vision camera, d represents parallax, X、 Y, Z They represent the distance components of the target in the world coordinate system respectively; W represents the weight parameter; Step A6: Use the iterative closest point method to align the point cloud coordinates into the same coordinate system to complete the registration operation of multiple image point clouds:

[0013] Where: R represents the rotation transformation matrix, U Represents the translation transformation matrix; p i Represents the corresponding point cloud p Any point inq i Represents the corresponding point cloud q The point with the shortest distance; The optimal transformation matrix is ​​obtained through iteration so that the two point clouds coincide with each other, thus completing the registration. Step A7: Use the Poisson surface reconstruction method to perform three-dimensional reconstruction on the registered point cloud, generate a three-dimensional model, and extract the fruit parameters, branch parameters and action distance of the bunch-type fruit based on the three-dimensional model.

[0014] Based on the further optimization of the above scheme, the specific method of generating a three-dimensional model by the Poisson surface reconstruction method is as follows: First, the point set of the registration point cloud J ( p 1 ,p 2 ,…,p n ) is converted to a triangulated network ST ( st 1 , st 2 , …, st n ), where each triangle consists of three vertices:

[0015] Where: Delaunay(·) express Delaunay Triangulation functions; Identify the boundary edges in the triangular network, that is, an edge that only appears in one triangle is a boundary edge, and construct a sparse matrix AX and zero vector L 0; For each edge (i,j), update the matrix: ; but:

[0016] Using triangulation and solved normal vectors F x Reconstruct 3D mesh and obtain smooth surface.

[0017] Based on the further optimization of the above scheme, the specific method of determining the deshading pressure strength for effectively separating the mother branch of the bunch-shaped fruit and the fruit it shades according to the parameter information and the spatial redundancy model in step B is: Step B1, preset a space margin threshold, if the space margin between the picking point and the blocked fruit is greater than the space margin threshold, the end effector enters the space margin and waits at the picking point for picking the mother branch of the clip-type fruit bunch; if the space margin between the picking point and the blocked fruit is less than the space margin threshold, the shading removal system is started until the space margin is greater than the space margin threshold; if the picked fruit bunch is not blocked, the end effector directly enters the picking point for picking; Step B2, using the spatial redundancy model to calculate the maximum tolerable distance between the mother branch and the obstructed fruit (i.e., the difference between the distance between the initially obstructed bunch of fruit and the bunch of fruit picking point generated in the three-dimensional reconstruction and the maximum distance after the obstruction is removed). If the maximum tolerable distance is greater than the spatial redundancy threshold, it means that the mother branch and the corresponding obstructed fruit can be effectively separated. If the maximum tolerable distance is not greater than the spatial redundancy threshold, it means that the mother branch and the corresponding obstructed fruit cannot be effectively separated. The fruit parameters and branch parameters of the bunch of fruit are extracted according to the reconstructed 3D model to obtain the effective separation and removal pressure:

[0018] Where: F Indicates the shielding force; D Indicates the maximum tolerable distance; P Indicates the effective separation and shielding pressure; S Indicates the area of ​​​​shading removal; V 2. V 1 represents the blowing gas velocity and the feeding gas velocity of the air spray gun respectively.

[0019] Based on further optimization of the above solution, the value range of the space margin threshold is 95 to 105 mm.

[0020] Based on the further optimization of the above scheme, the specific method of tracking and recording the motion characteristic data of the mother branch of the bunch-shaped fruit and the fruit covered by it in step B, and establishing the motion characteristic mapping table respectively is: Step B3: Detect local feature points in the image:

[0021] Where: H(f(x, y)) represents the Hessian matrix; det(H) The discriminant representing the eigenvalues ​​of the Hessian matrix; Represents a matrix that replaces function values ​​with image pixels; L xx 、L xy 、L yy represents the Gaussian second-order partial derivative , , Convolution with images; Represents the image pixel coordinates; The scale of the feature points is determined by the eigenvalues ​​of the Hessian matrix, the determinant of the Hessian matrix is ​​calculated, and the local extreme points in the image (i.e., the corresponding feature points in the image) are obtained; According to the feature point results matched by the reconstructed three-dimensional model, the feature points in the first frame image are matched with the feature points in the second frame image, so as to track the position changes of the feature points in different image frames and realize the tracking of the feature points in different image frames; Step B4: According to the pixel coordinates of the feature points, the speed of the mother branch and the occluded fruit is obtained by comparing the position changes of the feature points in adjacent image frames. v :

[0022] In the formula, Indicates the coordinate position of the pixel in the current frame, Indicates the coordinate position of the pixel in the previous frame, and the time interval between two adjacent frames of images; Obtain the acceleration of the mother branch and the obstructed fruit through the speed change rate a v :

[0023] Where: v 2 represents the speed of the pixel coordinates of the current frame, v 1 represents the speed of the pixel coordinates of the previous frame; At the same time, calculate the distance between the mother branch and the obstructed fruit d :

[0024] Where: (x 1 ,y 1 ) Represents the coordinates of the geometric center of the mother branch; (x 2 ,y 2 ) Represents the coordinates of the geometric center of the occluded fruit; The calculated speed, acceleration, and distance data between the mother branch and the obstructed fruit are recorded in rows, and each row corresponds to the data at a time point, thereby establishing a motion characteristic mapping table of the mother branch and its obstructed fruit.

[0025] Based on further optimization of the above scheme, the specific method of using the PID controller in step C to adjust the de-obstruction pressure in combination with the real-time pressure and motion characteristic mapping table is: According to the motion characteristic mapping table and the effective separation and de-obstruction pressure, real-time adjustment is made through the PID controller: The output adjustment amount of the PID controller is used as the adjustment amount of the deshading pressure, and is fed back to the entire system in real time. The PID control parameters are continuously adjusted according to the actual operation of the system: First determine the error value to estimate the difference between the actual pressure and the ideal effective separation and shielding pressure:

[0026] Where: e(t) Indicates the error value at the current moment; SP It indicates the ideal effective separation and shielding pressure strength set; PV Indicates the actual pressure; Then input the error value into the PID controller to obtain the output result:

[0027] Where: K p 、K i 、K d They represent proportional coefficient, integral coefficient and differential coefficient respectively; According to the output results of the PID controller, real-time feedback and adjustment are performed.

[0028] Based on the further optimization of the above scheme, the specific method of fitting the spatial trajectory according to the spatial coordinates of the mother branch of the bunch-shaped fruit and the fruit it blocks in step C is: Fit a horizontal plane curve and construct a fitting polynomial matrix:

[0029] Where: (x n ,y n ) Represents the plane coordinates of a point in the horizontal plane; represents the coefficient of the fitted curve; X n represents the horizontal coordinate matrix, Y n represents the ordinate matrix, A n represents the coefficient matrix; By solving the coefficient matrix A n , obtain the horizontal plane fitting curve; At the same time, fit the vertical plane curve and construct the fitting polynomial matrix:

[0030] Where: (y n ,z n ) Represents the plane coordinates of a point in the vertical plane; represents the coefficient of the fitted curve; Y n represents the horizontal coordinate matrix, Z n represents the ordinate matrix, B n represents the coefficient matrix; By solving the coefficient matrix B n , obtain the vertical plane fitting curve; The horizontal plane fitting curve and the vertical plane fitting curve are combined to obtain the spatial fitting curve, that is, the spatial trajectory.

[0031] Based on the further optimization of the above scheme, the specific method of sensing the picking point according to the spatial trajectory obtained by fitting in step D is: First, use Prim's algorithm to select a starting node as the root node of the tree and add it to the minimum spanning tree; initialize a set to store the nodes that have been added to the minimum spanning tree, and then initialize another set to store the nodes that have not yet been added to the minimum spanning tree; Add all the adjacent nodes of the starting node to the set, and select the edge with the smallest weight connected to the node that has been added to the minimum spanning tree from the set, add it to the minimum spanning tree, and add the corresponding node to the set; repeat the above steps until all nodes are added to the minimum spanning tree; Then, the maximum clique edge weight method is used to calculate the maximum clique edge weight in the minimum spanning tree and find the edge of the most densely connected nodes: G(V, E) is an undirected graph, where V Represents a node collection, i.e., a node , E represents the edge set, i.e., edge , w(e) Represents edge e At the same time, assuming N(v) Representation Node v j The neighbor node set of node v j The set of connected nodes, then the maximum clique edge weight MAX w for:

[0032] in: represents the edge of the minimum spanning tree; Then, for each node of the minimum spanning tree v j , and the edge weight of its maximum cluster MAX w Initialized to zero (i.e. ), traverse all edges of the minimum spanning tree and calculate the weight of the edges w(e) ; Then, for each node v j , find all edges connected to , and calculate the weights of these edges and obtain the edge with the largest weight: that is, for each node v j , compare its edge weight with the edge weight of the adjacent node. If the edge weight of the adjacent node is larger, update it to the edge weight of the adjacent node. If the edge weight of the adjacent node is smaller, do not update it until the edge weights of all nodes are no longer updated. The edge weight of each node finally obtained is the maximum clique edge weight of the minimum spanning tree. The picking points are obtained according to the edges of the nodes with the maximum clique edge weights of the minimum spanning tree.

[0033] The following are the technical effects of the solution of the present invention: The present application realizes the recognition of bunch-type fruit plants through the visual perception of a binocular stereo camera arranged at the end of a robotic arm, and based on the recognition of bunch-type fruit based on visual perception, performs three-dimensional reconstruction of bunches of bunch-type fruit with occlusion, and completes the analysis of the motion characteristics of the mother branches and the occluded fruits of the bunch-type fruit during the occlusion process, that is, under the action of the occlusion source; using the motion characteristics obtained by the analysis, not only can the real-time adjustment and control of the occlusion source be realized in conjunction with a PID controller, but also the complex motion between the mother branches and the occluded fruits can be effectively optimized, thereby fitting the motion trajectory of the mother branches and the occluded fruits, and combining the visual perception information of the spatial positioning of the picking point to establish a spatial motion model of the picking point under complex motion, thereby effectively guiding the robotic arm and the end effector to perform the clamping and picking operations of the picking point, not only completing the effective and precise separation between the picking point and the occluded fruit, but also avoiding the complex motion after the separation from affecting the clamping and picking operations of the end effector, reducing the difficulty in the automatic picking process of the bunch-type fruit, improving the automatic picking efficiency of the bunch-type fruit, and reducing the harvesting cost of the bunch-type fruit. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the structure of the picking robot in an embodiment of the present invention.

[0035] Figure 2 It is a structural schematic diagram of the shielding removal mechanism of the picking robot in an embodiment of the present invention.

[0036] Figure 3 This is a flow chart of obtaining a spatial curve for the deobstruction mechanism of the picking robot in an embodiment of the present invention.

[0037] Figure 4 It is a structural schematic diagram of PID control of the de-shading mechanism of the picking robot in an embodiment of the present invention.

[0038] Figure 5 is a graph of the spatial motion trajectory of the picking robot in an embodiment of the present invention; wherein, Figure 5 (a) is the picking point and the spatial motion trajectory curve. Figure 5 (b) is the spatial motion trajectory curve of the occluded fruit.

[0039] Figure 6 This is a schematic diagram of a picking robot sensing a picking point based on a spatial trajectory in an embodiment of the present invention.

[0040] Among them, 10, chassis; 20, walking mechanism; 30, industrial computer; 40, robotic arm; 50, end effector; 60, PID controller; 70, shielding removal mechanism; 71, vertical guide rail; 72, lead screw; 73, control motor; 74, slider; 75, connector; 76, rotating motor; 77, air spray gun; 78, electromagnetic control valve; 80, binocular stereo camera. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0042] Embodiment 1: A fruit intelligent shading removal and picking method, using a picking robot (such as a chassis 10, a walking mechanism 20, an industrial computer 30, a mechanical arm 40, an end effector 50, a PID controller 60, a shading removal mechanism 70, and a binocular stereo camera 80) Figure 1 As shown, the walking mechanism 20 is arranged at the lower side of the chassis 10 and is used to control the movement of the entire picking robot (as shown in FIG. Figure 1 As shown, in this embodiment, the walking mechanism 20 adopts a crawler-type walking mechanism); the industrial computer 30 and the PID controller 60 are arranged in the space of the chassis 10 (such as Figure 1As shown, the industrial computer 30 is arranged in the inner cavity of the chassis 10, the PID controller 60 is arranged on the end surface of the chassis 10), the robot arm 40 is arranged on the end surface of the chassis 10 (the robot arm 40 adopts a multi-axis robot arm, and the number of its axes is set according to actual conditions), the end effector 50 is arranged at the end of the robot arm 40 away from the chassis 10, and the binocular stereo camera 80 is installed on the upper side of the end effector 50 (that is, the binocular stereo camera 80 is also installed at the end of the robot arm 40 away from the chassis 10); the walking mechanism 20, the robot arm 40, the end effector 50, the PID controller 60, the shielding removal mechanism 70, and the binocular stereo camera 80 are electrically connected to the industrial computer 30 respectively. The shielding removal mechanism 70 is arranged on the end surface of the chassis 10, and includes a vertical guide rail 71, a lead screw 72, a control motor 73, a slider 74, a connector 75, a rotating motor 76, an air spray gun 77 and an electromagnetic control valve 78 (such as Figure 2 As shown in the figure, the vertical guide rail 71 is fixedly arranged on the end surface of the base 10 and a rotating lead screw 72 is arranged in the vertical guide rail 71, a control motor 73 is arranged at the bottom of the vertical guide rail 71 and the output shaft of the control motor 73 is fixedly connected to the bottom end of the lead screw 72, a slider 74 is slidably arranged in the vertical guide rail 71 and the middle part of the slider 74 is spirally penetrated by the lead screw 72 (that is, the lead screw 72 penetrates the middle part of the slider 74 and is threaded), a connecting piece 75 is fixedly arranged on one side of the slider 74 and the connecting piece 75 is slidably connected to the corresponding side of the vertical guide rail 71, a rotatable air spray gun 77 is arranged on the bottom surface of the connecting piece 75 and an electromagnetic control valve 78 is arranged at one end of the air spray gun 77 close to the connecting piece 75, a rotating motor 76 is arranged on the end surface of the connecting piece 75 and the end of the air spray gun 77 close to the connecting piece 75 is fixedly sleeved on the outer wall of the output shaft of the rotating motor 76, so that the air spray gun 77 is controlled to rotate around the central axis of the output shaft of the rotating motor 76 by the rotating motor 76.

[0043] Taking litchi as the picking target, the specific picking method is: Step A: Move the picking robot to the area to be picked by controlling the walking mechanism 20, and use the binocular stereo camera 80 to identify bunches of fruits: Step A1, the binocular stereo camera 80 recognizes the bunch of fruits through the pre-trained YOLOv8 model, and selects a frame of image for positioning; the image with overlapping parts of the fruit recognition frame is defined as a bunch of fruit with potential occlusion features; Step A2: Perform occlusion determination on the bunch of fruit with potential occlusion features:

[0044] Where: IoU represents the ratio of the intersection of images; S A , S B Respectively represent the area of ​​identification box A and identification box B; ax , c x Represents the horizontal coordinates of the two vertices of the recognition box A along the main diagonal; a y , c y Represents the ordinates of the two vertices of the identification box A along the main diagonal; b x , d x Represents the horizontal coordinates of the two vertices of the recognition box B along the main diagonal line; b y , d y Represents the ordinates of the two vertices of the identification box B along the main diagonal line; Preset occlusion threshold IoU d (In this embodiment, IoU d is 0.6), if IoU ≥ IoU d , then it is determined that the bunch of fruit has an occlusion feature, otherwise, it is not occluded.

[0045] And perform 3D reconstruction and extract various parameters of bunch-shaped fruits: Step A3: Use Canny edge detection to extract edge feature points of the bunch fruit image: First get the gradient strength of different pixels:

[0046] Where: G(x, y) represents the gradient amplitude; Indicates the gradient direction; G x , G y Represent the gradient of the image in the x and y directions respectively; Then, the non-maximum suppression and double threshold processing methods are used to retain the local maximum in the gradient direction and obtain the accurate detection frame edge: Preset high threshold G max With low threshold G min ,like G(x, y) ≥ G max , then it is a strong edge, if G min ≤ G(x, y) < G max , then it is a weak edge, if G(x, y) < G max , then it is non-edge; Step A4: using the nearest neighbor matching method to match the edge features of the extracted bunches of fruit: First measure the distance between different feature points:

[0047] Where: express x i , x j The distance between two feature points; Representing images x No. r attribute value; Then, feature points with close distances are taken out, and these feature points are counted to complete the matching of edge features of bunch-shaped fruit clusters; Step A5: Convert the matched feature points into three-dimensional space coordinates:

[0048] Where: Q Represents the reprojection matrix of the binocular stereo vision camera; T Represents the baseline distance of the binocular stereo vision camera; f Indicates focal length; x’, x Respectively represent the horizontal coordinates of the left and right camera views of the binocular stereo vision camera, d represents parallax, X、 Y, Z They represent the distance components of the target in the world coordinate system respectively; W represents the weight parameter; Step A6: Use the iterative closest point method to align the point cloud coordinates into the same coordinate system to complete the registration operation of multiple image point clouds:

[0049] Where: R represents the rotation transformation matrix, U Represents the translation transformation matrix; p i Represents the corresponding point cloud p Any point in q i Represents the corresponding point cloud q The point with the shortest distance; The optimal transformation matrix is ​​obtained through iteration so that the two point clouds coincide with each other, thus completing the registration. Step A7: Use the Poisson surface reconstruction method to perform three-dimensional reconstruction on the registered point cloud and generate a three-dimensional model, specifically: First, the point set of the registration point cloud J ( p 1,p 2 ,…,p n ) is converted to a triangulated network ST ( st 1 , st 2 , …, st n ), where each triangle consists of three vertices:

[0050] Where: Delaunay(·) express Delaunay Triangulation functions; Identify the boundary edges in the triangular network, that is, an edge that only appears in one triangle is a boundary edge, and construct a sparse matrix AX and zero vector L 0; For each edge (i,j), update the matrix:

[0051] but:

[0052] Using triangulation and solved normal vectors F x Reconstruct 3D mesh and obtain smooth surface.

[0053] The fruit parameters, branch parameters and action distance of bunch-shaped fruits are extracted based on the three-dimensional model.

[0054] Step B, determining the deshading pressure intensity for effectively separating the mother branch of the bunch-shaped fruit and the fruit it shades based on the parameter information and the spatial redundancy model: Step B1, preset the space margin threshold, the value range of the space margin threshold is 95-105mm (in this embodiment, the space margin threshold is preferably 100mm), if the space margin between the picking point and the obstructed fruit is greater than the space margin threshold, the end effector 50 enters the space margin and waits at the picking point to pick the mother branch of the clip-clipped bunch of fruit; if the space margin between the picking point and the obstructed fruit is less than the space margin threshold, the de-obstruction system is started until the space margin is greater than the space margin threshold; if the picking fruit bunch is not obstructed, the end effector 50 directly enters the picking point for picking; Step B2, using the spatial redundancy model to calculate the maximum tolerable distance between the mother branch and the obstructed fruit (i.e., the difference between the distance between the initially obstructed bunch of fruit and the bunch of fruit picking point generated in the three-dimensional reconstruction and the maximum distance after the obstruction is removed). If the maximum tolerable distance is greater than the spatial redundancy threshold, it means that the mother branch and the corresponding obstructed fruit can be effectively separated. If the maximum tolerable distance is not greater than the spatial redundancy threshold, it means that the mother branch and the corresponding obstructed fruit cannot be effectively separated. The fruit parameters and branch parameters of the bunch of fruit are extracted according to the reconstructed 3D model to obtain the effective separation and removal pressure:

[0055] Where: F Indicates the shielding force; D Indicates the maximum tolerable distance; P Indicates the effective separation and shielding pressure strength; S Indicates the area of ​​​​shading removal; V 2. V 1 represents the blowing gas velocity and the feeding gas velocity of the air spray gun respectively.

[0056] Track and record the motion characteristic data of the mother branch of the bunch-shaped fruit and its obstructed fruit, and establish motion characteristic mapping tables respectively: Step B3: Detect local feature points in the image:

[0057] Where: H(f(x, y)) represents the Hessian matrix; det(H) The discriminant representing the eigenvalues ​​of the Hessian matrix; Represents a matrix that replaces function values ​​with image pixels; L xx 、L xy 、L yy represents the Gaussian second-order partial derivative , , Convolution with images; Represents the image pixel coordinates; The scale of the feature points is determined by the eigenvalues ​​of the Hessian matrix, the determinant of the Hessian matrix is ​​calculated, and the local extreme points in the image (i.e., the corresponding feature points in the image) are obtained; According to the feature point results matched by the reconstructed three-dimensional model, the feature points in the first frame image are matched with the feature points in the second frame image, so as to track the position changes of the feature points in different image frames and realize the tracking of the feature points in different image frames; Step B4: According to the pixel coordinates of the feature points, the speed of the mother branch and the occluded fruit is obtained by comparing the position changes of the feature points in adjacent image frames. v :

[0058] In the formula, Indicates the coordinate position of the pixel in the current frame, Indicates the coordinate position of the pixel in the previous frame, and the time interval between two adjacent frames of images; Obtain the acceleration of the mother branch and the obstructed fruit through the speed change rate a v :

[0059] Where: v 2 represents the speed of the pixel coordinates of the current frame, v 1 represents the speed of the pixel coordinates of the previous frame; At the same time, calculate the distance between the mother branch and the obstructed fruit d :

[0060] Where: (x 1 ,y 1 ) Represents the coordinates of the geometric center of the mother branch; (x 2 ,y 2 ) Represents the coordinates of the geometric center of the occluded fruit; The calculated speed, acceleration, and distance data between the mother branch and the obstructed fruit are recorded in rows, and each row corresponds to the data at a time point, thereby establishing a motion characteristic mapping table of the mother branch and its obstructed fruit.

[0061] Step C: Start the shielding removal mechanism 70 to remove the shield (if there is an obstruction), and monitor the shielding removal pressure in real time (real-time monitoring can be performed by a pressure sensor set in the air spray gun 77, and the pressure sensor can be a conventional model in the art), and use the PID controller 60 to combine the real-time pressure and the motion characteristic mapping table to adjust the shielding removal pressure: According to the motion characteristic mapping table and the effective separation de-obstruction pressure, real-time adjustment is performed by the PID controller 60: The output adjustment amount of the PID controller 60 is used as the adjustment amount of the deshading pressure, and is fed back to the entire system in real time. The PID control parameters are continuously adjusted according to the actual operation of the system: First determine the error value to estimate the difference between the actual pressure and the ideal effective separation and shielding pressure:

[0062] Where: e(t) Indicates the error value at the current moment; SP It indicates the ideal effective separation and shielding pressure strength set; PV Indicates the actual pressure; The error value is then input into the PID controller 60 to obtain the output result:

[0063] Where: K p 、K i 、K d They represent proportional coefficient, integral coefficient and differential coefficient respectively; According to the output result of the PID controller 60, real-time feedback and adjustment are performed.

[0064] And the spatial trajectory is fitted according to the spatial coordinates of the mother branch of the bunch-shaped fruit and its obstructed fruit: Fit a horizontal plane curve and construct a fitting polynomial matrix:

[0065] Where: (x n ,y n ) Represents the plane coordinates of a point in the horizontal plane; represents the coefficient of the fitted curve; X n represents the horizontal coordinate matrix, Y n represents the ordinate matrix, A n represents the coefficient matrix; By solving the coefficient matrix A n , obtain the horizontal plane fitting curve; At the same time, fit the vertical plane curve and construct the fitting polynomial matrix:

[0066] Where: (y n ,z n ) Represents the plane coordinates of a point in the vertical plane; represents the coefficient of the fitted curve; Y n represents the horizontal coordinate matrix, Z n represents the ordinate matrix, B n represents the coefficient matrix; By solving the coefficient matrix B n , obtain the vertical plane fitting curve; The horizontal plane fitting curve and the vertical plane fitting curve are combined to obtain the spatial fitting curve, that is, the spatial trajectory.

[0067] Step D: Perceive the picking point based on the spatial trajectory obtained by fitting: First, use Prim's algorithm to select a starting node as the root node of the tree and add it to the minimum spanning tree; initialize a set to store the nodes that have been added to the minimum spanning tree, and then initialize another set to store the nodes that have not yet been added to the minimum spanning tree; Add all the adjacent nodes of the starting node to the set, and select the edge with the smallest weight connected to the node that has been added to the minimum spanning tree from the set, add it to the minimum spanning tree, and add the corresponding node to the set; repeat the above steps until all nodes are added to the minimum spanning tree; Then, the maximum clique edge weight method is used to calculate the maximum clique edge weight in the minimum spanning tree and find the edge of the most densely connected nodes: G(V, E) is an undirected graph, where V Represents a node collection, i.e., a node , E represents the edge set, i.e., edge , w(e) Represents edge e At the same time, assuming N(v) Representation Node v j The neighbor node set of node v j The set of connected nodes, then the maximum clique edge weight MAX w for:

[0068] in: represents the edge of the minimum spanning tree; Then, for each node of the minimum spanning tree v j , and the edge weight of its maximum cluster MAX w Initialized to zero (i.e. ), traverse all edges of the minimum spanning tree and calculate the weight of the edges w(e) ; Then, for each node v j , find all edges connected to , and calculate the weights of these edges and obtain the edge with the largest weight: that is, for each node v j, compare its edge weight with the edge weight of the adjacent node. If the edge weight of the adjacent node is larger, update it to the edge weight of the adjacent node. If the edge weight of the adjacent node is smaller, do not update it until the edge weights of all nodes are no longer updated. The edge weight of each node finally obtained is the maximum clique edge weight of the minimum spanning tree. The picking points are obtained according to the edges of the nodes with the maximum clique edge weights of the minimum spanning tree (the picking points obtained through the edges of the nodes with the maximum clique edge weights of the minimum spanning tree are potential picking point locations, and subsequent comparison and causal reasoning of the maximum clique edge weights of the minimum spanning tree are required to obtain accurate picking points).

[0069] Finally, the industrial computer 30 controls the robot arm 40 and the end effector 50 to move to the picking point, thereby completing the picking of the bunch-type fruits.

[0070] Embodiment 2: As another preferred embodiment of the present invention, based on the solution of Example 1, a storage frame is provided on the end surface of the chassis 10 for collecting the bunch-shaped fruits after picking; and a layer of buffer cotton is evenly laid on the inner wall of the storage frame.

Claims

1. A method for intelligent fruit picking by removing shading, characterized in that: A picking robot including a chassis, a walking mechanism, an industrial computer, a robotic arm, an end effector, a PID controller, a shielding removal mechanism, and a binocular stereo camera is used. The specific picking method is as follows: Step A: The picking robot moves to the area to be picked, and the binocular stereo camera identifies the bunch-shaped fruits, performs three-dimensional reconstruction, and extracts various parameters of the bunch-shaped fruits; Step B, determining the de-shading pressure intensity for effectively separating the mother branch of the bunch-shaped fruit and the fruit that is blocked by it according to the parameter information and the spatial redundancy model; tracking and recording the motion characteristic data of the mother branch of the bunch-shaped fruit and the fruit that is blocked by it, and establishing motion characteristic mapping tables respectively; Step C, starting the shading removal mechanism to remove the shading, and monitoring the shading removal pressure in real time, using a PID controller in combination with a real-time pressure and motion characteristic mapping table to adjust the shading removal pressure, and fitting the spatial trajectory according to the spatial coordinates of the mother branch of the bunch-type fruit and the fruit it covers; Step D: sensing the picking point according to the spatial trajectory obtained by fitting, so as to realize the dynamic picking of bunch-type fruits.

2. The intelligent fruit picking method according to claim 1, characterized in that: The walking mechanism is arranged on the lower side of the chassis; the industrial computer and the PID controller are arranged in the space of the chassis, the mechanical arm is arranged on the end face of the chassis, the end effector is arranged at the end of the mechanical arm away from the chassis, and the binocular stereo camera is installed on the upper side of the end effector; the shielding removal mechanism is arranged on the end face of the chassis, including a vertical guide rail, a lead screw, a control motor, a slider, a connecting piece, a rotating motor, an air spray gun and an electromagnetic control valve, the vertical guide rail is fixedly arranged on the end face of the base and a rotating lead screw is arranged in the vertical guide rail, a control motor is arranged at the bottom of the vertical guide rail and the output shaft of the control motor is fixedly connected to the bottom end of the lead screw, a slider is slidably arranged in the vertical guide rail and the middle part of the slider is penetrated by the lead screw spiral, a connecting piece is fixedly arranged on the side of one side of the slider and the connecting piece is slidably connected to the corresponding side of the vertical guide rail, a rotatable air spray gun is arranged on the bottom face of the connecting piece and an electromagnetic control valve is arranged on the end of the air spray gun close to the connecting piece, a rotating motor is arranged on the end face of the connecting piece and the end of the air spray gun close to the connecting piece is fixedly sleeved on the outer wall of the output shaft of the rotating motor.

3. The intelligent fruit picking method according to claim 1 or 2, characterized in that: The specific method for the binocular stereo camera to identify bunch-shaped fruits in step A is: Step A1, the binocular stereo camera recognizes the bunch of fruits through the pre-trained YOLOv8 model, and selects a frame of image for positioning; the image with overlapping parts of the fruit recognition frame is defined as a bunch of fruit with potential occlusion features; Step A2: Perform occlusion determination on the bunch of fruit with potential occlusion features: Where: IoU represents the ratio of the intersection of images; S A , S B Respectively represent the area of ​​identification box A and identification box B; a x , c x Represents the horizontal coordinates of the two vertices of the recognition box A along the main diagonal; a y , c y Represents the ordinates of the two vertices of the identification box A along the main diagonal; b x , d x Represents the horizontal coordinates of the two vertices of the recognition box B along the main diagonal line; b y , d y Represents the ordinates of the two vertices of the identification box B along the main diagonal line; Preset occlusion threshold IoU d ,like IoU ≥ IoU d , then it is determined that the bunch of fruit has an occlusion feature, otherwise, it is not occluded.

4. The intelligent fruit picking method according to claim 1 or 3, characterized in that: The specific method for performing three-dimensional reconstruction and extracting various parameters of string-shaped fruits in step A is as follows: Step A3: Use Canny edge detection to extract edge feature points of the bunch fruit image: First get the gradient strength of different pixels: Where: G(x,y) represents the gradient amplitude; Indicates the gradient direction; G x , G y Represent the gradient of the image in the x and y directions respectively; Then, the non-maximum suppression and double threshold processing methods are used to retain the local maximum in the gradient direction and obtain the accurate detection frame edge: Preset high threshold G max With low threshold G min ,like G(x,y)≥G max , then it is a strong edge, if G min ≤ G(x,y) <G max , then it is a weak edge, if G(x,y) <G max , then it is non-edge; Step A4: using the nearest neighbor matching method to match the edge features of the extracted bunches of fruit: First measure the distance between different feature points: Where: express x i , x j The distance between two feature points; Representing images x No. r attribute value; Then, feature points with close distances are taken out, and these feature points are counted to complete the matching of edge features of bunch-shaped fruit clusters; Step A5: Convert the matched feature points into three-dimensional space coordinates: Where: Q Represents the reprojection matrix of the binocular stereo vision camera; T Represents the baseline distance of the binocular stereo vision camera; f Indicates focal length; x'、x Respectively represent the horizontal coordinates of the left and right camera views of the binocular stereo vision camera, d represents parallax, X, Y, Z They represent the distance components of the target in the world coordinate system respectively; W represents the weight parameter; Step A6: Use the iterative closest point method to align the point cloud coordinates into the same coordinate system to complete the registration operation of multiple image point clouds: Where: R represents the rotation transformation matrix, U Represents the translation transformation matrix; p i Represents the corresponding point cloud p Any point in q i Represents the corresponding point cloud q The point with the shortest distance; The optimal transformation matrix is ​​obtained through iteration so that the two point clouds coincide with each other, thus completing the registration. Step A7: Use the Poisson surface reconstruction method to perform three-dimensional reconstruction on the registered point cloud, generate a three-dimensional model, and extract the fruit parameters, branch parameters and action distance of the bunch-type fruit based on the three-dimensional model.

5. The intelligent fruit picking method according to claim 4, characterized in that: In step B, the specific method for determining the deshading pressure strength for effectively separating the mother branch of the bunch-shaped fruit and the fruit it shades is as follows: Step B1, preset a space margin threshold, if the space margin between the picking point and the blocked fruit is greater than the space margin threshold, the end effector enters the space margin and waits at the picking point for picking the mother branch of the clip-type fruit bunch; if the space margin between the picking point and the blocked fruit is less than the space margin threshold, the shading removal system is started until the space margin is greater than the space margin threshold; if the picking fruit bunch is not blocked, the end effector directly enters the picking point for picking; Step B2, using the spatial margin model to calculate the maximum tolerable distance between the mother branch and the obstructed fruit, if the maximum tolerable distance is greater than the spatial margin threshold, it means that the mother branch and the corresponding obstructed fruit can be effectively separated, if the maximum tolerable distance is not greater than the spatial margin threshold, then they cannot be effectively separated; The fruit parameters and branch parameters of the bunch of fruit are extracted according to the reconstructed 3D model to obtain the effective separation and removal pressure: Where: F Indicates the shielding force; D Indicates the maximum tolerable distance; P Indicates the effective separation and shielding pressure; S Indicates the area of ​​​​shading removal; V 2. V 1 represents the blowing gas velocity and the feeding gas velocity of the air spray gun respectively.

6. The intelligent fruit picking method according to claim 5, characterized in that: The specific method of tracking and recording the motion characteristic data of the mother branch of the bunch-shaped fruit and the fruit covered by it in step B, and establishing the motion characteristic mapping table respectively is: Step B3: Detect local feature points in the image: Where: H(f(x,y)) represents the Hessian matrix; det(H) The discriminant representing the eigenvalues ​​of the Hessian matrix; Represents a matrix that replaces function values ​​with image pixels; L xx 、L xy 、L yy represents the Gaussian second-order partial derivative , , Convolution with images; Represents the image pixel coordinates; The scale of the feature points is determined by the eigenvalues ​​of the Hessian matrix, the determinant of the Hessian matrix is ​​calculated, and the local extreme points in the image are obtained; According to the feature point results matched by the reconstructed three-dimensional model, the feature points in the first frame image are matched with the feature points in the second frame image, so as to track the position changes of the feature points in different image frames and realize the tracking of the feature points in different image frames; Step B4: According to the pixel coordinates of the feature points, the speed of the mother branch and the occluded fruit is obtained by comparing the position changes of the feature points in adjacent image frames. v : In the formula, Indicates the coordinate position of the pixel in the current frame, Indicates the coordinate position of the pixel in the previous frame, and the time interval between two adjacent frames of images; Obtain the acceleration of the mother branch and the obstructed fruit through the speed change rate a v : Where: v 2 represents the speed of the pixel coordinates of the current frame, v 1 represents the speed of the pixel coordinates of the previous frame; At the same time, calculate the distance between the mother branch and the obstructed fruit d : Where: (x 1 ,y 1 ) Represents the coordinates of the geometric center of the mother branch; (x 2 ,y 2 ) Represents the coordinates of the geometric center of the occluded fruit; The calculated speed, acceleration, and distance data between the mother branch and the obstructed fruit are recorded in rows, and each row corresponds to the data at a time point, thereby establishing a motion characteristic mapping table of the mother branch and its obstructed fruit.

7. The intelligent fruit picking method according to claim 6, characterized in that: The specific method of fitting the spatial trajectory according to the spatial coordinates of the mother branch of the bunch-shaped fruit and the fruit it blocks in step C is: Fit a horizontal plane curve and construct a fitting polynomial matrix: Where: (x n ,y n ) Represents the plane coordinates of a point in the horizontal plane; represents the coefficient of the fitted curve; X n represents the horizontal coordinate matrix, Y n represents the ordinate matrix, A n represents the coefficient matrix; By solving the coefficient matrix A n , obtain the horizontal plane fitting curve; At the same time, fit the vertical plane curve and construct the fitting polynomial matrix: Where: (y n ,z n ) Represents the plane coordinates of a point in the vertical plane; represents the coefficient of the fitted curve; Y n represents the horizontal coordinate matrix, Z n represents the ordinate matrix, B n represents the coefficient matrix; By solving the coefficient matrix B n , obtain the vertical plane fitting curve; The horizontal plane fitting curve and the vertical plane fitting curve are combined to obtain the spatial fitting curve, that is, the spatial trajectory.

8. The intelligent fruit picking method according to claim 7, characterized in that: The specific method of sensing the picking point according to the spatial trajectory obtained by fitting in step D is: First, use Prim's algorithm to select a starting node as the root node of the tree and add it to the minimum spanning tree; initialize a set to store the nodes that have been added to the minimum spanning tree, and then initialize another set to store the nodes that have not yet been added to the minimum spanning tree; Add all the adjacent nodes of the starting node to the set, and select the edge with the smallest weight connected to the node that has been added to the minimum spanning tree from the set, add it to the minimum spanning tree, and add the corresponding node to the set; Repeat the above steps until all nodes are added to the minimum spanning tree; Then, the maximum clique edge weight method is used to calculate the maximum clique edge weight in the minimum spanning tree and find the edge of the most densely connected nodes: G(V,E) is an undirected graph, where V Represents a node collection, i.e., a node , E represents the edge set, i.e., edge , w (e) Represents edge e At the same time, assuming N(v) Representation Node v j The neighbor node set of node v j The set of connected nodes, then the maximum clique edge weight MAX w for: in: represents the edge of the minimum spanning tree; Then, for each node of the minimum spanning tree v j , and the edge weight of its maximum cluster MAX w Initialized to zero (i.e. ), traverse all edges of the minimum spanning tree and calculate the weight of the edges w(e) ; Then, for each node v j , find all edges connected to , and calculate the weights of these edges and obtain the edge with the largest weight: that is, for each node v j , compare its edge weight with the edge weight of the adjacent node. If the edge weight of the adjacent node is larger, update it to the edge weight of the adjacent node. If the edge weight of the adjacent node is smaller, do not update it until the edge weights of all nodes are no longer updated. The edge weight of each node finally obtained is the maximum clique edge weight of the minimum spanning tree. The picking points are obtained according to the edges of the nodes with the maximum clique edge weights of the minimum spanning tree.

Images