Fruit tree pruning robot and pruning method

Abstract

The invention provides a fruit tree pruning robot, and relates to the field of intelligent agriculture, the fruit tree pruning robot comprises a chassis (10), a supporting box body (20), a lifting mechanism (30), a mechanical arm (40), a tail end execution mechanism (50) and a visual identification assembly, the supporting box body (20) is arranged on the end face of the chassis (10), the lifting mechanism (30) is arranged in the supporting box body (20), and the lifting mechanism (30) is connected with a bottom joint of the mechanical arm (40); the tail end executing mechanism (50) is arranged at the end, away from the lifting mechanism (30), of the mechanical arm (40) and comprises a fixed flange base (51), a driver (52), a fixed rod (53), a push rod (54) and a disk saw (55), and the visual recognition assembly comprises a top binocular camera (61) and a bottom binocular camera (62). The pruning robot can effectively prune fruit tree branches, the pruning efficiency is high, and the pruning effect is good.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agriculture, and in particular to a fruit tree pruning robot and a pruning method. Background Art

[0002] Pruning fruit tree branches is a very important method in the process of fruit tree cultivation and planting. It can not only control the nutritional growth of fruit trees and promote more nutrients to flow to the fruits, thereby increasing the yield and quality of the fruits, but also promote the uniform distribution of branches in the crown and increase the ventilation and light transmittance of the crown. In addition, it can also timely prune away aging, disease-damaged or poorly growing branches, promote the germination and growth of new shoots, enhance the vitality of the tree, and extend the life of the fruit trees. At present, the pruning of fruit tree branches is mainly done by manual pruning. This pruning method is labor-intensive and has low pruning efficiency, and requires high professional skills of the operators. Secondly, pruning fruit branches requires climbing, which poses certain safety hazards.

[0003] With the development of smart agriculture, some parts of China are gradually using robots to prune fruit tree branches to avoid the time-consuming, low-efficiency, and potential safety hazards of manual pruning. However, most existing pruning robots have complex structures and high costs. They also have problems such as being unable to accurately reach the predetermined pruning position, large positioning errors at pruning points, and easy to accidentally cut non-target branches during pruning. In addition, existing robots are easily disturbed by external environmental factors when pruning branches in a dense range, resulting in poor pruning effects or even pruning failures, which in turn damages non-target branches, affects the secondary growth and germination of fruit trees, and reduces the economic value of fruit trees. Summary of the invention

[0004] In view of the problems existing in the above prior art, the purpose of the present invention is to provide a fruit tree pruning robot, which can effectively prune the branches of fruit trees, has high pruning efficiency, small positioning error, and will not cause problems such as mis-pruning and missed pruning, thereby ensuring the pruning effect.

[0005] Another object of the present invention is to provide a pruning method for a fruit tree pruning robot.

[0006] The purpose of the present invention is achieved through the following technical solutions: A fruit tree pruning robot comprises a chassis, a support box, a lifting mechanism, a mechanical arm, an end actuator and a visual recognition component, wherein the support box is arranged on the end surface of the chassis and the lifting mechanism is arranged in the support box, and the lifting mechanism is connected to the bottom joint of the mechanical arm; the end actuator is arranged at the end of the mechanical arm away from the lifting mechanism (i.e., the end joint of the mechanical arm), and comprises a fixed flange seat, a driver, a fixed rod, a push rod and a circular saw, the fixed flange seat is fixedly connected to the end joint of the mechanical arm, and the fixed rod and the driver are arranged at the end of the fixed flange seat away from the mechanical arm, the push rod is slidably arranged at the end of the fixed rod away from the fixed flange seat, and the rotating circular saw is arranged at the end of the push rod away from the fixed rod; the visual recognition component comprises an overhead binocular camera and a bottom binocular camera, the overhead binocular camera is arranged at the top of the end joint of the mechanical arm, and the bottom binocular camera is arranged at the front end of the chassis.

[0007] Based on further optimization of the above solution, a crawler-type walking mechanism is provided on the side of the chassis.

[0008] Based on further optimization of the above solution, the supporting box includes a box body, a cover plate and a limiting pillar. The box body is fixedly arranged on the end surface of the chassis and the cover plate is arranged above it. The limiting pillar is arranged on the end surface of the cover plate.

[0009] Based on the further optimization of the above scheme, the lifting mechanism includes a sliding guide column, a sliding guide cylinder, a lifting platform, a fixed pillar, a first lifting rod, a second lifting rod and a lifting drive. There are no less than two sliding guide columns and their lower ends are fixedly arranged on the bottom surface of the inner cavity of the box body. The sliding guide cylinder is slidably arranged on the upper end of the sliding guide column. The top ends of multiple sliding guide cylinders pass through the cover plate and are fixedly connected to the bottom surface of the same lifting platform. The mechanical arm is arranged on the end surface of the lifting platform; the fixed pillar is arranged on the bottom surface of the inner cavity of the box body, one end of the first lifting rod is slidably connected to the fixed pillar, and the other end is slidably connected to the second lifting rod, and the first lifting rod passes through the cover plate, and the second lifting rod is fixedly connected to the bottom surface of the lifting platform. The lifting drive is arranged on one side of the fixed pillar, which is used to control the extension and retraction of the first lifting rod and the second lifting rod.

[0010] A pruning method for a fruit tree pruning robot, using the above pruning robot, comprising: Step S1, target position acquisition: through the inertial sensor carried by the chassis and the bottom-mounted binocular camera, a fusion method is adopted to realize the fusion of inertial navigation and visual navigation, so as to realize the navigation of the pruning robot and reach the target fruit tree position; Step S2, height adjustment: the lifting platform is accurately made to reach a predetermined height through the lifting mechanism, the mechanical arm is driven to operate, and the position of the end actuator is adjusted to the same level as the target branch at the perspective position; Step S3, point cloud information acquisition: obtaining point cloud data of the target fruit tree through an overhead binocular camera and performing data processing; Step S4, skeleton feature extraction: separate the trunk, branches, and leaves of different parts of the fruit tree, and extract the skeleton features of the fruit tree to obtain the branch structure and connection relationship of the fruit tree; Step S5, trimming point acquisition: first, candidate trimming points are acquired through skeleton length, branch angle, and lesion information; then, the particle swarm optimization method is used to traverse and mark all candidate trimming points, and the optimal trimming point is acquired based on the set criteria; Step S6, branch pruning: the robot arm is gradually moved according to the optimal pruning point. When the end actuator reaches the pruning point, the push rod of the end actuator extends and moves from front to back, and stops moving when it reaches the end point, thereby shearing and separating the target branches and completing pruning.

[0011] Based on the further optimization of the above solution, the fusion method of step S1 is specifically as follows: Step S11, initializing relevant parameters of inertial navigation and visual navigation; Step S12: First, obtain the current position O1 of the pruning robot: And attitude angle ,in, Indicates the robot's rotation direction (i.e. x The angle of rotation of the axis, i.e. the roll angle, represents the rotation of the robot around its lateral axis (i.e. y The angle of rotation of the axis, i.e. the pitch angle, The robot rotates around the vertical axis (i.e. z The angle of rotation of the axis, i.e. the yaw angle; Then, the compensated position O2 of the lever arm (i.e. the distance between the navigation device and the measurement point, the same below) is obtained: , L x , L y , L z They represent the three directional components of the lever arm, specifically:

[0012] Where: L Indicates the actual length of the lever arm; Finally, the velocity vector after lever arm compensation is obtained V =( V x ,V y ,V z ), specifically:

[0013] Where: vx , v y , v z They represent the velocity vector of the pruning robot in the coordinate system. x axis, y axis, z The weight of the axis; Step S13: pre-process the data measured by the inertial sensor (i.e., IMU data) and convert the attitude angle in step S12 into a rotation matrix R :

[0014] Then, convert the acceleration:

[0015] Where: a1 represents the converted acceleration; a2 represents the measured acceleration; The velocity is updated through acceleration, and the position is updated using the updated velocity, specifically:

[0016]

[0017] Where: v 2 represents the speed at the next moment, v 1 represents the speed at the previous moment, Indicates the time interval between two segments (the time interval should be as small as possible); h 2 indicates the position at the next moment, h 1 indicates the position at the previous moment; Step S14, predicting the error information after navigation fusion, and updating the predicted value with data information and state feedback to correct the aforementioned position, speed and other parameters, compensate for the navigation fusion error and improve navigation accuracy.

[0018] Based on further optimization of the above solution, the preprocessing in step S13 includes debiasing, noise filtering and other processing.

[0019] Based on further optimization of the above solution, the error information after predicting navigation fusion in step S14 is specifically: First, define the error state vector:

[0020] Where: represents the position error, represents the speed error, Indicates acceleration deviation; The error feedback state Kalman filter model is used to predict the propagation of the error:

[0021]

[0022] Where: express k The error state estimate at time k -1 moment posterior estimate prediction), express k The error state estimate at time -1 (the corrected state estimate); express k The state transition matrix at time , which describes the system state from k -1 time has arrived k The changing relationship of moments; express k The prior covariance matrix at time , express k -1 posterior covariance matrix; express k The process noise covariance matrix at time t, which represents the influence of system model error and process noise; Then, the actual observations are integrated to update the state. Calculate the Kalman gain:

[0023] Where: K k express k The Kalman gain at the time, which determines the weight of the measurement value in the state update; H k express k The measurement matrix at the moment, which describes the relationship between the state vector and the measurement vector; R k express k The measurement noise covariance matrix at time instant, which represents the impact of measurement errors; Update the state estimate and covariance matrix:

[0024]

[0025] Where: Z k express k The measurement vector of the moment; I represents the identity matrix; Finally, after obtaining the posterior state Finally, the error is fed back to the navigation system to correct the navigation parameters:

[0026] Where: , , Respectively k Position, velocity, and acceleration correction values ​​at the moment; , , Respectively k The actual measured values ​​of position, velocity and acceleration at the moment.

[0027] Based on the further optimization of the above solution, step S3 is specifically as follows: Step S31, scanning the target branch at different distances and angles at a distant position to obtain sparse point cloud data, and generating dense point cloud data through multi-view stereo vision (MVS); Step S32: For each image, calculate its depth map by disparity; Step S33: fusing the depth map of each view into a unified three-dimensional point cloud through reprojection technology; Step S34: Fit a local plane, project each point onto the plane, and for each point, select its U The least square method is used to reduce noise among neighboring points. Step S35, using voxel grid downsampling to divide the point cloud into a uniform voxel grid, and retaining the voxel center as a representative point in each voxel, thereby retaining the connection between the overall and local features of the fruit tree branches.

[0028] Based on further optimization of the above solution, step S4 is specifically as follows: Step S41, using the U-Net model to segment the image, obtain segmentation masks of different regions of the fruit tree branches, and perform binarization processing on the segmented results to obtain binary images of different regions of the fruit tree branches; Step S42: for each pixel Q in the binary image ( i,j ), calculate the 8-neighborhood pixel values ​​Q1~Q8 and the number of foreground pixels N , if N≥2 and N≤6 and the neighborhood of the pixel contains one of the following conditions, namely:

[0029] Then the pixel Q( i,j ) is set to 0 and deleted directly; Repeat the iteration and continuously perform the refinement process until no pixel is deleted and then stop the iteration, so as to refine the outline of the binary image of the fruit tree branch into a skeleton; Step S43, calculating the length of the branches, the number of branch points and the trunk direction of the fruit tree by using the extracted skeleton features; Branch length: If the pixel sequence on the skeleton is Q={Q1,Q2,…,Q n}, No. x Pixels Q i The coordinates of are ( I x ,J x ), then the branch length LS is:

[0030] Number of branch points: Take pixel Q ( i,j ) as the center, take its 8-neighborhood pixel set, if the number of skeleton pixels in the neighborhood of the pixel point is not less than 3, then define the pixel point as a branch point, traverse all skeleton pixels, and count the number of all pixels that meet the conditions, that is, the number of branch points; Main direction: Extract the pixel coordinate set of the trunk part, S={q1,q2,…,q m}, No. y Pixels q i The coordinates of I y ,J y ), then its covariance matrix is:

[0031] Perform eigenvalue decomposition on the covariance matrix, where the eigenvector corresponding to the maximum eigenvalue is the trunk direction; The extracted skeleton feature image is compared with the original image and the segmentation result image to obtain the best optimization effect.

[0032] Based on further optimization of the above scheme, the specific steps of obtaining candidate pruning points through skeleton length, branch angle, and lesion information in step S5 are: Step S51, obtaining endpoints and branch points in the skeleton feature image. If there is only one foreground pixel in the eight neighborhood pixels of a pixel point, the pixel point is the endpoint, and the skeleton length is obtained through the Euclidean distance between the endpoint and each branch point; A length threshold is preset. If the obtained skeleton length is less than the length threshold, the branch is considered to be poorly grown, and the corresponding branch point is a candidate cutting point; Step S52: For each branch point Q f ( x f , y f ), through the adjacent connecting point A ( x a , y a ) and B ( x b , y b ) to obtain the corresponding branch angle :

[0033] A preset angle threshold is set. If the branch angle is less than the angle threshold, it indicates that the branches are too parallel and the corresponding branch point is a candidate cutting point. Step S53: using a deep learning model to judge the death or disease of branches by color, and identifying the dead or diseased branch points as candidate cutting points; Step S54, calculating the minimum distance between two branch points by Euclidean distance, and presetting a distance threshold. If the minimum distance between two branch points is less than the distance threshold, it is considered to be an over-dense point, and the over-dense point is used as a candidate cutting point.

[0034] The following are the technical effects of the technical solution of the present invention: The present invention adopts the method of fusing inertial navigation with visual system navigation, which not only takes advantage of the high update frequency of inertial navigation, but also takes advantage of the long-term reliable operation of visual navigation, and realizes real-time navigation by using fusion navigation, improves the positioning accuracy and real-time performance of navigation, and avoids navigation blind spots or missed target points of pruning robots, and then causes problems such as wrong pruning, missed pruning, and mis-pruning. The method of extracting the skeleton of a three-dimensional point cloud is adopted, and the candidate pruning points are obtained by combining the information such as skeleton length, branch angle, and lesion information, and then the particle swarm optimization method and the set standard are used to obtain the optimal pruning point, which can accurately and effectively obtain the characteristic information of fruit tree branches and the pruning points of fruit trees, thereby assisting the terminal actuator to accurately complete the pruning of fruit tree branches.

[0035] The pruning robot provided by the present invention can effectively avoid problems such as large positioning errors of the shearing point, poor pruning effect on branches in a dense range, and susceptibility to interference from adjacent branches during the shearing process, thereby ensuring the success rate and accuracy of shearing, and thus achieving efficient and rapid shearing of fruit tree branches. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the overall structure of the pruning robot in an embodiment of the present invention.

[0037] Figure 2Schematic diagram of the structure of the lifting mechanism of the pruning robot in an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the structure of the end effector of the pruning robot in an embodiment of the present invention.

[0039] Among them, 10, chassis; 20, supporting box; 30, lifting mechanism; 31, sliding guide column; 32, sliding guide cylinder; 33, lifting platform; 34, fixed pillar; 35, first lifting rod; 36, second lifting rod; 37, lifting drive; 40, robotic arm; 50, end actuator; 51, fixed flange seat; 52, drive; 53, fixed rod; 54, push rod; 55, circular saw; 61, top-mounted binocular camera; 62, bottom-mounted binocular camera. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below. In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are proposed to facilitate a thorough understanding of the embodiments of the present invention.

[0041] Embodiment 1: A fruit tree pruning robot comprises a chassis 10, a supporting box 20, a lifting mechanism 30, a mechanical arm 40, an end effector 50 and a visual recognition component (such as Figure 1 As shown, a crawler-type walking mechanism (such as Figure 1 As shown, the crawler-type walking mechanism can adopt a conventional walking mechanism in the art, which will not be discussed in detail in this embodiment). The support box 20 is arranged on the end surface of the chassis 10 and a lifting mechanism 30 is arranged in the support box 20. The support box 20 includes a box body, a cover plate and a limit support (such as Figure 1 As shown, the box body is fixedly arranged on the end surface of the chassis and a cover plate is arranged above it, and a limiting pillar is arranged on the end surface of the cover plate (for limiting the downward movement of the lifting platform 33). The lifting mechanism 30 is connected to the bottom joint of the mechanical arm 40, and the lifting mechanism 30 includes a sliding guide column 31, a sliding guide cylinder 32, a lifting platform 33, a fixed pillar 34, a first lifting rod 35, a second lifting rod 36 and a lifting driver 37 (as shown in FIG. Figure 2 As shown in FIG. 1 ), there are at least two sliding guide columns 31 and the lower ends thereof are fixedly arranged on the bottom surface of the inner cavity of the box body, and the upper ends of the sliding guide columns 31 are slidably arranged with sliding guide cylinders 32 (combined with Figure 1 and Figure 2As shown, in this embodiment, the number of sliding guide columns 31 and sliding guide cylinders 32 are both 4), the top ends of the plurality of sliding guide cylinders 32 pass through the cover plate and are fixedly connected to the bottom surface of the same lifting platform 33, and the robot arm 40 is arranged on the end surface of the lifting platform 33 (the robot arm 40 can be a multi-axis robot arm commonly used in the art); the fixed pillar 34 is arranged on the bottom surface of the inner cavity of the box body, one end of the first lifting rod 35 is slidably connected to the fixed pillar 34 (top end), and the other end is slidably connected to the second lifting rod 36 (bottom end), and the first lifting rod 35 passes through the cover plate, and the second lifting rod 36 is fixedly connected to the bottom surface of the lifting platform 33, and the lifting driver 37 is arranged on one side of the fixed pillar for controlling the extension and retraction of the first lifting rod 35 and the second lifting rod 36.

[0042] The end actuator 50 is disposed at one end of the robot arm 40 away from the lifting mechanism 30 (i.e., the end joint of the robot arm 40), and includes a fixed flange seat 51, a driver 52, a fixed rod 53, a push rod 54, and a circular saw 55 (such as Figure 3 As shown), the fixed flange seat 51 is fixedly connected to the end joint of the robot arm 40, and a fixed rod 53 and a driver 54 are set at the end of the fixed flange seat 51 away from the robot arm 40, a push rod 54 is slidably set at the end of the fixed rod 53 away from the fixed flange seat 51, and a rotating circular saw 55 is set at the end of the push rod 54 away from the fixed rod 53 (the rotation of the circular saw 55 can be controlled by a micro motor set on the push rod 54).

[0043] The visual recognition component includes an upper binocular camera 61 and a lower binocular camera 62. The upper binocular camera 61 is arranged at the top of the end joint of the robot arm 40, and the lower binocular camera 52 is arranged at the front end of the chassis 10 (such as Figure 1 as shown).

[0044] Embodiment 2: A pruning method for a fruit tree pruning robot, using the pruning robot as described in Example 1, comprising: Step S1, target position acquisition: through the inertial sensor carried by the chassis and the bottom-mounted binocular camera, a fusion method is adopted to realize the fusion of inertial navigation and visual navigation, so as to realize the navigation of the pruning robot and reach the target fruit tree position; specifically: Step S11, initializing the relevant parameters of inertial navigation and visual navigation (including position, speed, acceleration and other parameters); Step S12: First, obtain the current position O1 of the pruning robot: And attitude angle ,in, Indicates the robot's rotation direction (i.e. x The angle of rotation of the axis, i.e. the roll angle, represents the rotation of the robot around its lateral axis (i.e. y The angle of rotation of the axis, i.e. the pitch angle, The robot rotates around the vertical axis (i.e. z The angle of rotation of the axis, i.e. the yaw angle; Then, the compensated position O2 of the lever arm (i.e. the distance between the navigation device and the measurement point, the same below) is obtained: , L x , L y , L z They represent the three directional components of the lever arm, specifically:

[0045] Where: L Indicates the actual length of the lever arm; Finally, the velocity vector after lever arm compensation is obtained V =( V x ,V y ,V z ), specifically:

[0046] Where: v x , v y , v z They represent the velocity vector of the pruning robot in the coordinate system. x axis, y axis, z The weight of the axis; Step S13: pre-process the data measured by the inertial sensor (i.e., IMU data), including debiasing, noise filtering, etc., and convert the attitude angle in step S12 into a rotation matrix. R :

[0047] Then, convert the acceleration:

[0048] Where: a1 represents the converted acceleration; a2 represents the measured acceleration; The velocity is updated through acceleration, and the position is updated using the updated velocity, specifically:

[0049]

[0050] Where: v2 represents the speed at the next moment, v 1 represents the speed at the previous moment, Indicates the time interval between two segments (the time interval should be as small as possible); h 2 indicates the position at the next moment, h 1 indicates the position at the previous moment; Step S14: predict the error information after navigation fusion, and update the predicted value with data information and state feedback to correct the aforementioned parameters such as position and speed, compensate for the navigation fusion error, and improve navigation accuracy; the error information after navigation fusion is predicted to be: First, define the error state vector:

[0051] Where: represents the position error, represents the speed error, Indicates acceleration deviation; The error feedback state Kalman filter model is used to predict the propagation of the error:

[0052]

[0053] Where: express k The error state estimate at time k -1 moment posterior estimate prediction), express k The error state estimate at time -1 (the corrected state estimate); express k The state transition matrix at time , which describes the system state from k -1 time has arrived k The changing relationship of moments; express k The prior covariance matrix at time , express k -1 posterior covariance matrix; express k The process noise covariance matrix at time t, which represents the influence of system model error and process noise; Then, the actual observations are integrated to update the state. Calculate the Kalman gain:

[0054] Where: K k express k The Kalman gain at the time, which determines the weight of the measurement value in the state update;H k express k The measurement matrix at the moment, which describes the relationship between the state vector and the measurement vector; R k express k The measurement noise covariance matrix at time instant, which represents the impact of measurement errors; Update the state estimate and covariance matrix:

[0055]

[0056] Where: Z k express k The measurement vector of the moment; I represents the identity matrix; Finally, after obtaining the posterior state Finally, the error is fed back to the navigation system to correct the navigation parameters:

[0057] Where: , , Respectively k Position, velocity, and acceleration correction values ​​at the moment; , , Respectively k The measured values ​​of position, velocity and acceleration at the moment Step S2, height adjustment: the lifting platform 33 is accurately made to reach a predetermined height through the lifting mechanism 30, the mechanical arm 40 is driven to operate, and the position of the end actuator 50 is adjusted to the same level as the target branch at the perspective position; Specifically, first, the robot arm 40 is operated to observe whether the height of the robot arm 40 can reach the horizontal height of the target branch: if it can be reached, the robot arm 40 can work alone; if it cannot be reached, the lifting mechanism 30 starts to work and assists the robot arm 40 to lift the height. Among them, the lifting driver 37 at the bottom of the support box serves as a power source to provide power for the overall lifting of the lifting mechanism 30. The fixed support serves as the main supporting device and plays the role of overall support. The first lifting rod 35 and the second lifting rod 36 are the main lifting devices and play the role of performing the overall lifting. The sliding guide column 31 and the sliding guide cylinder 32 serve as auxiliary supporting devices and play the role of auxiliary support and guidance. The lifting platform 33 serves as an intermediate connecting device to transmit the lifting force and realize the overall lifting of the robot arm 40 and the end actuator 50. When the lifting mechanism 30 starts working, under the power of the lifting driver 37, the first lifting rod 35 is lifted first, applying thrust to the lifting platform 33, lifting the mechanical arm 40 and the end actuator 50 as a whole, and at the same time, the sliding guide cylinder 32 is also lifted to prevent the mechanical arm 40 and the end actuator 50 from tipping over due to uneven force. When the lifting height is insufficient, the second lifting rod 36 will lift and further lift the mechanical arm 40 and the end actuator 50 as a whole to reach a predetermined height position. When the trimming work is completed, the mechanical arm 40 completes the reset work first, and then the lifting mechanism is lowered to the initial position 30, completing a complete lifting work.

[0058] Step S3, point cloud information acquisition: obtain the point cloud data of the target fruit tree through the overhead binocular camera and perform data processing; specifically: Step S31, scanning the target branch at different distances and angles at a distant position to obtain sparse point cloud data, and generating dense point cloud data through multi-view stereo vision (MVS); Step S32: For each image, calculate its depth map by disparity:

[0059] Where: D(x,y) represents a depth map; f Indicates focal length; B represents the baseline distance (i.e. the distance between the two cameras); d (x,y) Indicates parallax; Step S33: fusing the depth map of each view into a unified three-dimensional point cloud through reprojection technology; First, set the internal parameter matrix K :

[0060] The pixel coordinates and depth values ​​are converted into three-dimensional coordinates using the intrinsic matrix. By multiplying the depth value with the inverse of the intrinsic matrix, the point in three-dimensional space is finally obtained. X :

[0061] Where: u , v ) represents pixel coordinates, D ( u , v ) represents the depth coordinate of the pixel; Step S34: Fit a local plane, project each point onto the plane, and for each point, select its U The least squares method is used to reduce noise:

[0062] Where: p i Indicate point p The i-th neighbor point of ; Step S35, using voxel grid downsampling to divide the point cloud into a uniform voxel grid, and retaining the voxel center as a representative point in each voxel, thereby retaining the connection between the overall and local features of the fruit tree branches.

[0063] Step S4, skeleton feature extraction: separate the trunk, branches, and leaves of different parts of the fruit tree, and extract the skeleton features of the fruit tree to obtain the branch structure and connection relationship of the fruit tree; specifically: Step S41: Use the U-Net model to segment the image, obtain segmentation masks of different regions of the fruit tree branches, and use the cross entropy loss function:

[0064] Where: represents the true label, represents the predicted probability, N The total number of samples is shown; The segmented results are binarized to obtain binary images of different regions of the fruit tree branches; Step S42: for each pixel Q in the binary image ( i,j ), calculate the 8-neighborhood pixel values ​​Q1~Q8 and the number of foreground pixels N , if N≥2 and N≤6 and the neighborhood of the pixel contains one of the following conditions, namely:

[0065] Then the pixel Q( i,j ) is set to 0 and deleted directly; Repeat the iteration and continuously perform the refinement process until no pixel is deleted and then stop the iteration, so as to refine the outline of the binary image of the fruit tree branch into a skeleton; Step S43, calculating the length of the branches, the number of branch points and the trunk direction of the fruit tree by using the extracted skeleton features; Branch length: If the pixel sequence on the skeleton is Q={Q1,Q2,…,Q n}, No. x Pixels Q i The coordinates of I x ,J x ), then the branch length LS is:

[0066] Number of branch points: Take pixel Q ( i,j ) as the center, take its 8-neighborhood pixel set, if the number of skeleton pixels in the neighborhood of the pixel point is not less than 3, then define the pixel point as a branch point, traverse all skeleton pixels, and count the number of all pixels that meet the conditions, that is, the number of branch points; Main direction: Extract the pixel coordinate set of the trunk part, S={q1,q2,…,q m}, No. y Pixels q i The coordinates of I y ,J y ), then its covariance matrix is:

[0067] Perform eigenvalue decomposition on the covariance matrix, where the eigenvector corresponding to the maximum eigenvalue is the trunk direction; The extracted skeleton feature image is compared with the original image and the segmentation result image, specifically: Assume that the skeleton feature image is I 1. The original image is I 2. For each image I 1 with image I 2 for wavelet decomposition, they are j The approximate components obtained at the layer scale are A j1 , A j2 , the detail components in the horizontal, vertical and diagonal directions obtained at the j-level scale are: D Hj1 , DVj1 , D Dj1 and D Hj2 , D Vj2 , D Dj2 , then the image I 1 with image I 2 Similarity measure at the jth level S j for:

[0068] Where: , Represents the weight coefficient, which is determined according to the actual situation; when S j When it is infinitely close to 1, the image is proved I 1 with image I 2 similarity is good, otherwise, the skeleton feature image is acquired again; similarly, the similarity measure between the skeleton feature image and the segmentation result image is obtained. If it is not close to 1, the segmentation result image and the skeleton feature image are acquired again to obtain the best optimization effect.

[0069] Step S5, pruning point acquisition: firstly, candidate pruning points are acquired through skeleton length, branch angle, and lesion information, specifically: Step S51, obtaining endpoints and branch points in the skeleton feature image. If there is only one foreground pixel in the eight neighborhood pixels of a pixel point, the pixel point is the endpoint, and the skeleton length is obtained through the Euclidean distance between the endpoint and each branch point; A length threshold is preset (in this embodiment, the length threshold is 30 cm). If the obtained skeleton length is less than the length threshold, it is considered that the branch is poorly grown, and the corresponding branch point is a candidate cutting point; Step S52: For each branch point Q f ( x f , y f ), through the adjacent connecting point A ( x a , y a ) and B ( x b , y b ) to obtain the corresponding branch angle :

[0070] A preset angle threshold (in this embodiment, the angle threshold is 15°). If the branch angle is less than the angle threshold, it indicates that the branches are too parallel, and the corresponding branch point is a candidate cutting point. Step S53: using a deep learning model to judge the death or disease of branches by color, and identifying the dead or diseased branch points as candidate cutting points; Among them, the specific training method of the deep learning model is: First, drone equipment is used to collect multispectral images, including RGB and near-infrared channel images, and the images are converted into colors. Convert RGB to Lab values:

[0071]

[0072] Where: X , Y , Z Represents the relative tristimulus values ​​in the RGB color space; X n , Y n , Z n Indicates the reference white point value; And use annotation tools (such as Labelmg) to mark the healthy / diseased / dead tree branches. Health Area: , lesion area: , Generate the mask:

[0073] In the formula, r Indicates the control of edge sharpness; The deep learning model design adopts a multi-branch hybrid architecture and a dynamic color attention mechanism. The multi-branch hybrid architecture includes an RGB branch, a Lab branch, and a priori code branch. The RGB branch inputs the original RGB image and uses EfficientNet-B3 to extract texture features; the Lab branch inputs a and b Channel, shallow CNN is used to extract color features; the prior code branch inputs the diffraction prior mask as the attention weight; the loss function uses Focal Loss as the main loss function and the color consistency loss function as the auxiliary loss function, and the color consistency is enhanced by contrastive learning loss function; Among them, the main loss function is:

[0074]

[0075] Where: It indicates the model's predicted probability that a sample belongs to the target category. Indicates adjustable focal length parameter (usually 0 to 5, select according to actual situation); Auxiliary loss function:

[0076] Where: , Respectively represent the predicted image in Lab color space and the real image in coordinates ( x , y ) a Channel value; , Respectively represent the predicted image and the real image in the Lab color space b Channel value statistics; N represents the number of statistical pixels; Contrastive learning loss function:

[0077]

[0078] Where: represents the anchor sample feature, Represents the characteristics of similar samples, Represents heterogeneous sample characteristics; Indicates the temperature coefficient (usually 0.1); Step S54, calculate the minimum distance between two branch points by Euclidean distance, preset a distance threshold (the distance threshold is set according to the specific fruit tree type, and is not specifically limited in this embodiment). If the minimum distance between two branch points is less than the distance threshold, it is considered to be an over-dense point, and the over-dense point is used as a candidate cutting point.

[0079] Then, the particle swarm optimization method is used to traverse and mark all candidate cutting points, and the best trimming point is obtained based on the set criteria. The specific criteria can be: first, determine whether steps S51 to S53 are met. If any one of steps S51 to S53 is met, it is directly determined as the best trimming point; then, the best trimming point after identification is excluded, and the density of the remaining branch points is calculated:

[0080] Where: N i Indicates the number of remaining branch points in the grid, AM Represents the area of ​​the grid; A density threshold is set (set according to the actual fruit type). If the density of the remaining branch points is not less than the density threshold, the best pruning point is further obtained through step S54, and finally all the best pruning points are integrated.

[0081] Step S6, branch pruning: the robot arm 40 is gradually moved according to the optimal pruning point. When the end actuator 50 reaches the pruning point, the push rod 54 of the end actuator 50 extends and moves from front to back (when the push rod 54 extends, the circular saw 55 is synchronously started to rotate), and stops moving when it reaches the end point, thereby achieving shearing and separation of the target branch and completing pruning.

Claims

1. A fruit tree pruning robot, characterized in that: It includes a chassis, a supporting box, a lifting mechanism, a robotic arm, an end actuator and a visual recognition component. The supporting box is arranged on the end surface of the chassis and the lifting mechanism is arranged in the supporting box, and the lifting mechanism is connected to the bottom joint of the robotic arm; the end actuator is arranged at the end of the robotic arm away from the lifting mechanism, and includes a fixed flange seat, a driver, a fixed rod, a push rod and a circular saw. The fixed flange seat is fixedly connected to the end joint of the robotic arm, and the fixed rod and the driver are arranged at the end of the fixed flange seat away from the robotic arm, a push rod is slidably arranged at the end of the fixed rod away from the fixed flange seat, and a rotating circular saw is arranged at the end of the push rod away from the fixed rod; the visual recognition component includes an overhead binocular camera and a bottom binocular camera. The overhead binocular camera is arranged at the top of the end joint of the robotic arm, and the bottom binocular camera is arranged at the front end of the chassis.

2. A fruit tree pruning robot according to claim 1, characterized in that: A crawler-type walking mechanism is arranged on the side of the chassis.

3. The fruit tree pruning robot according to claim 1, characterized in that: The supporting box body comprises a box body, a cover plate and a limiting pillar. The box body is fixedly arranged on the end surface of the chassis and the cover plate is arranged above it. The limiting pillar is arranged on the end surface of the cover plate.

4. A fruit tree pruning robot according to claim 1 or 3, characterized in that: The lifting mechanism includes a sliding guide column, a sliding guide cylinder, a lifting platform, a fixed pillar, a first lifting rod, a second lifting rod and a lifting driver. There are no less than two sliding guide columns and the lower ends of the sliding guide columns are fixedly arranged on the bottom surface of the inner cavity of the box body. The sliding guide cylinder is slidably arranged on the upper end of the sliding guide column. The top ends of the plurality of sliding guide cylinders pass through the cover plate and are fixedly connected to the bottom surface of the same lifting platform. The mechanical arm is arranged on the end surface of the lifting platform. The fixed pillar is arranged on the bottom surface of the inner cavity of the box body. One end of the first lifting rod is slidably connected to the fixed pillar and the other end is slidably connected to the second lifting rod. The first lifting rod passes through the cover plate and the second lifting rod is fixedly connected to the bottom surface of the lifting platform. The lifting driver is arranged on one side of the fixed pillar for controlling the extension and retraction of the first lifting rod and the second lifting rod.

5. A pruning method using a fruit tree pruning robot according to any one of claims 1 to 4, characterized in that: include: Step S1, target position acquisition: through the inertial sensor carried by the chassis and the bottom-mounted binocular camera, a fusion method is adopted to realize the fusion of inertial navigation and visual navigation, so as to realize the navigation of the pruning robot and reach the target fruit tree position; Step S2, height adjustment: the lifting platform is accurately made to reach a predetermined height through the lifting mechanism, the mechanical arm is driven to operate, and the position of the end actuator is adjusted to the same level as the target branch at the perspective position; Step S3, point cloud information acquisition: obtaining point cloud data of the target fruit tree through an overhead binocular camera and performing data processing; Step S4, skeleton feature extraction: separate the trunk, branches, and leaves of different parts of the fruit tree, and extract the skeleton features of the fruit tree to obtain the branch structure and connection relationship of the fruit tree; Step S5, trimming point acquisition: first, candidate trimming points are acquired through skeleton length, branch angle, and lesion information; then, the particle swarm optimization method is used to traverse and mark all candidate trimming points, and the optimal trimming point is acquired based on the set criteria; Step S6, branch pruning: the robot arm is gradually moved according to the optimal pruning point. When the end actuator reaches the pruning point, the push rod of the end actuator extends and moves from front to back, and stops moving when it reaches the end point, thereby shearing and separating the target branches and completing pruning.

6. The pruning method of a fruit tree pruning robot according to claim 5, characterized in that: The fusion method of step S1 is specifically as follows: Step S11, initializing relevant parameters of inertial navigation and visual navigation; Step S12: First, obtain the current position O1 of the pruning robot: And attitude angle ,in, Indicates the robot's rotation direction (i.e. x The angle of rotation of the axis, i.e. the roll angle, represents the rotation of the robot around its lateral axis (i.e. y The angle of rotation of the axis, i.e. the pitch angle, The robot rotates around the vertical axis (i.e. z The angle of rotation of the axis, i.e. the yaw angle; Then, the lever arm compensation position O2 is obtained: , L x , L y , L z They represent the three directional components of the lever arm, specifically: Where: L Indicates the actual length of the lever arm; Finally, the velocity vector after lever arm compensation is obtained V =( V x ,V y ,V z ), specifically: Where: v x , v y , v z They represent the velocity vector of the pruning robot in the coordinate system. x axis, y axis, z The weight of the axis; Step S13: pre-process the data measured by the inertial sensor (i.e., IMU data) and convert the attitude angle in step S12 into a rotation matrix R : Then, convert the acceleration: Where: a1 represents the converted acceleration; a2 represents the measured acceleration; The velocity is updated through acceleration, and the updated velocity is used to update the position, specifically: Where: v 2 represents the speed at the next moment, v 1 represents the speed at the previous moment, Indicates the time interval between two segments; h 2 indicates the position at the next moment, h 1 indicates the position at the previous moment; Step S14, predicting the error information after navigation fusion, and updating the predicted value with data information and state feedback to correct the aforementioned position, speed and other parameters, compensate for the navigation fusion error and improve navigation accuracy.

7. The pruning method of a fruit tree pruning robot according to claim 5, characterized in that: The error information after predicting navigation fusion in step S14 is specifically: First, define the error state vector: Where: represents the position error, represents the speed error, Indicates acceleration deviation; The error feedback state Kalman filter model is used to predict the propagation of the error: Where: express k The error state estimate at time t, express k Error state estimation at time -1; express k The state transition matrix at time , which describes the system state from k -1 time has arrived k The changing relationship of moments; express k The prior covariance matrix at time , express k -1 posterior covariance matrix; express k The process noise covariance matrix at time t, which represents the influence of system model error and process noise; Then, the actual observations are integrated to update the state. Calculate the Kalman gain: Where: K k express k The Kalman gain at the time, which determines the weight of the measurement value in the state update; H k express k The measurement matrix at the moment, which describes the relationship between the state vector and the measurement vector; R k express k The measurement noise covariance matrix at time instant, which represents the impact of measurement error; Update the state estimate and covariance matrix: Where: Z k express k The measurement vector of the moment; I represents the identity matrix; Finally, after obtaining the posterior state Finally, the error is fed back to the navigation system to correct the navigation parameters: Where: , , Respectively k Position, velocity, and acceleration correction values ​​at the moment; , , Respectively k The actual measured values ​​of position, velocity and acceleration at the moment.

8. The pruning method of a fruit tree pruning robot according to claim 5, characterized in that: The step S3 is specifically as follows: Step S31, scanning the target branch at different distances and angles at a distant position to obtain sparse point cloud data, and generating dense point cloud data through multi-view stereo vision; Step S32: for each image, calculate its depth map by disparity; Step S33: fusing the depth map of each view into a unified three-dimensional point cloud through reprojection technology; Step S34: Fit a local plane, project each point onto the plane, and for each point, select its U The least square method is used to reduce noise among neighboring points. Step S35, using voxel grid downsampling to divide the point cloud into a uniform voxel grid, and retaining the voxel center as a representative point in each voxel, thereby retaining the connection between the overall and local features of the fruit tree branches.

9. The pruning method of a fruit tree pruning robot according to claim 5, characterized in that: In step S5, the specific steps of obtaining candidate pruning points through skeleton length, branch angle, and lesion information are as follows: Step S51, obtaining endpoints and branch points in the skeleton feature image. If there is only one foreground pixel in the eight neighborhood pixels of a pixel point, the pixel point is the endpoint, and the skeleton length is obtained through the Euclidean distance between the endpoint and each branch point; A length threshold is preset. If the obtained skeleton length is less than the length threshold, the branch is considered to be poorly grown, and the corresponding branch point is a candidate cutting point; Step S52: For each branch point Q f ( x f , y f ), through the adjacent connecting point A ( x a , y a ) and B ( x b , y b ) to obtain the corresponding branch angle : A preset angle threshold is set. If the branch angle is less than the angle threshold, it indicates that the branches are too parallel and the corresponding branch point is a candidate cutting point. Step S53: using a deep learning model to judge the death or disease of branches by color, and identifying the dead or diseased branch points as candidate cutting points; Step S54, calculating the minimum distance between two branch points by Euclidean distance, and presetting a distance threshold. If the minimum distance between two branch points is less than the distance threshold, it is considered to be an over-dense point, and the over-dense point is used as a candidate cutting point.

Images