A device and method for intelligent pruning of litchi trees based on three-dimensional point cloud

Abstract

The application provides a device and method for intelligent pruning of litchi trees based on three-dimensional point clouds, and belongs to the technical field of litchi tree pruning. The device comprises a mechanical pruning shear, a mechanical arm, a mechanical arm lifting device, a power supply device, a device control and data processing center, a cloud platform camera, a depth camera I, a depth camera II, a depth camera III, a device mounting platform, a platform balancing device, a device carrying vehicle, and a post-pruning branch and leaf garbage bin. The cloud platform camera identifies the road conditions, plans the car path, identifies the litchi trees, then uses the car plate balancing method to balance the device mounting platform, uses two monocular depth cameras and one binocular depth camera to achieve accurate point cloud processing, uses the tree branch breakpoint repair method after three-dimensional modeling, and realizes the collaborative work of multiple mechanical arms, which can simultaneously prune two branches and speeds up the pruning speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of litchi tree pruning, and in particular to a device and method for realizing intelligent litchi tree pruning based on three-dimensional point clouds. BACKGROUND

[0002] In the growth process of litchi, pruning of litchi is a very important link. How to prune the branches of litchi trees during the budding period in spring and autumn and after fruit picking plays a very important role in the vigorous growth of litchi trees and the growth of high-quality litchi. In most areas, litchi tree pruning is still done manually. When a large area of litchi orchard needs to be pruned, a large amount of manpower needs to be invested in a short period of time in order to complete the pruning of litchi trees within the best pruning period, which may result in a shortage of pruning personnel or a large amount of labor costs, thereby affecting the pruning work and causing a sharp increase in pruning costs. Therefore, it is necessary to design a device and method for realizing intelligent litchi tree pruning based on three-dimensional point clouds. SUMMARY

[0003] The present application aims to provide a device and method for realizing intelligent litchi tree pruning based on three-dimensional point clouds, which solves the technical problems of low efficiency and large amount of manual labor in the existing manual pruning of litchi trees.

[0004] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0005] A device for realizing intelligent litchi tree pruning based on three-dimensional point clouds, comprising a mechanical pruning shears, a mechanical arm, a mechanical arm lifting device, a power supply device, a device control and data processing center, a pan-tilt camera, a depth camera I, a depth camera II, a depth camera III, a device installation platform, a platform balancing device, a device carrying vehicle and a pruning waste bin, the pruning waste bin is arranged below the device installation platform, the platform balancing device is arranged on the device carrying vehicle, the device installation platform is arranged at the top end of the platform balancing device, the pan-tilt camera is arranged at the center of the front of the device installation platform, the data processing and device control center, the power supply device and the mechanical arm lifting device are all arranged on the device installation platform, the mechanical pruning shears are arranged on the mechanical arm, the mechanical arm is arranged on the mechanical arm lifting device, the pan-tilt camera, the depth camera I, the depth camera II and the depth camera III are all arranged on the device installation platform, the pan-tilt camera, the depth camera I, the depth camera II and the depth camera III are all connected with the device control and data processing center, and the mechanical pruning shears, the mechanical arm, the mechanical arm lifting device and the platform balancing device are all connected with the device control and data processing center.

[0006] Further, the pruning waste bin is arranged between the wheels of the device installation platform, an infrared sensor is arranged in the waste bin for sensing the storage amount of branches in the waste bin, and the number of mechanical pruning shears, mechanical arms and mechanical arm lifting devices is two.

[0007] Further, the outer part of the mechanical pruning shears is set as a mechanical hand structure for grabbing the branches, and a retractable blade is arranged inside the mechanical hand structure.

[0008] Further, in order to keep the equipment mounting platform in a relatively horizontal state, the tilt angle of the equipment mounting platform is obtained through the coordinate angle difference between the world coordinates and the gimbal camera coordinates arranged in the equipment control and data processing center, and corresponding control instructions are generated, and under the control of the equipment control and data processing center, different amplitudes of the platform balancing device are realized to keep the equipment mounting platform relatively horizontal.

[0009] A pruning method of a litchi tree intelligent pruning device based on three-dimensional point clouds, the litchi tree intelligent pruning device is started, an intelligent pruning device automatic traveling method is adopted, image information in front of the device and on both sides is obtained through a gimbal camera, after processing by an equipment control and data processing center, path planning is performed, the equipment control center sends control instructions to control the vehicle to travel to a fixed distance below the litchi tree, then a trolley plate balancing method is adopted to adjust the platform balance, so that the equipment mounting platform is in a relatively horizontal state, which is beneficial to subsequent point cloud collection;

[0010] After the equipment mounting platform is balanced, the depth camera is calibrated, point clouds are collected from the front and the opposite side by the depth camera I, the depth camera II and the depth camera III, after the point clouds are completed, the discrete points and noise are removed in the equipment control and data processing center, three-dimensional reconstruction is performed, and the model is completed after the reconstruction by a breakpoint repair method;

[0011] The point clouds of the two angles are registered and semantically segmented, the first-order branches and mother branches are divided out, and the branches with the required pruning length and the redundant branches are selected by a preferred branch screening method;

[0012] The point clouds of each branch to be pruned are extracted in the equipment control and data processing center, and the pruning point determination method is adopted to determine the pruning point of each branch;

[0013] Finally, a top-down pruning method is adopted, the distance between the branches to be pruned and the mechanical pruning shears is sorted from small to large by the equipment control and data processing center, the path is planned from the smallest distance, then the equipment control and data processing center sends instructions to control the mechanical arm and the mechanical pruning shears to prune, after pruning, the grabbed branches are put into the branch storage garbage can, and the whole pruning operation is completed.

[0014] Further, the specific process of the intelligent pruning device automatic traveling is as follows:

[0015] Step 1.1: The PTZ camera acquires RGB images of the front and left and right sides of the intelligent pruning device for lychee trees;

[0016] Step 1.2: The equipment control and data processing center analyzes and processes the RGB image information, identifies and determines the location of the lychee tree, and plans a path to the bottom of the lychee tree based on the location of the lychee tree. The intelligent pruning device of the lychee tree moves forward quantitatively according to the planned path and identifies and detects whether there are other obstacles on the path in real time.

[0017] Step 1.3: Analyze the path information ahead based on the RGB image and divide the path planning into three scenarios: Scenario 1, no obstacles ahead; Scenario 2, obstacles ahead, but can be bypassed by going around to the left or right; Scenario 3, obstacles ahead, and it is impossible to avoid them.

[0018] Step 1.4: When it is case 1, the specific processing steps include the following:

[0019] Step 1.4.1: The data processing center of the equipment control and data processing center sends a forward command to the equipment control center;

[0020] Step 1.4.2: The equipment control center issues instructions to control the trolley to operate, realizing the quantitative forward movement and left and right turning of the intelligent pruning device;

[0021] Step 1.4.3: The intelligent pruning device moves forward in a quantitative manner. Each time it moves forward, it obtains the distance between its current position and the target lychee tree through the pan-tilt camera, and then continues to move forward along the planned path.

[0022] Step 1.4.4: Arrive at the lychee tree, issue the trolley board balancing method and subsequent pruning instructions. If the branches do not need pruning, re-identify new lychee trees until all lychee trees have no need for pruning.

[0023] Step 1.5: When it is case 2, the specific processing steps include the following:

[0024] Step 1.5.1: The intelligent pruning device plans a detour path to the left or right of the obstacle ahead;

[0025] Step 1.5.2: Use the PTZ camera to turn left or right to determine if there are other obstacles on the detour path. If there are no obstacles, plan the detour route and proceed with a fixed amount of movement.

[0026] Step 1.6: When it is case 3, the specific processing steps include the following:

[0027] Through the analysis of the two sides of the pan-tilt camera, if there is an obstacle, analyze whether it will affect the detour route, if not, plan the path and drive forward quantitatively, if it affects, give up the target lychee tree, select a new lychee tree target, the priority of the left side is greater than that of the right side, if there is no lychee tree in front, rotate the pan-tilt camera to identify from the left side, if there is no lychee tree on the left side, rotate the pan-tilt camera to the right side to identify the lychee tree;

[0028] Step 1.7: If there are new lychee tree targets on the left and right sides, repeat step 1.2, if there are no new lychee tree targets in front, left and right, stop the car and send a help instruction to the worker through the data processing center;

[0029] The car plate balancing method in step 1.4.4, the specific process is as follows:

[0030] Step 1.4.4.1: The platform balancing device receives the balance instruction;

[0031] Step 1.4.4.2: The data processing center manually specifies the world coordinates, i.e. the coordinates Xw, Yw, Zw on the ground, and specifies the positive direction of the three coordinate axes, the camera coordinates are Xc, Yc, Zc, since the camera coordinates exist on the slope, there is an angle difference between the camera coordinates and the world coordinates, the X, Y, Z axis angle difference is set as ψ, θ and φ, the length of the device installation platform is set as T, and the width is set as W;

[0032] Step 1.4.4.3: Calculate the included angle through two three-dimensional coordinate axes, take a point on the X, Y, Z axis of the camera coordinates and the world coordinates, and the included angle calculation model is: Where X W , Y W , Z W are the three-dimensional coordinate system of the world coordinates, X C , Y C , Z C are the three-dimensional coordinate system of the camera coordinates;

[0033] Step 1.4.4.4: Adjust the extension amplitude L of the platform balancing device, the extension amplitude L calculation model is: La=|Tsinα| or |Wsinα|, where La is the extension length, T and W represent the length and width of the device installation platform, and α represents the included angle between the two coordinates;

[0034] Step 1.4.4.5: The extension of the platform balancing device can be divided into the following four cases: Case 1: When the angle between the camera coordinates and the world coordinates X, Z axis coordinates is α>0, indicating that the front side of the trolley is high, then the platform balancing device III and the platform balancing device IV extend La=|Tsinψ| or |Tsinφ| amplitude, Case 2: When the angle between the camera coordinates and the world coordinates X axis coordinates is α<0, indicating that the rear side of the trolley is high, then the platform balancing device I and the platform balancing device II extend La=|Tsin(-φ)| or |Tsin(-ψ)| amplitude, Case 3: When the angle between the camera coordinates and the world coordinates Y, Z axis coordinates is α>0, indicating that the left side of the trolley is high, then the platform balancing device II and the platform balancing device IV extend La=|Wsinθ| or |Wsinφ| amplitude, Case 4: When the angle between the camera coordinates and the world coordinates Y axis coordinates is α<0, indicating that the right side of the trolley is high, then the platform balancing device I and the platform balancing device III extend La=|Wsin(-θ)| or |Wsin(-φ)| amplitude;

[0035] Step 1.4.4.6: After the extension of the platform balancing device, the angle α between the two coordinates is obtained again, and if α=0, the balancing adjustment is ended.

[0036] Further, the specific process of the breakpoint repair method is as follows:

[0037] Step 2.1: First, calibrate the camera, then obtain the depth image through the depth camera, convert it into point cloud data, and preprocess the collected point cloud to remove noise and discrete points, and reconstruct the point cloud three-dimensionally to form a three-dimensional litchi tree;

[0038] Step 2.2: Through three-dimensional reconstruction, a litchi tree with several breakpoints is obtained, and a breakpoint repair instruction is issued from the data processing center, the point cloud with breakpoints is sliced, the point cloud section near the breakpoint is obtained, and the center coordinates M of all point clouds on the section are calculated, and the calculation formula of the center point is:

[0039]

[0040] Where X i , Y i , Z i are the three-dimensional coordinates of each point cloud, and n is the total number of section point clouds.

[0041] Step 2.3: The point cloud breakpoint can be divided into the following two cases: Case 1, both sides of the point cloud have breakpoints, and Case 2, only one side of the point cloud has a breakpoint;

[0042] Step 2.3.1: When it is Case 1, the center coordinates of the point cloud on both sides are set as [M1, M2] and stored in a set;

[0043] Step 2.3.2: When it is case 2, set the point cloud center coordinates as C1, C2, C3, C4...CN, N is a positive integer;

[0044] Step 2.4: Randomly select a point cloud part with breakpoints on both sides, determine a straight line L1=A1x+B1y+C1z+D1 by the center points on both sides, define the direction of the straight line towards the center point with smaller z value as the repair direction, and determine the center point with smaller z value as the repair point Mf;

[0045] Step 2.5: Draw a circle with Mf as the center, and judge whether there are other breakpoints M[i] or C[i] within a distance of 3 cm;

[0046] Step 2.6: Judge the breakpoints into the following four cases: case 1, there are both breakpoints M[i] and C[i] within the distance range, case 2, there are no breakpoints within the distance range, case 3, there is only one or more breakpoints M[i] within the distance range, and case 4, there is only one or several breakpoints C[i] within the distance range;

[0047] Step 2.6.1: When it is case 1, connect Mf with multiple breakpoints, and form multiple straight lines L2, L3...LN by two points, N is a positive integer, and calculate the included angle a[i] between the straight lines and the straight line L1, the included angle formula is:

[0048]

[0049] Where A1, B1, C1 are the parameters of the straight line L1, and A i B i C i are the parameters of the straight line Li.

[0050] Since the branches are a relatively smooth curve, manually eliminate the breakpoints with an included angle greater than 20°, so the remaining included angle aS between the breakpoint straight line and L1 is <= 20°, then select the breakpoint straight line with the smallest included angle as the connection point, if there are the same number of included angles, calculate the distance S of the breakpoint M[i] or C[i] to Mf; the three-dimensional point cloud distance formula is:

[0051]

[0052] Where X1, Y1, Z1 are the three-dimensional coordinates of the repair point Mf, and X2, Y2, Z2 are the three-dimensional coordinates of the point M[i] or C[i]. Select the breakpoint with the smallest included angle or the same distance as the new connection point, and perform point cloud repair along the new connection point straight line, the new connection point has the following two cases: case 1, the new connection is M[i], and case 2, the new connection is C[i];

[0053] Step 2.6.1.1: If it is case 1, determine the new connection point as M[i], connect the two breakpoints in the M[i] set and establish the straight line as the new L1, and determine the point with smaller z value in the M[i] set as the new repair point Mf and the repair direction of L1, and execute step 2.5 again;

[0054] Step 2.6.1.2: If it is case 2, determine the new connection as C[i], and then perform point cloud repair along the straight line of the connection point, because the new connection point only has one breakpoint, which means that the other end has been connected to the trunk, i.e. the repaired Mf is connected to the trunk branch, so it represents the completion of the repair;

[0055] Step 2.6.2: If it is case 2, when there is no breakpoint within the distance, repair the point cloud by 3 cm in the repair direction, and there are two cases when extending: case 1, the extended end point does not intersect with any trunk or branch point cloud, case 2, the point cloud intersects with the trunk or branch point cloud during the extension;

[0056] Step 2.6.2.1: When it is case 1, find the new point in the cross section of the extended end point, and establish it as the new repair point Mf, the repair direction remains unchanged, and execute step 2.5 again;

[0057] Step 2.6.2.2: When it is case 2: if the point cloud intersects with the branch or trunk point cloud during the repair, stop the repair, i.e. the repair is completed;

[0058] Step 2.6.3: When it is case 3, perform step 2.6.1 and step 2.6.1.1;

[0059] Step 2.6.4: When it is case 4, perform step 2.6.1 and step 2.6.1.2;

[0060] Step 2.7: Whenever a branch is repaired, repeat the operation of step 2.4 until all breakpoints are repaired, i.e. the repair of all point cloud breakpoints is completed, and the data processing center executes subsequent point cloud pairing and semantic segmentation instructions.

[0061] Further, the specific process of the optimal branch selection method is as follows:

[0062] Step 3.1: After the semantic segmentation, the branch and the first branch are segmented, and the data processing center sets the parameter count to calculate the first branch on each branch;

[0063] Step 3.2: There are at most 4 first branches on a branch, and the size of the first branch is specified to be between 40-50 cm, and if there are too many branches, it will affect the development of the litchi tree and the yield of litchi;

[0064] Step 3.3: Slice the three-dimensional point cloud to extract the length of the first branch data[i], i is a positive integer, and put them into the set Rm, then Rm = {data1, data2, data3, data4...}, first put the data[i] > 50cm first branch into the pruning set R, and the data[i] < 40cm first branch into the pruning set R2;

[0065] Step 3.4: After the first round of screening, calculate the number of branches in the set Rm, if count < 4, the mother branch is screened, if count > 4, the optimal selection method is needed;

[0066] Step 3.5: After the long and short branches are removed, the remaining branches are between 40-50cm, but since a maximum of 4 are retained, pruning is needed, since the optimal length of the first branch is 40-50cm, the average value is 45cm, data[i] in Rm is operated data = |data-45|, which selects the branches with large difference from the average value for pruning, and data[i] in Rm is updated, then the set Rm is sorted from large to small, and the first count-4 branches are extracted and put into the set R2;

[0067] Step 3.6: After the selection of a mother branch is completed, steps 3.3 and 3.4 are repeated to select all first branches on the litchi tree and put them into the set R and the set R2, after the selection is completed, the data processing center issues the next instruction to extract the branches in the set R and the set R2 for pruning point selection.

[0068] Further, the specific process of pruning point determination is as follows:

[0069] Step 4.1: Extract the parameters of the branches in the set R and R2 through the data processing center, and fit the curve through the least square method, since the branches are curved, select the bending point as the pruning point for pruning, and automatically identify the pruning point;

[0070] Step 4.2: Select a branch in R and extract the point cloud coordinates near the root, bending point and top of the branch, train how to identify the bending point, root and top point through deep learning, then do point cloud slicing on the bending point to get the cross-sectional point cloud, and calculate the cross-sectional point cloud center coordinate F through the point cloud coordinates on the cross section:

[0071]

[0072] Where X i ,Y iZ i is the three-dimensional coordinate of the point cloud on each cross section, n is the number of point clouds of the current cross section;

[0073] Step 4.3: Assign a weight to each center point F[i] through the data processing center, F0=(x0+a0, y0+b0, z0+c0), F1=(x1+a1, y1+b1, z1+c1), F[i]=(x[i]+ai, y[i]+bi, z[i]+ci)......, and a distance optimization problem is constructed by using the least square method, that is, where di is the distance from the ith point cloud of the current cross section to the center point F of the current cross section, d is the average distance from the center point F of the current cross section to each point cloud of the current cross section, and a[i], b[i], and c[i] are solved to find the optimal point closest to all point clouds on the cross section, that is, F[i] is no longer the center point of the cross section, but the optimal point of the cross section;

[0074] Step 4.4: The optimal points F[i] of the root, bending point, and top end point are determined by the least square method, and each optimal point is connected to fit the original linear shape of the branch;

[0075] Step 4.5: After obtaining the curve, the target of the pruning point is placed on the bending point, and since the optimal point cloud coordinates are known, the length len[i] of the straight line connecting each two optimal points can be calculated, and the formula for calculating len[i] is:

[0076]

[0077] where X1, Y1, Z1, and X2, Y2, Z2 are the three-dimensional coordinates of two adjacent optimal points.

[0078] From the root optimal point, the length Ls of each straight line is accumulated, and the calculation formula is:

[0079]

[0080] where n is the nth branch;

[0081] Step 4.6: Extract the branches from the screened branch set R, and accumulate each segment. When Ls>50, stop accumulating, and determine the nth-1 bending point as the pruning point;

[0082] Step 4.7: For the branches in set R2, because the first-level branches on the parent branch are greater than the set value and the diseased branches are less than 40 cm, the whole root needs to be cut off, so the root optimal points of all branches in set R2 are directly selected as the pruning points, and all branches with determined pruning points are put into set R3;

[0083] Step 4.8: determine the pruning point of the branches in set R and set R2, repeat step 4.2 and step 4.6, 4.7, that is, complete the determination of the pruning point, and then the data processing center issues an instruction to calculate the distance from the mechanical arm to the pruning point of the branches in set R3, and plans the pruning from bottom to top.

[0084] Further, the specific process of the bottom-to-top pruning method is as follows:

[0085] Step 5.1: after the distance from the pruning point to the mechanical arm is calculated, the distance is sorted from small to large, and the pruning starts from the lowest branch, which can achieve the most efficient pruning and avoid many unnecessary path planning;

[0086] Step 5.2: set the mechanical arm coordinates and define a set Q for each branch in set R3, which contains a number x[i] and a distance d[i], so Q[i] = {x[i], d[i]};

[0087] Step 5.3: the conversion relationship between the world coordinates (Xw, Yw, Zw) and the camera coordinates (Xc, Yc, Zc) can be expressed as:

[0088]

[0089] where R is the rotation matrix from the world coordinate system to the camera coordinate system, and the specific expression of R is:

[0090]

[0091] where θ, φ are the rotation angles of the X, Y, Z axes of the world coordinates to the camera coordinates. T = [t1t2 t3] T is the translation vector from the world coordinate system to the camera coordinate system;

[0092] Step 5.4: then convert between camera coordinates and image coordinates g(x, y):

[0093]

[0094] where f is the focal length of the lens;

[0095] Step 5.5: through the conversion of the image g(x, y) and the pixel coordinates u(xu, yu), the image coordinate origin in the pixel mapping coordinates is g'(u, v):

[0096]

[0097] where dx and dy are the corresponding physical sizes of a single pixel point in the x-axis and y-axis directions;

[0098] Step 5.6: The distance of the mechanical arm to the pruning point can be calculated by coordinate transformation through the position of the mechanical arm coordinate in the world coordinate, and the distance can be calculated by three-dimensional coordinates Where X1, Y1, Z1 are the coordinates of the mechanical arm, and X i ,Y i ,Z i are the coordinates of the pruning point. And put into the set R4. After determining all the pruning distances d[i], the set R4 is quickly sorted from small to large, and after sorting, according to d[i] in the set Q[i] corresponding to the number x[i] to determine which branch, through the data processing center to determine the branch pruning point, send the pruning command to the equipment control center, the equipment control center sends the command to the mechanical arm lifting platform, the mechanical arm and the mechanical pruning shears to move to the appropriate position for pruning to complete the pruning from bottom to top.

[0099] The application has the following beneficial effects due to the adoption of the above technical scheme:

[0100] The application adopts a T-shaped track belt, realizes driving on steep slopes and uneven mountain roads, identifies road conditions through a gimbal camera, plans a trolley path, identifies litchi trees, then adopts a trolley plate balancing method to balance the equipment installation platform, and uses two monocular depth cameras and a binocular depth camera to realize accurate point cloud processing. After three-dimensional modeling, the tree branch breakpoint repair method is used to realize three-dimensional completion of the part of the whole litchi tree where the point cloud collection is incomplete, which is convenient for subsequent accurate judgment of the length and number of branches. After registration and semantic segmentation of the litchi tree, the optimal branch screening method is used to realize accurate automatic screening of the branches. After extracting the branch data, the pruning point confirmation method is used to realize intelligent branch pruning point selection. Finally, the bottom-up pruning method is used to realize the planning of the pruning path of the mechanical arm from the bottom branches of the tree, avoiding the need to bypass too many messy leaves when pruning the top branches. The combination of the mechanical hand and the blade is used when pruning the branches to realize fixed pruning point cutting. The residual branches are collected into the garbage can after pruning. The automatic point cloud of the litchi branches is realized by using machine vision technology, and the PLC automatic control technology is used to realize the cooperative work of multiple mechanical arms, which can simultaneously prune two branches, thereby speeding up the pruning speed. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 is a schematic view of the left side structure of the device of the application;

[0102] Figure 2 is a schematic view of the right side structure of the device of the application;

[0103] Figure 3 is a schematic view of the lower half structure of the device of the application;

[0104] Figure 4 is a schematic view of the structure of the pruning shears of the device of the present application;

[0105] Figure 5 is a flow chart of the balancing method of the trolley plate of the present application

[0106] Figure 6 is a flow chart of the method of repairing the branch break point of the present application;

[0107] Figure 7 is a flow chart of the method of screening the preferred branches of the present application;

[0108] Figure 8 is a flow chart of the method of determining the pruning point of the present application;

[0109] Figure 9 is a flow chart of the method of pruning from bottom to top of the present application;

[0110] Figure 10 is a flow chart of the automatic advancing method of the intelligent pruning device of the present application.

[0111] Figure Label: 107, mechanical pruning shears I; 108, mechanical pruning shears II; 15, mechanical arm I; 16, mechanical arm II; 105, mechanical arm lifting device I; 106, mechanical arm lifting device II; 108, power supply device; 110, equipment control center and data processing center; 101, pan-tilt camera; 104, depth camera I; 103, depth camera II; 109, depth camera III; 8, equipment installation platform; 112, platform balancing device I; 113, platform balancing device II; 111, platform balancing device III; 114, platform balancing device IV; 1, device carrier vehicle track; 115, device carrier vehicle rotating wheel; 102, branch storage garbage can. DETAILED DESCRIPTION

[0112] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the following preferred embodiments are given with reference to the accompanying drawings, and the present application is further described in detail. However, it should be noted that many details in the description are only for the reader to have a thorough understanding of one or more aspects of the present application, and the aspects of the present application can be realized without these specific details.

[0113] As Figures 1-3As shown, a three-dimensional point cloud-based intelligent litchi tree pruning device includes a mechanical pruning shear 107, a mechanical pruning shear 108, a mechanical arm, a mechanical arm lifting device, a power supply device 108, a device control center, a data processing center, a pan-tilt camera 101, a depth camera I 104, a depth camera II 103, a depth camera III 109, a device mounting platform 8, a platform balancing device, a device carrying vehicle, and a post-pruning branch and leaf garbage bin. The branch and leaf garbage bin is below the platform (between the two wheels), the platform balancing device and the platform are arranged on the device carrying vehicle, the device mounting platform 8 is arranged at the top end of the platform balancing device, the pan-tilt camera 101 is located at the front center of the device mounting platform 8, the data processing center, the device control center, the power supply device 108, and the mechanical arm lifting device are all arranged on the device mounting platform 8, the mechanical pruning shear is connected with the mechanical arm and the mechanical arm lifting device, the pan-tilt camera 101, the depth camera I 104, the depth camera II 103, and the depth camera III 109 are all arranged on the device mounting platform 8, and the pan-tilt camera 101, the depth camera I 104, the depth camera II 103, the depth camera III 109, and the device control center are all connected with the data processing center. The device control center and the data processing center 110 include the device control center and the data processing center.

[0114] Further, the pruning garbage bin is below the platform and located between the two wheels. An infrared sensor is arranged in the garbage bin for sensing the storage amount of branches in the garbage bin. The number of the mechanical pruning shears, the mechanical arm, and the mechanical arm lifting device is two.

[0115] The pan-tilt camera 101 is installed in front of the pruning device and is responsible for image acquisition of the front and both sides of the pruning trolley. The collected image information is processed and analyzed by the data processing center, and corresponding control instructions are generated. The device control center controls the trolley to move forward, backward, turn left and right, and the like, so as to realize the functions of quantitative forward movement and backward movement of the intelligent litchi tree pruning device, and move the intelligent litchi tree pruning device to a suitable position to prune the branches of the litchi tree.

[0116] The Kinect DK depth camera I 104 collects depth image information of the litchi tree on the left side of the intelligent pruning device, the Kinect2 DK depth camera II 103 collects depth image information of the litchi tree on the right side of the intelligent pruning device, and the ZED 2i depth camera collects depth image information of the overall litchi tree from the middle. Secondly, the real distance from the pruning point to the camera is obtained through the depth camera, which is convenient for path planning according to the distance d[i] between the mechanical arm and the pruning point.

[0117] The mechanical pruning shears I 107 and the mechanical pruning shears II 108 are respectively installed at the upper ends of the mechanical arm I 115 and the mechanical arm II 116, and the mechanical pruning shears mainly adopt a combination of mechanical pincers and blades, so that after the mechanical pruning shears move to a suitable position, the tree branches can be grabbed and pruning can be realized. Meanwhile, the mechanical arm I 115 and the mechanical arm II 116 are respectively installed at the upper ends of the mechanical arm lifting device I 105 and the mechanical arm lifting device II 106, and under the control of the equipment control center, the extension amplitude of the mechanical arm lifting device I 105 and the mechanical arm lifting device II 106 can be adjusted arbitrarily, the height of the mechanical arm is adjusted, the relative stability of the height difference between the mechanical arm and the picking target is maintained, and the pruning work at different heights is better adapted. And equipped with double pruning shears, the work efficiency can be better improved.

[0118] The equipment installation platform 8 is installed at the upper ends of the platform balancing device I 112, the platform balancing device II 113, the platform balancing device III 111 and the platform balancing device IV 114, and the above platform is jointly constituted by the data processing center, the equipment control center, the mechanical arm lifting device, the pan-tilt camera 101, the depth camera and the branch storage garbage can and other equipment to form an intelligent pruning trolley, which is convenient for the function realization of the intelligent pruning device of the litchi tree.

[0119] The platform balancing device I 112, the platform balancing device II 113, the platform balancing device III 111 and the platform balancing device IV 114 are respectively installed below the four corners of the equipment installation platform 8, and under the control of the equipment control center, the four balancing devices can be extended at different amplitudes, so that the equipment installation platform 8 maintains the same horizontal height.

[0120] The power supply device 108 provides stable electric energy for the mechanical pruning shears I 107, the mechanical pruning shears II 108, the mechanical arm I 115, the mechanical arm II 116, the mechanical arm lifting device I 105, the mechanical arm lifting device II 106, the equipment control center, the data processing center, the pan-tilt camera 101, the Kinect DK depth camera I 104, the Kinect DK depth camera II 103, the ZED 2i depth camera, the equipment installation platform 8, the platform balancing device I 112, the platform balancing device II 113, the platform balancing device III 111 and the platform balancing device IV 114.

[0121] The data processing center, the Kinect DK depth camera I 104, the Kinect DK depth camera II 103, the ZED 2i depth camera and the pan-tilt camera 101 are mainly responsible for processing and analyzing various image information, generating point clouds, preprocessing, modeling, repairing, screening, determining pruning points, path planning and the like;

[0122] The device control center controls the working state of the mechanical pruning shears, the mechanical arm, the mechanical arm lifting device, the platform balancing device, the pan-tilt camera 101 and the depth camera under the control of the control instructions.

[0123] The branch storage garbage can is installed below the device installation platform 8. After the branches are pruned, the mechanical pruning shears put the branches into the garbage can, thereby reducing the labor and time required for manual cleaning.

[0124] In order to improve the flexibility and stability of the pruning trolley, the device comprises a T-shaped track, a track wheel and a driving device. Two DC motors and corresponding transmission structures are arranged in the driving device. The power supply device 108 provides power, and the power supply is controlled by the device control center. The transmission structure transmits the power generated by the motor to the wheel to drive the rolling of the track, thereby realizing the movement of the litchi tree intelligent pruning device.

[0125] In order to facilitate the connection between devices, the connecting device realizes the fixed connection between devices by using a fixed pin.

[0126] In order to expand the range of litchi tree identification and obstacle avoidance planning, the device uses a pan-tilt camera 101, which can realize the shooting of the front environment, and can also rotate the camera to realize the shooting of the left and right environments.

[0127] In order to facilitate the accidental disposal of branches after litchi tree pruning, the pruning trolley is provided with a garbage can below the platform for collecting branches. The labor, labor time and labor cost of manual ground cleaning in the later stage are reduced.

[0128] In order to improve the cooperation of pruning and grabbing branches, the mechanical pruning shears are modified. The outside of the mechanical pruning shears is grabbed by the mechanical hand, and the inside of the mechanical pruning shears has a retractable blade. The inside blade is used to cut the branches after fixing the pruning point. The complex steps of first grabbing the branches by the mechanical hand and then pruning are omitted, and the installation of multiple mechanical arms is also omitted.

[0129] In order to improve the cooperation between each device, the device control center controls the working state of the device intelligent car, the platform balancing device, the camera shooting, the mechanical arm lifting device, the mechanical arm and the mechanical pruning shears under the control of the data processing center.

[0130] In order to adapt to the pruning work of litchi trees of different heights, the platform balancing device can realize lifting within a certain range, thereby realizing the lifting of the device platform at different heights.

[0131] In order to enhance the terrain adaptability of the pruning device, keep the equipment installation platform 8 in a relatively horizontal state, obtain the inclination angle of the platform through the coordinate angle difference between the world coordinates and the gimbal camera coordinates built in the data processing center, and generate corresponding control instructions, under the control of the equipment control center, realize the stretching and contraction of the platform balancing device I 112, II, III, IV with different amplitudes, and then keep the relative level of the equipment installation platform 8.

[0132] In order to adapt to the height and position changes of litchi trees and target branches in the actual pruning environment, the mechanical arm lifting device can realize a certain amplitude of lifting under the control of the equipment control center, so that the mechanical pruning shears are at a more appropriate working height.

[0133] In order to adapt to the pruning task of litchi trees with different heights, angles and directions, the mechanical arm can realize three degrees of freedom of motion posture change and multi-angle rotation of the pruning shears.

[0134] In order to better complete the motion path planning of the litchi tree intelligent pruning trolley, a gimbal camera 101 is adopted, and the camera has a rotating function, which can obtain the environmental information in front of and left and right of the picking device under the control of the equipment control center.

[0135] In order to improve the working efficiency of pruning and complete pruning within the best pruning period, the pruning device adopts double mechanical pruning shears, which can complete pruning more quickly and efficiently. The double mechanical pruning shears realize precise pruning from bottom to top under the control of the equipment control center.

[0136] In order to obtain the point cloud information of litchi trees more accurately and efficiently, and speed up the motion path planning of the mechanical arm, a ZED 2i binocular depth camera and two Kinect DK monocular depth cameras are adopted to obtain more complete three-dimensional point clouds of each litchi tree from both sides of the litchi tree. After processing and analysis by the data processing center, the motion path planning of the mechanical pruning shears is completed.

[0137] The mechanical arm lifting device adopts a servo electric cylinder, the platform balancing device adopts a servo electric cylinder, and the power supply device 108 adopts a detachable rechargeable battery. The equipment control center adopts an intelligent stm32 controller. The data processing center adopts a small computer.

[0138] The specific process of the pruning method is as follows:

[0139] The litchi tree intelligent pruning device is started, and an intelligent pruning device automatic traveling method is adopted. Image information in front of and on left and right sides of the device is acquired by the pan-tilt camera 101. After processing by a data processing center, path planning is performed. The device control center sends a control instruction to control the intelligent trolley to travel to a position 0.5 m away from the bottom of the litchi tree. Then, the trolley plate balancing method is adopted to adjust the platform balance, so that the equipment installation platform 8 is in a relatively horizontal state, which is beneficial to subsequent point cloud collection.

[0140] After the platform balance, the depth cameras are calibrated. The point cloud is collected by the Kinect DK depth camera I 104, the Kinect DK depth camera II 103 and the ZED 2i depth camera from the front and the opposite side at two different angles. After the point cloud is completed, the discrete points and noise are removed in the data processing center, and three-dimensional reconstruction is performed. After reconstruction, the breakpoint repair method is adopted to complete the model.

[0141] The point clouds at two angles are registered and semantically segmented to divide out the first-order branches and mother branches. The required pruning length branches and redundant branches are selected by the optimal branch selection method.

[0142] The point cloud of each required pruning branch is extracted in the data center, and the pruning point determination method is adopted to determine the pruning point of each branch.

[0143] Finally, the bottom-up pruning method is adopted. The distance between the required pruning branch and the mechanical pruning shears is sorted from small to large in the data processing center. The path is planned from the smallest distance, and then the device control center sends an instruction to control the mechanical arm and the mechanical pruning shears to prune. After pruning, the grabbed branches are put into the branch storage garbage can, and the whole pruning operation is completed.

[0144] The above pruning process can be operated by two mechanical arms synchronously. The two mechanical arms can work independently under the control of the device control center without affecting each other.

[0145] The device and method for realizing litchi tree intelligent pruning based on three-dimensional point cloud, the method comprising the following steps:

[0146] Step 1: The specific process of the intelligent pruning device automatic traveling method is as shown in Figure 10

[0147] Step 1.1: The pan-tilt camera 101 acquires the RGB images in front of and on left and right sides of the litchi tree intelligent pruning device.

[0148] ​Step 1.2: The data processing center analyzes and processes the RGB image information, identifies and determines the location of the litchi tree, plans a path under the litchi tree according to the location of the litchi tree, and the intelligent pruning device of the litchi tree quantitatively proceeds according to the planned path and detects in real time whether there are other obstacles on the path of travel.

[0149] Step 1.3: According to the path information in front of the RGB image, the path planning is divided into two situations: situation 1, no obstacles in front of the path; situation 2, there are obstacles in front of the path, but it can pass through the left or right side; situation 3, there are obstacles in front of the path and it cannot avoid the obstacles.

[0150] Step 1.4: When it is situation 1, the specific processing steps include the following:

[0151] Step 1.4.1: The data processing center issues a forward command to the device control center.

[0152] Step 1.4.2: The device control center issues instructions to control the car to run, realizing the quantitative forward movement and left and right turning actions of the intelligent pruning device.

[0153] Step 1.4.3: The forward distance of the intelligent pruning device is mainly in the form of quantitative forward movement. The current distance from the target litchi tree is obtained through the pan-tilt camera 101 each time it moves forward, and then it continues to move forward towards the planned path.

[0154] Step 1.4.4: When it comes to the litchi tree, the car plate balancing method and subsequent pruning instructions are issued. If the branches do not need to be pruned, a new litchi tree is identified. Until all litchi trees have no pruning needs.

[0155] Step 1.5: When it is situation 2, the specific processing steps include the following:

[0156] Step 1.5.1: The intelligent pruning device plans a detour path on the left or right side of the obstacle in front.

[0157] Step 1.5.2: The pan-tilt camera is turned left or right to determine whether there are other obstacles on the detour path. If there are no obstacles, the detour route is planned and the quantitative forward movement is performed.

[0158] Step 1.6: When it is situation 3, the specific processing steps include the following:

[0159] As analyzed by the pan-tilt camera 101 on both sides, if there is an obstacle, analyze whether it will affect the detour route. If not, plan the path and drive forward quantitatively. If it does, give up the target lychee tree and select a new lychee tree target. When selecting a new target, the priority on the left side is greater than that on the right side. If there is no lychee tree in front, rotate the pan-tilt camera to identify from the left side. If there is no lychee tree on the left side, rotate the pan-tilt camera to the right side to identify the lychee tree.

[0160] Step 1.7: If there are new lychee tree targets on the left and right sides, repeat step 1.2. If no new lychee tree target is found in front, left, and right, stop the car and send a help instruction to the worker through the data processing center.

[0161] Further, the car plate balancing method in step 1.4.4, the specific process is as follows Figure 5 :

[0162] Step 1.8: Receive the car platform balancing instruction.

[0163] Step 1.8.1: The data processing center manually specifies the world coordinates, i.e. the coordinates Xw, Yw, Zw on the ground, and specifies the positive direction of the three coordinate axes. The camera coordinates are Xc, Yc, Zc. Since the camera coordinates may exist on a slope, there is an angle difference between the camera coordinates and the world coordinates, and the X, Y, Z axis angle difference is set as ψ, θ and φ. The length of the device mounting platform 8 is set as T, and the width is set as W.

[0164] Step 1.8.2: Calculate the included angle size through two three-dimensional coordinate axes. Take a point on the X, Y, Z axes of the camera coordinates and the world coordinates, and the included angle calculation model is:

[0165] Where X W , Y W , Z W are the three-dimensional coordinate system of the world coordinates, and X C , Y C , Z C are the three-dimensional coordinate system of the camera coordinates.

[0166] Step 1.8.3: Adjust the extension amplitude L of the platform balancing device. The extension amplitude L calculation model is: La = |Tsinα| or |Wsinα|, where La is the extension length, T and W represent the length and width of the device mounting platform 8, and α represents the included angle between the two coordinates.

[0167] Step 1.8.4: The extension and retraction of the platform balancing device can be divided into the following four cases: Case 1: When the angle α between the camera coordinates and the world coordinates X and Z axes is greater than 0, it indicates that the front of the trolley is too high. In this case, the extension of platform balancing device III111 and platform balancing device IV114 is La = |Tsinψ| or |Tsinφ|. Case 2: When the angle α between the camera coordinates and the world coordinates X axis is less than 0, it indicates that the rear of the trolley is too high. In this case, the extension of platform balancing device I112 and platform balancing device II113 is La = |Tsin(-φ)| or |Tsin(-ψ)|. Case 3: When the angle α between the camera coordinates and the world coordinates Y and Z axes is greater than 0, it indicates that the left side of the trolley is too high. In this case, the extension of platform balancing device II113 and platform balancing device IV114 is La = |Wsinθ| or |Wsinφ|. Case 4: When the angle α between the camera coordinates and the world coordinate Y-axis is less than 0, it indicates that the right side of the trolley is too high, and the extension of the platform balancing device I112 and platform balancing device III111 is La = |Wsin(-θ)| or |Wsin(-φ)|.

[0168] Step 1.9: After passing through the stretching and balancing device, obtain the angle α between the two coordinates again. If α = 0, end the balancing adjustment.

[0169] Step 2: The specific process for repairing broken tree branches is as follows: Figure 6 As shown:

[0170] Step 2.1: First, calibrate the camera, then acquire depth images using a depth camera, convert them into point cloud data, and preprocess the collected point cloud to remove noise and some discrete points. After completing the above operations, a 3D lychee tree is constructed from the point cloud.

[0171] Step 2.2: A lychee tree with multiple breakpoints is obtained through 3D reconstruction. The data processing center issues a breakpoint repair command, slices all point clouds with breakpoints, obtains the point cloud cross-section near the breakpoint, and calculates the center coordinates M of all point clouds on that cross-section. The method for calculating the center point is as follows:

[0172]

[0173] Where X i ,Y i Z i Here are the three-dimensional coordinates of each point cloud, and n is the total number of point clouds in the cross section.

[0174] Step 2.3: Point cloud breakpoints can be divided into the following two cases: Case 1, breakpoints exist on both sides of the point cloud; Case 2, breakpoints exist on only one side of the point cloud.

[0175] Step 2.3.1: When it is situation 1, set the center coordinates of the part of point cloud as [M1, M2] and store them in a set.

[0176] Step 2.3.2: When it is situation 2, set the center coordinates of the part of point cloud as C1, C2, C3, C4...

[0177] Step 2.4: Randomly select a part of point cloud with a breakpoint on both sides, and a straight line L1=A1x+B1y+C1z+D1 can be determined by the center points on both sides. Define the direction of the straight line as the repair direction towards the center point with smaller z value, and establish the center point with smaller z value as the repair point Mf.

[0178] Step 2.5: Draw a circle with Mf as the center, and judge whether there are other breakpoints M[i] or C[i] within a distance of 3 cm.

[0179] Step 2.6: Determine the breakpoint into the following four situations: situation 1, there are breakpoints M[i] and C[i] within the distance range; situation 2, there are no breakpoints within the distance range; situation 3, there is only one or more breakpoints M[i] within the distance range; situation 4, there is only one or more breakpoints C[i] within the distance range.

[0180] Step 2.6.1: When it is situation 1, connect Mf with multiple breakpoints, and form multiple straight lines L2, L3... through two points, and calculate the included angle a[i] between the straight lines and the straight line L1, the included angle formula is:

[0181]

[0182] where A1, B1, C1 are the parameters of the straight line L1, and A i B i C i are the parameters of the straight line Li.

[0183] Since the branches are a relatively smooth curve, manually set the rejection of the included angle greater than 20°, so the included angle aS between the remaining breakpoint straight line and L1 is <= 20°. Next, select the breakpoint straight line with the smallest included angle as the connection point, if there are the same number of included angles, calculate the distance S of the breakpoint M[i] or C[i] to Mf. The formula for the distance of three-dimensional point cloud is:

[0184]

[0185] Wherein X1, Y1, Z1 are the three-dimensional coordinates of the repair point Mf, X2, Y2, Z2 are the three-dimensional coordinates of the point M[i] or C[i]. The minimum angle or the same angle distance is selected as the new connection point, and the point cloud is repaired along the new connection point. The new connection point has the following two situations: situation 1, the new connection is M[i]; situation 2, the new connection is C[i].

[0186] Step 2.6.1.1: If it is situation 1, the new connection point is determined as M[i], two breakpoints in the M[i] set are connected to establish the straight line as the new L1, and the point with smaller z value in the M[i] set is determined as the new repair point Mf and the repair direction of L1. Step 2.5 is executed again.

[0187] Step 2.6.1.2: If it is situation 2, the new connection is C[i], and the point cloud is repaired along the connection point, because the new connection point has only one breakpoint, which means that the other end has been connected to the trunk, that is, Mf is connected to the trunk after repair. Therefore, it represents that the repair is completed.

[0188] Step 2.6.2: If it is situation 2, when there is no breakpoint within the distance, the point cloud is repaired along the repair direction by 3 cm, and there are two situations during the extension: situation 1, the extended end point does not intersect with any trunk or branch point cloud; situation 2, the point cloud intersects with the trunk or branch point cloud during the extension.

[0189] Step 2.6.2.1: When it is situation 1, the new point cloud is obtained on the cross section of the extended end point, and the new repair point Mf is established, the repair direction is unchanged, and step 2.5 is executed again.

[0190] Step 2.6.2.2: When it is situation 2: if the point cloud intersects with the branch or trunk during the repair, the repair is stopped, that is, the repair is completed.

[0191] Step 2.6.3: When it is situation 3, step 2.6.1 and step 2.6.1.1 are performed.

[0192] Step 2.6.4: When it is situation 4, step 2.6.1 and step 2.6.1.2 are performed.

[0193] Step 2.7: Whenever a branch is repaired, step 2.4 is repeated until all breakpoints are repaired, that is, the repair of all point cloud breakpoints is completed. The data processing center executes subsequent point cloud pairing and semantic segmentation instructions.

[0194] Step 3: The specific process of the preferred branch screening method is shown in Figure 7

[0195] ​Step 3.1: When the main branch and the first branch are segmented by semantic segmentation, the data processing center sets the parameter count to calculate the first branch on each main branch.

[0196] Step 3.2: It is artificially stipulated that there are at most 4 first branches on a main branch, and the size of the first branch is stipulated to be between 40-50 cm. If there are too many branches, it will affect the development of the litchi tree and the yield of subsequent litchi.

[0197] Step 3.3: The data processing center slices the three-dimensional point cloud to extract the length data[i] of the first branch in the main branch and puts them into the set Rm, that is, Rm = {data1, data2, data3, data4,...}. First, put the data[i] > 50 cm first branch into the pruning set R, and the data[i] < 40 cm first branch into the pruning set R2.

[0198] Step 3.4: After the first round of screening, calculate the number of branches in the set Rm at this time. If count < 4, the main branch is screened, if count > 4, the optimal selection method needs to be used.

[0199] Step 3.5: After the long and short branches are removed, the remaining branches are between 40-50 cm in length, but since at most 4 are retained, pruning is still needed. Since the optimal length of the first branch is 40-50 cm, the average is 45 cm. Perform the data = |data-45| operation on data[i] in Rm, which selects the branches with larger differences from the average to prune. Update data[i] in Rm, and then sort the set Rm from large to small, and extract the first count-4 branches into the set R2.

[0200] Step 3.6: The above is the selection and judgment of a main branch, which needs to repeat steps 3.3, 3.4 to select and judge all first branches on the litchi tree and put them into sets R and R2. After screening, the data processing center issues the next instruction to extract the branches in sets R and R2 to select the pruning points.

[0201] Step 4: The specific process of the pruning point determination method is shown in Figure 8

[0202] Step 4.1: The data processing center extracts the parameters of the branches in sets R and R2, and fits the curve by the least squares method. Since the branches are curved, the bending points are selected as the pruning points for pruning. This method automatically identifies the pruning points.

[0203] ​Step 4.2: Select a branch in R, and extract the point cloud coordinates of the branch at the root, bending point (not more than one), and point near the top of the branch (how to identify the bending point, root, and top point can be trained by deep learning). Then perform point cloud slicing on the bending point to obtain the cross-sectional point cloud. The cross-sectional point cloud center coordinates F are calculated based on the point cloud coordinates on the cross section:

[0204]

[0205] where X i ,Y i ,Z i are the three-dimensional coordinates of each cross-sectional point cloud, and n is the number of point clouds in the current section.

[0206] Step 4.3: Assign weights to each center point F[i] through data processing center, F0 = (x0 + a0, y0 + b0, z0 + c0), F1 = (x1 + a1, y1 + b1, z1 + c1), F[i] = (x[i] + ai, y[i] + bi, z[i] + ci)......, and use the least squares method to construct a distance optimization problem, i.e. where di is the distance from the ith point cloud in the section to the center point F of the section, and d is the average distance from the center point F of the section to all point clouds in the section. Solve a[i], b[i], and c[i] to find the optimal point closest to all point clouds in the section. That is, F[i] is no longer the center point of the section, but the optimal point of the section.

[0207] Step 4.4: Determine the optimal points F of the root, bending point, and top point by the least squares method. Connect each optimal point to fit the original linear shape of the branch.

[0208] Step 4.5: After obtaining the curve, place the pruning point target on the bending point. Since the optimal point cloud coordinates are known, the length of the straight line connecting each two optimal points can be calculated as len[i], and the formula is:

[0209]

[0210] where X1, Y1, Z1, and X2, Y2, Z2 are the three-dimensional coordinates of the two adjacent optimal points.

[0211] From the root optimal point, accumulate the length of each straight line Ls, and the formula is:

[0212]

[0213] where n is the nth branch.

[0214] Step 4.6: Extracting the branches from the screened branch set R, accumulating each segment, stopping accumulation when Ls>50, and determining the n-1th bending point as the pruning point.

[0215] Step 4.7: For the branches in set R2, because there are too many primary branches on the parent branch, and the diseased branches are less than 40 cm, the whole branch needs to be cut off, so the root optimal point of all branches in set R2 is directly selected as the pruning point, and all branches with determined pruning points are put into set R3.

[0216] Step 4.8: Through the above operation, the pruning points of the branches in set R and set R2 can be determined, and only by repeating steps 4.2 and 4.6, 4.7, the determination of the pruning point is completed. Next, the data processing center issues an instruction to calculate the distance from the mechanical arm to the pruning point of the branches in set R3, and plans the pruning from bottom to top.

[0217] Step 5: The specific process of the pruning method from bottom to top is shown in Figure 9

[0218] Step 5.1: After calculating the distance from the pruning point to the mechanical arm, sort from small to large, and start pruning from the lowest branch, which can achieve the most efficient pruning and avoid many unnecessary path planning.

[0219] Step 5.2: Manually set the mechanical arm coordinates and define a set Q for each branch in set R3, which contains a number x[i] and a distance d[i], so Q[i] = {x[i], d[i]}.

[0220] Step 5.3: The conversion relationship between the world coordinates (Xw, Yw, Zw) and the camera coordinates (Xc, Yc, Zc) can be expressed as:

[0221]

[0222] where R is the rotation matrix from the world coordinate system to the camera coordinate system, and the specific expression of R is:

[0223]

[0224] where θ, φ are the rotation angles of the world coordinates to the camera coordinates X, Y, Z axes. T = [t1t2 t3] T is the translation vector from the world coordinate system to the camera coordinate system.

[0225] Step 5.4: Then convert between camera coordinates and image coordinates g(x, y):

[0226] ​

[0227] where f is the focal length of the lens.

[0228] Step 5.5: Through the conversion of the image g(x, y) and the pixel coordinates u(xu, yu), the image coordinate origin is g'(u, v) in the pixel mapping coordinates:

[0229]

[0230] where dx and dy are the corresponding physical sizes of a single pixel in the x-axis and y-axis directions.

[0231] Step 5.6: The distance from the mechanical arm to the pruning point can be calculated through the coordinate conversion of the mechanical arm coordinates in the world coordinates, and the distance can be calculated through the three-dimensional coordinates , where X1, Y1, Z1 are the mechanical arm coordinates, X i , Y i , and Z i are the pruning point coordinates. After determining all the pruning distances d[i], the set R4 is sorted in ascending order, and the corresponding number x[i] in the set Q[i] is determined according to d[i] to determine which branch is pruned. The pruning point is determined by the data processing center, and the pruning command is sent to the device control center. The device control center sends the command to the mechanical arm lifting platform, mechanical arm, and mechanical pruning shears to move to the appropriate position for pruning, completing the pruning from bottom to top.

[0232] Trolley plate balancing method: Through three-dimensional coordinate rotation, the X, Y, and Z rotation angles are solved, solving the problem of different angles at different slopes. Branch break point repair method: solves the problem of partial point cloud disconnection due to the influence of other branches and leaves, and does not fit into a complete curve. Optimal branch selection method: solves the problem of selecting all long or short branches and redundant branches on a mother branch, and realizes the selection of branches in a tree through the selection of all mother branches.

[0233] Pruning point determination method: solves the problem of extracting the required pruning branches, and through point cloud fitting curve, specifies pruning at the bending point of the branch. After fitting, it is judged which bending point to prune. From bottom to top pruning method: the pruning point of the branch in the set R3 is determined through the coordinate conversion of the visual system, solving the problem of calculating the distance from the mechanical pruning shears to the pruning point, and sorting all distances, solving the problem of pruning from bottom to top.

[0234] Trolley plate balancing method: the X, Y, Z rotation angles are solved by three-dimensional coordinate rotation, and the extension distance of the balancing device is calculated by the length and width of the trolley.

[0235] Optimal branch screening method: through the code, the national pruning standard of litchi tree is realized, and the pruning is classified. The branches with length greater than 50 and less than 40 are put into set R1, and the excess branches are put into set R2. Through screening and classification, the subsequent pruning point determination is facilitated.

[0236] Trolley plate balancing method: the rotation angles are obtained by the difference between the two three-dimensional coordinate angles, and the height adjustment of the four balancing devices is calculated.

[0237] Branch breakpoint repair method: the breakpoint repair direction and repair point are determined by using the center coordinates of the breakpoint cross-section point cloud, and the position of the next repair point is determined according to the angle, distance and other factors with the repair point as the center, so that the repair of the tree point cloud is realized step by step.

[0238] Optimal branch screening method: after semantic segmentation, this method screens all branches. Through this screening, all branches within the non-required range and the excess branches of each parent branch are screened out.

[0239] Pruning point determination method: the cross-section point cloud is obtained by slicing the bending point, the center point is calculated, and the least square method is used to calculate the optimal point of the cross-section. The identification of bending points, roots and end points can be realized by deep learning. The bending point is set at the bending point, and the pruning point is determined by the length of the line segment.

[0240] The method of pruning from bottom to top: using all the branches we have determined, pruning from the pruning point to the distance from the robot arm from small to large, instead of pruning the branches on a mother branch one by one. The distance between the three-dimensional point cloud coordinates and the three-dimensional coordinates of the robot arm is obtained through the coordinate transformation of the visual system, and the determined pruning points are extracted, the distance is sorted from small to large by using quicksort, and the double robot arms are used to prune the branches more efficiently.

[0241] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A pruning method for a litchi tree intelligent pruning device based on a three-dimensional point cloud, characterized by: The device for implementing pruning includes a mechanical pruning shears, a mechanical arm, a mechanical arm lifting device, a power supply device, a device control and data processing center, a pan-tilt camera, a depth camera I, a depth camera II, a depth camera III, a device mounting platform, a platform balancing device, a device carrying vehicle, and a post-pruning branch and leaf garbage can. The branch and leaf garbage can is arranged below the device mounting platform, the platform balancing device is arranged on the device carrying vehicle, the device mounting platform is arranged at the top end of the platform balancing device, the pan-tilt camera is arranged at the front center of the device mounting platform, the data processing and device control center, the power supply device, and the mechanical arm lifting device are all arranged on the device mounting platform, the mechanical pruning shears are arranged on the mechanical arm, the mechanical arm is arranged on the mechanical arm lifting device, the pan-tilt camera, the depth camera I, the depth camera II, and the depth camera III are all arranged on the device mounting platform, the pan-tilt camera, the depth camera I, the depth camera II, and the depth camera III are all connected with the device control and data processing center, and the mechanical pruning shears, the mechanical arm, the mechanical arm lifting device, and the platform balancing device are all connected with the device control and data processing center. The specific process of the pruning method is as follows: The litchi tree intelligent pruning device is started, the intelligent pruning device automatic advancing method is adopted, the image information in front of and on both sides of the device is obtained by the pan-tilt camera, after processing by the device control and data processing center, path planning is performed, the device control and data processing center sends a control instruction to control the vehicle to advance to a fixed distance below the litchi tree, then the trolley plate balancing method is adopted to adjust the platform balance, so that the device mounting platform is in a relatively horizontal state, which is conducive to subsequent point cloud collection. After the device mounting platform is balanced, the depth cameras are calibrated, point cloud collection is performed from the front and the opposite side by the depth camera I, the depth camera II, and the depth camera III, after the point cloud is completed, the discrete points and noise are removed in the device control and data processing center, three-dimensional reconstruction is performed, and after reconstruction, the breakpoint repair method is adopted to complete the model. The point clouds of the two angles are registered and semantically segmented, the first-order branches and mother branches are divided out, and the required pruning length branches and redundant branches are selected by the optimal branch selection method. The point cloud of each required pruning branch is extracted in the device control and data processing center, and the pruning point determination method is adopted to determine the pruning point of each branch. Finally, the from-bottom-to-top pruning method is adopted, the distance between the required pruning branches and the mechanical pruning shears is sorted from small to large by the device control and data processing center, the path is planned from the smallest distance, the device control and data processing center sends an instruction to control the mechanical arm and the mechanical pruning shears to prune, the grabbed branches are put into the branch storage garbage can after pruning, and the whole pruning operation is completed.

2. The method of claim 1, wherein the method is implemented by the device of claim 1. The specific process of the intelligent pruning device automatic advancing is as follows: Step 1.1: The pan-tilt camera obtains the RGB images in front of and on both sides of the litchi tree intelligent pruning device. Step 1.2: The device control and data processing center analyzes and processes the RGB image information, identifies and determines the location of the litchi tree, plans a path to the litchi tree, and the intelligent pruning device moves forward according to the planned path and detects whether there are other obstacles on the path in real time; Step 1.3: According to the analysis of the path information in front of the RGB image, the path planning is divided into three cases: case 1, no obstacles in front of the path; case 2, there are obstacles in front of the path, but they can be bypassed on the left or right side; case 3, there are obstacles in front of the path and they cannot be bypassed; Step 1.4: When it is case 1, the specific processing steps include the following: Step 1.4.1: The data processing center of the device control and data processing center sends a forward command to the device control center; Step 1.4.2: The device control center sends a command to control the car to run, realizing the quantitative forward movement and left and right turning of the intelligent pruning device; Step 1.4.3: The forward distance of the intelligent pruning device is in the form of quantitative forward movement, and the current distance from the target litchi tree is obtained by the pan-tilt camera each time, and then the device continues to move forward towards the planned path; Step 1.4.4: When it comes to the litchi tree, the car plate balancing method and subsequent pruning instructions are sent, and if the branches do not need to be pruned, a new litchi tree is identified until all litchi trees have no pruning needs; Step 1.5: When it is case 2, the specific processing steps include the following: Step 1.5.1: The intelligent pruning device plans a bypass path on the left or right side of the obstacle in front of it; Step 1.5.2: The pan-tilt camera is turned left or right to determine whether there are other obstacles on the bypass path, and if there are no obstacles, the bypass route is planned and the device moves forward quantitatively; Step 1.6: When it is case 3, the specific processing steps include the following: Through the analysis of the pan-tilt camera on both sides, if there are obstacles, it is analyzed whether they will affect the bypass route, if not, the path is planned and the device moves forward quantitatively, if they do, the target litchi tree is abandoned and a new litchi tree target is selected, the priority of the left side is greater than that of the right side, if there is no litchi tree in front, the pan-tilt camera is rotated to the left side to identify the litchi tree, if there is also no litchi tree on the left side, the pan-tilt camera is rotated to the right side to identify the litchi tree; Step 1.7: If there are new litchi tree targets on the left and right sides, step 1.2 is repeated, if there are no new litchi tree targets in front, left and right, the car is stopped and a help instruction is sent to the worker through the data processing center; The car plate balancing method in step 1.4.4 is as follows: Step 1.4.4.1: The platform balancing device receives the balancing instruction; Step 1.4.4.2: The data processing center manually specifies the world coordinates, i.e. the coordinates Xw, Yw, Zw on the horizontal ground and the positive directions of the three coordinate axes, and the camera coordinates are Xc, Yc, Zc, since the camera coordinates exist on a slope, there is an angle difference between the camera coordinates and the world coordinates, the X, Y, Z axis angle difference is set as ψ, θ and φ, the length of the device installation platform is set as T, and the width is set as W; Step 1.4.4.3: Calculate the angle between the two vectors on the three-dimensional coordinate axes, take a point on the X, Y, and Z axes of the camera coordinates and the world coordinates, and calculate the angle between the two vectors. The angle calculation model is as follows: Step 1.4.4.4: Adjust the extension range L of the platform balancing device. The extension range L calculation model is as follows: La = |Tsinα| or |Wsinα|, where La is the extension length, T and W represent the length and width of the equipment installation platform, and α represents the angle between the two coordinates. Step 1.4.4.5: The extension of the platform balancing device can be divided into the following four cases: Case 1: When the angle α between the camera coordinates and the world coordinates X and Z axis coordinates is greater than 0, it indicates that the front side of the trolley is higher, then the platform balancing device III and the platform balancing device IV are extended by La = |Tsinψ| or |Tsinφ| range, Case 2: When the angle α between the camera coordinates and the world coordinates X axis coordinates is less than 0, it indicates that the rear side of the trolley is higher, then the platform balancing device I and the platform balancing device II are extended by La = |Tsin(-φ)| or |Tsin(-ψ)| range, Case 3: When the angle α between the camera coordinates and the world coordinates Y and Z axis coordinates is greater than 0, it indicates that the left side of the trolley is higher, then the platform balancing device II and the platform balancing device IV are extended by La = |Wsinθ| or |Wsinφ| range, Case 4: When the angle α between the camera coordinates and the world coordinates Y axis coordinates is less than 0, it indicates that the right side of the trolley is higher, then the platform balancing device I and the platform balancing device III are extended by La = |Wsin(-θ)| or |Wsin(-φ)| range. Step 1.4.4.6: After the extension of the platform balancing device, the angle α between the two coordinates is obtained again. If α = 0, the balancing adjustment is completed.

3. The method of claim 2, wherein the method is implemented by the device of claim 1. The specific process of the breakpoint repair method is as follows: Step 2.1: First, calibrate the camera, then obtain the depth image through the depth camera, convert it to point cloud data, and preprocess the collected point cloud to remove noise and discrete points. The three-dimensional reconstruction of the point cloud forms a three-dimensional lychee tree. Step 2.2: Through three-dimensional reconstruction, a lychee tree with several breakpoints is obtained. The data processing center issues a breakpoint repair instruction, and all point clouds with breakpoints are sliced to obtain the point cloud section near the breakpoint. The center coordinates M of all point clouds on the section are calculated, and the calculation formula of the center point is as follows: Step 2.3: The point cloud breakpoint can be divided into the following two cases: Case 1, both sides of the point cloud have breakpoints, and Case 2, only one side of the point cloud has a breakpoint. Step 2.3.1: When Case 1 occurs, set the center coordinates of the point cloud on both sides as [M1, M2] and store them in a set. Step 2.3.2: When Case 2 occurs, set the center coordinates of the point cloud as C1, C2, C3, C4,..., CN, where N is a positive integer. Step 2.4: Randomly select a point cloud part with breakpoints on both sides, determine a straight line L1 = A1x + B1y + C1z + D1 from the center points on both sides. The direction of the straight line towards the center point with smaller z value is defined as the repair direction, and the center point with smaller z value is defined as the repair point Mf. Step 2.5: Draw a circle with Mf as the center, and determine whether there are other breakpoints M[i] or C[i] within a distance of 3 cm; Step 2.6: Determine the breakpoints into the following four situations: Situation 1, there are breakpoints M[i] and C[i] within the distance range, Situation 2, there are no breakpoints within the distance range, Situation 3, there is only one or more breakpoints M[i] within the distance range, Situation 4, there is only one or several breakpoints C[i] within the distance range; Step 2.6.1: When it is Situation 1, connect Mf with multiple breakpoints, and form multiple straight lines L2, L3,..., LN through two points, N is a positive integer, and calculate the included angle α between the straight lines and the straight line L1, the included angle formula is: wherein A1, B1, C1 are parameters of straight line L1, A i B i C i are parameters of straight line Li, since the branch is a relatively smooth curve, manually set the angle greater than 20° to be rejected, so the remaining angle αs between the breakpoint straight line and L1 is <= 20°, then select the breakpoint straight line with the smallest angle as the connection point, if there are the same number of angles, then calculate the distance S from the breakpoint M[i] or C[i] to Mf; the three-dimensional point cloud distance formula is: Select the breakpoint with the smallest included angle or the shortest distance with the same included angle as the new connection point, and perform point cloud repair along the straight line of the new connection point, the new connection point has the following two situations: Situation 1, the new connection is M[i], Situation 2, the new connection is C[i]; Step 2.6.1.1: If it is Situation 1, determine the new connection point as M[i], connect two breakpoints in the M[i] set and establish this straight line as the new L1, and determine the point with the smaller z value in the M[i] set as the new repair point Mf and the repair direction of L1, and execute Step 2.5 again; Step 2.6.1.2: If it is Situation 2, the new connection is C[i], then perform point cloud repair along the straight line of the connection point, because the new connection point only has one breakpoint, it means that the other end has been connected to the trunk, so Mf is connected to the trunk after repair, which represents the completion of the repair; Step 2.6.2: If it is Situation 2, there are no breakpoints within the distance, repair the point cloud by 3 cm in the repair direction, there are two situations during the extension: Situation 1, the extended point cloud does not intersect with any trunk or branch point cloud, Situation 2, the point cloud intersects with the trunk or branch point cloud during the extension; Step 2.6.2.1: When it is Situation 1, find the new point in the cross section of the extended point and establish it as the new repair point Mf, the repair direction remains unchanged, and Step 2.5 is executed again; Step 2.6.2.2: When it is Situation 2: If the point cloud intersects with the branch or trunk point cloud during the repair, stop the repair, which means the repair is completed; Step 2.6.3: When it is Situation 3, perform Step 2.6.1 and Step 2.6.1.1; Step 2.6.4: When it is Situation 4, perform Step 2.6.1 and Step 2.6.1.2; Step 2.7: Whenever a branch is repaired, repeat the operation of Step 2.4 until all breakpoints are repaired, which means the repair of all point cloud breakpoints is completed, and the data processing center executes subsequent point cloud pairing and semantic segmentation instructions.

4. The pruning method of the intelligent litchi tree pruning device based on a three-dimensional point cloud according to claim 3, characterized in that: The specific process of the optimal branch selection method is as follows: Step 3.1: After the semantic segmentation, the branch and the first-level branch are segmented, and the data processing center sets the parameter count to calculate the first-level branch on each branch; Step 3.2: There are at most 4 primary branches on one mother branch, and the size of the primary branch is specified between 40-50 cm, if there are too many branches, it will affect the development of litchi tree and the yield of subsequent litchi; Step 3.3: Through the data processing center, slice the three-dimensional point cloud to extract the length of the primary branch data[i], i is a positive integer, and put them into the set Rm, then Rm={data1, data2, data3, data4...}, first put data[i]>50cm primary branch into the pruning set R, and data[i]<40cm primary branch into the pruning set R2; Step 3.4: After the first round of screening, calculate the number of branches in the set Rm, if count<4, the mother branch is screened, if count>4, the optimal selection method is needed; Step 3.5: After the removal of long and short branches, the remaining branch length is between 40-50 cm, but since at most 4 are retained, pruning is needed, since the optimal length of the primary branch is 40-50 cm, the average value is 45 cm, data[i] in Rm is operated data=|data-45|, which selects the branches with large difference from the average value for pruning, and data[i] in Rm is updated, then the set Rm is sorted from large to small, and the first count-4 branches are extracted and put into the set R2; Step 3.6: After the selection and judgment of a mother branch, steps 3.3 and 3.4 are repeated to select all primary branches on the litchi tree and put them into sets R and R2, after screening, the data processing center issues the next instruction to extract the branches in sets R and R2 for pruning point selection.

5. The method of claim 4, wherein the method is implemented by the device of claim 1. The specific process of pruning point determination is as follows: Step 4.1: Through the data processing center, the parameters of the branches in sets R and R2 are extracted, and the three-dimensional point cloud of the branches is fitted by the least squares method, since the branches are curved, the bending points are selected as the pruning points for pruning, and the automatic identification of the pruning points is performed; Step 4.2: Select a branch in R, and extract the point cloud coordinates of the branch at the root, bending point, and top near the branch, train the recognition of the bending point, root, and top point through deep learning, then perform point cloud slicing on the bending point to obtain the cross-sectional point cloud, and calculate the cross-sectional point cloud center coordinate F: Where n is the number of point clouds in the current section; Step 4.3: Assign weights to each center point F[i] by the data processing center, F0= (x0+a0, y0+b0, z0+c0), F1= (x1+a1, y1+b1, z1+c1), F[i]= (x[i]+ai, y[i]+bi, z[i]+ci)......, and a distance optimization problem is constructed by using the least square method, that is Where di is the distance from the i-th point cloud of the section to the center point F of the section, d is the average distance from the center point F of the section to all point clouds in the section, and a[i], b[i], c[i] are solved to find the optimal point closest to all point clouds on the section, that is, F[i] is no longer the center point of the section, but the optimal point of the section. Step 4.4: The optimal points F of the root, bending point, and top point are determined by the least squares method, and each optimal point is connected to fit the original linear shape of the branch; Step 4.5: After obtaining the curve, the pruning point target is placed on the bending point, since the optimal point cloud coordinates are known, the length of the straight line connecting each two optimal points len[i] can be calculated, the formula for calculating len[i] is: From the root optimal point, the length of each straight line Ls is accumulated, the formula is: Where n is the nth branch; Step 4.6: Extracting branches from the screened branch set R, accumulating each segment, stopping accumulation when Ls>50, and determining the n-1th bending point as the pruning point; Step 4.7: For the branches in set R2, because the first-level branches on the parent branch are greater than the set value and the diseased branches are less than 40 cm, the whole branch needs to be cut off, so the root optimal point of all branches in set R2 is directly selected as the pruning point, and all branches with determined pruning points are put into set R3; Step 4.8: Determine the pruning points of branches in set R and set R2, repeat steps 4.2 and 4.6, 4.7, that is, complete the determination of the pruning point, and then the data processing center sends an instruction to calculate the distance from the mechanical arm to the pruning point of the branches in set R3, and plans the pruning from bottom to top. 6.The method of claim 1, wherein the method is implemented by a device for intelligent pruning of lychee trees based on a three-dimensional point cloud. The specific process of the bottom-to-top pruning method is as follows: Step 5.1: After calculating the distance from the pruning point to the mechanical arm, sort from small to large, and start pruning from the lowest branch, which can achieve the most efficient pruning and avoid unnecessary path planning; Step 5.2: Set the mechanical arm coordinates and define a set Q for each branch in set R3, which contains a number x[i] and a distance d[i], so Q[i]={x[i],d[i]}; Step 5.3: The conversion relationship between the world coordinates (Xw,Yw,Zw) and the camera coordinates (Xc,Yc,Zc) can be represented as: Where R is the rotation matrix from the world coordinate system to the camera coordinate system, and the specific expression of R is: wherein θ, φ are the rotation angles of the world coordinate to the camera coordinate X, Y, Z axes, T = [t1t2t3] T is the translation vector of the world coordinate system to the camera coordinate system; Step 5.4: Then convert between camera coordinates and image coordinates g(x,y): Where f is the lens focal length; Step 5.5: Through the conversion of image g(x,y) and pixel coordinates u(xu,yu), the image coordinate origin is mapped to pixel coordinates g'(u,v): Where dx and dy are the corresponding physical dimensions of a single pixel point in the x and y directions; Step 5.6: The distance from the robot arm to the pruning point can be calculated by coordinate transformation through the position of the robot arm coordinate in the world coordinate, and the distance can be calculated by three-dimensional coordinates After determining all the pruning distances d[i], the set R4 is sorted in ascending order, and the corresponding number x[i] in the set Q[i] is determined according to d[i] to determine which branch. The pruning point of the branch is determined by the data processing center, and the pruning command is sent to the device control center. The device control center sends the command to the robot arm lifting platform, the robot arm and the mechanical pruning shears to move to the appropriate position for pruning to complete the pruning of the branches from bottom to top.

7. The method of claim 1, wherein the method is implemented by the device of claim 1. The pruning garbage can is arranged between the equipment mounting platform and the wheels, and an infrared sensor is arranged in the garbage can for sensing the storage amount of branches in the garbage can. The number of mechanical pruning shears, mechanical arms and mechanical arm lifting devices is two.

8. The pruning method of the intelligent lychee tree pruning device based on a three-dimensional point cloud according to claim 7, characterized in that: The mechanical pruning shears are externally arranged as a mechanical hand structure for grabbing branches, and a retractable blade is arranged inside the mechanical hand structure. The mechanical hand structure fixes the pruning point and uses the internal blade to cut off the branches.

9. A pruning method for a litchi tree intelligent pruning device based on three-dimensional point cloud as described in claim 8, characterized in that: In order to keep the equipment mounting platform in a relatively horizontal state, the tilt angle of the equipment mounting platform is obtained by the coordinate angle difference between the world coordinates and the gimbal camera coordinates arranged in the equipment control and data processing center, and the corresponding control instruction is generated. Under the control of the equipment control and data processing center, the platform balancing device realizes different amplitude extension to keep the equipment mounting platform relatively horizontal.

Images