Fruit thinning robot and fruit thinning method thereof

Abstract

The invention belongs to the technical field of vegetable cultivation, and particularly relates to a fruit thinning robot and a fruit thinning method.The fruit thinning robot comprises a robot chassis, a six-axis mechanical arm, a multi-position fixed camera and a control system, the robot chassis is used for autonomously moving, and the multi-position fixed camera and the robot chassis are fixedly connected and installed; the multi-position fixed camera is composed of a plurality of cameras with different poses, the six-axis mechanical arm and the robot chassis are movably installed, and an electric cutting device is arranged at the tail end of the six-axis mechanical arm and used for cutting defective fruits and small fruits. According to the invention, tomatoes of different types and growth states can be identified through machine learning, so that the growth conditions of the tomatoes hung on branches are accurately judged, artificial fruit thinning is simulated, intelligent unmanned fruit thinning is realized, and efficient, accurate and convenient fruit thinning service is provided.

Description

Technical Field

[0001] The invention belongs to the technical field of vegetable cultivation, and particularly relates to a fruit thinning robot and a fruit thinning method thereof. Background Art

[0002] During the growth of tomatoes, without human intervention, too many young tomato fruits will grow on the same plant at the same time, resulting in a large number of secondary and small fruits. Secondary and small fruits will occupy a large amount of nutrients during the growth of tomatoes, resulting in a reduction in nutrients available for other large fruits, a decline in tomato fruit quality, and a reduction in sales prices. Therefore, fruit thinning is an essential step and link in the planting process. In traditional technology, fruit thinning is mostly done by workers manually cutting off secondary and small fruits. Through the fruit thinning process, sufficient nutrient supply can be guaranteed to ensure good growth of tomatoes.

[0003] In traditional technology, manual fruit thinning mostly uses manual tools, such as scissors, to cut off excess fruit. Or it uses semi-automatic handheld electric tools, such as electric flower shears. The disadvantages are as follows:

[0004] 1. Although the efficiency is improved compared with traditional tools, it still needs to search for inferior and small fruits for removal, and the efficiency is still relatively low;

[0005] 2. The fruit thinning process is complicated and can easily damage other retained fruits, resulting in an overall reduction in yield;

[0006] 3. Workers are prone to fatigue when working for long hours, which further reduces work efficiency;

[0007] 4. When workers are tired, it is very easy to miss fruits during the thinning process, resulting in a reduction in yield;

[0008] 5. Large-scale planting requires a lot of manpower and a lot of expenses, which further reduces the economic benefits;

[0009] 6. For large-scale tomato gardens, manual labor is often unable to complete the fruit thinning task within the specified time, causing further economic losses to the orchard. Summary of the invention

[0010] The purpose of the present invention is to provide a fruit thinning robot and a fruit thinning method thereof, so as to solve the problems raised in the background technology.

[0011] In order to achieve the above technical objectives, the technical solution adopted by the present invention is as follows:

[0012] A fruit thinning robot comprises a robot chassis, a six-axis robotic arm, a multi-camera fixed camera and a control system, wherein the robot chassis is used for autonomous movement, the multi-camera fixed camera is fixedly connected and installed with the robot chassis, the multi-camera fixed camera is composed of a plurality of cameras with different postures, the six-axis robotic arm is movably installed with the robot chassis, an electric cutter is arranged at the end of the six-axis robotic arm, and the electric cutter is used for cutting inferior fruits and small fruits.

[0013] The control system includes a calibration system, a three-dimensional reconstruction system, an intelligent recognition system, a path planning system and a motion control system;

[0014] The calibration system is used to perform camera calibration on an imaging system composed of multiple fixed cameras;

[0015] The intelligent identification system is used to identify inferior fruits and small fruits;

[0016] The path planning system is used to plan the optimal travel path of the six-axis robot;

[0017] The motion control system is used to control the six-axis robot arm to move along a path so that the electric cutter cuts inferior fruits and small fruits.

[0018] A fruit thinning method, using the above-mentioned fruit thinning robot, comprises the following steps:

[0019] Step 1: Use multiple fixed cameras to calibrate the camera to determine the internal and external parameters and distortion parameters of each camera;

[0020] Step 2: Use the YOLO algorithm to learn and identify the appearance of the fruit;

[0021] Step 3: Reconstruct the 3D point cloud model of the plant;

[0022] Step 4: Path planning of the six-axis robot end;

[0023] Step 5: Use electric trimmers to trim and thin the fruits;

[0024] Step 6: Repeat steps 2 to 5 until no more fruit can be identified on the plant that need to be cut.

[0025] In the step 1, a geometric model of camera imaging is established to determine the relationship between the three-dimensional geometric position of a point on the surface of the space object and its corresponding point in the image, so that the six-axis robot arm can be accurately positioned and cut by the electric cutter.

[0026] In the step 2, the YOLO algorithm learns and recognizes the appearance of the fruit and is responsible for scanning the fruit image in the tomato planting area, thereby identifying the state of the fruit on the plant and determining whether to crop it.

[0027] In the step three, the three-dimensional point cloud model of the tomato plant is reconstructed based on the images taken by multiple cameras. According to the distribution of the three-dimensional point cloud, the three-dimensional space is divided into a grid model, and the poorly growing fruits are removed. At the same time, obstacles such as the stems and leaves of the plant and the well-growing fruits need to be avoided during the removal process.

[0028] The path planning in step 4 is responsible for generating the optimal path from the starting point to the fruit to be cut at the end of the six-axis robot arm to avoid accidental damage to other fruits or plants.

[0029] In the step five, after the fruit image is captured, the fruit to be removed is marked by identification. In the step one, the coordinate system has been unified with the world coordinate system through camera calibration and the distortion of the camera shooting has been eliminated. The marked fruit is placed in the coordinate system to facilitate the subsequent cutting by the robotic arm.

[0030] The present invention can identify tomatoes of different types and growth states through machine learning, thereby accurately judging the growth status of hanging branch tomatoes, imitating manual fruit thinning, and realizing intelligent unmanned fruit thinning, thereby providing efficient, accurate and convenient fruit thinning services. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention is further illustrated by means of the non-limiting examples given in the accompanying drawings.

[0032] Figure 1 This is a schematic diagram of the layout of a fruit thinning robot according to the present invention;

[0033] Figure 2 It is a schematic diagram of the use process of the present invention;

[0034] Figure 3 It is a schematic diagram of the YOLOv5 model structure of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] like Figure 1-3 As shown, a fruit thinning robot of the present invention comprises a robot chassis, a six-axis robotic arm, a multi-position fixed camera and a control system, wherein the robot chassis is used for autonomous movement, the multi-position fixed camera is fixedly connected and installed with the robot chassis, the multi-position fixed camera is composed of a plurality of cameras with different postures, the six-axis robotic arm is movably installed with the robot chassis, an electric cutter is arranged at the end of the six-axis robotic arm, and the electric cutter is used for cutting inferior fruits and small fruits.

[0037] The control system includes a calibration system, a three-dimensional reconstruction system, an intelligent recognition system, a path planning system and a motion control system;

[0038] The calibration system is used to perform camera calibration on an imaging system composed of multiple fixed cameras;

[0039] The intelligent identification system is used to identify inferior fruits and small fruits;

[0040] The path planning system is used to plan the optimal travel path of the six-axis robot;

[0041] The motion control system is used to control the six-axis robot arm to move along a path so that the electric cutter cuts inferior fruits and small fruits.

[0042] A fruit thinning method, using the above-mentioned fruit thinning robot to perform the operation, comprises the following steps:

[0043] Step 1: Use multiple fixed cameras to calibrate the camera to determine the internal and external parameters and distortion parameters of each camera;

[0044] Step 2: Use the YOLO algorithm to learn and identify the appearance of the fruit;

[0045] Step 3: Reconstruct the 3D point cloud model of the plant;

[0046] Step 4: Path planning of the end of the six-axis robot arm;

[0047] Step 5: Use electric trimmers to trim and thin the fruits;

[0048] Step 6: Repeat steps 2 to 5 until no more fruit can be identified on the plant that need to be cut.

[0049] In the step 1, a geometric model of camera imaging is established to determine the relationship between the three-dimensional geometric position of a point on the surface of the space object and its corresponding point in the image, so that the six-axis robot arm can be accurately positioned and cut by the electric cutter.

[0050] The core of camera calibration is to determine the relationship between the three-dimensional geometric position of the object surface and its corresponding point in the image through the calibration plate, establish a projection imaging model, and solve the intrinsic parameters, extrinsic parameters and distortion parameters through matrix transformation calculation.

[0051] Radial Distortion: The lens of an actual camera always produces significant distortion at the edge of the imager. This phenomenon comes from the "barrel" or "fisheye" effect.

[0052] Tangential Distortion: Tangential distortion is caused by defects in lens manufacturing that make the lens itself not parallel to the image plane.

[0053] Rotation matrix: A matrix that, when multiplied by a vector, has the effect of changing the direction of the vector without changing its magnitude and preserves its chirality.

[0054] Focal Length: A measure of the convergence or divergence of light in an optical system. It refers to the distance from the optical center of the lens to the focus of light when parallel light is incident.

[0055] Step 1: Camera calibration

[0056] Principle

[0057] In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters. Camera calibration is an important step in the fruit thinning process, which enables the robot arm to accurately position and cut.

[0058] The core of camera calibration is to determine the relationship between the three-dimensional geometric position of the object surface and its corresponding point in the image through the calibration plate, establish a projection imaging model, and solve the intrinsic parameters, extrinsic parameters and distortion parameters through matrix transformation calculation.

[0059] Key concepts:

[0060] Radial Distortion: The lens of an actual camera always produces significant distortion at the edge of the imager. This phenomenon comes from the "barrel" or "fisheye" effect.

[0061] Tangential Distortion: Tangential distortion is caused by defects in lens manufacturing that make the lens itself not parallel to the image plane.

[0062] Rotation matrix: A matrix that, when multiplied by a vector, has the effect of changing the direction of the vector without changing its magnitude and preserves its chirality.

[0063] Focal Length: A measure of the convergence or divergence of light in an optical system. It refers to the distance from the optical center of the lens to the focus of light when parallel light is incident.

[0064] The specific steps are as follows:

[0065] 1. Introduce the camera coordinate system and the world coordinate system: The world coordinate system (world coordinate), also known as the measurement coordinate system, is a three-dimensional rectangular coordinate system that can be used as a reference to describe the spatial position of the camera and the object to be measured. The position of the world coordinate system can be freely determined according to actual conditions. The camera coordinate system (camera coordinate) is also a three-dimensional rectangular coordinate system. The origin is located at the optical center of the lens. The x and y axes are parallel to the two sides of the image plane respectively, and the z axis is the optical axis of the lens, which is perpendicular to the image plane. There are the following axial rotation matrices R1, R2, and R3:

[0066]

[0067] The result is a rotation matrix R=R1R2R3.

[0068] Add translation t = (a, b, c) T , then for any point (x ω, ,y ω ,z ω ) T , which can be mapped to the camera coordinate system by the following transformation:

[0069]

[0070] 2. Introduce pixel coordinate system and image coordinate system: The pixel coordinate system is a two-dimensional rectangular coordinate system that reflects the arrangement of pixels in the camera CCD / CMOS chip. Its origin is located in the upper left corner of the image, and the two coordinate axes are parallel to the two sides of the image plane. The unit of the pixel coordinate system is pixel point, which is not conducive to coordinate transformation. Therefore, it is necessary to establish an image coordinate system. Its unit is usually millimeter (mm). The origin is the intersection of the camera optical axis and the image plane (called the principal point). The point (x, y) in the pixel coordinate system and the point (u, v) in the image coordinate system have the following scaling and translation relationship:

[0071]

[0072] Can be organized into a transformation matrix

[0073]

[0074] 3. Convert the camera coordinate system to the image coordinate system: Through perspective projection, the mapping from the camera coordinate system to the image coordinate system can be obtained:

[0075]

[0076] Where Z c is the scale factor of the camera imaging.

[0077] 4. Combine the above transformations to get the transformation from world coordinate system to image coordinate system:

[0078]

[0079] in, is the external parameter of the camera, that is, the rotation and translation transformation relationship of the camera relative to the world coordinate system; is the intrinsic parameter of the camera, which is the inherent property of the camera; Z c Indicates the distance of an object from the optical center.

[0080] 5. Correct camera distortion: Distortion is an inherent characteristic of the camera itself and cannot be avoided during operation. However, distortion is an inherent characteristic of the camera itself and can be corrected after calibration once, without the need to recalibrate each time.

[0081] 5.1 Radial distortion: The lenses of actual cameras always produce significant distortion at the edges of the imager. This phenomenon comes from the "barrel" or "fisheye" effect. Light is bent more at the center of the lens than near the center. This phenomenon is more serious for commonly used ordinary lenses. Barrel distortion is very severe in cheap network cameras, but not obvious in high-end cameras because these lens systems do a lot of work to eliminate radial distortion. For radial distortion, the distortion is 0 at the center of the imager (optical center), and the distortion becomes more severe as you move to the edge. Radial distortion can be corrected using the following Taylor series expansion:

[0082] x corrected =x(1+k1r 2 +k2r 4 +k3r 6 )

[0083] y corrected =y(1+k1r 2 +k2r 4 +k3r 6 )

[0084] Where (x corrected ,y corrected ) is the real coordinate obtained by correction after distortion in the image, and r is the distance from the point to the imaging center.

[0085] 5.2 Tangential distortion: Tangential distortion is caused by defects in lens manufacturing that make the lens itself non-parallel to the image plane. It can be corrected by the following formula:

[0086] x correct =x+[2p1y+p2(r 2 +2x 2 )]

[0087] y correct =y+[2p1x+p2(r 2 +2y 2 )]

[0088] Where x and y are the real coordinates obtained by correcting the distortion in the image. R is the distance from the point to the center of the image.

[0089] 6. In this embodiment, Zhang's calibration method is used for calibration, which obtains multiple images with spatial coordinates by photographing a chessboard calibration plate from multiple directions, and calculates the intrinsic and extrinsic parameters and distortion parameters of the camera by inversely solving the above transformation matrix.

[0090] In the step 2, the YOLO algorithm learns and recognizes the appearance of the fruit and is responsible for scanning the fruit image in the tomato planting area, thereby identifying the state of the fruit on the plant and determining whether to crop it.

[0091] Principle

[0092] The core idea of ​​the YOLO algorithm is to use the whole image as input and regard the target detection task (in this invention, it is the real-time picture of the drone operation scene) as a single regression problem. It detects the target category, position and confidence through single-stage detection, grid division, multi-scale feature fusion, loss function design and other technical means.

[0093] YOLOv5 is the fifth version of the YOLO (You Only Look Once) target detection series. It improves the detection accuracy and speed based on the previous four versions of the algorithm. First, YOLOv5 adopts an improved network structure, which improves the network representation and learning capabilities by introducing deeper network layers and more convolution modules. Secondly, YOLOv5 uses feature fusion technology to fuse feature maps of different levels and scales, greatly improving the accuracy of target detection. In addition, YOLOv5 uses Mosaic data enhancement at the input end, referring to the CutMix data enhancement method. Mosaic data enhancement increases the number of images from the original two to four for splicing, and randomly scales, randomly crops, and randomly arranges the images. Using data enhancement can improve the problem of imbalanced data among small, medium, and large targets in the data set.

[0094] Key concepts:

[0095] 1. Grid cell: The input image is divided into a fixed number of cells, and each grid cell is responsible for detecting an object in the image.

[0096] 2. Convolutional layer: An important hierarchical structure in convolutional neural networks (CNNs). It extracts local features of the input image through convolution operations and generates feature maps for subsequent processing and analysis. The formula is as follows:

[0097]

[0098] 3. Pooling layer: A special convolutional layer used to downsample the input feature map, reduce the number of model parameters, and improve the generalization ability of the model. It imitates the human visual system to reduce the dimension of the data and represents the image with higher-level features. The purpose of implementing pooling includes reducing information redundancy, improving the scale invariance of the model, and preventing overfitting.

[0099] 4. Anchor box: used to predict the shape and position of the target box. The YOLO algorithm uses multiple anchor boxes to predict target boxes of different sizes and length ratios, which improves the accuracy of detection.

[0100] 5. Feature extraction: Extract image features through convolutional neural networks (such as Darknet) for subsequent target prediction.

[0101] 6. Object prediction: For each grid cell, predict the category of the object contained in the grid, the coordinates of the center point, and the width and height of the target box to achieve object detection.

[0102] 7. Feature fusion: Fuse feature maps of different levels and scales to adapt to objects of different scales and improve detection performance.

[0103] 8. Loss function design: Design a comprehensive loss function that takes into account the accuracy of target location, confidence, and category prediction, and optimizes network parameters by minimizing the loss function. The specific loss function formula is as follows:

[0104] 8.1: Position loss formula:

[0105]

[0106] 8.2: Size loss formula:

[0107]

[0108] 8.3: Confidence loss formula:

[0109]

[0110] 8.4: Prediction category loss:

[0111]

[0112] The algorithm steps are as follows:

[0113] Step 1: Use multiple cameras to take multi-view images of tomato plants, and transmit them to the neural network through the input and Mosaic data enhancement. The incoming images are pre-processed by distortion correction, noise removal, etc. to facilitate subsequent image processing.

[0114] Step 2: Send the processed image to the network backbone. In the network backbone, the image will first pass through the Conv module. The Conv module will first perform a convolution operation on the input image, and then perform a BatchNormalization operation on the convolved image to keep the input of each layer of the neural network in the same distribution, thereby accelerating convergence and preventing overfitting, while alleviating the gradient vanishing phenomenon of the low-level neural network during back propagation. The data processed by the Conv module and the BN operation is imported into the SiLu activation function, and its formula is: silu(x)=x*6(x). The data is then sent to the C3 module, which contains 3 standard convolutional layers, and the number is determined by the product of the n and depth_multiple parameters in the configuration file yaml. This module is the main module for learning residual features. Its structure is divided into two branches, one uses a specified number of Bottleneck stacks, and the other only passes through a basic convolution module, and finally the two branches are concat. Finally, the data is sent to the SPPF module, where a standard convolution module is used to halve the input channels, and then max pooling is performed with kernel-sizes of 5, 9, and 13 (for different kernel sizes, padding is adaptive). The results of the three max pooling operations are concat- ed with the data without pooling operations. The number of channels after the final merging is twice the original number, which can greatly improve the accuracy of complex multi-target detection.

[0115] Step 3: Import the data transformed by the above modules into the Neck module. YOLOv5 adopts the PANet structure in the Neck module, extracts the feature hierarchy within the network on the FPN (feature pyramid network), and introduces a bottom-up path. After the top-down feature fusion, the bottom-up feature fusion is performed again, so that the shallow position information can also be transmitted to the deep layer, thereby enhancing the network's positioning ability at multiple scales.

[0116] Step 4: Put the basically processed data into the Head module for target detection. Head expands the number of channels of the feature maps of different scales obtained in Neck through 1×1 convolution. The number of feature channels after expansion is (number of categories + 5) × number of anchors on each detection layer. The additional 5 channels correspond to the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box. The confidence here indicates the credibility of the prediction box. The value range is (0,1). The larger the value, the more likely there is a target in the prediction box. YOLOv5 uses a matching strategy based on the aspect ratio to screen the prior box and divide it into positive samples and negative samples. After the prediction is completed, a large number of redundant bounding boxes will be generated. The Non-maximum suppression method is used to calculate and compare the IOU (intersection of Union) of different bounding boxes with the highest confidence bounding box. If the IOU threshold of a bounding box and the highest confidence bounding box is greater than the IOU threshold we set, it means that the same object is predicted by two repeated bounding boxes, then remove this bounding box to reduce redundancy.

[0117] Step 5: Optimize the loss function. The total loss is composed of four types of losses: position loss, size loss, confidence loss, and prediction category loss. The calculation function formulas of the four losses are as follows:

[0118]

[0119] By calculating and comparing the loss function, if the loss value is less than the specified threshold, execute step 6; if the loss value is greater than the specified threshold, backpropagate the algorithm, repeat steps 1 to 4, optimize the parameters, and improve the detection accuracy of the YOLOv5 algorithm.

[0120] Step 6: Output the result. Finally, the identified target object (in this invention, small fruit, diseased fruit, or other tomato fruit that needs to be cut off) and its location information are output in the form of a bounding box in the image, which is convenient for the robot arm to remove the tomato in the subsequent process.

[0121] Step 3: Reconstruct the 3D mesh model of the tomato plant.

[0122] Principle description:

[0123] The present invention aims to remove tomatoes with poor growth. In the subsequent operation process, it is necessary to obtain the three-dimensional spatial position information of the target tomatoes, and at the same time, it is necessary to avoid obstacles such as the stems and leaves of the tomato plants and tomatoes with good growth during the removal process. Therefore, this invention adopts the SFM method to reconstruct the three-dimensional point cloud model of the tomato plant based on images taken by multiple cameras. Then, according to the distribution of the three-dimensional point cloud, the three-dimensional space is divided into a grid model.

[0124] Key concepts:

[0125] 1. Projection Matrix: The projection matrix is ​​a linear transformation matrix that maps data in three-dimensional space to the captured image of the camera.

[0126] 2. Least squares method: When solving an overdetermined system of equations, the least squares method finds a set of parameters by minimizing the sum of squares of the residuals (i.e., the difference between the observed values ​​and the model predicted values) so that the model predicted values ​​are as close as possible to the observed values.

[0127] The algorithm steps are as follows:

[0128] Step 1. Perform distortion correction on multiple images taken by multiple cameras at the same time, and use the SURF algorithm to obtain matching point pairs in the corrected images.

[0129] The SURF algorithm is an algorithm used for feature point extraction and description in image processing and computer vision. It mainly includes image preprocessing, scale space construction, feature point extraction, feature point main direction assignment, feature point descriptor generation, feature point matching and other steps.

[0130] Step 2: Use the intrinsic parameter matrix K of any camera obtained in step 1 i and the external parameter matrix [P i ,y i ], calculate the projection matrix of the camera:

[0131]

[0132] Step 3: For the obtained matching point coordinates, the positions of the corresponding 3D points are calculated through the triangulation algorithm to form a sparse 3D point cloud. The specific steps are as follows:

[0133] For a set of matching points under different perspectives There exists a point X=(x,y,z,1) in three-dimensional space corresponding to this set of matching points, and satisfies the following set of equations:

[0134] X i =P i X T ,i=1,2,...,N.

[0135] Arranged

[0136]

[0137] This system of equations is generally an overdetermined system of equations, and due to the noise of the acquired image and the error of the matching algorithm, this system of equations usually has no solution, so the least squares method is used to solve the approximate solution of X.

[0138] Step 4: Then, according to the recognition results of the tomato to be cut and the intact tomato plant in step 2, the 3D point cloud is divided into the following three types of points: 1. Points matching the tomato to be cut. 2. Other points contained in the tomato plant except the tomato to be cut. 3. Other points that do not belong to the tomato plant.

[0139] Step 5: Divide the space to be operated into three-dimensional grids, and then divide any grid into the following three categories: 1. If the grid contains type 1 points in step 4 and does not contain type 2 points, it is recorded as a target grid; 2. If the grid contains type 2 points, it is recorded as an obstacle grid; 3. If the grid does not contain any point in the three-dimensional point cloud, or only contains type 3 points, it is recorded as a movable path.

[0140] Step 4: Path planning

[0141] Principle

[0142] Path planning is the core part of the robotic arm shearing system. It is responsible for generating the optimal path from the starting point to the tomato to be cut at the end of the robotic arm to avoid accidentally damaging other fruits or plants. The present invention adopts the JPS algorithm, which is a path planning algorithm suitable for quickly calculating the optimal path. It is optimized based on the A* algorithm and can more efficiently find the optimal path.

[0143] The core idea of ​​the JPS (Jump Point Search) algorithm is to reduce the number of nodes that need to be evaluated by maintaining a dynamic open list, which contains all known nodes with the lowest cost. The algorithm uses the heuristic function h(n) to estimate the cost from the current node n to the target point, and searches for forced neighbors and jump points to find the path with the lowest cost more efficiently.

[0144] Key concepts:

[0145] 1.openList: A collection of valid points needs to be saved during the pathfinding process. This list stores the points that can be explored.

[0146] 2.closeList: A collection of valid points needs to be saved during the pathfinding process. This list stores the points that have been explored.

[0147] 3.g(n): represents the cost from the starting node of the current record to node n.

[0148] 4.h(n): represents the estimated cost from the currently recorded node n to the target point.

[0149] 5.f(n): represents the sum of the cost from the starting node to node n and from node n to the end point of the current record. That is:

[0150] f(n)=g(n)+h(n)

[0151] 6. Forced neighbor: If there is an obstacle among the adjacent nodes of node n, and the distance cost of node n's parent node p to reach node m through node n is smaller than the distance cost of any path to node m without passing through node n, then node m is called a forced neighbor of node n.

[0152] 7. Jump point:

[0153] If node n is a starting point or a destination point, then n is a hop point.

[0154] If node n has neighbors and they are forced neighbors, then n is a hop point.

[0155] If the parent node to node n is a diagonal movement and node n can reach the jump point by moving horizontally or vertically, then n is a jump point.

[0156] The algorithm steps are as follows:

[0157] Step 1: Initialize, create openList to store the points that can be explored, and closeList to store the points that have been explored to prevent repeated searches. Grid the three-dimensional space, and treat each grid as a node. Mark the obstacles in the grid. Determine the starting node and the target node, and add the starting node to the openList.

[0158] Step 2: Define the node evaluation function f(n) = g(n) + h(n), and use Manhattan distance to calculate the cost from the current node to the target node and the starting node. The Manhattan distance calculation formula is as follows:

[0159] d(i,j)=|x i -x j |-|y i -y j |

[0160] Where i and j are two target points, and d(i,j) represents the Manhattan distance between the two points.

[0161] Step 3: Starting from the starting node, search other nodes, first search in a straight line, then search in an oblique direction. If an obstacle, jump point or boundary is found in a certain direction, the search ends. If a jump point is found, it is added to the openList.

[0162] Step 4: If the search in the diagonal direction is not completed in a single round, move forward in the diagonal direction and repeat step 3 until the search is completed. If all directions starting from this node have been searched, remove this node from the openList and add it to the closeList. The specific method is as follows:

[0163] Step 4.1: Assign g(n) = infinity to all other nodes in the graph

[0164] Step 4.2: Loop (until the queue is empty or the target node is found), if the queue is originally empty, return false

[0165] Step 4.3: For all unexpanded neighbors of node n, first search in all straight line directions. If the boundary is touched, stop exploring in this direction; if the jump point is touched, mark the touched jump point as node m and stop searching in this direction.

[0166] Step 4.4: Then, perform a single-step search in all oblique directions. If the boundary is touched, stop the exploration in this direction; if the jump point is touched, mark the touched jump point as node m and stop the search in this direction. If the boundary or jump point is not touched, search in the horizontal decomposition direction of the current direction of travel. If the boundary is touched, stop the exploration in this direction; if the jump point is touched, mark the touched jump point as node m and stop the search in this direction.

[0167] (If n has a parent node, the search direction only needs to be the direction of the line connecting the parent node and n, the horizontal decomposition direction, and the direction of the line connecting n and the mandatory neighbor)

[0168] Step 4.5: If g(m) = infinity, it means that node m is not expanded, so let g(m) = g(n) + Cost mn , where Cost mn Indicates the distance cost between node n and node m. And add node "m" to openList. (#openList.push(m)), delete n from openList and add it to closeList

[0169] Step 4.6: If g(m)>g(n)+Cost mn , then let g(m)=g(n)+Cost mn , do not add it to the openList, and delete n from the openList and add it to the closeList. This operation greatly reduces the number of nodes in the openlist of the JPS algorithm, greatly improving the efficiency of the algorithm.

[0170] Step 4.7: End the loop.

[0171] Step 5: Take the node with the lowest cost in openList, i.e. the node with the smallest f(n), as the new parent node, search all straight and diagonal directions, and repeat steps 3 and 4. Until openList is empty or the target node is found. If openList is empty and the target node has not been found, false is returned. If the target node is found, connect all the points in closeList to get the desired path.

[0172] Step 5: Use the algorithm to control the robotic arm to remove the target fruit.

[0173] Principle description:

[0174] The present invention uses a six-axis (six-degree-of-freedom) robotic arm to perform pruning operations. After capturing the tomato image, the tomatoes to be removed are marked by identification. In step one, the coordinate system has been unified to the world coordinate system through camera calibration and the distortion of camera shooting has been eliminated. The marked tomatoes are placed in the coordinate system to facilitate subsequent cropping by the robotic arm. Finally, the tomatoes are cropped by adjusting the robotic arm posture and kinematic modeling to complete the operation.

[0175] The core idea of ​​this operation is to adjust and manipulate the robotic arm based on kinematic modeling, by controlling the joints and links to control the rigid body position and posture, thereby controlling the robotic arm to accurately prune tomato fruits.

[0176] Key concepts:

[0177] 1. Rotation matrix: A matrix that changes the direction of a vector but does not change its magnitude when multiplied by a vector and maintains its chirality. It is used to describe the posture of a rigid body. It consists of three parts: basic rotation, vector rotation, and rotation matrix.

[0178] 2. Euler angles: used to describe the orientation of a rigid body relative to a fixed coordinate system. They can also represent the orientation of a moving reference system or the orientation of a basis in three-dimensional linear algebra.

[0179] The algorithm steps are as follows:

[0180] 1. Describe the position of the rigid body: Let O-x0y0z0 be a spatial Cartesian coordinate system, a0, b0, c0 are unit vectors parallel to the coordinate axes, and O1 is the center of mass of the rigid body. The position of point O' relative to the reference coordinate system can be expressed as O' = O 1x a+O 1y b+O 1z c, where O 1x a,O 1y b,O 1z c is the component of vector O1 on the reference coordinate axis. Therefore, the position of the center of mass of the rigid body relative to the reference coordinate system can be expressed as a 3*1 vector, which is the position vector, as follows:

[0181]

[0182] 2. Description of the rigid body posture: Let O-x1y1z1 be a Cartesian coordinate system in space and fixed to the rigid body. a1, b1, c1 are unit vectors of the rigid body coordinate system, and the components of each unit vector are the direction cosines of the coordinate axis of the rigid body coordinate system O-x1y1z1 relative to the reference coordinate system O-x0y0z0. a1, b1, c1 can be expressed in the reference coordinate system O-x0y0z0 as:

[0183]

[0184] The position of a rigid body in space can be represented by the position of the center of mass of the rigid body relative to the reference coordinate system, while the posture of the rigid body in space can be represented by the three components of the unit vector of the coordinate system fixed to the rigid body relative to the reference system. The vector needs to be converted into a rotation matrix to further visualize the posture of the rigid body and facilitate subsequent kinematic analysis. Each unit vector of the new coordinate system O-x0′y0′z0′ can be described by its component relative to the reference coordinate system O-x0y0z0. That is:

[0185]

[0186] Therefore, the rotation matrix of the new coordinate system O-x0'y0'z0' about the coordinate axis z0 is:

[0187]

[0188] Similarly, there is a rotation matrix about the coordinate axis x0, y0:

[0189]

[0190] Then we can use the rotation matrix R z (α)R y (β)R x (γ) is used to represent the position of the rigid body.

[0191] 3. Trajectory planning: After determining the position and posture of the rigid body, the rigid body is kinematically modeled to complete the trajectory planning. The present invention mainly uses the Denavit-Hartenberg method to perform kinematic modeling.

[0192] Select axis z along the direction of joint i+1 i ;

[0193] The origin o i Placed on axis z i With axis z i-1 At the intersection of the common perpendicular lines;

[0194] The origin o i Further placed on the common perpendicular line and axis z i-1 At the intersection of

[0195] Along the z axis i-1 and z i The common perpendicular line is selected as axis x i , the direction is from joint i to joint i+1;

[0196] Finally, select the y axis i Make it equal to x i and z i A right-handed coordinate system is formed.

[0197] The definition of the joint coordinate system given by the DH method is not unique when the following situations occur:

[0198] For coordinate system 0, only the direction of axis z0 is specified, so x0 can be chosen arbitrarily;

[0199] For coordinate system n (i.e., degree of freedom n of the robot arm), since joint n+1 does not exist, although x n Must be aligned with axis z n-1 vertical, but z n The definition of is not unique. Usually, if joint n is a revolute joint, z n Along the z n-1 direction;

[0200] When two adjacent rotation axes are parallel, the common perpendicular line between them is not unique;

[0201] When two adjacent rotation axes intersect, x i The direction can be chosen arbitrarily;

[0202] When joint i is mobile, z i-1 The direction can be arbitrary.

[0203] 4. According to the path planning in step 4, the final planning of the motion trajectory of the end of the robotic arm is completed, thereby completing the process of automated pruning of tomatoes.

[0204] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A fruit thinning robot, comprising a robot chassis, a six-axis robotic arm, a multi-position fixed camera and a control system, characterized in that: The robot chassis is used for autonomous movement, the multi-camera fixed camera is fixedly connected and installed with the robot chassis, the multi-camera fixed camera is composed of a plurality of cameras with different postures, the six-axis robotic arm is movably installed with the robot chassis, an electric cutter is provided at the end of the six-axis robotic arm, and the electric cutter is used for cutting inferior fruits and small fruits.

2. A fruit thinning robot according to claim 1, characterized in that: The control system includes a calibration system, a three-dimensional reconstruction system, an intelligent recognition system, a path planning system and a motion control system; The calibration system is used to perform camera calibration on an imaging system composed of multiple fixed cameras; The intelligent identification system is used to identify inferior fruits and small fruits; The path planning system is used to plan the optimal travel path of the six-axis robot; The motion control system is used to control the six-axis robot arm to move along a path so that the electric cutter cuts inferior fruits and small fruits.

3. A fruit thinning method, characterized in that: The operation using the fruit thinning robot according to any one of claims 1-2 comprises the following steps: Step 1: Use multiple fixed cameras to calibrate the camera to determine the internal and external parameters and distortion parameters of each camera; Step 2: Use the YOLO algorithm to learn and identify the appearance of the fruit; Step 3: Reconstruct the 3D point cloud model of the plant; Step 4: Path planning of the six-axis robot end; Step 5: Use electric trimmers to trim and thin the fruits; Step 6: Repeat steps 2 to 5 until no more fruit can be identified on the plant that need to be cut.

4. A fruit thinning method according to claim 3, characterized in that: In the step 1, a geometric model of camera imaging is established to determine the relationship between the three-dimensional geometric position of a point on the surface of the space object and its corresponding point in the image, so that the six-axis robot arm can be accurately positioned and cut by the electric cutter.

5. A fruit thinning method according to claim 3, characterized in that: In the step 2, the YOLO algorithm learns and recognizes the appearance of the fruit and is responsible for scanning the fruit image in the tomato planting area, thereby identifying the state of the fruit on the plant and determining whether to crop it.

6. A fruit thinning method according to claim 3, characterized in that: In the step three, the three-dimensional point cloud model of the tomato plant is reconstructed based on the images taken by multiple cameras. According to the distribution of the three-dimensional point cloud, the three-dimensional space is divided into a grid model, and the poorly growing fruits are removed. At the same time, obstacles such as the stems and leaves of the plant and the well-growing fruits need to be avoided during the removal process.

7. A fruit thinning method according to claim 3, characterized in that: The path planning in step 4 is responsible for generating the optimal path from the starting point to the fruit to be cut at the end of the six-axis robot arm to avoid accidental damage to other fruits or plants.

8. A fruit thinning method according to claim 3, characterized in that: In the step five, after the fruit image is captured, the fruit to be removed is marked by identification. In the step one, the coordinate system has been unified with the world coordinate system through camera calibration and the distortion of the camera shooting has been eliminated. The marked fruit is placed in the coordinate system to facilitate the subsequent cutting by the robotic arm.

Images