A plum picking and grading robot and picking method

Abstract

This invention provides a plum picking and grading robot, relating to the field of fruit picking, including a base support mechanism (10), a collection box, a grading and sorting mechanism, a picking robotic arm (40), an end effector (50), a storage mechanism, and a visual recognition system; the collection box is located at the front end of the base support mechanism (10), including a good fruit collection trough (21) and a bad fruit collection trough (22); the grading and sorting mechanism and the picking robotic arm (40) are respectively located on the end face of the base support mechanism (10), and the end effector (50) is located at the end of the picking robotic arm (40) away from the base support mechanism (10), and the storage mechanism is located on the end face of the base support mechanism (10); the visual recognition system includes a positioning binocular camera (71) and two recognition binocular cameras (72). This robot can realize automatic picking and sorting in the plum picking process, effectively improving picking efficiency, improving picking accuracy, and avoiding problems such as mispicking and missed picking.

Description

Technical Field

[0001] This invention relates to the field of fruit picking technology, specifically to a plum picking and grading robot and picking method. Background Technology

[0002] The plum is a fruit rich in nutrients and with a unique flavor. It is known as a third-generation functional fruit, rich in vitamins A, B, and C, which are essential for the body's immune system and metabolism, as well as minerals such as calcium, iron, zinc, and potassium, which are helpful for bone health, blood circulation, and cell function.

[0003] Currently, Kashgar region in Xinjiang is one of the main distribution and production areas of plum cultivation resources and fruit in my country. Due to the significant differences in the ripening period of plums and the stringent quality requirements during harvesting, the plum harvest still relies primarily on manual selective picking. This method is labor-intensive, inefficient, and costly. Furthermore, it demands a high level of expertise from the pickers, which can lead to missed harvest times and severely impact the yield and quality of the plums. With the popularization of mechanized and automated agriculture, some regions are gradually adopting robots for plum harvesting, which can effectively reduce labor costs and intensity and improve harvesting efficiency. However, due to the special characteristics of plums (i.e., significant differences in ripening period) and their delicate nature, existing plum harvesting robots often suffer from drawbacks such as complex structures, high costs, easy damage to fruit during harvesting, inability to simultaneously perform picking and sorting, and the need for manual separation. These shortcomings prevent maximizing harvesting efficiency and can increase the rate of defective fruit during the harvesting process. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention aims to provide a plum picking and grading robot. This robot can automatically pick and sort plums during the picking process, thereby effectively improving picking efficiency and avoiding unnecessary screening steps before and after picking. At the same time, the robot can effectively protect the plums during the picking process, avoiding damage to the plums caused by picking, thus ensuring fruit quality while improving picking efficiency.

[0005] Another objective of this invention is to provide a harvesting method for the aforementioned plum picking and grading robot.

[0006] A plum picking and grading robot includes a base support mechanism, a collection box, a grading and sorting mechanism, a picking robotic arm, an end effector, a storage mechanism, and a visual recognition system. The collection box is located at the front end of the base support mechanism and includes a good fruit collection slot and a bad fruit collection slot. The grading and sorting mechanism and the picking robotic arm are respectively located on the end face of the base support mechanism, and the end effector is located at the end of the picking robotic arm away from the base support mechanism. The storage mechanism is located on the end face of the base support mechanism and is connected to both the end effector and the grading and sorting mechanism. The visual recognition system includes one positioning binocular camera and two recognition binocular cameras. The positioning binocular camera is located on the end effector, and the recognition binocular cameras are located on the end face of the base support mechanism corresponding to the grading and sorting mechanism.

[0007] Based on further optimization of the above scheme, the base support mechanism includes a chassis, support studs, a connecting plate, track wheels and tracks. Multiple evenly distributed vertical support studs are provided on the end face of the chassis, and the end of the support studs away from the chassis is connected to the same connecting plate. Two track wheels are provided on each side of the chassis, and the two track wheels on the same side are connected by tracks. The bad fruit collection trough is provided between the chassis and the connecting plate, and the good fruit collection trough is provided at the front end of the chassis and is fixedly connected to the bad fruit collection trough.

[0008] Based on further optimization of the above scheme, the grading and screening mechanism includes a screening disc, a first baffle, a first slide rail, a second baffle, a slide rail for damaged fruit, a slide rail for good fruit, and a second slide rail. The screening disc is fixedly mounted on the end face of the connecting plate and has a conical structure with a larger diameter at the top and a smaller diameter at the bottom. A coaxial rotating shaft is provided in the middle of the screening disc, and a first baffle is provided on the outer wall of the rotating shaft. The first baffle rotates within the screening disc via the rotating shaft. A discharge port is opened on the side of the screening disc near the good fruit collection trough for discharging material. A first slide is provided at the opening, which is an inclined track structure that slopes from the screening disc to the good fruit collection trough. The end of the first slide away from the screening disc is connected to the good fruit slide, and a bad fruit slide is provided on the side of the end of the first slide away from the screening disc. A second baffle is rotatably provided at the connection between the good fruit slide and the bad fruit slide. A discharge hole is opened on the connecting plate at the end of the bad fruit slide away from the first slide, and the discharge hole corresponds to the bad fruit collection trough. The end of the good fruit slide away from the first slide is connected to the second slide provided in the good fruit collection trough.

[0009] Based on further optimization of the above scheme, the end effector includes a fixed housing, a drive motor, a first gear, a second gear, a first rack, a first protective ring, a second rack, a pressing plate, and a second protective ring. The fixed housing is located at the end of the harvesting robot arm away from the base support mechanism, and the drive motor is located at the bottom of the inner cavity of the fixed housing. The output shaft of the drive motor is coaxially mounted with a gear shaft, and the end of the gear shaft away from the drive motor is rotatably connected to the top surface of the inner cavity of the fixed housing. The outer wall of the gear shaft is sequentially fitted with the first gear and the second gear from bottom to top. The inner wall of one side of the fixed housing is slidably mounted with the first rack corresponding to the first gear, and the end of the first rack away from the harvesting robot arm is mounted with a first rack via a sliding support. The first protective ring has one end away from the first rack that passes through the corresponding side wall of the fixed housing and is slidably connected. A separation block is provided on the top surface of the end of the first protective ring away from the fixed housing, and a separation groove with a built-in separation blade is opened on the side of the separation block near the fixed housing. The second rack is slidably set on the other inner wall of the fixed housing (i.e., the inner wall opposite to the first rack) and corresponding to the second gear. The end of the second rack near the first protective ring passes through the corresponding side wall of the fixed housing and is provided with a pressing plate. A top block is provided on the side of the pressing plate near the separation block corresponding to the separation groove. The second protective ring is fixedly set on the outer wall of the fixed housing and below the first protective ring. The second protective ring is set parallel to the first protective plate.

[0010] To ensure smooth cutting by the end effector, based on further optimization of the above scheme, a limiting rod is provided on the side of the extrusion plate away from the top block and on the side of the first rack. The limiting rod passes through the corresponding side wall of the fixed housing, and a limiting block is provided on the side of the limiting rod away from the extrusion plate.

[0011] Based on further optimization of the above scheme, both the first protective ring and the second protective ring include an inner ring and an outer ring, the inner ring being made of plastic and the outer ring being made of biopolymer material.

[0012] Based on further optimization of the above scheme, the collection mechanism includes a connecting hose, a buffer mechanism, a transition collection box, and a guide chute. The transition collection box is fixedly installed on the end face of the base support mechanism and located on one side of the screening disc. The distance between the bottom surface of the transition collection box and the end face of the base support mechanism is greater than the distance between the top surface of the screening disc and the end face of the base support mechanism (i.e., the height of the transition collection box is greater than the height of the screening disc). One end of the connecting hose is located on the bottom surface of the second protective ring and communicates with the inner ring of the second protective ring, and the other end communicates with one side of the transition collection box. A buffer mechanism is provided at the connection between the connecting hose and the transition collection box (e.g., buffering is achieved through evenly distributed rubber protrusions). A guide chute is provided on the side of the transition collection box near the screening disc. The guide chute is an inclined groove that slopes from the transition collection box towards the screening disc.

[0013] Based on further optimization of the above scheme, the bottom surface of the inner cavity of the transition collection box is set as an inclined structure that slopes from the transition collection box to the screening disc (the inclination of the inclined structure is consistent with the inclination of the guide chute), and the inner cavity wall of the transition collection box is uniformly covered with a sponge buffer layer.

[0014] A harvesting method using the aforementioned plum harvesting and grading robot includes:

[0015] Step S1, Identification and Positioning: The binocular camera is positioned at a distance from the plum fruit in a horizontal orientation to scan and obtain an image of the plum fruit in the distance. The image is then segmented and the target is identified.

[0016] Step S2, Selection of new plum fruit clusters: The target new plum fruit is tracked using a target tracking algorithm to identify the number of mature new plum fruit; when the number of mature new plum fruit in the same cluster exceeds 60% of the total number of fruit in the cluster, the cluster is marked as the target new plum fruit cluster;

[0017] Step S3, Picking point location: Select the lowest point of the target new plum fruit cluster as the picking point reference position, and move it up and down 3-5cm along the lowest point of the picking point reference position in the longitudinal direction of the fruit to obtain the picking point.

[0018] Step S4, Fruit Harvesting: The harvesting robotic arm is moved step by step according to the harvesting point to further identify and locate the new plum fruit in close-up view; when the end effector reaches the harvesting point, the relative distance between the harvesting point and the upper edge of the new plum fruit (i.e., the apex of the new plum fruit) is obtained, and then the distance is increased by 3-5 cm in the longitudinal direction based on the base to obtain the end point of the end effector; the end effector moves from bottom to top according to the harvesting point and the end point to realize the envelopment and shearing separation of the new plum fruit, thus completing the harvesting;

[0019] Step S5, Fruit sorting and collection: After harvesting, the fruits fall into the grading and sorting mechanism through the collection mechanism. After sorting by the grading and sorting mechanism, good fruits and bad fruits are separated and collected in the good fruit collection trough and bad fruit collection trough respectively.

[0020] Based on further optimization of the above scheme, step S1 specifically includes:

[0021] Step S11: First, perform histogram calculation on the input image to obtain the frequency distribution of each pixel value in the image; then sample the target image and divide its histogram interval into M sub-intervals, setting the width of each sub-interval to U; then, map the image grayscale values ​​to each sub-interval and calculate the pixel frequency in each sub-interval.

[0022] Step S12: First, calculate the gray level distribution of all pixel values ​​in each sub-interval to obtain the gray level mean in each sub-interval, and calculate the intra-class variance of each sub-interval to represent the degree of dispersion of pixel values ​​in that interval; then, iterate through all possible thresholds, calculate the intra-class variance corresponding to each threshold, and select the threshold that maximizes the intra-class variance as the optimal threshold.

[0023] Step S13: Divide the image into two regions, foreground and background, according to the optimal threshold: the region greater than the optimal threshold is the foreground and the region less than the optimal threshold is the background, thereby separating the target image from the background.

[0024] Step S14: Optimize the segmented image using grayscale morphology to achieve image segmentation and recognition, and divide and locate the sub-regions of the segmented image.

[0025] Based on further optimization of the above scheme, the specific steps in step S2, which involve using a target tracking algorithm to track the target new plum fruits and identify the number of ripe new plum fruits, are as follows:

[0026] Step S21: First, object detection is applied to detect candidate targets from video frames, and the bounding box and confidence score of each selected target are obtained; then the detected candidate targets are grouped according to their temporal and spatial correlation, and the degree of overlap between two candidate targets is measured.

[0027] Step S22: Match the detection result of the current frame with the trajectory of the previous frame, and use the Hungarian algorithm to obtain the matching method with the minimum cost. The cost includes calculations based on factors such as distance, speed, and confidence. Complete the initial matching. For candidate targets that do not exist in the previous trajectory matching, perform a second matching with the nearest trajectory.

[0028] Step S23: Based on the matching results, generate new trajectories. Each generated trajectory includes a series of bounding boxes and corresponding confidence scores. Then, use a Kalman filter to predict the trajectory position of the next frame and correct the predicted trajectory based on the new detection results.

[0029] Step S24: Preset a confidence score threshold range. Compare the confidence score in the trajectory with the confidence score threshold range. The lower limit of the confidence score threshold range indicates high maturity, and the upper limit of the confidence score threshold range indicates low maturity, thus distinguishing the maturity of the fruit. Then, filter the trajectories according to the trajectory length and confidence score, and remove unstable and unreasonable trajectories to obtain the number of fruits.

[0030] Based on further optimization of the above scheme, the confidence score threshold range is [0,1].

[0031] Based on further optimization of the above scheme, the method for obtaining the position of the lowest point of the new plum fruit in step S3 is as follows:

[0032] First, images from different perspectives are acquired through multi-view image acquisition. Then, feature points are extracted from two or more images to find feature point pairs corresponding to the same target point (i.e., the lowest point, the new plum fruit). Next, for each matching feature point pair, triangulation is used to calculate a 3D point cloud, thereby obtaining the spatial coordinates of the target point. The triangulation depth is used in this process. d for:

[0033]

[0034] In the formula: f Indicates the camera's focal length. b Indicates the distance between images from different viewpoints. This indicates the angle between images viewed from different perspectives.

[0035] Based on the further optimization of the above scheme, the specific steps in step S4 for further identification and localization of the new plum fruit in close-up view are as follows:

[0036] First, the SVM algorithm is used to obtain an optimal classification hyperplane. A model is built, and a linear equation for the hyperplane is defined to divide the sample data into two classes.

[0037]

[0038] In the formula: w Represents the weight vector; X Denotes a hyperplane, with the direction perpendicular to the hyperplane. X These are the sample data points; b This represents the bias term, which determines the distance between the hyperplane and the origin;

[0039] Then, the Lagrange multiplier method is used to solve the problem and obtain the weight vector of the hyperplane linear equation. w With bias term b ;

[0040] Then, the model is optimized by selecting any two variables through iteration.

[0041] Finally, the data is classified using weight vectors and bias terms. The distance from each data point to the hyperplane is calculated, and the category is determined by whether the value is positive or negative. The classification linear equation is as follows:

[0042]

[0043] In the formula: sign() The category of the function range prediction label is -1 or 1;

[0044] By using decision functions to determine the category of pixels in an image, image segmentation and target recognition and localization can be completed.

[0045] Based on further optimization of the above scheme, the method for obtaining the upper edge of the new plum fruit (i.e., the vertex of the new plum fruit) in step S4 is as follows:

[0046] First, obtain the two-dimensional coordinates of the corresponding feature points in the image. u,v The feature point is then calculated by taking its two-dimensional coordinates and its depth information D; and then, its three-dimensional coordinates are obtained.

[0047]

[0048] In the formula: fx , fy Indicates the camera's focal length. cx , cy This indicates the coordinates of the camera's principal point.

[0049] Based on further optimization of the above scheme, the discrimination method for the grading and screening mechanism to screen new plum fruits in step S5 is as follows:

[0050] Step S51, Data Collection: Collect a dataset containing images of various new plum fruits and their corresponding labels; the dataset covers different task labels, including fruit size, presence or absence of cracks, presence or absence of damage, presence or absence of fruit stems, etc.

[0051] Step S52: Set the fruit size as a regression problem, labeled as the size of the new plum fruit: calculate the minimum bounding rectangle of the fruit image, obtain the maximum longitudinal and transverse diameters of the fruit, compare with the average data (the average data is obtained through a large amount of experimental data), and determine whether the fruit size meets the standard; set the presence or absence of cracks as a binary classification problem, labeled as cracked 1 and no crack 0; set the presence or absence of damage as a binary classification problem, labeled as damaged 1 and no damage 0; set the presence or absence of fruit stem as a binary classification problem, labeled as with fruit stem 1 and without fruit stem 0.

[0052] Step S53, Model Design: The ResNet neural network model is used as the screening model. The first four ResNet Blocks of the ResNet neural network model are selected as shared convolutional layers to extract features from the input image. After the shared convolutional layers, multiple task-specific branches are set up. Each branch processes a specific task. Each task-specific branch consists of several convolutional layers and fully connected layers, which are optimized for different tasks.

[0053] Step S54: Train the screening model using a dataset containing multi-task labels, providing corresponding labels for each task during the training process.

[0054] Step S55: After training is complete, use the validation set to evaluate the performance of the screening model and complete the training.

[0055] The following are the technical effects of the present invention:

[0056] This application employs an end effector mechanism composed of a fixed housing, a drive motor, a first gear, a second gear, a first rack, a first protective ring, a second rack, a pressing plate, and a second protective ring. This mechanism enables a harvesting mode that envelops and cuts the fruit from bottom to top. Firstly, it separates target and non-target fruits, accurately grasping individual target fruits and improving harvesting efficiency. Secondly, it avoids damage to non-target fruits during harvesting, keeping them outside the harvesting area and ensuring the overall quality of the cluster. Thirdly, it avoids the influence of the surrounding environment on the target fruit during harvesting, achieving stable and damage-free fruit harvesting, preventing deviations that could damage the fruit and ensuring its integrity and quality. Fourthly, it ensures the overall flexibility of the end effector mechanism, preventing its movement from affecting the fruit cluster and thus impacting positioning and harvesting accuracy, leading to missed or incorrect harvesting. The grading and screening mechanism, consisting of a screening disc, a first baffle, a first slide, a second baffle, a slide for damaged fruit, a slide for good fruit, and a second slide, works in conjunction with an end-effector to achieve simultaneous fruit screening during harvesting. This avoids the need for additional manual screening after harvesting, improving efficiency and preventing fruit damage during subsequent screening processes. Furthermore, the grading and screening mechanism employs a three-stage screening system to identify and screen qualified and unqualified fruits, effectively preventing the mixing of good and bad fruits and thus ensuring overall quality.

[0057] Furthermore, based on the growth characteristics of new plum fruits, this application employs a method combining long-range recognition and positioning with target tracking, along with close-range recognition and positioning, to set the optimal working path. The entire method boasts high accuracy and stability, effectively saving working time and improving harvesting efficiency. Combined with the structural design of the end effector, it efficiently harvests new plum fruits while avoiding damage to the new plum fruits and surrounding fruits, preventing issues such as misharvesting and missed harvesting, and significantly improving the automation level of fruit identification and detection. Moreover, through comprehensive data coverage, precise annotation, optimized model structure, synchronous training, and performance verification for fruit screening, it can accurately and efficiently screen out bad and good fruits, ensuring the quality of harvested fruits. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall structure of the harvesting and grading robot in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the base support mechanism of the harvesting and grading robot in an embodiment of the present invention.

[0060] Figure 3 This is a top view of the harvesting and grading robot in an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of the end effector of the harvesting and grading robot in an embodiment of the present invention.

[0062] Figure 5 for Figure 4 A sectional view along line AA.

[0063] Figure 6 This is a flowchart illustrating the harvesting and grading robot in an embodiment of the present invention.

[0064] Among them, 10. Base support mechanism; 11. Chassis; 12. Support stud; 13. Connecting plate; 130. Feeding hole; 14. Track wheel; 15. Track; 21. Good fruit collection trough; 22. Bad fruit collection trough; 31. Screening disc; 32. First baffle; 33. First slide rail; 34. Second baffle; 35. Bad fruit slide rail; 36. Good fruit slide rail; 37. Second slide rail; 40. Harvesting robotic arm; 50. End effector; 51. Fixing 52. Housing; 53. Drive motor; 54. Gear shaft; 55. First gear; 56. Second gear; 57. First rack; 58. First protective ring; 59. Separating block; 50. Sliding support; 51. Second rack; 52. Extrusion plate; 53. Top block; 54. Limiting rod; 55. Second protective ring; 66. Connecting hose; 67. Transition collection box; 68. Guide chute; 79. Positioning binocular camera; 70. Identifying binocular camera. Detailed Implementation

[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0066] Example 1:

[0067] A new plum picking and grading robot, such as Figure 1 As shown: Includes a base support mechanism 10, a collection box, a grading and sorting mechanism, a harvesting robotic arm 40, an end effector 50, a storage mechanism, and a visual recognition system; Figure 2 As shown: The base support mechanism 10 includes a chassis 11, support studs 12, a connecting plate 13, track wheels 14, and tracks 15. Multiple evenly distributed vertical support studs 12 are provided on the end face of the chassis 11 (in conjunction with...). Figure 1 and Figure 2As shown, the number of support studs 12 is set according to the actual situation. In this embodiment, eight support studs 12 are used. The end of the support stud 12 away from the chassis 11 is connected to the same connecting plate 13 (see...). Figure 2 As shown, the connecting plate 13 is arranged parallel to the chassis 11, and the two sides of the chassis 11 (i.e. Figure 2 Two tracked wheels 14 are respectively installed on the front and rear sides (as shown), and the two tracked wheels 14 on the same side are connected by a track 15. The collection box is located at the front end of the base support mechanism 10 (i.e., Figure 1 (as shown on the lower right side) and it is located between the chassis 11 and the connecting plate 13 (combined) Figure 1 and Figure 2 As shown, it includes a good fruit collection tank 21 and a bad fruit collection tank 22. The bad fruit collection tank 22 is located between the chassis 11 and the connecting plate 13, and the good fruit collection tank 21 is located at the front end of the chassis 11 and the good fruit collection tank 21 and the bad fruit collection tank 22 are fixedly connected.

[0068] The grading and screening mechanism and the harvesting robotic arm 40 are respectively set on the end face of the base support mechanism (specifically the connecting plate 13), combined with Figure 1 and Figure 3 As shown, the grading and screening mechanism includes a screening disc 31, a first baffle 32, a first slide rail 33, a second baffle 34, a sluice rail for damaged fruit 35, a sluice rail for good fruit 36, and a second slide rail 37. The screening disc 31 is fixedly mounted on the end face of the connecting plate 13 by a bracket, and the screening disc 31 has a conical structure with a larger diameter at the top and a smaller diameter at the bottom (i.e., the longitudinal section of the screening disc is an inverted isosceles trapezoid). A coaxial rotating shaft is set in the middle of the screening disc 31, and the first baffle 32 is set on the outer wall of the rotating shaft. The first baffle 32 is rotated within the screening disc 31 by the rotating shaft (the rotating shaft can be driven to rotate by a motor set at the bottom of the screening disc 31, thereby causing the first baffle 32 to rotate around the central axis of the rotating shaft). The side of the screening disc 31 near the good fruit collection trough 21 (i.e., Figure 3 The right side shown has a discharge port and a first chute 33 is provided at the discharge port. The first chute 33 is for the flow from the screening disc 31 to the good fruit collection tank 21 (i.e., from the screening disc 31 to the good fruit collection tank 21). Figure 3 The inclined track structure (shown from left to right) is as follows: the end of the first slide 33 away from the screening disk 31 is connected to the good fruit slide 36, and the side of the first slide 33 away from the screening disk 31 is provided with the bad fruit slide 35 (see reference). Figure 1 , Figure 3As shown), at the connection point (inner wall of the corner) between the good fruit slide 36 and the bad fruit slide 35, a second baffle 34 is rotatably installed (the second baffle 34 can be driven to rotate by a motor installed on the outer wall of the slide). A discharge hole 130 is opened at the end of the bad fruit slide 35 away from the first slide 33, and the discharge hole 130 corresponds to the bad fruit collection trough 22. The end of the good fruit slide 36 away from the first slide 33 is connected to the second slide 37 installed in the good fruit collection trough 21 (in conjunction with...). Figure 1 and Figure 3 (As shown).

[0069] The harvesting robotic arm 40 adopts a multi-axis robotic arm commonly used in this field, typically with 3 to 6 axes. See [link / reference]. Figure 1 As shown, a six-axis robotic arm is used in this embodiment.

[0070] An end effector 50 is provided at the end of the harvesting robotic arm 40 away from the base support mechanism 10. The end effector includes a fixed housing 51, a drive motor 52, a first gear 53, a second gear 54, a first rack 55, a first protective ring 56, a second rack 57, a pressing plate 58, and a second protective ring 59. The fixed housing 51 is located at the end of the harvesting robotic arm 40 away from the base support mechanism 10, and the drive motor 52 is located at the bottom of the inner cavity of the fixed housing 51 (see...). Figure 4 As shown), a gear shaft 520 is coaxially mounted on the output shaft of the drive motor 52, and the end of the gear shaft 520 away from the drive motor 52 is rotatably connected to the top surface of the inner cavity of the fixed housing 51; the outer wall of the gear shaft 520 is sequentially (fixedly) sleeved with the first gear 53 and the second gear 54 from bottom to top, and the inner wall of one side of the fixed housing 51 is slidably provided with the first rack 55 corresponding to the first gear 53 (see...). Figure 5 As shown, in this embodiment, a first rack 55 is provided on the upper side of the fixed housing 51, and a first protective ring 56 is provided at the end of the first rack 55 away from the harvesting robotic arm 40 via a sliding support 562 (the end of the sliding support 562 away from the first rack 55, i.e., Figure 4 The lower end shown is slidably connected to the inner wall of the fixed housing 51, thereby achieving stable sliding of the first protective ring 56. The end of the first protective ring 56 away from the first rack 55 passes through the corresponding side wall of the fixed housing 51 and is slidably connected. A separation block 561 is provided on the top surface of the end of the first protective ring 56 away from the fixed housing 51 (i.e., the end located on the outside of the fixed housing 51), and a separation groove with a built-in separation blade is opened on the side of the separation block 561 near the fixed housing 51 (e.g., ...). Figure 4 As shown); the inner wall of the other side of the fixed housing 51 (i.e., the inner wall opposite to the first rack 55, as shown). Figure 5 The fixed housing 51 (lower side wall shown) and the second rack 57 is slidably arranged corresponding to the second gear 54 (see...) Figure 5As shown, the length of the second rack 57 is greater than the length of the first rack 55. The end of the second rack 57 near the first protective ring 56 penetrates the corresponding side wall of the fixed housing 51 and is provided with a pressing plate 58. A top block 581 is provided on the side of the pressing plate 58 near the separating block 561, corresponding to the separating groove. To ensure smooth cutting by the end effector 50, a limiting rod 582 is provided on the side of the pressing plate 58 away from the top block 581 and located on the side of the first rack 55 (e.g., ...). Figure 5 As shown, the limiting rod 582 passes through the corresponding side wall of the fixed housing 51, and a limiting block is provided on the side of the limiting rod 582 away from the extrusion plate 58; a second protective ring 59 is fixedly installed on the outer wall of the fixed housing 51 and located below the first protective ring 56, and the second protective ring 59 is arranged parallel to the first protective plate 56. Both the first protective ring 56 and the second protective ring 59 include an inner ring and an outer ring. The inner ring is made of plastic, and the outer ring is made of biopolymer material (both plastic and biopolymer materials can be common materials in this field).

[0071] The storage mechanism is located on the end face of the base support mechanism 10 (specifically, the connecting plate 13) and is connected to the end actuator 50 and the grading and screening mechanism respectively; for example Figure 1 As shown, the collection mechanism includes a connecting hose 61, a buffer mechanism, a transition collection box 62, and a guide chute 63. The transition collection box 62 is fixedly mounted on the end face of the base support mechanism 10 (i.e., the connecting plate 13) by a bracket and is located on one side of the screening disc 31. The distance between the bottom surface of the transition collection box 62 and the end face of the base support mechanism 10 is greater than the distance between the top surface of the screening disc 31 and the end face of the base support mechanism 10 (i.e., the height of the transition collection box 62 is greater than the height of the screening disc 31). Figure 1 As shown, one end of the connecting hose 61 is located on the bottom surface of the second protective ring 59 and communicates with the inner ring of the second protective ring 59, and the other end is communicated with one side of the transition collection box 62. A buffer mechanism is provided at the connection between the connecting hose 61 and the transition collection box 62 (for example, buffering by evenly distributed rubber protrusions or other similar methods; this application does not make specific limitations, and those skilled in the art can set it according to the actual situation). A guide groove 63 is provided on the side of the transition collection box 62 near the screening disc 31. The guide groove 63 is an inclined groove that slopes from the transition collection box 62 to the screening disc 31. The bottom surface of the inner cavity of the transition collection box 62 is set as an inclined structure that slopes from the transition collection box 62 to the screening disc 31 (the inclination of the inclined structure is consistent with the inclination of the guide groove 63), and a sponge buffer layer is evenly laid on the inner wall of the transition collection box 62 (to avoid damage to the skin of the slipping fruit).

[0072] The visual recognition system includes one positioning binocular camera 71 and two recognition binocular cameras 72. The positioning binocular camera 71 is mounted on the end effector 50 (specifically, on the top surface of the fixed housing 51). The recognition binocular cameras 72 correspond to a grading and screening mechanism mounted on the end face of the base support mechanism 10 (specifically, the grading disc 31 corresponding to the recognition binocular cameras 72 is mounted on its upper and lower sides, and the grading disc 31 is made of transparent material, such as...). Figure 1 (As shown).

[0073] Example 2:

[0074] As a preferred embodiment of the present invention, a harvesting method using a plum harvesting and grading robot as described in Example 1 includes:

[0075] Step S1, Identification and Localization: The binocular camera 71 scans horizontally at a distance from the plum fruit to obtain an image of the plum fruit in the distance. The image is then segmented and target is identified. Specifically:

[0076] Step S11: First, perform histogram calculation on the input image to obtain the frequency distribution of each pixel value in the image; then sample the target image and divide its histogram interval into M sub-intervals, setting the width of each sub-interval to U; then, map the image grayscale values ​​to each sub-interval and calculate the pixel frequency in each sub-interval.

[0077] Step S12: First, calculate the gray level distribution of all pixel values ​​in each sub-interval to obtain the gray level mean in each sub-interval, and calculate the intra-class variance of each sub-interval to represent the degree of dispersion of pixel values ​​in that interval; then, iterate through all possible thresholds, calculate the intra-class variance corresponding to each threshold, and select the threshold that maximizes the intra-class variance as the optimal threshold.

[0078] Step S13: Divide the image into two regions, foreground and background, according to the optimal threshold: the region greater than the optimal threshold is the foreground and the region less than the optimal threshold is the background, thereby separating the target image from the background.

[0079] Step S14: Optimize the segmented image using grayscale morphology (grayscale morphology can use conventional techniques in this field, and is not specifically limited in this embodiment) to achieve image segmentation and recognition, and to divide and locate the sub-regions of the segmented image.

[0080] Step S2, Selection of New Plum Fruit Clusters: A target tracking algorithm is used to track the target new plum fruits and identify the number of mature new plum fruits. Specifically:

[0081] Step S21: First, object detection is applied to detect candidate targets from video frames, and the bounding box and confidence score of each selected target are obtained; then the detected candidate targets are grouped according to their temporal and spatial correlation, and the degree of overlap between two candidate targets is measured.

[0082] Step S22: Match the detection result of the current frame with the trajectory of the previous frame, and use the Hungarian algorithm to obtain the matching method with the minimum cost (the conventional Hungarian algorithm in this field can be used). The cost includes calculations based on factors such as distance, speed, and confidence, and the initial matching is completed. For candidate targets that do not exist in the previous trajectory matching, perform a second matching with the nearest trajectory.

[0083] Step S23: Based on the matching results, generate new trajectories. Each generated trajectory includes a series of bounding boxes and corresponding confidence scores. Then, use a Kalman filter to predict the trajectory position of the next frame and correct the predicted trajectory based on the new detection results.

[0084] Step S24: Preset a confidence score threshold range. Compare the confidence score in the trajectory with the confidence score threshold range. In this embodiment, the confidence score threshold range is [0,1]. The lower limit of the confidence score threshold range indicates high maturity, and the upper limit of the confidence score threshold range indicates low maturity (i.e., the closer to 0, the higher the maturity, and the closer to 1, the lower the maturity), thus distinguishing the maturity of the fruit. Then, the trajectory is filtered according to the trajectory length and confidence score to remove unstable and unreasonable trajectories, thereby obtaining the number of fruits.

[0085] When the number of mature new plum fruits in a cluster exceeds 60% of the total number of fruits in that cluster, the cluster is designated as the target new plum fruit cluster.

[0086] Step S3, Picking point location: Select the lowest point of the target new plum fruit cluster as the picking point reference position, and move it up and down 3-5cm along the lowest point of the picking point reference position in the longitudinal direction of the fruit to obtain the picking point.

[0087] The method for obtaining the location of the lowest point new plum fruit is as follows: First, images from different perspectives are acquired through multi-view image acquisition; then, feature points are extracted from two or more images to find feature point pairs corresponding to the same target point (i.e., the lowest point new plum fruit); subsequently, for each matching feature point pair, a three-dimensional point cloud is calculated using triangulation to obtain the spatial coordinates of the target point. The triangulation depth... d for:

[0088]

[0089] In the formula: fIndicates the camera's focal length. b Indicates the distance between images from different viewpoints. This indicates the angle between images viewed from different perspectives.

[0090] Step S4, Fruit Harvesting: Based on the harvesting point (i.e., the sub-area corresponding to the harvesting point), the harvesting robotic arm 40 is gradually moved (using the positioning binocular camera 71) to further identify and locate the new plum fruits in close-up view, specifically:

[0091] First, the SVM algorithm is used to obtain an optimal classification hyperplane. A model is built, and a linear equation for the hyperplane is defined to divide the sample data into two classes.

[0092]

[0093] In the formula: w Represents the weight vector; X Denotes a hyperplane, with the direction perpendicular to the hyperplane. X These are the sample data points; b This represents the bias term, which determines the distance between the hyperplane and the origin;

[0094] Then, the Lagrange multiplier method is used to solve the problem and obtain the weight vector of the hyperplane linear equation. w With bias term b ;

[0095] Then, the model is optimized by selecting any two variables through iteration.

[0096] Finally, the data is classified using weight vectors and bias terms. The distance from each data point to the hyperplane is calculated, and the category is determined by whether the value is positive or negative. The classification linear equation is as follows:

[0097]

[0098] In the formula: sign() The category of the function range prediction label is -1 or 1;

[0099] By using decision functions to determine the category of pixels in an image, image segmentation and target recognition and localization can be completed.

[0100] When the end effector 50 reaches the picking point, the relative distance between the picking point and the upper edge of the new plum fruit (i.e., the apex of the new plum fruit) is obtained, and then the distance is increased by 3-5 cm in the longitudinal direction based on this distance to obtain the end point of the end effector 50; wherein, the method for obtaining the upper edge of the new plum fruit (i.e., the apex of the new plum fruit) is as follows:

[0101] First, obtain the two-dimensional coordinates of the corresponding feature points in the image. u,vThe feature point is then calculated by taking its two-dimensional coordinates and its depth information D; and then, its three-dimensional coordinates are obtained.

[0102]

[0103] In the formula: fx , fy Indicates the camera's focal length. cx , cy This indicates the coordinates of the camera's principal point.

[0104] The end effector 50 moves from bottom to top according to the picking point and the end point to achieve the envelopment and shearing separation of the new plum fruit, thus completing the picking. Specifically, the picking robotic arm 40 controls the end effector 50 to move from bottom to top, so that the target fruit to be picked passes through the first protective ring 56 and the second protective ring 59 sequentially from top to bottom until the end effector 50 reaches the end point; then, the drive motor 52 rotates, which drives the first gear 53 and the second gear 54 to rotate simultaneously through the gear shaft 520, and the first rack 55 drives the first protective ring 56 to move towards the end point. Figure 5 Sliding to the left in the indicated direction, the second rack 56 drives the extrusion plate 58 to move as shown. Figure 5 The fruit slides to the right in the direction shown, thereby pressing against the fruit stalk. The fruit stalk is cut off by the cross-squeezing action of the separating blade and the top block 581. The cut fruit falls into the screening disc 31 through the connecting hose 61, the transition collection box 62, and the guide chute 63.

[0105] Step S5, Fruit sorting and collection: After harvesting, the fruits fall into the grading and sorting mechanism through the collection mechanism. After sorting by the grading and sorting mechanism, good fruits and bad fruits are separated and collected in the good fruit collection trough 21 and bad fruit collection trough 22 respectively.

[0106] The grading and screening agency uses the following method to screen new plum fruits:

[0107] Step S51, Data Collection: Collect a dataset containing images of various new plum fruits and their corresponding labels; the dataset covers different task labels, including fruit size, presence or absence of cracks, presence or absence of damage, presence or absence of fruit stems, etc.

[0108] Step S52: Set the fruit size as a regression problem, labeled as the size of the new plum fruit: calculate the minimum bounding rectangle of the fruit image, obtain the maximum longitudinal and transverse diameters of the fruit, compare with the average data (the average data is obtained through a large amount of experimental data), and determine whether the fruit size meets the standard; set the presence or absence of cracks as a binary classification problem, labeled as cracked 1 and no crack 0; set the presence or absence of damage as a binary classification problem, labeled as damaged 1 and no damage 0; set the presence or absence of fruit stem as a binary classification problem, labeled as with fruit stem 1 and without fruit stem 0.

[0109] Step S53, Model Design: The ResNet neural network model is used as the screening model. The first four ResNet Blocks of the ResNet neural network model are selected as shared convolutional layers to extract features from the input image. After the shared convolutional layers, multiple task-specific branches are set up. Each branch processes a specific task. Each task-specific branch consists of several convolutional layers and fully connected layers, which are optimized for different tasks.

[0110] Step S54: Train the screening model using a dataset containing multi-task labels, providing corresponding labels for each task during the training process.

[0111] Step S55: After training is complete, use the validation set to evaluate the performance of the screening model and complete the training.

[0112] The binocular camera 72 identifies the new plums in the screening disc 31, determining their size, presence of cracks, damage, and presence of stems, etc. (During this process, the first baffle 32 rotates continuously, firstly to turn the new plums over and prevent the four corners from being detected, and secondly to move the new plums to the discharge port; the first baffle 32 is made of transparent material). If the plums are qualified, the second baffle 34 rotates, closing the interface between the bad fruit slide 35 and the first slide 33. The plums then pass through the first slide 33, the good fruit slide 36, and the second slide 37 into the good fruit collection trough 21. If the plums are unqualified, the second baffle 34 rotates, closing the interface between the good fruit slide 36 and the first slide 33. The plums then pass through the first slide 33, the bad fruit slide 36, and the discharge hole 130 into the bad fruit collection trough 22, thus achieving the grading and screening of the new plums.

Claims

1. A method for harvesting plums using a plum-harvesting and grading robot, characterized in that: The harvesting and grading robot includes a base support mechanism, a collection box, a grading and sorting mechanism, a harvesting robotic arm, an end effector, a storage mechanism, and a visual recognition system. The collection box is located at the front end of the base support mechanism, including a good fruit collection slot and a bad fruit collection slot; the grading and sorting mechanism and the picking robotic arm are respectively located on the end face of the base support mechanism, and the end effector is located at the end of the picking robotic arm away from the base support mechanism; the storage mechanism is located on the end face of the base support mechanism and is connected to the end effector and the grading and sorting mechanism respectively; the visual recognition system includes one positioning binocular camera and two recognition binocular cameras. The positioning binocular camera is located on the end effector, and the recognition binocular camera is located on the end face of the base support mechanism corresponding to the grading and sorting mechanism; the end effector includes a fixed box, a drive motor, a first gear, a second gear, a first rack, a first protective ring, a second rack, a pressing plate, and a second protective ring. The fixed box is located at the end of the picking robotic arm away from the base support mechanism, and the bottom of the inner cavity of the fixed box is provided with A drive motor is installed, and a gear shaft is coaxially mounted on the output shaft of the drive motor. The end of the gear shaft away from the drive motor is rotatably connected to the top surface of the inner cavity of the fixed box. A first gear and a second gear are sequentially sleeved on the outer wall of the gear shaft from bottom to top. A first rack is slidably mounted on the inner wall of one side of the fixed box corresponding to the first gear. A first protective ring is mounted on the end of the first rack away from the harvesting robot arm via a sliding support. The end of the first protective ring away from the first rack passes through the corresponding side wall of the fixed box and is slidably connected. A separation block is mounted on the top surface of the end of the first protective ring away from the fixed box, and a separation groove with a built-in separation blade is opened on the side of the separation block near the fixed box. A second rack is slidably mounted on the inner wall of the other side of the fixed box corresponding to the second gear. The end of the second rack near the first protective ring passes through the corresponding side wall of the fixed box and is mounted on a pressing plate. A top block is mounted on the side of the pressing plate near the separation block corresponding to the separation groove. A second protective ring is fixedly installed on the outer wall of the fixed box and located below the first protective ring. The second protective ring is arranged parallel to the first protective plate. Specific harvesting methods include: Step S1, Identification and Positioning: The binocular camera is positioned at a distance from the plum fruit in a horizontal orientation to scan and obtain an image of the plum fruit in the distance. The image is then segmented and the target is identified. Step S2, Selection of new plum fruit clusters: The target new plum fruit is tracked using a target tracking algorithm to identify the number of mature new plum fruit; when the number of mature new plum fruit in the same cluster exceeds 60% of the total number of fruit in the cluster, the cluster is marked as the target new plum fruit cluster; Step S3, Picking point location: Select the lowest point of the target new plum fruit cluster as the picking point reference position, and move it up and down 3-5cm along the lowest point of the picking point reference position in the longitudinal direction of the fruit to obtain the picking point. Step S4, Fruit Harvesting: The harvesting robotic arm is moved step by step according to the harvesting point to further identify and locate the new plums in close-up view; when the end effector reaches the harvesting point, the relative distance between the harvesting point and the upper edge of the new plum is obtained, and then the distance is increased by 3-5 cm in the longitudinal direction to obtain the end point of the end effector; the end effector moves from bottom to top according to the harvesting point and the end point to realize the envelopment and shearing separation of the new plums, thus completing the harvesting; Step S5, Fruit sorting and collection: After harvesting, the fruits fall into the grading and sorting mechanism through the collection mechanism. After sorting by the grading and sorting mechanism, good fruits and bad fruits are separated and collected in the good fruit collection trough and bad fruit collection trough respectively.

2. The harvesting method of a plum picking and grading robot according to claim 1, characterized in that: The base support mechanism includes a chassis, support studs, a connecting plate, track wheels, and tracks. Multiple evenly distributed vertical support studs are provided on the end face of the chassis, and the end of the support studs away from the chassis is connected to the same connecting plate. Two track wheels are provided on each side of the chassis, and the two track wheels on the same side are connected by tracks. A bad fruit collection trough is provided between the chassis and the connecting plate, and a good fruit collection trough is provided at the front end of the chassis, and the good fruit collection trough and the bad fruit collection trough are fixedly connected.

3. The harvesting method of a plum picking and grading robot according to claim 1 or 2, characterized in that: The grading and screening mechanism includes a screening disc, a first baffle, a first slide rail, a second baffle, a slide rail for damaged fruit, a slide rail for good fruit, and a second slide rail. The screening disc is fixedly mounted on the end face of the connecting plate and has a conical structure with a larger diameter at the top and a smaller diameter at the bottom. A coaxial rotating shaft is provided in the middle of the screening disc, and a first baffle is provided on the outer wall of the rotating shaft. The first baffle rotates within the screening disc via the rotating shaft. A discharge port is opened on the side of the screening disc near the good fruit collection trough, and a first baffle is provided at the discharge port. The slide is a sloping track structure that extends from the screening disc to the good fruit collection trough. The end of the first slide away from the screening disc is connected to the good fruit slide, and a bad fruit slide is set on the side of the end of the first slide away from the screening disc. A second baffle is rotatably set at the connection between the good fruit slide and the bad fruit slide. A discharge hole is opened on the connecting plate at the end of the bad fruit slide away from the first slide, and the discharge hole corresponds to the bad fruit collection trough. The end of the good fruit slide away from the first slide is connected to the second slide set in the good fruit collection trough.

4. The harvesting method of a plum picking and grading robot according to claim 3, characterized in that: The collection mechanism includes a connecting hose, a buffer mechanism, a transition collection box, and a guide chute. The transition collection box is fixedly installed on the end face of the base support mechanism and located on one side of the screening disc. The distance between the bottom surface of the transition collection box and the end face of the base support mechanism is greater than the distance between the top surface of the screening disc and the end face of the base support mechanism. One end of the connecting hose is located on the bottom surface of the second protective ring and communicates with the inner ring of the second protective ring, and the other end communicates with one side of the transition collection box. A buffer mechanism is provided at the connection between the connecting hose and the transition collection box. A guide chute is provided on the side of the transition collection box near the screening disc. The guide chute is an inclined groove that slopes from the transition collection box towards the screening disc.

5. The harvesting method of a plum picking and grading robot according to claim 4, characterized in that: Step S1 specifically involves: Step S11: First, perform histogram calculation on the input image to obtain the frequency distribution of each pixel value in the image; then sample the target image and divide its histogram interval into M sub-intervals, setting the width of each sub-interval to U; then, map the image grayscale values ​​to each sub-interval and calculate the pixel frequency in each sub-interval. Step S12: First, calculate the gray level distribution of all pixel values ​​in each sub-interval, obtain the gray level mean in each sub-interval, and calculate the intra-class variance of each sub-interval to represent the degree of dispersion of pixel values ​​in that interval. Then, iterate through all possible thresholds, calculate the intra-class variance for each threshold, and select the threshold that maximizes the intra-class variance as the optimal threshold. Step S13: Divide the image into two regions, foreground and background, according to the optimal threshold: the region greater than the optimal threshold is the foreground and the region less than the optimal threshold is the background, thereby separating the target image from the background. Step S14: Optimize the segmented image using grayscale morphology to achieve image segmentation and recognition, and divide and locate the sub-regions of the segmented image.

6. The harvesting method of a plum picking and grading robot according to claim 5, characterized in that: In step S2, the target tracking algorithm is used to track the target new plum fruits and identify the number of mature new plum fruits. Specifically, this involves: Step S21: First, object detection is applied to detect candidate targets from video frames, and the bounding box and confidence score of each selected target are obtained; then the detected candidate targets are grouped according to their temporal and spatial correlation, and the degree of overlap between two candidate targets is measured. Step S22: Match the detection result of the current frame with the trajectory of the previous frame, and use the Hungarian algorithm to obtain the matching method with the minimum cost. The cost includes calculations based on distance, speed, and confidence factors to complete the initial matching; for candidate targets that do not exist in the previous trajectory matching, perform a second matching with the nearest trajectory. Step S23: Based on the matching results, generate new trajectories. Each generated trajectory includes a series of bounding boxes and corresponding confidence scores. Then, use a Kalman filter to predict the trajectory position of the next frame and correct the predicted trajectory based on the new detection results. Step S24: Preset a confidence score threshold range. Compare the confidence score in the trajectory with the confidence score threshold range. The lower limit of the confidence score threshold range indicates high maturity, and the upper limit of the confidence score threshold range indicates low maturity, thus distinguishing the maturity of the fruit. Then, filter the trajectories according to the trajectory length and confidence score, and remove unstable and unreasonable trajectories to obtain the number of fruits.

7. The harvesting method of a plum picking and grading robot according to claim 6, characterized in that: The method for obtaining the position of the lowest point of the new plum fruit in step S3 is as follows: First, images from different perspectives are acquired through multi-view image acquisition. Then, feature points are extracted from two or more images to find feature point pairs corresponding to the same target point. Next, for each matching feature point pair, triangulation is used to calculate a 3D point cloud, thereby obtaining the spatial coordinates of the target point. The triangulation depth is used in this process. d for: In the formula: f Indicates the camera's focal length. b Indicates the distance between images from different viewpoints. This indicates the angle between images viewed from different perspectives.

8. The harvesting method of a plum picking and grading robot according to claim 7, characterized in that: The method for obtaining the upper edge of the new plum fruit in step S4 is as follows: First, obtain the two-dimensional coordinates of the corresponding feature points in the image. u,v And the depth information D of the two-dimensional coordinates; Then, the three-dimensional coordinates of the feature point are calculated: In the formula: fx , fy Indicates the camera's focal length. cx , cy This indicates the coordinates of the camera's principal point.

Images