A deep learning-based shearing and clamping integrated fruit picking device and application thereof

Abstract

The application discloses a kind of based on deep learning's clamping and shearing integrated cluster fruit harvesting device and its application, the clamping and shearing integrated cluster fruit harvesting device includes end effector, mechanical arm, collection frame, industrial computer, visual identification module, support platform;The end effector, including cutting mechanism and clamping mechanism;End effector is located at the end of mechanical arm, and mechanical arm and industrial computer are installed on support platform, and collection frame is located in front of support platform.The application uses deep learning network in the identification of cluster fruit, in addition, different from prior art is that clamping and cutting are synchronous and the technical scheme of double-motor-driven clamping and cutting, the application can make clamping and cutting action asynchronous by single motor, simplify the structure of end effector, reduce the weight of end effector, reduce the load requirement to mechanical arm, and expand the selection range of working mechanical arm.

Description

Technical Field

[0001] This invention belongs to the field of agricultural machinery, specifically relating to a deep learning-based integrated clamping and shearing fruit harvesting device and its application. Background Technology

[0002] Currently, fruit harvesting machinery is mainly divided into single-fruit harvesting machinery and bunch-fruit harvesting machinery. The former has a simple structure, but can only harvest one fruit per action, resulting in low harvesting efficiency. The latter can harvest more efficiently, but the clamping and cutting of the fruit stem often requires two or more drive devices, leading to problems such as complex structure, large weight, and difficult maintenance. Patent 202211147213.8 discloses a fruit harvesting robot equipped with an end effector. The mechanical structure includes a vehicle body, a harvesting robotic arm, and an end effector. The end effector uses a pneumatic robotic gripper to complete the harvesting by cutting and clamping the fruit stem. However, this end effector is connected to an external air pump and uses pneumatic drive to squeeze and cut the fruit stem. To ensure the success rate of stem cutting, a high-pressure air pump and drive device are required, resulting in a complex structure and large size. The clamped fruit stem is prone to slippage, directly affecting the cutting efficiency. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a deep learning-based integrated clamping and shearing fruit harvesting device and its application. The device of this invention has a simple structure and only requires a single motor to drive the clamping and shearing actions. It has stable fruit stem clamping, strong shearing ability, and high shearing success rate.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A deep learning-based integrated clamping and shearing fruit harvesting device includes an end effector 1, a robotic arm 2, a collection frame 3, an industrial control computer 4, a vision recognition module, and a support platform; the end effector is located at the end of the robotic arm, the robotic arm and the industrial control computer are mounted on the support platform, and the collection frame is located in front of the support platform.

[0006] The end effector 1 includes a cutting mechanism and a clamping mechanism.

[0007] The cutting mechanism includes a cutting blade 9, a crescent-shaped toothed blade 15, a base plate 10, a blade reinforcing plate 7, and connecting rods 8. Two lower connecting rods 8 are connected to the lower part of the cutting blade 9. The other end of one lower connecting rod is connected to the base plate 10, and the other end of the other lower connecting rod is connected to the motor drive shaft 11. A connecting rod is connected to the upper part of the cutting blade 9, and the other end of the upper connecting rod is connected to the blade reinforcing plate 7. The lower end of the blade reinforcing plate 7 is connected to the base plate 10. The crescent-shaped toothed blade 15 is directly connected to the base plate.

[0008] The working principle of the cutting mechanism of this invention is as follows: the cutting blade 9, the base plate 10, and the two lower connecting rods form a parallelogram double crank mechanism. The upper connecting rod of the cutting blade 9 acts as a parallel rod, avoiding the phenomenon of uncertain motion of the driven member when the four rods of the parallelogram mechanism are in the same straight line position. When the stepper motor 13 drives the lower connecting rod 8 through the motor drive shaft 11, the cutting blade 9 will move parallel to the trajectory of a circle with the other end of the upper connecting rod as the center and the length of the upper connecting rod as the radius. When the motor is powered on, the movement speed of the cutting blade 9 can be divided into a speed component pointing towards the axis of the fruit-bearing branch and a speed component in the same direction as the blade, thereby generating tensile and shearing forces on the branch. However, most existing end effectors only generate shearing or tensile forces for cutting. The former has high operating efficiency but low cutting capacity and high power requirements for the blade and drive components, while the latter has excellent cutting capacity but low operating efficiency and also high power requirements for the drive components. The cutting mechanism of this invention combines the working methods of both, achieving the advantages of both with lower hardware requirements.

[0009] The clamping mechanism includes a base plate 10, a cam 12, a stepper motor 13, a cam pressure plate 19, a spring 17, a clamping pad connecting rod 18, a clamping pad 16, a motor drive shaft 11, and a motor fixing plate 20. The stepper motor 13 is mounted on the motor fixing plate 20, which is fixed to the base plate 10. The cam 12 is connected to the motor drive shaft 11. The cam pressure plate 19 is connected to the two clamping pads 16 respectively through two clamping pad connecting rods 18. A spring 17 is sleeved on the clamping pad connecting rod 18.

[0010] The clamping pad 16 is made of soft silicone material. The clamping surface is designed with biomimetic patterns of tree frog feet and is covered with prisms arranged in a hexagonal shape. The top of the prisms is designed with a semi-circular groove.

[0011] The working principle of the clamping mechanism of this invention is as follows: When the stepper motor 13 is working, it drives the cam 12 to rotate through the motor transmission shaft 11, thereby driving the cam pressure plate 19. The motion law of the cam pressure plate 19 is determined by the contour shape of the cam 12. When the contour of the cam 12 with changing radius vector contacts the cam pressure plate 19, the cam pressure plate 19 will reciprocate; while when the arc segment contour with the rotation center of the cam 12 as the center contacts the cam pressure plate 19, the cam pressure plate 19 will remain stationary. Therefore, as the cam 12 rotates continuously, the cam pressure plate 19 can obtain intermittent, predictable movement, thereby enabling the cutting mechanism and the clamping mechanism to be driven simultaneously by the same stepper motor, realizing the asynchronous movement of the cutting blade 9 and the clamping pad 16. This ensures that the fruit stems are fixed by the clamping pad 16 before cutting begins, increasing the stability and success rate of cutting. The cutting blade 9 then cuts the fruit stems. After the cutting blade 9 completes its cutting stroke, the clamping pad 16 remains in a clamping state. When the robotic arm moves to the designated position, the stepper motor 13 is controlled to release the clamping pad 16.

[0012] The visual recognition module includes a binocular camera 6; the binocular camera 6 is fixed on a connecting plate 14, which is connected to the base plate 10. The binocular camera 6 includes two camera lenses, a camera photosensitive element, and a camera processor; the camera photosensitive element receives external light signals through the camera lenses, converts the light signals into electrical signals, and transmits them to the camera processor; the camera processor processes the electrical signals into image information and transmits the image information to an industrial control computer. The industrial control computer uses an improved YOLOv5 algorithm to identify the clustered fruits and the fruit stems that need to be cut and clamped in the image information.

[0013] A deep learning-based method for harvesting bunched fruits using a clamping and shearing mechanism employs the aforementioned device and includes the following steps:

[0014] (1) Acquire the image of the current field of view: Use a binocular camera to capture images of the left and right lenses at the current position;

[0015] (2) Fruit cluster recognition: Based on the left view in the captured image, the improved YOLOv5 deep learning network trained in the industrial control computer is used to identify all the fruit clusters in the left view, and each fruit cluster is assigned a serial number from left to right in the image. Then the length, width and center point of the minimum bounding rectangle of each fruit cluster are calculated.

[0016] (3) Obtain the three-dimensional coordinates of the string of fruits: Use Zhang Zhengyou's binocular calibration method to perform binocular calibration on the binocular camera and obtain binocular calibration parameters such as eigenvalue matrix, intrinsic parameter matrix, rotation matrix, and distortion coefficient; after identifying the string of fruits, use the calibrated intrinsic parameter matrix, rotation matrix, and distortion coefficient to perform epipolar calibration on the left and right images of the camera to obtain calibrated left and right calibrated views, and then use the SGBM algorithm to perform stereo matching on the left and right calibrated views to obtain disparity maps; multiply the disparity map pixels corresponding to the minimum bounding rectangle of the string of fruits with the eigenvalue matrix obtained by binocular calibration to obtain the real-world three-dimensional coordinates of each pixel of the string of fruits, thereby obtaining the three-dimensional coordinates of each string of fruits;

[0017] (4) Move the end effector to the pre-picking position: After obtaining the three-dimensional coordinates of each bunch of fruit, the industrial control computer performs forward and inverse kinematics on the posture of the robotic arm, calculates the trajectory planning of the robotic arm and the walking chassis, moves the end effector to the same horizontal line as the bunch of fruit, and then performs step (2) to obtain the image of the bunch of fruit at close range.

[0018] (5) Cutting point confirmation: The fruit target recognition is performed on the fruit string image obtained at close range. An equilateral triangle region with a vertex angle of 150 degrees and a height equal to the width of the fruit string bounding box is constructed as the region of interest, with the center of the target fruit string bounding box as the vertex. The image region containing the fruit stem target is extracted and the fruit stem is detected. The center of the fruit stem detection box is used as the cutting point.

[0019] (6) Reaching the cutting point: The industrial control computer converts the real-world three-dimensional coordinates of the calculated cutting point into three-dimensional coordinates under the coordinates of the robotic arm base, and controls the robotic arm to drive the end effector to reach the cutting point.

[0020] (7) Clamping the fruit stem: The industrial control computer controls the stepper motor 13 to rotate the cam 12 by 90 degrees. Driven by the cam 12, the cam pressure plate 19 clamps the fruit stem with the clamping pad 16, and at the same time the cutting blade 9 moves to the initial cutting position.

[0021] (8) Cutting the fruit stem: The industrial control computer controls the output shaft of the stepper motor 13 to rotate 180 degrees, so that the cutting blade 9 performs the cutting action. The cutting blade 9 and the toothed blade on the crescent toothed blade 15 will cut the fruit stem. After the cutting is completed, the cutting blade 9 moves to the cutting end position, and the clamping pad 16 keeps clamping the fruit stem under the action of the cam 12.

[0022] (9) Place the fruit bunches in the designated position: After the industrial control computer controls the robotic arm to move the end effector to the preset position, it controls the stepper motor 13 to rotate the cam 12 by 90 degrees. At this time, the spring 17 on the clamping pad connecting rod 18 causes the clamping pad 16 to release the fruit stem, and the fruit bunches fall into the designated collection box.

[0023] In step (2), the improved YOLOv5 deep learning network is an improvement on the YOLOv5 fruit cluster recognition network. There are four structural blocks in the YOLOv5 backbone network that mainly learn the residual features in the vectors transmitted by the network. The CA attention module with an attention mechanism is added to the fourth structural block. The CA attention module can not only extract the information of vectors between different channels in the vectors operated in the network, but also extract position information in long-distance vector operations. Through training, the network can become more sensitive to the information of fruit clusters and fruit branches in complex environments, thereby obtaining a higher accuracy recognition performance.

[0024] In step (5), the principle of fruit stalk detection is as follows: Since complex environments may contain pseudo-fruit stalks that are highly similar to fruit stalks, such as dead branches, in order to improve the accuracy of fruit stalk detection, this invention first uses the YOLOv5 network to identify the target fruit cluster with obvious features and obtain the position of the target fruit cluster in the image. At the same time, taking the center of the predicted box of the fruit cluster as the starting point, an equilateral triangle with a height of twice the width of the predicted box and a vertex angle of 150 degrees is constructed upward as the region of interest where the target fruit stalk exists. Because the larger the region of interest, the more likely it is to have pseudo-fruit stalks, this paper chooses to use the shape of a triangle to construct the region of interest of the fruit stalk in order to avoid the situation where pseudo-fruit stalks appear in the region of interest. Then, the image of the region of interest is extracted, and the image is filled with black pixels to form a rectangular image for YOLOv5 to detect the target fruit stalk, and the predicted box of the fruit stalk is obtained. The center of the predicted box of the fruit stalk is taken as the position of the fruit stalk.

[0025] Compared with the prior art, the present invention has the following advantages and effects:

[0026] (1) This invention uses a deep learning network for the identification of clustered fruits. Compared with other schemes that use traditional algorithms, the deep learning network identification scheme has the advantages of fast identification speed, adaptability to different lighting environments, and low probability of misidentification, thereby improving the efficiency of harvesting work.

[0027] (2) The present invention uses a single stepper motor to achieve asynchronous clamping and cutting; unlike the existing technology which uses synchronous clamping and cutting and dual-motor driven clamping and cutting, the present invention can perform clamping and cutting actions asynchronously with only a single motor, which simplifies the structure of the end effector, reduces the weight of the end effector, reduces the load requirements on the robotic arm, and expands the selection range of working robotic arms.

[0028] (3) When the cutting blade of the present invention is working, it moves parallel to a circle with the other end of the upper connecting rod as the center and the length of the upper connecting rod as the radius. The speed of the movement during operation can be divided into a speed component perpendicular to the axis of the fruit-bearing branch and a speed component in the same direction as the blade, so that the cutting blade generates tensile force when slicing the branch and shear force when squeezing the branch. Compared with the existing end effectors that only generate shear force or tensile force for cutting, the present invention has both cutting performance and cutting efficiency.

[0029] (4) The clamping mechanism of the present invention adopts a cam mechanism. During the clamping stage of the end effector, since the cam mechanism has a self-locking capability, it is not necessary to keep the motor powered on in order to maintain the clamping action, which reduces the power supply requirements while maintaining the clamping performance.

[0030] (5) The clamping mechanism of the present invention is made of soft silicone material. The clamping surface is designed with biomimetic patterns of tree frog feet and covered with prisms. The prisms are arranged in a hexagonal shape and the top of the prisms is designed with a semi-circular groove. When squeezed by the fruit stalk, it will undergo elastic deformation to form a tiny suction cup. It can provide pressure and friction before the end effector performs the cutting action, fix the fruit stalk and protect the fruit stalk while generating sufficient friction to ensure that the string of fruits will not fall off due to the fruit stalk being cut off. Attached Figure Description

[0031] Figure 1 This is a front view of the fruit harvesting device of the present invention.

[0032] Figure 2 This is a left view of the fruit harvesting device of the present invention.

[0033] Figure 3 This is a side view of the fruit harvesting device of the present invention.

[0034] Figure 4 This is a front view of the end effector.

[0035] Figure 5 This is the left view of the end effector.

[0036] Figure 6 This is a side view of the end effector.

[0037] Figure 7 A schematic diagram of the supporting platform.

[0038] Figure 8 This is a microstructure diagram of a soft silicone clamping pad.

[0039] Figure 9 This is a flowchart of the end effector's workflow.

[0040] Figure 10 This is a flowchart for visual recognition.

[0041] The components include: 1. End effector; 2. Robotic arm; 3. Collection box; 4. Industrial computer; 5. Support platform; 6. Binocular camera; 7. Blade reinforcement plate; 8. Linkage rod; 9. Cutting blade; 10. Base plate; 11. Motor drive shaft; 12. Cam; 13. Stepper motor; 14. Connecting plate; 15. Crescent toothed blade; 16. Clamping pad; 17. Spring; 18. Clamping pad connecting rod; 19. Cam pressure plate; 20. Motor mounting plate; 21. Drive wheel; 22. Track; 23. Load-bearing wheel; 24. Chassis cover; 25. Frame; 26. Flange; 27. Robot base. Detailed Implementation

[0042] To facilitate understanding of the present invention, specific embodiments will be described in detail below. These embodiments will help those skilled in the art to further understand the present invention; however, they are not intended to limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements to the present invention without departing from its conceptual framework, and these modifications and improvements all fall within the scope of protection of the present invention.

[0043] Example 1

[0044] like Figure 1 , Figure 2 , Figure 3 As shown, a deep learning-based integrated fruit harvesting device includes an end effector 1, a robotic arm 2, a collection frame 3, an industrial computer 4, a vision recognition module, and a support platform, i.e., a tracked chassis 5. The end effector is located at the end of the robotic arm, the robotic arm and the industrial computer are mounted on the support platform, and the collection frame is located in front of the support platform. The vision recognition module includes a binocular camera 6; the binocular camera 6 is fixed on a connecting plate 14, which is connected to a base plate 10. The connecting plate 14 and the robotic arm flange 26 are connected by screws, the robotic arm base 27 is mounted on the chassis cover 24 by screws, the collection frame 3 is mounted in front of the chassis by screws, and the industrial computer 4 is mounted on the chassis cover 24. Figure 7 As shown, the chassis 5 includes drive wheels 21, tracks 22, load-bearing wheels 23, and a frame 25. The frame is equipped with components such as batteries, chassis control boxes, motors, and reducers.

[0045] like Figure 4 , Figure 5 , Figure 6As shown, the end effector 1 includes a cutting mechanism and a clamping mechanism. The cutting mechanism includes a cutting blade 9, a crescent-shaped toothed blade 15, a base plate 10, a blade reinforcement plate 7, a connecting rod 8, a cam 12, a stepper motor 13, a motor drive shaft 11, and a motor mounting plate 20. The base plate 10, the connecting plate 14, and the motor mounting plate 20 form the main frame of the entire end effector, and all components are mounted on these three parts. The end effector is connected to the robotic arm flange 26 with screws through six through holes on the connecting plate 14. Two lower connecting rods 8 are connected below the cutting blade 9; the other end of one lower connecting rod is connected to the base plate 10, and the other end of the other lower connecting rod is connected to the motor drive shaft 11. A connecting rod is connected above the cutting blade 9, and the other end of the upper connecting rod is connected to the blade reinforcement plate 7. The lower end of the blade reinforcement plate 7 is connected to the base plate 10 with screws. The crescent-shaped toothed blade 15 is directly connected to the base plate. The clamping mechanism includes a base plate 10, a cam 12, a stepper motor 13, a cam pressure plate 19, a spring 17, clamping pad connecting rods 18, soft silicone clamping pads 16, a motor drive shaft 11, and a motor fixing plate 20. The stepper motor 13 is mounted on the motor fixing plate 20 with screws, and the motor fixing plate 20 is fixed to the base plate 10. The cam 12 is connected to the motor drive shaft 11. The cam pressure plate 19 is connected to two soft silicone clamping pads 16 via two clamping pad connecting rods 18, and each clamping pad connecting rod 18 is fitted with a spring 17. The clamping pads 16 are made of soft silicone material, and the clamping surface is designed with a biomimetic texture resembling a tree frog's foot, covered with prisms arranged in a hexagonal shape, with a semi-circular groove designed on the top of each prism. Figure 8 The image shown is a microstructure diagram of a soft silicone clamping pad.

[0046] The visual recognition module includes a binocular camera 6, an industrial control computer 4, and a connection board 14. The binocular camera 6 integrates two camera lenses, a camera image sensor, and a camera processor that can be connected to the robot's industrial control computer via a USB cable. The robot's industrial control computer is a desktop computer running a Linux system. The robot's industrial control computer uses the YOLOv5 algorithm to identify the clusters of fruit and the fruit stems that need to be cut and held in the image.

[0047] To address the challenging working environments in agricultural settings, characterized by a diverse array of objects including leaves, branches, and fruits against a cluttered background, the existing YOLOv5 fruit cluster recognition network was improved. This resulted in an enhanced YOLOv5 deep learning network. Specifically, the YOLOv5 backbone network contains four structural blocks that primarily learn the residual features of the vectors transmitted through the network. In the fourth structural block, an efficient attention mechanism, the CA attention module, was added. This module not only extracts information about vectors from different channels but also extracts positional information from long-distance vector operations. Through training, the network becomes more sensitive to information about fruit clusters and branches in complex environments, resulting in higher recognition accuracy. Furthermore, this invention addresses the possibility of pseudo-fruit stalks, such as dead branches, in complex environments. A corresponding algorithm for fruit stalk recognition was developed, such as… Figure 9 As shown, the YOLOv5 network identifies the distinctive cluster of fruits and obtains their position in the image. Simultaneously, starting from the center of the predicted bounding box of the cluster, an equilateral triangle with a height twice the width of the predicted bounding box and a vertices angle of 150 degrees is constructed upwards as the region of interest (ROI) for the fruit stem. Since a larger ROI increases the likelihood of false stems, this invention uses a triangle shape to construct the ROI, thus avoiding the appearance of false stems within the ROI. The image of the ROI is then extracted and padded with black pixels to form a rectangular image for YOLOv5 detection of the target fruit stem, obtaining the predicted bounding box of the stem. The center of the predicted bounding box is then used as the position of the fruit stem.

[0048] Then, using the left and right views of the binocular camera 6, stereo matching is performed to obtain the real-world 3D coordinates of the identified fruit stem. The industrial control computer then controls the chassis and robotic arm to move the end effector to the position of the fruit stem based on the calculated 3D coordinates. The industrial control computer controls the rotating shaft of the stepper motor 13 to rotate 90 degrees. Driven by the cam 12, the soft silicone clamping pad 16 clamps the fruit stem. At the same time, the rotating shaft 11 of the motor drives the cutting blade 9 to the initial cutting position. The industrial control computer controls the rotating shaft of the stepper motor 13 to rotate 180 degrees again. The cutting blade 9 completes the cutting operation and moves to the cutting end position. However, at this time, the soft silicone clamping pad 16 is still in the clamping state, and the fruit stem is in a state of being cut but suspended and clamped. The robot industrial control computer controls the robotic arm to move the end effector above the collection box, and then controls the output shaft 11 of the stepper motor to rotate 90 degrees. Under the action of the spring 17, the soft silicone clamping pad 16 releases the fruit stem, and the string of fruits falls into the collection box.

[0049] Specifically, the target for harvesting is the cluster of tomatoes, such as... Figure 10 As shown, the working steps of the integrated clamping and shearing fruit harvesting device in this embodiment are as follows:

[0050] 1) Acquire the image of the current field of view: The industrial control computer controls the binocular camera 6 to capture images of the left and right lenses of the pair of cameras at the current position via a wired connection;

[0051] 2) Fruit Cluster Recognition: Using the left view in the captured image as a baseline, a trained improved YOLOv5 deep learning network is used in the industrial control computer to identify all tomato clusters in the left view. Each cluster is assigned a number from left to right in the image. Then, the length, width, center point, target type, and corresponding confidence score of the minimum bounding rectangle for each cluster are calculated. Figure 10 The confidence score of the detected cross-identified results is 0.92. If the network calculates a confidence score of less than 0.5 for the target, it will be considered a misidentification and will be ignored.

[0052] 3) Obtaining the 3D coordinates of the fruit clusters: The camera is calibrated using Zhang Zhengyou's binaural calibration method to obtain binaural calibration parameters such as the intrinsic matrix, intrinsic parameter matrix, rotation matrix, and distortion coefficients. After identifying the tomato clusters, the epipolar calibration of the left and right images of the camera is performed using the obtained intrinsic parameter matrix, rotation matrix, and distortion coefficients to obtain calibrated left and right views. Then, the SGBM algorithm is used to perform stereo matching on the left and right calibrated views to obtain disparity maps. The disparity map pixels corresponding to the minimum bounding rectangle of the fruit clusters are multiplied by the intrinsic matrix obtained from the binaural calibration to obtain the real-world 3D coordinates of each pixel in the fruit cluster, thus obtaining the 3D coordinates of each fruit cluster.

[0053] 4) The end effector moves to the pre-harvesting position: After obtaining the three-dimensional coordinates of each bunch of fruit, the industrial control computer performs forward and inverse kinematics on the posture of the robotic arm, calculates the trajectory planning of the robotic arm and the walking chassis, moves the end effector to the same horizontal line as the bunch of fruit, and then performs step 2 to obtain the image of the bunch of fruit at close range.

[0054] 5) Cropping Point Confirmation: For the fruit cluster image obtained in the above steps, fruit cluster target recognition is performed. Using the center of the bounding rectangle of the target fruit cluster as the vertex, an equilateral triangle region with a vertex angle of 150 degrees and a height equal to the width of the bounding rectangle is constructed as the region of interest. The image region containing the fruit stem target is extracted, and the improved YOLOv5 is used for fruit stem detection. The bounding rectangle of the fruit stem target and the confidence score of the fruit stem target are calculated, such as... Figure 10 The confidence level of the detected fruit stem target is 0.96. If the confidence level is lower than 0.5, it will be regarded as a false identification and will not be used as the operation object. Finally, the center of the fruit stem detection box is used as the cutting point.

[0055] 6) Reaching the cutting point: The industrial control computer converts the calculated real-world three-dimensional coordinates of the cutting point into three-dimensional coordinates under the coordinates of the robotic arm base, and controls the robotic arm to drive the end effector to reach the cutting point;

[0056] 7) Clamping the fruit stem: The industrial control computer controls the stepper motor 13 to rotate the cam 12 by 90 degrees. Driven by the cam 12, the cam pressure plate 19 clamps the fruit stem with the soft silicone clamping pad 16, while the cutting blade 9 moves to the initial cutting position.

[0057] 8) Cutting the fruit stem: The industrial control computer controls the output shaft of the stepper motor 13 to rotate 180 degrees, so that the cutting blade 9 completes the cutting action and moves to the cutting end position. At the same time, under the action of the cam 12, the silicone clamping pad 16 keeps clamping the fruit stem.

[0058] 9) Place the fruit bunches in the designated position: After the industrial control computer controls the robotic arm to move the end effector to the preset position, it controls the stepper motor 13 to rotate the cam 12 by 90 degrees. At this time, the spring 17 on the clamping pad connecting rod 18 causes the silicone clamping pad 16 to release the fruit stem, and the fruit bunches fall into the designated collection box.

[0059] It is understood that the above specific description of the present invention is only for illustrating the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that local modifications or equivalent substitutions can still be made to the present invention to achieve the same technical effect; as long as the use needs are met, they are all within the protection scope of the present invention.

Claims

1. A deep learning-based method for harvesting bunched fruits using a combination of clamping and shearing, characterized in that: It is a fruit harvesting device that uses a clamping and shearing integrated mechanism, including an end effector, a robotic arm, a collection frame, an industrial control computer, a vision recognition module, and a support platform; the end effector includes a cutting mechanism and a clamping mechanism; the end effector is located at the end of the robotic arm, the robotic arm and the industrial control computer are mounted on the support platform, and the collection frame is located in front of the support platform; The cutting mechanism includes a cutting blade, a crescent-shaped toothed blade, a base plate, a blade reinforcing plate, and connecting rods. Two lower connecting rods are connected to the lower part of the cutting blade; one lower connecting rod's other end is connected to the base plate, and the other lower connecting rod's other end is connected to the motor drive shaft. An upper connecting rod is connected to the upper part of the cutting blade, and its other end is connected to the blade reinforcing plate. The lower end of the blade reinforcing plate is connected to the base plate. The crescent-shaped toothed blade is directly connected to the base plate. The clamping mechanism includes a base plate, a cam, a stepper motor, a cam pressure plate, a spring, a clamping pad connecting rod, a clamping pad, a motor drive shaft, and a motor fixing plate. The stepper motor... The system is mounted on a motor mounting plate, which is fixed to a base plate. A cam is connected to the motor drive shaft. The cam pressure plate is connected to two clamping pads via two clamping pad connecting rods, each with a spring. The vision recognition module includes a binocular camera. The binocular camera is fixed to a connecting plate, which is connected to the base plate. The binocular camera includes two camera lenses, a camera photosensitive element, and a camera processor. The camera photosensitive element receives external light signals through the camera lenses, converts the light signals into electrical signals, and transmits them to the camera processor. The camera processor processes the electrical signals into image information and transmits the image information to an industrial control computer. The harvesting method includes the following steps: Step 1: Acquire the image of the current field of view: Use a stereo camera to capture images of the left and right lenses at the current position; Step 2: Fruit Cluster Recognition: Based on the left view in the captured image, the improved YOLOv5 deep learning network trained in the industrial control computer is used to identify all the fruit clusters in the left view, and each fruit cluster is assigned a number from left to right in the image. Then, the length, width and center point of the minimum bounding rectangle of each fruit cluster are calculated. Step 3: Obtain the 3D coordinates of the string of objects: Use Zhang Zhengyou's binocular calibration method to perform binocular calibration on the stereo camera, and obtain the binocular calibration parameters, including the eigenvalue matrix, intrinsic parameter matrix, rotation matrix, and distortion coefficients. After identifying the string of objects, use the obtained intrinsic parameter matrix, rotation matrix, and distortion coefficients to perform epipolar calibration on the left and right images of the camera, and obtain the calibrated left and right calibrated views. Then, use the SGBM algorithm to perform stereo matching on the left and right calibrated views to obtain the disparity map. Multiply the disparity map pixels corresponding to the minimum bounding rectangle of the string of objects with the eigenvalue matrix obtained by binocular calibration to obtain the real-world 3D coordinates of each pixel of the string of objects, thereby obtaining the 3D coordinates of each string of objects. Step 4: Move the end effector to the pre-harvesting position: After obtaining the three-dimensional coordinates of each bunch of fruit, the industrial control computer performs forward and inverse kinematics on the posture of the robotic arm, calculates the trajectory planning of the robotic arm and the walking chassis, moves the end effector to the same horizontal line as the bunch of fruit, and then performs Step 2 to obtain the image of the bunch of fruit at close range. Step 5: Cutting point confirmation: Perform fruit target recognition on the fruit cluster image obtained at close range. With the center of the bounding rectangle of the target fruit cluster as the vertex, construct an equilateral triangle region with a vertex angle of 150 degrees and a height equal to the width of the bounding rectangle of the fruit cluster as the region of interest. Extract the image region containing the fruit stem target and perform fruit stem detection. Use the center of the fruit stem detection box as the cutting point. Step 6: Reaching the cutting point: The industrial control computer converts the calculated real-world 3D coordinates of the cutting point into 3D coordinates under the coordinates of the robotic arm base, and controls the robotic arm to drive the end effector to reach the cutting point; Step 7: Clamping the fruit stem: The industrial control computer controls the stepper motor to rotate the cam 90 degrees. Driven by the cam, the cam pressure plate clamps the fruit stem with the clamping pad, while the cutting blade moves to the initial cutting position. Step 8: Cutting the fruit stem: The industrial control computer controls the output shaft of the stepper motor to rotate 180 degrees, so that the cutting blade performs the cutting action. The cutting blade and the toothed blade on the crescent toothed blade will cut the fruit stem. After the cutting is completed, the cutting blade moves to the cutting end position, and at the same time, the clamping pad holds the fruit stem under the action of the cam. Step 9: Place the fruit bunches in the designated location: After the industrial control computer controls the robotic arm to move the end effector to the preset position, it controls the stepper motor to rotate the cam 90 degrees. At this time, the spring on the clamping pad connecting rod causes the clamping pad to release the fruit stem, and the fruit bunches fall into the designated collection box.

2. The harvesting method for bunched fruits using a clamping and shearing mechanism according to claim 1, characterized in that: In step 2, the improved YOLOv5 deep learning network is an improvement on the YOLOv5 fruit cluster recognition network. The YOLOv5 backbone network has four main structural blocks that learn the residual features in the vectors transmitted by the network. The fourth structural block adds a CA attention module with an attention mechanism. The CA attention module can not only extract information between different channels of vectors in the vectors operated in the network, but also extract positional information in long-distance vector operations. Through training, the network can become more sensitive to information about fruit clusters and fruit branches in complex environments, thereby achieving higher recognition accuracy.

3. The harvesting method for bunched fruits using a clamping and shearing mechanism according to claim 1, characterized in that: The clamping pad is made of soft silicone material, and the clamping surface is designed with biomimetic patterns of tree frog feet. It is covered with prisms arranged in a hexagonal shape, and the top of the prisms is designed with a semi-circular groove.

Images