Multi-arm picking robot based on YOLOv8 recognition algorithm and fruit recognition method

Abstract

The invention relates to the technical field of agricultural fruit picking, and discloses a multi-arm picking robot based on a YOLOv8 recognition algorithm and a fruit recognition method. The robot comprises a main frame, three groups of picking execution arms which are arranged in parallel at equal intervals, a replaceable end effector, a depth camera and a tomato real-time detection model optimized based on YOLOv8. An RGB-D image is collected through a depth camera, an LSKA attention mechanism is embedded in an optimization model behind an SPPF module of a YOLOv8 backbone network, the feature extraction capability of a dense fruit scene is improved, and a prediction bounding box is output; image coordinates are converted into a Cartesian coordinate system of the mechanical arm in combination with depth information, the optimal path of three-arm cooperation is planned through a PLC, transverse and longitudinal moving shafts are preferentially driven, and finally an oblique telescopic shaft is controlled to avoid obstacles. Through the multi-arm parallel structure, the high-precision recognition algorithm and the modular design, the problems that a traditional picking robot is low in efficiency and poor in adaptability are solved, and the robot has the advantages of being high in robustness and low in cost and is suitable for large-scale precise operation of agriculture.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural fruit picking, and in particular to a multi-arm picking robot and a fruit recognition method based on the YOLOv8 recognition algorithm. Background Art

[0002] In my country's orchards, harvesting ripe fruit is the most time-consuming and labor-intensive process, largely done manually. The low rate of mechanization in harvesting has led to rising labor costs in fruit cultivation, which now account for 50% to 70% of total production costs.

[0003] my country is a major producer of fruits and vegetables, and tomatoes are a key crop. Harvesting is labor-intensive and inefficient. Agricultural robots can improve production efficiency, address labor shortages, reduce labor intensity and costs, and enable large-scale, diversified, and precision agricultural production.

[0004] The current problems faced by tomato production can be solved by designing a multi-arm rapid harvesting robot for a standard tomato greenhouse. However, the tomato harvesting robots currently used in facility agriculture environments have the following problems and difficulties: First, the target detection model has a slow reasoning speed and a low recognition success rate when faced with high-density tomato fruit scenes, which cannot meet the real-time performance requirements of the harvesting robot; the single-arm tomato harvesting robot has a long operation cycle and low picking efficiency and cannot meet production needs.

[0005] Therefore, we propose a multi-arm picking robot and fruit recognition method based on YOLOv8 recognition algorithm to solve the problems in the above background. Summary of the Invention

[0006] This invention provides a multi-arm harvesting robot and fruit recognition method based on the YOLOv8 recognition algorithm. These methods address the existing issues of slow reasoning speed and low recognition success rate in high-density tomato fruit harvesting robots, resulting in inability to meet the real-time performance requirements and low picking efficiency. Targeting the high crop density and high yield of vertical tomato greenhouses, a multi-arm parallel integrated structure and fruit recognition method are designed. Key design elements include the configuration of an oblique grasping actuator, the structural design of a horizontal linear track conveying module, a replaceable end-effector structure, a large-volume collection device, and a YOLOv8-based real-time tomato detection model. These features ensure a total system load of 50 kg, enabling the harvest of 400 fruits in a single operation. During continuous harvesting, the system achieves real-time reasoning performance and a harvesting rate of over 8 fruits per minute, effectively improving the mechanical efficiency of tomato harvesting.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a multi-arm picking robot based on the YOLOv8 recognition algorithm, comprising a main frame and multiple picking execution arms installed on the main frame, the bottom of the main frame is provided with a transport and collection mechanism for collecting and transmitting fruits, the picking execution arm is installed on one side of the transport and collection mechanism, and the movable end of the picking execution arm is installed with a replaceable end effector; a lower computer display screen and an upper computer display screen are fixedly installed on one side of the main frame, and a depth camera and a tomato real-time detection device for analyzing images from the depth camera based on the YOLOV8 algorithm are installed on the main frame, the tomato real-time detection device is electrically connected to a PLC controller, and the PLC controller is used to control the motion of the picking execution arm and the replaceable end effector.

[0008] Preferably, there are three groups of picking execution arms and they are arranged in parallel and equidistantly, namely the first robotic arm, the second robotic arm, and the third robotic arm. The picking execution arms include an X-axis module, a Y-axis module, and a Z-axis module. The replaceable end effector is connected to the Z-axis module. The X-axis module is horizontally arranged and fixedly connected to the main frame. The Y-axis module is vertically connected to the X-axis module, and the Z-axis module is tilted relative to the Y-axis module.

[0009] A fruit recognition method for a multi-arm picking robot based on the YOLOv8 recognition algorithm includes the following steps:

[0010] S1, collects RGB-D images of the target area in real time through the depth camera installed on the main frame, where the RGB image is used for fruit detection and the depth information is used to calculate the three-dimensional space coordinates;

[0011] S2. Input the RGB image into a real-time tomato detection model optimized based on YOLOv8. This model adds an LSKA attention mechanism after the feature pyramid module SPPF in the backbone network to enhance feature extraction in dense fruit scenes and generate a predicted bounding box containing the ripe tomato fruit.

[0012] S3. Based on the coordinates of the center point of the predicted bounding box and the depth information from the depth camera, a coordinate conversion algorithm is used to map the image coordinate system to the Cartesian coordinate system of the robotic arm, generating the target picking coordinates for the grasping device. Based on the target picking coordinates, a path planning algorithm is used to determine the optimal motion paths for the first, second, and third robotic arms. The sequence of the horizontal and vertical axes is prioritized, and the extension and retraction of the diagonal axis is finally controlled to prevent interference between the robotic arms and the vines.

[0013] S4. Transmits the target coordinates and path instructions to the PLC controller, which drives the X-axis motor, Y-axis motor, and Z-axis motor to move in coordination, causing the replaceable end effector to reach the target position.

[0014] The flexible gripper is controlled by the gripper opening and closing motor to close and grab the fruit, and then the rotary motor drives the rotating platform to complete the fruit twisting and separation;

[0015] S5. During the continuous picking process, dynamically adjust the conveyor belt speed and the picking sequence of the robotic arm to ensure that no less than 8 fruits are picked per minute.

[0016] Compared to existing technologies, the present invention achieves the following beneficial effects: The present invention designs a multi-arm agricultural robot with a parallel multi-arm structure and an optimized recognition algorithm based on the YOLOv8 model, including a three-arm oblique grasping actuator. Compared to most commercially available six-degree-of-freedom articulated harvesting robots, it has a larger coverage area and is suitable for harvesting with single-sided coverage. This solves the problem of the limited flexibility and range of six-degree-of-freedom harvesting robots in greenhouse tomato environments. The slide module structure ensures precise grasping and faster execution of harvesting commands from the host computer. Its flexibility adapts to tomato fields with varying planting densities and layouts, enabling intelligent and refined harvesting strategies. The replaceable grasping mechanism and the twisting action at the end solve the problem of difficult-to-remove thick fruit stems, adapting to various harvesting tasks. The actuator can be adaptively replaced according to the target fruit. The high-performance, low-latency image inference framework used in the robot efficiently infers the YOLOv8 model, meeting the performance requirements of real-time detection. By introducing the LSKA attention mechanism, the present invention improves the model's feature extraction capabilities in dense scenes, enabling more accurate recognition of ripe tomatoes. The sliding multi-arm parallel picking robot and fruit recognition algorithm have lower maintenance and design costs, are more suitable for picking operations in facility tomato cultivation environments, and have higher work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0018] Figure 2 is a schematic diagram of a picking arm of the present invention;

[0019] Figure 3 is a schematic diagram of the replaceable gripping device of the present invention;

[0020] Figure 4 It is a schematic diagram of the assembly of the longitudinal moving shaft and the transverse moving shaft of the present invention;

[0021] Figure 5 Schematic diagram of the limiting device of the oblique telescopic rod of the present invention;

[0022] Figure 6 is a structural diagram of the camera platform of the present invention;

[0023] Figure 7 is a schematic diagram of a transmission device of the present invention;

[0024] Figure 8 It is an expanded parts diagram of the longitudinal movable shaft of the present invention;

[0025] Figure 9 It is the inference flow chart of the image recognition method of the present invention.

[0026] Among them: 1. First robotic arm; 2. Second robotic arm; 3. Third robotic arm; 4. Replaceable end effector; 5. Depth camera; 6. Lower computer display screen; 7. Upper computer display screen; 8. Transport module; 9. Collection module; 10. Main frame; 101. Grasping device; 102. End connecting plate; 103. Oblique moving axis; 104. Longitudinal moving axis; 105. X-axis motor; 106. Transverse moving axis; 107. Y-axis motor; 108. Z-axis motor; 109. Oblique connecting plate; 111. T-shaped connecting plate; 201. Flexible gripper; 202. Gripper opening and closing motor; 203. Gripper base; 204. Rotating motor; 205. Gripper fixing plate; 20 6. Motor housing; 207. Rotating platform; 301. Sealed slide; 302. Slide base plate; 303. Y-axis module base; 304. Y-axis slide base; 305. Y-axis guide rail; 306. Reinforcement rib base; 307. Reinforcement rib plate; 308. X-axis slide; 310. Slide fixing plate; 311. X-axis motor bracket; 401. Positive limit switch; 402. Z-axis base plate; 403. Limit metal sheet; 404. Negative limit switch; 405. Z-axis guide rail; 406. Z-axis motor bracket; 501. Camera angle adjuster; 502. Camera bracket; 601. Anti-drop baffle; 602. Conveyor belt; 603. Anti-roll baffle; 604. Transmission speed control console. DETAILED DESCRIPTION

[0027] The specific embodiments of the present invention are described in detail below, but it should be understood that the protection scope of the present invention is not limited by the specific embodiments.

[0028] Reference Attachment Figure 1 and 2 As shown, this embodiment discloses a multi-arm picking robot based on the YOLOv8 recognition algorithm, including a main frame 10 and multiple picking execution arms installed on the main frame 10. The bottom of the main frame 10 is provided with a transport and collection mechanism for collecting and transporting fruits, and the movable end of the picking execution arm is installed with a replaceable end effector 4; the multiple picking execution arms in this embodiment work in parallel.

[0029] A lower computer display screen 6 and an upper computer display screen 7 are fixedly installed on one side of the main frame 10. A depth camera 5 and a tomato real-time detection device that analyzes the images of the depth camera 5 based on the YOLOV8 algorithm are installed on the main frame 10. The tomato real-time detection device is electrically connected to a PLC controller, which is used to control the motion of the picking execution arm and the replaceable end effector 4.

[0030] In one embodiment of the above solution, Figure 1 and Figure 2 As shown, there are three groups of picking execution arms and they are arranged in parallel and equidistantly, namely the first robotic arm 1, the second robotic arm 2, and the third robotic arm 3. The picking execution arms include an X-axis module, a Y-axis module, and a Z-axis module. The replaceable end effector 4 is connected to the Z-axis module. The X-axis module is horizontally arranged and fixedly connected to the main frame 10. The Y-axis module is vertically connected to the X-axis module, and the Z-axis module is tilted relative to the Y-axis module.

[0031] The specific implementation is that the grasping device 101 is connected to the oblique connecting shaft through the end connecting plate 102, the transverse moving shaft 106 and the longitudinal moving shaft 104 are installed vertically at 90° through the T-shaped connecting plate 111, and the two moving shafts are fixed by the triangular reinforcing rib plate 307 to avoid swinging during operation. The transverse moving shaft 106 is installed on the main frame 10 through four pairs of bolts, and the oblique moving shaft 103 is fixed to the longitudinal moving shaft 104 through the oblique connecting plate 109. The slide module on the rear side of the oblique connecting plate 109 can be fixed on the rail in the Y-axis direction. The X-axis motor 105 is fixed to the end of the transverse moving shaft 106 through four pairs of fastening bolts. The output shaft of the X-axis motor 105 is connected to the transverse moving module. The input torque of the X-axis motor 105 drives the slide rail to realize linear sliding on the X-axis slide 308, which is the transverse moving shaft 106. The Y-axis motor 107 and the Z-axis motor 108 are installed in the same manner. The only difference in the Z-axis is that it is installed at an oblique angle. The movement methods and principles of the three axes are the same.

[0032] In another embodiment of the above X-axis module, Figure 2 and Figure 4As shown, the X-axis module includes a transverse moving shaft 106 and a driving source. The transverse moving shaft 106 is fixedly mounted on the main frame 10. A linear sliding assembly is provided on the transverse moving shaft 106. An X-axis guide rail is provided on the transverse moving shaft 106. An X-axis slide 308 is slidably connected to the X-axis guide rail. The side of the transverse moving shaft 106 is fixedly connected to a sealed slide 301. The outer sliding portion of the sealed slide 301 is slidably matched with a slide base plate 302. The side of the X-axis slide 308 is fixedly connected to a slide fixing plate 310. The inner side of the slide fixing plate 310 is fixedly connected to the slide base plate 302. The driving source is the X-axis motor 105. The X-axis motor 105 drives the linear sliding assembly. The linear sliding assembly drives the X-axis slide 308 to slide on the X-axis guide rail.

[0033] The linear sliding assembly is an existing technology and can adopt a screw drive or a belt drive; when the belt drive method is adopted, a belt and a belt gear are installed inside the lateral moving shaft 106, and the output shaft of the X-axis motor 105 is connected to the belt gear inside the lateral moving shaft 106, and the belt is connected to the X-axis slide 308. After the X-axis motor 105 outputs torque, the belt gear rotates and drives the belt transmission, and the belt drives the X-axis slide 308 to slide along the X-axis guide rail. The installation and drive method between the Z axis and the Y axis are the same. The screw drive listed therein is an existing technical means and will not be repeated here.

[0034] In another embodiment of the Y-axis module, Figure 4 As shown, the Y-axis module includes a longitudinal moving shaft 104, a linear sliding assembly and a Y-axis motor 107. The longitudinal moving shaft 104 and the transverse moving shaft 106 are arranged at 90 degrees. The upper part of the X-axis slide 308 is fixedly connected to the Y-axis module base 303, and the Y-axis slide base 304 is fixedly connected to the Y-axis module base 303. The surface of the main frame 10 of the longitudinal moving shaft 104 along the axial direction is provided with a Y-axis guide rail 305 main frame 10, and the side of the main frame 10 of the longitudinal moving shaft 104 is provided with a reinforcing rib base 306 main frame 10. The reinforcing rib base 306 main frame 10 They are respectively fixedly connected to the sides of the longitudinal moving axis 104 main frame 10 and the Y-axis slide base 304 main frame 10. The Y-axis module base 303 main frame 10 is fixedly connected to the reinforcing rib plate 307 main frame 10. The Y-axis guide rail 305 main frame 10 is slidably connected to the oblique connecting plate 109 main frame 10. The Y-axis motor 107 is installed at one end of the longitudinal moving axis 104 main frame 10 and is used to drive the linear sliding assembly on the longitudinal moving axis 104 main frame 10. The linear sliding assembly drives the oblique connecting plate 109 main frame 10 to slide on the Y-axis guide rail 305 main frame 10.

[0035] In another embodiment, the specific assembly diagram of the Y-axis module and the X-axis module is shown. The X-axis and the Y-axis are installed in the same plane at a 90° angle. The X-axis module is horizontal to the ground, and the Y-axis is installed vertically above the X-axis. First, the Y-axis guide rail 305 and the Y-axis slide base 304 are fixed to the rear reinforcing rib base 306 by means of openings using three bolts. At the same time, the Y-axis slide base 304 is fixed to the Y-axis module base 303. The Y-axis module base 303 is installed with a reinforcing rib base 306 and a reinforcing rib plate 307. The function of the reinforcing rib plate 307 is to prevent the Y-axis slide rail from shaking and to assist in installation. The X-axis slide 308 is responsible for connecting the belt gear inside the X-axis. The X-axis motor 105 outputs torque. The X-axis motor 105 is fixed to the end of the X-axis module via the X-axis motor bracket 311 to achieve horizontal movement of the Y-axis in the X-axis direction. The sealed slide 301, the slide base 302 and the slide fixing plate 310 are combined into an auxiliary side slide rail to control the stability of the slide movement. The installation and drive method between the Z axis and the Y axis are the same.

[0036] like Figure 2 and Figure 3 As shown, the replaceable end effector 4 includes a gripping device 101 and a gripper fixing plate 205, wherein the gripper fixing plate 205 is an L-shaped angle plate, which is used to fix the gripping device 101 to the oblique moving shaft 103; the gripping device 101 includes at least two flexible grippers 201, a deformation frame, and a gripper opening and closing motor 202, wherein the flexible grippers 201 are all made of rubber material, and the output shaft of the gripper opening and closing motor 202 is connected to the inside of the deformation frame. The four flexible grippers 201 are mounted on the deformation frame, and the opening and closing of the flexible grippers 201 are controlled by the forward and reverse rotation of the motor. The end of the gripper opening and closing motor 202 away from the deformation frame is fixedly connected to a rotating platform 207;

[0037] The end of the clamp fixing plate 205 close to the rotating platform 207 is fixedly connected to the motor housing 206 main frame 10, and the rotating motor 204 main frame 10 is fixedly installed inside the motor housing 206 main frame 10. The end of the motor housing 206 main frame 10 close to the flexible clamp 201 is fixedly connected to the clamp base 203 main frame 10, and the clamp base 203 main frame 10 is rotatably connected to the rotating platform 207 main frame 10. The output shaft of the rotating motor 204 passes through the clamp base 203 main frame 10 and is fixedly connected to the center of the rotating platform 207 main frame 10.

[0038] The specific implementation of the above scheme is that the replaceable end effector 4 includes a head grasping device 101 and a tail clamping claw fixing plate 205, and the clamping claw fixing plate 205 is a 90° angle iron, which fixes the grasping device 101 to the oblique moving shaft 103. This example is a four-claw grasping device 101, which is suitable for grasping round fruits. Four flexible clamping claws 201 are used at the end, and the material is rubber material to reduce the damage problem during the grasping process and increase the contact friction. The output shaft of the clamping claw opening and closing motor 202 is a small screw, which is connected to the four flexible clamping claws 201 through a deformation frame. The opening and closing of the flexible clamping claws 201 is controlled by the forward and reverse rotation of the motor. Behind the clamping claw opening and closing motor 202 is a rotating platform 207. The rotating platform 207 has an opening and is connected to the rear rotating motor 204. The rotating platform 207 is directly the output shaft surface of the rotating motor 204. The knob action of the rotating motor 204 simulates the dragging action of the human hand after the fruit grasping action is completed. The entire gripping device 101 is connected via a clamping claw fixing plate 205 , and a suitable gripping device 101 can be selected according to different crops.

[0039] In a further embodiment, Figure 3 As shown, the deformation frame includes a U-shaped seat and a linkage plate. The side of the U-shaped seat away from the U-shaped opening is fixedly connected to the jaw opening and closing motor 202. The linkage plate is movably arranged inside the U-shaped seat. The inner side of the flexible jaw 201 is provided with a support foot, which is rotatably connected to the linkage plate. The output shaft of the jaw opening and closing motor 202 is a small lead screw, and the middle part of the linkage plate is threadedly connected to the small lead screw. In the above scheme, the output shaft of the jaw opening and closing motor 202 is threadedly matched with the middle part of the linkage plate, and the four corners of the linkage plate are rotatably connected to the four flexible jaws 201. When the jaw opening and closing motor 202 rotates in the forward or reverse direction, the linkage plate moves along the axial direction of the output shaft. When the linkage plate moves outward, it drives the four flexible jaws 201 to open outward. When it moves inward, it drives the four flexible jaws 201 to contract and grasp the tomato fruit.

[0040] In one embodiment of the above-mentioned Z-axis module, the Z-axis module includes an oblique moving shaft 103, which is mounted on the outer side of the oblique connecting plate 109. A Z-axis guide rail 405 is provided on the oblique moving shaft 103, and a linear sliding assembly is installed in the Z-axis guide rail 405. The Z-axis guide rail 405 is slidably connected to a Z-axis base plate 402, and the linear sliding assembly is connected to a side of the Z-axis base plate 402 close to the Z-axis guide rail 405; one end of the oblique moving shaft 103 is fixedly connected to a Z-axis motor bracket 409. 06. A Z-axis motor 108 is installed on the Z-axis motor bracket 406; a rotation drive device is installed on the oblique connecting plate 109, and the Z-axis base plate 402 is connected to the driving end of the rotation drive device. The Z-axis motor 108 drives the linear sliding assembly, and the linear sliding assembly drives the Z-axis base plate 402 to slide on the Z-axis guide rail 405, so that the oblique moving shaft 103 slides obliquely with the Z-axis base plate 402 as the base point, and the rotation drive device drives the oblique moving shaft 103 to rotate, thereby adjusting the inclination angle of the oblique moving shaft 103.

[0041] like Figure 5 As shown, in another embodiment, a limit mechanism is installed on the oblique moving shaft 103, and the limit mechanism includes a positive limit switch 401 and a negative limit switch 404. The positive limit switch 401 and the negative limit switch 404 are installed on the Z-axis guide rail 405; two limit metal sheets 403 are installed on one side of the Z-axis base plate 402 close to the positive limit switch 401 and the negative limit switch 404. The positive limit switch 401 and the negative limit switch 404 are both mechanical mechanisms. When the limit metal sheet 403 reaches the preset limit position, the mechanical contact of the limit switch is triggered, and the physical action is immediately converted into an electrical signal.

[0042] A more specific implementation involves opening holes in the upper side of the Z-axis base plate 402. Each limit metal plate 403 is mounted using two screws. A positive limit switch 401 and a negative limit switch 404 are mounted on the Z-axis guide rail 405. These switches are mechanical mechanisms. When the limit metal plate 403 reaches the preset limit position, i.e., the limit switch installation position, the mechanical contacts of the limit switches are triggered, and the physical action is converted into an electrical signal. The installation position of the limit switches can be adjusted as needed on the Z-axis guide rail 405.

[0043] In another embodiment, a mounting platform of the depth camera 5 is fixedly connected to the main frame 10, and the mounting platform includes a camera angle adjuster 501 and a camera bracket 502. The back side of the depth camera 5 is fixedly connected to the camera angle adjuster 501, and the camera angle adjuster 501 is fixedly mounted on the lower camera bracket 502; a specific embodiment, such as Figure 6The figure shows the mounting platform structure of the depth camera 5. The mounting platform includes a camera angle adjuster 501 and a camera bracket 502. The depth camera 5 is connected to the camera angle adjuster 501 via back bolts. The camera angle adjuster 501 is mounted on the lower camera bracket 502 with two pairs of M12 screws. The camera bracket 502 is directly connected to the main frame 10 and fixed with four sets of M16 screws. The minimum tilt angle of the camera angle adjuster 501 is -30°, and the maximum tilt angle is 60°.

[0044] like Figure 7 It is a structural diagram of the transmission device, the transportation and collection mechanism includes a transportation module 8, the transportation module 8 is installed on the main frame 10, and the picking execution arm is installed on one side of the transportation module 8; the transportation module 8 includes a conveyor belt 602 located directly below the origin position of the picking execution arm, the storage box is located below the conveyor belt 602 and is connected to one end of the conveyor belt 602, a transmission speed control console 604 is installed on the side of the conveyor belt 602, an anti-roll baffle 603 is installed on the surface of the conveyor belt 602, and anti-drop baffles 601 are installed on both sides of the conveyor belt 602, and the collection module 9 includes a storage box for storing tomato fruits.

[0045] After picking, the tomatoes are transferred from the transport module 8 to the collection module 9. A transmission speed control console 604 controls the speed of the conveyor belt 602. Anti-roll baffles 603 are installed on the surface of the conveyor belt 602, each 30 cm apart and secured with M12 screws. Anti-drop baffles 601 are installed on both sides of the conveyor belt 602 to prevent the fruits from falling to the ground after picking. The direction and speed of the conveyor belt 602 are controlled by the transmission speed control console 604, which is mounted on the right end of the conveyor. The default transmission direction is from the first robotic arm 1 to the third robotic arm 3, and the conveyor belt 602 speed is 5 cm / s.

[0046] like Figure 9 As shown in FIG, a fruit recognition method of a multi-arm picking robot based on the YOLOv8 recognition algorithm includes the following steps:

[0047] S1. The depth camera 5 is mounted on the main frame 10 to collect real-time RGB-D images of the target area, where the RGB image is used for fruit detection and the depth information is used to calculate the three-dimensional space coordinates;

[0048] S2. Input the RGB image into a real-time tomato detection model optimized based on YOLOv8. This model adds an LSKA attention mechanism after the feature pyramid module SPPF in the backbone network to enhance feature extraction in dense fruit scenes and generate a predicted bounding box containing the ripe tomato fruit.

[0049] S3. Based on the predicted bounding box center point coordinates and the depth information of the depth camera 5, the image coordinate system is mapped to the Cartesian coordinate system of the robotic arm through a coordinate conversion algorithm to generate the target picking coordinates of the grasping device 101;

[0050] Based on the target picking coordinates, the optimal movement paths of the first robotic arm 1, the second robotic arm 2, and the third robotic arm 3 are determined through a path planning algorithm. The movement sequence of the horizontal movement axis 106 and the vertical movement axis 104 is prioritized, and finally the extension and retraction movement of the oblique movement axis 103 is controlled to avoid interference between the robotic arms and the vines.

[0051] S4 transmits the target coordinates and path instructions to the PLC controller, drives the X-axis motor 105, Y-axis motor 107, Z-axis motor 108 to move in coordination, so that the replaceable end effector 4 reaches the target position;

[0052] The flexible clamping jaws 201 are controlled by the clamping jaw opening and closing motor 202 to close and grab the fruit, and then the rotating motor 204 drives the rotating platform 207 to complete the fruit twisting and separation;

[0053] S5. During the continuous picking process, dynamically adjust the speed of the conveyor belt 602 and the picking sequence of the robotic arm to ensure that no less than 8 fruits are picked per minute.

[0054] Further explanation of the PLC controller and the real-time tomato detection device that analyzes images from the depth camera 5 using the YOLO V8 algorithm: The PLC controller receives the detection results from the edge device and executes a coordinate conversion algorithm to convert the image coordinate system into the robotic arm's Cartesian coordinate system. This further controls the three-axis motion of the picking arm (X-axis module, Y-axis module, and Z-axis module) and the gripping action of the gripping device 101.

[0055] The real-time tomato detection device includes an edge computing device and a data transmission module for deploying the YOLOv8 model;

[0056] Edge computing devices include hardware platforms, typically embedded devices such as the NVIDIA Jetson series, with integrated GPU acceleration to meet real-time inference requirements. Simultaneously, model optimization is performed by adding the LSKA (Local Spatial Kernel Attention) mechanism after YOLOv8's SPPF (Spatial Pyramid Pooling Fusion) module to enhance feature extraction in dense fruit scenes. The input size is set to 640×640 pixels to accommodate high-resolution detection requirements, and the output generates a predicted bounding box containing the center coordinates of the ripe tomato fruit, its category, and its confidence score.

[0057] The data transmission module transmits the RGB-D image of the depth camera 5 to the edge computing device via USB / Ethernet, and sends the detection results (target coordinates, depth information) to the PLC controller via CAN bus or RS-485 protocol.

[0058] The PLC controller, real-time tomato detection equipment, depth camera 5 and picking execution arm work together. The working principle is that the depth camera 5 collects RGB-D images in real time, the RGB images are input into the YOLOv8 model, and the depth information is stored separately for subsequent coordinate conversion; in the further image processing stage, the image size is uniformly adjusted to 640×640 pixels.

[0059] Step 2: Perform feature extraction. The optimized YOLOv8 model extracts features through convolutional layers. The LSKA attention mechanism focuses on dense fruit areas to generate a predicted bounding box of the ripe tomato and outputs the center point coordinates (X, Y), width, height, and confidence level.

[0060] Step 3: Combined with the calibration parameters of the depth camera 5, the image coordinate system (X, Y) is converted to the robotic arm Cartesian coordinate system (X, Y, Z).

[0061] Prioritize driving the transverse moving axis 106 and the longitudinal moving axis 104, and finally control the extension and retraction of the oblique moving axis 103, and set the elevation angle of the oblique moving axis 103 to be inclined at 30 degrees to avoid interference with vines.

[0062] Furthermore, the three picking execution arms work together to divide the picking areas of the first robotic arm 1, the second robotic arm 2, and the third robotic arm 3, optimize the total stroke, and the average picking time of a single arm is ≤20 seconds.

[0063] Step 4: The PLC controller controls the picking arm, driving the X-axis motor 105 and Y-axis motor 107 to the target coordinate (X, Y) position. The Z-axis motor 108 controls the extension and retraction of the oblique motion axis 103, causing the flexible gripper 201 to contact the fruit. The gripper opening and closing motor 202 closes the gripper, and the rotation motor 204 twists the fruit, simulating a manual dragging motion. Furthermore, when the positive limit switch 401 or negative limit switch 404 of the Z-axis guide 405 is triggered, the motor stops immediately to prevent mechanical overtravel.

[0064] Step 5: After picking is completed, the gripping device 101 returns to the original position and releases the fruit to the conveyor belt 602. The transmission speed control console 604 dynamically adjusts the speed of the conveyor belt 602 according to the fruit density to ensure synchronization with the picking rhythm.

[0065] like Figure 9As shown in the figure, the real-time tomato detection model is optimized based on the YOLOv8 architecture and is mainly divided into three parts: the backbone network (Backbone), the neck network (Neck) and the head network (Head). Combined with the LSKA attention mechanism and multi-scale feature fusion technology, it significantly improves the detection accuracy and real-time performance in dense fruit scenes. The following is a further detailed description of each module: 1. The backbone network (Backbone) extracts basic features from the input image and enhances the representation ability of key areas through multi-layer convolution and attention mechanisms. The input format is: 640×640×3 RGB image, which is suitable for high-resolution detection requirements.

[0066] Its core modules include Conv (convolutional layer), C2f module, SPPF (Spatial Pyramid Pooling Fusion) and LSKA (Local Spatial Kernel Attention). Among them, Conv (convolutional layer) extracts low-level features such as edges and textures through 3×3 convolution kernels to construct basic feature maps.

[0067] Among them, the C2f module is an improved cross-stage local feature fusion module that combines multi-branch convolution operations to fuse features of different scales and enhance the model's sensitivity to small target fruits; the spatial pyramid pooling fusion module (SPPF) captures contextual information of different receptive fields through multi-scale pooling (such as 5×5, 9×9, 13×13), improving the model's ability to detect occluded or overlapping fruits; the newly added local spatial kernel attention mechanism (LSKA) dynamically adjusts the convolution kernel weights to focus on the feature responses of dense fruit areas; for example, higher weights are assigned to fruit edges and areas with high color contrast to suppress background interference.

[0068] 2. Neck Network: The neck network is responsible for feature fusion and enhancement, integrating the multi-scale features output by the backbone network to provide rich context for detection tasks. Its main operations include upsampling, concatenation, and reuse of the C2f module. Upsampling scales deep feature maps and aligns them with shallow feature maps to preserve detail. Concatenation concatenates feature maps from different layers in the backbone network, for example, deep semantic features with shallow detail features, enhancing multi-scale object detection capabilities. Reuse of the C2f module further optimizes the fused features and improves feature expression.

[0069] 3. The head network (Head) generates the final detection results based on the fused feature maps, including the predicted bounding box location, category, and confidence. The core modules include the Conv (detection convolutional layer) and the Detect module. The Conv (detection convolutional layer) compresses the number of channels through 1×1 convolution and outputs the object location and category probability. The Detect module integrates anchor box matching and non-maximum suppression (NMS) to generate the final predicted bounding box. The output format is the center point X / Y coordinates, width and height W / H, category, and confidence.

[0070] This optimized model incorporates an attention enhancement mechanism, embedding an LSKA module after the SPPF. This allows the model to focus more on the fruit area in dense scenes and reduce interference from vines or leaves. The image input size is optimized to a high-resolution 640×640 pixel input, balancing detection accuracy and computational efficiency to adapt to the computing power of embedded devices. The model is pre-trained using the PyTorch framework and deployed on NVIDIA Jetson edge devices, leveraging GPU acceleration for real-time inference (single-frame processing time ≤ 50ms). Simultaneously, the detection results (center point coordinates + depth information) are transmitted to the PLC controller via the CAN bus, driving the coordinated movement of the three-axis robotic arm to ensure efficient picking, with an average picking time of ≤ 20 seconds for a single picking arm.

[0071] The inference process of the above optimization model is as follows: the RGB image of depth camera 5 is scaled to 640×640 pixels and normalized; the backbone network extracts multi-scale features through Conv, C2f, SPPF+LSKA, focusing on fruit-dense areas; the neck network further integrates multi-level features through upsampling and splicing to generate a high-semantic feature map; the head network predicts the bounding box, combines the depth information to convert it into the robot arm coordinate system, and plans the optimal picking path.

[0072] The optimization effect of this model is manifested in the following aspects: first, the accuracy is improved. The LSKA attention mechanism reduces the missed detection rate of dense fruits by 15%, and it has real-time performance. The edge device inference frame rate is ≥30 FPS, which meets the demand of picking ≥10 fruits per minute. At the same time, SPPF multi-scale fusion effectively copes with complex scenes with fruits of different sizes and severe occlusion. Through the above optimization, the model realizes efficient and accurate fruit recognition and positioning in the tomato picking robot, providing reliable technical support for automated agriculture.

[0073] The specific fruit picking recognition method is as follows: a picking robot enters a protected tomato orchard and begins work. It follows a fixed track through the orchard, enters the row, and selects a target tomato wall on one side. The vision system uses a depth camera 5 and a second depth camera 5 to capture image information of the current robotic arm's target area. The algorithm model uses the optimized YOLOv8 tomato recognition algorithm, and the real-time RGB-D image provided by the depth camera 5 is input to the algorithm model. The input size of the feature map of each image frame is converted into a single-channel feature map through a convolutional layer to obtain the global context of the input feature map. The input size includes the batch size, number of channels, and size of the feature map. The model is deployed using the Pytorch deep learning framework. After processing such as convolution and attention mechanisms, it infers tomatoes that meet the picking criteria and calculates the corresponding center point coordinates for the next step. The robotic arm's picking path determines picking efficiency. By dividing the picking area for each robotic arm, the optimal path is determined to minimize the total picking distance, and the picking sequence for the target ripe fruit is determined. Testing has shown that the average picking time for each arm is ≤20 seconds. Through coordinate transformation, the camera coordinates are converted to the working coordinates of the robotic arm. The control system performs tomato picking according to the host computer's instructions, and the corresponding motors output torque. First, the coordinate movements of the X- and Y-axes are planned, and finally, the Z-axis telescopic shaft is extended and retracted to prevent the robotic arm from becoming entangled with the tomato vines. After the target tomato is picked, the X-axis motor 105 outputs torque, driving the X-axis slide 308 to move to the x-coordinate of the target fruit. Simultaneously, the Y-axis motor 107 outputs torque, driving the oblique telescopic shaft to move to the y-axis coordinate. Because the oblique telescopic shaft has an upward tilt angle, the y-coordinate is converted using trigonometric functions, bringing the gripping device 101 closer to the fruit. The image depth information read by the depth camera 5 is output to the controller, which determines the travel distance of the Z-axis. Driven by the Z-axis motor 108, the oblique moving shaft 103 moves through a gear rack, causing the four flexible grippers 201 of the gripping device 101 to make preliminary contact with the target tomato fruit. After the flexible gripper 201 completes pre-contact with the target tomato fruit, the three-axis slide motion completes all movements, and the target picking position is reached. The gripper opening and closing motor 202 outputs torque, driving the small screw at the end output shaft to control the four flexible grippers 201 to complete the gripping action. After the grippers contact the tomato fruit, the flexible material deforms according to the maturity and firmness of the tomato, mimicking the gripping state of a hand. The rotating motor 204 outputs torque, and its end output shaft is the rotating platform 207. The motor output rotates in the positive direction one circle, that is, the gripping device 101 completes one circle of rotation and then resets. The gripping device 101 completes the gripping action. After completing their respective gripping tasks, the first robotic arm 1, the second robotic arm 2, and the third robotic arm 3 perform a return to zero motion. The return to zero position is the initial position. At this time, the gripping device 101 is directly above the conveyor belt 602, completing the initial picking action.

[0074] In another embodiment, after completing their picking task, the three robotic arms each return to their zero position. After the robotic arms lift the gripped fruit to a certain height, the jaw opening and closing motor 202 controls the flexible jaws 201 to open and release the fruit. The tomato fruit, protected by the anti-drop baffle 601, falls onto the conveyor belt 602, completing the picking operation for each fruit. The fruit is then transferred to the collection module 9. The transmission speed control unit 604, based on information processed by the visual module and the number of fruits in the current work area, adaptively adjusts the speed of the conveyor belt 602 to accommodate the falling rate of the three robotic arms. Based on test data, our harvesting robot has demonstrated a consistent ability to pick at least 10 fruits per minute. This achievement not only demonstrates the high efficiency and reliability of robotics in orchard management but also demonstrates its potential to increase productivity and reduce labor costs.

[0075] The above disclosures are only a few specific embodiments of the present invention. However, the embodiments of the present invention are not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A multi-arm picking robot based on the YOLOv8 recognition algorithm, comprising a main frame (10) and a plurality of picking execution arms mounted on the main frame (10), wherein a transport and collection mechanism for collecting and transporting fruits is provided at the bottom of the main frame (10), and the picking execution arms are mounted on one side of the transport and collection mechanism, characterized in that: The movable end of the picking execution arm is equipped with a replaceable end effector (4); A lower computer display screen (6) and an upper computer display screen (7) are fixedly mounted on one side of the main frame (10). A depth camera (5) and a tomato real-time detection device for analyzing images from the depth camera (5) based on a YOLOV8 algorithm are mounted on the main frame (10). The tomato real-time detection device is electrically connected to a PLC controller, and the PLC controller is used to control the motion of a picking execution arm and a replaceable end effector (4).

2. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 1, characterized in that: The picking execution arms are provided in three groups and are arranged in parallel and equidistantly, namely a first robotic arm (1), a second robotic arm (2), and a third robotic arm (3). The picking execution arms include an X-axis module, a Y-axis module, and a Z-axis module. The replaceable end effector (4) is connected to the Z-axis module. The X-axis module is arranged horizontally and fixedly connected to the main frame (10). The Y-axis module is vertically connected to the X-axis module. The Z-axis module is arranged tilted relative to the Y-axis module.

3. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 2, characterized in that: The X-axis module comprises a transverse moving shaft (106) and a driving source, wherein the transverse moving shaft (106) is fixedly mounted on the main frame (10), a linear sliding assembly is provided on the transverse moving shaft (106), an X-axis guide rail is provided on the transverse moving shaft (106), an X-axis slide (308) is slidably connected to the X-axis guide rail, a side of the transverse moving shaft (106) is fixedly connected to a sealed slide (301), an outer sliding engagement of the sealed slide (301) is provided with a slide base plate (302), a side of the X-axis slide (308) is fixedly connected to a slide fixing plate (310), an inner side of the slide fixing plate (310) is fixedly connected to the slide base plate (302); the driving source is an X-axis motor, the X-axis motor drives the linear sliding assembly, and the linear sliding assembly drives the X-axis slide (308) to slide on the X-axis guide rail.

4. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 2, characterized in that: The Y-axis module comprises a longitudinal moving shaft (104), a linear sliding assembly and a Y-axis motor (107), wherein the longitudinal moving shaft (104) and the transverse moving shaft (106) are arranged at 90 degrees, and the upper portion of the X-axis slide (308) is fixedly connected to the Y-axis module base (303); A Y-axis slide base (304) is fixedly connected to the Y-axis module base (303), a Y-axis guide rail (305) is provided on the surface of the longitudinal movable shaft (104) along the axial direction, a reinforcing rib base (306) is provided on the side of the longitudinal movable shaft (104), the reinforcing rib base (306) is fixedly connected to the side of the longitudinal movable shaft (104) and the Y-axis slide base (304), respectively, the Y-axis module base (303) is fixedly connected to a reinforcing rib plate (307), an oblique connecting plate (109) is slidably connected to the Y-axis guide rail (305), a Y-axis motor (107) is installed at one end of the longitudinal movable shaft (104), and is used to drive a linear sliding assembly on the longitudinal movable shaft (104), and the linear sliding assembly drives the oblique connecting plate (109) to slide on the Y-axis guide rail (305).

5. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 2, characterized in that: The Z-axis module comprises an oblique moving shaft (103), the oblique moving shaft (103) being mounted on the outside of the oblique connecting plate (109), a Z-axis guide rail (405) being provided on the oblique moving shaft (103), a linear sliding assembly being mounted in the Z-axis guide rail (405), and a Z-axis base plate (402) being slidably connected to the Z-axis guide rail (405), the linear sliding assembly being connected to a side of the Z-axis base plate (402) close to the Z-axis guide rail (405); One end of the oblique moving shaft (103) is fixedly connected to a Z-axis motor bracket (406), and a Z-axis motor (108) is installed on the Z-axis motor bracket (406); a rotation drive device is installed on the oblique connecting plate (109), and the Z-axis base plate (402) is connected to the driving end of the rotation drive device. The Z-axis motor (108) drives the linear sliding component, and the linear sliding component drives the Z-axis base plate (402) to slide on the Z-axis guide rail (405), so that the oblique moving shaft (103) slides obliquely with the Z-axis base plate (402) as a base point, and the rotation drive device drives the oblique moving shaft (103) to rotate; A limit mechanism is installed on the oblique moving shaft (103), the limit mechanism comprising a positive limit switch (401) and a negative limit switch (404), the positive limit switch (401) and the negative limit switch (404) being installed on a Z-axis guide rail (405); Two limit metal sheets (403) are installed on one side of the Z-axis base plate (402) close to the positive limit switch (401) and the negative limit switch (404). The positive limit switch (401) and the negative limit switch (404) are both mechanical mechanisms. When the limit metal sheet (403) reaches a preset limit position, the mechanical contact of the limit switch is triggered, and the physical action is then converted into an electrical signal.

6. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 5, characterized in that: The replaceable end effector (4) comprises a gripping device (101) and a clamping claw fixing plate (205), wherein the clamping claw fixing plate (205) is an L-shaped angle plate, and the angle plate is used for fixedly connecting the gripping device (101) and the oblique moving shaft (103); The gripping device (101) comprises at least two flexible clamping jaws (201), a deformation frame, and a clamping jaw opening and closing motor (202). The flexible clamping jaws (201) are all made of rubber material. The output shaft of the clamping jaw opening and closing motor (202) is connected to the inside of the deformation frame. The four flexible clamping jaws (201) are mounted on the deformation frame. The opening and closing of the flexible clamping jaws (201) are controlled by the forward and reverse rotation of the motor. The end of the clamping jaw opening and closing motor (202) away from the deformation frame is fixedly connected to a rotating platform (207). One end of the clamp fixing plate (205) close to the rotating platform (207) is fixedly connected to the motor housing (206), and a rotating motor (204) is fixedly installed inside the motor housing (206). One end of the motor housing (206) close to the flexible clamp (201) is fixedly connected to the clamp base (203), and the clamp base (203) is rotatably connected to the rotating platform (207). The output shaft of the rotating motor (204) passes through the clamp base (203) and is fixedly connected to the center of the rotating platform (207).

7. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 6, characterized in that: The deformation frame comprises a U-shaped seat and a linkage plate, wherein a side of the U-shaped seat away from the U-shaped opening is fixedly connected to the clamping jaw opening and closing motor (202), the linkage plate is movably arranged inside the U-shaped seat, a support foot is arranged on the inner side of the flexible clamping jaw (201), and the support foot is rotatably connected to the linkage plate, the output shaft of the clamping jaw opening and closing motor (202) is a small screw, and the middle part of the linkage plate is threadedly connected to the small screw.

8. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 1, characterized in that: A mounting platform for a depth camera (5) is fixedly connected to the main frame (10), the mounting platform comprising a camera angle adjuster (501) and a camera bracket (502), the back side of the depth camera (5) is fixedly connected to the camera angle adjuster (501), and the camera angle adjuster (501) is fixedly mounted on the lower camera bracket (502).

9. The multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 1, characterized in that: The transport and collection mechanism comprises a conveyor belt (602) located directly below the origin of the picking execution arm and a storage box, the storage box being located below the conveyor belt (602) and being connected to one end of the conveyor belt (602), a conveyor speed control console (604) being installed on the side of the conveyor belt (602), an anti-roll baffle (603) being installed on the surface of the conveyor belt (602), and anti-drop baffles (601) being installed on both sides of the conveyor belt (602).

10. The fruit recognition method of the multi-arm picking robot based on the YOLOv8 recognition algorithm according to claim 1, characterized in that: The following steps are involved: S1. The depth camera (5) installed on the main frame (10) collects RGB-D images of the target area in real time, wherein the RGB image is used for fruit detection and the depth information is used to calculate the three-dimensional space coordinates; S2. Input the RGB image into a real-time tomato detection model optimized based on YOLOv8. This model adds an LSKA attention mechanism after the feature pyramid module SPPF in the backbone network to enhance feature extraction in dense fruit scenes and generate a predicted bounding box containing the ripe tomato fruit. S3. Based on the center point coordinates of the predicted bounding box and the depth information of the depth camera (5), the image coordinate system is mapped to the Cartesian coordinate system of the robotic arm through a coordinate conversion algorithm to generate the target picking coordinates of the grasping device (101); Based on the target picking coordinates, the optimal movement paths of the first robotic arm (1), the second robotic arm (2), and the third robotic arm (3) are determined through a path planning algorithm, the movement sequence of the horizontal moving axis (106) and the vertical moving axis (104) is prioritized, and finally the telescopic movement of the oblique moving axis (103) is controlled to avoid interference between the robotic arms and the vines; S4. transmitting the target coordinates and path instructions to the PLC controller, driving the X-axis motor, the Y-axis motor (107), and the Z-axis motor (108) to move in coordination, so that the replaceable end effector (4) reaches the target position; The flexible clamp (201) is controlled by the clamp opening and closing motor (202) to close and grasp the fruit, and then the rotating motor (204) drives the rotating platform (207) to complete the twisting and separation of the fruit; S5. During the continuous picking process, the speed of the conveyor belt (602) and the picking sequence of the robotic arm are dynamically adjusted to ensure that no less than 8 fruits are picked per minute.

Images