Intelligent picking device and method based on visual recognition and motion control

Abstract

This invention discloses an intelligent harvesting device and method based on visual recognition and motion control, belonging to the field of agricultural robot technology. It includes: a body, and integrated within the body a visual recognition module, a main control module, a mechanical execution module, a travel module, a power supply module, and an interaction module. The visual recognition module, with an embedded computer as its core, is connected to an image acquisition unit and equipped with a lightweight target detection algorithm. It is used to acquire crop images, identify mature fruits, and calculate the three-dimensional spatial coordinates of the fruits, transmitting the coordinate data to the main control module. The travel module adopts a dual-motor differential transmission structure to drive the overall movement of the device, adapting to inter-row operations. By replacing the traditional multi-degree-of-freedom robotic arm with a lightweight three-axis movement structure, combined with an adaptive shearing mechanism and a multi-algorithm fusion control scheme, it achieves low-cost, high-success-rate automated harvesting of solanaceous crops in facility agriculture.

Description

Technical Field

[0001] This invention relates to an intelligent harvesting device and method based on visual recognition and motion control, belonging to the field of agricultural robot technology. Specifically, it relates to machine vision recognition, multi-axis motion cooperative control, intelligent fault diagnosis, and automated harvesting technology for facility agriculture in agricultural robots. Background Technology

[0002] In current agricultural production, the harvesting of solanaceous crops is still mainly done manually, which presents three major pain points: 1) High labor costs: The planting area of ​​solanaceous crops in facility agriculture in my country exceeds 3 million mu, and the cost of manual harvesting accounts for 35%-50% of the total production cost, and the labor shortage problem is becoming increasingly prominent; 2) Low harvesting efficiency: The average efficiency of manual tomato harvesting is about 150 tomatoes / hour, and it is easily affected by fatigue and mood, resulting in large fluctuations in efficiency; 3) High fruit damage rate: The consistency of manual operation is poor, and improper control of the cutting force of the fruit stem can easily lead to fruit damage, with a damage rate of 8%-12%, reducing the marketable fruit rate.

[0003] Chinese patent application publication number CN119949150A discloses a convenient multi-head harvesting device based on deep learning vision. It adopts a patting harvesting structure and is only suitable for fruits with hard outer skin, such as camellia fruit. The damage rate for soft-skinned solanaceous crops such as tomatoes and strawberries exceeds 30%. It lacks a positioning accuracy control mechanism and the harvesting success rate is less than 80%.

[0004] Chinese patent application publication number CN117694106A discloses an intelligent harvesting robot and an intelligent fruit and vegetable harvesting method. It adopts a flexible robotic arm structure, but the overall cost is high, and the technical solution of the three-axis lightweight transmission structure is not disclosed, making it difficult for small and medium-sized growers to afford the procurement cost.

[0005] Chinese patent application publication number CN101273688A discloses a flexible picking device for a citrus picking robot. It uses a 5-DOF robotic arm as the execution structure. The picking process requires adsorption followed by shearing. The picking time for a single fruit is ≥8 seconds. The heavy weight of the end effector leads to high overall energy consumption and continuous operation time of less than 4 hours.

[0006] Current technologies generally suffer from three core defects: First, they mostly use multi-degree-of-freedom robotic arms as actuators, which are complex in structure, costly, and difficult to maintain, making them unsuitable for small- to medium-scale facility agriculture. Second, the shearing mechanisms mostly use fixed-force shearing, which easily damages the fruit skin, with a fruit damage rate generally exceeding 10%. Third, the accuracy matching between visual positioning and motion control is low, resulting in a harvesting success rate of less than 85%, and a precision control system integrating multiple algorithms has not been formed. Fourth, the robustness and scene adaptability of the algorithms are insufficient: general visual algorithms show a sharp drop in recognition rate in complex field scenes such as foliage occlusion, overlapping fruits, and backlighting or low light, lacking dynamic target tracking and coordinate compensation mechanisms, leading to a significant decrease in the actual harvesting success rate in the field; motion control mostly uses single-axis independent PID control, resulting in large multi-axis collaborative synchronization errors, lacking load adaptive capability and equipment health pre-diagnosis function, leading to a high failure rate in continuous field operations, and a fault-free operating time of less than 40 hours. Summary of the Invention

[0007] The purpose of this invention is to propose an intelligent harvesting device and method based on visual recognition and motion control. By replacing the traditional multi-degree-of-freedom robotic arm with a lightweight three-axis moving structure, and in conjunction with an adaptive shearing mechanism and a control scheme that integrates multiple algorithms, it achieves low-cost, high-success-rate automated harvesting of solanaceous crops in facility agriculture.

[0008] The present invention discloses an intelligent harvesting device based on visual recognition and motion control, comprising: The machine body, and the vision recognition module, main control module, mechanical execution module, movement module, power supply module and interaction module integrated in the machine body; The visual recognition module is based on an embedded computer, connected to an image acquisition unit, and equipped with a lightweight target detection algorithm. It is used to acquire crop images, identify mature fruits, calculate the three-dimensional spatial coordinates of the fruits, and transmit the coordinate data to the main control module. The visual recognition module is equipped with an improved YOLOv5-lite occlusion robustness detection algorithm, a multispectral fusion maturity grading algorithm, and a fruit stalk instance segmentation algorithm, which can realize fruit recognition in occlusion scenes, accurate positioning of fruit stalk cutting points, and maturity grading and screening, and is adapted to complex field lighting and plant growth environments. The main control module uses a microcontroller. After receiving the coordinate data output by the visual recognition module, it generates control signals to drive the mechanical execution module and the travel module to complete the corresponding actions. The main control module incorporates a three-axis cross-coupled synchronous control algorithm, an adaptive fuzzy PID control algorithm, a force-position hybrid adaptive control algorithm, and a multi-module time-series collaborative scheduling algorithm to achieve multi-axis collaborative high-precision motion, adaptive shear control, and stable time-series scheduling across all modules, ensuring the stability and accuracy of continuous operation of the device. The mechanical execution module includes a three-axis moving picking structure, a crank shears, and a collection frame. The three-axis moving picking structure is used to drive the crank shears to position the fruit stalk. The crank shears are used to cut the fruit stalk. The cut fruit falls into the collection frame through a pipe. The three-axis mobile picking structure achieves synchronous movement of the three axes through a cross-coupling control algorithm, with a synchronous movement error of ≤±0.03mm, avoiding trajectory deviation and mechanical vibration during the three-axis linkage process, and improving the picking positioning accuracy; The crank shears achieve precise control of shearing force and shearing position through a force-position hybrid adaptive control algorithm. The shearing force can be adjusted in real time according to the thickness and toughness of the fruit stalk to avoid damage to the fruit and the blade. The traveling module adopts a dual-motor differential transmission structure to drive the overall movement of the device, which is suitable for inter-row operation scenarios. The travel module achieves precise control of travel and steering through an adaptive fuzzy PID + load feedforward compensation control algorithm, adapting to different loads and ground slopes in inter-row operation scenarios, and improving travel positioning accuracy and steering flexibility. The power module provides power support with an appropriate voltage for each module, and the interaction module is a Bluetooth module used to realize command interaction and status feedback between the device and an external terminal.

[0009] Preferably, the visual recognition module includes a Raspberry Pi 4B single-board computer, a camera, a serial port module, and a display screen; the lightweight target detection algorithm on the Raspberry Pi 4B single-board computer is trained based on a self-made crop dataset, with a mature fruit recognition accuracy of ≥95% and a spatial coordinate calculation error of ≤±0.3mm. The coordinate data is transmitted to the main control module via the serial port module using a serial communication protocol.

[0010] Preferably, the core of the main control module is a development board, which generates pulse signals with an accuracy of 1μs through a timer to drive the power components of the mechanical execution module and the travel module respectively.

[0011] Preferably, the three-axis moving picking structure includes an X-axis drive unit, a Y-axis drive unit, a Z-axis drive unit, and a TB6600 drive module: The X-axis transmission unit includes an X-axis stepper motor, an X-axis slide rail, an X-axis aluminum alloy connector, an X-axis drive gear, an X-axis driven synchronous pulley, and an X-axis synchronous belt; The Y-axis transmission unit includes a Y-axis stepper motor, a Y-axis slide rail, a Y-axis speed-increasing gear set, a Y-axis synchronous belt, a Z-axis drive gear, and a Z-axis dual-output gear set; The Z-axis transmission unit includes a Z-axis stepper motor, a Z-axis trapezoidal lead screw, a Z-axis coupling, a Z-axis trapezoidal lead screw nut seat, a Z-axis slider, and a Z-axis guide shaft; The X and Y axes use synchronous belt drive, and the slider and synchronous belt are rigidly connected by a connector. The connector has a threaded hole on its surface to fix the crank shears. The Z axis uses lead screw drive. The left end of the X axis slide rail is equipped with an X axis limit switch, the rear end of the Y axis slide rail is equipped with a Y axis limit switch, and the end of the trapezoidal lead screw of the Z axis is equipped with a Z axis limit switch for three-axis position calibration.

[0012] Preferably, the crank shears include a servo motor, a crank, a connecting rod, a moving blade, a fixed blade, a slide groove, an M3 bolt, an adjusting screw, and a hose; the servo motor drives the crank and connecting rod to move the moving blade back and forth along the slide groove, and the shearing force can be adaptively adjusted in the range of 50-200N by a pulse width modulation signal, and the cut fruit slides down to the collection frame by its own weight through the hose.

[0013] Preferably, the travel module includes a DC motor, a primary chain assembly, an intermediate drive shaft, a secondary chain assembly, wheel axles, and a BTS7960 drive module; the primary chain assembly includes a primary drive sprocket, a primary driven sprocket, and a primary chain; the secondary chain assembly includes a secondary drive sprocket, a secondary driven sprocket, and a secondary chain; the DC motor drives the intermediate drive shaft through the primary chain assembly, and then the drive shaft is split into two paths that connect to the front and rear wheel axles through the secondary chain assembly. Steering is achieved by adjusting the speed difference between the two DC motors, with a minimum turning radius ≤300mm, suitable for ridge spacing operations of 1.2m-1.5m.

[0014] Preferably, the power supply module includes a 24V battery, an LM2596-5V step-down module, and an LM2596-3.3V step-down module; a multi-stage step-down circuit is used to power different components, and each power supply branch integrates an overcurrent protection unit.

[0015] The intelligent harvesting method based on visual recognition and motion control described in this invention, applied to the intelligent harvesting device based on visual recognition and motion control, includes the following steps: S1: Device initialization: The power module supplies power to each component, the main control module completes the peripheral clock configuration, and drives the stepper motors of each axis to move; the vision recognition module loads the target detection model and starts real-time frame acquisition. S2: Visual Recognition and Coordinate Calculation: The visual recognition module runs a target detection algorithm to identify the ripe fruit and stem areas, and calculates the 3D world coordinates of the fruit and stem through a perspective transformation algorithm. The coordinate data is then encapsulated and transmitted to the main control module. The transmission process uses a double verification mechanism of frame header and frame tail + checksum. After completing the fruit target detection, the module executes the stem instance segmentation and 3D pose calculation optimization sub-steps to achieve pixel-level positioning and high-precision coordinate calculation of the stem cutting point. S2-1 dynamic visual tracking and inter-frame compensation: Real-time tracking of the target fruit and the cutting point of the fruit stem, calculation of dynamic offset and completion of coordinate compensation. S3: Multi-target path planning: If the visual recognition module recognizes multiple mature fruits at the same time, the main control module calculates the Euclidean distance of all target fruits based on the current position of the actuator using a greedy algorithm, and generates the picking order from the closest to the furthest. The global optimal picking path can be generated through an improved ant colony global path planning algorithm to reduce the three-axis idle travel and improve the efficiency of multi-target picking. S4: Travel and Positioning: The main control module generates a differential control strategy based on the target coordinates, driving the travel module to move to the target position directly in front of the fruit. During the movement, the encoder provides real-time feedback of the rotation speed and dynamically adjusts the PWM duty cycle. The travel process adopts an adaptive fuzzy PID + load feedforward compensation control algorithm to adjust the motor speed in real time, improving the travel and positioning accuracy and load adaptability. S5: Multi-axis cooperative positioning: The main control module controls the X / Y / Z axes to move synchronously using an S-shaped acceleration and deceleration curve, accurately positioning the crank shears to the fruit stem; the three-axis cooperative motion adopts a cross-coupling synchronous control algorithm to compensate for the three-axis following error in real time and improve the accuracy of the three-axis synchronous motion; S6: Adaptive Shearing and Recovery: The main control module matches the shearing force corresponding to the current crop type, drives the crank shears to complete the fruit stem shearing, and the fruit slides into the collection box; the shearing process adopts a force-position hybrid adaptive control algorithm to realize dual closed-loop control of shearing force and shearing position, improve the shearing success rate and reduce the fruit damage rate; S7: Safety and Status Monitoring: During operation, real-time data on motor current, equipment temperature, and collection basket weight are collected. When an anomaly is triggered, tiered protection logic is executed, and harvesting data and fault logs are stored simultaneously. During operation, equipment health status prediction and fault pre-diagnosis sub-steps are performed to provide early warnings of equipment faults, optimize tiered fault self-recovery strategies, and improve the stability of continuous operation of the device.

[0016] Preferably, the post-processing of the target detection algorithm in step S2 uses a non-maximum suppression algorithm, and the cross-union ratio (CUI) threshold for overlapping bounding boxes is 0.5. The CUI calculation formula is as follows: ; in, To predict the bounding box region, For the true target bounding box region, only bounding boxes with IOU < 0.5 and maturity confidence higher than 0.7 are retained as valid recognition results; The coordinate calculation employs a perspective transformation method, using a pre-calibrated homography matrix to convert image coordinates to world coordinates. The conversion formula is as follows: ; Where s is the scale factor, ( , (u,v) represents the world plane coordinates, (u,v) represents the image pixel coordinates, and H represents the 3×3 homography matrix. The coordinate transformation error is ≤±0.3mm, obtained through pre-calibration using a calibration plate.

[0017] Preferably, the number of stepper motor drive pulses in step S4 is calculated using the following formula: ; Where N is the required number of pulses, L is the target movement distance, and P is the lead of the lead screw / synchronous belt. This refers to the microstepping factor of the TB6600 driver module. This represents the number of pulses required per step angle per revolution of the stepper motor. In step S3, differential steering uses a PID control algorithm to adjust the motor speed. The PID output formula is: ; Where u(t) is the PWM duty cycle output value of the DC motor, and e(t) is the deviation between the target position and the current position. , , These are the proportional, integral, and derivative coefficients, respectively, and the travel positioning error is ≤ ±50mm.

[0018] Compared with existing technologies, the intelligent harvesting device and method based on visual recognition and motion control of the present invention exhibits the following beneficial effects in terms of technical performance and practical application: 1. High harvesting efficiency: The device can harvest 500-600 solanaceous crops per hour, which is 3-4 times higher than manual harvesting; the continuous working time is ≥8 hours (powered by 24V battery), which can meet the daily harvesting needs. 2. Low fruit damage rate: High harvesting accuracy through YOLOv5-lite precise identification (accuracy ≥ 97%) and three-axis high-precision positioning (error ≤ ± 0.1 mm): Through the fusion of YOLOv5-lite target detection + perspective transformation coordinate calculation + trapezoidal acceleration and deceleration motion control algorithms, combined with adjustable shearing force (50-200N), the overall harvesting success rate is ≥ 92%, and the fruit damage rate is ≤ 4%, which is more than 50% lower than manual harvesting.

[0019] 3. Strong terrain adaptability: The differential travel structure has a minimum turning radius of ≤300mm, which can be adapted to greenhouse row spacing of 1.2m-1.5m without modifying the planting environment; the device weighs ≤30kg and can be manually transported to different plots.

[0020] 4. High ease of operation: Supports Bluetooth remote control (distance ≤10m), and mobile APP can monitor status and set parameters in real time; modular design reduces maintenance costs by 30%, and component replacement time is ≤10 minutes.

[0021] 5. By changing the training dataset and adjusting the shearing force parameters, it can adapt to the harvesting needs of various solanaceous and berry crops such as tomatoes, eggplants, and strawberries. The ridge spacing can be adapted from 1.2m to 1.5m, covering the mainstream specifications for facility agriculture planting.

[0022] 6. Strong adaptability to complex scenes: Through improved visual detection algorithms, the fruit recognition rate is ≥92% in occluded scenes, the accuracy fluctuation in backlight and low light environments is ≤2%, and the accuracy of locating the fruit stem cutting point is ≥98%. It can adapt to field operation scenarios with different lighting and different plant growth stages, and the actual harvesting success rate in the field is ≥95%. 7. Significantly improved control accuracy and operational stability: Through multi-axis cross-coupling control and adaptive PID algorithm, the synchronous motion error of the three axes is ≤ ±0.03mm, and the positioning error is ≤ ±20mm; the newly added equipment health pre-diagnosis function reduces the failure rate of continuous field operation by more than 60%, and the fault-free continuous operation time is ≥120 hours.

[0023] 8. Further optimization of harvesting efficiency: Through the improved ant colony global path planning algorithm, the three-axis empty travel is reduced by more than 30% in multi-target harvesting scenarios, and the harvesting efficiency is increased by 25% compared with the original solution, with 600-700 solanaceous crops per hour. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall structure of the intelligent harvesting device based on visual recognition and motion control according to the present invention. Figure 2 This is a schematic diagram of the movement module of the intelligent harvesting device based on visual recognition and motion control according to the present invention; Figure 3 This is a schematic diagram of the three-axis movement structure of the intelligent harvesting device based on visual recognition and motion control according to the present invention; Figure 4 This is a schematic diagram of the crank-handle scissor structure of the intelligent harvesting device based on visual recognition and motion control according to the present invention; Figure 5 This is a front view of the overall structure of the intelligent harvesting device based on visual recognition and motion control according to the present invention; Figure 6 This is a left view of the overall structure of the intelligent harvesting device based on visual recognition and motion control according to the present invention; Figure 7 This is a top view of the overall structure of the intelligent harvesting device based on visual recognition and motion control according to the present invention. Figure 8 This is a schematic diagram of the three-axis moving platform structure of the present invention; Figure 9 This is a flowchart of the intelligent harvesting method based on visual recognition and motion control according to the present invention; In the diagram: 1. Main body; 2. Vision recognition module; 21. Raspberry Pi 4B single-board computer; 22. Camera; 3. Main control module; 31. Development board; 4. Mechanical execution module; 41. Three-axis moving picking structure; 42. Crank shears; 43. Collection frame; 5. Traveling module; 51. DC motor; 52. First-stage chain assembly; 521. First-stage drive sprocket; 522. First-stage driven sprocket; 523. First-stage chain; 53. Intermediate drive shaft; 54. Second-stage chain assembly; 541. Second-stage drive sprocket; 542. Second-stage driven sprocket; 543. Second-stage chain; 55. Wheel and axle; 4111. X-axis stepper motor; 4112. X-axis slide rail; 4114. X-axis 4115. Driven gear; X-axis driven synchronous belt pulley; 4116. X-axis synchronous belt; 4121. Y-axis stepper motor; 4122. Y-axis slide rail; 4123. Y-axis slider connector; 4124. Y-axis speed-increasing gear set; 4126. Y-axis synchronous belt; 4127. Z-axis drive gear; 4128. Z-axis dual output gear set; 4132. Z-axis trapezoidal lead screw; 4133-1. X-axis limit switch; 4133-2. Y-axis limit switch; 4133-3. Z-axis limit switch; 4135. Z-axis trapezoidal lead screw nut seat; 4136. Z-axis slider; 421. Servo motor; 422. Crank; 423. Connecting rod; 424. Moving blade; 425. Fixed blade; 426. Slide groove. Detailed Implementation

[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0026] Example 1: like Figures 1-8 As shown, the intelligent harvesting device based on visual recognition and motion control according to the present invention includes: The device includes a main body 1, and integrated within the main body 1 are a vision recognition module 2, a main control module 3, a mechanical execution module 4, a movement module 5, a power supply module, and a Bluetooth interaction module. The vision recognition module 2 is responsible for recognizing mature fruits and calculating their coordinates. The core of the main control module 3 is a development board 31, which generates pulse signals with an accuracy of 1μs through a timer to drive the power components of the mechanical execution module and the movement module 5. The main control module 3 uses an STM32F103ZET6 microcontroller as its core to coordinate the collaborative work of each module. The mechanical execution module 4 completes fruit positioning and cutting. The movement module 5 enables flexible movement of the device. The power supply module supplies power to each component. The Bluetooth interaction module supports remote control and parameter adjustment, thus realizing fully automated operation of the "recognition-positioning-cutting-recycling" process for solanaceous crops.

[0027] In this example, the visual recognition module 2 includes a Raspberry Pi 4B single-board computer, a camera 22, and a CH340GTTL to serial port module. The camera 22 is installed below the crank shears 42 and acquires real-time images of tomato / eggplant crops in the field, capturing 30 frames per second and transmitting them to the Raspberry Pi 4B single-board computer. The Raspberry Pi 4B single-board computer is equipped with a YOLOv5-lite object detection algorithm trained on a self-made dataset containing 5000 tomato / eggplant images each, with a model weight file ≤ 5. The system has an MB (megabyte) memory, an inference speed ≥15fps, and achieves a mature fruit identification accuracy ≥95% after image preprocessing and feature extraction, with a false recognition rate ≤4% and a missed recognition rate ≤1%. It converts image coordinates to world coordinates using a perspective transformation algorithm, with a calculation error ≤±0.3mm after calibration. The coordinate data is sent to the main control module 3 via the CH340G module using the UART protocol. The frame format is "0xAA + X coordinate 2 bytes + Y coordinate 2 bytes + Z coordinate 2 bytes + 0x55", with a frame length of 8 bytes and a data transmission rate ≥100kbps. The CH340G module uses a 3.3V output and connects to the STM32's USART1 serial port. The TXD is connected to PA10, and the RXD is connected to PA9. The serial port parameters are a baud rate of 9600, 8 data bits, 1 stop bit, and no parity.

[0028] In this example, development board 31 uses an STM32F103ZET6 microcontroller. After receiving instructions from the vision recognition module 2, it generates pulse signals through the TIM4 timer with a timing accuracy of 1μs to control the stepper motor speed of mechanical execution module 4 (1000rpm), the servo motor rotation angle (0-180°), and the start / stop of the DC motor of travel module 5. At the same time, it connects to the Bluetooth interaction module through the USART2 serial port to receive remote control commands and provide feedback on the device's operating status.

[0029] In this example, the mechanical execution module 4 includes a three-axis moving picking structure 41, a crank shears 42, and a collection frame 43. The three-axis moving picking structure 41 is used to drive the crank shears 42 to position the fruit stalk. The crank shears 42 is used to cut the fruit stalk. The cut fruit falls into the collection frame 43 through the pipe. The three-axis mobile harvesting structure 41 includes an X-axis drive unit, a Y-axis drive unit, a Z-axis drive unit, and a TB6600 drive module. The X-axis transmission unit includes an X-axis stepper motor 4111, an X-axis slide rail 4112, an X-axis drive gear 4114, an X-axis driven synchronous pulley 4115, and an X-axis synchronous belt 4116; The Y-axis transmission unit includes a Y-axis stepper motor 4121, a Y-axis slide rail 4122, a Y-axis slider connector 4123, a Y-axis speed-increasing gear set 4124, a Y-axis synchronous belt 4126, a Z-axis drive gear 4127, and a Z-axis dual-output gear set 4128. The Z-axis transmission unit includes a Z-axis stepper motor, a Z-axis trapezoidal lead screw 4132, a Z-axis coupling, a Z-axis trapezoidal lead screw nut seat 4135, a Z-axis slider 4136, and a Z-axis guide shaft; The X and Y axes use synchronous belt drive, and the slider and synchronous belt are rigidly connected by a connector. The connector has a threaded hole on its surface to fix the crank scissor 42. The Z axis uses lead screw drive. The left end of the X axis slide rail 4112 is equipped with an X axis limit switch 4133-1, the rear end of the Y axis slide rail 4122 is equipped with a Y axis limit switch 4133-2, and the end of the Z axis trapezoidal lead screw 4132 is equipped with a Z axis limit switch 4133-3 for three-axis position calibration.

[0030] Three-axis moving structure assembly: 1) X-axis assembly: The X-axis slide rail 4112 (700mm in length) is fixed to the middle mounting plate with M4 bolts. An aluminum alloy connector (5mm thick) is installed on the slide rail slider. The X-axis stepper motor 4111 (42BYGH60) is fixed to one end of the slide rail with a motor bracket. The X-axis drive gear 4114 (module 1.0, number of teeth 20) is installed on the motor output shaft. The X-axis driven synchronous pulley 4115 (same model) is installed on the other end of the slide rail. The X-axis synchronous belt 4116 (GT2 type, 600mm in length) is wound around the two gears. The connector and the synchronous belt are fixed with buckles. An X-axis limit switch 4133-1, model D4V-8108Z, is installed on the left end of the X-axis slide rail 4112. The switch pin is connected to the PA4 pin of the STM32. This switch is used for X-axis position calibration and travel limitation during equipment initialization.

[0031] 2) Y-axis assembly: The Y-axis slide rail 4122 (300mm in length) is vertically fixed to the X-axis connector. The same model Y-axis stepper motor 4121 is used to drive the Y-axis synchronous belt 4126 through the 1.35:1 Y-axis speed-increasing gear set 4124 (module 2, number of teeth 25 / 34). The Y-axis slider connector 4123 fixes the synchronous belt. The Y-axis limit switch 4133-2, model D4V-8108Z, is installed at the rear end of the Y-axis slide rail 4122. The switch pin is connected to the PA5 pin of the STM32. This switch is used for Y-axis position calibration and travel limitation during device initialization.

[0032] 3) Z-axis assembly: The Z-axis trapezoidal lead screw 4132 (400mm in length) is vertically fixed to the Y-axis slider connector 4123 via the lead screw support seat. The Z-axis stepper motor (57BYG250H) is connected to the Z-axis drive gear 4127 (module 2, number of teeth 42) 1:1.5 Z-axis dual output gear set 4128 (module 2, number of teeth 42 / 64) via a coupling (flexible coupling, bore diameter 8mm / 12mm) to drive the synchronous belt and drive the lead screw. The Z-axis trapezoidal lead screw nut seat 4135 is fixed to the Z-axis slider 4136 (adapter to the lead screw), and the slider slides along the Z-axis guide shaft (diameter 8mm, length 400mm). A Z-axis limit switch 4133-3, model D4V-8108Z, is installed at the bottom of the lead screw. The switch pin is connected to the PA0 pin of the STM32. This switch is used for Z-axis position calibration and travel limitation during equipment initialization.

[0033] All three-axis stepper motors use the TB6600 driver module. The stepper motor is set to 8 microsteps and corresponding current of 1.0A and 2.0A via DIP switches. The ENA pin is connected to the PB3 pin of the STM3231 at a high level for enable.

[0034] The crank shears 42 include a servo motor 421, a crank 422, a connecting rod 423, a moving blade 424, a fixed blade 425, a slide groove 426, an M3 bolt, an adjusting screw, and a hose. The servo motor 421 drives the crank 422 and the connecting rod 423 to move the moving blade 424 back and forth along the slide groove 426. The shearing force can be adaptively adjusted in the range of 50-200N by a pulse width modulation signal. The cut fruit slides down the hose to the collection frame 43 by its own weight.

[0035] Crank scissor assembly: 1) Fix the fixed blade 425 (PLA, 50mm in length) to the bottom of the Z-axis slider 4136 with an M3 bolt, with the fixed blade edge facing upwards; 2) Fix the servo motor 421 (MG996R) to one side of the fixed blade with a servo motor bracket (aluminum alloy, 3mm thick), and install the crank 422 (PAL, 30mm in length) on the servo motor output shaft; 3) Hinge one end of the connecting rod 423 (50mm in length) to the crank 422 with an M3 bolt, and the other end to the moving blade 424 (same material as the fixed blade, 50mm in length); the moving blade 424 is installed in the slide groove 426 (PLA, 40mm in length) and can slide along the slide groove; 4) Install the adjusting screw (M2.5) on the side of the fixed blade 425 to adjust the blade edge gap between the moving blade and the fixed blade (0.1-0.3mm); install a flexible hose (200mm inner diameter, 1000mm in length) under the scissors, with the end of the hose aligned with the collection frame 43.

[0036] The crank shears 42 include a servo motor 421 of MG996R, a crank 422 with a length of 20mm, a connecting rod 423 with a length of 75.5mm, a moving blade 424 made of PLA material with a cutting edge angle of 30°, and a fixed blade 425 fixed to the end of the Z-axis slide rail. The servo motor 421 drives the crank 422 to perform circular motion, and the connecting rod 423 drives the moving blade 424 to perform linear reciprocating motion along the slide groove 426 with a length of 20mm. The moving blade 424 and the connecting rod 423 are hinged by an M3 bolt, and the gap between the bolt and the connecting rod hole is ≤0. 1mm; the blade gap between the fixed blade 425 and the moving blade 424 can be adjusted within the range of 0.1-0.3mm via the M2.5 adjusting screw; the shearing force is adjusted by the servo motor PWM duty cycle of 5%-10%, with an adjustment range of 50-200N, to adapt to the toughness of different solanaceous crops' fruit stalks; a flexible aluminum foil tube with an inner diameter of 200mm and a length of 1000mm is installed below the shears, and the end of the tube is connected to a 10L volume polypropylene collection frame 43. The fruit slides down the tube by its own weight into the collection frame, avoiding collision damage.

[0037] The travel module 5 includes a DC motor 51, a primary chain assembly 52, an intermediate drive shaft 53, a secondary chain assembly 54, wheel axles 55, and a BTS7960 drive module. The primary chain assembly 52 includes a primary drive sprocket 521, a primary driven sprocket 522, and a primary chain 523. The secondary chain assembly 54 includes a secondary drive sprocket 541, a secondary driven sprocket 542, and a secondary chain 543. The DC motor 51 drives the intermediate drive shaft 53 through the primary chain assembly 52, and then splits into two paths to connect the front and rear wheel axles 55 through the secondary chain assembly 54. Steering is achieved by adjusting the speed difference between the two DC motors 51, with a minimum turning radius of ≤300mm, suitable for ridge spacing operations of 1.2m-1.5m.

[0038] Assembly of the travel module: 1) Fix the DC motor 51 (Mingyang JGB36GP-55524V) to the bottom left side of the machine body through the motor bracket (45# steel, 5mm thickness), and install the first-stage drive sprocket 521 (08A-1, 12 teeth) on the motor output shaft. 2) The intermediate drive shaft 53 (diameter 12mm, 45# steel) is fixed to the center of the bottom of the machine body through the bearing seat (model UCP205). The two ends of the shaft are equipped with a first-stage driven sprocket 522 (same model, 24 teeth) and a second-stage drive sprocket 541 (same model, 12 teeth). 3) The front and rear axles 55 (15mm in diameter, 45# steel) are fixed to the front and rear ends of the bottom of the machine body through bearing seats. Wheels (rubber material, 100mm in diameter) and secondary driven sprockets 542 (same model, 24 teeth) are installed at both ends of the axle. 4) Install the first-stage chain 523 (08A-1, length 300mm) to connect the first-stage drive sprocket 521 and the first-stage driven sprocket 522, and install the second-stage chain 543 (length 200mm) to connect the second-stage drive sprocket 541 and the second-stage driven sprocket 542.

[0039] In this example, the travel module 5 includes a Mingyang JGB36GP-55524V DC motor with a rated speed of 150 rpm and a torque of 12 kg. The system includes a 08A-1 type primary chain assembly, a 12.7mm pitch, 12mm diameter 45# steel intermediate drive shaft, and two identical 08A-1 type secondary chain assemblies. A DC motor 51 is controlled by a BTS7960 driver module. The module's logic terminal VCC is connected to a 5V power supply, the power terminal VS is connected to a 24V power supply, and the IN_L / IN_R pins are connected to the STM32's PB0 / PB1 pins for PWM output at a frequency of 20kHz. The BTS7960 driver module integrates overcurrent protection threshold. 30A, over-temperature protection threshold of 150℃ and short-circuit protection. When an abnormality is detected, a low level is output through the IS pin to the PB2 pin of the STM32 to trigger the motor to stop urgently. The DC motor 51 drives the intermediate transmission shaft 53 through the first-stage chain group 52, and then splits into two paths to connect to the 15mm diameter wheel axle 55 through the second-stage chain group 54. By using the speed difference of 0-150rpm between the left and right motors, the minimum turning radius ≤300mm is achieved, which is suitable for greenhouse row spacing of 1.2m-1.5m.

[0040] In this example, the Bluetooth module is an HC05 Bluetooth 3.0 module, connected to the STM32's USART2 serial port with a baud rate of 9600, data bits, stop bits, and no parity. It supports AT command configuration, such as AT+NAME=SmartPicker to set the device name, and defaults to slave mode. It supports sending start / stop commands, picking mode switching commands, and parameter debugging commands via an Android 8.0 or later mobile app. Command response latency is ≤100ms, Bluetooth communication distance is ≤10m without obstructions, and the communication error rate is ≤10%. -5 The mobile app can display the device status, battery level, harvest quantity, and motor temperature in real time, and can also set harvesting parameters such as maturity threshold and shearing force.

[0041] The power supply module includes a 24V battery, an LM2596-5V step-down module, and an LM2596-3.3V step-down module; it uses a multi-stage step-down circuit to power different components, and each power supply branch integrates an overcurrent protection unit.

[0042] Working principle and usage process of this invention: 1. Equipment Initialization: After the device is started, the 24V polymer lithium battery outputs 5V and 3.3V voltages respectively through the LM2596 step-down module to power each module; the development board 31 completes the initialization of GPIO ports, timers, serial ports and driver modules, configures USART1 visual data reception, USART2 Bluetooth communication parameters, TIM2 DC motor PWM control, 20kHz, TIM3 servo PWM control, 50Hz, TIM4 stepper motor pulse generation, and enters the ready state with 1μs precision; the Raspberry Pi 4B single-board computer starts the Raspbian 11 system, loads the YOLOv5-lite target detection model, and the camera 22 begins to acquire crop images in real time.

[0043] 2. Target Recognition and Coordinate Calculation: The Raspberry Pi 4B single-board computer uses the YOLOv5-lite algorithm to preprocess the acquired images, performing grayscale conversion, noise reduction, and feature extraction via the CSPDarknet53 lightweight network. It then identifies ripe fruits and outputs the image coordinates. Using a perspective transformation algorithm based on pre-calibrated camera intrinsic parameters, it converts the image coordinates into world coordinates, i.e., the target position of the three-axis moving structure. The coordinate data frame is then sent to the STM32 via the CH340G serial port module, with the frame format being "0xAA + X coordinate + Y coordinate + Z coordinate + 0x55".

[0044] 3. Movement and Positioning: After receiving the coordinate data, the STM32 verifies the frame header 0xAA and the frame tail 0x55, parses the X / Y coordinates, and calculates the distance the device moves. It generates a PWM signal through the TIM2 timer to control the BTS7960 driver module to adjust the speed and direction of the DC motor 51. When turning is required, the device can be flexibly turned by adjusting the speed difference between the left and right motors, so that the device moves to the front of the fruit with an error of ≤±50mm. After it is in place, the STM32 controls the DC motor to stop.

[0045] 4. Precise Three-Axis Positioning: Based on the analyzed three-dimensional coordinates, the STM32 generates pulse signals through the TIM4 timer to drive the TB6600 driver module to control the X, Y, and Z axis stepper motors. The number of pulses is calculated according to "number of pulses = target distance / lead × subdivision", realizing coordinated control of X-axis lateral movement, Y-axis forward and backward movement, and Z-axis vertical movement, so that the crank shears 42 is precisely positioned at the fruit stem with a positioning error ≤ ±0.1mm. The X and Y axes use synchronous belt drive, and the Z axis uses lead screw drive. An X-axis limit switch 4133-1 is set at the left end of the X-axis slide rail 4112, a Y-axis limit switch 4133-2 is set at the rear end of the Y-axis slide rail 4122, and a Z-axis limit switch 4133-3 is set at the end of the Z-axis trapezoidal lead screw 4132 for three-axis position calibration. If the Z-axis movement triggers the Z-axis limit switch 4133-3, when the Z-axis moves to the limit position 10mm from the bottom, it will trigger an STM32 interrupt to complete the calibration or cut off the power supply to the Z-axis stepper motor and send an alarm message via Bluetooth module.

[0046] 5. Cutting and Recycling: The STM32 outputs a signal with the corresponding PWM duty cycle to the servo motor 421 of the MG996R via the TIM3 timer. The cutting force is set according to the crop type: 8% duty cycle for tomatoes, with a cutting force of 100-150N; 10% duty cycle for eggplants, with a cutting force of 150-200N. The servo motor 421 drives the crank 422 to make a circular motion, and the connecting rod 423 drives the moving blade 424 to reciprocate along the slide 426, which works with the fixed blade 425 to cut the fruit stem. The cut fruit slides down the aluminum foil tube by its own weight into the collection frame 43 to avoid collision damage.

[0047] 6. Cyclic Operation and Remote Interaction: After a single harvest is completed, the STM32 sends a "completion signal" to the Raspberry Pi 4B single-board computer to trigger the next round of image acquisition and target recognition, realizing continuous automated harvesting. During the operation, users can view the device's operating status in real time through a mobile APP, and send commands to adjust harvesting parameters such as maturity threshold, shearing force, or start and stop the device in an emergency according to actual needs.

[0048] It should be noted that in this example, the Raspberry Pi 4B single-board computer uses the Python pyserial 3.5 library to transmit and receive data, ensuring that there are no dropped frames in the visual data transmission; the STM32's GPIO ports are configured strictly according to the signal connection specifications, PB0 / PB1 control the DC motor, PB3-PB9 control the stepper motor, PA1 controls the servo motor, and PA0 detects the limit switch to ensure coordinated action of each actuator; the overcurrent and overtemperature protection functions of the BTS7960 driver module, the microstepping adjustment function of the TB6600 driver module, and the self-resetting fuse of the power module together improve the stability and reliability of the device in complex environments such as greenhouses and open fields.

[0049] To verify the practical application effect of the present invention, a comparative experiment was conducted at a greenhouse solanaceous crop planting base. The experimental conditions were as follows: the greenhouse row spacing was 1.2m, and 5 rows each of tomatoes (variety: Zhongza 105) and eggplants (variety: Heiliren) were planted, with each row being 50m long and the fruits being uniformly mature; the experimental equipment consisted of 1 device of the present invention, 2 skilled harvesters (harvesting experience ≥3 years), and 1 existing traditional solanaceous harvesting equipment (model: AGRI-600).

[0050] The experimental results are as follows: Table 1 Comparison of Harvesting Efficiency

[0051] Table 2 Comparison of Fruit Damage Rate

[0052] Table 3 Positioning accuracy and steering adaptability test

[0053] Table 4 Energy Consumption and Ease of Operation

[0054] Experimental conclusions: The device of this invention improves harvesting efficiency by 3-4 times compared to manual labor and by 40%-50% compared to existing traditional equipment; the fruit damage rate is reduced by more than 60% compared to manual labor and by more than 70% compared to existing traditional equipment; the positioning accuracy and steering adaptability fully meet the requirements of greenhouse operation with a row spacing of 1.2m, and it has lower energy consumption and is more convenient to operate, which can effectively reduce labor costs and improve agricultural production efficiency.

[0055] In this example, the improved YOLOv5-lite occlusion robustness detection algorithm is based on the ARM architecture of a Raspberry Pi 4B single-board computer for operator quantization and model pruning. After quantization, the model weight is ≤3.5MB and the inference speed is ≥20fps, meeting the real-time operation requirements of the embedded platform. The self-made crop dataset adds images of occlusion, overlap, backlight, and low light scenes, fruit stalk segmentation and annotation images, and multispectral maturity annotation images, covering the entire growth cycle and all scene conditions of tomatoes, eggplants, and strawberries. The Zhang's calibration method is used to complete the camera intrinsic parameter calibration, and the nine-point calibration method is used to complete the hand-eye collaborative extrinsic parameter calibration of the camera and the three-axis moving structure. The calibration process is repeated multiple times and the average value is taken to reduce the calibration error.

[0056] In this example, all electronic control algorithms are developed based on the STM32F103ZET6 microcontroller. The interrupt priority hierarchical scheduling mechanism of the multi-module timing collaborative scheduling algorithm is adopted, with the safety protection algorithm having the highest priority, followed by motion control, vision reception, and state prediction. The single algorithm operation cycle is ≤1ms, which meets the real-time control requirements. The parameters of each control algorithm are optimally calibrated through experiments to ensure stable field operations.

[0057] Example 2: like Figure 9 As shown, the intelligent harvesting method based on visual recognition and motion control described in this invention, applied to the intelligent harvesting device based on visual recognition and motion control as described in Example 1, includes the following steps: S1 device initialization: Power module 6 supplies power to each component, main control module 3 completes peripheral clock configuration, drives each axis stepper motor to move towards the limit switch, and records the initial position to complete origin calibration after triggering the limit switch; visual recognition module 2 loads target detection model, and camera 22 starts real-time frame acquisition. S2 Visual Recognition and Coordinate Calculation: The visual recognition module 2 runs the target detection algorithm to identify the mature fruit and fruit stem area, and calculates the three-dimensional world coordinates of the fruit and fruit stem through the perspective transformation algorithm. After encapsulating the coordinate data, it is transmitted to the main control module 3 through the serial port module 23. The transmission process adopts a double verification mechanism of frame header and frame tail + checksum. S3 Multi-Target Path Planning: If the visual recognition module 2 recognizes multiple ripe fruits at the same time, the main control module 3 calculates the Euclidean distance of all target fruits based on the current position of the actuator using a greedy algorithm, and generates a picking order from the closest to the furthest distance. S4 Movement and Positioning: The main control module 3 generates a differential control strategy based on the target coordinates, which drives the DC motor 51 of the movement module 5 to move to the target position directly in front of the fruit. During the movement, the encoder provides real-time feedback on the rotation speed and dynamically adjusts the PWM duty cycle output by the BTS7960 drive module. S5 Multi-Axis Collaborative Positioning: The main control module 3 generates drive pulses for the X-axis stepper motor 4111, Y-axis stepper motor 4121, and Z-axis stepper motor 4131 through the TB6600 drive module, controlling the X / Y / Z axes to move synchronously using an S-shaped acceleration and deceleration curve, and accurately positioning the crank shears 42 to the fruit stem. S6 Adaptive Cutting and Recycling: The main control module 3 matches the cutting force corresponding to the current crop type, drives the servo motor 421 to drive the crank shears 42 to complete the fruit stem cutting, and the fruit slides down the hose by its own weight to the collection frame 43; S7 Safety and Status Monitoring: During operation, the system collects motor current data from the INA219 current sensor, equipment temperature data from the DS18B20 temperature sensor, and collection basket weight data from the FSR402 pressure sensor in real time. When an abnormality is triggered, the system executes graded protection logic and simultaneously stores the harvesting data and fault logs to the W25Q64 Flash memory. Status data is synchronized to an external terminal in real time via Bluetooth module 71.

[0058] The hierarchical protection logic in step S7 is as follows: ① Current protection: The motor current is collected every 10ms by the INA219 current sensor. The stall current threshold of DC motor 51 is 5A, and the stall current threshold of X-axis stepper motor 4111 / Y-axis stepper motor 4121 / Z-axis stepper motor 4131 is 2A. When the current exceeds the threshold and lasts for ≥100ms, the power supply of the corresponding motor is cut off and a stall alarm is sent through Bluetooth module 71. ② Temperature protection: The device temperature is collected every 1 second by the DS18B20 temperature sensor. When the temperature exceeds 70℃, the corresponding motor speed is reduced to 50% of the rated value and the cooling fan is started. When the temperature continues to exceed 80℃, the corresponding axis movement is stopped and an overheat alarm is sent through the Bluetooth module 71. ③ Emergency stop protection: The physical emergency stop button triggers the highest priority interrupt. After triggering, it immediately cuts off the power to all motors, locks the servo motor 421 of the crank scissor 42, and sends an emergency stop alarm via Bluetooth module. ④ Full basket protection: The weight data of the collection basket 43 is collected every 500ms via the FSR402 pressure sensor. When the pressure exceeds 5kg, the harvesting process is stopped and a message is displayed that the collection basket is full. A new graded fault self-recovery strategy has been added: minor faults will automatically retry up to 3 times; moderate faults will execute the "stop → reverse movement 1mm → restart" operation; and severe faults will execute a shutdown alarm and wait for manual reset. The self-recovery success rate for minor faults is 100%, and the self-recovery success rate for moderate faults is ≥90%. All harvesting data and fault logs are stored in the W25Q64 Flash memory, which supports batch export via Bluetooth module (71). At the same time, the underlying interface is encapsulated through the hardware abstraction layer, which supports adaptation to different models of controllers, drivers and actuators. The crop type configuration can be switched with one click through the terminal.

[0059] Main control program comprehensive debugging (including basic functions, safety protection, status monitoring, algorithm optimization, fault diagnosis and compatibility expansion): 1) Create an STM32F103ZET6 project in the Keil5 environment, import the STM32 standard library V3.5.0, configure the system clock to 72MHz (external 8MHz crystal oscillator frequency multiplied to 72MHz by PLL), and write the full module initialization function: (A) Serial port initialization: USART1 (PA9 / PA10) is configured to 9600bps, 8 data bits, 1 stop bit, and no parity bit, and the interrupt priority is set to 2 (higher than the timer interrupt to receive visual data first), for receiving visual data; USART2 (PA2 / PA3) is configured to 9600bps for Bluetooth communication (HC-05 module, baud rate synchronous setting). (B) Timer initialization: TIM2 (PWM mode, 20kHz) is used for DC motor speed control, with ARR set to 3599 and PSC set to 0 to achieve precise duty cycle adjustment; TIM3 (PWM mode, 50Hz) is used for servo control, with ARR set to 1999 and PSC set to 719 to generate a PWM signal with a period of 20ms; TIM4 (timer mode, 1μs) is used for stepper motor pulse generation, with ARR set to 9 and PSC set to 7 to achieve a timing accuracy of 1μs. (C) GPIO and basic sensor initialization: PB0 / PB1 (push-pull output, initial level low) are used for BTS7960 control (PB0 controls the left motor direction, PB1 controls the right motor direction); PB3-PB9 (push-pull output, initial level low) are used for stepper motor control (PB3-PB5 are X-axis pulse / direction / enable, PB6-PB8 are Y-axis, PB9 is Z-axis enable); PA0, PA4, PA5 (pull-up input, falling edge triggers interrupt) are used for Z-axis, X-axis, and Y-axis limit switch detection, respectively. When the device is powered on and initialized, the STM32 drives each axis stepper motor to move towards the limit switch at 50% of the rated speed. After triggering the switch, it stops immediately and records the initial position to complete the calibration. During operation, triggering the switch will cut off the power supply of the corresponding axis and send an alarm code (0xEE + axis number + 0xFF) through USART2. (D) Safety protection hardware initialization: PB6 / PB7 are multiplexed as I2C1_SCL / SDA, connected to INA219 current sensor (to collect DC / stepper motor current); PA15 (single bus interface) is connected to DS18B20 temperature sensor (attached to the stepper motor driver and servo surface); PA6 (pull-up input, falling edge triggers the highest priority interrupt) is connected to the physical emergency stop button; PB10 (push-pull output, initial level is low) controls the cooling fan; PA7 (analog input) is connected to the FSR402 pressure sensor at the bottom of the fruit collection basket; (E) Storage and indicator hardware initialization: PB12-PB15 are multiplexed as SPI2_NSS / SCK / MISO / MOSI and connected to W25Q64 Flash memory (for data acquisition and fault log storage); PC0-PC3 (push-pull output, initial level is low) are connected to LED indicator group (PC0 green, PC1 yellow, PC2 red, PC3 blue); (F) Reserved function interface initialization: one set each of I2C2 (PB10 / PB11), SPI1 (PA5-PA7) and USART3 (PB10 / PB11) interfaces are reserved to reserve the hardware foundation for future expansion.

[0060] 2) Write motor control and core algorithm optimization functions: (A) DC motor control: Adjust the motor speed by changing the PWM duty cycle (0-100%) of TIM2. The correspondence between duty cycle and speed is calibrated by encoder (E6B2-CWZ6C, 1000 lines): 50% duty cycle corresponds to 75rpm speed, with an error ≤±2rpm; Write differential steering function (when turning left, the duty cycle of the left motor decreases by 30% while the right motor remains unchanged, and vice versa when turning right), with a steering radius error ≤±10mm; At the same time, integrate INA219 current data. If the current exceeds the threshold, the stall protection is triggered. (B) Stepper Motor Control: An S-shaped acceleration / deceleration function is used, dividing the motion process into "acceleration phase (increasing acceleration → decreasing acceleration), constant speed phase, and deceleration phase (increasing deceleration → decreasing deceleration)". Smooth speed transitions are achieved by adjusting the jerk value (the optimal value, tested, is 1000 mm / s³), preventing stepper motor step loss or overshoot. Laser displacement sensor testing shows that the positioning error fluctuation during acceleration and deceleration decreased from ±0.1 mm to ±0.03 mm. The pulse count calculation formula is "pulse count = (target distance / lead) × subdivision × 200". For example, if the Z-axis movement is 10 mm and the lead is 5 m... m, 8 subdivisions, pulse count = (10 / 5) × 8 × 200 = 3200; Write a three-axis linkage function, adopt a synchronous start and synchronous stop control strategy to achieve synchronous movement of X / Y / Z axes, linkage error ≤ ±0.05mm; Add a multi-target picking path planning function: For scenarios where the vision system detects multiple ripe fruits at the same time, based on the greedy algorithm, take the current position of the robotic arm as the starting point, calculate the Euclidean distance of all target fruits, prioritize the fruit with the closest distance as the next picking target, and replan the remaining target paths after completion. After testing, the multi-target picking efficiency is improved by more than 25% compared with random order; (C) Servo control: The servo angle is adjusted by the PWM duty cycle (5%-10%) of TIM3. The correspondence between duty cycle and angle is calibrated by an angle sensor: 5% corresponds to 0°, 10% corresponds to 180°, and the angle error is ≤±1°. A shearing force adjustment function is written, and the relationship between duty cycle and force is calibrated based on a pressure sensor (FSR402): 6% duty cycle corresponds to 50N, 10% corresponds to 200N, and the force error is ≤±10N. Adaptive adjustment logic for stem shearing position is added: A "stem region detection branch" is added to the YOLOv5-lite model output (the labeling dataset is supplemented with stem position labels, a total of 3000 images). The top 1 / 5 of the fruit bounding box is identified as the stem region. The center coordinates of the stem are calculated and the Z-axis shearing position is adjusted (5mm higher than the center of the fruit). After testing, the stem shearing success rate is improved from 85% to 98%.

[0061] 3) Harvesting process, safety protection and fault self-recovery integrated function: (A) Receive visual data: Receive coordinate frames through USART1 interrupt, first check the frame header (0xAA) and frame tail (0x55), then calculate the checksum (X+Y+Z lower 8 bits) and compare it with the received checksum. If the checksum passes, parse the X / Y / Z coordinates and send an ACK frame (0xBB+0xCC). If the check fails or a complete frame is not received within 1000ms, discard the current frame and request retransmission. (B) Travel control: The device calculates the travel distance based on the X / Y coordinates, calls the DC motor function to move to the target position, and the speed is fed back in real time through the encoder during the movement. If the actual speed deviates from the set speed by more than 10%, the duty cycle is adjusted. The movement error is tested to be ≤ ±50mm.

[0062] (C) Three-axis positioning: Call the stepper motor function to drive the X / Y / Z axes to move to the target coordinates according to the S-shaped acceleration and deceleration curve. If a non-target limit switch is triggered during the positioning process, the system will stop immediately and sound an alarm. The positioning error is ≤ ±0.1mm as tested by the laser displacement sensor (KEYENCELK-G5000).

[0063] (D) Cutting and Recycling: Call the servo function to set the cutting duty cycle according to the crop type (120N for 8% of tomato and 200N for 10% of eggplant), drive the shears to cut the fruit stem, and reset the servo after a 0.5s delay after cutting; after a 1-second delay (when the fruit falls into the collection basket), trigger the next round of the process; if the servo load exceeds the threshold during the cutting process (detected by changes in PWM duty cycle), the cutting will stop and an alarm will be triggered; (E) Safety protection logic: Read INA219 current data every 10ms, set DC motor stall current threshold to 5A and stepper motor to 2A. If the current exceeds the threshold and the duration is ≥100ms, immediately cut off the power supply of the corresponding motor and send a "stall alarm" code (0xEE+0x01+0xFF) via Bluetooth; Read DS18B20 temperature data every 1s, set temperature threshold to 70℃. If the temperature exceeds the threshold, reduce the speed of the corresponding motor to 50% of the rated speed and start the cooling fan. If the temperature continues to exceed 80℃, stop the movement of the corresponding axis and send an "overheat alarm" code (0xEE+0x02+0xFF); The emergency stop button triggers the highest priority interrupt. After triggering, immediately cut off the power supply of all motors, lock the servo motor, and send an "emergency stop trigger" code (0xEE+0x03+0xFF) via Bluetooth; Read the pressure sensor data of the collection basket every 500ms. If the pressure exceeds 5kg (corresponding to the upper limit of capacity), stop the harvesting process and prompt "collection basket is full".

[0064] (F) Status Monitoring and Data Recording: Define a "system status structure" containing fields such as visual communication status, motor operation status, sensor data, and harvesting progress. The structure data is updated every 50ms, and the status data is sent to the mobile monitoring APP in real time via USART2 (frame format is "0xDD + status data (20 bytes) + checksum (1 byte) + 0xEE"). The W25Q64 Flash is managed using the FATFS file system. After each harvest, "crop type, harvesting time, coordinate data, and shearing intensity" are written to the Flash (each record occupies 32 bytes, and ≥200,000 records can be stored). Historical data can be exported in batches via the Bluetooth APP. Write LED indicator control functions: PC0 green light indicates normal system operation, flashing indicates harvesting in progress; PC1 yellow light indicates sensor warning, flashing indicates communication abnormality; PC2 red light indicates fault shutdown, flashing indicates emergency stop triggering; PC3 blue light flashing indicates data storage in progress.

[0065] (G) Fault Diagnosis and Graded Self-Recovery: A "Fault Diagnosis Knowledge Base" is established, containing "fault phenomena, possible causes, and handling measures". The fault type is automatically identified by matching the knowledge base with status monitoring data. Graded recovery strategies are set: for minor faults (such as visual data verification failure), "automatic resend request (up to 3 times)" is executed; for moderate faults (such as stepper motor temporary stall), "stop → reverse movement 1mm → restart" is executed; for severe faults (such as continuous high temperature), "stop → send alarm → wait for manual reset" is executed. After testing, the self-recovery success rate for minor faults is 100%, and for moderate faults it is 90%. The fault type, occurrence time, and handling measures are written to the Flash fault log area, and historical faults can be queried through the APP.

[0066] (H) Compatibility and Scalability Configuration: Write Hardware Abstraction Layer (HAL) functions to encapsulate low-level operations such as motor control and sensor acquisition into a unified interface. Support for adaptation to different models of STM32 chips (such as F103C8T6, F407ZGT6), DC motor drivers (such as L298N), and servos (such as MG996R) through macro definitions; Set up a "Crop Type Configuration Table" which includes parameters such as shearing force and fruit stalk detection threshold for crops such as tomatoes, eggplants, and cucumbers. Support one-click switching of crop types via Bluetooth APP without reprogramming.

[0067] Added adaptive fruit stem localization logic: After the target detection model outputs the fruit bounding box, it identifies the top 1 / 5 of the bounding box as the fruit stem area, calculates the center coordinates of the fruit stem, and adjusts the cutting position to 5mm above the center of the fruit. The success rate of fruit stem cutting is ≥98%.

[0068] This algorithm is optimized based on the 32-bit ARM architecture platform of Raspberry Pi 4B. The model uses the lightweight version of YOLOv5-lite-s. The training configuration is as follows: the input image size is 640×640 pixels, the batch size is 16, the initial learning rate is 0.01, the cosine annealing learning rate decay strategy is adopted, the training epochs are 200, and the training dataset contains 30,000 images of three crops: tomatoes, eggplants and strawberries. It covers three lighting environments: sunny day, cloudy day and night with supplemental lighting; three shooting angles: front, side and occlusion; and three maturity states: fully ripe, semi-ripe and unripe. Data augmentation uses three methods: random flipping, random cropping and brightness perturbation.

[0069] The actual runtime inference speed is 15fps. After the target detection outputs the bounding boxes, non-maximum suppression is used for filtering. The IOU threshold is set to 0.5, and the confidence threshold is set to 0.7. The specific calculation process is as follows:

[0070] in, To predict the bounding box region, For the true target bounding box region, only bounding boxes with IOU < 0.5 and maturity confidence higher than 0.7 are retained as valid recognition results.

[0071] This algorithm pre-calibrates the OV5640 camera using a 10mm×10mm checkerboard calibration board, acquiring 15 images of the calibration board from different angles. The camera intrinsic parameter matrix and radial distortion coefficient k1 are then calculated using the calibration method. 0.235, k2=0.062, tangential distortion coefficient p1= 0.003, p2=0.001, complete camera distortion correction.

[0072] The coordinate transformation uses a pre-calibrated 3×3 homography matrix H, and the specific transformation formula is as follows:

[0073] Where s is the scale factor, with a value of 1.12, ( , Let (u,v) be the world plane coordinates, (u,v) be the image pixel coordinates, and H be a 3×3 homography matrix. The specific values ​​of the H matrix are:

[0074] The converted coordinate error is ≤ ±0.3mm, which fully meets the requirements for harvesting positioning accuracy.

[0075] The number of stepper motor drive pulses in step S4 is calculated using the following formula:

[0076] Where N is the required number of pulses, L is the target movement distance, and P is the lead of the lead screw / synchronous belt. This refers to the microstepping factor of the TB6600 driver module. This represents the number of pulses required per step angle per revolution of the stepper motor. The stepper motor motion is controlled using an S-shaped acceleration / deceleration curve, dividing the motion process into an acceleration phase (increasing acceleration → decreasing acceleration), a constant speed phase, and a deceleration phase (increasing deceleration → decreasing deceleration). The Jerk value for acceleration is set to 1000 mm / s². 3 This achieves a smooth speed transition; the X / Y / Z axes employ a three-axis linkage control strategy with synchronous start and synchronous stop.

[0077] The lightweight object detection algorithm used in the visual recognition module is an improved YOLOv5-lite occlusion robustness detection algorithm. The residual module of the CSPDarknet53 backbone network embeds a CBAM hybrid attention mechanism at its end, and the neck network adopts a lightweight Bi-FPN bidirectional feature fusion structure. The loss function uses an improved CIoU loss, as shown in the formula:

[0078] in, The standard CIoU bounding box regression loss is used, λ is the occlusion penalty coefficient (ranging from 0.8 to 1.2), and Locc is the feature loss of the occluded fruit region. For fruits with a branch and leaf occlusion rate of ≥30%, the edge contour and HSV color features of the fruit are extracted through feature enhancement branch to achieve effective recognition of occluded fruits. The post-processing of the algorithm adopts adaptive threshold non-maximum suppression. For overlapping fruits, the cross-union ratio threshold is dynamically adjusted to 0.3-0.6. The average recognition accuracy of mature fruits is ≥98.5%, the recognition accuracy in occluded scenarios is ≥92%, the model weight file is ≤3.5MB, and the inference speed is ≥20fps.

[0079] In step S2, after completing the fruit target detection, a new sub-step for fruit stem instance segmentation and 3D pose calculation optimization is added: Based on the lightweight SOLOv2 instance segmentation network, the top 1 / 5 region of the detected mature fruit is segmented at the pixel level, the center line of the fruit stem and the coordinates of the optimal cutting point are extracted, and the spatial orientation and cutting angle of the fruit stem are output; a hand-eye collaborative calibration optimization algorithm is adopted to establish the transformation relationship between the camera coordinate system and the three-axis moving picking structure coordinate system, and the 3D coordinate calculation is optimized by combining the principle of binocular vision parallax, so that the fruit stem cutting point positioning is more accurate; the coordinate calculation error is ≤ ±0.15mm, and the fruit stem cutting point positioning accuracy is ≥98%.

[0080] Between steps S2 and S3, a new sub-step of dynamic visual tracking and inter-frame compensation is added: an improved KCF kernel correlation filter lightweight tracking algorithm is used to track the identified target fruit and the fruit stem cutting point in real time. The pixel offset caused by the movement of the device and the swaying of branches and leaves is calculated by the inter-frame difference method, and the offset compensation value is output to the main control module 3. The main control module 3 corrects the motion coordinates of the three-axis moving picking structure 41 in real time, with a compensation frequency of ≥20fps and a positioning error fluctuation of ≤±0.05mm in dynamic scenes.

[0081] The image acquisition unit is an RGB+near-infrared dual-spectrum camera. The visual recognition module 2 is equipped with a multi-spectral fusion maturity grading recognition algorithm. It extracts the internal sugar and water content characteristics of the fruit through near-infrared spectroscopy, combines the appearance color and fruit shape characteristics of the RGB image, and uses a random forest classifier to grade the fruit maturity into 4 levels. Users can set the harvest maturity threshold through the interactive module.

[0082] The main control module 3 has a built-in three-axis cross-coupling synchronous control algorithm for coordinated motion control of the X / Y / Z axes. The algorithm collects the following error between the actual position and the target position of the three axes in real time, establishes a cross-coupling error compensation model, and compensates for the motion deviation of the three axes in real time. The three-axis synchronous motion error is ≤ ±0.03mm, and the single-axis positioning error is ≤ ±0.05mm, avoiding trajectory deviation and mechanical vibration during the three-axis linkage process.

[0083] The dual-motor differential transmission structure of the travel module 5 adopts an adaptive fuzzy PID + load feedforward compensation control algorithm. The algorithm takes the deviation between the target travel distance and the actual encoder feedback value and the rate of change of the deviation as input, and adjusts the proportional, integral and derivative coefficients of the PID online through fuzzy inference rules. At the same time, it introduces load feedforward compensation terms for the weight of the collection frame and the ground slope to improve the accuracy of travel and steering control. The travel positioning error is ≤ ±20mm, the steering radius control error is ≤ ±10mm, and the speed fluctuation under tire slippage conditions is ≤5%.

[0084] The crank shears 42 are equipped with a force-position hybrid adaptive control algorithm. The main control module 3 collects the torque changes during the shearing process in real time through servo motor current feedback, and establishes a dual closed-loop control of shearing force and position: Position loop: controls the travel of the moving blade 424, with a positioning accuracy of ≤±0.1mm; Force loop: adaptively adjusts the output torque of the servo motor 421 based on a fuzzy PID algorithm, with a real-time adjustment range of shearing force of 20-250N. When a shearing force mutation rate of ≥50% is detected, the moving blade is immediately triggered to reverse and retract, avoiding fruit damage and blade breakage; the success rate of fruit stem shearing is ≥99%, and the fruit damage rate during the shearing process is ≤1.5%.

[0085] In step S3, an improved ant colony global path planning algorithm is added to the multi-objective path planning. Taking the current position of the three-axis moving picking structure 41 as the starting point and the cutting points of all mature fruits as the target nodes, the path planning objective function is established by combining the acceleration constraints and travel limit constraints of the three-axis motion. The ant colony search process is optimized by the pheromone adaptive update rule, and the globally optimal picking path is output. Compared with the original greedy algorithm, the three-axis idle travel is reduced by more than 30%, and the multi-objective picking efficiency is improved by more than 25%.

[0086] In step S7, a new sub-step for equipment health status prediction and fault pre-diagnosis is added. The main control module 3 has a built-in lightweight LSTM time series prediction algorithm to collect real-time time series data of motor current, temperature, vibration, and battery voltage. It performs trend prediction on faults such as motor bearing wear, battery degradation, shear blade wear, and line aging, and outputs early warning information 24 hours in advance. The fault prediction accuracy is ≥90%. At the same time, the graded fault self-recovery strategy is optimized, with a 100% success rate for minor faults and a ≥95% success rate for moderate faults.

[0087] The main control module 3 has a built-in multi-module timing coordination scheduling algorithm and adopts an interrupt priority hierarchical scheduling mechanism. The interrupt priority of the safety protection algorithm is set to level 0 (highest), the priority of the motion control algorithm is set to level 1, the priority of the visual data receiving algorithm is set to level 2, and the priority of the status monitoring and fault prediction algorithm is set to level 3. The single algorithm operation cycle is ≤1ms, and the instruction response delay between modules is ≤5ms, so as to avoid instruction conflicts and timing disorders when multiple modules run in parallel.

[0088] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0089] In this embodiment, the dynamic visual tracking and inter-frame compensation sub-steps calculate the target offset through inter-frame difference, convert it into world coordinate compensation values, and send them to the main control module in real time to achieve accurate positioning of dynamic scenes.

[0090] In this embodiment, the equipment health status prediction and fault pre-diagnosis sub-step predicts the equipment operation trend through time series data, identifies potential faults in advance and performs graded self-recovery, automatically retrying for minor faults, automatically resetting for moderate faults, and shutting down and alarming for severe faults, thereby improving the reliability of continuous operation.

Claims

1. An intelligent harvesting device based on visual recognition and motion control, characterized in that, include: The body (1), and the visual recognition module (2), main control module (3), mechanical execution module (4), movement module (5), power supply module and interaction module integrated in the body (1); The visual recognition module (2) is based on an embedded computer, connected to an image acquisition unit, and equipped with a lightweight target detection algorithm. It is used to acquire crop images, identify mature fruits, calculate the three-dimensional spatial coordinates of the fruits, and transmit the coordinate data to the main control module (3). The main control module (3) adopts a microcontroller. After receiving the coordinate data output by the visual recognition module (2), it generates control signals to drive the mechanical execution module and the travel module (5) to complete the corresponding actions. The mechanical execution module (4) includes a three-axis moving picking structure (41), a crank shears (42), and a collection frame (43). The three-axis moving picking structure (41) is used to drive the crank shears (42) to position the fruit stalk. The crank shears (42) is used to cut the fruit stalk. The cut fruit falls into the collection frame (43) through the pipeline. The traveling module (5) adopts a dual-motor differential transmission structure to drive the overall movement of the device and adapt to the inter-row operation scenario. The power module provides power support with an appropriate voltage for each module, and the interaction module is a Bluetooth module used to realize command interaction and status feedback between the device and an external terminal.

2. The intelligent harvesting device based on visual recognition and motion control according to claim 1, characterized in that: The visual recognition module (2) includes a Raspberry Pi 4B single-board computer, a camera (22), a serial port module and a display screen; the lightweight target detection algorithm on the Raspberry Pi 4B single-board computer is trained based on a self-made crop dataset, with a mature fruit recognition accuracy of ≥95% and a spatial coordinate calculation error of ≤±0.3mm. The coordinate data is transmitted to the main control module (3) through the serial port module using a serial communication protocol.

3. The intelligent harvesting device based on visual recognition and motion control according to claim 1, characterized in that: The core of the main control module (3) is the development board (31), which generates a pulse signal with an accuracy of 1μs through a timer to drive the power components of the mechanical execution module and the travel module (5) respectively.

4. The intelligent harvesting device based on visual recognition and motion control according to claim 1, characterized in that: The three-axis mobile picking structure (41) includes an X-axis transmission unit, a Y-axis transmission unit, a Z-axis transmission unit, and a TB6600 drive module: The X-axis transmission unit includes an X-axis stepper motor (4111), an X-axis slide rail (4112), an X-axis drive gear (4114), an X-axis driven synchronous pulley (4115), and an X-axis synchronous belt (4116). The Y-axis transmission unit includes a Y-axis stepper motor (4121), a Y-axis slide rail (4122), a Y-axis slider connector (4123), a Y-axis speed-increasing gear set (4124), a Y-axis synchronous belt (4126), a Z-axis drive gear (4127), and a Z-axis dual-output gear set (4128). The Z-axis transmission unit includes a Z-axis stepper motor, a Z-axis trapezoidal lead screw (4132), a Z-axis coupling, a Z-axis trapezoidal lead screw nut seat (4135), a Z-axis slider (4136), and a Z-axis guide shaft; The X and Y axes are driven by synchronous belts. The slider and the synchronous belt are rigidly connected by a connector. The connector has a threaded hole on its surface for fixing the crank shears (42). The Z axis is driven by a lead screw. The left end of the X axis slide rail (4112) is equipped with an X axis limit switch (4133-1), the rear end of the Y axis slide rail (4122) is equipped with a Y axis limit switch (4133-2), and the end of the Z axis trapezoidal lead screw (4132) is equipped with a Z axis limit switch (4133-3) for three-axis position calibration.

5. The intelligent harvesting device based on visual recognition and motion control according to claim 1, characterized in that: The crank shears (42) include a servo motor (421), a crank (422), a connecting rod (423), a moving blade (424), a fixed blade (425), a slide (426), an M3 bolt, an adjusting screw, and a hose. The servo motor (421) drives the crank (422) and the connecting rod (423) to move the moving blade (424) back and forth along the slide (426). The shearing force can be adaptively adjusted in the range of 50-200N by a pulse width modulation signal. The cut fruit slides down the hose to the collection frame (43) by its own weight.

6. The intelligent harvesting device based on visual recognition and motion control according to claim 1, characterized in that: The traveling module (5) includes a DC motor (51), a primary chain assembly (52), an intermediate drive shaft (53), a secondary chain assembly (54), a wheel axle (55), and a BTS7960 drive module. The primary chain assembly (52) includes a primary drive sprocket (521), a primary driven sprocket (522), and a primary chain (523). The secondary chain assembly (54) includes a secondary drive sprocket (541), a secondary driven sprocket (542), and a secondary chain (543). The DC motor (51) drives the intermediate drive shaft (53) through the primary chain assembly (52), and then connects the front and rear wheel axles (55) through the secondary chain assembly (54). The steering is achieved by adjusting the speed difference between the two DC motors (51), with a minimum steering radius of ≤300mm, suitable for ridge spacing operation scenarios of 1.2m-1.5m.

7. The intelligent harvesting device based on visual recognition and motion control according to claim 1, characterized in that, The power module includes a 24V battery, an LM2596-5V step-down module, and an LM2596-3.3V step-down module; it uses a multi-stage step-down circuit to power different components, and each power supply branch integrates an overcurrent protection unit.

8. A smart harvesting method based on visual recognition and motion control, applied to the smart harvesting device based on visual recognition and motion control as described in any one of claims 1-7, characterized in that, Includes the following steps: S1: Device initialization: The power module (6) supplies power to each component, the main control module (3) completes the peripheral clock configuration and drives the stepper motors of each axis to move; the vision recognition module (2) loads the target detection model and starts real-time frame acquisition; S2: Visual recognition and coordinate calculation: The visual recognition module (2) runs the target detection algorithm to identify the mature fruit and fruit stem area, calculates the three-dimensional world coordinates of the fruit and fruit stem through the perspective transformation algorithm, encapsulates the coordinate data and transmits it to the main control module (3). The transmission process adopts a double verification mechanism of frame header and frame tail + checksum. S3: Multi-target path planning: If the visual recognition module (2) recognizes multiple mature fruits at the same time, the main control module (3) calculates the Euclidean distance of all target fruits based on the greedy algorithm, starting from the current position of the execution mechanism, and generates the picking order from near to far. S4: Travel and positioning: The main control module (3) generates a differential control strategy based on the target coordinates, drives the travel module (5) to move to the target position directly in front of the fruit, and feeds back the rotation speed in real time through the encoder and dynamically adjusts the PWM duty cycle during the movement. S5: Multi-axis collaborative positioning: The main control module (3) controls the X / Y / Z axes to move synchronously using an S-shaped acceleration and deceleration curve, and precisely positions the crank shears (42) to the fruit stem; S6: Adaptive shearing and recycling: The main control module (3) matches the shearing force corresponding to the current crop type, drives the crank shears (42) to complete the fruit stem shearing, and the fruit slides into the collection box (43). S7: Safety and Status Monitoring: During operation, the system collects data on motor current, equipment temperature, and collection basket weight in real time. When an abnormality is triggered, the system executes graded protection logic and simultaneously stores harvesting data and fault logs.

9. The intelligent harvesting method based on visual recognition and motion control according to claim 8, characterized in that, In step S2, the post-processing of the target detection algorithm uses a non-maximum suppression algorithm, and the cross-union ratio (CUI) threshold for overlapping bounding boxes is 0.

5. The CUI calculation formula is as follows: ； in, To predict the bounding box region, For the true target bounding box region, only bounding boxes with IOU < 0.5 and maturity confidence higher than 0.7 are retained as valid recognition results; The coordinate calculation employs a perspective transformation method, using a pre-calibrated homography matrix to convert image coordinates to world coordinates. The conversion formula is as follows: ； Where s is the scale factor, ( , (u,v) represents the world plane coordinates, (u,v) represents the image pixel coordinates, and H is a 3×3 homography matrix, which is obtained through pre-calibration using a calibration plate.

10. The intelligent harvesting method based on visual recognition and motion control according to claim 8, characterized in that, The number of stepper motor drive pulses in step S4 is calculated using the following formula: ； Where N is the required number of pulses, L is the target movement distance, and P is the lead of the lead screw / synchronous belt. This refers to the microstepping factor of the TB6600 driver module. The number of pulses corresponding to the step angle required per revolution of the stepper motor is given. The stepper motor motion is controlled by an S-shaped acceleration and deceleration curve.

Images