Rigid-flexible coupling sequence driving picking device and intelligent control method thereof
By using a double-layer X-shaped telescopic mechanism and an integrated rigid-flexible coupling end effector, combined with intelligent control methods, the problems of complex structure and difficult force control of the harvesting robot have been solved, achieving efficient and low-cost fruit harvesting.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHWEST A & F UNIV
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-12
AI Technical Summary
Existing harvesting robots suffer from problems such as complex structure due to multiple drive sources, large end-effector inertia, high cost, poor reliability, and difficulty in force control under complex transmission chains. Furthermore, traditional control methods have failed to effectively address the high-dimensional nonlinearity and time-varying characteristics of rigid-flexible coupled systems.
Employing a double-layer X-shaped telescopic mechanism and an integrated rigid-flexible coupling end effector, the system controls multiple sequences of actions through a single motor. By combining a forward state estimation network, a stem breakage detection network, and an inverse control decision network, it achieves sensorless precision force control and online adaptive learning.
Significantly reduces the number of drive sources, lowers costs, improves harvesting efficiency, simplifies the structure, enhances reliability and accuracy, and achieves lightweight and efficient harvesting.
Smart Images
Figure CN122004046A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of agricultural robots and intelligent control technology, and in particular to a rigid-flexible coupling sequence-driven harvesting device and its intelligent control method. Background Technology
[0002] Fruit harvesting is a labor-intensive agricultural activity, and as the world's largest fruit producer, my country faces prominent challenges such as a high proportion of labor required for harvesting, rising labor costs, and seasonal labor shortages. Harvesting robots represent a key technological solution to this dilemma, but existing technologies suffer from two major bottlenecks that severely restrict their commercial application: Bottlenecks in drive mechanisms: Existing harvesting devices mostly adopt a multi-drive source solution, with independent motors configured for clamping, separating, and approaching actions. This results in large end mass, high inertia, low harvesting efficiency, complex structure, poor reliability, difficulty in control and coordination, and high cost. Some pneumatic or underactuated solutions suffer from problems such as dependence on complex air sources, slow response, low precision, or difficulty in controlling spring parameters, unstable timing, and poor adaptability. They have not achieved integrated mechanical logic drive with a lower drive source and multiple action sequences.
[0003] Bottlenecks in control methods: The harvesting device has a long transmission chain with rigid-flexible coupling. The mapping relationship between motor signals and the states of terminal clamping force and breaking torque exhibits high-dimensional nonlinearity and time-varying characteristics. Traditional physical modeling results in serious error accumulation, while end-sensor solutions suffer from high cost, poor reliability, severe drift, and response delay. Traditional machine learning methods have limited expressive power, and deep learning control technology for this complex system, especially multimodal fusion, transient fracture detection, and online adaptive learning solutions, has not yet achieved effective breakthroughs. Summary of the Invention
[0004] The purpose of this invention is to provide a rigid-flexible coupling sequence-driven harvesting device and its intelligent control method, which solves the technical problems of existing harvesting robots, such as complex structure, large end-effector inertia, high cost, poor reliability, and difficulty in force control under complex transmission chain conditions caused by multiple drive sources.
[0005] To achieve the above objectives, the present invention provides a rigid-flexible coupling sequence-driven harvesting device, comprising a drive mechanism, a drive rope, and an integrated rigid-flexible coupling end effector; the drive mechanism is fixedly mounted on a movable base; the drive rope connects the drive mechanism and the integrated rigid-flexible coupling end effector; the integrated rigid-flexible coupling end effector is a one-piece molded continuous flexible structure, mounted above the drive mechanism.
[0006] Preferably, the drive mechanism adopts a double-layer X-shaped telescopic mechanism, including an outer X-shaped mechanism, an inner X-shaped mechanism, and a drive motor; the drive motor is installed in a reserved fixed position on the movable base; the lower end of the outer X-shaped mechanism is fixed on the movable base, and the upper end is the moving end, which extends and retracts in the vertical direction; the inner X-shaped mechanism is nested inside the outer X-shaped mechanism, and its size is slightly smaller than the outer layer; its lower end is fixed on the movable base and a lower limit position of the limit trigger mechanism is set nearby; its upper end is connected to an integrated rigid-flexible coupling end actuator and an upper limit position of the limit trigger mechanism is set nearby; a damping mechanism is set between the hinge nodes of the inner X-shaped mechanism.
[0007] Preferably, the outer X-shaped mechanism and the inner X-shaped mechanism are rhomboid retractable structures.
[0008] Preferably, the integrated rigid-flexible coupling end effector includes a wrist elastic sheet, a gripper elastic sheet, and a flexible finger; the lower end of the wrist elastic sheet is connected to the upper platform of the inner X-shaped mechanism, and the upper end is connected to the flexible gripper; the outer periphery of the gripper elastic sheet is connected to the flexible finger.
[0009] Preferably, there are two drive ropes; one end of one rope is connected to the elastic sheet of the hand claw, and the other end is connected to the outer X-shaped mechanism; the other rope is connected to the elastic sheet of the wrist and the outer X-shaped mechanism at both ends respectively.
[0010] This invention also provides an intelligent control method for a rigid-flexible coupling sequence-driven harvesting device, comprising the following steps: S1. Design and train the forward state estimation network, the fruit stalk breakage detection network, and the inverse control decision network; S2. Data Acquisition: Real-time acquisition of drive motor signal timing information and RGB-D visual images; drive motor signal timing information includes drive motor current, voltage, and rotation angle signals within a certain time window. S3. Fruit Recognition: Based on RGB-D visual images, the fruit location, type, and maturity are identified through a deep convolutional neural network. The fruit firmness and weight are estimated, and the desired targets are determined. The desired targets include the desired gripper closure degree, the desired clamping force, the desired breaking torque, and the desired operation stage. S4. Forward State Estimation: The current terminal state is estimated by using the forward state estimation network based on the timing of the drive motor signal collected in S2 and the fruit information obtained in S3. This includes the degree of gripper closure, actual clamping force, wrist flexion angle, breaking torque, probability of fruit stem breakage, and operation stage identifier. S5. Reverse Control Decision and Execution: Based on the work stage identifier, combined with the forward state estimation results and the desired target, instructions are generated through the reverse control decision network, which drives the motor through the execution control module. This process is divided into five stages: Ascending and positioning stage: Based on the Z-axis height target of fruit recognition, the reverse control decision network generates a forward rotation command for the drive motor, the outer X-shaped mechanism rises, the drive rope connected to the wrist elastic sheet is tightened, the wrist returns to center, thereby driving the hand claw elastic sheet to recover, the hand claw opens, and the integrated rigid-flexible coupling end effector reaches below the fruit tree. Clamping stage: Force closed-loop control is adopted to monitor and estimate the clamping force in real time. Based on the error between the actual clamping force and the expected clamping force, the drive motor current is adjusted until the clamping force reaches the target value. Breaking stage: The motor reverses, the outer X-shaped mechanism descends, the drive rope connecting the elastic sheet of the claw tightens, the claw tightens, thereby causing the elastic sheet of the wrist to recover, the wrist deflects, and the execution control module adopts torque progressive control, so that the breaking torque gradually increases from zero, and the rate of increase is adjusted according to the characteristics of the fruit; Fracture response and fruit harvesting stage: After the fruit stem fracture detection network detects the fracture, the execution control module immediately issues a stop command, drives the motor to brake urgently, and then switches to the rising fruit harvesting stage; Reset phase: Constant speed control is adopted, the drive motor rotates at a constant speed, the outer X-shaped mechanism returns to the upper limit position, the inner X-shaped mechanism resets synchronously, the wrist returns to the center, the claws open fully, the fruit falls, and it waits for the next picking cycle. S6. Online learning and adaptation: After each harvesting cycle, the complete data sequence and result evaluation are automatically recorded; new data are added to the training set according to the set period, and the network parameters are fine-tuned with a small learning rate to update each network module; after fine-tuning, the model performance is evaluated, and if the accuracy is improved, the network model in the main controller is updated, so that time-varying factors, including the system's adaptation to material creep, mechanical wear and new fruit varieties, are maintained to ensure long-term stable control.
[0011] Preferably, the design of the forward state estimation network in step S1 specifically includes: Step 1: Design the input module: Divide the input branches according to signal type, including: electrical signal branch, visual branch, and fruit feature branch; the electrical signal branch inputs the timing sequence of motor signals, including: historical current, voltage, rotation angle and its derivative; the visual branch inputs the fruit and claw feature vectors extracted from the visual network; the fruit feature branch inputs fruit attributes, including: category, maturity, estimated hardness, and estimated weight. Step 2: Design the output module: Based on the data type of the output status, divide the output into three categories: regression task output, probability task output, and classification task output; regression task output includes: gripper closure degree, clamping force, wrist flexion angle, and breaking torque; probability task output is the probability of fruit stem breakage; classification task output is the operation stage identifier. Step 3: Build the network architecture: Build an architecture with multiple branches for input, feature fusion, and multiple tasks for output; the electrical signal branch uses a one-dimensional convolutional neural network or a long short-term memory network to process time series, the visual branch uses a convolutional neural network to extract image features, the fruit attribute branch uses a fully connected layer for processing, and the features of each branch are fused through splicing or attention mechanism, and then high-level semantic features are extracted through a multi-layer fully connected network, and finally each target variable is predicted. Step 4: Set the loss function: use mean squared error or smoothing loss for regression tasks, and cross-entropy loss for classification tasks. The total loss is the weighted sum of the losses of each task. Step 5: Network Training: Supervised learning is employed. Training data is collected by temporarily installing auxiliary sensors on the harvesting device to record motor signals, visual information, fruit attributes, and the actual labels measured by the sensors. A large number of samples are collected, covering different fruits, environments, and mechanical states. Data augmentation is used to expand the sample size. Standard deep learning training procedures are used to train on the training set, monitor performance on the validation set, and evaluate generalization ability on the test set. After training, all auxiliary sensors are removed, and inference is performed based on motor signals and visual information.
[0012] Preferably, the design of the fruit stalk breakage detection network in step S1 specifically includes: Step 1: Design network input: Extract the current, power, and angular acceleration of the drive motor within a short-time signal window as input data; Step 2: Design the network output: The network output is a binary classification or continuous probability; Step 3: Build a network architecture using a bidirectional LSTM, Transformer encoder, or one-dimensional convolutional network to automatically capture mutation features; Step 4: Design the loss function: Use class-weighted or special loss functions to handle the sample imbalance problem; Step 5: Set network evaluation criteria: use recall, precision, and detection latency as core indicators, and use them as the basis for judging training objectives and generalization ability; Step 6: Train the network: Supervised learning is adopted, and the training data consists of the current, power, and angular acceleration of the drive motor within a short-time signal window; manual annotation or auxiliary sensors are used to match the real labels to the signal window; the loss function in Step 4 is used to handle the sample imbalance problem; the samples in the signal window and the real labels are divided into training set, validation set, and test set according to the ratio; a standard deep learning training process is used, the Adam optimizer is selected, and an appropriate learning rate and batch size are set; the training set is input into the network, and training is performed for several rounds until convergence; the performance is monitored on the validation set; after training is completed, the metrics in Step 5 are calculated on the test set; if the metrics are met, the model training is complete; otherwise, rework and optimization are performed.
[0013] Preferably, the design of the inverse control decision network in step S1 specifically includes: Step 1: Design network input: Input data includes the forward state estimate obtained in step S3, the desired target value obtained in step S2, the state error between the forward state estimate and the desired target value, and historical control commands; Step 2: Design network output: Output data includes current command, speed command, and stop signal; Step 3: Build a multi-layer fully connected network or recurrent neural network; Step 4, Network Training: Training methods include supervised learning, reinforcement learning, or a hybrid approach.
[0014] Therefore, the present invention employs the above-mentioned rigid-flexible coupling sequence-driven harvesting device and intelligent control method, which has the following beneficial effects: (1) Minimization of drive source, simplification of structure and reduction of cost: The double-layer X-shaped telescopic mechanism realizes multi-sequence actions controlled by a single motor, which greatly reduces the number of drive sources compared to the multiple motors required by existing technologies. The cost of motors, reducers and transmission mechanisms is significantly reduced, and the elimination of multi-motor synchronous controllers and coordination sensors further reduces costs and significantly improves economic feasibility; (2) Significantly reduced end-efficiency and improved efficiency: The motor is placed at the base, and only a lightweight flexible end effector and rope are retained at the end, resulting in a significant reduction in end-efficiency mass and inertia. The acceleration of the robotic arm is significantly improved, the movement speed is faster, the single picking time is shortened, and the picking efficiency is greatly improved. The reduction in end-efficiency also reduces the load on the robotic arm joints, reducing the cost and energy consumption of the entire system; (3) Integrated Z-axis motion reduces system complexity: The double-layer X-shaped mechanism automatically switches between overall Z-axis motion and driving force transmission through the pure mechanical logic of the limit trigger mechanism, eliminating the need for a separate Z-axis linear module and control system. This eliminates the problems of weight, cost, failure points, and control complexity caused by an independent Z-axis mechanism, resulting in a simpler and more compact system structure; (4) Sensorless Precise Force Control: The forward state estimation network estimates the end clamping force and breaking torque from the motor signal, eliminating the need for expensive force sensors at the end, thus saving significant costs. Deep learning methods offer high force estimation accuracy. The sensorless design eliminates problems such as sensor drift, failure, and frequent calibration, improving system reliability and operational continuity.
[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the overall structure of a rigid-flexible coupling sequence-driven harvesting device according to the present invention. Figure 2 This is a schematic diagram of the closing and twisting of the terminal actuator of the present invention; Figure 3 This is a flowchart of the working cycle of a rigid-flexible coupling sequence-driven harvesting method according to the present invention.
[0017] Figure Labels 1.1 Drive motor; 1.2 Outer X-shaped mechanism; 1.3 Inner X-shaped mechanism; 2.1 Wrist elastic sheet; 2.2 Hand elastic sheet; 2.3 Flexible fingers; 2.4 Support component; 3. Drive rope. Detailed Implementation
[0018] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0019] Example 1: A rigid-flexible coupling sequence-driven kiwifruit harvesting device and its intelligent control method.
[0020] I. Specific Implementation Methods of the Mechanical Device (a) Overall structural composition, such as Figure 1 As shown The harvesting device consists of four main parts: a mobile base, a drive mechanism, a drive rope 3, and an integrated rigid-flexible coupling end effector.
[0021] The mobile base employs an XY two-dimensional Cartesian mechanism to position the end effector in the horizontal plane. The double-layered X-shaped telescopic mechanism of the drive mechanism serves as the Z-axis motion mechanism, enabling vertical telescopic movement and driving force transmission. A drive rope connects the telescopic mechanism to the end effector, transmitting the driving force. An integrated rigid-flexible coupling end effector is installed at the end of the telescopic mechanism to perform the clamping and breaking actions on the fruit.
[0022] (II) Manufacturing and Assembly of XY Two-Dimensional Cartesian Mechanisms The XY two-dimensional Cartesian mechanism consists of an X-axis slide rail, a Y-axis slide rail, a moving platform, and a drive system, forming a two-dimensional positioning mechanism in the horizontal plane.
[0023] The X-axis guide rail uses aluminum alloy profiles and is fixed to the ground support frame. The length of the guide rail is determined according to the working area. Linear guide rail pairs are installed on the guide rail surface, and the slider slides on the guide rail with low friction.
[0024] The Y-axis slide rail is mounted on the X-axis slider and is arranged perpendicular to the X-axis guide rail. The Y-axis slide rail also uses aluminum alloy profiles and linear guide pairs. The Y-axis slider slides on the Y-axis guide rail and supports the entire Z-axis telescopic mechanism and the end effector.
[0025] The drive system employs either synchronous belt drive or ball screw drive. Each of the X and Y directions is equipped with an independent motor and transmission mechanism. The motor is fixed to the end of the guide rail and drives the slider to move along the guide rail via the synchronous belt or ball screw. The motor is equipped with an encoder to provide position feedback.
[0026] The mobile platform is a mounting plate on the Y-axis slider, vertically fixed to the top of the series-connected X-shaped telescopic mechanism. The platform is made of aluminum alloy or steel and has sufficient rigidity to bear dynamic loads.
[0027] During assembly, first fix and level the X-axis guide rail, then install the X-axis slider and drive system. Install the Y-axis guide rail on the X-axis slider, ensuring it is perpendicular to the X-axis, then install the Y-axis slider and drive system. Finally, install the moving platform on the Y-axis slider. During debugging, ensure smooth, uninterrupted movement of both axes and that the positioning accuracy meets requirements.
[0028] (III) Manufacturing and assembly of the drive mechanism The drive mechanism comprises a double-layer X-shaped telescopic mechanism, including an outer X-shaped mechanism 1.2, an inner X-shaped mechanism 1.3, and a drive motor 1.1. The outer X-shaped mechanism 1.2 and the inner X-shaped mechanism 1.3 are composed of multiple X-shaped units connected in series, enabling a wide range of telescopic movement along the Z-axis and multi-functional switching with a single motor. Each X-shaped unit consists of four connecting rods linked together to form an X-shape. The connecting rods are made of hollow aluminum alloy profiles, which are lightweight and high-strength. The four connecting rods are of equal length and are connected in pairs via a central hinge to form an X shape. The four ends are the upper left, upper right, lower left, and lower right vertices, which are connected to adjacent units or a fixed platform via top hinges.
[0029] When the X-angle changes, the distance between the upper and lower hinges changes, achieving telescopic movement. When the X-angle is close to 180 degrees, the unit is in a fully extended state, with the vertical distance at its maximum; when the X-angle decreases, the unit contracts, and the vertical distance decreases.
[0030] The hinges use ball bearings or self-lubricating bearings and are made of stainless steel, ensuring flexible rotation, low friction, and long service life.
[0031] Multiple X-shaped units are connected in series vertically to form a Z-axis mechanism that can extend and retract over a wide range. The number of units connected in series is determined by the required extension and retraction range, and is usually three to six units.
[0032] The units are connected by top hinges. The lower top hinge of the lowest unit is fixed to the moving platform, and the upper top hinge of the highest unit is connected to an integrated rigid-flexible coupling end effector. All units move synchronously and extend and retract in a coordinated manner as a whole.
[0033] The double-layer X-shaped mechanism, with its inner and outer double-layer nested design, is key to achieving multi-functional switching of a single motor.
[0034] The outer X-shaped mechanism 1.2 is the active layer directly driven by a motor. All units in the outer layer are synchronously linked via rigid connecting rods. The driving method is as follows: drive cables or push rods are connected to the two lower vertices of the lowest X-shaped unit, with the other end of the drive cable connected to the drum or lead screw nut of the drive motor 1.1. When the drive motor 1.1 rotates forward, the cable tightens or the push rod extends, causing the X-shaped unit to unfold and lengthen; when the drive motor 1.1 rotates in reverse, the cable loosens or the push rod shortens, causing the X-shaped unit to contract and shorten. Due to the rigid linkage of each unit, the entire outer layer moves synchronously.
[0035] The inner X-shaped mechanism 1.3 is nested inside the outer X-shaped mechanism 1.2. Each unit is slightly smaller than the outer unit and is arranged coaxially with the outer unit. The inner X-shaped mechanism 1.3 is not directly driven by the motor, but is coupled to the outer X-shaped mechanism 1.2 through a limiting mechanism.
[0036] The fixed end of the drive rope 3 is connected to the uppermost hinge of the outer X-shaped mechanism 1.2. The relative movement of the inner and outer layers changes the length of the drive rope, thereby controlling the action of the end effector.
[0037] Drive motor 1.1 and transmission system The drive motor 1.1 is a DC servo motor or stepper motor, fixedly mounted on the mobile platform or base. The drive motor 1.1 is connected to the drum or lead screw through a reducer. The drive motor 1.1 drives the drum to wind up and unwind the cable in both forward and reverse directions, or drives the lead screw nut to move and push the outer X-shaped mechanism 1.2 to extend and retract.
[0038] The drive motor 1.1 is equipped with an encoder that provides real-time feedback on rotation angle and speed information. The driver receives control commands, controls the current and speed of the drive motor 1.1, and provides feedback on the status of the drive motor 1.1.
[0039] (iv) Manufacturing of integrated rigid-flexible coupling end effector The integrated rigid-flexible coupling end effector is manufactured using a one-piece molding process and achieves underactuated sequence actions driven by a single rope through a stiffness gradient design.
[0040] 1. Selection of molding method Integrated continuous structures are manufactured using multi-material 3D printing or segmented casting methods.
[0041] Multi-material 3D printing: This involves using a dual-nozzle or multi-nozzle 3D printer to simultaneously use thermoplastic elastomer materials (such as TPU series) with varying hardness. During printing, different areas use materials of varying hardness according to a preset hardness distribution, achieving a stiffness gradient. Different infill rates and internal structures (solid, honeycomb, corrugated, etc.) can also be set during printing to further adjust the stiffness.
[0042] Segmented casting: Using a multi-segment mold, liquid silicone rubber or polyurethane materials of different hardness are poured sequentially. The interfaces of each segment of material are chemically bonded in a partially cured state, forming a seamless continuum.
[0043] 2. Structural segment design The integrated rigid-flexible coupling end effector is divided into the following sections from top to bottom: Wrist: Comprising a flexible wrist sheet 2.1 and a support member 2.4, it is made of a high-hardness, flexible material and has a hollow tubular structure. The length and wall thickness of the wrist segment are designed to give it high bending stiffness. The function of the wrist segment is to bend after the fruit is grasped by the claws, applying a breaking torque to the fruit stalk.
[0044] The gripper comprises an elastic sheet 2.2 and flexible fingers 2.3. The flexible fingers 2.3 consist of three or four fingers evenly distributed circumferentially. The flexible fingers 2.3 are made of a low-hardness flexible material with an internal design featuring a high-permeability honeycomb or corrugated structure, significantly reducing bending stiffness. The outer surface of the flexible fingers 2.3 is covered with an ultra-soft material layer, providing a soft contact interface. The inner side of the flexible fingers 2.3 features a fine, friction-enhancing texture, increasing the coefficient of friction and improving gripping reliability.
[0045] 3. Implementation of stiffness gradient The stiffness ratio of the wrist to the claw is a key design parameter. Through various means such as material hardness selection, cross-sectional moment of inertia design, and hollowing ratio adjustment, the wrist stiffness is made to be much greater than the claw stiffness.
[0046] When the stiffness ratio is sufficiently high, during the initial stage of low rope tension, the low-stiffness gripper preferentially bends to hold the fruit, while the high-stiffness wrist remains largely unchanged. After the gripper closes, the rope tension continues to increase, and the wrist begins to bend, applying a breaking torque. This sequential triggering requires no sensor judgment or control switching; it is entirely achieved automatically by the mechanical stiffness characteristics.
[0047] After the sample is made, a bending test is conducted to verify whether the stiffness ratio meets the design requirements. If it does not meet the requirements, the material hardness, wall thickness, or perforation ratio is adjusted, and the sample is remade.
[0048] 4. Rope distribution and finger actuation, such as Figure 2 As shown There are two drive ropes: one connects the outer X-shaped mechanism and the wrist elastic plate, and the other connects the outer X-shaped mechanism and the claw elastic plate. One end of the flexible finger is connected to an elastic sheet for the hand, and the other end is connected to an elastic sheet for the wrist.
[0049] 5. Hollow Channel Design The integrated rigid-flexible coupling end effector has a hollow channel running from top to bottom at its center, with a diameter designed according to the size of the fruit. The channel serves as a fruit collection channel.
[0050] The fingers are arranged circumferentially around the hollow channel. When the fingers are open, the entrance to the hollow channel is open, allowing the fruit to enter. When the fingers hold the fruit, the fruit will not fall even when the fingers are closed. After the fingers are released and opened, the fruit falls through the channel under the influence of gravity, passes through the wrist, and enters the collection container.
[0051] The inner wall of the passage is smooth, and guide ribs are installed when necessary to help the fruit fall smoothly.
[0052] (v) Assembly and commissioning of the complete mechanical system Assemble the parts in order: First, assemble the XY Cartesian base, fix it on the ground or a moving vehicle, and adjust the X and Y directions to ensure smooth and accurate movement.
[0053] Install a double-layer X-shaped telescopic mechanism on the mobile platform, fixing the bottommost X-shaped unit. Install the outer layer units and connect the linkage rods, install the inner layer units, set the motion damping mechanism, and adjust the damping force. Install the limit trigger mechanism and adjust the limit position.
[0054] Install the drive motor 1.1 and transmission system, connect the drive cable to the outer mechanism, and test the forward and reverse rotation of the motor to make the outer layer extend and retract smoothly. Verify that the limit trigger mechanism operates reliably.
[0055] Install drive ropes 3, there are two in total; one end of one rope is connected to the elastic plate 2.2 of the hand claw, and the other end is connected to the outer X-shaped mechanism 1.2; the other rope is connected to the elastic plate 2.1 of the wrist and the outer X-shaped mechanism 1.2 at both ends respectively.
[0056] Install an integrated rigid-flexible coupling end effector and connect it to the top of the double-layer X-shaped telescopic mechanism.
[0057] Overall debugging: Complete one full work cycle using the manual or electric drive device, and observe whether the actions at each stage are correct: When ascending to the upper limit, the rope slackens, the gripper opens and returns to center; during descent to the lower limit, the rope tightens, the gripper clamps, and the wrist bends as the rope continues to tighten. Adjust the limit positions and damping forces until the actions at each stage are coordinated and reliable.
[0058] II. Detailed Implementation of the Control System (a) Hardware system configuration The control system hardware includes a main controller, motor drivers, cameras, and power supplies.
[0059] The main controller uses an industrial computer or embedded controller with sufficient processor performance to run deep learning inference. An operating system and deep learning framework are installed.
[0060] Three motor drivers control the X-axis motor, Y-axis motor, and Z-axis telescopic mechanism drive motor respectively. The drivers are connected to the main controller via a communication bus, receiving control commands and providing feedback on motor current, voltage, rotation angle, and other information.
[0061] The camera, either a color industrial camera or an RGB-D camera, is mounted above or to the side of the mobile platform, covering the work area. The camera connects to the main controller via USB or Ethernet.
[0062] The power system provides a stable power supply to all components.
[0063] (II) Software System Architecture The control software adopts a modular design, mainly including a data acquisition module, a fruit recognition module, a forward state estimation network, a fruit stem breakage detection network, a reverse control decision network, an execution control module, and an online learning module.
[0064] The data acquisition module collects the current, voltage, and rotation angle signals of the three motors, as well as camera images, in real time, and timestamps the data before storing it in a buffer.
[0065] The fruit recognition module runs a target detection network to identify the location, type, and maturity of the fruit from the image, estimate the size, hardness, weight, and other attributes of the fruit, and determine the expected clamping force and expected breaking torque.
[0066] The forward state estimation network is the core module. It runs a deep neural network to estimate the current state of the end effector from the Z-axis motor signal and fruit attributes, including the gripper closure degree, clamping force, wrist flexion angle, breaking torque, etc.
[0067] The fruit stalk breakage detection network is specifically designed to monitor fruit stalk breakage events, using the abrupt change characteristics of the Z-axis motor signal to determine in real time whether a breakage has occurred.
[0068] The inverse control decision network generates control commands based on the state estimate and the desired objective, and sends them to the motor driver.
[0069] The execution control module drives the motor according to control commands to achieve closed-loop control. The online learning module periodically fine-tunes network parameters using newly collected task data to improve control performance.
[0070] (III) Training of Deep Learning Networks 1. Training data collection Before formal application, force and torque sensors are temporarily installed on the end effector to measure the actual clamping force and breaking torque as labels.
[0071] Extensive harvesting experiments were conducted, involving the picking of different fruits and operations under varying environmental conditions. For each harvest, complete Z-axis motor signal timing (current, voltage, rotation angle and its derivative), camera images, fruit attributes, sensor measurements of the actual labels, and operational results were recorded.
[0072] The number of samples collected reached a sufficient scale, covering various working conditions.
[0073] 2. Network Structure Design The state estimation network uses a temporal convolutional network or a recurrent neural network to process the temporal sequence of the motor signal and extract temporal features. Simultaneously, fully connected layers are used to process fruit attribute features. The features from the two branches are fused and then output as a state estimate through a multi-layer fully connected network.
[0074] The network input consists of motor signals (time series of current, voltage, rotation angle, power, speed, and angular acceleration) and fruit attribute vectors within a time window.
[0075] The network output includes continuous values such as gripper closure, clamping force, wrist flexion angle, breaking torque, and breakage probability.
[0076] The fracture detection network uses a recurrent neural network or a one-dimensional convolutional network to process motor signals within a short time window, identify abrupt changes such as sudden drops in current, power reduction, and sudden increases in angular acceleration, and output a binary result or probability value for fracture judgment.
[0077] 3. Training process The network is trained using the collected data. The data is divided into training set, validation set, and test set.
[0078] The loss function is a weighted sum of the losses from each task; mean squared error is used for regression tasks, and cross-entropy loss is used for classification tasks.
[0079] Using the standard deep learning training process, employing optimizers such as Adam, setting appropriate learning rates and batch sizes, and training for several rounds until convergence.
[0080] Monitor performance on the validation set to prevent overfitting. Evaluate generalization ability on the test set after training.
[0081] 4. Deploy the application After training is complete, remove the temporarily installed force and torque sensors. Deploy the trained network model onto the main controller.
[0082] In actual operation, the network relies solely on the Z-axis motor signal and fruit attributes for inference, achieving sensorless force estimation and state estimation.
[0083] (iv) Implementation of control strategies, such as Figure 3 As shown 1. Rising Positioning Phase After system initialization, the fruit recognition module identifies the target fruit from the camera image and determines its XY coordinates.
[0084] The controller sends commands to the X and Y motors, driving the mobile platform to move the telescopic mechanism directly under the fruit.
[0085] The Z-axis motor rotates forward, driving the outer X-shaped mechanism 1.2 to unfold and extend upward, while the inner layer remains stationary, the rope slackens, and the grippers open.
[0086] When the outer X-shaped mechanism 1.2 reaches its upper limit, the limit trigger mechanism automatically engages, locking the inner and outer layers. The motor continues to rotate forward, and the inner and outer layers as a whole continue to rise, raising the end effector to an appropriate height below the fruit.
[0087] During this phase, the gripper remains open, and the system moves rapidly as a rigid whole.
[0088] 2. Clamping stage The Z-axis motor reverses direction, causing the outer X-shaped mechanism 1.2 to retract, while the inner X-shaped mechanism 1.3 tends to maintain its position due to damping. The outer X-shaped mechanism 1.2 moves downward relative to the inner X-shaped mechanism 1.3, causing the rope fixed to the top of the inner X-shaped mechanism 1.3 to be pulled downward taut.
[0089] Once the rope is taut, the tension is transmitted to the end effector. Due to the stiffness gradient design, the different spring stiffnesses cause the low-stiffness gripper to bend first, while the flexible fingers 2.3 close towards the center to clamp the fruit. The state estimation network estimates the clamping force in real time, and the control decision module adjusts the motor current based on the clamping force error to achieve closed-loop force control, ensuring that the clamping force reaches the desired value.
[0090] 3. Breaking stage After the flexible finger 2.3 is fully closed, the hand reaches its deformation limit. The motor continues to reverse, increasing the rope tension. The different spring stiffnesses achieve a high-stiffness wrist, which begins to bend, applying a breaking torque to the fruit stem. The torque gradually increases.
[0091] 4. Fracture Response and Harvesting Stage When the fruit stalk breaks, the motor load suddenly decreases, the current drops sharply, the power decreases abruptly, and the angular acceleration increases abruptly. The fracture detection network captures these characteristics and immediately outputs a fracture signal.
[0092] After descent, the motor switches to forward rotation, the outer layer unfolds, and the rope slackens. The wrist of the integrated rigid-flexible coupling end effector returns to center, and the gripper opens outward using elastic restoring force to release the fruit.
[0093] The fruit falls from the hollow center of the hand under the influence of gravity, and then falls into the collection container through the hollow channel.
[0094] 5. Reset Phase The outer X-shaped mechanism 1.2 returns to its upper limit position, and the system resets. Ready for the next harvest.
[0095] 6. Exception Handling If the clamping force fails to reach the expected value for an extended period, the clamping is deemed a failure. The rope is then released, the position adjusted, and the process repeated.
[0096] If the breaking torque continues to increase but no breakage is detected for a long time, the stem may be too hard. Stop the operation after reaching the upper limit of torque or time to avoid damaging the mechanism.
[0097] If any abnormalities such as motor failure or communication interruption are detected, the machine will immediately stop and an alarm will sound.
[0098] (v) Implementation of online learning For each harvesting operation, the system records complete data: motor signal timing, fruit attributes, network estimation, control commands, and operation results (success / failure, fruit damage, and operation time).
[0099] Run the online learning program periodically (e.g., after a certain number of assignments or after each day's assignments). Extract recent assignment data from the database and select high-quality samples to add to the training set. Fine-tune the network using a small learning rate and train it for a few epochs.
[0100] After fine-tuning, evaluate performance on the validation set. If performance improves, update the network parameter file and load the new parameters on the next startup.
[0101] Online learning enables the system to adapt to time-varying factors such as material creep, mechanical wear, temperature changes, and new fruit varieties, maintaining long-term stable and high-precision control.
[0102] Therefore, the present invention adopts the above-mentioned rigid-flexible coupling sequence-driven picking device and its intelligent control method, which can systematically overcome the two core bottlenecks of the existing picking robots: complex multi-drive source structure and difficult control of long transmission chains, and realize lightweight, low-cost and high-precision coordination of picking operations.
[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A rigid-flexible coupling sequence-driven harvesting device, characterized in that: It includes a drive mechanism, a drive rope, and an integrated rigid-flexible coupling end effector; the drive mechanism is fixedly mounted on a mobile base; the drive rope connects the drive mechanism and the integrated rigid-flexible coupling end effector; the integrated rigid-flexible coupling end effector is a one-piece molded continuous flexible structure, mounted above the drive mechanism.
2. The rigid-flexible coupling sequence-driven harvesting device according to claim 1, characterized in that, The drive mechanism adopts a double-layer X-shaped telescopic mechanism, including an outer X-shaped mechanism, an inner X-shaped mechanism, and a drive motor; the drive motor is installed in a reserved fixed position on the movable base; the lower end of the outer X-shaped mechanism is fixed on the movable base, and the upper end is the moving end, which extends and retracts in the vertical direction; the inner X-shaped mechanism is nested inside the outer X-shaped mechanism, and its size is slightly smaller than the outer layer. Its lower end is fixed on the movable base and a lower limit of the limit trigger mechanism is set nearby, and its upper end is connected to an integrated rigid-flexible coupling end actuator and an upper limit of the limit trigger mechanism is set nearby; a damping mechanism is set between the hinge nodes of the inner X-shaped mechanism.
3. The rigid-flexible coupling sequence-driven harvesting device according to claim 2, characterized in that: The outer X-shaped mechanism and the inner X-shaped mechanism are rhomboid retractable structures.
4. The rigid-flexible coupling sequence-driven harvesting device according to claim 2, characterized in that, The integrated rigid-flexible coupling end effector includes a support, a wrist elastic sheet, a gripper elastic sheet, and a flexible finger; the lower ends of the wrist elastic sheet and the support are connected to the upper platform of the inner X-shaped mechanism, and the upper ends are connected to the flexible finger; the outer periphery of the gripper elastic sheet is connected to the flexible finger.
5. The rigid-flexible coupling sequence-driven harvesting device according to claim 4, characterized in that, There are two drive ropes; one end of one rope is connected to the elastic plate of the hand claw, and the other end is connected to the outer X-shaped mechanism; the other rope is connected to the elastic plate of the wrist and the outer X-shaped mechanism at both ends respectively.
6. An intelligent control method applied to the rigid-flexible coupling sequence-driven harvesting device described in claims 1-5, characterized in that, Includes the following steps: S1. Design and train the forward state estimation network, the fruit stalk breakage detection network, and the inverse control decision network; S2. Data Acquisition: Real-time acquisition of drive motor signal timing information and RGB-D visual images; drive motor signal timing information includes drive motor current, voltage, and rotation angle signals within a certain time window. S3. Fruit Recognition: Based on RGB-D visual images, the fruit location, type, and maturity are identified through a deep convolutional neural network. The fruit firmness and weight are estimated, and the desired targets are determined. The desired targets include the desired gripper closure degree, the desired clamping force, the desired breaking torque, and the desired operation stage. S4. Forward State Estimation: The current terminal state is estimated by using the forward state estimation network based on the timing of the drive motor signal collected in S2 and the fruit information obtained in S3. This includes the degree of gripper closure, actual clamping force, wrist flexion angle, breaking torque, probability of fruit stem breakage, and operation stage identifier. S5. Reverse Control Decision and Execution: Based on the work stage identifier, combined with the forward state estimation results and the desired target, instructions are generated through the reverse control decision network, which drives the motor through the execution control module. This process is divided into five stages: Ascending and positioning stage: Based on the Z-axis height target of fruit recognition, the reverse control decision network generates a forward rotation command for the drive motor, and the integrated rigid-flexible coupling end effector reaches the bottom of the fruit tree; Clamping stage: Force closed-loop control is adopted to monitor and estimate the clamping force in real time. Based on the error between the actual clamping force and the expected clamping force, the drive motor current is adjusted until the clamping force reaches the target value. Breaking stage: The execution control module adopts torque progressive control, so that the breaking torque gradually increases from zero, and the rate of increase is adjusted according to the characteristics of the fruit; Fracture response and fruit harvesting stage: After the fruit stem fracture detection network detects the fracture, the execution control module immediately issues a stop command, drives the motor to brake urgently, and then switches to the rising fruit harvesting stage; Reset phase: Constant speed control is adopted, the drive motor rotates at a constant speed, the outer X-shaped mechanism returns to the upper limit position, the inner X-shaped mechanism resets synchronously, the wrist returns to the center, the claws open fully, the fruit falls, and it waits for the next picking cycle. S6. Online learning and adaptation: After each harvesting cycle, the complete data sequence and result evaluation are automatically recorded; new data are added to the training set according to the set period, and the network parameters are fine-tuned with a small learning rate to update each network module. After fine-tuning, the model performance is evaluated. If the accuracy is improved, the network model in the main controller is updated to enable the system to adapt to time-varying factors, including material creep, mechanical wear, and new fruit varieties, and maintain long-term stable control.
7. The intelligent control method for a rigid-flexible coupling sequence-driven harvesting device according to claim 6, characterized in that, The design of the forward state estimation network in step S1 specifically includes: Step 1: Design the input module: Divide the input branches according to signal type, including: electrical signal branch, visual branch, and fruit feature branch; the electrical signal branch inputs the timing sequence of motor signals, including: historical current, voltage, rotation angle and its derivative; the visual branch inputs the fruit and claw feature vectors extracted from the visual network; the fruit feature branch inputs fruit attributes, including: category, maturity, estimated hardness, and estimated weight. Step 2: Design the output module: Based on the data type of the output status, divide the output into three categories: regression task output, probability task output, and classification task output; regression task output includes: gripper closure degree, clamping force, wrist flexion angle, and breaking torque; probability task output is the probability of fruit stem breakage; classification task output is the operation stage identifier. Step 3: Build the network architecture: Build an architecture with multiple branches for input, feature fusion, and multiple tasks for output; the electrical signal branch uses a one-dimensional convolutional neural network or a long short-term memory network to process time series, the visual branch uses a convolutional neural network to extract image features, the fruit attribute branch uses a fully connected layer for processing, and the features of each branch are fused through splicing or attention mechanism, and then high-level semantic features are extracted through a multi-layer fully connected network, and finally each target variable is predicted. Step 4: Set the loss function: use mean squared error or smoothing loss for regression tasks, and cross-entropy loss for classification tasks. The total loss is the weighted sum of the losses of each task. Step 5: Network Training: Supervised learning is employed. Training data is collected by temporarily installing auxiliary sensors on the harvesting device to record motor signals, visual information, fruit attributes, and the actual labels measured by the sensors. A large number of samples are collected, covering different fruits, environments, and mechanical states. Data augmentation is used to expand the sample size. Standard deep learning training procedures are used to train on the training set, monitor performance on the validation set, and evaluate generalization ability on the test set. After training, all auxiliary sensors are removed, and inference is performed based on motor signals and visual information.
8. The intelligent control method for a rigid-flexible coupling sequence-driven harvesting device according to claim 6, characterized in that, The design of the fruit stalk breakage detection network in step S1 specifically includes: Step 1: Design network input: Extract the current, power, and angular acceleration of the drive motor within a short-time signal window as input data; Step 2: Design the network output: The network output is a binary classification or continuous probability; Step 3: Build a network architecture using a bidirectional LSTM, Transformer encoder, or one-dimensional convolutional network to automatically capture mutation features; Step 4: Design the loss function: Use class-weighted or special loss functions to handle the sample imbalance problem; Step 5: Set network evaluation criteria: use recall, precision, and detection latency as core indicators, and use them as the basis for judging training objectives and generalization ability; Step 6: Train the network: Supervised learning is adopted, and the training data consists of the current, power, and angular acceleration of the drive motor within a short-time signal window; manual annotation or auxiliary sensors are used to match the real labels to the signal window; the loss function in Step 4 is used to handle the sample imbalance problem; the samples in the signal window and the real labels are divided into training set, validation set, and test set according to the ratio; a standard deep learning training process is used, the Adam optimizer is selected, and an appropriate learning rate and batch size are set; the training set is input into the network, and training is performed for several rounds until convergence; the performance is monitored on the validation set; after training is completed, the metrics in Step 5 are calculated on the test set; if the metrics are met, the model training is complete; otherwise, rework and optimization are performed.
9. The intelligent control method for a rigid-flexible coupling sequence-driven harvesting device according to claim 6, characterized in that, The design of the inverse control decision network in step S1 specifically includes: Step 1: Design network input: Input data includes the forward state estimate obtained in step S3, the desired target value obtained in step S2, the state error between the forward state estimate and the desired target value, and historical control commands; Step 2: Design network output: Output data includes current command, speed command, and stop signal; Step 3: Build a multi-layer fully connected network or recurrent neural network; Step 4, Network Training: Training methods include supervised learning, reinforcement learning, or a hybrid approach.