A fruit continuous picking control system and method based on simulink and ros

Abstract

This invention discloses a coherent fruit-harvesting control system and method based on Simulink and ROS. The system includes forward and inverse kinematic models of a robotic arm based on the MATLAB / Simulink software platform, establishing forward and inverse kinematic control system modules for the robotic arm; building a network connection and communication mechanism between the Linux operating system and the ROS system, publishing data to Simulink according to the ROS message format, and constructing a fruit detection vision system module; controlling the opening and closing of fingers based on the state of the harvester controller, establishing two message formats under ROS, and completing the construction of the harvester system control module; planning the motion trajectory of the robotic arm based on target localization and the forward and inverse kinematic models, controlling the operation of the robotic arm's end effector, and completing the entire process from the approach point to the target fruit, the robotic arm harvesting the fruit, and collecting the fruit. The entire system integrates a vision recognition system, a harvester gripping system, and a robotic arm motion control system, solving the problems of difficult integration of subsystems and long development cycles in traditional harvesting robots.

Description

Technical Field

[0001] This invention relates to the field of agricultural robot control, and in particular to a fruit harvesting control system and method based on Simulink and ROS. Background Technology

[0002] With the continuous development of agricultural automation, fruit-harvesting robots are gradually becoming an important tool to replace manual harvesting. However, existing fruit-harvesting robots still have many shortcomings in terms of control precision, work efficiency, and adaptability to complex environments. Especially in orchard environments, traditional robot control systems face significant limitations due to the complexity of fruit distribution and variable working conditions. Currently, many harvesting robots still rely on manual programming and debugging, which is time-consuming, labor-intensive, and prone to program errors, making it difficult to achieve multi-arm collaborative operation and precise control. In addition, existing robot control systems often lack effective path planning and obstacle avoidance strategies, causing the robotic arms to easily collide during movement, affecting work quality and harvesting efficiency. Summary of the Invention

[0003] This invention addresses the problems of insufficient control precision, integration difficulties, and low harvesting efficiency of existing fruit-harvesting robots in complex orchard environments. It proposes a fruit-harvesting control system and method based on Simulink and ROS. This system integrates the modular modeling and simulation advantages of Simulink with the real-time communication and control capabilities of ROS, forming a highly efficient and intelligent harvesting control system. Through innovative control strategies and modular design, it effectively improves the robot's adaptability and operational efficiency in complex environments.

[0004] The technical solution of the present invention is as follows:

[0005] A fruit-harvesting control system based on Simulink and ROS includes:

[0006] The robotic arm control module establishes the forward kinematics model and inverse kinematics model of the robotic arm based on the Simulink software platform. The system control module plans the motion trajectory of the robotic arm through the forward kinematics model and inverse kinematics model, which is used to control the robotic arm end effector to accurately perform the tasks of approaching, grasping, picking and collecting the target fruit.

[0007] The vision recognition module, installed in the Linux operating system, is used to acquire the three-dimensional coordinates and posture information of the fruit in real time through the ROS system, and publish it to Simulink according to the ROS message format to provide the fruit position data required by the robotic arm control module.

[0008] The harvester control module controls the opening and closing of the harvester by controlling the high and low level states of the harvester controller I / O port. Based on the identified fruit position and the movement trajectory of the robotic arm, the module adjusts the operating posture of the harvester in real time to ensure accurate fruit grasping and safe release.

[0009] The communication control module is used to establish a network connection and communication mechanism between the Linux operating system and the ROS system. It transmits the fruit position data obtained by the vision recognition module to the robotic arm control module and synchronizes the motion commands of the robotic arm control module to the picking hand control module, so as to enable data interaction and collaborative control between the modules.

[0010] An integrated control module is designed based on Simulink to implement the harvesting control strategy. It integrates a vision recognition module, a robotic arm control module, and a harvester control module. The harvesting control strategy is generated into C++ code and directly deployed to the ROS system to complete the harvesting work.

[0011] The robotic arm control module:

[0012] Specifically, it is used to establish the forward and inverse kinematic models of the robotic arm using the DH parameter method, and to perform trajectory planning using the principle of minimizing state difference. The principle of minimizing state difference selects the motion path with the least change by comparing the difference between the current state of the robotic arm and the target state, so as to enable the robotic arm end to accurately locate and grasp the fruit.

[0013] A method for a coherent fruit harvesting control system based on Simulink and ROS includes the following steps:

[0014] 1) The robotic arm moves to the initial position: The robotic arm first moves to the set initial position, which is located on one side of the fruit collection box, so as to avoid the robotic arm blocking the camera and improve the overall work efficiency;

[0015] 2) Setting an approach point: When grabbing the target fruit, the robotic arm is first moved to the set approach point, which is an intermediate position to avoid the robotic arm colliding with other fruits or branches when approaching the target fruit.

[0016] 3) Path planning and grasping: Calculate the motion path of the robotic arm using forward and inverse kinematics models to ensure that the end effector of the robotic arm moves smoothly from the approach point to the target fruit position;

[0017] 4) Perform the grasping action: The picker adjusts the grasping posture at the approach point, closes the fingers and performs a rotating and horizontal picking action to ensure the fruit is safely picked;

[0018] 5) Fruit placement: After harvesting, the robotic arm moves the fruit to the fruit collection box and places it safely, then returns to the initial position to prepare for the next harvesting operation;

[0019] 6) Repeat steps 2 to 5: Continue the picking process until all fruits within the designated area have been picked.

[0020] The beneficial technical effects of this invention are as follows:

[0021] 1. By combining Simulink and ROS, the development process of the control system was simplified, enabling the harvesting robot to operate with high precision in complex orchard environments.

[0022] 2. The modular design facilitates the independent optimization and integration of each function, enhancing the system's scalability and adaptability, and enabling it to quickly adapt to the harvesting needs of different types of orchards.

[0023] 3. Real-time data exchange is achieved through a custom communication mechanism, and the efficient collaboration between modules significantly improves the overall system performance and operational efficiency, enabling the harvesting robot to maintain a stable and efficient operating state in a variable environment. Attached Figure Description

[0024] Figure 1 This is a structural block diagram of a motion control system for a fruit-harvesting robot based on Simulink and ROS, provided in one embodiment of the present invention.

[0025] Figure 2 This is a structural block diagram of a fruit-harvesting robot motion control system based on Simulink and ROS, provided in another embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the workflow of a fruit-harvesting robot motion control system based on Simulink and ROS, provided in one embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of a forward kinematics Simulink model provided in one embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of a Simulink model based on inverse kinematics with minimum state difference provided in an embodiment of the present invention.

[0029] Figure 6 This is a schematic diagram of the Simulink and ROS communication modules of a visual recognition and positioning system provided in one embodiment of the present invention.

[0030] Figure 7 This is a Simulink and ROS communication module for the forward and inverse kinematics control system of a robotic arm provided in one embodiment of the present invention.

[0031] Figure 8This is a communication module between Simulink and ROS for the harvester opening and closing control system provided in one embodiment of the present invention.

[0032] Figure 9 This is a schematic diagram of a fruit coherent harvesting control strategy model based on Simulink and ROS provided in one embodiment of the present invention.

[0033] Figure 10 This is a schematic diagram of the initial pose control model of a robotic arm provided in one embodiment of the present invention.

[0034] Figure 11 This is a schematic diagram of a harvester opening control model provided in one embodiment of the present invention.

[0035] Figure 12 This is a schematic diagram of a visual recognition and positioning data acquisition model provided in one embodiment of the present invention.

[0036] Figure 13 This is a schematic diagram of the approach point control model of the robotic arm and the picking hand grasping the fruit according to an embodiment of the present invention.

[0037] Figure 14 This is a schematic diagram of a control model for the movement of a robotic arm to the position of holding the fruit, provided in one embodiment of the present invention.

[0038] Figure 15 A schematic diagram of a harvester closure control model provided in one embodiment of the present invention.

[0039] Figure 16 This is a schematic diagram of a torsion-pulling harvesting control model provided in one embodiment of the present invention, which involves the coordinated operation of a robotic arm and a harvesting hand.

[0040] Figure 17 This is a schematic diagram of a control model for the movement of a robotic arm and a picking hand to a fruit collection box, provided in one embodiment of the present invention.

[0041] Figure 18 This is a schematic diagram of a harvester opening control model provided in one embodiment of the present invention.

[0042] Figure 19 This is a schematic diagram of a control model for restoring the harvester to a natural state, provided in one embodiment of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the protection scope of the present invention.

[0044] Figure 1As shown, this invention provides a continuous fruit harvesting control system based on Simulink and ROS, comprising:

[0045] The robotic arm control module, based on the Simulink software platform, establishes forward and inverse kinematic models of the robotic arm. By solving these models, the system module plans the robotic arm's motion trajectory to precisely control its end effector to approach, grasp, pick, and collect target fruit. This module leverages Simulink's graphical modeling and simulation capabilities to simplify complex kinematic calculations into drag-and-drop operations, and integrates with ROS for real-time data interaction, enabling precise positioning and motion control of the robotic arm in varying environments. Through customized control strategies and seamless integration with ROS, efficient command response and precise robotic arm operation are achieved.

[0046] The visual recognition module, installed on the Linux operating system, is used to acquire the three-dimensional coordinates and posture information of the fruit in real time through the ROS system. This information is then published to Simulink according to the ROS message format, providing the robotic arm control module with the fruit position data required by the system. This module integrates advanced image processing technology and machine learning algorithms, enabling accurate identification of the target fruit's position, size, and posture under complex backgrounds and various lighting conditions. Data processing is optimized using parallel computing, significantly improving the speed and accuracy of visual recognition and providing reliable target data for subsequent grasping operations.

[0047] The harvester control module controls the opening and closing of the harvester hand through the high and low level states of the harvester hand controller's I / O ports. Based on the detected fruit position and the robotic arm's movement trajectory, it adjusts the harvester hand's operating posture in real time to ensure precise fruit grasping and safe release. This module senses and adjusts based on feedback from different fruit characteristics, employing a flexible grasping and force control strategy to ensure the fruit is not damaged during harvesting. The harvester hand's structural design incorporates biomimetic principles, enabling it to adapt to various fruit shapes and improving the stability and reliability of the grasping process.

[0048] The communication control module is used to establish a network connection and communication mechanism between the Linux operating system and the ROS system. It transmits the fruit position data acquired by the vision recognition module to the robotic arm control module and synchronizes the motion commands of the robotic arm control module to the harvester control module, so as to enable data interaction and collaborative control between the modules. The Simulink simulation interface provides the function of real-time monitoring and adjustment of the communication process, effectively reducing data transmission latency and improving the overall system response speed.

[0049] The integrated control module, based on Simulink, designs the harvesting control strategy, integrating a vision recognition module, a robotic arm control module, and a harvester control module. The harvesting control strategy is generated into C++ code and directly deployed to the ROS system to complete the harvesting operation. The communication control module handles real-time communication between Simulink and the ROS system. Through an asynchronous one-way ROS Topic mechanism, modules exchange data using a custom message format. This module utilizes a custom communication protocol and data format to optimize transmission speed and data synchronization capabilities, ensuring efficient collaboration among modules in complex operating environments. Simulink's simulation interface provides real-time monitoring and adjustment of the communication process, effectively reducing data transmission latency and improving the overall system response speed. The integrated control module designs and simulates the harvesting control strategy in Simulink, encapsulating the recognition, grasping, picking, and placement stages as independent modules. Logical judgments connect these steps to form a closed-loop control system. The integrated control module supports flexible switching between various harvesting strategies, such as planar grasping, rotational grasping, and two-handed coordination, to adapt to different types and states of fruit. Through simulation testing and optimization, each control strategy can be verified before deployment, ensuring safety and efficiency in actual operation.

[0050] Specifically, the robotic arm control module 1:

[0051] Specifically, this method is used to establish forward and inverse kinematic models of a robotic arm using the DH parameter method, and to perform trajectory planning using the principle of minimizing state difference. This principle selects the motion path with the least change by comparing the difference between the current state and the target state of the robotic arm, thus enabling the robotic arm's end effector to accurately locate and grasp the fruit. During path planning, the spatial distribution of the fruit, the kinematic constraints of the robotic arm, and the influence of environmental obstacles are considered, and the motion trajectory is dynamically adjusted to avoid collisions and interference.

[0052] The robotic arm control module 1 includes a dedicated forward kinematics model and an inverse kinematics model encapsulated in Simulink:

[0053] The forward kinematics model is modularly modeled using Simulink. It calculates the position and orientation of the end effector in space by using the angles of each joint of the robotic arm, and encapsulates it in a graphical and modular way in Simulink. It sets the initial pose of the robotic arm and provides an accurate initial state reference for motion path planning. The inverse kinematics model is built and encapsulated in Simulink. It calculates the motion path of the end effector by solving the change values ​​of each joint angle in reverse and combining them with the target position. Through Simulink's simulation and verification functions, it controls the precise motion control of the robotic arm from the current pose to the target pose, thereby optimizing the path for grasping and collecting the fruit.

[0054] The visual recognition module 2 is specifically used for:

[0055] The three-dimensional spatial coordinates of the fruit are obtained by a depth camera, and the size, shape and position of the fruit are detected in real time by combining the fruit recognition algorithm. The detection results are published to Simulink through a custom message format based on geometry_msgs / Point, which is then provided to the robotic arm control module for path planning.

[0056] The communication control module 3 is specifically used for:

[0057] An asynchronous one-way ROS Topic communication mechanism is adopted to publish and subscribe to each subsystem through a custom message format, so as to realize data transmission and command synchronization between the vision recognition module, the robotic arm control module and the harvester control module.

[0058] This communication control module uses the interface between Simulink and ROS to exchange data, and monitors and adjusts message transmission through a graphical interface to ensure real-time collaboration and efficient integration of various modules, thereby enhancing the stability and response speed of the system.

[0059] The integrated control module 5 is embedded in Simulink and uses SwitchCase Block logic to define the picking control strategy. It encapsulates each step of fruit identification, grasping, picking and placing as an independent module, and connects each step through logical judgment conditions to achieve automated and continuous operation.

[0060] The integrated control module 5 is also used to respond to feedback signals from the vision recognition module, the robotic arm control module, and the harvester control module, optimize the control process through simulation testing, and generate C++ code to be deployed to the ROS system.

[0061] The harvesting control strategy includes a fully automated control process encompassing fruit recognition, robotic arm movement, hand grasping, and fruit placement. Utilizing the integration of Simulink and ROS, deployment is achieved with a single click via ROS's Roslaunch tool, supporting automatic parameter adjustment and error detection. Specifically, fruit recognition involves setting an approximation point before identifying the target fruit, guiding the robotic arm to a safe intermediate position. The robotic arm movement employs an adaptive algorithm to adjust its speed and the hand grasping force, achieving non-destructive fruit handling. Simulink and ROS simulation tests are used to optimize the parameter configurations for each stage.

[0062] This invention also discloses a method for a continuous fruit harvesting control system based on Simulink and ROS, comprising the following steps:

[0063] 1) The robotic arm moves to the initial position: The robotic arm first moves to the set initial position, which is located on one side of the fruit collection box, so as to avoid the robotic arm blocking the camera and improve the overall work efficiency;

[0064] 2) Setting an approach point: When grabbing the target fruit, the robotic arm is first moved to the set approach point, which is an intermediate position to avoid the robotic arm colliding with other fruits or branches when approaching the target fruit.

[0065] 3) Path planning and grasping: Calculate the motion path of the robotic arm using forward and inverse kinematics models to ensure that the end effector of the robotic arm moves smoothly from the approach point to the target fruit position;

[0066] 4) Perform the grasping action: The picker adjusts the grasping posture at the approach point, closes the fingers and performs a rotating and horizontal picking action to ensure the fruit is safely picked;

[0067] 5) Fruit placement: After harvesting, the robotic arm moves the fruit to the fruit collection box and places it safely, then returns to the initial position to prepare for the next harvesting operation;

[0068] 6) Repeat steps 2 to 5: Continue the picking process until all fruits within the designated area have been picked.

[0069] This invention discloses a fruit-harvesting control method and system based on Simulink and ROS. The system's key feature lies in its unique logic control strategy design, which tightly integrates Simulink's simulation and modular modeling advantages with ROS's real-time communication and control capabilities. Simulink provides an intuitive graphical modeling environment, allowing control algorithms, path planning, and robotic arm motion models to be quickly built and optimized in a modular manner. Through the logic control strategy designed in this invention, these models and algorithms are seamlessly integrated into the ROS system, achieving real-time data transmission and precise execution of control commands. This control method not only effectively integrates Simulink's dynamic simulation capabilities with ROS's distributed computing power but also enables real-time adjustment and optimization of the coordination between modules during the harvesting process, significantly improving the robot system's response speed, control accuracy, and adaptability to complex environments. Through this innovative control strategy, this invention overcomes the bottlenecks of traditional harvesting robot technology, providing reliable technical support for achieving intelligent and stable automatic fruit harvesting.

[0070] like Figure 1 The present invention provides a structural block diagram of a fruit-harvesting robot motion control system based on Simulink and ROS, which includes a hardware structure 100 and a software system 200.

[0071] The hardware structure 100 includes an industrial computer 300, a driver board 400, a sensor 500, and an actuator 600. The driver board 400 includes a power module 410, a driver module 420, and a USB module 430. The software system 200 includes a Simulink simulation platform 210 and a ROS operating platform 220.

[0072] Figure 1 As shown, the industrial computer 300 is used to install the Simulink simulation platform 210 and the ROS operating platform 220, serving as the core processor of the motion control system for the fruit-harvesting robot. The industrial computer 300 is connected to the driver board 400 via a USB module 430. The driver board 400 includes a power module 410 and a driver module 420, where the driver module 420 is used to process and execute the control decisions transmitted by the industrial computer 300. The sensor 500 is connected to the industrial computer 300 and transmits the collected three-dimensional information of the fruit to the industrial computer 300. The driver module 420 drives the actuator 600 to operate according to the control commands of the industrial computer 300. The power module 410 provides power support for the other modules on the driver board 400.

[0073] The motion control system for the fruit-harvesting robot based on Simulink and ROS in this embodiment of the invention can be divided into two levels: one is the workflow of the software system 200, including the Simulink simulation platform 210 and the ROS operating platform 220; the other is the motion control process of the hardware structure 100 with the industrial computer 300 and the drive module 420 as the core.

[0074] In software system 200, the Simulink simulation platform 210 is responsible for the design and simulation of the control strategy, including the robotic arm control module, vision recognition module, harvester control module, communication control module, and integrated control module. The ROS operating platform 220 is installed under a Linux operating system (such as Ubuntu) and exchanges data with the Simulink simulation platform 210 through the ROS communication mechanism to realize the real-time transmission and execution of the control strategy.

[0075] like Figure 2 As shown, the actuator 600 includes a robotic arm 700, a harvester 900, a motor 620, and an air pump 610. The drive module 420 drives the motor 620 according to the control instructions of the industrial computer 300, controlling the movement and grasping actions of the robotic arm 700 and the harvester 900; the air pump 610 provides air pressure support for the harvester 900, enabling precise grasping and release of the fruit.

[0076] Further reference Figure 2The hardware structure 100 also includes a depth camera 500, which acquires the three-dimensional coordinates and posture data of the fruit in real time through the ROS system and transmits the data to the Simulink simulation platform 210 for path planning and control command generation.

[0077] The hardware structure 100 also includes an encoder 630, which is used to provide real-time feedback on the motion status of the motor 620 to ensure the precise movement and position control of the robotic arm 700, thereby enabling precise operation during the fruit grasping and placement process.

[0078] The hardware structure 100 also includes an input / output interface 450, which is used to realize data interaction and debugging between various modules, ensuring the stability and efficient operation of the entire robot motion control system.

[0079] The hardware structure 100 also includes a power supply module 410, which provides a stable power supply to all modules to ensure the reliable operation of each component.

[0080] Figure 3 This is a flowchart illustrating the motion control system of a fruit-harvesting robot based on Simulink and ROS, according to an embodiment of the present invention. This method is applied to applications such as... Figure 1 or Figure 2 The image shows a motion control system for a fruit-harvesting robot based on Simulink and ROS. This method achieves automated harvesting from "recognition—grasping—picking—fruit placement" through the coordinated work of a vision system, a robotic arm control system, and a harvester's hand control system. The following follows... Figure 3 The five stages are described in detail, with each step explained in its implementation:

[0081] Preparation Phase: The system initializes the robot system by starting the Roslaunch node, ensuring smooth communication between modules. The industrial computer 300 connects to the driver board 400 via USB module 430, and the driver board 400 provides power to all hardware in the system via power module 410. The robotic arm 700 and the picking hand 900, as actuators, receive instructions according to the ROS topic message format and move to the preset initial pose under the control of the driver board, ensuring that the robotic arm does not obstruct the vision system's camera field of view. At this time, the vision system begins to acquire image data and identify the position of the target fruit. Simultaneously, the picking hand module receives instructions and begins to open its fingers, preparing for subsequent grasping. Optionally, refer to the reference... Figure 2 The drive board 400 is connected to the industrial computer 300 via a USB interface to ensure that the robotic arm and harvester can accurately execute control commands.

[0082] Target detection stage: The vision system acquires environmental images in real time using a depth camera 500, extracts RGB image data, and detects the position of the target fruit using image processing algorithms. The depth camera 500 connects to the industrial control computer 300 via USB to acquire the 3D coordinate information of the target fruit and transmit it to the robotic arm control system. Using the target detection algorithm, the system detects the target fruit based on the RGB image data and performs 3D calibration based on the recognition results, converting the target fruit's coordinate information into a coordinate format that the robotic arm can process, providing accurate position information for subsequent actions. Optionally, in conjunction with reference... Figure 1 The depth camera 500 is connected to the industrial control computer 300 to ensure that the vision system provides real-time and accurate environmental perception data in complex orchard environments.

[0083] Grasping Phase: The robotic arm generates an approximation point based on the three-dimensional coordinates of the target fruit and calculates the motion trajectory using an inverse kinematic model, then moves to the approximation point. The industrial computer 300 controls the movement of the robotic arm and the harvester via the drive board 400, ensuring the robotic arm avoids collisions with the fruit and surrounding obstacles as it approaches the target fruit. At the approximation point, the robotic arm adjusts the attitude of its end effector, ensuring alignment with the target fruit, ready to grasp. At this time, the harvester closes its fingers according to control commands, ensuring accurate grasping of the target fruit. Optional, refer to the reference... Figure 2 The algorithm for optimizing the movement path of the robotic arm adopts a dynamic path adjustment strategy to ensure that it can smoothly perform grasping tasks even when encountering complex obstacles in the orchard environment.

[0084] Harvesting Stage: When the robotic arm's end effector reaches the target fruit, the harvester executes a horizontal harvesting mode to ensure accurate fruit grasping. The robotic arm's end joint rotates, maintaining the x and z coordinates while adding a fixed value to the y coordinate to ensure smooth harvesting. The rotation angle and translation distance are adjusted based on the target fruit's position and the fruit collection box's position to ensure the harvested fruit can be easily placed into the collection box. Throughout the process, the industrial computer 300 coordinates the movements of the robotic arm and harvester via the drive board 400, ensuring the accuracy and stability of the harvesting process. Optional, refer to the reference... Figure 2 The control precision of the harvester can be further optimized by adjusting the control module to ensure adaptability to different types of fruit.

[0085] Fruit placement stage: After fruit grasping, the robotic arm's end effector delivers the fruit to the collection box. During placement, the harvester executes a pressure release action according to control commands to ensure the fruit is safely released into the collection box and to prevent damage during placement. After the fruit placement operation is completed, the robotic arm's end effector returns to its initial position, preparing for the next round of harvesting, ensuring the system can operate efficiently and stably continuously. Optional, refer to the reference... Figure 1During this stage, the movement path of the robotic arm is adjusted through an optimization algorithm, which further improves the accuracy and stability of fruit delivery. In particular, when the position of the fruit collection box changes, the system can quickly adjust the movement path of the robotic arm.

[0086] Optional, refer to the reference Figure 1 In the hardware structure 100, the system controls various components of the robot through the interconnection of the industrial control computer 300, drive board 400, sensor 500, and actuator 600. The industrial control computer 300, as the core control unit of the system, is responsible for data exchange with various modules. The drive board 400 is connected to the industrial control computer 300 via a USB module 430 and provides power to all hardware in the system via a power module 410, ensuring the normal operation of the hardware system. The actuator module 440 is responsible for executing the robot's control commands and drives the joints of the robotic arm 700 through the drive module 420.

[0087] Optional, refer to the reference Figure 2 In the software system 200, the Simulink simulation platform 210 and the ROS operating platform 220 coordinate the operation of the robotic arm control, vision recognition, and communication modules to ensure the efficient operation of the robot system. The ROS platform 220 performs data transmission and real-time control through the mutual coordination of modules such as Node 1 and Node 2. The Simulink simulation platform 210 simulates the harvesting task to ensure that the system has undergone rigorous testing and verification before hardware deployment, maximizing harvesting efficiency and accuracy.

[0088] Reference Figure 4 It shows a schematic diagram of the robot's forward kinematics Simulink model.

[0089] Optional, refer to the reference Figure 4 It shows a schematic diagram of the robot's forward kinematics Simulink model. Figure 4 This demonstrates how to use a Simulink model to perform forward kinematics calculations to determine the position and orientation of a robotic arm's end effector. The model takes an angle as input (…). θ The model uses forward kinematics (ForwardSolver_MDH) to calculate and output the end effector pose (Pos) of the robotic arm. The output includes position (X, Y, Z coordinates) and orientation (Roll, Pitch, Yaw angles), providing precise position data for the robotic arm's end effector. Through the Simulink platform, each joint angle of the robotic arm affects the position and orientation of the end effector, ensuring that the robotic arm can perform precise motion control according to the set goals.

[0090] Optional, refer to the reference Figure 4The forward kinematics model, using Matlab's simulation capabilities, simulates the actual position of the robotic arm at different joint angles, providing data support for subsequent control system optimization. Specifically, theta represents the angle of each joint, while Pos represents the calculated end-effector pose, including its coordinates and orientation angles in three-dimensional space. This model can adjust joint angles and motion trajectories in real time, ensuring the accuracy and stability of the robotic arm, especially for effective control when performing tasks in complex environments.

[0091] Reference Figure 5 It shows a schematic diagram of a Simulink model based on inverse kinematics with minimum state difference.

[0092] Optional, refer to the reference Figure 5 It shows a schematic diagram of a Simulink model based on inverse kinematics with minimum state difference. Figure 5 This demonstrates how to use inverse kinematics algorithms to calculate the angles of each joint of a robotic arm. θ This model achieves precise control of the target position (Pos) by inputting the position of the robotic arm's end effector (such as X, Y, Z coordinates and attitude angles) as target data and using inverse kinematics to calculate the corresponding joint angles to achieve the desired end effector position.

[0093] The model uses the InverseSolver_MDH module to handle inverse kinematics calculations and further optimizes the solution process using AllSolverTheta to ensure the robotic arm can find the optimal path during movement, minimize joint angle differences, and guarantee precise positioning. To further optimize the solution process, the round module is used to round the calculation results to ensure that the joint angle values ​​conform to the actual motion constraints of the robotic arm.

[0094] Optional, refer to the reference Figure 5 To select the optimal solution, the system employs the "minimum state difference" principle among multiple inverse kinematics solutions. This principle calculates the state difference (i.e., the difference between the current state of the robotic arm and the state of the upcoming picking action) for each solution and selects the solution with the smallest difference. In practical applications, after providing the positioning pose, the inverse kinematics model calculates eight possible solutions. The system then selects the solution that minimizes the change in the current robotic arm state, avoiding excessive joint angle adjustments and unnecessary movements.

[0095] The model calculated using this method θ The (joint angle) is transmitted to the control system, and the robotic arm performs precise movements based on this control signal. Optionally, this can be combined with a reference... Figure 2The output of the inverse kinematics model is controlled by the Simulink simulation platform of the software system 200, ensuring that control commands can be effectively transmitted to the hardware through the ROS operating platform, thus guaranteeing the accuracy and stability of the robotic arm when performing complex tasks.

[0096] Reference Figure 6 It shows a schematic diagram of the communication modules of Simulink and ROS in the visual recognition and positioning system.

[0097] Optional, refer to the reference Figure 6 This module demonstrates how to transmit and process visual data in real time using ROS and Simulink systems. In this module, the vision system communicates with the Simulink system via a ROS topic / chatter, receiving data (such as the 3D coordinates of a target) from the vision sensor.

[0098] In ROS, the IsNew signal is used to check if a new message (Msg) has been received. If a new message is received, the data (such as the target's X, Y, Z coordinates and W value) is transmitted to the Simulink system. The data is written to the Vision Data Store through a message passing mechanism, ensuring that the data can be read and used in real time in Simulink for subsequent target localization and control.

[0099] Optionally, this communication module supports multi-channel data transmission, enabling it to handle various visual recognition tasks, such as real-time localization and obstacle detection, ensuring the robot can perform precise operations in complex environments. Through efficient communication between ROS and Simulink, visual data can be rapidly fed back to the control system, achieving real-time dynamic control. Specifically, the vision system uses the `geometry_msgs` data type in ROS to transmit target position information. `geometry_msgs / Point` describes the target position (x, y, z), and `geometry_msgs / Quaternion` describes the target's quaternion orientation (x, y, z, w), converting the quaternions to Euler angles (r, p, y) when necessary for use by the robot arm's inverse kinematics model. Through this mechanism, the vision system's output provides the robot arm with the necessary position information (x, y, z) and orientation information (r, p, y).

[0100] Reference Figure 7 It shows a schematic diagram of the communication modules of Simulink and ROS for the forward and inverse kinematics control system of the robotic arm.

[0101] Optional, refer to the reference Figure 7This module demonstrates how to transmit data and send control commands between ROS and the Simulink system. In this module, ROS publishes a target message via the topic ` / rocr6_msgs / Goal`, indicating the target position or motion command of the robotic arm. This message data is transmitted to the Simulink system via the data bus (Bus) and further assigned to the control parameters of different robotic arm joints (A1 to A6).

[0102] In Simulink, data is extracted from the Goal Message and read via the CMD Data StoreRead module. Then, the data is converted into a form suitable for robotic arm control (such as target joint angles) using a custom format. This data is passed to the control command (Cmd) via the Arm Bus Assignment module and finally sent to the robotic arm actuator via the ROS system's publish mechanism ( / aubo5_msgs / goal).

[0103] Optional, refer to the reference Figure 7 The robotic arm's joint angle information is communicated via a custom message format. In this system, the standard ROS database cannot fully meet the complex data transmission requirements, necessitating a custom message format. First, a `Goal.msg` file is created, defining the six joint angles of the robotic arm (a1, a2, a3, a4, a5, a6), with these angles in radians (float64 type). Second, a `Feedback.msg` file is defined to provide feedback on the status information of each joint of the robotic arm. The feedback format includes `Joint shoulder_pan_joint`, `Joint shoulder_lift_joint`, `Jointelbow_joint`, `Joint wrist_1_joint`, `Joint wrist_2_joint`, and `Joint wrist_3_joint`, also in radians, used for real-time monitoring and adjustment of the robotic arm's motion.

[0104] Reference Figure 8 The diagram shows a schematic of the communication module between Simulink and ROS for the harvester's opening and closing control system.

[0105] Figure 8This demonstrates how to coordinate the opening and closing of a harvester's hand using a Simulink system in conjunction with a ROS system. The harvester controls the opening and closing of its fingers by using two states—high and low—of the controller's I / O ports. When U_DO_00 is 0 and U_DO_01 is 1, the finger is closed; when U_DO_00 is 1 and U_DO_01 is 0, the finger is open; when U_DO_00 is 0 and U_DO_01 is 1, the finger closes again; and when U_DO_00 is 1 and U_DO_01 is 1, the finger is released.

[0106] In the Simulink model, this is controlled through the geometry_msgs / Quaternion message format. In this message format, orientation contains four float64 variables: x, y, z, and w, where w takes values ​​of 0, 1, and 2, which are used to control different states of the picking hand: w=0 represents fingers closed, w=1 represents fingers open, and w=2 represents fingers relaxed.

[0107] This information is transmitted to the robotic arm control system via the ROS publishing mechanism / gripper. The data bus in the Simulink model distributes the state of the harvester (X, Y, Z coordinates and W angle) to the control module, ensuring that the harvester of the robotic arm end effector can perform the predetermined actions.

[0108] Reference Figure 9 It shows a schematic diagram of a fruit coherent harvesting control strategy model based on Simulink and ROS.

[0109] Optional, refer to the reference Figure 9 This model uses a SwitchCase Block system to modularize each key step in the harvesting process, ensuring that each stage has a clear execution logic. Specifically, Figure 9 The core is to control the order and execution of picking tasks through various "case" blocks.

[0110] Figure 9 The SwitchCase Block structure triggers different "cases" based on different control conditions, with each "case" representing a picking operation stage. By transmitting signals, the system can progressively complete all operations from step 1 (initialization) to step 9 (fruit placement). These steps are executed in the following order:

[0111] 1) Initialization (case 0): First, the system enters the initial position to ensure that the robotic arm is in the correct starting position.

[0112] 2) Open fingers (case 1): After the initial pose is completed, the system instructs the grasping hand to open its fingers in preparation for grasping.

[0113] 3) Visual data acquisition (case 2): The vision system begins to identify the target and acquire the three-dimensional coordinate information of the fruit.

[0114] 4) Approaching the target (case 3): The robotic arm approaches the target fruit based on feedback information from the vision system, preparing to grab it.

[0115] 5) Grasping the fruit target (case 4): The robotic arm moves to the fruit and grasps the target, ensuring that the fingers are closed.

[0116] 6) Harvesting mode (case 6): The harvester performs a horizontal pulling mode to ensure the safe harvesting of the fruit.

[0117] 7) Move to collection box (case 7): The harvested fruit is transported to the collection box.

[0118] 8) Fruit placement operation (case 8): The robotic arm places the fruit into the fruit collection box and releases its fingers.

[0119] 9) Pressure release operation (case 9): After the fruit is released, the robotic arm performs a pressure release action to ensure that the fruit falls smoothly into the fruit collection box.

[0120] Optional, refer to the reference Figure 3 The implementation of this control strategy is accomplished through close collaboration between ROS and Simulink. Figure 3 The five stages demonstrated (preparation stage, target detection stage, grasping stage, picking stage, and fruit placement stage) coordinate between modules through a message passing mechanism, ensuring efficient communication and data synchronization between each module. Each "case" block connects to the external control system via ROS topics, and every movement and state change of the robotic arm and picking hand is transmitted to the control system in real time, ensuring that the system automatically adjusts according to the current state. The steps of each stage are executed sequentially, ensuring the accuracy and continuity of the picking process. Starting from the preparation stage, through target detection, grasping, picking, and fruit placement, the entire operation process is finally completed.

[0121] Optional, refer to the reference Figure 9 This demonstrates a model of a coherent fruit harvesting control strategy based on Simulink and ROS. Figure 9 In this process, the harvesting process is broken down into multiple independent and orderly steps. Each step proceeds only after the previous one has been fully executed, ensuring the logic and continuity of the control process.

[0122] Each "case" block represents an independent operation step, covering the entire process from initializing the robotic arm's position to placing the fruit. These steps include key aspects such as target positioning, robotic arm movement, and the actions of the picking hand.

[0123] Optional, Figure 9 The SwitchCase Block control strategy design in this context has the following characteristics:

[0124] 1) Modularization of steps: Each step is encapsulated as an independent module, and the modules are linked through data transmission to ensure that the system executes smoothly according to the predetermined process.

[0125] 2) Real-time monitoring and debugging: Through the integration of Simulink and ROS, the execution status of each step can be monitored in real time, ensuring accurate execution of operations. If a problem occurs at a certain stage, the problematic module can be quickly located and corrected.

[0126] 3) Graphical design: The control strategy is designed through a graphical interface, which facilitates development, debugging, and subsequent system expansion and secondary development.

[0127] Reference Figure 10 It shows a schematic diagram of the initial pose control model of the robotic arm.

[0128] Optional, refer to the reference Figure 10 The system first sends the target rotation angles of the six joints to the robotic arm via the Goal.msg message format. These angle values ​​are then combined... Figure 4 The forward model was obtained through simulation. The robotic arm adjusts the angle of each joint according to these target angles to achieve the initial pose.

[0129] The symbol 0 in the diagram is written into the global variable rad1, representing the initial state of the forward kinematics model. The symbol [A] represents the Goto-From module, used to transmit signals between data points within the same subsystem, ensuring accurate signal transmission and effective coordination between modules.

[0130] Optionally, the workflow of the control model is started in Case [0], and the joint angles of the robotic arm are updated according to the target rotation angle. By continuously calculating the angle difference of each joint of the robotic arm, when the absolute difference between the angles of all joints and the target angle is less than 0.001 radians, the Case [0] condition is terminated, and the system enters Case [1], that is, it begins to execute the next stage of the action.

[0131] In Case [1], the system calls the feedback information (Feedback.msg) of the 6 joints in ROS and performs absolute difference calculation with the target angle in real time. Only after the discrimination condition is met will it proceed to the next operation. Through this control strategy, the robotic arm can accurately adjust the joint angles according to the feedback information to ensure that its pose meets the target requirements.

[0132] Reference Figure 11 It shows a schematic diagram of the control model for opening the harvester.

[0133] Optional, refer to the reference Figure 11 The control logic is located in the Case [1] module and is used to execute the opening operation of the picking hand. The system first encodes the finger opening instruction into a quadruple [0 0 0 2] according to the geometry_msgs / Quaternion message format and writes it into the hand global variable. This variable serves as an intermediate communication medium and can be used by Figure 8 The communication module shown reads the data and publishes it to the corresponding harvester control interface via ROS.

[0134] Subsequently, the system reads the state value in the `hand` variable and performs logical judgment on the `w` element: if the current value meets the set condition (e.g., `w≥2`), it determines that the opening action of the picking hand has been completed; if `w<3`, the system converts it into a numerical signal and compares it with a threshold to confirm whether the opening action meets the judgment condition for entering the next control state. When the opening state judgment is valid, the signal is output to the global variable `A` via the logic judgment module, indicating that the current opening operation has been completed and the next control step can be entered.

[0135] Optional, refer to the reference Figure 2 The system's hardware structure, through the coordinated operation of the industrial control computer 300, driver board 400, sensor 500, and actuator 600, ensures precise control and stable operation of the harvester. The industrial control computer 300 is responsible for receiving control signals and transmitting instructions to the harvester control module through the driver board 400, ensuring that the opening and closing movements of the fingers can be executed accurately.

[0136] Reference Figure 12 It shows a schematic diagram of a visual recognition and positioning data acquisition model.

[0137] Optional, refer to the reference Figure 12 The Simulink implementation structure of the visual recognition and localization data acquisition model is shown. The model is located in the Case [2] module and is used to receive, judge and filter data from the vision system in order to control whether to proceed to the subsequent target detection and pose calculation steps.

[0138] In this module, the system first reads the global variable data written by the vision system through the Vision Data Store Read node. This variable contains the three-dimensional coordinate information output by the vision recognition module. Then, the model performs logical judgments on the coordinate values ​​(such as whether it is less than 1000, whether it is a positive number, etc.) and summarizes the signals that meet the conditions into the selector module. If the judgment logic is valid, the model will write the data value into a new global variable data2 (implemented through the Vision Data2 Store Write module) and generate a control signal to write into variable A, which serves as the basis for judgment when entering the target approximation stage (Case [3]).

[0139] Optional, refer to the reference Figure 2 The system, through the interconnection and cooperation of the industrial control computer 300, drive board 400, sensor 500, and actuator 600, ensures the accurate transmission and processing of vision system data, further guaranteeing the stability of target positioning and control. (Refer to reference...) Figure 13 It shows a schematic diagram of the control model for the robotic arm and the picking hand to grasp the fruit before approaching the fruit.

[0140] Optional, refer to the reference Figure 13 This model corresponds to the Case [3] module, which is mainly used to handle the control process of the robotic arm end moving from the current position to the target approach point.

[0141] In this module, the system first reads the 3D coordinate data of the target fruit through the Vision Data2 Store Read node. Then, it performs a validity check on the coordinate values ​​using logical judgments, and superimposes the valid coordinate values ​​with a set of preset offsets (such as [1.18, 1.22, 0.3]) to generate the 3D pose information of the approximation point. This approximation point data is processed by the Reshape module and written to the global variable rad, serving as the input for the subsequent inverse kinematics module.

[0142] The main purpose of designing the approach point is to reserve a safe distance before performing the grasping action, allowing the robotic arm's end effector sufficient space for attitude adjustment as it approaches the target fruit. This design helps reduce the risk of structural interference in complex orchard settings, especially reducing the possibility of collisions caused by blind spots or obstructions, thereby improving the feasibility of the grasping path and the stability of system operation.

[0143] Meanwhile, the system obtains the current joint angle feedback by subscribing to the ROS topic / aubo5_msgs / feedback and compares it with the angle value obtained by inverse solving of the approximation point for joint-by-joint error. When the absolute difference of all joints is less than the set threshold (e.g., 0.01), it is determined that the robotic arm has accurately reached the approximation point, and the system writes control variable A to trigger the execution of the grasping action in the Case [4] stage.

[0144] Reference Figure 14 It shows a schematic diagram of the control model for the robotic arm to move to the position of holding the fruit.

[0145] Optional, refer to the reference Figure 14 This model corresponds to the Case [4] module, which is mainly used to control the end of the robotic arm to move further from the approach point to the precise grasping position of the target fruit.

[0146] The system reads the 3D coordinate data of the target fruit through the Vision Data2 Store Read node and corrects the Z-axis coordinate to compensate for the spatial deviation when the end effector contacts the fruit. The correction value is set by empirical parameters. After the coordinate data passes the validity check, it is combined with the position correction vector to generate the target grasping pose information. This pose information is converted into a data format by the Reshape module and written to the global variable rad, which serves as the input to the inverse kinematics model.

[0147] Position information input to Figure 5 The inverse kinematics model shown calculates the corresponding six joint angles and outputs them to control the movements of each joint of the robotic arm. Simultaneously, the system obtains the current joint angle state of the robotic arm by subscribing to the ` / aubo5_msgs / feedback` topic in ROS. This feedback angle is related to... Figure 5 The target angle obtained by solving the inverse kinematics model is compared joint by joint, and the system sets a threshold to determine whether the error range meets the control requirements.

[0148] When the absolute difference of all joint angles is less than the set threshold (e.g., 0.01), the system will output a control signal to variable A as the logical basis for triggering the subsequent Case [5] picking action. This mechanism realizes the precise alignment of the pre-grabbing posture and the closed-loop confirmation of the state, ensuring the reliability and continuity of the picking action.

[0149] Reference Figure 15 It shows a schematic diagram of the harvester's closed control model.

[0150] Optional, refer to the reference Figure 15 This model corresponds to the Case [5] module, which is used to control the picking hand to complete the closing operation after the end of the robotic arm has reached the fruit grasping position.

[0151] The control logic of this module and Figure 11 The control model shown is consistent with the one used for opening the picking hand, also employing the geometry_msgs / Quaternion message format. Control commands are written to global variables in quaternion form, and then... Figure 8 The communication module shown sends a message to the harvester 900. The system sets the variable w to the value corresponding to the closed state, and the control signal is transmitted from the industrial computer 300 to the driver module 420 via the USB module 430, thereby controlling the harvester 900 to perform the finger closing action.

[0152] Meanwhile, the system obtains the current position and feedback status information of the robotic arm by subscribing to the ROS topic / aubo5_msgs / feedback, and calculates the difference between the movement value of the harvester's end effector in the Y direction and the set target. When the difference is less than the set threshold (e.g., 0.01), and the feedback status meets the execution judgment requirements, the system writes a control signal to variable A as the trigger condition for subsequent harvesting mode execution steps.

[0153] Optional, refer to the reference Figure 2 The system, through the collaboration of the industrial control computer 300, the driver board 400, and the USB module 430, ensures the precise execution of the harvester's movements while maintaining good coordination with the robotic arm and vision system. (Refer to reference...) Figure 16 It shows a schematic diagram of a torsion-pulling harvesting control model in which the robotic arm and the harvesting hand work together.

[0154] Optional, refer to the reference Figure 16 This model corresponds to the Case [6] module, which is mainly used to control the end of the robotic arm to execute a specific picking mode after the picking hand has closed and grasped the fruit, so as to realize the fruit detachment and picking.

[0155] The system reads the end effector pose data (data5) stored in the previous stage from the global variable rad, and triggers the start of the current stage action by variable A. This action controls the end effector to perform a certain amplitude of perturbation in spatial coordinates to form a combined lateral and rotational motion. Specifically, the model adjusts the end effector pose in real time by setting the translation amounts in the X and Y directions, thereby achieving pull path control of the end effector.

[0156] The joint control signals of the robotic arm are generated by the industrial computer 300 and transmitted to the drive module 420 via the USB module 430, driving the robotic arm 700 in the actuator 600 to realize the end-effector trajectory transformation. After detecting the disturbance completion status (such as whether the translational offset reaches the set value of 0.1), the model outputs a judgment signal to variable A as the switching condition for the subsequent result placement stage.

[0157] Reference Figure 17It shows a schematic diagram of the control model for the movement of the robotic arm and the picking hand to the fruit collection box.

[0158] Optional, refer to the reference Figure 17 This model corresponds to the Case [7] module, which is mainly used to control the end of the robotic arm to move from the current working position to the specified position of the fruit collection box.

[0159] In this model, the system first reads the target pose data stored in the global variable rad (by... Figure 5 The data is calculated using the inverse kinematics model shown, and the Reshape module is used to format the data into a format suitable for processing by the inverse kinematics model. The target position (collection position) is then written to the global variable collection position and transmitted to the control system via the ROS topic message mechanism to guide the movement of the robotic arm's end effector.

[0160] The angles of the six joints of the robotic arm (joint 1 to joint 6) are calculated, and a suitable feedback control system ensures that the end effector can accurately move to the designated position in the fruit collection box. During this process, Figure 5 The inverse kinematics model, as shown, comes into play again, calculating the joint angles required for the robotic arm's end effector to reach the fruit collection box based on the target position. During the inverse kinematics calculation, the target data (such as X, Y, and Z coordinates and Roll, Pitch, and Yaw angles) is passed as input to the inverse kinematics model to solve for the appropriate joint angles, enabling the robotic arm's end effector to precisely align with the fruit collection box.

[0161] By monitoring the difference between the target position and the actual position in real time, when the error is less than a set threshold (e.g., 0.1), the control signal of the robotic arm is written to A, indicating that the target position has been successfully reached. At this time, the control signal ensures that each joint of the robotic arm reaches the target pose, ensuring that it can successfully complete the docking of the fruit collection box.

[0162] Optional, refer to the reference Figure 1 and Figure 2 In the hardware structure 100, the collaboration of the industrial control computer 300, driver board 400, USB module 430, and power module 410 ensures the stability and accuracy of the robotic arm control system. The industrial control computer 300, as the core control unit of the system, communicates with the robotic arm through the driver board 400. The power module 410 provides the necessary power support for the entire system, ensuring its normal operation. Simultaneously, the sensor 500 and actuator 600 work closely with other parts of the system to ensure the robotic arm can accurately perform various tasks.

[0163] Reference Figure 18 It shows a schematic diagram of the control model for opening the harvester.

[0164] Optional, refer to the reference Figure 18 The model and Figure 11 The control models shown are consistent, both controlling the opening state of the harvester through the geometry_msgs / Quaternion message format. The specific control logic is the same as... Figure 11 The description is the same. In the Case[1] module, the picker operates according to the set quaternion values. The signal is exchanged with the Simulink system through the ROS topic to control the finger opening state.

[0165] Reference Figure 19 It shows a schematic diagram of a control model for restoring the harvester to a natural state.

[0166] Optional, refer to the reference Figure 19 The model is located in the Case [9] module and is used to perform the operation of restoring the picker to a natural state, marking the completion of a picking cycle.

[0167] The system first encodes the instruction to restore the picking hand to its natural state into a quadruple [0 0 0 3] according to the geometry_msgs / Quaternion message format, and writes it to the global variable 'hand'. This variable serves as an intermediate communication medium and can be... Figure 8 The communication module reads the data and publishes it to the harvester control interface via ROS. At this time, the system reads the status value in the hand variable and performs a logical judgment on the w element: if the w value is greater than or equal to 3, it means that the harvester has completed the release action.

[0168] Once the relaxation action is completed, the system will update the value of variable A through a logical judgment signal, indicating that the picking cycle has ended and triggering a re-entry into Case [1] to start a new picking cycle. At this time, the picking hand returns to its initial open state, and the robotic arm is ready to perform the next round of picking tasks.

[0169] A harvesting process control system model was established using the MATLAB / Simulink toolbox. After simulation testing, the control algorithm model can automatically generate C++ code and be applied to the project. This process not only shortens the project development cycle and improves development efficiency, but also effectively avoids the problem of repeated debugging required by traditional hand-written code. Through the tight integration of Simulink and the ROS system, after the simulation model is verified to be error-free, the control algorithm can be quickly converted into deployable C++ code and directly applied to the robot hardware, further improving the system's development efficiency and stability.

[0170] It will be apparent to those skilled in the art that the above specific examples are merely preferred embodiments of the present invention. Therefore, any improvements or modifications that may be made by those skilled in the art to certain parts of the present invention still embody the principles of the present invention and achieve the objectives of the present invention, and all fall within the scope of protection of the present invention.

Claims

1. A continuous fruit harvesting control system based on Simulink and ROS, characterized in that, include: The robotic arm control module establishes the forward kinematics model and inverse kinematics model of the robotic arm based on the Simulink software platform. The system control module plans the motion trajectory of the robotic arm through the forward kinematics model and inverse kinematics model, which is used to control the robotic arm end effector to accurately perform the tasks of approaching, grasping, picking and collecting the target fruit. The vision recognition module, installed in the Linux operating system, is used to acquire the three-dimensional coordinates and posture information of the fruit in real time through the ROS system, and publish it to Simulink according to the ROS message format to provide the fruit position data required by the robotic arm control module. The harvester control module controls the opening and closing of the harvester by controlling the high and low level states of the harvester controller I / O port. Based on the identified fruit position and the movement trajectory of the robotic arm, the module adjusts the operating posture of the harvester in real time to ensure accurate fruit grasping and safe release. The communication control module is used to establish a network connection and communication mechanism between the Linux operating system and the ROS system. It transmits the fruit position data obtained by the vision recognition module to the robotic arm control module and synchronizes the motion commands of the robotic arm control module to the picking hand control module, so as to enable data interaction and collaborative control between the modules. An integrated control module was designed based on Simulink to implement a harvesting control strategy. This module integrates a vision recognition module, a robotic arm control module, and a harvester control module. The harvesting control strategy is then generated into C++ code and directly deployed to the ROS system to complete the harvesting process. The robotic arm control module: Specifically, it is used to establish the forward and inverse kinematic models of the robotic arm using the DH parameter method, and to perform trajectory planning using the principle of minimizing state difference. The principle of minimizing state difference selects the motion path with the least change by comparing the difference between the current state of the robotic arm and the target state, so as to enable the robotic arm end to accurately locate and grasp the fruit. The visual recognition module is specifically used for: The three-dimensional spatial coordinates of the fruit are obtained by a depth camera, and the size, shape and position of the fruit are detected in real time by combining the fruit recognition algorithm. The detection results are published to Simulink through a custom message format based on geometry_msgs / Point, which is provided to the robotic arm control module for path planning. The communication control module is specifically used for: An asynchronous one-way ROS Topic communication mechanism is adopted to publish and subscribe to each subsystem through a custom message format, so as to realize data transmission and command synchronization between the vision recognition module, the robotic arm control module and the harvester control module. This communication control module uses the interface between Simulink and ROS to exchange data, and monitors and adjusts message transmission through a graphical interface to ensure real-time collaboration and efficient integration of various modules, thereby enhancing the stability and response speed of the system.

2. The fruit harvesting control system based on Simulink and ROS according to claim 1, characterized in that, The robotic arm control module includes a forward kinematics model and an inverse kinematics model encapsulated in Simulink: Among them, the forward kinematics model is based on modular modeling in Simulink. It calculates the position and orientation of the end effector in space by the angle of each joint of the robotic arm, and encapsulates it in a graphical and modular way in Simulink to set the initial pose of the robotic arm and provide an accurate initial state reference for motion path planning. The inverse kinematics model is built and encapsulated in Simulink. By solving the change values ​​of each joint angle in reverse, and combining the target position, the motion path of the robotic arm end is calculated. Through Simulink's simulation and verification functions, the precise motion control of the robotic arm from the current pose to the target pose is controlled, thereby optimizing the path for grasping and collecting the fruit.

3. The fruit harvesting control system based on Simulink and ROS according to claim 1, characterized in that, The integrated control module is embedded in Simulink and uses SwitchCase Block logic to define the picking control strategy. It encapsulates each step of fruit identification, grasping, picking and placing as an independent module, and connects each step through logical judgment conditions to achieve automated and coherent operation.

4. The fruit harvesting control system based on Simulink and ROS according to claim 2, characterized in that, The integrated control module is also used to respond to feedback signals from the vision recognition module, the robotic arm control module, and the harvester control module, optimize the control process through simulation testing, and generate C++ code to be deployed to the ROS system.

5. The fruit harvesting control system based on Simulink and ROS according to claim 2, characterized in that, The harvesting control strategy includes a fully automated control process encompassing fruit recognition, robotic arm movement, hand grasping, and fruit placement. Utilizing the integration of Simulink and ROS, the strategy enables one-click deployment via ROS's Roslaunch tool, supporting automatic parameter adjustment and error detection.

6. The fruit harvesting control system based on Simulink and ROS according to claim 2, characterized in that, The fruit recognition specifically includes: setting an approximation point before recognizing the target fruit, the approximation point being used to guide the robotic arm to first enter a safe intermediate position; the robotic arm movement is achieved by adjusting the movement speed of the robotic arm and the grasping force of the picking hand through an adaptive algorithm to realize non-destructive processing of the fruit, and the parameter configuration of each step is optimized by using Simulink and ROS simulation tests.

7. A method for a continuous fruit harvesting control system based on Simulink and ROS, characterized in that, Includes the following steps: 1) The robotic arm moves to the initial position: The robotic arm first moves to the set initial position, which is located on one side of the fruit collection box, so as to avoid the robotic arm blocking the camera and improve the overall work efficiency; 2) Setting an approach point: When grabbing the target fruit, the robotic arm is first moved to the set approach point, which is an intermediate position to avoid the robotic arm colliding with other fruits or branches when approaching the target fruit. 3) Path planning and grasping: Calculate the motion path of the robotic arm using forward and inverse kinematics models to ensure that the end effector of the robotic arm moves smoothly from the approach point to the target fruit position; 4) Perform the grasping action: The picker adjusts the grasping posture at the approach point, closes the fingers and performs a rotating and horizontal picking action to ensure the fruit is safely picked; 5) Fruit placement: After harvesting, the robotic arm moves the fruit to the fruit collection box and places it safely, then returns to the initial position to prepare for the next harvesting operation; 6) Repeat steps 2 to 5: Continue the picking process until all fruits within the designated area have been picked.

Images