Green plum fruit picking robot based on multi-source data fusion and picking method thereof

Abstract

The invention discloses a green plum fruit picking robot based on multi-source data fusion and a picking method thereof, and relates to the technical field of agricultural intelligent equipment. The robot comprises a crawler-type self-adaptive chassis, a multi-degree-of-freedom mechanical arm, a multi-source sensing module, a bionic flexible end effector, an intelligent control unit and a fruit collecting box, and all the parts are electrically connected with the intelligent control unit. The multi-source sensing module synchronously collects vision, spectrum, touch, acoustics and environment data, and accurate recognition and positioning of mature fruits are achieved through hierarchical weighted attention fusion algorithm processing. And fruit damage is reduced in combination with a path planning algorithm introducing a flexible obstacle avoidance coefficient and a flexible picking execution mechanism. The self-adaptive chassis can adapt to complex terrains, and parameters are dynamically optimized through reinforcement learning in the picking process. The green plum picking machine solves the problems that traditional picking is low in efficiency and high in damage rate, and existing equipment is poor in adaptability, and the intelligent level and operation quality of green plum picking are improved.

Description

Technical Field

[0001] This invention relates to the field of agricultural intelligent equipment technology, and in particular to a plum fruit picking robot based on multi-source data fusion and its picking method. Background Technology

[0002] Green plums are widely cultivated in complex terrains such as mountains and hills. Their fruit is characterized by its thin skin, tender flesh, and concentrated ripening period, requiring extremely high efficiency and precision in harvesting. Currently, green plum harvesting is still mainly done manually, which presents several technical bottlenecks: Firstly, manual harvesting requires workers to climb tree branches or use simple tools to reach higher ground, resulting in high labor intensity and low efficiency. Furthermore, the steep terrain and narrow forest paths in mountainous areas easily lead to falls, collisions, and other safety accidents. Secondly, judging fruit ripeness relies primarily on vision and experience, making it difficult to accurately distinguish between ripe and unripe fruits, as well as those damaged by pests or diseases. Additionally, the obstruction of branches and leaves easily leads to mis-picking and missed harvesting, resulting in inconsistent fruit quality and directly impacting subsequent processing quality and economic benefits.

[0003] With the development of agricultural mechanization and intelligence, some fruit and vegetable harvesting robots have begun to be applied in production, but there are still obvious shortcomings in specialized equipment for plum harvesting:

[0004] Most existing harvesting robots use wheeled chassis, which are difficult to adapt to the uneven terrain of plum planting areas and lack an effective posture adjustment mechanism. When the chassis tilts during operation, it can easily lead to a decrease in the motion accuracy of the robotic arm and make it impossible to complete the harvesting action stably. At the same time, although tracked chassis have a certain off-road capability, their surface anti-slip performance is insufficient, and they are prone to slipping in muddy or steep areas, affecting the stability of operation.

[0005] Most harvesting robots rely on a single visual sensor for fruit identification. Affected by factors such as changes in light intensity, shading by branches and leaves, and similar fruit and background colors, it is difficult to accurately extract the fruit outline and location information, and it is even more difficult to effectively distinguish the fruit's maturity and pest and disease status. Although some devices have introduced multiple sensors, they have not established an effective data fusion mechanism and have failed to make full use of the complementarity of the sensory data from various dimensions, resulting in large identification errors and high rates of mis-harvesting and missed harvesting.

[0006] Existing end effectors mostly adopt rigid structure designs, which lack flexibility in adjusting the gripping force and have poor adaptability to the spherical contour of plum fruits, easily causing damage to the peel and crushing of the flesh. At the same time, the path planning algorithms of robotic arms are mostly based on simple distance calculations, which do not fully consider the complexity of the distribution of tree branches and leaves, resulting in poor obstacle avoidance. During the movement, they are prone to collisions with branches and fruits, further aggravating fruit damage.

[0007] Low level of intelligence in operation: Existing equipment lacks a dynamic perception and correction mechanism for operating environment parameters. The detection accuracy of sensors is easily affected by environmental interference. Moreover, the picking parameters are mostly fixed settings and cannot be optimized in real time according to the fruit growth status and tree structure differences. As a result, the equipment is not adaptable to different planting scenarios and plum trees of different ages, making it difficult to meet the needs of large-scale and refined picking.

[0008] Therefore, this application aims to develop a plum fruit picking robot and its picking method based on multi-source data fusion, which can adapt to complex terrain, accurately identify target fruits, and perform flexible and low-damage picking. Summary of the Invention

[0009] One objective of this invention is to propose a plum fruit harvesting robot and its harvesting method based on multi-source data fusion. This invention can adapt to complex terrains such as mountains and hills commonly found in plum cultivation. Through the collaborative fusion of multi-source sensing data, it achieves accurate identification and positioning of mature fruits, branches and leaves, and fruits affected by pests and diseases. Relying on a flexible harvesting structure and intelligent path planning mechanism, it reduces fruit harvesting damage. At the same time, it continuously adapts to different operating scenarios and tree growth states through the dynamic optimization capability of algorithms, significantly improving the intelligence level, operating efficiency, and harvest quality of plum harvesting. It effectively solves the technical problems of high labor intensity and high safety risks in traditional manual harvesting, as well as the poor environmental adaptability, insufficient recognition accuracy, and high damage rate of existing harvesting equipment.

[0010] According to an embodiment of the present invention, a plum fruit picking robot based on multi-source data fusion includes a tracked adaptive chassis, a multi-degree-of-freedom robotic arm, a multi-source sensing module, a biomimetic flexible end effector, an intelligent control unit, and a fruit collection box.

[0011] The multi-degree-of-freedom robotic arm includes a rotating base fixedly installed on the front side of the top of the frame. The rotating base is provided with a large arm, a connecting arm, and a telescopic forearm. The bionic flexible end effector includes a mounting seat fixedly installed on the front end of the telescopic forearm. The mounting seat is equipped with a gripper. The fruit collection box is fixedly installed on the top of the frame.

[0012] The tracked adaptive chassis, multi-degree-of-freedom robotic arm, multi-source sensing module, biomimetic flexible end effector, and fruit collection box are all electrically connected to the intelligent control unit.

[0013] The multi-source sensing module includes a sensing component consisting of an RGB-D camera, a polarization camera, an acoustic sensor, and a hyperspectral sensor integrated at the end of the retractable forearm for synchronously collecting multi-dimensional data of plum trees and fruits; a flexible tactile sensor set inside the gripper; and an environmental sensor set at the front end of the frame.

[0014] The intelligent control unit is installed inside the workshop. The control unit has a built-in data processing module and a motion control module. The data processing module runs a multi-source data fusion algorithm to analyze the sensed data and output target recognition results and picking parameters. The motion control module outputs control commands for the multi-degree-of-freedom robotic arm and the bionic flexible end effector based on the above results.

[0015] Furthermore, the adaptive chassis includes a frame and track wheels symmetrically arranged on both sides of the frame, and has a built-in hydraulic leveling component and an inertial measurement unit. The track wheels have biomimetic anti-slip ridges on their surfaces. The hydraulic leveling component includes at least three sets of hydraulic outriggers. The inertial measurement unit collects chassis tilt data in real time and transmits it to the intelligent control unit. The intelligent control unit controls the extension and retraction of the hydraulic outriggers to achieve chassis leveling adjustment. The frame is equipped with a motor unit for driving the track wheels.

[0016] Furthermore, the multi-degree-of-freedom robotic arm includes multiple motion joints, namely a rotating base rotation joint, a connecting arm pitch joint, and a telescopic forearm pitch joint. The telescopic forearm can freely extend and retract its length, and the biomimetic flexible end effector is connected to the telescopic forearm via a rotation joint.

[0017] Furthermore, in the multi-source sensing module, the RGB-D camera acquires the three-dimensional coordinates and surface texture data of the fruit, the polarization camera distinguishes the fruit from branches and leaves by the difference in polarization degree, the hyperspectral sensor collects the spectral information of the fruit's characteristic bands, the flexible tactile sensor collects the contact force data between the end effector and the fruit, the acoustic sensor collects the vibration and breakage sound signals of the fruit stalk during the picking process, and the environmental sensor collects the temperature, humidity and light intensity data of the working environment.

[0018] Furthermore, the biomimetic flexible end effector has a three-finger arc-shaped structure, made of flexible silicone material, with biomimetic protrusions on the inner side of the fingertips. The end effector has a built-in drive component and a force feedback component. The drive component controls the opening and closing angle of the gripper, and the force feedback component is linked with the flexible tactile sensor to realize real-time monitoring and adjustment of the gripping force.

[0019] A plum harvesting method based on a multi-source data fusion-based plum harvesting robot includes the following steps:

[0020] S1. Robot deployment and initialization: Move the robot to the work area. The chassis inertial measurement unit and hydraulic leveling components work together to complete the horizontal adjustment. The multi-degree-of-freedom robotic arm and bionic flexible end effector are subjected to extension tests.

[0021] S2. Multi-source data acquisition and target recognition: The multi-degree-of-freedom robotic arm drives the sensing components to scan the tree, simultaneously acquiring visual, spectral and acoustic data. Meanwhile, the environmental sensor can collect environmental data, which, after preprocessing, is input into the multi-source data fusion algorithm to output the spatial coordinates and growth status information of the harvestable fruits.

[0022] S3. Harvesting path planning: Based on the spatial coordinates of the fruit and the information of tree obstacles, an improved path planning algorithm is used to generate the motion trajectory of the robotic arm.

[0023] S4. Flexible picking execution: The multi-degree-of-freedom robotic arm moves to the target position according to the planned trajectory, and the bionic flexible end effector performs the grasping and fruit stem separation action. During the process, tactile and acoustic data are collected in real time and fed back to the intelligent control unit.

[0024] S5. Fruit collection and parameter optimization: After harvesting, the robotic arm with degrees of freedom transfers the fruit to the fruit collection box. The intelligent control unit records the harvesting parameters and updates the algorithm model. Step SS is repeated until the job is completed.

[0025] Furthermore, the data preprocessing in step S2 can use Gaussian filtering to remove noise from visual data, remove branch and leaf point clouds using the RANSAC algorithm, use standard normal variable transformation to eliminate baseline drift and extract feature band values ​​from spectral data, use wavelet thresholding to reduce noise from acoustic and tactile data, and use moving average filtering to smooth fluctuations from environmental data.

[0026] Furthermore, the multi-source data fusion algorithm in step S2 is a hierarchical weighted attention fusion algorithm, which includes two core steps: feature weight allocation and decision fusion. The feature weight allocation is implemented using an attention mechanism, and the weight calculation formula is as follows:

[0027]

[0028] in, The final attention weights for the i-th type of perceptual data are... The basic correlation score between the i-th type of perceived data and the ripeness of plums is given, with a value range of [0,1]. The higher the correlation, the closer the score is to 1. K is the environmental correction coefficient for the i-th type of sensing data, dynamically adjusted by the light, temperature, and humidity data collected by the environmental sensor, with a value range of [0.8, 1.2]. K decreases when environmental interference is high and increases when environmental interference is low. To represent the total number of types of perceived data. =5, It is a natural exponential function;

[0029] The decision fusion process employs an improved DS evidence theory, combining the weights of each data point to calculate the overall trust level of the decision proposition. When the trust level of the mature and harvestable proposition is higher than that of other propositions and the difference is ≥0.3, it is determined to be the target fruit.

[0030] Furthermore, the improved path planning algorithm in step S3 is an AI algorithm that introduces a flexible obstacle avoidance coefficient, and the path cost function is defined as:

[0031]

[0032] in, The total cost of the path. The cost is the actual distance from the current node to the starting node. The estimated distance cost from the current node to the target node is calculated using Euclidean distance. This is the flexible obstacle avoidance coefficient, with a value range of [0.5, 1.5]. It is applied when the distance between the path and the obstacle is greater than 5cm. Take 0.5, when the distance is ≤5cm Take 1.5, The obstacle cost is inversely proportional to the distance from the node to the obstacle; the closer the distance, the larger the value of O.

[0033] Furthermore, the flexible harvesting execution step S4 includes:

[0034] S. Pre-positioning: The multi-degree-of-freedom robotic arm drives the bionic flexible end effector to move to the side of the fruit, and adjusts the opening and closing angle of the gripper based on visual data;

[0035] S42. Flexible gripping: The gripper closes slowly and stops when the force detected by the tactile sensor reaches the threshold output by the fusion algorithm.

[0036] S43. Fruit stalk separation: The wrist joint drives the gripper to apply a torsional force along the growth direction of the fruit stalk. The acoustic sensor collects the sound signal in real time. When the signal frequency is in the characteristic frequency band of mature fruit stalk breakage, the separation is determined to be successful.

[0037] The parameter optimization in step S5 is achieved through a reinforcement learning model, using fruit damage as a reward signal to update the feature weights of the fusion algorithm.

[0038] The beneficial effects of this invention are:

[0039] 1. This invention is based on a five-dimensional perception architecture of vision, spectrum, touch, acoustics and environment. It dynamically allocates the weights of each source data through a hierarchical weighted attention fusion algorithm, uses the polarization degree difference of a polarization camera to distinguish fruits from branches and leaves, captures the characteristic bands of maturity by a hyperspectral sensor, and combines environmental parameters to correct the reliability of the data. This solves the problem that the single vision of the existing technology is easily interfered with by light, and achieves accurate differentiation between mature fruits and pests, branches and leaves, avoiding misharvesting and missed harvesting.

[0040] 2. The flexible structure of the end effector in this invention is adapted to the shape of plum fruit. Combined with the force feedback and real-time perception linkage mechanism, it realizes flexible control of grasping force and separation action during the picking process. At the same time, combined with optimized path planning logic, it can efficiently avoid tree obstacles and significantly reduce mechanical damage during fruit picking.

[0041] 3. In this invention, the adaptive chassis posture adjustment function ensures stable operation of the robot in uneven working environments. At the same time, it relies on reinforcement learning model to record harvesting process data and dynamically update algorithm parameters, enabling the robot to continuously adapt to different plum tree growth states and changes in the working environment, thereby improving the adaptability and harvesting efficiency of long-term operations. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a schematic diagram of the overall structure of the plum fruit picking robot based on multi-source data fusion proposed in this invention.

[0044] Figure 2 This is a schematic diagram of the internal structure of the frame of the plum fruit picking robot based on multi-source data fusion proposed in this invention.

[0045] Figure 3 This is a schematic diagram of the biomimetic flexible end effector structure of the plum fruit picking robot based on multi-source data fusion proposed in this invention.

[0046] Figure 4 This is a flowchart illustrating the harvesting method of the plum fruit harvesting robot based on multi-source data fusion proposed in this invention.

[0047] In the diagram: 1. Tracked adaptive chassis; 11. Frame; 12. Tracked wheels; 13. Fruit collection box; 14. Motor unit; 2. Multi-degree-of-freedom robotic arm; 21. Rotating base; 22. Main arm; 23. Connecting arm; 24. Telescopic forearm; 3. Bionic flexible end effector; 31. Mounting base; 32. Gripper; 4. Multi-source sensing module; 41. Sensing components; 42. Environmental sensor; 43. Flexible tactile sensor; 5. Intelligent control unit. Detailed Implementation

[0048] To make the technical means and objectives and effects of the present invention easier to understand, the embodiments of the present invention will be described in detail below with reference to specific illustrations.

[0049] The following is in conjunction with the appendix Figures 1 to 4 The specific embodiments of the present invention will be described in further detail. These embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0050] like Figure 1 As shown, the plum fruit harvesting robot based on multi-source data fusion disclosed in this invention achieves low-damage harvesting of plums in complex terrain through the collaborative processing of multi-source sensing data and the precise coordination of various execution components. The connection relationship and working logic of its components are as follows:

[0051] The tracked adaptive chassis 1 serves as the foundation for the robot's load-bearing and movement; its structure is as follows: Figure 1 and Figure 2 As shown, the vehicle includes a frame 11 and track wheels 12 symmetrically arranged on both sides of the frame 11. The frame 11 integrates a motor unit 14, a hydraulic leveling assembly and an inertial measurement unit. The intelligent control unit 5 is fixedly installed in the mounting cavity inside the frame 11, forming a compact layout with the various components.

[0052] The biomimetic anti-slip texture on the surface of the track wheel 12 adopts the concave-convex texture design of lizard's paw, which can enhance the fit with the muddy mountain road surface. The motor unit 14 drives the track wheel 12 to rotate through the reduction mechanism, providing stable power for the robot.

[0053] Once the robot moves to the work position next to the plum tree, the inertial measurement unit immediately starts and continuously collects the tilt angle data of the chassis. This data is transmitted to the intelligent control unit 5 in real time. The motion control module in the control unit compares the data with the preset level threshold. If the chassis tilt is detected to be outside the allowable range, it sends a telescopic command to the hydraulic leveling component. The three sets of hydraulic outriggers correspond to the front sides and rear center of the frame 11, respectively. The attitude of the frame 11 is adjusted by differential telescopic adjustment until the data fed back by the inertial measurement unit meets the level requirements, ensuring that the subsequent operation of the robotic arm is not affected by the terrain tilt.

[0054] The multi-degree-of-freedom robotic arm 2 is installed on the front top of the frame 11 and is the core component for achieving the coverage of the picking range. Its structure consists of a rotating base 21, a large arm 22, a connecting arm 23 and a telescopic small arm 24 connected in sequence. The components are connected by rotating joints to form multiple degrees of freedom of movement.

[0055] The rotating base 21 is fixed to the mounting platform of the frame 11 by bolts. The internal drive motor can drive the entire robotic arm to rotate 360° to meet the picking needs in different directions. One end of the upper arm 22 is hinged to the rotating base 21, and the other end is connected to the telescopic lower arm 24 through the connecting arm 23. The connecting joints of the connecting arm 23, the upper arm 22, and the telescopic lower arm 24 are equipped with angle sensors to provide real-time feedback on the joint rotation angle. The telescopic lower arm 24 adopts a nested structure and achieves free adjustment of the arm length through the built-in screw transmission mechanism. Its end is connected to the bionic flexible end effector 3 through a rotating joint. The rotating joint integrates a micro inertial measurement unit, which can capture the spatial attitude data of the end effector in real time and avoid unexpected collisions between the end effector and fruits and branches during the picking process.

[0056] When not in operation, the retractable forearm 24 is retracted to its shortest state, and the connecting arm 23 and the upper arm 22 are folded and fitted together, greatly reducing the overall size of the robot and making it easier to move in the narrow passage of the plum forest.

[0057] The multi-source sensing module 4 adopts a distributed layout and centralized acquisition design principle to ensure the comprehensiveness and synchronization of data acquisition.

[0058] The sensing component 41, consisting of an RGB-D camera, a polarization camera, an acoustic sensor, and a hyperspectral sensor, is integrated and fixed to the outer end of the telescopic forearm 24 via a bracket, maintaining a suitable distance from the bionic flexible end effector 3. This ensures that the picking action is not affected and that the sensor can accurately target the fruit area. The flexible tactile sensor 43 is fixed to the inner contact surface of the gripper 32 by adhesive bonding. It has three collection points, each corresponding to the middle inner part of the three-finger gripper 32, to ensure that the contact force can be fully sensed during grasping. The environmental sensor 42 is installed at the front end of the frame 11, at the same height as the middle and lower branches of the plum tree, and can collect real-time temperature, humidity, and light intensity data of the working area.

[0059] All sensor data lines converge to the intelligent control unit 5 through wiring holes inside the robotic arm. The data processing module within the control unit uniformly controls the data acquisition timing, enabling synchronous acquisition of multi-source data.

[0060] like Figure 3 As shown, the biomimetic flexible end effector 3 includes a mounting base 31 and a three-finger arc-shaped gripper 32. The mounting base 31 is fixedly connected to the front end of the telescopic arm 24 via a flange, and integrates a drive component and a force feedback component inside.

[0061] The drive assembly consists of three micro servos, each controlling the opening and closing of one of the three grippers 32. The output shafts of the servos are connected to the grippers 32 via a linkage mechanism, allowing for adjustment of the gripper 32's opening and closing angle from 0° to 90°. The grippers 32 are integrally molded from flexible silicone material, with an arc-shaped structure that matches the spherical contour of the plum fruit. The biomimetic protrusions on the inner side increase friction during gripping while preventing damage to the fruit peel from rigid contact. The force feedback assembly is directly linked to the flexible tactile sensor 43 via a signal line. When the tactile sensor detects that the contact force reaches a preset threshold, the force feedback assembly immediately sends a stop command to the drive assembly to prevent the grippers 32 from closing excessively and causing damage to the fruit.

[0062] The fruit collection box 13 is fixed to the rear of the top of the frame 11 by a snap-fit ​​structure. The inside of the box is lined with a flexible sponge cushioning layer to receive the picked plums and reduce the impact damage during the fall. The intelligent control unit 5, as the brain of the robot, has a built-in data processing module and motion control module that interact with each other through an internal bus. The data processing module is responsible for running a multi-source data fusion algorithm to preprocess and fuse the raw data collected by each sensor. The motion control module generates control commands for the movement of the robotic arm joints and the action of the end effector based on the analysis results.

[0063] like Figure 4 As shown, the specific steps of the plum picking method based on the above robot structure are as follows:

[0064] S1. Robot Deployment and Initialization: The operator controls the robot to move to the target plum tree through the remote controller. After starting the initialization program, the inertial measurement unit and hydraulic leveling component of the chassis automatically cooperate to complete the horizontal adjustment. At the same time, the multi-degree-of-freedom robotic arm 2 slowly unfolds from the folded state, and each joint moves to the initial position in sequence. The bionic flexible end effector 3 completes an opening and closing action test to ensure that each component operates normally.

[0065] S2. Multi-source data acquisition and target recognition: The intelligent control unit 5 controls the robotic arm to drive the sensing component 41 to perform a spiral scan along the canopy contour of the plum tree. During the scan, the sensing component 41 simultaneously acquires visual data, spectral data, and acoustic background data of the tree, while the environmental sensor 42 continuously acquires environmental parameters. The acquired visual data is first filtered by Gaussian to remove noise, and then the RANSAC algorithm is used to remove the point cloud of branches and leaves, retaining the three-dimensional coordinates and texture information of the candidate fruit area. The polarization camera further distinguishes between fruits and branches and leaves by analyzing the difference in polarization degree. The spectral data is transformed by standard normal variables to eliminate baseline drift, and the characteristic band values ​​related to the maturity of plums are extracted. The acoustic data and tactile data are processed by wavelet thresholding for noise reduction. The environmental data is smoothed by moving average filtering.

[0066] The preprocessed multi-source data is input into a hierarchical weighted attention fusion algorithm. First, the weights of each data point are calculated using an attention mechanism. The weight calculation formula is as follows:

[0067]

[0068] in, The final attention weights for the i-th type of perceptual data are... The basic correlation score between the i-th type of perceived data and the ripeness of plums is given, with a value range of [0,1]. The higher the correlation, the closer the score is to 1. K is the environmental correction coefficient for the i-th type of sensing data, dynamically adjusted by the light, temperature, and humidity data collected by the environmental sensor 42, with a value range of [0.8, 1.2]. K decreases when environmental interference is high and increases when environmental interference is low. To represent the total number of types of perceived data. =5, It is a natural exponential function;

[0069] in The values ​​are preset based on the correlation between different sensors and maturity, and the spectral data... The highest value, The value is dynamically adjusted by the light intensity collected by the environmental sensor 42, such as visual data under strong light conditions. The value decreases, and the spectral data The value increased.

[0070] If the fused decision result determines that the fruit is ripe and ready for harvest, then the spatial coordinates and growth status information of the fruit will be output.

[0071] S3. Harvesting Path Planning: The motion control module generates the robotic arm's motion trajectory based on the spatial coordinates of the fruit and the obstacle point cloud data of the tree, using a path planning algorithm that incorporates a flexible obstacle avoidance coefficient.

[0072] The cost function of this algorithm is defined as:

[0073]

[0074] in, The total cost of the path. The cost is the actual distance from the current node to the starting node. The estimated distance cost from the current node to the target node is calculated using Euclidean distance. This is the flexible obstacle avoidance coefficient, with a value range of [0.5, 1.5]. It is applied when the distance between the path and the obstacle is greater than 5cm. Take 0.5, when the distance is ≤5cm Take 1.5, The obstacle cost is inversely proportional to the distance from the node to the obstacle; the closer the distance, the larger the value of O.

[0075] This cost function ensures that the robotic arm optimizes its movement path while avoiding obstacles such as branches and leaves.

[0076] S4. Flexible harvesting is implemented, which consists of three sub-steps:

[0077] S41, Pre-positioning stage: The robotic arm moves the end effector to the side of the fruit according to the planned trajectory, and adjusts the opening and closing angle of the gripper 32 based on visual data to keep the gripper 32 and the fruit in a suitable alignment relationship.

[0078] S42, during the flexible gripping stage, the drive component controls the gripper 32 to close slowly. When the flexible tactile sensor 43 detects that the contact force reaches the threshold output by the fusion algorithm, the force feedback component immediately controls the gripper 32 to stop closing.

[0079] S43, during the fruit stalk separation stage, the rotating joint at the end of the telescopic forearm 24 drives the gripper 32 to slowly apply torsional force along the growth direction of the fruit stalk. The acoustic sensor collects the vibration and fracture sound signals of the fruit stalk in real time. When the signal frequency is in the fracture characteristic frequency band of the mature fruit stalk, the data processing module determines that the fruit stalk separation is successful, and the robotic arm drives the fruit away from the branch area.

[0080] S5. Fruit collection and parameter optimization: The robotic arm moves the gripper 32, which is holding fruit, to the top of the fruit collection box 13. The drive component controls the gripper 32 to open, and the fruit falls into the buffer layer inside the collection box.

[0081] Meanwhile, the intelligent control unit 5 records the grasping force threshold, torsional force magnitude, and fruit damage during this harvest. This data is then used as a reward signal to input into the reinforcement learning model, which automatically updates the feature weight parameters in the multi-source data fusion algorithm, making the parameter settings for subsequent harvests more accurate.

[0082] After completing one harvest, the robotic arm returns to the scanning start position and repeats steps S2 to S5 until all the harvestable fruits on the current plum tree are harvested. Then the robot moves to the next target tree and continues the cycle.

[0083] In this embodiment, the robot's components work together to achieve accurate identification and status judgment of plum fruits through multi-source data fusion. Combined with flexible execution structure and dynamic parameter adjustment, the fruit damage rate is effectively reduced. At the same time, the adaptive chassis ensures the stability of operation in complex terrain, greatly improving the efficiency and quality of plum picking.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A plum fruit harvesting robot based on multi-source data fusion, characterized in that, It includes a tracked adaptive chassis (1), a multi-degree-of-freedom robotic arm (2), a multi-source sensing module (4), a bionic flexible end effector (3), an intelligent control unit (5), and a fruit collection box (13). The multi-degree-of-freedom robotic arm (2) includes a rotating base (21) fixedly installed on the front side of the top of the frame (11). The rotating base (21) is provided with a large arm (22), a connecting arm (23) and a telescopic small arm (24). The bionic flexible end effector (3) includes a mounting seat (31) fixedly installed on the front end of the telescopic small arm (24). The mounting seat (31) is equipped with a gripper (32). The fruit collection box (13) is fixedly installed on the top of the frame (11). The tracked adaptive chassis (1), multi-degree-of-freedom robotic arm (2), multi-source sensing module (4), bionic flexible end effector (3) and fruit collection box (13) are all electrically connected to the intelligent control unit (5); The multi-source sensing module (4) includes a sensing component (41) consisting of an RGB-D camera, a polarization camera, an acoustic sensor and a hyperspectral sensor integrated at the end of the telescopic arm (24) for synchronously collecting multi-dimensional data of plum trees and fruits, a flexible tactile sensor (43) set inside the gripper (31), and an environmental sensor (42) set at the front end of the frame (11). The intelligent control unit (5) is installed inside the workshop (1). The control unit (5) has a built-in data processing module and a motion control module. The data processing module runs a multi-source data fusion algorithm to analyze the perceived data and output the target recognition result and picking parameters. The motion control module outputs control commands for the multi-degree-of-freedom robotic arm (2) and the bionic flexible end effector (3) based on the above results.

2. The plum fruit harvesting robot based on multi-source data fusion according to claim 1, characterized in that, The adaptive chassis (1) includes a frame (11) and track wheels (12) symmetrically arranged on both sides of the frame (11), and has a built-in hydraulic leveling component and an inertial measurement unit. The track wheels (12) have biomimetic anti-slip ridges on their surfaces. The hydraulic leveling component includes at least three sets of hydraulic outriggers. The inertial measurement unit collects chassis tilt data in real time and transmits it to the intelligent control unit (5). The intelligent control unit (5) controls the extension and retraction of the hydraulic outriggers to achieve chassis level adjustment. The frame (1) is equipped with a motor unit (14) for driving the track wheels (12).

3. The plum fruit harvesting robot based on multi-source data fusion according to claim 1, characterized in that, The multi-degree-of-freedom robotic arm (2) includes multiple motion joints, namely a rotating base (21) rotation joint, a connecting arm (23) pitch joint, and a telescopic forearm (24) pitch joint. The telescopic forearm (24) can freely extend and retract its arm length. The bionic flexible end effector (3) is connected to the telescopic forearm (24) through a rotation joint.

4. The plum fruit harvesting robot based on multi-source data fusion according to claim 1, characterized in that, In the multi-source sensing module (4), the RGB-D camera acquires the three-dimensional coordinates and surface texture data of the fruit, the polarization camera distinguishes the fruit from the branches and leaves by the difference in polarization degree, the hyperspectral sensor collects the spectral information of the fruit's characteristic band, the flexible tactile sensor (43) collects the contact force data between the end effector and the fruit, the acoustic sensor collects the vibration and breakage sound signal of the fruit stalk during the picking process, and the environmental sensor (42) collects the temperature, humidity and light intensity data of the working environment.

5. The plum fruit harvesting robot based on multi-source data fusion according to claim 1, characterized in that, The biomimetic flexible end effector (3) is a three-finger arc structure made of flexible silicone material. The inner side of the fingertip is provided with biomimetic protrusions. The end effector has a built-in drive component and force feedback component. The drive component controls the opening and closing angle of the gripper, and the force feedback component is linked with the flexible tactile sensor to realize real-time monitoring and adjustment of the gripping force.

6. A plum harvesting method based on multi-source data fusion using a plum fruit harvesting robot, characterized in that, The plum fruit harvesting robot based on multi-source data fusion according to any one of claims 1-5 includes the following steps: S1. Robot deployment and initialization: Move the robot to the work area. The chassis inertial measurement unit and hydraulic leveling component work together to complete the horizontal adjustment. The multi-degree-of-freedom robotic arm (2) and the bionic flexible end effector (3) are subjected to extension tests. S2, Multi-source data acquisition and target recognition: The multi-degree-of-freedom robotic arm (2) drives the sensing component (41) to scan the tree and simultaneously collect visual, spectral and acoustic data. The environmental sensor (42) can collect environmental data, which is then preprocessed and input into the multi-source data fusion algorithm to output the spatial coordinates and growth status information of the harvestable fruit. S3. Harvesting path planning: Based on the spatial coordinates of the fruit and the information of tree obstacles, an improved path planning algorithm is used to generate the motion trajectory of the robotic arm. S4. Flexible picking execution: The multi-degree-of-freedom robotic arm (2) moves to the target position according to the planned trajectory, and the bionic flexible end effector (3) performs the grasping and fruit stem separation action. During the process, tactile and acoustic data are collected in real time and fed back to the intelligent control unit (5). S5. Fruit collection and parameter optimization: After the harvest is completed, the free-degree robotic arm (2) transfers the fruit to the fruit collection box (13). The intelligent control unit (5) records the harvest parameters and updates the algorithm model. Repeat steps S2-S5 until the job is completed.

7. The plum harvesting method based on multi-source data fusion using a plum fruit harvesting robot according to claim 6, characterized in that, The data preprocessing in step S2 can be performed as follows: Gaussian filtering can be used to remove noise from visual data, and branch and leaf point clouds can be removed by using the RANSAC algorithm; standard normal variable transformation can be used to eliminate baseline drift and extract feature band values ​​from spectral data; wavelet thresholding can be used to reduce noise from acoustic and tactile data; and moving average filtering can be used to smooth fluctuations in environmental data.

8. The plum harvesting method based on multi-source data fusion using a plum fruit harvesting robot according to claim 6, characterized in that, The multi-source data fusion algorithm in step S2 is a hierarchical weighted attention fusion algorithm, which includes two core steps: feature weight allocation and decision fusion. The feature weight allocation is implemented using an attention mechanism, and the weight calculation formula is as follows: in, The final attention weights for the i-th type of perceptual data are... The basic correlation score between the i-th type of perceived data and the ripeness of plums is given, with a value range of [0,1]. The higher the correlation, the closer the score is to 1. K is the environmental correction coefficient for the i-th type of sensing data, dynamically adjusted by the light, temperature, and humidity data collected by the environmental sensor, with a value range of [0.8, 1.2]. K decreases when environmental interference is high and increases when environmental interference is low. To represent the total number of types of perceived data. =5, It is a natural exponential function; The decision fusion process employs an improved DS evidence theory, combining the weights of each data point to calculate the overall trust level of the decision proposition. When the trust level of the mature and harvestable proposition is higher than that of other propositions and the difference is ≥0.3, it is determined to be the target fruit.

9. The plum harvesting method of the plum fruit harvesting robot based on multi-source data fusion according to claim 6, characterized in that, The improved path planning algorithm in step S3 is an AI algorithm that introduces a flexible obstacle avoidance coefficient, and the path cost function is defined as: in, The total cost of the path. The cost is the actual distance from the current node to the starting node. The estimated distance cost from the current node to the target node is calculated using Euclidean distance. This is the flexible obstacle avoidance coefficient, with a value range of [0.5, 1.5]. It is applied when the distance between the path and the obstacle is greater than 5cm. Take 0.5, when the distance is ≤5cm Take 1.5, The obstacle cost is inversely proportional to the distance from the node to the obstacle; the closer the distance, the larger the value of O.

10. The plum harvesting method based on multi-source data fusion using a plum fruit harvesting robot according to claim 6, characterized in that, The flexible harvesting process in step S4 includes: S41, Pre-positioning, the multi-degree-of-freedom robotic arm (2) drives the bionic flexible end effector (3) to move to the side of the fruit, and adjusts the opening and closing angle of the gripper (32) based on visual data; S42, Flexible gripping, the gripper (32) slowly closes, and stops when the force value detected by the tactile sensor (43) reaches the threshold output by the fusion algorithm; S43. Fruit stalk separation: The wrist joint drives the gripper to apply a torsional force along the growth direction of the fruit stalk. The acoustic sensor collects the sound signal in real time. When the signal frequency is in the characteristic frequency band of mature fruit stalk breakage, the separation is determined to be successful. The parameter optimization in step S5 is achieved through a reinforcement learning model, using fruit damage as a reward signal to update the feature weights of the fusion algorithm.

Images