A fruit stem clamping and shearing agricultural robot and a picking control method thereof

Abstract

This invention provides a fruit stalk clamping and shearing agricultural robot and its harvesting control method. The robot includes a control cabinet, a robotic arm, and a visual navigation system mounted on a tracked chassis support platform. The control cabinet contains a host computer control system and a slave computer control system. A movable end effector with an identification and positioning system and a supplementary lighting device is installed at the end of the robotic arm. The movable end effector can flexibly adapt to the harvesting needs of different crops and complex working conditions. The host computer control system combines data collected by a depth camera and LiDAR, and uses the YOLOv8-MC model to identify target crops and their key points in the images, significantly improving the crop identification accuracy in complex environments, reducing the probability of missed or incorrect harvesting, and minimizing damage to the fruit. When constructing a global map, the host computer control system performs depth correction and map optimization for changes in illumination, further improving the robot's path planning and operational accuracy.

Description

Technical Field

[0001] This invention belongs to the field of agricultural harvesting equipment design technology, and particularly relates to an agricultural robot for clamping and shearing fruit stalks and its harvesting control method. Background Technology

[0002] Agricultural automation and intelligentization are important directions for the development of modern agriculture. Fruit and vegetable harvesting robots are of great significance in addressing labor shortages and improving harvesting efficiency. Currently, there are some robotic devices on the market for automated harvesting, but many suffer from limited applicability and insufficient recognition and harvesting accuracy. Therefore, researching an agricultural robot capable of clamping and shearing fruit stalks to meet the harvesting needs of various crops has become an important topic in current technological development.

[0003] Existing harvesting robots employ a design combining a robotic arm and an end effector. They acquire images through vision sensors and utilize algorithms to identify and harvest target crops. Typically, these robots rely on cameras and target detection algorithms for crop identification and harvesting using a simple gripping mechanism. However, in practical operations, this design is constrained by several factors, including: the efficiency and accuracy of identification and harvesting are significantly affected by environmental changes and foliage obstruction, easily leading to missed or incorrect harvests; most harvesting robots are designed for only one type of crop, lacking compatibility with different crop varieties, thus limiting their application scope; and existing robot path planning algorithms often employ a single-weight evaluation mechanism, making it difficult to make efficient decisions quickly in environments with a mix of dynamic changes and static obstacles.

[0004] To address the above problems, this invention presents a fruit stalk clamping and shearing agricultural robot and its harvesting control method. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a fruit stalk clamping and shearing agricultural robot and its harvesting control method. It overcomes the problems of insufficient recognition accuracy, limited applicability, low environmental adaptability, and insufficient path planning ability in existing technologies. By optimizing the visual recognition algorithm, enhancing multi-crop adaptability, improving the path planning strategy, and designing an intelligent end effector, the robot can efficiently and accurately complete the automatic harvesting tasks of various crops in complex working environments.

[0006] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0007] A fruit stalk clamping and shearing agricultural robot includes a tracked chassis with a support platform mounted on it. A control cabinet, a robotic arm, and a vision navigation system are mounted on the support platform. A collection frame is mounted above the control cabinet. The control cabinet contains a host computer control system and a slave computer control system. The host computer control system includes a visual display screen and an embedded industrial computer, while the slave computer control system includes a robotic arm controller. A movable end effector is mounted at the end of the robotic arm, and the movable end effector is equipped with an identification and positioning system and a supplementary lighting device. A power supply system, a drive module, and an air pump are also arranged within the tracked chassis.

[0008] Furthermore, the movable end effector is mounted on the end of the robotic arm via an end flange, which is connected to a movable adjustment slide rail. A movable adjustment slider is slidably mounted on the movable adjustment slide rail, and an electric actuator is mounted below the movable adjustment slider. Air ducts are mounted on both sides of the electric actuator and are connected to an air pump. A motion guide rail is mounted at the front end of the electric actuator, and an ultrasonic sensor is mounted below the motion guide rail. Two grippers are mounted at the front end of the motion guide rail, and force sensors, blades, and rubber pads are embedded on opposite sides of the two grippers. The movable adjustment slider slides to adjust its position on the movable adjustment slide rail, thereby indirectly adjusting the position of the two grippers to adapt to ridges with different planting spacings.

[0009] Furthermore, the visual navigation system includes a gimbal housing fixed on a support platform, a servo motor installed inside the gimbal housing, the power output end of the servo motor passing through the top of the gimbal housing and connected to the gimbal rotation base, and a second depth camera fixed on the gimbal rotation base.

[0010] Furthermore, the identification and positioning system includes a first depth camera mounted above the motion guide rail and a lidar mounted above the movable adjustment slider.

[0011] Furthermore, the supplementary lighting device is installed on the end flange and includes a light sensor. The light sensor detects the ambient light intensity and transmits the detection result to the embedded industrial control computer. The embedded industrial control computer adjusts the brightness of the supplementary light according to the detection result.

[0012] A method for harvesting multiple target crops using the above-mentioned fruit stalk clamping and shearing agricultural robot includes the following steps:

[0013] Step 1: First, use the first depth camera of the recognition and positioning system to acquire the original image of the target crop. Then, manually mark key points on the visualization display screen at a position 2-3 cm above the root of the fruit stalk: mark the highest point of the fruit stalk as P1, the midpoint of the fruit stalk as P2, and the root of the fruit stalk as P3; mark the lowest point of the central axis of the target crop passing through point P3 as P4; mark the left and right endpoints as P5 and P6 respectively; then use a bounding box to select and mark the target crop and its fruit stalk, with P3 and P4 reflecting the length of the target crop, and P5 and P6 reflecting the curvature of the target crop; create a dataset from the marked image data and input it into the YOLOv8-MC model in the embedded industrial control computer for training to obtain a key point recognition model for the target crop and its fruit stalk for subsequent actual detection and recognition.

[0014] Step 2: The second depth camera of the visual navigation system acquires the working environment information and transmits it to the embedded industrial computer. The embedded industrial computer uses a multimodal information fusion map building algorithm to build a global map.

[0015] Step 3: The embedded industrial computer sends instructions to the gimbal servo motor of the vision navigation system, which indirectly drives the second depth camera to rotate and updates the surrounding dynamic and static obstacle information in real time; the information collected by the second depth camera is transmitted to the embedded industrial computer for preliminary analysis, and then the information is added to the global map to perform local path planning and determine the robot's motion trajectory.

[0016] Step 4: The embedded industrial computer makes a decision and sends instructions to the drive module to drive the tracked chassis to perform motion control according to the linear velocity and angular velocity corresponding to the optimal trajectory, so as to reach the designated target crop picking area;

[0017] Step 5: The embedded industrial computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm. The movable end effector, recognition and positioning system, and supplementary lighting device move accordingly to reach the image acquisition pose.

[0018] Step 6: The first depth camera acquires images and depth image data of the target crop, and the LiDAR acquires 3D point cloud data. The image data and point cloud data are fused and packaged and then transmitted to the embedded industrial control computer. The embedded industrial control computer inputs the fused data into the multimodal information fusion and multi-crop recognition module, analyzes the geometric features of the target crop's fruit stalk, optimizes the gripping angle of the fingers on the movable end effector, and outputs the world coordinates of the picking point of the target crop at the fruit stalk.

[0019] Step 7: The embedded industrial computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm, driving the movable end effector to move so that the blade reaches the target crop picking point; when branches and leaves block the target crop or affect the movement of the movable end effector, the embedded industrial computer sends instructions to the air pump to blow away the branches and leaves on both sides of the target crop through the air duct; the embedded industrial computer calculates the precise force required for clamping based on the shape and material of the fruit stalk, and the electric actuator drives the two clamping fingers to move closer to each other along the motion guide rail; when the rubber pad is in contact with the surface of the fruit stalk, the force sensor monitors the clamping force, and the electric actuator adjusts the relative movement between the clamping fingers according to the real-time feedback to ensure that the fruit stalk is firmly clamped; then, the blade and the clamping fingers work together to cut the fruit stalk;

[0020] Step 8: During the clamping and cutting process, the ultrasonic sensor detects the distance between the target crops in real time to ensure proper clamping. If clamping failure is detected, the embedded industrial control computer will terminate the remaining actions through a feedback mechanism and repeat steps 5, 6, and 7 until the harvest is successful. After successful harvesting, the embedded industrial control computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm, driving the movable end effector to move the harvested crops into the collection box.

[0021] Step 9: After harvesting the target crops in the field of view of the first depth camera, repeat steps 2 to 8 to continue harvesting the target crops in other areas.

[0022] Furthermore, in step 2, the specific process of global map construction is as follows:

[0023] Step 2.1: The light sensor transmits the collected light intensity data to the embedded industrial computer. The embedded industrial computer analyzes the light intensity data and compensates for and corrects the affected depth value according to the compensation formula.

[0024]

[0025] in, For the current pixel Depth value at that location, The raw depth values ​​acquired by the second depth camera. Errors caused by changes in illumination; For light intensity data, and For calibration parameters;

[0026] Step 2.2: Divide the illumination effects into two categories: direct reflection interference and information loss in shadow areas, and design corresponding correction methods for each; infer the lost depth information in the shadow areas using the texture features of the RGB images; perform depth compensation through neighborhood depth point interpolation to update the point cloud data of the reflection interference area:

[0027]

[0028] in, The depth value after compensation. For pixels The surrounding neighborhood, for Pixels within, For pixels The original depth value at that location, The weighting coefficients are as follows:

[0029]

[0030] Step 2.3: Accumulate depth data from multiple frames to eliminate depth errors caused by changes in illumination within a single frame. The fusion formula is as follows:

[0031]

[0032] in, The depth value after weighted fusion. Here, T represents the depth value per frame, and T is the number of frames in the time series. Time weighting; For attenuation parameters;

[0033] Step 2.4: Update local point cloud frames:

[0034]

[0035]

[0036] in, For the updated depth value, , , To update the weighting factors;

[0037] Step 2.5: Project the updated local point cloud frames onto the 3D map, integrate and reconstruct them to generate a continuous global map.

[0038] Furthermore, in step 6, the multimodal information fusion multi-crop identification module includes an identification and localization processing unit, a depth registration processing unit, and a coordinate system transformation processing unit, which are respectively used to realize the identification of the target crop and its fruit stalk and the key point localization processing, depth registration processing, and coordinate system transformation processing. The specific data processing process of the identification and localization processing unit is as follows:

[0039] The identification and localization processing unit utilizes an improved YOLOv8 model, specifically the YOLOv8-MC model, to identify and locate key points of target crops and their fruit stems in the image. The improvements to the YOLOv8 model include adding an attention mechanism module at the end of the feature extraction network and adding a BiFPN module before each convolutional layer in the feature fusion network, as detailed below:

[0040] An attention mechanism module is added at the end of the feature extraction network to integrate the extracted features in both channel and spatial dimensions, focusing on the phenotypic features of the target crop. The integration formula is as follows:

[0041]

[0042]

[0043]

[0044]

[0045] in, This is a one-dimensional channel attention feature map; The sigmoid activation function is used; SVM is the support vector machine classification algorithm. , These represent the average pooling layer and the max pooling layer, respectively; F is the input feature map. , These are channel attention feature maps obtained after passing through the average pooling layer and the max pooling layer, respectively. , Let C represent tensors of size (C / r, C) and (C, C / r), respectively, where C is the number of channels and r is the scaling factor. , These are spatial attention feature maps obtained after passing through the average pooling layer and the max pooling layer, respectively. , All are two-dimensional spatial attention feature maps; The kernel size is 7×7; For element-wise multiplication; F1 is the output after spatial attention; F2 is the final output feature map;

[0046] A BiFPN module is added before each convolutional layer in the feature fusion network; an adaptive weight allocation mechanism is introduced to dynamically allocate weights according to the importance of crop features, enabling information flow between feature maps of different scales and performing feature fusion; the fused features are input into the output network, non-maximum suppression is performed, and the final target bounding box and key points of the target crop are output and mapped onto the original image.

[0047] Furthermore, the depth registration processing unit registers the image marked with the target box and key points with the depth image and the 3D point cloud data, extracts the pixel coordinates of the key points P1 and P2 of the target crop fruit stalk, and inputs them into the depth image and the 3D point cloud data to search for matching the depth values ​​of key points P1 and P2. If the difference between the two depth values ​​is less than a fixed threshold, the positioning is accurate; otherwise, the identification and positioning are re-performed.

[0048] The coordinate system transformation processing unit combines pixel coordinates and depth values ​​to transform the pixel coordinates of key point P2 to coordinates in the camera coordinate system. Then, by combining the transformation relationship between the camera coordinate system and the world coordinate system and the position of the front blade of the movable end effector, the three-dimensional coordinate values ​​are transformed, thus obtaining the final picking point coordinates.

[0049] The present invention has the following beneficial effects:

[0050] The fruit stalk clamping and shearing agricultural robot provided by this invention, especially the design of the mobile end effector, can flexibly adapt to the harvesting needs of different crops and complex working conditions. It is suitable for harvesting various crops and has significant advantages in improving the working efficiency of harvesting robots, adapting to complex working environments, and expanding the scope of applications.

[0051] The multimodal information fusion and multi-crop recognition module designed in this invention, combined with depth cameras and LiDAR, significantly improves the accuracy of crop recognition by robots in complex environments. This precise recognition and harvesting technology reduces the chance of missed or incorrect harvesting, minimizes damage to fruits, and increases the market value of crops.

[0052] The path planning and obstacle avoidance algorithm designed in this invention selects the optimal path by evaluating dynamic and static obstacles in the working environment in real time, thus avoiding the situation where traditional robots get stuck in local minima and cannot reach the target point in complex environments. This innovation can significantly improve the robot's adaptability in complex working environments, reduce the risk of work interruption or damage caused by improper path planning, and thus ensure long-term, efficient and stable operation. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall structure of the fruit stalk clamping and shearing agricultural robot described in this invention;

[0054] Figure 2 This is a schematic diagram of the side structure of the fruit stalk clamping and shearing agricultural robot of the present invention;

[0055] Figure 3 This is a schematic diagram of the rear structure of the fruit stalk clamping and shearing agricultural robot described in this invention;

[0056] Figure 4 This is a schematic diagram of the execution device structure described in this invention;

[0057] Figure 5 This is a schematic diagram of the movable end effector structure described in this invention;

[0058] Figure 6 This is a schematic diagram of the electric gimbal structure described in this invention;

[0059] Figure 7 This is a schematic diagram of the DAV_DWA local path planning algorithm described in this invention;

[0060] Figure 8 This is a schematic diagram of the YOLOv8-MC model described in this invention.

[0061] In the diagram: 1-tracked chassis; 2-mobile end effector; 201-first gripper finger; 202-second gripper finger; 203-first air duct; 204-second air duct; 205-moving adjustment slider; 206-moving adjustment slide rail; 207-end flange; 208-electric actuator; 209-ultrasonic sensor; 3-identification and positioning system; 301-first depth camera; 302-LiDAR; 4-lighting device; 401-lighting lamp; 402-light sensor; 5-robotic arm; 6-support platform; 7-visual navigation system; 701-second depth camera; 702-gimbal housing; 703-servo motor; 704-gimbal rotating base; 8-control cabinet; 801-visual display screen; 9-collection frame. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0063] like Figures 1 to 3 As shown, the fruit stalk clamping and shearing agricultural robot of the present invention includes a tracked chassis 1, a power supply system, a drive module, an air pump, a movable end effector 2, a recognition and positioning system 3, a supplementary lighting device 4, a robotic arm 5, a support platform 6, a visual navigation system 7, a control cabinet 8, and a collection frame 9. The upper computer control system is signal-connected to the drive module, air pump, movable end effector 2, recognition and positioning system 3, supplementary lighting device 4, lower computer control system, and visual navigation system 7. The upper computer control system receives data, performs data analysis, makes scientific decisions, and issues commands to control each module to perform corresponding actions. The power supply system, drive module, and air pump are all located within the tracked chassis 1. The power supply system acts as a battery, powering the entire robot. The drive module drives the motor of the tracked chassis 1, and the air pump supplies air to the air duct on the movable end effector 2.

[0064] like Figures 1 to 3As shown, a support platform 6 is mounted on the tracked chassis 1, and a control cabinet 8 and a robotic arm 5 are mounted on the support platform 6. A collection box 9 is mounted on the top of the control cabinet 8. The control cabinet 8 contains a host computer control system and a slave computer control system. The host computer control system includes a visual display screen 801 and an embedded industrial computer. The slave computer control system includes a robotic arm controller, which receives instructions from the embedded industrial computer to control the robotic arm 5 to rotate at corresponding angles along each axis.

[0065] like Figure 3 , 5 As shown, the movable end effector 2 includes a first gripper finger 201, a second gripper finger 202, a first air duct 203, a second air duct 204, a movable adjustment slider 205, a movable adjustment slide rail 206, an end flange 207, an electric actuator 208, and an ultrasonic sensor 209. The movable end effector 2 is mounted on the end of the robotic arm 5 via the end flange 207, which is connected to the movable adjustment slide rail 206. The movable adjustment slider 205 is slidably mounted on the movable adjustment slide rail 206, and the electric actuator 208 is mounted below the movable adjustment slider 205. The device 208 has a first air duct 203 and a second air duct 204 installed on both sides. A motion guide rail is installed at the front end of the electric actuator 208, and an ultrasonic sensor 209 is installed below the motion guide rail. A first gripper finger 201 and a second gripper finger 202 are installed at the front end of the motion guide rail. Force sensors, blades, and rubber pads are embedded on opposite sides of the first gripper finger 201 and the second gripper finger 202. The movable adjustment slider 205 can slide and adjust its position on the movable adjustment slide rail 206, thereby indirectly adjusting the positions of the first gripper finger 201 and the second gripper finger 202 to adapt to ridges with different planting spacing. This invention utilizes a lidar 302 in conjunction with a first depth camera 301 to achieve precise identification and positioning of the target crop, and uses the ultrasonic sensor 209 to determine whether harvesting has been successful. Furthermore, in agricultural environments, branches and leaves may obstruct the target crop, making harvesting difficult for the movable end effector 2. Therefore, this invention installs two air ducts on both sides of the electric actuator 208 to blow away obstructions on both sides of the target crop, thus exposing the target crop and greatly improving the harvesting success rate.

[0066] like Figure 1 , 2As shown in Figure 6, the visual navigation system 7 is installed at the front end of the support platform 6 and includes a second depth camera 701, a gimbal housing 702, a servo motor 703, and a gimbal rotation base 704. The gimbal housing 702 is fixed on the support platform 6, and the servo motor 703 is installed inside the gimbal housing 702. The power output end of the servo motor 703 passes through the top of the gimbal housing 702 and is connected to the gimbal rotation base 704. The second depth camera 701 is fixed on the gimbal rotation base 704. The servo motor 703 can drive the second depth camera 701 to rotate in order to scan the depth information of different areas and assist in subsequent path planning.

[0067] like Figure 4 , 5 As shown, the identification and positioning system 3 includes a first depth camera 301 and a lidar 302; the first depth camera 301 is installed above the motion guide rail of the movable end effector 2 for identifying target crops, and the lidar 302 is installed above the movable adjustment slider 205.

[0068] like Figure 4 , 5 As shown, the supplementary lighting device 4 is installed on the end flange 207 and includes a supplementary light 401 and a light sensor 402. The light sensor 402 detects the ambient light intensity and transmits the detection result to the embedded industrial control computer. The embedded industrial control computer automatically adjusts the brightness of the supplementary light 401 according to the detection result.

[0069] The harvesting method for various target crops using the fruit stalk clamping and shearing agricultural robot described in this invention includes the following steps:

[0070] Step 1: Before the robot performs the actual harvesting operation, the first depth camera 301 is used to acquire the original image of the target crop, and the image size is normalized. Then, key points are manually marked on the visualization display screen 801 at a position 2-3 cm above the root of the fruit stalk of the target crop: the highest point is marked as P1, the middle point as P2, and the root of the fruit stalk as P3; the lowest point of the target crop through the central axis of P3 is marked as P4; the left and right endpoints are marked as P5 and P6 respectively; then, a target bounding box is used to select and mark the target crop and its fruit stalk. P3 and P4 reflect the length of the target crop, and P5 and P6 reflect the curvature of the target crop, which can help determine the pose of the target crop; the marked image data is made into a dataset and input into the YOLOv8-MC model in the embedded industrial control computer for training to obtain the key point recognition model of the target crop and its fruit stalk for subsequent actual detection and recognition.

[0071] Step 2: The second depth camera 701 collects the working environment information and transmits it to the embedded industrial control computer. The working environment information includes RGB images and depth images. The embedded industrial control computer uses a multimodal information fusion map building algorithm to build a global map. In addition, in the facility agriculture environment, the lighting conditions change due to changes in light-transmitting film, reflective surface or natural light. Such changes in lighting can easily lead to distortion of depth camera data, especially the jump in depth value or error accumulation, which reduces the accuracy of the generated global map and thus affects the robot's path planning and operation accuracy. Therefore, when the embedded industrial control computer of this invention converts the environmental information collected by the second depth camera 701 into a global map, it performs depth correction and map optimization for lighting changes.

[0072] The specific process of constructing a global map is as follows:

[0073] Step 2.1: The light sensor 402 transmits the collected light intensity data to the embedded industrial computer. The embedded industrial computer analyzes the light intensity data and compensates and corrects the affected depth value according to the compensation formula.

[0074]

[0075] in, For the current pixel Depth value at that location, The raw depth values ​​acquired by the second depth camera 701 Errors caused by changes in illumination; For light intensity data, and For calibration parameters;

[0076] Step 2.2: Divide the illumination effects into two categories: direct reflection interference and information loss in shadow areas, and design corresponding correction methods for each. Infer the lost depth information in the shadow areas using the texture features of the RGB images. Perform depth compensation through neighborhood depth point interpolation to update the point cloud data of the reflection interference area, as shown in the following formula:

[0077]

[0078] in, The depth value after compensation. For pixels The surrounding neighborhood, for Pixels within, For pixels The original depth value at that location, The weighting coefficients are defined as follows:

[0079]

[0080] in, For attenuation parameters;

[0081] Step 2.3: Accumulate depth data from multiple frames to eliminate depth errors caused by changes in illumination within a single frame. The fusion formula is as follows:

[0082]

[0083] in, The depth value after weighted fusion. Here, T represents the depth value per frame, and T is the number of frames in the time series. Time weighting;

[0084] Step 2.4: Update the local point cloud frame, using the following formula:

[0085]

[0086]

[0087] in, For the updated depth value, , , To update the weighting factors;

[0088] Step 2.5: Project the updated local point cloud frames onto the 3D map, integrate and reconstruct them to generate a continuous global map.

[0089] Step 3: The embedded industrial control computer sends commands to the gimbal servo motor 703, controlling the gears on the gimbal servo motor 703 to rotate, driving the gimbal rotating base 704 to rotate. The second depth camera 701 on the gimbal rotating base 704 rotates accordingly, updating the surrounding dynamic and static obstacle information in real time. The information collected by the second depth camera 701 is transmitted to the embedded industrial control computer for preliminary analysis, and then the information is added to the global map. Local path planning is performed using the Dual obstacle cost function, Adaptive weights and Virtualtarget_Dynamic Window Approach (DAV_DWA) based on the dual obstacle cost function, adaptive weights and virtual target method to determine the robot's motion trajectory. Figure 7 As shown ( Figure 7 The horizontal and vertical coordinates in the graph have no practical meaning; they merely form a plane to represent the robot's position. The specific processing procedure is as follows:

[0090] Step 3.1: Apply different evaluation methods to dynamic / static obstacles; divide the distance between the robot and the obstacle into two components, including: the distance evaluation function with respect to dynamic obstacles. Distance evaluation function between the object and a static obstacle Where v represents the robot's linear velocity and w represents the robot's angular velocity; different safety thresholds are set for the two types of obstacles for differentiated processing, and they are incorporated into two evaluation functions respectively. The trajectory evaluation function is defined as:

[0091]

[0092] in, For trajectory evaluation function; Here is the azimuth evaluation function, which represents the angular deviation between the predicted orientation of the robot's trajectory endpoint and the target position at a set speed. , These are the distance evaluation functions from the robot trajectory to dynamic obstacles and static obstacles, respectively; This is the current robot linear velocity evaluation function; , , and These are the weighting coefficients; For normalization parameters;

[0093] Step 3.2: Adaptively adjust the weights of each evaluation function using a fuzzy logic algorithm; input the dynamic / static obstacle distances into the fuzzy logic algorithm and dynamically adjust them. , , and The optimal path is found using four weighted parameters, and the specific process is as follows:

[0094] The minimum distances between the robot and dynamic and static obstacles are acquired using a second depth camera 701, and these distances are then used as input variables for fuzzification. Z-shaped, Gaussian, and S-shaped membership functions are then used to evaluate the input variables, and the fuzzy controller dynamically adjusts the parameters. , , and Four weight parameters; by adjusting the weight coefficients, the robot can prioritize obstacle avoidance and reduce speed when encountering dynamic obstacles, while increasing speed and reducing obstacle avoidance priority when encountering static obstacles that are far away.

[0095] Step 3.3: Introduce virtual target points so that the robot can still escape even when it reaches a local minimum; the specific process of creating virtual target points is as follows:

[0096] The location where the robot's speed is below a predetermined threshold and the duration of this state exceeds a specific time is identified as a local minimum location; obstacles with a distance of less than or equal to 2p between adjacent obstacles are detected and added to a local minimum obstacle group, where p is the robot radius; the outermost obstacle in the obstacle group that is closest to the robot is designated as the target obstacle, and a virtual target point is set at twice the radius outside the line connecting the robot and the obstacle; when the robot reaches the virtual target point, the virtual target point is switched to the actual target point, thus completing the local path planning.

[0097] Step 4: The embedded industrial control computer sends the processed information to the visualization display screen 801 to visualize the operation process; the embedded industrial control computer makes a decision and sends instructions to the drive module to drive the tracked chassis 1 to perform motion control according to the linear velocity and angular velocity corresponding to the optimal trajectory, so as to reach the designated target crop picking area.

[0098] Step 5: The embedded industrial control computer sends instructions to the robotic arm controller, which controls the rotation of each axis of the robotic arm 5. The movable end effector 2, the identification and positioning system 3, and the supplementary lighting device 4 move accordingly to reach the image acquisition pose.

[0099] Step 6: The first depth camera 301 acquires target crop images and depth image data, and the lidar 302 acquires 3D point cloud data. The image data and point cloud data are fused and packaged, and then transmitted to the embedded industrial control computer for analysis. The embedded industrial control computer inputs the fused data into the multimodal information fusion multi-crop recognition module to analyze the geometric features of the target crop stalk, such as length, angle, and distance, optimizes the clamping angle of the first gripper finger 201 and the second gripper finger 202 on the movable end effector 2, and outputs the world coordinates of the harvesting point of the target crop at the stalk. Specifically, the multimodal information fusion multi-crop recognition module includes a recognition and positioning processing unit, a depth registration processing unit, and a coordinate system transformation processing unit, which are used to realize the recognition of the target crop and its stalk and key point positioning processing, depth registration processing, and coordinate system transformation processing, respectively. The specific process is as follows:

[0100] Step 6.1: The identification and localization processing unit utilizes an improved YOLOv8 model, specifically the YOLOv8-multi-crop (YOLOv8-MC) model, to identify and locate key points of target crops and their fruit stalks in the image. The improvements to the YOLOv8 model include adding an attention mechanism module at the end of the feature extraction network and adding a BiFPN module before each convolutional layer in the feature fusion network. Figure 8 As shown; the specific improvement methods are as follows:

[0101] An attention mechanism module is added at the end of the feature extraction network to integrate the extracted features in both channel and spatial dimensions, focusing on the phenotypic features of the target crop. The integration formula is as follows:

[0102]

[0103]

[0104]

[0105]

[0106] in, This is a one-dimensional channel attention feature map; The sigmoid activation function is used; SVM is the support vector machine classification algorithm. , These represent the average pooling layer and the max pooling layer, respectively; F is the input feature map. , These are channel attention feature maps obtained after passing through the average pooling layer and the max pooling layer, respectively. , Let C represent tensors of size (C / r, C) and (C, C / r), respectively, where C is the number of channels and r is the scaling factor. , These are spatial attention feature maps obtained after passing through the average pooling layer and the max pooling layer, respectively. , All are two-dimensional spatial attention feature maps; The kernel size is 7×7; For element-wise multiplication; F1 is the output after spatial attention; F2 is the final output feature map;

[0107] A BiFPN module is added before each convolutional layer in the feature fusion network; an adaptive weight allocation mechanism is introduced to dynamically allocate weights according to the importance of crop features, enabling information flow between feature maps of different scales and performing feature fusion; the fused features are input into and output to the network, non-maximum suppression is performed, and the final target bounding box and key points of the target crop are output and mapped onto the original image;

[0108] Step 6.2: The depth registration processing unit registers the image marked with the target bounding box and key points with the depth image and 3D point cloud data; it extracts the pixel coordinates of the key points P1 and P2 of the target crop fruit stalk and inputs them into the depth image and 3D point cloud data to search for and match the depth values ​​of key points P1 and P2; if the difference between the two depth values ​​is less than a fixed threshold, the localization is accurate; otherwise, the localization is re-identified and re-located.

[0109] Step 6.3: The coordinate system transformation processing unit combines the pixel coordinates and depth values ​​to transform the pixel coordinates of key point P2 to the coordinates in the camera coordinate system. Then, by combining the transformation relationship between the camera coordinate system and the world coordinate system and the position of the front blade of the movable end effector 2, the transformation of the three-dimensional coordinate values ​​is completed, that is, the final picking point coordinates are obtained.

[0110] Step 7: The embedded industrial control computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm 5, driving the movable end effector 2 to move so that the blade reaches the target crop picking point; when branches and leaves block the target crop or affect the movement path of the movable end effector 2, the embedded industrial control computer sends instructions to the air pump to blow away the branches and leaves on both sides of the target crop through the first air guide pipe 203 and the second air guide pipe 204; the embedded industrial control computer calculates the precise force required for clamping according to the shape and material of the fruit stalk, and the electric driver 208 drives the first clamping finger 201 and the second clamping finger 202 to move closer to each other along the motion guide rail; when the rubber pad is in contact with the surface of the fruit stalk, the force sensor monitors the clamping force, and the electric driver 208 adjusts the relative movement between the clamping fingers according to the real-time feedback to ensure that the fruit stalk is firmly clamped; then, the blade and the clamping fingers cooperate to cut the fruit stalk.

[0111] Step 8: During the clamping and cutting process, the ultrasonic sensor 209 detects the distance between the target crops in real time to ensure that the clamping is in place; if clamping failure is detected (if the stem is not clamped firmly and falls or is not cut successfully), the embedded industrial control computer will terminate the remaining actions through the feedback mechanism and repeat steps 5, 6, and 7 until the harvest is successful; after the harvest is successful, the embedded industrial control computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm 5, which drives the movable end effector 2 to move the harvested crops into the collection box 9.

[0112] Step 9: After harvesting the target crops in the field of view of the first depth camera 301, repeat steps 2, 3, 4, 5, 6, and 7 to continue harvesting the target crops in other areas.

[0113] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A fruit stalk clamping and shearing agricultural robot, characterized in that, The system includes a tracked chassis (1), a support platform (6) mounted on the tracked chassis (1), a control cabinet (8), a robotic arm (5), and a vision navigation system (7) mounted on the support platform (6). A collection box (9) is mounted above the control cabinet (8). The control cabinet (8) contains a host computer control system and a slave computer control system. The host computer control system includes a visual display screen (801) and an embedded industrial computer. The slave computer control system includes a robotic arm controller. A movable end effector (2) is mounted at the end of the robotic arm (5). An identification and positioning system (3) and a supplementary lighting device (4) are mounted on the movable end effector (2). The tracked chassis (1) also contains a power supply system, a drive module, and an air pump. The visual navigation system (7) includes a gimbal housing (702) fixed on a support platform (6). A servo motor (703) is installed inside the gimbal housing (702). The power output end of the servo motor (703) passes through the top of the gimbal housing (702) and is connected to the gimbal rotating base (704). A second depth camera (701) is fixed on the gimbal rotating base (704). The movable end effector (2) is installed at the end of the robotic arm (5) via the end flange (207). The end flange (207) is connected to the movable adjustment slide rail (206). A movable adjustment slider (205) is slidably installed on the movable adjustment slide rail (206). An electric driver (208) is installed below the movable adjustment slider (205). Air pipes are installed on both sides of the electric driver (208) and connected to an air pump. A motion guide rail is installed at the front end of the electric driver (208), and an ultrasonic sensor (209) is installed below the motion guide rail. Two grippers are installed at the front end of the motion guide rail. A force sensor, a blade, and a rubber pad are embedded on the opposite side of the two grippers. The movable adjustment slider (205) slides and adjusts its position on the movable adjustment slide rail (206), thereby indirectly adjusting the position of the two grippers to adapt to ridges with different planting spacing. The supplementary lighting device (4) is installed on the end flange (207) and includes a light sensor (402). The light sensor (402) detects the ambient light intensity and transmits the detection result to the embedded industrial computer. The embedded industrial computer adjusts the brightness of the supplementary light (401) according to the detection result. The second depth camera (701) of the visual navigation system (7) collects the working environment information and transmits it to the embedded industrial control computer. The embedded industrial control computer uses a multimodal information fusion map building algorithm to build a global map. The specific process of global map building is as follows: The light sensor (402) of the supplementary lighting device (4) transmits the collected light intensity data to the embedded industrial control computer. The embedded industrial control computer analyzes the light intensity data and compensates and corrects the affected depth value according to the compensation formula. ； in, For the current pixel Depth value at that location, The raw depth values ​​acquired by the second depth camera (701) Errors caused by changes in illumination; For light intensity data, and For calibration parameters; The effects of illumination are categorized into two types: direct reflection interference and information loss in shadow areas, and corresponding correction methods are designed for each. The lost depth information in shadow areas is inferred from the texture features of RGB images. Depth compensation is performed through interpolation of neighboring depth points to update the point cloud data of the reflection interference area. ； in, The depth value after compensation. For pixels The surrounding neighborhood, for Pixels within, For pixels The original depth value at that location, The weighting coefficients are as follows: ； in, For attenuation parameters; Accumulate depth data from multiple frames to eliminate depth errors caused by changes in illumination within a single frame. The fusion formula is as follows: ； in, The depth value after weighted fusion. Here, T represents the depth value per frame, and T is the number of frames in the time series. Time weighting; Update local point cloud frames: ； ； in, For the updated depth value, , , To update the weighting factors; The updated local point cloud frames are projected onto the 3D map, integrated and reconstructed to generate a continuous global map.

2. The fruit stalk clamping and shearing agricultural robot according to claim 1, characterized in that, The identification and positioning system (3) includes a first depth camera (301) installed above the motion guide rail and a lidar (302) installed above the movable adjustment slider (205).

3. A method for harvesting multiple target crops using the fruit stalk clamping and shearing agricultural robot as described in claim 1, characterized in that, The process includes the following: Step 1: First, use the first depth camera (301) of the recognition and positioning system (3) to collect the original image of the target crop. Then, manually mark the key points on the visualization display screen (801) at a position 2-3 cm above the root of the fruit stalk of the target crop: mark the highest point of the fruit stalk as P1, the middle point of the fruit stalk as P2, and the root of the fruit stalk as P3; mark the lowest point of the central axis of the target crop passing through point P3 as P4; mark the left endpoint and the right endpoint as P5 and P6 respectively; then use a target box to select and mark the target crop and its fruit stalk. P3 and P4 reflect the length of the target crop, and P5 and P6 reflect the curvature of the target crop; make the marked image data into a dataset and input it into the YOLOv8-MC model in the embedded industrial control computer for training to obtain the key point recognition model of the target crop and its fruit stalk for subsequent actual detection and recognition. Step 2: The second depth camera (701) of the visual navigation system (7) collects the working environment information and transmits it to the embedded industrial control computer. The embedded industrial control computer uses a multimodal information fusion map building algorithm to build a global map. Step 3: The embedded industrial control computer sends instructions to the gimbal servo motor (703) of the vision navigation system (7), which indirectly drives the second depth camera (701) to rotate and update the surrounding dynamic and static obstacle information in real time; The information collected by the second depth camera (701) is transmitted to the embedded industrial control computer for preliminary analysis, and then the information is added to the global map to perform local path planning and determine the robot's motion trajectory. Step 4: The embedded industrial control computer makes a decision and sends instructions to the drive module to drive the tracked chassis (1) to perform motion control according to the linear velocity and angular velocity corresponding to the optimal trajectory, and reach the designated target crop picking area; Step 5: The embedded industrial control computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm (5). The movable end effector (2), the identification and positioning system (3), and the supplementary lighting device (4) move accordingly to reach the image acquisition pose. Step 6: The first depth camera (301) acquires target crop images and depth image data, and the lidar (302) acquires three-dimensional point cloud data. The image data and point cloud data are fused and packaged and then transmitted to the embedded industrial control computer. The embedded industrial control computer inputs the fused data into the multimodal information fusion multi-crop recognition module, analyzes the geometric features of the target crop stalk, optimizes the gripping angle of the finger on the movable end effector (2), and outputs the world coordinates of the picking point of the target crop at the stalk. Step 7: The embedded industrial computer sends instructions to the robotic arm controller to control the rotation of each axis of the robotic arm (5), driving the movable end effector (2) to move, and the blade reaches the target crop picking point; when the branches and leaves block the target crop or affect the movement of the movable end effector (2), the embedded industrial computer sends instructions to the air pump to blow away the branches and leaves on both sides of the target crop through the air pipe; the embedded industrial computer calculates the clamping force required according to the shape and material of the fruit stalk, and the electric driver (208) drives the two clamping fingers to move closer to each other along the motion guide rail; when the rubber pad is in contact with the surface of the fruit stalk, the force sensor monitors the clamping force, and the electric driver (208) adjusts the relative movement between the clamping fingers according to the real-time feedback to ensure that the fruit stalk is firmly clamped; then the blade and the clamping fingers work together to cut the fruit stalk; Step 8: During the clamping and cutting process, the ultrasonic sensor (209) detects the distance between the target crops in real time to ensure that the clamping is in place; if the clamping failure is detected, the embedded industrial control computer will terminate the remaining actions through the feedback mechanism and repeat steps 5, 6, and 7 until the harvest is successful; then the embedded industrial control computer controls the rotation of each axis of the robotic arm (5) through the robotic arm controller, driving the movable end effector (2) to move the harvested crops into the collection box (9); Step 9: After harvesting the target crops in the field of view of the first depth camera (301), repeat steps 2 to 8 to continue harvesting the target crops in other areas.

4. The method for harvesting multiple target crops according to claim 3, characterized in that, In step 6, the multimodal information fusion multi-crop recognition module includes a recognition and localization processing unit, a depth registration processing unit, and a coordinate system transformation processing unit, which are used to realize the recognition of the target crop and its fruit stalk and key point localization processing, depth registration processing, and coordinate system transformation processing, respectively. The depth registration processing unit registers the image marked with the target box and key points with the depth image and the 3D point cloud data, extracts the pixel coordinates of the key points P1 and P2 of the target crop fruit stalk, and inputs them into the depth image and the 3D point cloud data to search for matching the depth values ​​of key points P1 and P2. If the difference between the two depth values ​​is less than a fixed threshold, the localization is accurate; otherwise, the recognition and localization are re-performed. The coordinate system transformation processing unit combines pixel coordinates and depth values ​​to transform the pixel coordinates of key point P2 to coordinates in the camera coordinate system. Then, it combines the transformation relationship between the camera coordinate system and the world coordinate system with the position of the front blade of the movable end effector (2) to complete the transformation of the three-dimensional coordinate values, that is, to obtain the final picking point coordinates.

Images