Fruit stem clamping and shearing agricultural robot and picking control method thereof

Abstract

The invention provides a fruit stem clamping and shearing agricultural robot and a picking control method thereof.The robot comprises a control cabinet, a mechanical arm and a visual navigation system which are installed on a crawler-type chassis supporting platform, and an upper computer control system and a lower computer control system are installed in the control cabinet; and the tail end of the mechanical arm is provided with a movable end effector with an identifying and positioning system and a light supplementing device. The movable end effector can flexibly adapt to picking requirements and complex working conditions of different crops; the upper computer control system is combined with a depth camera and a laser radar to collect data, and a YOLOv8-MC model is utilized to perform identification and key point positioning on a target crop in an image, so that the crop identification precision in a complex environment is remarkably improved, the probability of missing collection and mistaken collection is reduced, and the damage to fruits is reduced; when the upper computer control system constructs a global map, depth correction and map optimization are carried out on illumination changes, and the path planning and operation accuracy of the robot is further improved.

Description

Technical Field

[0001] The invention belongs to the technical field of agricultural harvesting equipment design, and in particular relates to a fruit stalk clamping and shearing agricultural robot and a picking control method thereof. Background Art

[0002] Agricultural automation and intelligence are important directions for the development of modern agriculture. Fruit and vegetable picking robots are of great significance in coping with labor shortages and improving picking efficiency. At present, there are some robot devices for automatic picking on the market, but most of them have problems such as limited applicability and insufficient recognition and picking accuracy. Therefore, the study of a fruit stem clamping and shearing agricultural robot that can adapt to the picking needs of various crops has become an important topic in the current technological development.

[0003] Existing harvesting robots use a design that combines a robotic arm with an end effector, collect images through visual sensors, and use algorithms to identify and harvest target crops. Usually, these robots rely on cameras and target detection algorithms to complete crop recognition, and harvest crops through a simple gripping mechanism. However, in actual operations, this design is constrained by many factors, mainly including: the efficiency and accuracy of recognition and harvesting are greatly affected by factors such as environmental changes and occlusion of branches and leaves, and are prone to missed or mis-harvesting problems; most harvesting robots are designed only for a certain type of crop and lack the ability to be compatible with different types of crops, resulting in limited application scope; the path planning algorithms of existing robots mostly use a single weight evaluation mechanism, which makes it difficult to make efficient decisions quickly in an environment mixed with dynamic changes and static obstacles.

[0004] In view of the above problems, the present invention designs a fruit stalk clamping and shearing agricultural robot and a picking control method thereof. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a fruit stalk clamping and shearing agricultural robot and a picking control method thereof, which overcomes the problems of insufficient recognition accuracy, single applicability, low environmental adaptability and insufficient path planning ability in the prior art. By optimizing the visual recognition algorithm, enhancing multi-crop adaptability, improving the path planning strategy and designing an intelligent end effector, the robot can efficiently and accurately complete the automatic picking tasks of various crops in a complex working environment.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] A fruit stalk clamping and shearing agricultural robot comprises a crawler chassis, a supporting platform is installed on the crawler chassis, a control cabinet, a mechanical arm, and a visual navigation system are installed on the supporting platform, a collection frame is installed above the control cabinet, and an upper computer control system and a lower computer control system are installed in the control cabinet. The upper computer control system comprises a visual display screen and an embedded industrial computer, and the lower computer control system comprises a mechanical arm controller; a movable end effector is installed at the end of the mechanical arm, and an identification and positioning system and a fill light device are installed on the movable end effector; a power supply system, a drive module, and an air pump are also arranged in the crawler chassis.

[0008] Furthermore, the movable end effector is installed at the end of the robotic arm through an end flange, and the end flange is connected to a movable adjustment rail, on which a movable adjustment slider is slidably installed, and an electric driver is installed below the movable adjustment slider, and air ducts are installed on both sides of the electric driver, and the air ducts are connected to the air pump; a motion guide rail is installed at the front end of the electric driver, and an ultrasonic sensor is installed below the motion guide rail; two clamping fingers are installed at the front end of the motion guide rail, and force sensors, blades, and rubber pads are embedded on the opposite sides of the two clamping fingers; the movable adjustment slider slides to adjust its position on the movable adjustment rail, thereby indirectly adjusting the positions of the two clamping fingers to adapt to ridges with different planting spacings.

[0009] Furthermore, the visual navigation system includes a gimbal housing fixed on a supporting platform, a servo motor is installed in the gimbal housing, a power output end of the servo motor passes through the top of the gimbal housing and is connected to a gimbal rotating base, and a second depth camera is fixed on the gimbal rotating base.

[0010] Furthermore, the identification and positioning system includes a first depth camera installed above the moving guide rail and a laser radar installed above the moving adjustment slider.

[0011] Furthermore, the fill light device is installed on the end flange and includes a light sensor, which detects the ambient light intensity and transmits the detection result to the embedded industrial computer, and the embedded industrial computer adjusts the brightness of the fill light according to the detection result.

[0012] A method for picking multiple target crops using the above-mentioned fruit stem clamping and shearing agricultural robot includes the following process:

[0013] Step 1: First, use the first depth camera of the recognition and positioning system to collect the original image of the target crop, and then manually mark the key points 2 to 3 cm above the root of the fruit stalk of the target crop on the visual display screen: the highest point of the fruit stalk is marked as P1, the midpoint of the fruit stalk is marked as P2, and the root of the fruit stalk is marked as P3; the lowest point of the central axis of the target crop passing through point P3 is marked as P4; the left end point and the right end point are marked as P5 and P6 respectively; then use a target frame to select and mark the target crop and its fruit stalk, P3 and P4 reflect the length of the target crop, and P5 and P6 reflect the curvature of the target crop; the labeled image data is made into a data set, and input into the YOLOv8-MC model in the embedded industrial computer for training, and the key point recognition model of the target crop and its fruit stalk is obtained for subsequent actual detection and recognition;

[0014] Step 2: The second depth camera of the visual navigation system collects the working environment information and transmits it to the embedded industrial computer, which uses the multimodal information fusion map construction algorithm to construct a global map;

[0015] Step 3: The embedded industrial computer sends instructions to the pan / tilt servo motor of the visual navigation system, indirectly driving the second depth camera to rotate and update the surrounding dynamic and static obstacle information in real time; the information collected by the second depth camera is transmitted to the embedded industrial computer for preliminary analysis, and then the information is added to the global map to perform local path planning and determine the robot's motion trajectory;

[0016] Step 4: The embedded industrial computer makes a decision and sends a command to the drive module, driving the crawler chassis to perform motion control according to the linear velocity and angular velocity corresponding to the optimal trajectory to reach the designated target crop picking area;

[0017] Step 5: The embedded industrial computer sends instructions to the robot arm controller to control the rotation of each axis of the robot arm. The movable end effector, identification and positioning system, and fill light device move accordingly to reach the image acquisition position.

[0018] Step 6: The first depth camera collects the target crop image and depth image data, the laser radar collects the three-dimensional point cloud data, and the image data and the point cloud data are fused and packaged and transmitted to the embedded industrial computer; the embedded industrial computer inputs the fused data into the multi-modal information fusion multi-crop recognition module, analyzes the geometric characteristics of the target crop fruit stalk, optimizes the gripping angle of the gripper on the movable end effector, and outputs the world coordinates of the picking point of the target crop at the fruit stalk;

[0019] Step 7: The embedded industrial computer sends instructions to the robot arm controller to control the rotation of each axis of the robot arm, drive the movable end effector to move, and make the blade reach the target crop picking point; when the branches and leaves block the target crop or affect the travel route of the movable end effector, the embedded industrial computer sends instructions to the air pump to blow away the branches and leaves on both sides of the target crop through the air duct; the embedded industrial computer calculates the precise force required for clamping according to the shape and material of the fruit stalk, and the electric drive drives the two clamping fingers to approach each other along the motion guide rail; when the rubber pad fits the surface of the fruit stalk, the force sensor monitors the clamping force, and the electric drive adjusts the relative movement between the clamping fingers according to real-time feedback to ensure that the fruit stalk is firmly clamped; then, the blade cooperates with the clamping fingers to achieve fruit stalk cutting;

[0020] Step 8: During the clamping and cutting process, the ultrasonic sensor detects the distance between the target crops in real time to ensure that the crops are clamped in place. If the clamping fails, the embedded industrial computer will terminate the remaining actions through the feedback mechanism and repeat steps 5, 6, and 7 until the crops are picked successfully. After the crops are picked successfully, the embedded industrial computer sends instructions to the robot arm controller to control the rotation of each axis of the robot arm, driving the movable end effector to move the picked crops to the collection frame.

[0021] Step 9: After picking the target crops in the field of view of the first depth camera, repeat steps 2 to 8 to continue picking target crops in other areas.

[0022] Furthermore, in step 2, the specific process of constructing the global map is as follows:

[0023] Step 2.1: The light sensor transmits the collected light intensity data to the embedded industrial computer, which analyzes the light intensity data and compensates the affected depth value according to the compensation formula:

[0024] D current (x,y)=D raw (x,y)+E light (x,y)

[0025] =D raw (x,y)+[μ×I(x,y)+η]

[0026] Among them, D current (x, y) is the depth value of the current pixel (x, y), D raw (x, y) is the original depth value collected by the second depth camera, E light (x, y) is the error caused by illumination change; I(x, y) is the light intensity data, μ and η are correction parameters;

[0027] Step 2.2: Divide the lighting effects into two categories: direct reflection interference and shadow area information loss, and design corresponding correction methods for each. Infer the lost depth information of the shadow area through the texture features of the RGB image. Perform depth compensation through neighborhood depth point interpolation and update the point cloud data of the reflection interference area:

[0028]

[0029] Among them, D filled (x, y) is the compensated depth value, ￥(x, y) is the neighborhood around the pixel point (x, y), (ξ, ζ) is the pixel point within ￥(x, y), D raw (ξ,ζ) is the original depth value at the pixel point (ξ,ζ), w c (ξ,ζ) is the weight coefficient, which is as follows:

[0030]

[0031] Step 2.3: Accumulate multi-frame depth data to eliminate the depth error caused by single-frame illumination changes. The fusion formula is as follows:

[0032]

[0033] Among them, D fused (x, y) is the depth value after weighted fusion, D t (x, y) is the depth value of each frame, T is the number of time series frames, w t is the time weight; κ is the attenuation parameter;

[0034] Step 2.4: Update the local point cloud frame:

[0035] D update (x,y)=ρ1×[D raw (x,y)+E light (x,y)]+ρ2×D filled (x,y)+ρ3×D fused (x,y)

[0036] ρ1+ρ2+ρ3=1

[0037] Where Dupdate(x,y) is the updated depth value, ρ1, ρ2, ρ3 are update weight factors;

[0038] Step 2.5: Project the updated local point cloud frame into the 3D map, perform integration and reconstruction, and generate a continuous global map.

[0039] Furthermore, in step 6, the multi-modal information fusion multi-crop recognition module includes an identification and positioning processing unit, a depth registration processing unit, and a coordinate system conversion processing unit, which are respectively used to realize target crop and fruit stalk recognition and key point positioning processing, depth registration processing, and coordinate system conversion processing, wherein the specific data processing process of the identification and positioning processing unit is as follows:

[0040] The recognition and positioning processing unit uses an improved YOLOv8 model, that is, a YOLOv8-MC model to identify and locate key points of the target crop and its fruit stem in the image; the improvement method of the YOLOv8 model includes adding an attention mechanism module at the end of the feature extraction network and adding a BiFPN module before each convolution layer in the feature fusion network, as follows:

[0041] An attention mechanism module is added at the end of the feature extraction network to integrate the extracted features in the channel dimension and spatial dimension, focusing on the phenotypic characteristics of the target crop. The integration formula is as follows:

[0042]

[0043]

[0044] Among them, M c (F) is a one-dimensional channel attention feature map; σ is the sigmoid activation function; SVM is the support vector machine classification algorithm; AvgPool and MaxPool represent the average pooling layer and the maximum pooling layer respectively; F is the input feature map; are the channel attention feature maps obtained after the average pooling layer and the maximum pooling layer respectively; W0 and W1 represent tensors of size (C / r, C) and (C, C / r), respectively, where C is the number of channels and r is the reduction ratio; are the spatial attention feature maps obtained after the average pooling layer and the maximum pooling layer respectively; M s (F), M s (F1) are both two-dimensional spatial attention feature maps; f 7×7 is a convolution kernel of size 7×7; is element-by-element multiplication; F1 is the output after spatial attention; F2 is the final output feature map;

[0045] A BiFPN module is added before each convolutional layer in the feature fusion network. An adaptive weight allocation mechanism is introduced to dynamically allocate weights according to the importance of crop features, so that information flows between feature maps of different scales and feature fusion is performed. The fused features are input into the output network for non-maximum suppression processing, and the final target box and key points of the target crop are output and mapped to the original image.

[0046] Furthermore, the depth registration processing unit registers the image marked with the target frame and key points with the depth image and the three-dimensional point cloud data, extracts the pixel coordinates of the key points P1 and P2 of the fruit stalk of the target crop, inputs them into the depth image and the three-dimensional point cloud data for searching, matches the depth values ​​of the key points P1 and P2, and if the difference between the depth values ​​of the two is less than a fixed threshold, the positioning is accurate, otherwise the identification and positioning are performed again;

[0047] The coordinate system conversion processing unit combines the pixel coordinates and the depth value to convert the pixel coordinates of the key point P2 into the coordinates in the camera coordinate system, and then combines the conversion relationship between the camera coordinate system and the world coordinate system and the position of the blade at the front end of the movable end effector to complete the conversion of the three-dimensional coordinate value, that is, to obtain the final picking point coordinates.

[0048] The present invention has the following beneficial effects:

[0049] The fruit stalk clamping and shearing agricultural robot provided by the present invention, especially the design of the movable end effector, can flexibly adapt to the picking needs and complex working conditions of different crops, is suitable for picking various crops, and has significant advantages in improving the operating efficiency of the picking robot, adapting to complex working environments and expanding the scope of application.

[0050] The multi-modal information fusion multi-crop recognition module designed in the present invention, combined with a depth camera and a lidar, significantly improves the robot's crop recognition accuracy in complex environments; this precise recognition and picking technology reduces the chances of missed and mispicked crops, reduces damage to the fruit, and increases the market value of the crops.

[0051] The path planning and obstacle avoidance algorithm designed by the present invention selects the optimal path by real-time evaluation of dynamic and static obstacles in the working environment, thus avoiding the situation where traditional robots fall into the local minimum and cannot reach the target point in complex environments. This innovation can significantly improve the adaptability of robots in complex working environments, reduce the risk of interruption or damage caused by improper path planning, and thus ensure long-term efficient and stable operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the overall structure of the fruit stalk clamping and shearing agricultural robot of the present invention;

[0053] Figure 2 This is a schematic diagram of the side structure of the fruit stalk clamping and shearing agricultural robot of the present invention;

[0054] Figure 3 This is a schematic diagram of the back structure of the fruit stalk clamping and shearing agricultural robot of the present invention;

[0055] Figure 4 This is a schematic diagram of the structure of the execution device of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of the movable end effector of the present invention;

[0057] Figure 6 This is a schematic diagram of the electric pan-tilt structure of the present invention;

[0058] Figure 7 This is a schematic diagram of the DAV_DWA local path planning algorithm of the present invention;

[0059] Figure 8 Schematic diagram of the YOLOv8-MC model described in the present invention.

[0060] In the figure: 1-tracked chassis; 2-movable end effector; 201-first gripping finger; 202-second gripping finger; 203-first air duct; 204-second air duct; 205-movable adjustment slider; 206-movable adjustment slide rail; 207-end flange; 208-electric drive; 209-ultrasonic sensor; 3-identification and positioning system; 301-first depth camera; 302-laser radar; 4-fill light device; 401-fill light; 402-light sensor; 5-mechanical arm; 6-support platform; 7-visual navigation system; 701-second depth camera; 702-pan head housing; 703-servo motor; 704-pan head rotating base; 8-control cabinet; 801-visual display screen; 9-collection frame. DETAILED DESCRIPTION

[0061] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0062] like Figures 1 to 3 As shown, the fruit stem clamping and shearing agricultural robot of the present invention comprises a crawler chassis 1, a power supply system, a drive module, an air pump, a movable end effector 2, an identification and positioning system 3, a fill light device 4, a mechanical arm 5, a support platform 6, a visual navigation system 7, a control cabinet 8, and a collection frame 9. The host computer control system is connected to the drive module, the air pump, the movable end effector 2, the identification and positioning system 3, the fill light device 4, the lower computer control system, and the visual navigation system 7 by signals. The host computer control system receives data for data analysis and makes scientific decisions, and issues instructions to control each module to perform corresponding actions. The power supply system, the drive module, and the air pump are all arranged in the crawler chassis 1. The power supply system is equivalent to a battery, which supplies power to the entire robot. The drive module is used to drive the motor of the crawler chassis 1, and the air pump is used to supply air to the air guide tube on the movable end effector 2.

[0063] like Figures 1 to 3As shown, a support platform 6 is installed on the crawler chassis 1, a control cabinet 8 and a robotic arm 5 are installed above the support platform 6, and a collection frame 9 is installed above the control cabinet 8; the control cabinet 8 contains an upper computer control system and a lower computer control system; the upper computer control system includes a visual display screen 801 and an embedded industrial computer; the lower computer control system includes a robotic arm controller, and the robotic arm controller receives instructions from the embedded industrial computer to control the axes of the robotic arm 5 to rotate to corresponding angles.

[0064] like Figure 3 , 5 As shown, the movable end effector 2 includes a first gripping finger 201, a second gripping finger 202, a first air duct 203, a second air duct 204, a movable adjustment slider 205, a movable adjustment slide rail 206, an end flange 207, an electric driver 208, and an ultrasonic sensor 209; the movable end effector 2 is installed at the end of the robot arm 5 through the end flange 207, the end flange 207 is connected to the movable adjustment slide rail 206, the movable adjustment slider 205 is slidably installed on the movable adjustment slide rail 206, and the electric driver 208 is installed below the movable adjustment slider 205. The first air guide tube 203 and the second air guide tube 204 are respectively installed on both sides of the device 208, a moving guide rail is installed at the front end of the electric driver 208, an ultrasonic sensor 209 is installed below the moving guide rail, a first clamping finger 201 and a second clamping finger 202 are installed at the front end of the moving guide rail, and the first clamping finger 201 and the second clamping finger 202 are embedded with force sensors, blades, and rubber pads on the opposite sides; the movable adjustment slider 205 can slide and adjust the position on the movable adjustment slide rail 206, and then indirectly adjust the position of the first clamping finger 201 and the second clamping finger 202 to adapt to ridges with different planting spacings. The present invention uses a laser radar 302 in conjunction with a first depth camera 301 to achieve accurate identification and positioning of target crops, and uses an ultrasonic sensor 209 to determine whether the picking is successful; in addition, there is a problem in the agricultural environment that branches and leaves block the target crops, making it difficult for the movable end effector 2 to pick, so the present invention installs two air guide tubes on both sides of the electric driver 208, which can blow away the obstructions on both sides of the target crops, so that the target crops can be exposed, greatly improving the success rate of picking.

[0065] like Figure 1 , 2As shown in , 6, the visual navigation system 7 is installed at the front end of the supporting platform 6, including a second depth camera 701, a gimbal housing 702, a servo motor 703, and a gimbal rotating base 704; the gimbal housing 702 is fixed on the supporting platform 6, and a servo motor 703 is installed in the gimbal housing 702. The power output end of the servo motor 703 passes through the top of the gimbal housing 702 and is connected to the gimbal rotating base 704. The second depth camera 701 is fixed on the gimbal rotating base 704, and the servo motor 703 can drive the second depth camera 701 to rotate to scan the depth information of different areas to assist in the subsequent path planning.

[0066] like Figure 4 , 5 As shown, the identification and positioning system 3 includes a first depth camera 301 and a laser radar 302 ; the first depth camera 301 is installed above the motion guide rail of the movable end effector 2 for identifying the target crop, and the laser radar 302 is installed above the movable adjustment slider 205 .

[0067] like Figure 4 , 5 As shown, the fill light device 4 is installed on the end flange 207, including a fill light 401 and a light sensor 402; the light sensor 402 detects the ambient light intensity and transmits the detection result to the embedded industrial computer, and the embedded industrial computer automatically adjusts the brightness of the fill light 401 according to the detection result.

[0068] A method for picking multiple target crops using the fruit stalk clamping and shearing agricultural robot of the present invention comprises the following steps:

[0069] Step 1: Before the robot performs the actual picking operation, the first depth camera 301 is used to collect the original image of the target crop and the image size is normalized; then the key points are manually marked on the visual display screen 801 at a position 2 to 3 cm above the root of the fruit stalk of the target crop: the highest point is marked as P1, the midpoint is marked as P2, and the root of the fruit stalk is marked as P3; the lowest point of the central axis of the target crop through P3 is marked as P4; the left end point and the right end point are marked as P5 and P6 respectively; then a target frame is used to select and mark the target crop and its fruit stalk, P3 and P4 reflect the length of the target crop, combined with P5 and P6 to reflect the curvature of the target crop, which can assist in determining the position and posture of the target crop; the labeled image data is made into a data set, and input into the YOLOv8-MC model in the embedded industrial computer for training, and a key point recognition model of the target crop and its fruit stalk is obtained for subsequent actual detection and recognition.

[0070] Step 2: The second depth camera 701 collects working environment information and transmits it to the embedded industrial computer. The working environment information includes RGB images and depth images. The embedded industrial computer uses a multimodal information fusion map construction algorithm to construct a global map. In addition, in a facility agriculture environment, lighting conditions change due to changes in transparent films, reflective surfaces or natural light. Such lighting changes can easily lead to distortion of depth camera data, especially jumps in depth values ​​or accumulation of errors, which reduces the accuracy of the generated global map, thereby affecting the robot's path planning and operation accuracy. Therefore, when the embedded industrial computer of the present invention converts the environmental information collected by the second depth camera 701 into a global map, it performs depth correction and map optimization on the lighting changes.

[0071] The specific process of building a global map is as follows:

[0072] Step 2.1: The light sensor 402 transmits the collected light intensity data to the embedded industrial computer, which analyzes the light intensity data and compensates the affected depth value according to the compensation formula:

[0073] D current (x,y)=D raw (x,y)+E light (x,y)

[0074] =D raw (x,y)+[μ×I(x,y)+η]

[0075] Among them, D current (x, y) is the depth value of the current pixel (x, y), D raw (x, y) is the original depth value collected by the second depth camera 701, E light (x, y) is the error caused by illumination change; I(x, y) is the light intensity data, μ and η are correction parameters;

[0076] Step 2.2: Divide the illumination influence into two categories: direct reflection interference and shadow area information loss, and design corresponding correction methods respectively; infer the lost depth information of the shadow area through the texture features of the RGB image; perform depth compensation through neighborhood depth point interpolation, and update the point cloud data of the reflection interference area. The formula is as follows:

[0077]

[0078] Among them, D filled (x, y) is the compensated depth value, ￥(x, y) is the neighborhood around the pixel point (x, y), (ξ, ζ) is the pixel point within ￥(x, y), D raw (ξ,ζ) is the original depth value at the pixel point (ξ,ζ), w c (ξ,ζ) is the weight coefficient, which is defined as follows:

[0079]

[0080] Among them, κ is the attenuation parameter;

[0081] Step 2.3: Accumulate multi-frame depth data to eliminate the depth error caused by single-frame illumination changes. The fusion formula is as follows:

[0082]

[0083] Among them, D fused (x, y) is the depth value after weighted fusion, D t (x, y) is the depth value of each frame, T is the number of time series frames, w t is the time weight;

[0084] Step 2.4: Update the local point cloud frame. The formula is as follows:

[0085] D update (x,y)=ρ1×[D raw (x,y)+E light (x,y)]+ρ2×D filled (x,y)+ρ3×D fused (x,y)

[0086] ρ1+ρ2+ρ3=1

[0087] Where Dupdate(x,y) is the updated depth value, ρ1, ρ2, ρ3 are update weight factors;

[0088] Step 2.5: Project the updated local point cloud frame into the 3D map, perform integration and reconstruction, and generate a continuous global map.

[0089] Step 3: The embedded industrial computer sends a command to the pan-tilt servo motor 703 to control the gear on the pan-tilt servo motor 703 to rotate, drive the pan-tilt rotating base 704 to rotate, and the second depth camera 701 on the pan-tilt rotating base 704 rotates accordingly, and updates the surrounding dynamic and static obstacle information in real time; the information collected by the second depth camera 701 is transmitted to the embedded industrial computer for preliminary analysis, and then the information is added to the global map, and the local path planning is performed by the dynamic window method (Dual obstacle cost function, Adaptive weights and Virtualtarget_Dynamic Window Approach, DAV_DWA) based on the dual obstacle cost function, adaptive weights and virtual target method to determine the robot motion trajectory; Figure 7 As shown ( Figure 7The horizontal and vertical coordinates in the image have no practical meaning, they only form a plane to show the position of the robot). The specific processing process is as follows:

[0090] Step 3.1: Use different evaluation methods for dynamic / static obstacles; divide the distance between the robot and the obstacle into two components, including: the distance evaluation function between the robot and the dynamic obstacle dist_dynamic(v,w), and the distance evaluation function between the robot and the static obstacle dist_static(v,w), where v represents the robot linear velocity and w represents the robot angular velocity; set different safety thresholds for the two types of obstacles for differentiated processing, and include them in the two evaluation functions respectively. The trajectory evaluation function is defined as:

[0091] G(v,w)=σ(α×heading(v,w)+γ×velocity(v,w)+β1×dist_dynamic(v,w)+β2×dist_static(v,w))

[0092] Among them, G(v,w) is the trajectory evaluation function; heading(v,w) is the azimuth evaluation function, which means the angle deviation between the orientation of the robot's expected trajectory end point and the target position at the set speed; dist_dynamic(v,w) and dist_static(v,w) are the distance evaluation functions from the robot trajectory to dynamic obstacles and static obstacles respectively; velocity(v,w) is the current robot linear velocity evaluation function; α, β1, β2 and γ are weight coefficients; σ is the normalization processing parameter;

[0093] Step 3.2: Adaptively adjust the weights of each evaluation function according to the fuzzy logic algorithm; input the dynamic / static obstacle distance into the fuzzy logic algorithm, and find the optimal path by dynamically adjusting the four weight parameters α, β1, β2 and γ. The specific process is as follows:

[0094] The minimum distance between the robot and the dynamic obstacle and the static obstacle is obtained through the second depth camera 701, and these distances are used as input variables for fuzzy processing; then the input variables are evaluated using Z-type, Gaussian-type, and S-type membership functions, and the four weight parameters α, β1, β2, and γ are dynamically adjusted through the fuzzy controller; by adjusting the weight coefficients, the robot can give priority to avoiding obstacles and reduce the speed when encountering dynamic obstacles, and increase the speed and reduce the obstacle avoidance priority when the static obstacle is far away;

[0095] Step 3.3: Introduce a virtual target point so that the robot can still escape when it reaches the local minimum. The specific process of creating a virtual target point is as follows:

[0096] The position where the robot's speed is lower than a predetermined threshold and the state lasts for more than a certain period of time is determined as the local minimum position; obstacles with a distance between adjacent obstacles less than or equal to 2p are detected and added to a local minimum obstacle group, where p is the robot radius; the outermost obstacle in the obstacle group that is closer to the robot is defined as the target obstacle, and a virtual target point is set at 2 times the radius outside the direction of the line connecting the robot and the obstacle; when the robot reaches the virtual target point, the virtual target point is switched to the actual target point, and the local path planning is completed.

[0097] Step 4: The embedded industrial computer sends the processed information to the visual display screen 801 to visualize the operation process; the embedded industrial computer makes a decision and sends instructions to the drive module to drive the crawler chassis 1 to perform motion control according to the linear velocity and angular velocity corresponding to the optimal trajectory to reach the designated target crop picking area.

[0098] Step 5: The embedded industrial computer sends instructions to the robot arm controller, and the robot arm controller controls the rotation of each axis of the robot arm 5, and the movable end effector 2, the identification and positioning system 3, and the fill light device 4 move accordingly to reach the image acquisition posture.

[0099] Step 6: The first depth camera 301 collects the target crop image and depth image data, the laser radar 302 collects the three-dimensional point cloud data, and the image data and the point cloud data are fused and packaged and transmitted to the embedded industrial computer for analysis; the embedded industrial computer inputs the fused data into the multi-modal information fusion multi-crop recognition module, analyzes the geometric features of the fruit stalk of the target crop, such as length, angle, distance, etc., optimizes the clamping angle of the first clamping finger 201 and the second clamping finger 202 on the movable end effector 2, and outputs the world coordinates of the picking point of the target crop at the fruit stalk; specifically, the multi-modal information fusion multi-crop recognition module includes an identification and positioning processing unit, a depth registration processing unit, and a coordinate system conversion processing unit, which are respectively used to realize the identification of the target crop and its fruit stalk and key point positioning processing, depth registration processing, and coordinate system conversion processing. The specific process is as follows:

[0100] Step 6.1: The recognition and positioning processing unit uses an improved YOLOv8 model, namely, a YOLOv8-multi-crop (YOLOv8-MC) model to identify and locate key points of the target crop and its fruit stem in the image; the improvement method of the YOLOv8 model includes adding an attention mechanism module at the end of the feature extraction network and adding a BiFPN module before each convolution layer in the feature fusion network, such as Figure 8 The specific improvements are as follows:

[0101] An attention mechanism module is added at the end of the feature extraction network to integrate the extracted features in the channel dimension and spatial dimension, focusing on the phenotypic characteristics of the target crop. The integration formula is as follows:

[0102]

[0103] Among them, M c (F) is a one-dimensional channel attention feature map; σ is the sigmoid activation function; SVM is the support vector machine classification algorithm; AvgPool and MaxPool represent the average pooling layer and the maximum pooling layer respectively; F is the input feature map; are the channel attention feature maps obtained after the average pooling layer and the maximum pooling layer respectively; W0 and W1 represent tensors of size (C / r, C) and (C, C / r), respectively, where C is the number of channels and r is the reduction ratio; are the spatial attention feature maps obtained after the average pooling layer and the maximum pooling layer respectively; M s (F), M s (F1) are both two-dimensional spatial attention feature maps; f 7×7 is a convolution kernel of size 7×7; is element-by-element multiplication; F1 is the output after spatial attention; F2 is the final output feature map;

[0104] Add a BiFPN module before each convolutional layer in the feature fusion network; introduce an adaptive weight allocation mechanism to dynamically allocate weights according to the importance of crop features, flow information between feature maps of different scales, and perform feature fusion; input the fused features into the output network, perform non-maximum suppression processing, output the final target box and key points of the target crop, and map them to the original image;

[0105] Step 6.2: The depth registration processing unit registers the image marked with the target frame and key points with the depth image and the three-dimensional point cloud data; extracts the pixel coordinates of the key points P1 and P2 of the fruit stalk of the target crop, inputs them into the depth image and the three-dimensional point cloud data for searching, and matches the depth values ​​of the key points P1 and P2; if the difference between the two depth values ​​is less than a fixed threshold, the positioning is accurate, otherwise the identification and positioning are performed again;

[0106] Step 6.3: The coordinate system conversion processing unit combines the pixel coordinates and the depth value to convert the pixel coordinates of the key point P2 into the coordinates in the camera coordinate system, and then combines the conversion relationship between the camera coordinate system and the world coordinate system and the position of the front blade of the movable end effector 2 to complete the conversion of the three-dimensional coordinate value, that is, to obtain the final picking point coordinates.

[0107] Step 7: The embedded industrial computer sends instructions to the robot arm controller to control the rotation of each axis of the robot arm 5, drive the movable end effector 2 to move, and make the blade reach the target crop picking point; when the branches and leaves block the target crop or affect the travel route of the movable end effector 2, the embedded industrial computer sends instructions to the air pump to blow away the branches and leaves on both sides of the target crop through the first air duct 203 and the second air duct 204; the embedded industrial computer calculates the precise force required for clamping according to the shape and material of the fruit stalk, and the electric driver 208 drives the first clamping finger 201 and the second clamping finger 202 to approach each other along the motion guide rail; when the rubber pad fits the surface of the fruit stalk, the force sensor monitors the clamping force, and the electric driver 208 adjusts the relative movement between the clamping fingers according to the real-time feedback to ensure that the fruit stalk is firmly clamped; then, the blade and the clamping fingers cooperate to achieve fruit stalk cutting.

[0108] Step 8: During the clamping and cutting process, the ultrasonic sensor 209 detects the distance between the target crops in real time to ensure that the clamping is in place; if a clamping failure is detected (if the stem is not clamped firmly and falls off or the cutting is not successful), the embedded industrial computer will terminate the remaining actions through the feedback mechanism and repeat steps 5, 6, and 7 until the picking is successful; after the picking is successful, the embedded industrial computer sends instructions to the robot arm controller to control the rotation of each axis of the robot arm 5, driving the movable end effector 2 to move the picked crops to the collection frame 9.

[0109] Step 9: After picking the target crops in the field of view of the first depth camera 301, repeat steps 2, 3, 4, 5, 6, and 7 to continue picking target crops in other areas.

[0110] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A fruit stem clamping and shearing agricultural robot, characterized in that: The invention comprises a crawler chassis (1), a support platform (6) is installed on the crawler chassis (1), a control cabinet (8), a mechanical arm (5), and a visual navigation system (7) are installed on the support platform (6), a collection frame (9) is installed above the control cabinet (8), a host computer control system and a lower computer control system are installed in the control cabinet (8), the host computer control system comprises a visual display screen (801) and an embedded industrial computer, and the lower computer control system comprises a mechanical arm controller; a movable end effector (2) is installed at the end of the mechanical arm (5), and an identification and positioning system (3) and a fill light device (4) are installed on the movable end effector (2); a power supply system, a drive module, and an air pump are also arranged in the crawler chassis (1).

2. The fruit stem gripping and shearing agricultural robot according to claim 1, characterized in that: The movable end effector (2) is installed at the end of the mechanical arm (5) through an end flange (207), and the end flange (207) is connected to a movable adjustment slide rail (206). A movable adjustment slider (205) is slidably installed on the movable adjustment slide rail (206), and an electric driver (208) is installed below the movable adjustment slider (205). Air guide tubes are installed on both sides of the electric driver (208), and the air guide tubes are connected to an air pump; a motion guide rail is installed at the front end of the electric driver (208), and an ultrasonic sensor (209) is installed below the motion guide rail; two clamping fingers are installed at the front end of the motion guide rail, and force sensors, blades, and rubber pads are embedded on the opposite sides of the two clamping fingers; the movable adjustment slider (205) slides to adjust the position on the movable adjustment slide rail (206), thereby indirectly adjusting the positions of the two clamping fingers to adapt to ridges with different planting spacings.

3. The fruit stem gripping and shearing agricultural robot according to claim 1, characterized in that: The visual navigation system (7) comprises a pan-tilt housing (702) fixed on a supporting platform (6), a servo motor (703) being installed in the pan-tilt housing (702), a power output end of the servo motor (703) passing through the top of the pan-tilt housing (702) and connected to a pan-tilt rotating base (704), and a second depth camera (701) being fixed on the pan-tilt rotating base (704).

4. The fruit stem gripping and shearing agricultural robot according to claim 2, characterized in that: The identification and positioning system (3) comprises a first depth camera (301) installed above the moving guide rail and a laser radar (302) installed above the moving adjustment slider (205).

5. The fruit stem gripping and shearing agricultural robot according to claim 2, characterized in that: The fill light device (4) is mounted on the end flange (207), and comprises a light sensor (402). The light sensor (402) detects the ambient light intensity and transmits the detection result to the embedded industrial computer. The embedded industrial computer adjusts the brightness of the fill light (401) according to the detection result.

6. A method for harvesting multiple target crops using the fruit stem clamping and shearing agricultural robot according to claim 2, characterized in that: The process includes the following: Step 1: first, using the first depth camera (301) of the recognition and positioning system (3) to collect the original image of the target crop, and then manually marking the key points of the target crop fruit stalk 2 to 3 cm above the root on the visual display screen (801): the highest point of the fruit stalk is marked as P1, the midpoint of the fruit stalk is marked as P2, and the root of the fruit stalk is marked as P3; the lowest point of the central axis of the target crop passing through point P3 is marked as P4; the left end point and the right end point are marked as P5 and P6 respectively; then a target frame is used to select and mark the target crop and its fruit stalk, P3 and P4 reflect the length of the target crop, and P5 and P6 reflect the curvature of the target crop; the marked image data is made into a data set, and input into the YOLOv8-MC model in the embedded industrial computer for training, so as to obtain a key point recognition model of the target crop and its fruit stalk for subsequent actual detection and recognition; Step 2: The second depth camera (701) of the visual navigation system (7) collects the working environment information and transmits it to the embedded industrial computer, and the embedded industrial computer uses a multi-modal information fusion map construction algorithm to construct a global map; Step 3: The embedded industrial computer sends a command to the pan / tilt servo motor (703) of the visual navigation system (7), indirectly driving the second depth camera (701) to rotate, and updating the surrounding dynamic and static obstacle information in real time; the information collected by the second depth camera (701) is transmitted to the embedded industrial computer for preliminary analysis, and then the information is added to the global map to perform local path planning and determine the robot's motion trajectory; Step 4: The embedded industrial computer makes a decision and sends a command to the drive module to drive the crawler chassis (1) to perform motion control according to the linear velocity and angular velocity corresponding to the optimal trajectory to reach the designated target crop picking area; Step 5: The embedded industrial computer sends a command to the robot arm controller to control the rotation of each axis of the robot arm (5), and the movable end effector (2), the identification and positioning system (3), and the fill light device (4) move accordingly to reach the image acquisition position; Step 6: The first depth camera (301) collects target crop images and depth image data, the laser radar (302) collects three-dimensional point cloud data, and the image data and point cloud data are fused, packaged, and transmitted to the embedded industrial computer; The embedded industrial computer inputs the fused data into the multi-modal information fusion multi-crop recognition module, analyzes the geometric features of the fruit stem of the target crop, optimizes the gripping angle of the gripping fingers on the movable end effector (2), and outputs the world coordinates of the picking point of the target crop at the fruit stem; Step 7: The embedded industrial computer sends a command to the robot arm controller to control the rotation of each axis of the robot arm (5), drive the movable end effector (2) to move, and the blade reaches the target crop picking point; when the branches and leaves block the target crop or affect the travel route of the movable end effector (2), the embedded industrial computer sends a command to the air pump to blow away the branches and leaves on both sides of the target crop through the air duct; the embedded industrial computer calculates the force required for clamping according to the shape and material of the fruit stalk, and the electric driver (208) drives the two clamping fingers to approach each other along the motion guide rail; when the rubber pad is in contact with the surface of the fruit stalk, the force sensor monitors the clamping force, and the electric driver (208) adjusts the relative movement between the clamping fingers according to the real-time feedback to ensure that the fruit stalk is firmly clamped; then the blade cooperates with the clamping fingers to achieve fruit stalk cutting; Step 8: During the clamping and cutting process, the ultrasonic sensor (209) detects the distance between the target crops in real time to ensure that the crops are clamped in place; if a clamping failure is detected, the embedded industrial computer will terminate the remaining actions through a feedback mechanism and repeat steps 5, 6, and 7 until the crops are picked successfully; then the embedded industrial computer controls the rotation of each axis of the robotic arm (5) through the robotic arm controller, driving the movable end effector (2) to move the picked crops to the collection frame (9); Step 9: After picking the target crops in the field of view of the first depth camera (301), repeat steps 2 to 8 to continue picking target crops in other areas.

7. The method for picking multiple target crops according to claim 6, characterized in that: In step 2, the specific process of building the global map is as follows: Step 2.1: The light sensor (402) of the light-filling device (4) transmits the collected light intensity data to the embedded industrial computer, which analyzes the light intensity data and compensates the affected depth value according to the compensation formula: D current (x,y)=D raw (x,y)+E light (x,y) =D raw (x,y)+[μ×I(x,y)+η] Among them, D current (x, y) is the depth value of the current pixel (x, y), D raw (x, y) is the original depth value collected by the second depth camera (701), E light (x, y) is the error caused by illumination change; I(x, y) is the light intensity data, μ and η are correction parameters; Step 2.2: Divide the lighting effects into two categories: direct reflection interference and shadow area information loss, and design corresponding correction methods for each. Infer the lost depth information of the shadow area through the texture features of the RGB image. Perform depth compensation through neighborhood depth point interpolation and update the point cloud data of the reflection interference area: Among them, D filled (x, y) is the compensated depth value, ￥(x, y) is the neighborhood around the pixel point (x, y), (ξ, ζ) is the pixel point within ￥(x, y), D raw (ξ,ζ) is the original depth value at the pixel point (ξ,ζ), w c (ξ,ζ) is the weight coefficient, which is as follows: Step 2.3: Accumulate multi-frame depth data to eliminate the depth error caused by single-frame illumination changes. The fusion formula is as follows: Among them, D fused (x, y) is the depth value after weighted fusion, D t (x, y) is the depth value of each frame, T is the number of time series frames, w t is the time weight; κ is the attenuation parameter; Step 2.4: Update the local point cloud frame: D update (x,y)=ρ1×[D raw (x,y)+E light (x,y)]+ρ2×D filled (x,y)+ρ3×D fused (x,y) ρ1+ρ2+ρ3=1 Where Dupdate(x,y) is the updated depth value, ρ1, ρ2, ρ3 are update weight factors; Step 2.5: Project the updated local point cloud frame into the 3D map, perform integration and reconstruction, and generate a continuous global map.

8. The method for picking multiple target crops according to claim 6, characterized in that: In step 6, the multi-modal information fusion multi-crop recognition module includes an identification and positioning processing unit, a depth registration processing unit, and a coordinate system conversion processing unit, which are respectively used to realize target crop and fruit stalk recognition and key point positioning processing, depth registration processing, and coordinate system conversion processing. The specific data processing process of the identification and positioning processing unit is as follows: The recognition and positioning processing unit uses an improved YOLOv8 model, that is, a YOLOv8-MC model to identify and locate key points of the target crop and its fruit stem in the image; the improvement method of the YOLOv8 model includes adding an attention mechanism module at the end of the feature extraction network and adding a BiFPN module before each convolution layer in the feature fusion network, as follows: An attention mechanism module is added at the end of the feature extraction network to integrate the extracted features in the channel dimension and spatial dimension, focusing on the phenotypic characteristics of the target crop. The integration formula is as follows: Among them, M c (F) is a one-dimensional channel attention feature map; σ is the sigmoid activation function; SVM is the support vector machine classification algorithm; AvgPool and MaxPool represent the average pooling layer and the maximum pooling layer respectively; F is the input feature map; are the channel attention feature maps obtained after the average pooling layer and the maximum pooling layer respectively; W0 and W1 represent tensors of size (C / r, C) and (C, C / r), respectively, where C is the number of channels and r is the reduction ratio; are the spatial attention feature maps obtained after the average pooling layer and the maximum pooling layer respectively; M s (F), M s (F1) are both two-dimensional spatial attention feature maps; f 7×7 is a convolution kernel of size 7×7; is element-by-element multiplication; F1 is the output after spatial attention; F2 is the final output feature map; A BiFPN module is added before each convolutional layer in the feature fusion network. An adaptive weight allocation mechanism is introduced to dynamically allocate weights according to the importance of crop features, so that information flows between feature maps of different scales and feature fusion is performed. The fused features are input into the output network for non-maximum suppression processing, and the final target box and key points of the target crop are output and mapped to the original image.

9. The method for picking multiple target crops according to claim 8, characterized in that: The depth registration processing unit registers the image marked with the target frame and key points with the depth image and the three-dimensional point cloud data, extracts the pixel coordinates of the key points P1 and P2 of the fruit stalk of the target crop, inputs them into the depth image and the three-dimensional point cloud data for searching, matches the depth values ​​of the key points P1 and P2, and if the difference between the depth values ​​of the two is less than a fixed threshold, the positioning is accurate, otherwise the identification and positioning are performed again; The coordinate system conversion processing unit combines the pixel coordinates and the depth value to convert the pixel coordinates of the key point P2 into coordinates in the camera coordinate system, and then combines the conversion relationship between the camera coordinate system and the world coordinate system and the position of the front blade of the movable end effector (2) to complete the conversion of the three-dimensional coordinate value, that is, to obtain the final picking point coordinates.

Images