Apple picking robot based on visual guidance and picking method

Abstract

The application relates to the field of agricultural automation equipment, in particular to an apple picking robot based on visual guidance and a picking method, which recognizes apples and determines position information by using a camera, controls a mobile chassis to move in a large range according to position information of a control system, realizes height and angle adjustment in combination with a lifting column and a small arm, and then picks apples by using a fruit picking opening and conveys the apples to a collecting frame by using a mesh belt. The application greatly reduces the labor intensity of manual picking and improves the picking efficiency. Then, real-time pose information of the apples is acquired by using a camera, a mechanical arm is guided to adjust actions to complete apple tracking and grabbing, so that accurate and efficient picking work is realized, the labor intensity of manual picking is reduced, and the picking efficiency is improved. The robot can automatically recognize target apples and pick the apples, so that the picking efficiency and productivity are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural automation equipment, in particular to an apple picking robot based on visual guidance and a picking method. BACKGROUND

[0002] The picking season of fruits and vegetables is usually short, so it is necessary to complete the picking work in a short time after the fruits and vegetables are ripe. At present, manual picking is still the main method for picking apples, and traditional manual picking has the disadvantages of high cost, low efficiency and high labor intensity. At the same time, with the acceleration of urbanization, the shortage of rural labor is serious, and there are many safety problems in the picking process of the old people in rural areas.

[0003] In recent years, with the rapid development of robot technology, agricultural robots have gradually entered the field of agricultural production, effectively solving the above-mentioned problems of manual picking, liberating labor and improving labor productivity. However, when picking apples in windy, rainy and other weather conditions, the apples will sway, and the existing apple picking robots will have problems such as mis-grabbing, empty grabbing and collision during picking, resulting in large errors in the grabbing of the mechanical arm and affecting the picking efficiency. SUMMARY

[0004] In view of the problem that the apple picking robot in the prior art is affected by the grabbing error and the picking efficiency is affected, the present application provides an apple picking robot based on visual guidance and a picking method.

[0005] The present application is realized by the following technical solutions:

[0006] An apple picking robot based on visual guidance, comprising a robot body, a visual recognition system, a visual servo system, a control system, the robot body comprising a base, a lifting column, a horizontal rotating motor, a vertical rotating motor, a mechanical arm, a fruit picking port and a hose, the lifting column being fixedly installed on the robot base, the horizontal rotating motor being fixedly installed on the top of the lifting column, the vertical rotating motor being fixedly installed on the top of the horizontal rotating motor, the output shaft of the horizontal rotating motor being perpendicular to the output shaft of the vertical rotating motor, the output shaft of the vertical rotating motor being connected with the mechanical arm, the fruit picking port and the hose being both arranged at the end of the mechanical arm, and the hose being located below the fruit picking port;

[0007] The visual recognition system and the visual servo system are both arranged at the end of the mechanical arm, the output ends of the visual recognition system and the visual servo system are connected with the input end of the control system, and the output end of the control system is connected with the control end of the mechanical arm.

[0008] Preferably, the lifting column comprises a fixed tube, a lifting motor, a lifting screw rod and a lifting telescopic tube, the bottom of the fixed tube is fixedly connected with the base, the shell of the lifting motor is fixedly connected with the base, the output shaft of the lifting motor is vertically upward, the output shaft of the lifting motor is connected with one end of the lifting screw rod, the lifting telescopic tube is slidably arranged in the fixed tube, and the lifting telescopic tube is threadedly connected with the lifting screw rod.

[0009] Preferably, the shell of the horizontal rotation motor is fixed to the inner top of the lifting telescopic tube, the output shaft of the horizontal rotation motor penetrates through the top of the lifting telescopic tube and is connected with the rotating holder, and the vertical rotation motor is mounted on the rotating holder;

[0010] The rotating holder is provided with two spaced parallel support seats, one end of the mechanical arm is rotatably arranged between the two support seats through a rotating shaft, the shell of the vertical rotation motor is fixedly connected with any one of the support seats, and the output shaft of the vertical rotation motor is fixedly connected with the rotating shaft.

[0011] Preferably, the end of the mechanical arm comprises an arm fixed tube, an arm telescopic screw rod, an arm telescopic tube and an arm telescopic motor, one end of the arm is rotatably connected with the support seat, the arm telescopic motor is fixedly arranged in the arm fixed tube, the output shaft of the arm telescopic motor is fixedly connected with the arm telescopic screw rod, the arm telescopic tube is threadedly connected with the arm telescopic screw rod, and the fruit picking mouth is arranged at the other end of the arm telescopic tube away from the arm telescopic motor.

[0012] Preferably, the fruit picking mouth comprises a bracket and two opposite parallel air cylinders, the bracket is fixedly connected with one end of the arm telescopic arm away from the arm telescopic motor, the output ends of the two air cylinders are provided with arc-shaped clamping pieces, and the apple is fixed by relative movement of the clamping pieces.

[0013] A picking method of an apple picking robot based on visual guidance, comprising the following steps:

[0014] S1, the robot moves to the side of the apple tree, target detection is performed through a visual recognition system, an apple tracking template is obtained in the system, an estimated pose thereof is sent to a control system, the control system drives a mechanical arm to reach a specified area according to the target estimated pose;

[0015] S2, after the mechanical arm is in place, a visual tracking of a to-be-grasped apple is performed through an improved SiamRPN++ tracking model in a visual servo system to obtain real-time pose information;

[0016] S3, apple coordinates are converted from a visual coordinate system to a mechanical arm coordinate system through hand-eye calibration;

[0017] S4, the mechanical arm is adjusted in real time according to a pose error to complete accurate tracking of the apple, and the apple is picked, and the picked apple is transported to subsequent equipment through a hose of the picking mouth.

[0018] Preferably, in S2, the improved SiamRPN++ tracking model comprises a feature extraction network, a target prediction network and a background feature perception module, the feature extraction network adopts a five-stage ResNet-50 as a feature extraction backbone network, and a SimAM non-parameter attention module is introduced to focus on extracting target picking apple features; the target prediction network is used for predicting the pose of the apple, after the network receives the target features, the target confidence and the target bounding box are obtained through the fusion of three classification branches and regression branches after further processing; the background feature perception module detects whether the target is occluded based on the confidence of the apple occlusion detection method to define a confidence threshold to distinguish whether the target is occluded; in each tracking period, the mean value of the confidence is calculated, and when the mean value is lower than the threshold , the occlusion occurs.

[0019] Preferably, the SimAM non-parameter attention module combines a spatial attention mechanism and a channel attention mechanism, and the last three-stage features obtained by the feature extraction network in SiamRPN++ are enhanced through SimAM, so that the apple target can be better focused without introducing additional parameters.

[0020] Preferably, the RPN module comprises a classification branch and a regression branch, the target prediction network changes the channel dimension of the template image features and the search image features through a convolution layer, and then inputs the features into the two branches as inputs, and then performs correlation calculation on the two branches, and through training data and minimizing a loss function, the corresponding target prediction network can be obtained, the prediction maps of the three networks are combined by weighting and summing, the confidence of each prediction region corresponding to the position is obtained, and the region with the best confidence score is taken as the final tracking region.

[0021] Preferably, the background feature perception module comprises initialization, feature updating and feature perception.

[0022] In the initial stage of target tracking, the algorithm first locks the initial frame, denoted as , extracts a global scale invariant feature transformation feature point set in the frame, and records the target position in the initial frame.

[0023] Feature updating is to determine whether to update the current feature for the moving apple by evaluating the richness of the feature points and the position change of the bounding box;

[0024] Feature perception is that the algorithm matches the current feature information with the previously stored feature information to judge the overall motion trend and spatial position change of the feature.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] This invention discloses a vision-guided apple-picking robot. It uses a camera to identify apples and determine their location. Based on this location information, a control system moves the mobile chassis over a wide range, adjusting the height and angle using a lifting column and forearm. The apples are then picked through a picking opening and conveyed to a collection box via a conveyor belt. This invention significantly reduces the labor intensity of manual picking and improves efficiency. The robot then uses a camera to acquire real-time apple pose information, guiding the robotic arm to adjust its movements to track and grasp the apples, thus achieving accurate and efficient picking. This robot can automatically identify and pick target apples, improving picking efficiency and productivity.

[0027] This invention discloses a visually guided apple-picking robot picking method, which is a real-time visually guided precision tracking algorithm. The SiamRPN++ algorithm of the visual servo system is used to visually track the apple to be picked and obtain real-time pose information, thereby improving the accuracy and efficiency of picking.

[0028] Furthermore, the apple pose is updated in real time via vision. An apple tracking model based on SiamRPN++ is constructed, and the SimAM self-attention mechanism is introduced to improve the tracker's ability to extract apple features. An apple background feature perception algorithm is also proposed to effectively deal with the dynamic interference problem in the visual tracking process. Attached Figure Description

[0029] Figure 1 This is an overall structural diagram of a vision-guided apple-picking robot according to the present invention;

[0030] Figure 2 This is a cross-sectional view of a vision-guided apple-picking robot according to the present invention.

[0031] Figure 3 This is a flowchart of a vision-guided apple-picking robot according to the present invention.

[0032] In the diagram: 1. Base; 2. Lifting column; 21. Lifting telescopic tube; 22. Lifting motor; 23. Lifting screw; 3. Horizontal rotary motor; 4. Vertical rotary motor; 5. Robotic arm; 51. Forearm telescopic tube; 52. Forearm telescopic motor; 53. Forearm telescopic screw; 6. Camera; 7. Fruit picking opening; 8. Flexible hose; 9. Support; 10. Cylinder; Detailed Implementation

[0033] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0034] This invention discloses a vision-guided apple-picking robot, referring to... Figure 1, including a robot body, a visual recognition system, a visual servo system, and a control system.

[0035] The robot body includes a base, a lifting column, a horizontal rotation motor, a vertical rotation motor, a mechanical arm, a fruit picking mouth, and a hose.

[0036] The horizontal rotation motor is fixedly installed on the top of the lifting column, and the vertical rotation motor is fixedly installed on the top of the horizontal rotation motor.

[0037] The horizontal rotation motor is fixedly installed on the top of the lifting column, and the vertical rotation motor is fixedly installed on the top of the horizontal rotation motor.

[0038] The vertical rotation motor is fixedly connected with any one of the support seats, and the output shaft of the vertical rotation motor is fixedly connected with the rotating shaft.

[0039] The fruit picking mouth and the hose are both arranged at the end of the mechanical arm, and the hose is located below the fruit picking mouth.

[0040] The visual recognition system and the visual servo system are both arranged at the end of the mechanical arm, the output ends of the visual recognition system and the visual servo system are connected with the input end of the control system, and the output end of the control system is connected with the control end of the mechanical arm.

[0041] The apple picking robot based on visual guidance adjusts the working height of the robot by vertically lifting the telescopic pipe through the lifting motor and the lifting screw rod, the horizontal rotation motor is used for driving the robot body to rotate in the horizontal plane, and the vertical rotation motor is used for realizing the rotation of the mechanical arm in the vertical plane, the telescopic motor drives the forearm telescopic pipe to realize the telescoping of the mechanical arm, so as to approach the apple picking area, the visual recognition system and the visual servo system are used for capturing the position information of the apple in real time, and sending the information to the control system, and the control system is used for receiving the apple position information and driving the mechanical arm to pick accurately according to the received information.

[0042] The apple picking method of the apple picking robot based on visual guidance comprises the following steps: Figure 3

[0043] S1, the robot moves to the side of the apple tree, target detection is carried out through the visual recognition system, the apple tracking template is obtained in the system, and the estimated pose thereof is sent to the control system, and the control system drives the mechanical arm to reach the specified area according to the target estimated pose.

[0044] S2, after the mechanical arm is in place, the improved SiamRPN++ tracking model in the visual servo system is used to track the apple to be grabbed to obtain real-time pose information.

[0045] The improved SiamRPN++ tracking model comprises a feature extraction network, a target prediction network and a background feature perception module, the feature extraction network adopts a five-stage ResNet-50 as a feature extraction backbone network, and introduces a SimAM non-parameter attention module to focus on extracting target apple features; the target prediction network is used for predicting the pose of the apple, after the network receives the target features, the target confidence and the target bounding box are obtained after three classification branches and regression branches are fused and further processed; the background feature perception module detects whether the target is blocked based on a confidence-based apple occlusion detection method, and defines a confidence threshold to distinguish whether the target is blocked; in each tracking cycle, the mean value of the confidence is calculated, and when the mean value is lower than the threshold , the occlusion occurs.

[0046] Specifically, the SimAM non-parameter attention module combines a spatial attention mechanism and a channel attention mechanism, and the energy function formula is as follows:

[0047] (1)

[0048] Wherein, and are target neurons and other neurons of the input features; is an index in space, ​is the response value of the corresponding position, and is the weight and offset transformation, is the regularization coefficient, is the linear change on the neuron, is the number of neurons on the channel. By calculating the variance of all neurons and the mean The minimum energy function can be obtained:

[0049] (2)

[0050] From equation (2), the lower the energy, the higher the distinction between the neurons in the t region and other regions. The scaling algorithm is used to optimize the features:

[0051] (3)

[0052] where E is all , the sigmoid function is used to limit the larger value in E, X is the original feature, is the optimized feature. By SimAM enhancing the last three-stage features obtained by the feature extraction network in SiamRPN++, the apple target is better focused without introducing additional parameters.

[0053] The RPN module includes a classification branch and a regression branch. The target prediction network changes the channel dimension of the template image feature and the search image feature through the convolution layer, and then inputs them into the two branches, and performs correlation calculation on the two branches respectively. Specifically:

[0054] (4)

[0055] (5)

[0056] where, is the correlation calculation, contains a 2k channel, which represents whether the anchor point at the corresponding position is activated. contains a 4k channel vector, which represents the distance between the measured anchor frame and the corresponding actual frame. When training the network using multiple anchors, the classification loss uses cross-entropy loss, and the smooth L1 loss with normalized coordinates is used for regression. The normalized distance is passed through the smooth L1 loss:

[0057] (6)

[0058] where, is the difference between the predicted value and the true value, is the parameter of control smoothness. The final loss function is:

[0059] (7)

[0060] where, is the cross-entropy loss function, which measures the consistency between the model prediction and the ground truth, is the regression loss function, which measures the difference between the predicted position parameter and the true value. The formula is as follows:

[0061] (8)

[0062] (9)

[0063] where, is the true label, is the probability of the model predicting an apple. is the difference between the predicted position parameter and the true value.

[0064] By training the data and minimizing the loss function, the corresponding target prediction network can be obtained. By combining the prediction maps of the three networks through weighted sum, the confidence score of each prediction region corresponding to the position is obtained, and the region with the best confidence score is taken as the final tracking region.

[0065] In the process of visual tracking of apples, branches and leaves may cause certain occlusion to the apple target. When the target is occluded, the tracker will lose part of the target features, leading to tracking failure. To solve this problem, a background feature perception module is designed to correct the pose when the target is occluded. An apple occlusion detection method based on confidence is proposed to detect whether the target is occluded. A confidence threshold is defined to distinguish whether the target is occluded; in each tracking cycle, the mean value of the confidence is calculated, and when the mean value is lower than the threshold , occlusion occurs.

[0066] The background feature perception module includes initialization, feature update and feature perception.

[0067] Among them, the initialization is in the initial stage of target tracking. The algorithm first locks the initial frame, denoted as ; in this frame, the global scale-invariant feature transformation feature point set is extracted, and the target position in the initial frame is recorded.

[0068] Feature update is to determine whether to update the current feature for moving apples by evaluating the richness of feature points and the position change of the bounding box. The update criterion is:

[0069] (10)

[0070] in, Let be the current frame at time t, and let its set of background feature points be: The target box position is , The number of feature sets, It is a boolean function that returns True when the target bounding box is at the edge of the image. To adjust the parameters, adjustments need to be made based on the on-site situation. The current feature point and target bounding box positions are updated only when the richness of the current feature points exceeds the product of the adjustment parameter and the number of stored feature points, and the target is not currently at the image edge.

[0071] When an apple is obscured, only a small portion of its feature points are visible, or they are completely invisible, but a large amount of background feature information still exists around it. Feature perception is an algorithm that matches current feature information with previously stored feature information to determine the overall movement trend and spatial position changes of the features.

[0072] Assume the set of feature points at time t is The currently stored set of historical feature points is For each feature point, obtain its feature descriptor vector. .for Each feature point in the data is in Found corresponding Make it satisfy ,in Let be the Euclidean distance between vectors; the two feature points with the smallest distance are the most closely matched. For each pair of matched points... ,have , where H is the homography matrix. The coordinates of the stored target bounding box in the historical frame are . ,in It is the coordinate of the top left corner of the target. If the width and height of the bounding box are given, then the corner points can be converted into a set of points in a homogeneous coordinate system:

[0073] (11)

[0074] Transform the point set using the obtained homography matrix:

[0075] Transformed_points=H (12)

[0076] The transformed point set represents the target's position in the latest frame. The smallest bounding box enclosing these points is then obtained. To ensure the accuracy and continuity of target tracking, the updated center coordinates of the target bounding box and its original width and height are calculated:

[0077] (13)

[0078] The obtained box is the target frame obtained by the background perception module, and the target position of visual tracking failure can be corrected through the target frame.

[0079] S3, the apple coordinates are converted from the visual coordinate system to the mechanical arm coordinate system through hand-eye calibration, so that the mechanical arm can accurately grasp the apple.

[0080] S4, the mechanical arm is adjusted in real time according to the pose error to complete accurate tracking of the apple, and the apple is picked, and the picked apple is transported to subsequent equipment by the hose of the picking port.

[0081] The picking method of the apple picking robot based on visual guidance firstly obtains the initial pose of the target to be grasped and a tracking template through a visual recognition module, and a control system is used for strategy distribution, so that the apple is distributed to the corresponding mechanical arm for grasping; when the target apple enters the grasping working area of the mechanical arm, a single target tracking model based on a twin network is used to obtain real-time pose information of the apple, and the mechanical arm action is adjusted in real time to complete grasping.

[0082] The specific process is: the fruit and vegetable image of the current position to be picked is shot, the fruit tree image is subjected to target detection in a visual recognition system, an apple tracking template is obtained and its pose is estimated and sent to a control system, the control system drives the mechanical arm to reach the specified area according to the estimated pose of the target. After the mechanical arm reaches the position, the SiamRPN++ algorithm of the visual servo system is used to track the apple to be picked to obtain real-time pose information; then the apple coordinates are converted from the visual coordinate system to the mechanical arm coordinate system through hand-eye calibration; finally, the mechanical arm is adjusted in real time according to the pose error to complete accurate tracking of the apple, and the mechanical arm is driven to pick the apple accurately; after the target apple is picked by the picking port, the apple is sent to the storage box through the mesh belt. Through the integration of advanced visual recognition, visual servo and control system, efficient and accurate apple picking is realized, and the labor intensity is also reduced.

[0083] The above only describes the preferred embodiments of the present application, and does not limit the technical solutions of the present application in any way, and those skilled in the art should understand that the technical solutions can be modified and replaced in several simple ways without departing from the spirit and principles of the present application, and these modifications and replacements also belong to the protection scope covered by the claims.

Claims

1. A picking method of a vision-guided apple picking robot, characterized by, The method comprises the following steps: S1, the robot moves to the side of the apple tree, detects the target through the visual recognition system, acquires the apple tracking template in the system, and sends the estimated pose thereof to the control system, which drives the mechanical arm to reach the specified area according to the target estimated pose; S2, after the mechanical arm is in place, the improved SiamRPN++ tracking model in the visual servo system is used to track the apple to be picked to obtain real-time pose information; The improved SiamRPN++ tracking model comprises a feature extraction network, a target prediction network and a background feature perception module, the feature extraction network adopts a five-stage ResNet-50 as a feature extraction backbone network, and introduces a SimAM parameter-free attention module to focus on extracting the picking apple features of the target; the target prediction network is used for predicting the pose of the apple, and after the network receives the target features, the target confidence and the target bounding box are obtained through the fusion of three classification branches and regression branches after further processing; The background feature perception module detects whether the target is blocked based on a confidence-based apple blockage detection method, and defines a confidence threshold to distinguish whether the target is blocked; in each tracking period, the confidence mean value is calculated, and when the mean value is lower than the threshold , the blockage occurs; S3, the apple coordinates are converted from the visual coordinate system to the mechanical arm coordinate system through hand-eye calibration; S4, the mechanical arm is adjusted in real time according to the pose error to complete accurate tracking of the apple, and the picked apple is transported to subsequent equipment by the hose of the picking port.

2. The picking method according to claim 1, characterized in that, The SimAM parameter-free attention module combines spatial attention mechanism and channel attention mechanism, and enhances the last three-stage features obtained by the feature extraction network in SiamRPN++ through SimAM, so that the apple target can be better focused without introducing additional parameters.

3. The picking method according to claim 1, characterized in that, The RPN module includes classification and regression branches, and the target prediction network uses template graph features. and search graph features After changing the channel dimension by using convolutional layers as input to two branches, correlation calculations are performed on the two branches respectively. By training data and minimizing the loss function, the corresponding target prediction network can be obtained. By weighted summing and combining the prediction maps of the three networks, each prediction region corresponds to the confidence level of the corresponding position. The region with the best confidence score is taken as the final tracking region.

4. The picking method according to claim 1, characterized in that, The background feature perception module comprises initialization, feature update and feature perception; Wherein, initialization in the initial stage of target tracking, the algorithm first locks the initial frame, denoted as ; In the frame, the global scale invariant feature transform feature point set is extracted, and the target position in the initial frame is recorded; The feature update is to determine whether to update the current features by evaluating the richness of the feature points and the position change of the bounding box for the moving apple; The feature perception is to match the current feature information with the previously stored feature information to determine the overall motion trend and spatial position change of the features.

5. A vision-guided apple picking robot for implementing the picking method according to any one of claims 1 to 4, characterized in that, The robot body comprises a base, a lifting column, a horizontal rotating motor, a vertical rotating motor, a mechanical arm, a fruit picking port and a hose, the lifting column is fixedly installed on the robot base, the horizontal rotating motor is fixedly installed on the top of the lifting column, the vertical rotating motor is fixedly installed on the top of the horizontal rotating motor, the output shaft of the horizontal rotating motor is perpendicular to the output shaft of the vertical rotating motor, the output shaft of the vertical rotating motor is connected with the mechanical arm, the fruit picking port and the hose are arranged at the end of the mechanical arm, and the hose is located below the fruit picking port; The visual recognition system and the visual servo system are arranged at the end of the mechanical arm, the output ends of the visual recognition system and the visual servo system are connected with the input end of the control system, and the output end of the control system is connected with the control end of the mechanical arm.

6. The vision-guided apple picking robot according to claim 5, wherein, The lifting column comprises a fixed tube, a lifting motor, a lifting screw and a lifting telescopic tube, the bottom of the fixed tube is fixedly connected with the base, the shell of the lifting motor is fixedly connected with the base, the output shaft of the lifting motor is vertically upward, the output shaft of the lifting motor is connected with one end of the lifting screw, and the lifting telescopic tube is slidingly arranged in the fixed tube and is threadedly connected with the lifting screw.

7. The vision-guided apple picking robot according to claim 5, wherein, The housing of the horizontal rotating motor is fixed to the inner top of the lifting telescopic pipe, the output shaft of the horizontal rotating motor penetrates the top of the lifting telescopic pipe and is connected with the rotating holder, and the vertical rotating motor is installed on the rotating holder; The rotating holder is provided with two parallel support seats, one end of the mechanical arm is rotatably arranged between the two support seats, the housing of the vertical rotating motor is fixedly connected with any support seat, and the output shaft of the vertical rotating motor is fixedly connected with the rotating shaft.

8. The vision-guided apple picking robot according to claim 7, wherein, The end of the mechanical arm comprises an arm fixing pipe, an arm telescopic screw rod, an arm telescopic pipe and an arm telescopic motor, one end of the arm is rotatably connected with the support seat, the arm telescopic motor is fixed in the arm fixing pipe, the output shaft of the arm telescopic motor is fixedly connected with the arm telescopic screw rod, the arm telescopic pipe is threadedly connected with the arm telescopic screw rod, and the fruit picking opening is arranged at the other end of the arm telescopic pipe away from the arm telescopic motor.

9. The vision-guided apple picking robot according to claim 8, characterized in that, The fruit picking opening comprises a support and two opposite parallel air cylinders, the support is fixedly connected with one end of the arm telescopic arm away from the arm telescopic motor, the output ends of the two air cylinders are provided with arc-shaped clamping pieces, and the apples are fixed by relative movement of the clamping pieces.

Images