Deep learning-based skewered tomato composite picking method
Through the YOLOv8 model, the picking points are identified and positioned in the tomato skewers, combined with the ripening degree judgment, the complex structure and occlusion problems of tomato skewers are solved, and the precise picking of tomatoes is achieved, and the picking efficiency and quality are improved.
Patent Information
- Application Number
- CN202411129143.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-12
AI Technical Summary
The tomato bunches have complex structures, and there may be occlusion and overlap between the fruits, which leads to difficulty in detection and picking, and it is difficult for the prior art to achieve accurate identification and efficient picking.
The deep learning object detection and key point detection model based on YOLOv8 is adopted to identify tomato strings and picking points through images, combined with maturity judgment, and the best picking method is selected to achieve accurate identification and positioning of tomatoes.
It improves the efficiency and quality of tomato picking, reduces fruit damage, is suitable for large-scale agricultural production, and reduces labor intensity and cost.
Smart Images

Figure CN118941960A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and robot picking of tomatoes on a vine, and in particular to a composite picking method of tomatoes on a vine based on deep learning. Background Art
[0002] With the development of modern agricultural technology, automation and intelligence are increasingly being used in agriculture. As a common economic crop, tomatoes require precise judgment and operation during their picking process. At present, target detection algorithms based on deep learning have shown great potential in fruit recognition and positioning. YOLOv8 is a new version of the YOLO series with higher detection accuracy and speed, providing new technical support for intelligent picking in the agricultural field.
[0003] The structure of tomato bunches is complex, and there may be occlusion and overlap between fruits, which brings challenges to detection and picking. With the advanced object detection and key point detection capabilities of deep learning, these problems can be effectively solved, and accurate identification and positioning of tomato fruits can be achieved, thus providing reliable technical support for intelligent picking.
[0004] In view of the shortcomings of the prior art, the present invention provides a composite picking method for tomatoes on a bunch based on YOLOv8 to solve the problem of selecting a suitable picking method due to the complex structure of the tomato bunch and possible occlusion and overlap between the fruits. Summary of the invention
[0005] In view of the defects in the prior art, the object of the present invention is to provide a composite picking method of tomatoes on a string based on deep learning.
[0006] A composite method for picking tomatoes on a vine based on deep learning provided by the present invention comprises the following steps:
[0007] Step S1: Collect the tomato images, and label the tomatoes and divide the data set according to the target detection model and key point detection model respectively;
[0008] Step S2: the target detection model identifies the rectangular frame of the tomato bunches in the image and the individual small tomatoes in each tomato bunch detection target image, and classifies the tomatoes into ripe fruits and unripe fruits according to the maturity of the tomato fruit;
[0009] Step S3: using a key point detection model to identify the rectangular frame of tomato bunches, picking points and key points in the image;
[0010] Step S4: according to the result of the tomato bunch target detection in step S2, the maturity of the tomato bunch is determined. If the maturity is determined to be greater than or equal to the first set value, step S5 is executed; if the maturity is between the first set value and the second set value, step S6 is executed; if the maturity is less than or equal to the second set value, no processing is performed;
[0011] Step S5: judging whether the tomato bunches in the detected image are ripe and combining the picking point information in step S3, picking the whole bunch of tomatoes;
[0012] Step S6: Pick the tomatoes on the bunch individually based on the ripe fruits identified in the target detection model in step S2 and the key point information in step S3.
[0013] Preferably, the target detection model adopts a target detection model based on a YOLOv8 network; and the key point detection model adopts a key point detection model based on a YOLOv8 network.
[0014] Preferably, in step 2, the target detection model based on the YOLOv8 network recognizes the rectangular frame of the tomato bunches in the image and the individual small tomatoes in each tomato bunch detection target image, and divides the tomato into ripe fruits and unripe fruits according to the maturity of the tomato grains; the ripe fruits are marked in red, and the unripe fruits are marked in green.
[0015] Preferably, in step S3, the picking points include the connection between the main stem and the fruit stalk, the connection between the fruit stalk and the first tomato from top to bottom, and the connection between the fruit stalk and the last tomato from top to bottom; the key points include the centroid of each tomato in each bunch.
[0016] Preferably, in step S4, according to the result of the tomato bunch target detection in S2, a statistical function is called to count the number of tomato fruits in the mature and immature stages of each bunch of tomatoes in the detection image, and the number of ripe fruits of each bunch of tomatoes is divided by the total number of tomatoes in each bunch to obtain the maturity of the bunch of tomatoes.
[0017] Preferably, in step S4, the first set value is 90% and the second set value is 50%.
[0018] Preferably, the step S1 comprises: collecting images of tomatoes on a string in a greenhouse, sorting them, annotating the tomato images using the Labelme tool, and then performing data enhancement, and establishing two image data sets based on the collected tomato images, wherein the division ratio of each data set is: training set: validation set = 8:2;
[0019] Through the tomato target detection model based on the YOLOv8 network, the target detection model is trained using the corresponding data set. After the training is completed, the tomato target detection model based on the YOLOv8 network is obtained;
[0020] Through the key point detection model of tomato on a vine based on the YOLOv8 network, the corresponding data set is used to train the target detection model. After the training is completed, the key point detection model of tomato on a vine based on the YOLOv8 network is obtained.
[0021] Preferably, step S3 includes the following:
[0022] The backbone layer in the YOLOv8-based object detection model extracts features from the input tomato image, where the backbone network includes the Conv module, the C2f module, and the SPPF module;
[0023] The head layer in the YOLOv8-based target detection model defines the detection head of the model, which is used for the network structure of the final target detection and produces the final detection result, including the upsampling layer, feature map concatenation layer, C2f module and detection module.
[0024] Preferably, the key point detection model of YOLOv8 has an additional branch head in the head layer for key point detection compared to the target detection model of YOLOv8.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] 1. This invention uses a deep learning target detection algorithm to efficiently and accurately identify the color characteristics of single tomatoes and whole bunches of tomatoes, thereby judging the maturity of tomatoes. Compared with traditional image processing methods, YOLOv8 can still maintain high detection accuracy under complex backgrounds and lighting conditions.
[0027] 2. The present invention uses a depth camera and a YOLOv8 key point detection model to obtain the image coordinates and depth information of the tomato stalk picking point and the fruit center. This method can provide accurate three-dimensional spatial positioning, provide an accurate target position for the tomato picking robot, and reduce fruit damage during the picking process.
[0028] 3. The present invention combines the target detection model and the key point detection model to provide a picking scheme for single tomatoes and whole bunches of tomatoes. According to the maturity and spatial position information of the tomatoes, the picking robot can select the best picking method (single or whole bunch), thereby improving the picking efficiency and meeting different picking needs.
[0029] 4. The present invention significantly improves the efficiency of tomato picking through automated tomato detection and picking technology, and reduces the labor intensity and cost of manual picking. This method is not only suitable for large-scale agricultural production, but also can improve the quality of fruit picking while reducing the damage rate to the fruit, thereby increasing the commodity value of tomatoes. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0031] Figure 1 A schematic diagram of the method provided by the present invention;
[0032] Figure 2 This is a schematic diagram of target detection and annotation provided by the present invention;
[0033] Figure 3 A schematic diagram of key point detection and annotation provided by the present invention;
[0034] Figure 4 The YOLOv8 target detection network structure diagram provided by the present invention;
[0035] Figure 5 This is a diagram of the YOLOv8 key point detection network structure provided by the present invention. DETAILED DESCRIPTION
[0036] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0037] like Figures 1 to 5 As shown, a composite method for picking tomatoes on a string based on deep learning provided by the present invention comprises the following steps:
[0038] Step S1, collecting bunch tomato images through a depth camera, and annotating and dividing the images according to the target detection and key point detection models respectively;
[0039] Step S2, using the target detection model based on the YOLOv8 network to identify the rectangular frame of the tomato bunch in the image and the individual small tomatoes in each tomato bunch detection target image, and classifying the tomato fruit into ripe fruit and unripe fruit according to the maturity of the tomato fruit; "red" is the ripe fruit, i.e., red fruit, and "green" is the unripe fruit, i.e., green fruit;
[0040] Step S3, using the YOLOv8 network-based key point detection model to identify the rectangular frame of the tomato bunches in the image, the picking points and the key points, the picking points including the connection between the main stem and the fruit stalk, the connection between the fruit stalk and the first tomato from top to bottom, and the connection between the fruit stalk and the last tomato from top to bottom, and the key points including the centroid of each tomato in each bunch;
[0041] Step S4, according to the result of the tomato bunch target detection in step S2, call the statistical function to count the number of tomato fruits in the mature and immature stages of each bunch of tomatoes in the detection image, and divide the number of mature fruits of each bunch of tomatoes by the total number of tomatoes in each bunch, so as to obtain the maturity of the bunch of tomatoes. If the maturity is greater than or equal to 90%, it is judged as yes, and S5 is executed. If the maturity is between 50% and 90%, S6 is executed. Otherwise, no processing is performed.
[0042] Step S5, judging that the tomato bunch in the detection image is ripe and combining the picking point information of the key point detection model in S3, the robot arm can be controlled to pick the whole bunch of tomatoes by means of a sleeve;
[0043] Step S6, combining the red fruit identified in the target detection model in step S2 and the key point information of the key point detection model in step S3, the robot arm can be controlled to calculate the path to achieve single-grain picking of the bunch tomatoes;
[0044] In one embodiment: images of tomatoes on a string in a greenhouse are collected, sorted, and labeled using the Labelme tool, and two image data sets are established based on the collected tomato images, and the division ratio of each data set is: training set: validation set = 8:2;
[0045] Through the tomato target detection model based on the YOLOv8 network, the target detection model is trained using the corresponding data set. After the training is completed, the tomato target detection model based on the YOLOv8 network is obtained;
[0046] Through the key point detection model of tomato on a vine based on the YOLOv8 network, the corresponding data set is used to train the target detection model. After the training is completed, the key point detection model of tomato on a vine based on the YOLOv8 network is obtained;
[0047] In one embodiment: extracting features from an input tomato image based on a backbone layer in a YOLOv8 object detection model, wherein the backbone network adopts a structure similar to CSPDarknet, including Conv, C2f modules (including multiple convolutional layers and feature fusion layers) and SPPF module (spatial pyramid pooling module);
[0048] The head layer in the YOLOv8-based target detection model defines the detection head of the model, that is, the network structure used for final target detection, which produces the final detection result, including the upsampling layer (nn.Upsample), the feature map concatenation layer (Concat), the C2f module, and the detection (Detect) module;
[0049] Compared with the target detection model of YOLOv8, the key point detection model of YOLOv8 has an additional branch head in the head layer for key point detection.
[0050] The following is a more detailed description:
[0051] Step S1: Use the depth camera to collect images of tomatoes in the greenhouse, sort them out, and use the Labelme tool to annotate the tomato images. The annotated images are as follows: Figure 2 , 3 As shown, two image datasets are established based on the collected tomato images, and the division ratio of each dataset is: training set: verification set = 8:2;
[0052] Through the tomato target detection model based on the YOLOv8 network, the target detection model is trained using the corresponding data set. During the training process, the stochastic gradient descent (SGD) optimizer and multi-scale training strategy are used to improve the robustness and generalization ability of the model. After the training is completed, the tomato target detection model based on the YOLOv8 network is obtained.
[0053] Through the key point detection model of tomato on a vine based on the YOLOv8 network, the corresponding data set is used to train the target detection model. After the training is completed, the key point detection model of tomato on a vine based on the YOLOv8 network is obtained;
[0054] Step S2, using the target detection model based on the YOLOv8 network to identify the rectangular frame of the tomato bunch in the image and the individual small tomatoes in each tomato bunch detection target image, and classifying the tomato fruit into ripe fruit and unripe fruit according to the maturity of the tomato fruit; "red" is the ripe fruit, i.e., red fruit, and "green" is the unripe fruit, i.e., green fruit;
[0055] Among them, Figure 4 As shown in the figure, the YOLOv8 target detection model is divided into four parts: input, backbone, neck and head. The backbone in the target detection model based on YOLOv8 extracts features from the input tomato image. The backbone adopts a structure similar to CSPDarknet, including Conv, C2f modules (including multiple convolutional layers and feature fusion layers) and SPPF module (spatial pyramid pooling module) to improve the efficiency and accuracy of feature extraction; the Conv module includes convolutional layer, batch normalization layer and activation function layer; the C2f module is mainly designed with reference to the ideas of C3 module and ELAN, and divides the input feature map into two parts in one dimension, which reduces the amount of calculation and enriches the gradient flow information, which helps to improve the nonlinear representation ability of the model. The SPPF module connects the backbone network and the neck network, extracts the tomato string features through different convolution kernels, and then fuses the feature information of different scales;
[0056] The neck network (Neck) uses the PAN-FPN structure. The FPN structure samples from top to bottom to extract semantic features, improving the multi-scale detection accuracy of the model. The PAN structure samples from bottom to top to extract rich location information and splices it with high-resolution feature maps to enhance target positioning capabilities.
[0057] The head network (Head) defines the detection head of the model, that is, the network structure used for final target detection, which produces the final detection result, including the upsampling layer (nn.Upsample), the feature map concatenation layer (Concat), the C2f module and the detection (Detect) module, and finally outputs the target category, confidence and target box coordinates; the target detection model output tensor shape is S,S,(B×5+C), where S is the feature map size, B is the number of bounding boxes predicted by each unit, each bounding box has 5 parameters (x, y, w, h, confidence), and C is the number of categories;
[0058] Step S3, using the YOLOv8 network-based key point detection model to identify the rectangular frame of the tomato bunches in the image, the picking points and the key points, the picking points including the connection between the main stem and the fruit stalk (a), the connection between the fruit stalk and the first tomato from top to bottom (b), and the connection between the fruit stalk and the last tomato from top to bottom (c), and the key points including the centroid of each tomato in each bunch;
[0059] Among them, Figure 5 As shown in the figure, the key point detection model of YOLOv8 has an additional branch head in the head layer compared to the target detection model of YOLOv8, which is used for key point detection. The branch head consists of three convolutional layers. The sizes of the convolution kernels are 3×3, 3×3, and 1×1, respectively, and the step sizes are 1. The first two convolutional layers are activated by the activation function (ReLU), and the last convolutional layer does not need to be activated by the activation function, and directly outputs the key point prediction results, including the normalized coordinates and confidence of the key points relative to the input image.
[0060] The model outputs the coordinates of the key points and target boxes and the corresponding confidence; the shape is S,S,(K×3)S, where S is the feature map size, K is the number of key points for each target, and each key point has 3 parameters (x, y, confidence);
[0061] Step S4, according to the result of the tomato bunch target detection in step S2, call the statistical function to count the number of tomato fruits in the mature and immature stages of each bunch of tomatoes in the detection image, and divide the number of mature fruits of each bunch of tomatoes by the total number of tomatoes in each bunch, so as to obtain the maturity of the bunch of tomatoes. If the maturity is greater than or equal to 90%, it is judged as yes, and S5 is executed. If the maturity is between 50% and 90%, S6 is executed. Otherwise, no processing is performed.
[0062] Step S5, judging that the tomato bunches in the detection image are ripe, combining the picking point information of the key point detection model in step S3, controlling the robot arm to reach the connection point (c) between the fruit stalk and the last tomato from top to bottom at -5 cm on the y coordinate, and moving upward along the positive direction of the y axis to the center of the connection point between the main stem and the fruit stalk (a) and the connection point between the fruit stalk and the first tomato from top to bottom (b) by means of a sleeve, and controlling the end effector to perform a shearing operation, thereby realizing the picking of the tomato bunches;
[0063] Step S6: Combining the red fruit identified in the target detection model in step S2 with the key point information of the key point detection model in step S3, the robot arm can be controlled to calculate the path to achieve single-grain picking of tomatoes on the bunch.
[0064] In summary, in the tomato picking process, the present invention realizes the accurate identification and positioning of tomatoes through the advanced YOLOv8 target detection and key point detection technology, provides reliable technical support for the tomato picking robot, and has significant application value and broad market prospects.
[0065] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.
Claims
1. A composite picking method for tomatoes on a string based on deep learning, It is characterized in that The steps include: Step S1: Collect the tomato images, and label the tomatoes and divide the data set according to the target detection model and key point detection model respectively; Step S2: the target detection model identifies the rectangular frame of the tomato bunches in the image and the individual small tomatoes in each tomato bunch detection target image, and classifies the tomatoes into ripe fruits and unripe fruits according to the maturity of the tomato fruit; Step S3: using a key point detection model to identify the rectangular frame of tomato bunches, picking points and key points in the image; Step S4: according to the result of the tomato bunch target detection in step S2, the maturity of the tomato bunch is determined. If the maturity is determined to be greater than or equal to the first set value, step S5 is executed; if the maturity is between the first set value and the second set value, step S6 is executed; if the maturity is less than or equal to the second set value, no processing is performed; Step S5: judging whether the tomato bunches in the detected image are ripe and combining the picking point information in step S3, picking the whole bunch of tomatoes; Step S6: Pick the tomatoes on the bunch individually based on the ripe fruits identified in the target detection model in step S2 and the key point information in step S3.
2. The method for composite picking of tomatoes on a string based on deep learning according to claim 1, It is characterized in that The target detection model adopts a target detection model based on the YOLOv8 network; the key point detection model adopts a key point detection model based on the YOLOv8 network.
3. The method for composite picking of tomatoes based on deep learning according to claim 1, It is characterized in that In the step 2, the target detection model based on the YOLOv8 network recognizes the rectangular frame of the tomato bunch in the image and the individual small tomatoes in each tomato bunch detection target image, and divides the tomato into ripe fruits and unripe fruits according to the maturity of the tomato fruit; the ripe fruits are marked in red, and the unripe fruits are marked in green.
4. The method for composite picking of tomatoes based on deep learning according to claim 1, It is characterized in that In step S3, the picking points include the connection between the main stem and the fruit stalk, the connection between the fruit stalk and the first tomato from top to bottom, and the connection between the fruit stalk and the last tomato from top to bottom; the key points include the centroid of each tomato in each bunch.
5. The method for composite picking of tomatoes based on deep learning according to claim 1, It is characterized in that In step S4, according to the result of the tomato bunch target detection in S2, a statistical function is called to count the number of tomato fruits in the mature and immature stages of each bunch of tomatoes in the detection image, and the number of mature fruits of each bunch of tomatoes is divided by the total number of tomatoes in each bunch to obtain the maturity of the bunch of tomatoes.
6. The method for composite picking of tomatoes on a string based on deep learning according to claim 1, It is characterized in that In step S4, the first set value is 90% and the second set value is 50%.
7. The method for composite picking of tomatoes on a string based on deep learning according to claim 1, It is characterized in that The step S1 comprises: collecting images of tomatoes on a string in a greenhouse, sorting them, annotating the tomato images using the Labelme tool, and then performing data enhancement, and establishing two image data sets based on the collected tomato images, with each data set being divided into a training set: a validation set = 8:2; Through the tomato target detection model based on the YOLOv8 network, the target detection model is trained using the corresponding data set. After the training is completed, the tomato target detection model based on the YOLOv8 network is obtained; Through the key point detection model of tomato on a vine based on the YOLOv8 network, the corresponding data set is used to train the target detection model. After the training is completed, the key point detection model of tomato on a vine based on the YOLOv8 network is obtained.
8. The method for composite picking of tomatoes on a string based on deep learning according to claim 1, It is characterized in that Step S3 includes the following: The backbone layer in the YOLOv8-based object detection model extracts features from the input tomato image, where the backbone network includes the Conv module, the C2f module, and the SPPF module; The head layer in the YOLOv8-based target detection model defines the detection head of the model, which is used for the network structure of the final target detection and produces the final detection result, including the upsampling layer, feature map concatenation layer, C2f module and detection module.
9. The method for composite picking of tomatoes on a string based on deep learning according to claim 8, It is characterized in that Compared with the target detection model of YOLOv8, the key point detection model of YOLOv8 has an additional branch head in the head layer for key point detection.
Citation Information
Patent Citations
Plant and fruit detection algorithm of tomato picking robot
CN115984704A
Method for detecting maturity of string tomatoes based on improved YOLOv5 network
CN116030456A
Apple picking manipulator auxiliary grabbing method based on key point detection
CN117173244A
Polar coordinate type double-arm tomato picking robot
CN117796228A
Device for picking fruit
US20200396900A1
Cited By
Method for detecting maturity of string tomatoes based on improved YOLOv5 network
CN116030456A
Method for detecting maturity of string tomatoes based on improved YOLOv5 network
CN116030456B
Spatial heterogeneous visual positioning system of tomato picking robot
CN119772891A
Automatic tomato picking method and system based on machine vision
CN119790827A
Method and device for detecting picking key points of tomato fruits in red ripe stage based on multi-scale attention mechanism
CN119810473A
