Plant and fruit detection algorithm for a tomato picking robot
By optimizing a multi-task convolutional neural network model, collision-free tomato picking by a robot was achieved, solving the problems of detection efficiency in vision systems and deployment of embedded devices, and providing efficient visual information support.
Patent Information
- Application Number
- CN202310096462.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-10
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2043-02-10
AI Technical Summary
Existing vision systems for tomato harvesting robots struggle to efficiently acquire tomato posture and stem position, resulting in inefficient and collision-free harvesting. Furthermore, deep learning models are difficult to deploy in embedded devices.
A multi-task convolutional neural network model is adopted, which combines the YOLO V5s model with semantic segmentation branches and key point prediction parameters. Through sparse training, channel pruning and fine-tuning, the model is optimized to achieve tomato target, pose detection and stem semantic segmentation, thereby reducing model weights and speeding up detection.
It provides information support for collision-free tomato harvesting robots. The model is fast, accurate, and suitable for embedded devices, meeting the needs of complex greenhouse environments.
Smart Images
Figure CN115984704B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of machine vision, in particular to a plant and fruit detection algorithm for a tomato picking robot. BACKGROUND
[0002] With the aggravation of China's population aging, the shortage of agricultural labor force and the rise of labor cost. In order to solve this problem, agricultural robots are being widely concerned. The design of picking robot is to perceive the agricultural environment through various sensors, obtain perception information, and replace human to perform picking operation. Although picking robot has a foreseeable future in the field of agriculture, but at present it cannot compete with manual picking due to various technical limitations. Research shows that improving the performance of visual guidance control is conducive to breaking through the existing application bottleneck.
[0003] The vision system of the tomato picking robot faces two arduous tasks, namely the accurate acquisition of tomato posture and the accurate acquisition of stem position. Tomato posture and stem position can help determine the end effector posture of the robot to achieve collision-free picking. Comprehensive visual information acquisition is of great significance to the research and development of tomato robot. The existing visual algorithms in the field of agriculture mainly include the following categories: 1) only target detection algorithm; 2) only target detection and semantic segmentation; 3) only posture detection. No algorithm can efficiently and comprehensively complete tomato target detection, posture detection and stem semantic segmentation.
[0004] And the convolutional neural network has strong feature extraction ability and robustness, so the method based on deep learning has achieved good results in target detection and semantic segmentation field. However, in order to extract more accurate and effective features, researchers usually increase the network depth of deep learning model and increase the parameter quantity. The above reasons make the model have better performance, but also produce too many redundant convolution kernels, the calculation cost is expensive, it is difficult to deploy in embedded device, which limits the development of agricultural robot. SUMMARY
[0005] The purpose of the present application is to overcome the deficiencies in the above background art, and to provide a plant and fruit detection algorithm for a tomato picking robot, which should realize collision-free picking and has the characteristics of fast detection speed and high precision.
[0006] The technical solution of the present application is:
[0007] A plant and fruit detection algorithm for a tomato picking robot, comprising the following steps:
[0008] 1-1) Collecting tomato image;
[0009] 1-2) Inputting the tomato image into a multi-task convolutional neural network model for processing;
[0010] 1-3) Get the bounding box, key points and stem of the tomato on the tomato image;
[0011] The method for establishing the multi-task convolutional neural network model is:
[0012] 2-1) Constructing a data set: constructing a tomato key point data set and a tomato stem data set;
[0013] 2-2) Data preprocessing: normalizing and data enhancing the tomato key point data set and the tomato stem data set;
[0014] 2-3) Selecting YOLO V5s as a basic network model; adding key point prediction parameters in the detection branch of the Head network of the basic network model to realize bounding box regression and key point prediction, adding a semantic segmentation module in the Head network of the basic network model to realize tomato stem semantic segmentation; the splicing layer of the 16th layer of the basic network model sequentially passes through a convolutional layer, an up-sampling layer, a C3 module, a convolutional layer, an up-sampling layer, a convolutional layer, a C3 module, a convolutional layer, and an up-sampling layer, and then connects the semantic segmentation module;
[0015] 2-4) Determining model training hyperparameters, constructing a model loss function, taking F1 as a target detection performance indicator, taking mIoU as a semantic segmentation performance indicator, and taking dlmk as a key point performance indicator, inputting the data set into the basic network model for training and testing to obtain a multi-task convolutional neural network model.
[0016] The method for establishing the multi-task convolutional neural network model further comprises: 2-5) Channel pruning of the basic network model: sparse training, channel pruning, and fine-tuning of the network model, so that the model reduces model weights and speeds up detection while maintaining model accuracy.
[0017] In the step 2-4),
[0018] The F1 formula is:
[0019]
[0020] Wherein, TP represents true positive, FP represents false positive, and FN represents false negative;
[0021] The d lmk The formula is:
[0022]
[0023] Wherein, n represents the number of key points, x pi represents the x-coordinate of the predicted key point, y pi represents the y-coordinate of the predicted key point, x tix coordinates of label key points, y ti y coordinates of label key points
[0024] The mIoU formula is:
[0025]
[0026] Wherein, K represents the number of semantic segmentation categories.
[0027] In the step 2-4), the optimizer is Adam, the training period is 700 times, the batch size is 16, and the picture pixel is 800.
[0028] The sparse training is: the BN layer coefficient of the basic network model is constrained by using an L1 regularization function based on L1 norm, so that the BN layer gamma factor is sparse, and the model is adjusted in the sparse direction.
[0029] The optimization target of the sparse training is:
[0030]
[0031] Wherein, x represents input, y represents target, w represents trainable weight, l represents the loss of the difference between actual output and target, |m i | represents the absolute value of the scaling factor of the i-th layer, lambda represents the sparsity, L represents the total number of network layers, and h represents the function expression of the trainable layer.
[0032] The channel pruning is: after the sparse training is completed, the proportion of model pruning needs to be determined, so as to determine the threshold value of BN layer pruning, and the channel pruning is performed on the model.
[0033] The model fine-tuning is: the basic network model after the channel pruning is completed is further trained by using a data set.
[0034] The loss function is composed of a boundary box regression loss function, a mask loss function, a key point loss function and a confidence loss function; the key point loss function is a Wing loss function; and the mask loss function is a cross-entropy loss function.
[0035] The beneficial effects of the present application are:
[0036] The application can complete tomato target, posture detection and stem semantic segmentation in one model by adding a semantic segmentation branch module and key point prediction parameters to the YOLO V5s model, effectively extracts comprehensive visual information, and the tomato target and posture of the visual information can help the picking robot determine the end effector picking posture, and the stem position of the visual information can help the robot realize collision-free picking, providing information support for the tomato picking robot; The model proposed by the application is fast, small in weight and high in precision, and uses channel pruning algorithm, sparse training, model pruning and fine tuning to further reduce the model weight and speed, reaching 3.2MB and 5.5ms, which can better deploy in mobile terminal and embedded device, and meet the needs of tomato picking robot in complex greenhouse environment. BRIEF DESCRIPTION OF DRAWINGS
[0037] Figure 1 is a structure diagram of the multi-task convolutional neural network model of the application.
[0038] Figure 2 is a data annotation schematic diagram of the application.
[0039] Figure 3 is a data set construction schematic diagram of the application.
[0040] Figure 4 is a key point parameter schematic diagram of the application.
[0041] Figure 5 is a channel pruning flowchart of the application.
[0042] Figure 6 is a channel pruning principle diagram of the application.
[0043] Figure 7 is a sparse training training diagram of the application.
[0044] Figure 8 is a channel change diagram after channel pruning of the application.
[0045] Figure 9 is an output result diagram of the application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.
[0047] EMBODIMENT
[0048] A plant and fruit detection algorithm for a tomato picking robot, comprising the following steps:
[0049] 1-1) Collecting tomato images;
[0050] 1-2) Inputting the tomato images into a multi-task convolutional neural network model (YOLO-MCNN) for processing;
[0051] 1-3) Obtaining the bounding box, key points, and stem of the tomato on the tomato image Figure 9 ) and connecting the key points of the tomato and the center point of the bounding box as the 2D pose of the tomato.
[0052] The picking robot determines the end effector picking pose according to the bounding box, 2D pose position, and stem position, realizes collision-free picking, and improves the picking quality and picking speed.
[0053] The method for establishing the multi-task convolutional neural network model is as follows:
[0054] 2-1) Constructing a data set: collecting tomato original images, constructing a tomato key point data set and a tomato stem data set;
[0055] Figure 2 For the data-annotated images, the tomato key point data set is made for the tomato key points and the tomato target as the target, the tomato original images are manually labeled, the tomato is labeled with a rectangular frame, the key points are labeled with a dot, the labels are generated and saved in JSON format, and the tomato stem data set is made for the stem as the target, the tomato original images are manually labeled, the stem is labeled with a mask, the labels are generated and saved in JOSN format.
[0056] The two data sets have the same images and different labels, a total of 804 images, and are divided into a training set and a test set in a ratio of 4:1;
[0057] The label format of the tomato key point data set is (x, y, h, w, x k , y k ), wherein x, y, w, and h are the center point position and length and width of the tomato bounding box, and x k , y k are the key point positions; the label format of the tomato stem data set is (x1, y1, x2, y2…x n , y n ), wherein x1, y1, x2, y2…x n , y n are the positions of the boundary points forming the mask;
[0058] 2-2) Data preprocessing: normalizing and data enhancing the tomato key point data set and the tomato stem data set;
[0059] 2-3) Due to the advantages of YOLO V5s model, such as low weight and fast detection speed, YOLO V5s was chosen as the basic network model;
[0060] The basic network model includes a Backbone network, a Neck network, and a Head network, and is mainly composed of a Focus module, a C3 module, and a Conv module.
[0061] The basic network model was improved to construct a multi-task convolutional neural network model (YOLO-MCNN);
[0062] Key point prediction parameters are added to the detection branch of the Head network of the basic network model to achieve bounding box regression and key point prediction. A semantic segmentation module is added to the Head network of the basic network model to achieve semantic segmentation of tomato stems.
[0063] The 16th layer of the basic network model is a splicing layer that sequentially passes through a convolutional module (Conv module), an upsampling layer, a C3 module, a convolutional module (Conv module), an upsampling layer, a convolutional module (Conv module), a C3 module, a convolutional module (Conv module), an upsampling layer, and then connects to the semantic segmentation module.
[0064] The essence of semantic segmentation is to classify each pixel. The semantic segmentation module classifies each feature point through 1*1 convolution. Since the 16th layer already contains object information and features of different proportions, we choose to restore the feature layer to the original image size through 3 upsampling layers on the 16th layer for semantic segmentation. Upsampling can make low-resolution images containing high-level abstract features high-resolution while retaining high-level abstract features. At the same time, the C3 module and convolution module are added during the upsampling process to further extract accurate features.
[0065] like Figure 4 As shown, the detection branch introduces keypoint prediction parameters (length × width × number of anchor boxes × 2), transforming the original parameter confidence, class probability, and bounding box parameters into parameter confidence, class probability, bounding box parameters (x, y and w, h of the bounding box center point), and keypoint location parameters (x and y of the keypoint).
[0066] The Focus module uses a slicing operation to split a high-resolution feature map into multiple low-resolution feature maps, thus completing the downsampling of the feature layer while preserving the original information.
[0067] The C3 module has two types: C3_x_1 and C3_x_2. Figure 1 This includes C3_1_1, C3_3_1, and C3_1_2, which are used to enhance the learning capabilities of deep learning networks;
[0068] The Conv module is composed of a convolution layer, a normalization layer and a SiLu activation function, and is used for extracting features of a feature layer;
[0069] 2-4) Determine the model training hyperparameters, construct the model loss function, and take F1 as the target detection performance indicator, take mIoU as the semantic segmentation performance indicator, and take dlmk as the key point performance indicator; input the data set into the basic network model for training and testing; the optimizer is Adam, the training period is 700 times, the batch size is 16, and the picture pixel is 800;
[0070] ① The F1 formula is:
[0071]
[0072] Among them, TP represents true positive, FP represents false positive, and FN represents false negative.
[0073] ② The d lmk formula is:
[0074]
[0075] Among them, n represents the number of key points, x pi represents the x coordinate of the predicted key point, y pi represents the y coordinate of the predicted key point, x ti represents the x coordinate of the label key point, and y ti represents the y coordinate of the label key point.
[0076] ③ The mIoU formula is:
[0077]
[0078] Among them, K represents the number of semantic segmentation categories.
[0079] ④ The loss function L is composed of a bounding box regression loss function L box , a mask loss function L mask , a key point loss function L key , and a confidence loss function L conf (L=L box +L mask +L key +L conf ).
[0080] The key point loss function is Wing loss, and the mask loss function is cross-entropy loss.
[0081] 2-5) Channel pruning of the base network model: sparse training, channel pruning, fine-tuning of the model, so that the model reduces the model weight and speeds up the detection while maintaining the model accuracy.
[0082] The sparse training is to constrain the BN layer coefficient of the base network model by using an L1 regularization function based on L1 norm, so as to make the BN layer gamma factor sparse and adjust the model in the sparse direction.
[0083] The optimization target of the sparse training is:
[0084]
[0085] Wherein, x represents input, y represents target, w represents trainable weight, l represents the loss of the difference between actual output and target, |m i | represents the absolute value of the scaling factor of the i-th layer, λ represents the sparsity, L represents the total number of layers of the network, and h represents the function expression of the trainable layer.
[0086] The channel pruning is to determine the pruning proportion of the model after the sparse training is completed, so as to determine the threshold of the BN layer pruning, and then the channel pruning of the model is performed.
[0087] The model fine-tuning is to continue training the base network model after the channel pruning is completed by using the data set. After the model pruning, although the network parameter quantity is significantly reduced, the accuracy is lost to a certain extent. Fortunately, the fine-tuning of the model can restore the model accuracy.
[0088] In the above formula, i represents the number of layers;
[0089] 2-6) Obtain a multi-task convolutional neural network model (YOLO-MCNN).
[0090] Test
[0091] The performance comparison of the multi-task convolutional neural network model is shown in Tables 1, 2 and 3:
[0092] In terms of target detection, YOLO V3 achieves the highest F1 of 0.88, but the model is too large, with a size of 123.5 MB.
[0093] The F1 of YOLO-MCNN is 87.6, which is only 0.004 lower than that of YOLO V3 and 0.022 and 0.123 higher than those of YOLO V3-tiny and YOLO V4-tiny.
[0094] YOLO-MCNN has the smallest size of 15.4 MB compared with other network models, and the speed is 6.8 ms. Although it is not the best, it meets the real-time requirement and completes more tasks.
[0095] In terms of semantic segmentation: the YOLO-MCNN network achieves the highest mIoU and the fastest inference speed of 72.4% and 6.8ms. Compared with Deeplabv3-Resnet50, Deeplabv3-Resnet101, FCN-Resnet50, FCN-Resnet101, Cednet and Enet, the mIoU is increased by 1.3%, 0.6%, 0.2%, 0.5%, 0.5% and 0.4% respectively, and the speed is increased by 16.4ms, 36.8ms, 17.5ms, 36.5ms, 31.8ms and 22.9ms respectively.
[0096] In terms of key point detection: the key point detection accuracy dlmk of the YOLO-MCNN network is 6.78 pixels, which is negligible in the actual 1080x1920 pixel image.
[0097] Model Precision (%) Recall (%) F1(％) Model size (MB) Speed (ms) YOLO V3 87.5 88.5 88 123.5 8.3 YOLO V3-tiny 86.9 84 85.4 17.4 2.2 YOLO V4-tiny 89.3 65.1 75.3 24.4 2.7 YOLO-MCNN 88.1 87.2 87.6 15.4 6.8
[0098] Table 1 (performance comparison of model target detection)
[0099] Model mIoU (%) Model size (MB) Speed (ms) Deeplabv3-Resnet50 71.1 167.6 23.2 Deeplabv3-Resnet101 71.8 243.9 43.6 FCN-Resnet50 72.2 140.8 24.3 FCN-Resnet101 71.9 217.1 43.3 Cednet 71.9 9.4 38.6 Enet 72 1.7 29.7 YOLO-MCNN 72.4 15.4 6.8
[0100] Table 2 (performance comparison of model semantic segmentation)
[0101] Model Precision (%) Recall (%) F1(％) mIoU (%) dlmk Model size (MB) Speed (ms) YOLO-MCNN 88.1 87.2 87.6 72.4 6.78 15.4 6.8
[0102] Table 3 (YOLO-MCNN performance)
[0103] The channel pruning performance of the multi-task convolutional neural network model is shown in Table 4:
[0104] The multi-task convolutional neural network YOLO-MCNN is sparsely trained, and the sparse training epoch is set to 1500 and sr is set to 0.0006. As shown in Figure 5 , the BN layer γ factor is well sparsified, and γ gradually approaches 0 from the normal distribution as the training progresses, but not all 0.
[0105] The channel changes after pruning are shown in Figure 6 , compared with before pruning, F1 increases by 0.007, mIoU decreases by 0.4%, dlmk decreases by 0.57 pixels, model size decreases by 79.2%, and speed increases by 1.3ms.
[0106] Figure 7 The model output result graph is shown in
[0107] Model Precision (%) Recall (%) F1(％) mIoU (%) dlmk Model size (MB) Speed (ms) YOLO-MCNN 88.1 87.2 87.6 72.4 6.78 15.4 6.8 Fine-tuning 84.9 92 88.3 72 7.35 3.2MB 5.5
[0108] Table 4 (Model performance changes during pruning process)
[0109] Finally, it should be noted that the above-mentioned are only specific embodiments of the present application. Obviously, the present application is not limited to the above-mentioned embodiments, but can have many variations. All variations that can be directly derived or inferred from the disclosure of the present application by those of ordinary skill in the art should be considered within the scope of the present application.
Claims
1. A plant and fruit detection algorithm for a tomato picking robot, comprising the following steps: 1-1) Collecting a tomato image; 1-2) Inputting the tomato image into a multi-task convolutional neural network model for processing; 1-3) Obtaining the bounding box, key points, and stem of the tomato on the tomato image; The method for establishing the multi-task convolutional neural network model is as follows: 2-1) Building a dataset: building a tomato key point dataset and a tomato stem dataset; 2-2) Data preprocessing: normalizing and data enhancing the tomato key point dataset and the tomato stem dataset; 2-3) Selecting YOLO V5s as a basic network model; adding key point prediction parameters to the detection branch of the Head network of the basic network model to realize bounding box regression and key point prediction, adding a semantic segmentation module to the Head network of the basic network model to realize tomato stem semantic segmentation; and connecting the semantic segmentation module after the 16th layer of the basic network model passes through a convolutional layer, an upsampling layer, a C3 module, a convolutional layer, an upsampling layer, a convolutional layer, a C3 module, a convolutional layer, and an upsampling layer in sequence; 2-4) Determining model training hyperparameters, constructing a model loss function, taking F1 as the target detection performance indicator, taking mIoU as the semantic segmentation performance indicator, taking dlmk as the key point performance indicator, inputting the dataset into the basic network model for training and testing, and obtaining the multi-task convolutional neural network model.
2. The plant and fruit detection algorithm for a tomato picking robot according to claim 1, characterized in that: The method for establishing the multi-task convolutional neural network model further comprises: 2-5) Channel pruning of the basic network model: sparse training, channel pruning, and fine-tuning of the network model to reduce the model weight and speed up the detection speed while maintaining the model accuracy.
3. The plant and fruit detection algorithm for a tomato picking robot according to claim 2, characterized in that: In the step 2-4), The F1 formula is: Where TP represents true positive, FP represents false positive, and FN represents false negative. The d lmk The formula is: wherein n represents the number of key points, x pi represents the x coordinate of a predicted key point, y pi represents the y coordinate of a predicted key point, x ti represents the x coordinate of a labeled key point, y ti represents the y coordinate of a labeled key point; The mIoU formula is: Where K represents the number of semantic segmentation categories.
4. The plant and fruit detection algorithm of claim 3, wherein: In the step 2-4), the optimizer is Adam, the training period is 700 times, the batch size is 16, and the picture pixel is 800.
5. The plant and fruit detection algorithm of a tomato picking robot according to claim 4, characterized in that: The sparse training is to constrain the BN layer coefficients of the basic network model by using an L1 regularization function based on the L1 norm to make the BN layer γ factor sparse and adjust the model in the sparse direction. The optimization target of the sparse training is: where x represents input, y represents target, w represents trainable weight, l represents loss of difference between actual output and target, |m i | represents the absolute value of the i-th layer scaling factor, λ represents the sparsity, L represents the total number of layers of the network, and h represents the function expression of the trainable layer.
6. The plant and fruit detection algorithm of a tomato picking robot according to claim 5, characterized in that: The channel pruning is to determine the pruning proportion of the model after the sparse training is completed, thereby determining the threshold for pruning the BN layer, and pruning the channels of the model.
7. The plant and fruit detection algorithm of a tomato picking robot according to claim 6, characterized in that: The fine-tuning of the model is to continue training the basic network model after channel pruning using the dataset.
8. The plant and fruit detection algorithm of a tomato picking robot according to claim 7, characterized in that: The loss function is composed of a bounding box regression loss function, a mask loss function, a key point loss function, and a confidence loss function; the key point loss function is a Wing loss function; and the mask loss function is a cross-entropy loss function.