Method for sensing and identifying an obstacle branch of apple picking

Abstract

The application provides a method for perceiving and identifying apple picking obstacle branches, comprising the following steps: S1: collecting a plurality of images of branches and apples; S2: using labelImg to label all branch data sets in the pictures, subdividing the labeled data sets, using labelImg to label branches that only block apples, and obtaining a branch set with a red apple background; S3: using labelme to pretreat the subdivided branch set and label the branch contour with a rectangular shape feature; S4: improving the Dice coefficient; S5: improving the YOLOv4 function; S6: predicting; and S7: obtaining the optimal data set. The method uses semantic segmentation to pretreat the branch set, optimizes the problem that the branch features are not obvious, improves the YOLOv4 model, and proposes a BIOU bounding box regression loss function suitable for branches, and simultaneously improves the recognition rate of samples with low confidence and samples with high confidence.

Description

Technical Field

[0001] This invention relates to machine perception and recognition methods, and more specifically to a method for perceiving and recognizing branches that obstruct apple picking. Background Technology

[0002] Apples are one of the most commonly consumed fruits, and my country's apple planting area has reached tens of millions of acres. Therefore, the application of harvesting robots in apple harvesting has a very broad prospect. In the mid-to-late 1920s, American scholars Schertz and Brown first proposed using robots in the fruit harvesting process to simplify the harvesting process. However, due to the diverse planting methods and production models of apple trees in my country, the low standardization of orchard construction and planting, the complex environment within orchards, and the presence of many obstacles, which pose certain safety hazards, harvesting robots are not suitable for identification and harvesting operations, thus limiting their widespread application.

[0003] Currently, fruit picking identification methods mainly rely on image acquisition, preprocessing, and recognition. Information about obstructing tree branches is primarily obtained through radar point cloud data collection. Patent document CN 114431005 A discloses an intelligent agricultural fruit picking identification and positioning method, system, and device, which uses normalized preprocessed RGB images as input into a model for prediction. Patent document 201210367554.Y discloses a device and method for identifying, classifying, and spatially locating picking objects based on panoramic stereo vision, using a binocular stereo panoramic vision sensor for perception and identification. Patent 202111669823.Y discloses a training method for a tree branch identification model, a tree branch recognition method, and a device. This method trains and identifies the tree branch model using radar point cloud data collection.

[0004] The methods disclosed in the aforementioned patent documents only acquire data through sensing images or point clouds. They do not effectively separate the complex background environment of the object under test, nor do they propose identification methods for tree branches of different shapes. In actual detection and identification, they have limitations and have disadvantages such as low accuracy and poor real-time performance. Summary of the Invention

[0005] To address the obstacle of identifying tree branches during apple harvesting, specifically branches obscuring the apple's location when a robotic arm is harvesting, this paper proposes a preprocessing method for tree branches. It also proposes a method combining semantic segmentation and YOLOv4 to obtain semantic skeletons and identify the tree branch locations, and proposes an improved bounding box regression loss function to enhance the accuracy of tree branch recognition.

[0006] The specific plan is as follows:

[0007] A method for sensing and identifying apple picking obstacles on tree branches includes the following steps:

[0008] S1. Image Acquisition: Collect several images of tree branches and apples;

[0009] S2, Labeling and Subdivision: LabelImg is used to label all tree branches in the image dataset. The labeled dataset is then subdivided: labelImg is used to label only the branches that obscure the apples, resulting in a set of branches with a red apple background. The subdivided branch sets possess certain similar features. In S2, the bounding boxes of each branch are labeled separately to obtain the dataset of all branches. During training, due to the lack of obvious shape and background features, both training and recognition results are poor. Subdividing the labeled dataset by labeling only the branches that obscure the apples yields a set of branches with rectangular shapes and a uniform red background, which significantly improves the training and prediction performance of the network model.

[0010] S3. Preprocessing: Use labelme to preprocess the subdivided tree branch set and label the tree branch outlines with rectangular shape features;

[0011] S4. Improve Dice coefficient: Improve the Dice coefficient in the semantic segmentation network model by feeding the preprocessed tree branch set into the improved semantic segmentation model for training;

[0012] S5. Improved YOLOv4 function: Improve the bounding box regression loss function in YOLOv4 by feeding the subdivided tree set into the improved YOLOv4 network model for training;

[0013] S6. Prediction: Input the image to be tested and predict the branches that are blocking the apple in the image;

[0014] S7. Obtain the optimal dataset: Feed the prediction results obtained in S6 back to the YOLOv4 network model, change the weight file for comparative training, select the weight file with the best prediction effect, and at the same time obtain the tree set with the best training effect.

[0015] To enhance the tree branch preprocessing effect, a Dice coefficient for calculating tree branch sample similarity is proposed. To improve the training effect of semantic segmentation on the tree branch set and better segment the tree branch envelope, the Dice coefficient s used for calculating sample similarity in semantic segmentation is increased by a fixed proportion based on its range. This increase is used to enhance the branches with lower semantic skeleton recognition efficiency. The original Dice coefficient s is:

[0016]

[0017] The improved Dice coefficient s′ is:

[0018]

[0019] In the above formula, X is the number of true value elements in the segmentation; Y is the number of elements in the prediction model.

[0020] Loss of training branches for semantic segmentation (Dice') Loss =1-s'.

[0021] YOLO is a novel object detection technique characterized by its ability to achieve both rapid detection and high accuracy. YOLO integrates object region prediction and object category prediction into a single neural network model, enabling rapid learning of object detection and recognition while maintaining high accuracy, making it more suitable for practical research environments. YOLO v1, the starting point of YOLO, relies on object recognition labeled data, thus its performance is not significant for the unconventional shapes of objects like apple branches. YOLO v2 is a joint training method for detection and classification. YOLO v3 improves upon YOLO v2 by directly generating bounding box coordinates and probabilities for each class through regression, significantly improving detection speed. YOLO v4 achieves a balance between detection speed and accuracy compared to YOLO v3. This solution improves upon YOLO v4 for the special case of apple branches. In S5, YOLO v4 calculates the loss value based on the CIOU of the bounding box regression loss function:

[0022]

[0023]

[0024]

[0025]

[0026] In the formula ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box;

[0027] C is the diagonal distance of the smallest closure region that can simultaneously contain both the predicted bounding box and the ground truth bounding box;

[0028] A is the prediction box;

[0029] B represents the true bounding box;

[0030] IOU represents the degree of overlap between the predicted bounding box and the ground truth bounding box;

[0031] b specifies the position of a point within the target bounding box;

[0032] b gt To predict the position of the points in the bounding box;

[0033] α is a parameter used to balance the proportions;

[0034] v is a parameter used to measure the consistency of aspect ratio between the prior bounding box and the target bounding box;

[0035] ω gt Indicates the width of the target bounding box;

[0036] h gt Indicates the height of the target bounding box;

[0037] ω represents the width of the prediction box;

[0038] h represents the height of the prediction box.

[0039] To calculate the regression loss for tree branches, a novel bounding box regression loss function, BIOU, is defined based on the YOLOv4 tree branch prediction bounding boxes and the ground truth bounding boxes. This function only displays tree branch prediction bounding boxes with a confidence level above 0.5. For each small interval where the confidence level exceeds 0.5, the confidence level of the tree branch prediction is increased to varying degrees, thus enhancing the prediction performance. The BIOU formula is as follows:

[0040]

[0041] The corresponding loss of the tree branches is B. Loss =1-BIOU.

[0042] Beneficial effects:

[0043] (1) This invention provides a method for perceiving and recognizing branches that hinder apple picking. It uses semantic segmentation to remove small branches and preprocesses the branch set to optimize the problem of unclear branch features. By improving the YOLOv4 model, a BIOU bounding box regression loss function suitable for branches is proposed. After fine-tuning, when the prediction confidence reaches above 0.5, the recognition rate of samples with low confidence and samples with high confidence is improved at the same time.

[0044] (2) This invention provides a method for perceiving and recognizing branches that obstruct apple picking. In the case where the shape features and background features of apple branches are not obvious, a detailed scene annotation method is used. Using the labelImg annotation tool, on the basis of branches with only rectangular outlines, a part of the branches that obscure the red apple is selected, that is, the part of the branches with red apple skin as the background. The network adds obvious color features when training branches, which solves the problem of indistinct color features in complex scenes.

[0045] (3) This invention provides a method for perceiving and recognizing branches that hinder apple picking. The Dice coefficients s used for calculating sample similarity in semantic segmentation are increased by a fixed proportion according to the range, which is used to enhance the branches with low semantic skeleton recognition efficiency. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method for sensing and identifying obstructed branches during apple harvesting in this invention.

[0047] Figure 2 This is a comparison chart of prediction results before and after the modification of the YOLOv4 network model in the embodiment.

[0048] Figure 3 This is a comparison chart of prediction results before and after the modification of the semantic segmentation network model in the embodiment. Detailed Implementation

[0049] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments and accompanying drawings. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0050] Example:

[0051] A method for sensing and identifying apple picking obstacles on tree branches includes the following steps:

[0052] S1. Image Acquisition: Collect several images of tree branches and apples;

[0053] S2. Labeling and Segmentation: Use labelImg to label all tree branches in the image dataset. Then, segment the labeled dataset and use labelImg to label only the branches that obscure the apples.

[0054] Obtain a set of tree branches with a red apple background;

[0055] S3. Preprocessing: Use labelme to preprocess the subdivided tree branch set and label the tree branch outlines with rectangular shape features;

[0056] S4. Improve Dice coefficient: Improve the Dice coefficient in the semantic segmentation network model by feeding the preprocessed tree branch set into the improved semantic segmentation model for training;

[0057] S5. Improved YOLOv4 function: Improve the bounding box regression loss function in YOLOv4 by feeding the subdivided tree set into the improved YOLOv4 network model for training;

[0058] S6. Prediction: Input the image to be tested and predict the branches that are blocking the apple in the image;

[0059] S7. Obtain the optimal dataset: Feed the prediction results obtained in S6 back to the YOLOv4 network model, change the weight file for comparative training, select the weight file with the best prediction effect, and at the same time obtain the tree set with the best training effect.

[0060] The above steps were used to process 1210 images of tree branches and apples. First, all rectangles containing tree branches were labeled using labelImg, resulting in a total of 6449 bounding box samples. Among them, there were 5195 bounding box samples in the training set, 592 bounding box samples in the validation set, and 662 bounding box samples in the test set.

[0061] 829 images of tree branches obscuring apples were selected from all images. The number of samples containing borders was 3783, labeled with labelImg. These 3783 border samples were preprocessed, and small branches that would affect the training effect were removed by labelme, forming the rectangular outline of the tree branches. The branches were then divided into training and validation sets in a 9:1 ratio according to the number of images for training purposes.

[0062] The improved Dice coefficient s′ in semantic segmentation is:

[0063]

[0064] In the above formula, X is the number of true value elements in the segmentation; Y is the number of elements in the prediction model.

[0065] The improved Dice coefficient is applied to the semantic segmentation model;

[0066] YOLOv4 calculates the loss value based on the CIOU bounding box regression loss function as follows:

[0067]

[0068]

[0069]

[0070]

[0071] YOLOv4's boundary regression loss function BIOU, which is based on tree-based predicted bounding boxes and ground truth bounding boxes, is as follows:

[0072]

[0073] The corresponding loss of the tree branches is B. Loss =1-BIOU, applying the tree-boundary regression loss function BIOU to the YOLOv4 model;

[0074] Load the dataset, train the model, and compare the training results using AP and F1 scores respectively.

[0075]

[0076]

[0077] P – Prediction accuracy;

[0078] R – Recall;

[0079]

[0080]

[0081] TP – The actual number of positive samples;

[0082] FP – Number of spurious positive samples;

[0083] FN – Number of spurious negative samples;

[0084] The experiment concluded that:

[0085] The YOLOv4 model before the loss function was trained on the entire tree branch set, and the AP (Average Precision) and F1 (F1 Score) values ​​were 71.00% and 60.87%, respectively.

[0086] The YOLOv4 model before the loss function was improved achieved an AP value of 89.00% and an F1 value of 86.58% when training on a set of branches that obscure apples.

[0087] The YOLOv4 model with the improved loss function trained on the entire tree branch set achieved an AP and F1 score of 71.00% and 67.15%, respectively.

[0088] The YOLOv4 model with the improved loss function trained on a set of branches that obscure apple trees achieved an AP value of 91.00% and an F1 score of 90.51%.

[0089] Figure 2 This is a comparison chart of the prediction results before and after the modification of the YOLOv4 network model. Figure 3 This is a comparison chart of the prediction results before and after the modification of the semantic segmentation network model. The loss curve of the BIOU loss function on the training set is shown below. Figure 2 The BIOU_train_loss curve is shown, and the loss curve on the validation set is as follows. Figure 2 The BIOU_val_loss curve is shown in the figure; the loss curves of the model using YOLOv4 Loss on the training set and validation set are respectively shown by... Figure 2 The YOLOv4_train_loss and YOLOv4_val_loss curves are shown in the figure; the loss curves for semantic segmentation on the training and validation sets are given by... Figure 3The yuyi_train_loss and yuyi_val_loss curves are shown in the figure; the loss curve of the fine-tuned semantic segmentation on the training set is shown in the figure. Figure 3 As shown in Byuyi_train_loss, the loss curve on the validation set is as follows: Figure 3 As shown in Byuyi_val_loss.

[0090] Figure 2 Figure 3 These are all graphs showing the change in loss values ​​after 100 epochs, used to evaluate the trained model. The weights are saved once per epoch. Figure 2 It can be seen that the model trained using the BIOU loss function has a better fit on the training set than YOLOv4. YOLOv4 stabilizes on the validation set after 50 epochs. The BIOU validation set loss is not within the scope of the BIOU loss function modification, hence the slight fluctuations. The semantic segmentation training loss value is relatively small; the modification of the Dice coefficient based on the tree-branch model did not significantly change the loss value. Both the improved and unimproved models gradually converged after 60 epochs.

[0091] After training is complete, the optimal dataset is obtained as the set of branches that occlude the apple. The optimal network model is YOLOv4 with an improved loss function. Based on this, the input image to be tested is used for prediction, and the best prediction result is obtained.

[0092] As a further improvement, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for sensing and identifying obstructed branches when picking apples, characterized in that, Includes the following steps: S1. Image Acquisition: Collect several images of tree branches and apples; S2, Labeling and Segmentation: Use labelImg to label all tree branch datasets in the image, segment the labeled dataset, and use labelImg to label only the branches that cover the apples to obtain the tree branch set with red apples as the background. S3. Preprocessing: Use labelme to preprocess the subdivided tree branch set and label the tree branch outlines with rectangular shape features; S4. Improve Dice coefficient: Improve the Dice coefficient in the semantic segmentation network model by feeding the preprocessed tree branch set into the improved model for training; S5. Improved YOLOv4 function: Improve the bounding box regression loss function in YOLOv4 by feeding the subdivided tree set into the improved YOLOv4 network model for training; S6. Prediction: Input the image to be tested and predict the branches that are blocking the apple in the image; S7. Obtain the optimal dataset: Feed the prediction results obtained in S6 back to the YOLOv4 network model, change the weight file for comparative training, select the weight file with the best prediction effect, and at the same time obtain the tree set with the best training effect. The Dice coefficient s before the improvement was: The improved Dice coefficient s′ is: In the above formula, X represents the true value element of the segmentation; Y represents the predictive model element. In S5, YOLOv4 calculates the loss value based on the CIOU bounding box regression loss function as follows: In the formula The Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c is the diagonal distance of the smallest closure region that can simultaneously contain both the predicted bounding box and the ground truth bounding box; A is the prediction box; B represents the true bounding box; IOU represents the degree of overlap between the predicted bounding box and the ground truth bounding box; To specify the location of the center point of the target bounding box; This indicates the position of the center point of the prediction box; It is a parameter used to balance the proportions; It is a parameter used to measure the consistency of aspect ratio between the prior bounding box and the target bounding box; Indicates the width of the target bounding box; Indicates the height of the target bounding box; Indicates the width of the prediction box; Indicates the height of the prediction box; In S5, the YOLOv4 bounding box regression loss function BIOU, based on tree-based predicted bounding boxes and ground truth bounding boxes, is: 。 2. The method for sensing and identifying apple picking obstacles by tree branches according to claim 1, characterized in that, Loss of training branches for semantic segmentation .

3. The method for sensing and identifying apple picking obstacles by tree branches according to claim 1, characterized in that, The corresponding loss of tree branches is .

Images