Lawn obstacle detection method based on lightweight YOLOv5s model
Through the lawn obstacle detection method based on the lightweight YOLOv5s model, the problem of low obstacle detection efficiency of the mowing robot is solved, and efficient and convenient lawn obstacle detection is achieved, which is suitable for real-time environments and maintains detection accuracy and speed.
Patent Information
- Application Number
- CN202111318483.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-11-09
AI Technical Summary
The obstacle detection system of existing mowing robots is not efficient and vulnerable to damage, making it difficult to be suitable for mowing robots with limited computing resources while ensuring detection accuracy and speed.
The lawn obstacle detection method based on the lightweight YOLOv5s model is adopted, and the detection accuracy and speed are improved through image data acquisition, model improvement, pruning training and feature fusion, which is suitable for obstacle detection in the lawn environment.
It realizes efficient and convenient lawn obstacle detection, improves detection accuracy and speed, is suitable for real-time environments, and reduces the amount of model parameters and inference time.
Smart Images

Figure CN114049554B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a lawn environment obstacle detection method, and in particular to a lawn obstacle detection method based on a lightweight YOLOv5s model. Background Art
[0002] Manual mowing in a lawn environment is a monotonous and time-consuming task, and there are also threats from lawn slopes, obstacles, gravel, and even harmful insects in the grass. Although the emergence of mowing robots has solved the above problems, the obstacle avoidance systems of mowing robots on the market rely on contact sensors, wire technology and other traditional means, such as the AutoMower produced by Sweden's Husqvarna and the Sileno city produced by Germany's Gardena, which are inefficient and easy to damage obstacles.
[0003] With the rapid development of deep learning in recent years, the object detection technology based on deep learning has enabled lawn mowing robots to identify obstacles more intelligently and effectively. By detecting the types of obstacles, relevant obstacle avoidance strategies can be formulated. At the same time, the green plants and public facilities in the lawn environment can be supervised for later maintenance.
[0004] At present, there are two types of target detection methods based on deep learning. One is a two-stage algorithm, such as R-CNN, SPP-Net, and Fast R-CNN. These algorithms first generate candidate regions and then classify them. Although the accuracy is improved, the speed does not meet the real-time requirements of lawn mowing robots. The other is a single-stage algorithm, such as the YOLO series and the SSD series. This type of algorithm discards the step of generating candidate regions and regards object detection as a regression problem. Although it greatly improves the detection speed, it also leads to a decrease in accuracy.
[0005] The computing resources of lawn mowing robots are limited and dangerous, so the model needs to be compressed while ensuring detection accuracy and speed. Therefore, how to improve the detection accuracy and speed when detecting obstacles in the lawn environment has become a technical problem that needs to be solved urgently. Summary of the invention
[0006] Purpose of the invention: The purpose of the present invention is to provide a lawn obstacle detection method based on a lightweight YOLOv5s model, thereby improving the detection accuracy and detection speed, making the detection process more efficient and convenient.
[0007] Technical solution: The lawn obstacle detection method based on the lightweight YOLOv5s model of the present invention comprises the following steps:
[0008] (1) Collect image data including tree trunks, spherical shrubs, and lawn lamp posts, randomly divide the collected image data and some human image data in Pascal VOC 2007 into training sets and test sets, and annotate them;
[0009] (2) Lightweight improvement of the target detection network model YOLOv5s;
[0010] (3) Use the training set to train the improved target detection network model YOLOv5s to obtain the optimal detection network model;
[0011] (4) using a pruning method to perform pruning training on the optimal detection model obtained in step (3);
[0012] (5) Use the pruned trained model obtained in step (4) to detect the image data in the test set, and perform detection accuracy and lightweight evaluation on the detection results of the test set.
[0013] Step (1) specifically includes the following steps:
[0014] (1.1) Collect at least 10,000 sets of image data, with at least 1,000 sets of image data for each category;
[0015] (1.2) Clean the data set and remove low-quality image data;
[0016] (1.3) Annotate the dataset in YOLO format;
[0017] (1.4) Divide the human image data into training set and test set according to the set ratio.
[0018] Step (2) specifically includes the following steps:
[0019] (2.1) Divide the feature map of the first 1×1 convolution output in the Bottleneck structure of the C3 module in the YOLOv5s model into 4 subsets with equal number of channels;
[0020] (2.2) Starting from the second subset x2, a 3×3 convolution operation is performed. The output feature map subset y2 is added to x3 and input into the corresponding 3×3 convolution kernel. The output feature map subset y3 is added to x4 and input into the corresponding 3×3 convolution kernel to obtain the output feature map subset y4.
[0021] (2.3) Concatenate the four sets of output feature maps y1, y2, y3, and y4;
[0022] (2.4) Input the concatenated feature map into 1×1 convolution for feature fusion;
[0023] (2.5) A Squeeze-and-Excitation module is added after the 1×1 convolution to recalibrate the original features in the channel dimension.
[0024] Step (3) specifically includes the following steps:
[0025] (3.1) Setting initial parameters, including image input size, momentum parameter, weight decay regularization term parameter, image angle change parameter, saturation and exposure change parameter, hue change parameter, initial learning rate, and number of training iterations;
[0026] (3.2) Use the adaptive anchor box method to automatically cluster the anchor box size in the lawn environment obstacle dataset;
[0027] (3.3) Set the IOU threshold for loss calculation and enable multi-scale training, where the loss function uses CIOUloss.
[0028] Step (4) specifically includes the following steps:
[0029] (4.1) Load the improved model parameters in step (3), specifically, use the torch.load function to load the learning parameters of the lightweight improved YOLOv5s model, mainly to screen out the filters (or convolutional layers) that need to be pruned;
[0030] (4.2) For all convolution kernels in each filter, calculate the sum of the Euclidean distances between the convolution kernel and other convolution kernels, and prune N × pruning rate convolution kernels in order from small to large; N represents the total number of convolution kernels;
[0031] (4.3) Adjust the pruned network and train it for one epoch, and force the pruned convolution kernel gradient to 0;
[0032] (4.4) After iterating steps (4.2) and (4.3) multiple times, a zero-pruning model is obtained;
[0033] (4.5) Perform a zero removal operation on the pruned model in step (4.4) to obtain the pruned improved YOLOv5s model.
[0034] In step (5), the mean average precision (mAP) value of the four target categories in the dataset is used as the evaluation indicator of detection accuracy.
[0035] The number of model parameters, frame rate, model weight size, and inference speed are used as lightweight evaluation indicators.
[0036] In step (4), the optimal detection model obtained in step (3) is pruned and trained using a median-based filter pruning method.
[0037] In step (1.3), use LabelImg software to annotate the dataset in YOLO format.
[0038] In step (2.3), the feature map subsets y1, y2, y3, y4 are expressed as follows:
[0039]
[0040] Beneficial effects: Compared with the prior art, the present invention has the following advantages: The lawn obstacle detection method based on the lightweight YOLOv5s model of the present invention has high detection accuracy, detection speed, good real-time performance and lightweight performance. Compared with the prior art, the detection method of the present invention is more efficient and convenient. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a structural diagram of the Bottleneck module in the lawn obstacle detection method based on the lightweight YOLOv5s model of the present invention;
[0042] Figure 2 This is a structural diagram of the improved YOLOv5s model after pruning of the present invention;
[0043] Figure 3 It is a training curve diagram of the present invention;
[0044] Figure 4 This is a pruning flowchart for improving the YOLOv5s model in the present invention. DETAILED DESCRIPTION
[0045] Example:
[0046] (1) Image collection: Use the smartphone camera to collect image data of tree trunks, spherical shrubs, lawn lamp posts, and people in parks and communities, and extract some person category image data from the Pascal voc2007 public dataset. The number of images in each sub-dataset is shown in Table 1.
[0047] Table 1 Dataset analysis table
[0048]
[0049] Then the image data set is divided and labeled: the four sub-datasets of spherical shrubs, tree trunks, lawn lamp posts, and human images are mixed and shuffled, and randomly divided into training sets and test sets in a ratio of 8:2. In this embodiment, the final training set contains 7292 positive samples and the test set contains 1841 positive samples. The training set and the test set are independent of each other. The specific division is shown in Table 1, and labelImg is used for data annotation to form an annotation file suitable for YOLO model training.
[0050] (2) The target detection network model YOLOv5s is lightweight and improved. The improvement process is as follows: Figure 1 As shown, the specific steps are as follows:
[0051] (2.1) First, the feature map of the first 1×1 convolution output in the Bottleneck structure of the C3 module in the target detection network model YOLOv5s is evenly divided into 4 subsets with equal number of channels. A 3×3 convolution operation is performed starting from the second subset x2. The feature map y2 output by the convolution operation is added to x3 and input into the corresponding 3×3 convolution kernel. The output feature map y3 is added to x4 and input into the corresponding 3×3 convolution kernel to obtain the output feature map y4.
[0052] (2.2) Then the four sets of output feature maps y1, y2, y3, and y4 are concatenated and the concatenated feature maps are input into a 1×1 convolution for feature fusion.
[0053] (2.3) Finally, the SE (Squeeze-and-Excitation) module is added after the 1×1 convolution to recalibrate the original features in the channel dimension. The improved YOLOv5s model after pruning is as follows: Figure 2 shown.
[0054] (3) Use the training set to train the improved YOLOv5s model to obtain the optimal detection network model. The specific process is as follows:
[0055] Set the initial parameters of the improved YOLOv5s model, including image input size, momentum parameter, weight decay regularization parameter, image angle change parameter, saturation and exposure change parameter, hue change parameter, and initial learning rate. Set batch size and epoch according to the video memory size, and adjust epoch and learning rate parameters during training to improve the accuracy of the improved YOLOv5s model. The training curve is as follows: Figure 3 shown.
[0056] (4) Improve the pruning process of the YOLOv5s model such as Figure 4 shown.
[0057] Step (4.1), select 95 filter layers from all the learning parameters in the improved YOLOv5s model;
[0058] Step (4.2), for each convolution kernel in the filter, calculate the sum of the Euclidean distances between the convolution kernel and all other convolution kernels, where the convolution kernel with the smallest sum of Euclidean distances is the required geometric median. After that, set the N×pruning rate convolution kernels closest to the geometric median to zero, where N represents the total number of convolution kernels.
[0059] In step (4.3), the pruned network is adjusted and trained for one epoch, and the gradient of the pruned convolution kernel is forced to 0.
[0060] Step (4.4), in this embodiment, after iterating step (4.2) and step (4.3) 50 times, a zero-pruning model is obtained.
[0061] Step (4.5), finally, the pruned model in step (4.4) is de-zeroed to obtain the pruned improved YOLOv5s model. The pruning training parameters of the improved YOLOv5s model are shown in Table 2, and the pruning results are shown in Table 3. In this embodiment, after filter pruning, the number of parameters of the improved YOLOv5s model is reduced by 18%, the mAP is reduced by 1.7%, the model size is reduced by 2.1, and the inference speed is reduced by 5ms. Filter pruning effectively simplifies and accelerates the improved YOLOv5s model.
[0062] Table 2 Filter pruning parameter setting table
[0063]
[0064] Table 3 Pruning results
[0065]
[0066] (5) The pruned improved YOLOv5 model is evaluated according to the evaluation index, and the evaluation results are shown in Table 4. In this embodiment, the calculation amount of the improved YOLOv5s is reduced by 23.8%, and the weight size is reduced by 2.7MB. This is because after the 3×3 convolution in the original Bottleneck is divided into 3 convolutions, the model parameters are greatly reduced. The accuracy of the improved YOLOv5s remains almost unchanged, indicating that the hierarchical residual structure in step (2.2) compensates the accuracy of the convolution grouping well. In this embodiment, the reasoning speed of the pruned improved YOLOv5s model is increased by 3.1ms compared with YOLOv5s. The increase in the total amount of data read by the improved YOLOv5s model leads to a slower reasoning speed. The reasoning speed of the pruned improved YOLOv5s model is 69.8ms, which is 5ms less than that of the improved YOLOv5s, and exceeds that of YOLOv5s. It can be seen that the lawn environment obstacle detection method based on the lightweight YOLOv5s model proposed in the present invention effectively accelerates the model and maintains accuracy.
[0067] Table 4. Comparison of improved model performance
[0068]
Claims
1. A lawn obstacle detection method based on a lightweight YOLOv5s model, characterized in that: The detection method comprises the following steps: (1) Collect image data containing tree trunks, shrubs, lawn lamp posts, and people, divide the collected image data into training sets and test sets, and annotate them; (2) Lightweight improvement of the target detection network model YOLOv5s model; the process is as follows: (2.1) Divide the feature map of the first 1×1 convolution output in the Bottleneck structure of the C3 module in the YOLOv5s model into 4 subsets with equal number of channels; (2.2) Starting from the second subset x2, a 3×3 convolution operation is performed. The output feature map subset y2 is added to x3 and input into the corresponding 3×3 convolution kernel. The obtained output feature map subset y3 is added to x4 and input into the corresponding 3×3 convolution kernel to obtain the output feature map subset y4. (2.3) Concatenate the four output feature map subsets y1, y2, y3, and y4; (2.4) Input the concatenated feature map subset into 1×1 convolution for feature fusion; (2.5) Add the SE module after the 1×1 convolution to recalibrate the original features in the channel dimension; (3) Use the training set to train the improved target detection network model YOLOv5s model to obtain the optimal detection network model; the process is: (3.1) Setting initial parameters, including image input size, momentum parameter, weight decay regularization term parameter, image angle change parameter, saturation and exposure change parameter, hue change parameter, initial learning rate, and number of training iterations; (3.2) Use the adaptive anchor box method to cluster the anchor box sizes in the lawn obstacle dataset; (3.3) Set the IOU threshold for loss calculation and enable multi-scale training; (4) using a pruning method to perform pruning training on the optimal detection model obtained in step (3); (5) Use the pruned model trained in step (4) to detect the image data in the test set, and perform detection accuracy and lightweight evaluation on the detection results of the test set.
2. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 1, characterized in that: Step (1) includes the following: (1.1) Collect at least 1,000 images for each image category; (1.2) Clean the data set; (1.3) Annotate the dataset in YOLO format; (1.4) Divide the human image data into training set and test set according to the set ratio.
3. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 1, characterized in that: Step (4) includes the following: (4.1) Loading the improved model parameters in step (3), i.e., screening out multiple filters; (4.2) For all convolution kernels in each filter, calculate the sum of the Euclidean distances between the convolution kernel and other convolution kernels, and prune N × pruning rate convolution kernels in order from small to large; N represents the total number of convolution kernels; (4.3) Adjust the pruned model and train it for one epoch, and force the pruned convolution kernel gradient to 0; (4.4) After iterating steps (4.2) and (4.3) multiple times, the zero-pruning model is obtained; (4.5) Perform a zero removal operation on the pruned model in step (4.4) to obtain the pruned improved YOLOv5s model.
4. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 1, characterized in that: In step (5), the mean average precision (mAP) of the four target categories in the dataset is used as the evaluation indicator of detection accuracy.
5. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 1, characterized in that: In step (5), the model parameter quantity, frame rate, model weight size, and inference speed are used as lightweight evaluation indicators.
6. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 1, characterized in that: In step (4), the optimal detection model obtained in step (3) is pruned and trained using a median-based filter pruning method.
7. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 2, characterized in that: In step (1.3), use LabelImg software to annotate the dataset in YOLO format.
8. The lawn obstacle detection method based on the lightweight YOLOv5s model according to claim 1, characterized in that: In step (2.3), the feature map subsets y1, y2, y3, y4 are expressed as follows:
