A method for detecting pecan fruit based on improved YOLOv5

Abstract

The application discloses a kind of based on improved YOLOv5 pecan fruit detection method, including steps: S1, according to the physical properties and morphological characteristics of pecan, collect pecan under different time, different illumination sample image, according to sample feature first carries out pretreatment to collection image, establishes target fruit identification and detection dataset;S2, using improved YOLOv5s network to the identification and detection dataset is trained, and verifies the robustness and real-time performance of algorithm;Wherein, the improved YOLOv5s network is: on the structure of YOLOv5s target detection algorithm is improved, by introducing SE attention module and StageBlock module, increase the detection accuracy of algorithm to target fruit under specific environment.The comparative experiment finds that when fruit target is less, the algorithm of the application is improved by 5.4% than the original algorithm, the accuracy reaches 97.2%, and when detecting dense, the improved algorithm of the application is better in small target detection performance, and the detection speed can reach 54.3fps / s.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for detecting thin-shelled pecan fruits based on an improved YOLOv5. Background Technology

[0002] Pecan (Caryaillinoinensis Koch) belongs to the genus Caryaillino in the family Juglandaceae. Native to the United States and northern Mexico, it is also known as the American pecan and is one of the world's most famous nut-bearing tree species. Pecans are large with thin shells, have a high kernel yield, and are rich in various amino acids, vitamins, and trace elements beneficial to the human body, making them an important woody oilseed plant. Pecan trees are tall and upright with a beautiful shape, making them excellent ornamental, shade-providing, and street tree species. Their wood has a fine texture and is tough, making it an ideal material for pianos, handicrafts, and high-end furniture. Pecans have been introduced to my country for over 100 years, mainly through manual cultivation and management, with very little large-scale automated harvesting machinery. Compared to mechanized production abroad, there is still a significant gap. Furthermore, agricultural automation is the future direction of agricultural informatization and plays a crucial role in improving farmers' income. As an agricultural cash crop, pecans hold an important position in agricultural income. Therefore, research on automated harvesting of pecan fruits is essential. Among them, conducting research on the detection, location, and harvesting of thin-shelled pecans is of forward-looking significance for improving the mechanization of thin-shelled pecan management in the future.

[0003] Fruit harvesting is typically done manually, requiring a large workforce, accounting for more than one-third of the total labor force needed for orchard management. Traditional manual harvesting is not only inefficient but also incurs high labor costs due to the need for a large workforce. Furthermore, the strenuous harvesting work can cause physical harm to workers, reducing both efficiency and manpower. To improve agricultural production management, people are constantly seeking new methods to increase production efficiency, enabling more production tasks to be completed with less labor, thereby reducing production management costs and maximizing production benefits. In manual harvesting, manual labor alone is insufficient; sometimes mechanical tools are needed. With the development of agricultural automation and robotics technologies, there is a growing demand for tools to replace labor-intensive methods. The demand for automated harvesting robots in agricultural production is on the rise, with automated fruit and vegetable harvesting robots being the most common example. This is an important manifestation of modern agriculture and an inevitable result of the mechanization of modern agriculture. Therefore, to meet the increasing demand for labor, the use of automated harvesting robots can improve the efficiency of fruit and vegetable harvesting.

[0004] In computer vision, image recognition methods based on traditional digital image processing techniques are relatively mature. These algorithms utilize features such as color, texture, and geometry in images to identify targets. However, when it comes to identifying thin-shelled pecan fruits in unstructured environments, the robustness of traditional digital image processing algorithms is greatly reduced, ultimately resulting in a recognition rate that fails to meet harvesting requirements. The success of fruit harvesting largely depends on the correct identification of the target; therefore, the key to successful fruit harvesting is accurately identifying the fruit from the tree.

[0005] Kang et al. (Kang Hanwen, Chen Chao. Fruit detection, segmentation and 3D visualization of environments in apple orchards[J]. Computers and Electronics in Agriculture, 2020, 171: 105302(6).) developed the deep neural network DaSNet-v2 to detect and segment fruits and branches in apple orchard environments, and gave performance evaluation parameters for different trunks of apples and branches. Tian et al. (Tian Yunong, Yang Guodong, Wang Zhe, et al. Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J]. Computers and Electronics in Agriculture, 2019, 157: 417–426.) proposed an improved YOLO-V3 model to detect apples at different growth stages in orchards, so as to effectively improve fruit detection under the conditions of apple overlap and occlusion. Koirala et al. (Koirala A., Walsh KB, Wang Z., et al. Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of 'MangoYOLO'[J]. Precision Agriculture, 2019, 20: 1107–1135(6).) adopted the "MangoYOLO" architecture, which is an improvement on YOLO, to estimate mango fruit yield. Sun et al. (Sun Jun, He Xiaofei, Ge Xiao, et al. Detection of Key Organs in Tomato Based on Deep Migration Learning in a Complex Background[J]. Agriculture, 2018, 8: 196(12).) proposed an improved tomato tissue detection method based on convolutional neural networks. Based on the original Faster R-CNN algorithm, a ResNet-50 with residual blocks was used instead of the traditional VGG16 feature extraction network, and K-means clustering was used to improve detection accuracy.The aforementioned studies mainly focused on the detection of individual fruits such as apples, citrus fruits, and mangoes, which are relatively large and have distinct color and texture characteristics. However, there are fewer studies on the detection of small fruits with no obvious texture characteristics and whose color is similar to the background color, and the accuracy of these studies is relatively low, such as for thin-shelled pecans.

[0006] Currently, color-based fruit target detection methods still have some problems in natural unstructured environments, especially in fruit detection under high overlap conditions: (1) some color threshold-based methods cannot identify fruits with similar background colors; (2) some shape feature-based methods cannot detect clustered fruits; (3) some methods cannot detect fruits of different sizes. Summary of the Invention

[0007] The purpose of this invention is to provide a method for detecting thin-shelled pecan fruits based on an improved YOLOv5s, aiming to solve the problems existing in the prior art mentioned above.

[0008] This invention is implemented as follows: a method for detecting thin-shelled pecan fruits based on an improved YOLOv5, the method comprising the following steps:

[0009] S1. Based on the physical characteristics and morphological features of thin-shelled pecans, collect sample images of thin-shelled pecans under different times and different light conditions. Based on the sample features, preprocess the collected images to establish a target fruit recognition and detection dataset.

[0010] S2. The improved YOLOv5s network is used to train the recognition and detection dataset, and the robustness and real-time performance of the algorithm are verified. The improved YOLOv5s network is an improvement on the structure of the YOLOv5s target detection algorithm by introducing the SE attention module and the StageBlock module to increase the detection accuracy of the target fruit in a specific environment.

[0011] Preferably, step S2 is followed by the following step:

[0012] S3. Calculate the area of ​​the overlapping fruit region and the number of overlapping fruits to cluster the pecan fruits.

[0013] Preferably, the preprocessing includes dataset sample labeling and illumination correction for unevenly exposed photos.

[0014] Preferably, in step S2, the SE attention module is specifically used to: obtain the global features of each channel by performing global max pooling on the input feature map, obtain a 1*1*C vector, process the vector using a fully connected layer to learn the nonlinear relationship between different channels, apply the Sigmoid activation function as weights to the input feature map, that is, assign different weights to different channels, and add the calculated result to the original feature map.

[0015] Preferably, in step S2, the StageBlock module is specifically used to: process the input feature maps using different convolutions, with the number of output channels being 1 / 8 and 1 / 4 of the number of input channels, respectively, and process them using two basic blocks; then, downsample the feature map with 1 / 8 of the channels by a factor of 2, upsample the feature map with 1 / 4 of the channels by a factor of 2, fuse them, process them using the ReLU activation function, and then process them using two basic blocks, upsample the feature map with 1 / 4 of the channels by a factor of 2, reduce the number of channels to 1 / 8 of the original, fuse it with the feature map with 1 / 8 of the channels, and use convolution to restore the number of channels to the original size.

[0016] Preferably, step S3 includes the following specific process: determining whether the overlap area of ​​the two walnut test boxes is greater than a threshold; if so, clustering the two target detection boxes into a larger new detection box, repeating the above process until no new detection boxes can be generated, and finally obtaining the clustering of the entire image.

[0017] This invention involves the construction of deep neural network models and improvements to the traditional YOLOv5. To facilitate understanding of this invention, a basic overview of deep learning networks, the YOLO detection network series, and the YOLOv5 architecture model are introduced here.

[0018] 1. Overview of Deep Learning

[0019] Deep learning is a major branch of machine learning, originating from research on artificial neural networks. Proposed by Hinton et al. (Hinton Geoffrey E., Osindero Simon, Teh Yee-Whye. A fast learning algorithm for deep belief nets[J]. Neural computation, 2006, 18: 1527–1554(7).), it is an algorithm that uses multiple processing layers containing complex structures or multiple nonlinear transformations to perform high-level abstraction of data. Convolutional neural networks were already used commercially as early as the 1980s. When Lecun et al. (Lecun Y., Bottou L., Bengio Y., et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86: 2278–2324(11).) invented the LeNet network, it is considered the initial network model for deep learning. However, limited by the hardware capabilities and dataset size at the time, deep learning has achieved limited success in recent years and remains in its nascent stage. With the rise of massively parallel computing and distributed computing, and the creation of the large-scale ImageNet dataset by Fei-Fei Li and others at Stanford University (Deng Jia, Dong Wei, Socher Richard, et al. ImageNet: A large-scale hierarchical image database [C] / / Staff, I. 2009 IEEE Conference on Computer Vision and Pattern Recognition. [Place of publication not identified]: IEEE, 2009: 248–255.) and the annual challenge, deep learning entered a new era in 2012 with the invention of the AlexNet network by Hinton et al. Currently, deep learning applications cover image recognition, autonomous driving, natural language processing, intelligent search engines, and robotics, and will have a huge and far-reaching impact on these fields.

[0020] (1) Traditional Neural Networks

[0021] The basic processing unit of the human brain is the single neuron shown in the diagram. A neuron receives the excitation signal from the previous neuron from its dendrites, generating excitation or inhibition, and transmits neurotransmitters along the axon through the synapse to the next neuron, thereby completing the transmission of information.

[0022] Similarly, assuming the working mechanism of the neuron in the figure can be represented by the abstract diagram on the right, the input signal received by the neuron is X, the intensity of the neurotransmitter is set with weight W, and it contains a bias value b, then the output Y of the neuron can be represented by equation (1):

[0023]

[0024] Introducing nonlinear functions as activation functions allows deep neural networks to theoretically approximate any function. However, the sigmoid function restricts real values ​​to the interval (0,1). When neuron activation approaches 0 and 1, it is prone to oversaturation, and the gradient also approaches zero, making it difficult to propagate errors back to earlier layers of the network during backpropagation. While the Tanh function exhibits relatively less gradient vanishing, like the sigmoid function, it involves exponential operations in its computation, making it time-consuming.

[0025] (2) Convolutional Neural Network

[0026] Convolutional Neural Networks (CNNs) (SZEGEDY C, LIU W, JIA Y, et al. Going Deeper with Convolutions[R], 2014. http: / / arxiv.org / pdf / 1409.4842v1.) are commonly used in object detection and image classification algorithms. They are a type of feedforward neural network that includes convolutional computation and has a deep structure. They are also one of the representative algorithms of deep learning. They are mainly composed of a series of different basic layer structures such as convolutional layers, pooling layers, and fully connected layers, each of which has a different function.

[0027] As the most fundamental component of convolutional neural networks, convolution essentially uses the parameters of the convolution kernel to extract features from the data and obtains the results through matrix multiplication and summation operations. Figure 1 The diagram shows the operation process of a basic two-dimensional convolution, with the formula y = ωx + b. Here, the input feature map x is 1×3×3 in size, meaning it has 1 input channel and 3 in both length and width. The convolution kernel ω is 3×3 in size, and the bias b is 1. To ensure that the output feature dimension is consistent with the input feature dimension, padding is required, which is done using 0. The convolution kernel parameters are multiplied bit by bit with the corresponding pixels and then summed to obtain the result of one calculation, thus obtaining the rightmost output feature map.

[0028] After convolution, the height and width of the output feature map are calculated as follows:

[0029]

[0030]

[0031] Where: padding is the padding width, stride is the stride used in the convolution, and H... in W in H k and W k These represent the width and height of the input feature map and the convolution kernel, respectively. The size of the output feature map can be adjusted by modifying the size of the convolution kernel, as well as padding and stride. During convolution, a 3×3 kernel is commonly used, with padding and stride set to 1 to ensure the output feature map size is the same as the input feature map size.

[0032] Nonlinear computation in neurons is generally implemented by activation functions. The following are some common activation functions in neural network models.

[0033] 1) Sigmoid function

[0034] The Sigmoid function, also known as the Logistic function, simulates the characteristics of biological neurons, and its function expression is shown in equation (4).

[0035]

[0036] The Sigmoid function curve and gradient curve are as follows: Figure 2 As shown.

[0037] 2) Tanh function

[0038] The Tanh function is an improvement on the Sigmoid function. Compared with the Sigmoid function, the Tanh function has a mean output of 0, which can speed up the convergence of the network. However, the Tanh function also suffers from the vanishing gradient phenomenon.

[0039]

[0040] The Tanh function and its gradient curve are as follows: Figure 3 As shown.

[0041] 3) ReLU function

[0042] The ReLU function is an activation function proposed to address the vanishing gradient problem present in the Sigmoid and Tanh functions, and it is one of the most widely used activation functions in convolutional neural networks. Because the gradient remains unchanged when x is greater than 0, ReLU solves the vanishing gradient problem present in the Tanh function. The expression for the ReLU function is shown in equation (6).

[0043]

[0044] The ReLU function and its gradient curve are as follows: Figure 4 As shown.

[0045] In convolutional neural networks, pooling layers are often added between convolutional layers to reduce the number of parameters in the feature map, improve computational speed, and increase the receptive field; it's a downsampling operation. Pooling layers do not involve convolution operations; common pooling operations are max pooling and average pooling. Max pooling selects a filter operator similar to a convolution kernel, slides it across the output matrix of the previous layer, and calculates the maximum value of the intersection element as the new image point. Finally, iterates through all pixels in the image to obtain the max-pooled image. The advantage of max pooling is that it only retains the maximum value (feature) within the region, ignoring other values ​​and reducing noise. Average pooling, on the other hand, calculates the average value over the sliding region of the filter operator. Its advantage is that it considers every pixel; by adding all pixel values ​​and then averaging, it increases the model's robustness to noise. Figure 5 The diagram shows the process of max pooling and average pooling with a kernel size and step size of 2.

[0046] Depend on Figure 5 It can be seen that the pooling layer has no learnable parameters; only the pooling function needs to be specified. Assuming the input feature map has dimensions h and w, a sliding step size of s, and a pooling window size of k×k, the size of the output feature map is calculated using formula (7):

[0047]

[0048] Fully connected layers typically connect to the feature map output by the convolutional network. Their function is classification. A key characteristic is that each node is connected to all nodes in the layers above and below it, but has no connections to other non-adjacent layers. For example... Figure 6 As shown in the diagram, the leftmost column is the input layer, the rightmost column is the output layer, the middle three columns are the hidden layers, and the output layer has four output nodes, indicating that it can ultimately predict four categories.

[0049] (3) Classification of target detection algorithms

[0050] Object detection algorithms can be broadly classified into two categories: one-stage object detection algorithms and two-stage object detection algorithms.

[0051] One-stage object detection algorithms do not generate candidate boxes; they directly transform the problem of object bounding box localization into a regression problem. A single detection process yields the final detection result, quickly providing the object's category and location information. Common one-stage object detection algorithms include the SSD (LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single Shot MultiBox Detector [R], 2015. https: / / arxiv.org / pdf / 1512.02325.) and the YOLO (REDMON J, DIVVALA S, GIRSHICK R, et al. You Only Look Once: Unified, Real-Time Object Detection [R], 2015) series.

[0052] The two-stage object detection algorithm generates a series of candidate boxes as samples, and then uses a convolutional neural network to locate and classify the samples. Common two-stage target detection algorithms include: R-CNN (GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation [R], 2013. https: / / arxiv.org / pdf / 1311.2524.), Fast R-CNN (Girshick Ross. Fast R-CNN [C] / / Vision, IIC o C.2015 IEEE International Conference on Computer Vision:11-18 December 2015,Santiago,Chile:proceedings.Piscataway,NJ:IEEE,2015:1440–1448.),Faster R-CNN(Ren Shaoqing,HeKaiming,Girshick Ross,et al.Faster R-CNN:Towards Real-Time Object Detectionwith Region Proposal Networks[J].IEEE transactions on pattern analysis and machine Intelligence, 2017, 39: 1137–1149(6). Due to the differences between the two methods, their performance also differs; the former is superior in algorithm speed, while the latter is superior in detection accuracy and positioning precision.

[0053] 2. Object detection based on YOLOv5s model

[0054] Object detection models extract features from the input image using a backbone network and output a predicted bounding box. The predicted bounding box is then compared with the ground truth bounding boxes to determine if it represents a real target. If it is a real target, the bounding box is classified, its coordinates are regressed, and finally, it is displayed on the image. Convolutional neural networks (CNNs) effectively extract feature information from image data through convolutional operations. Most commonly used object detection algorithms are deep network algorithms based on CNNs, mainly divided into two categories: two-stage algorithms based on candidate regions and one-stage algorithms based on regression. Two-stage models involve a region proposal process, and the proposed regions are then fed into a classifier. One-stage algorithms typically use segmentation algorithms to create a grid of input images, then set several anchor boxes in each grid as predicted targets, and finally determine whether a real target is present by comparing the intersection-over-union (IoU) of the anchor boxes with the ground truth bounding boxes.

[0055] (1) Introduction to YOLO Series

[0056] The YOLO algorithm, proposed by Redmon et al. in 2016, is a convolutional neural network that can predict the location and class of multiple boxes at once. The network design strategy of the YOLO algorithm continues the core idea of ​​GoogleNet (Szegedy Christian, Liu Wei, Jia Yangqing, et al. Going deeper with convolutions[C] / / .Computer Vision and Pattern Recognition(CVPR), 2015 IEEE Conference on:Date,7-12June2015.[Piscataway,New Jersey]:[EEE],2015:1–9.), truly achieving end-to-end object detection and leveraging its speed advantage, but its accuracy has decreased. However, the YOLO 9000 algorithm proposed by Redmon et al. in 2016 (Redmon Joseph, Farhadi Ali. YOLO 9000: Better, Faster, Stronger [C] / / . 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017-2017: 6517–6525.) improved the accuracy of the original YOLO algorithm while maintaining its speed. The improvements are mainly in two aspects: 1) a series of improvements were made to the original YOLO detection framework to compensate for the lack of detection accuracy; 2) a method was proposed that combines object detection and object training into one. The YOLOv2 algorithm (Chang Yang-Lang, Anagaw Amare, Chang Lena, et al. Ship Detection Based on YOLOv2 for SAR Imagery[J]. Remote Sensing, 2019, 11: 786(7).) trains a network that uses downsampling to dynamically adjust under specific conditions. This mechanism allows the network to predict images of different sizes, achieving a balance between detection speed and accuracy.

[0057] The YOLO series of algorithms uses a single CNN to segment the input image into several grid cells. It employs an anchor box mechanism to select all cells containing potential objects and simultaneously predicts bounding boxes. The output indicates the predicted probabilities of each object, the width and height of the bounding box, and its horizontal and vertical coordinates. Unlike two-stage classification algorithms such as R-CNN, which are based on classification, YOLO, as a typical regression-based algorithm, does not require selecting Regions of Interest (ROIs) in the image. Instead, it directly predicts the category and bounding box of objects in the image all at once. This significantly reduces computational cost and improves detection speed while maintaining high accuracy, meeting the requirements of real-time forest fire monitoring. Since its inception in 2015, the YOLO series of algorithms has undergone several design improvements and performance optimizations.

[0058] (2) YOLOv5s model

[0059] The YOLOv5 model (ZHU X, LYU S, WANG X, et al. TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios [R], 2021. https: / / arxiv.org / pdf / 2108.11539.) was publicly released by Ultralytics on June 9, 2020. It is an improvement on the YOLOv3 model (REDMON J, FARHADI A. YOLOv3: An Incremental Improvement [R], 2018. https: / / arxiv.org / pdf / 1804.02767.) and includes four models: YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. YOLOv5s is the shallowest of the four versions, with the smallest feature map width. The other three versions increase the number of layers and widen the feature maps based on YOLOv5s, thus continuously enhancing their feature extraction and fusion capabilities. Considering the experimental hardware platform conditions, model detection accuracy, and network detection speed requirements, this invention selects the YOLOv5s model with the smallest network depth and width for training. The YOLOv5 model consists of a backbone network, a neck network, a head network, and an output.

[0060] (1) Input terminal

[0061] The input side has three main features: a) It uses Mosaic data augmentation, including random scaling, cropping, and arrangement for stitching; b) It embeds code that adaptively calculates the optimal anchor boxes for different datasets. By updating the difference between the area of ​​the predicted box output and the ground truth in each iteration, and then updating the network parameters in reverse, it calculates the optimal anchor box value, reducing the waste of model training costs caused by inappropriate initial anchor box settings; c) When using the model for inference testing, it uses adaptive image scaling to reduce black borders in the image, improving the information redundancy caused by excessive black border filling when uniformly scaling the original image to a standard size in existing object detection algorithms, thereby improving the inference speed of object detection.

[0062] (2) Backbone Network

[0063] The backbone network of the YOLOv5s model mainly consists of Focus, BottleneckCSP, and SSP networks, including modules such as Focus, Conv convolutional blocks, BottleneckCSP, and SSP. The Focus module slices the image to reduce its height and width, and then uses Concat to combine the height and width of the sliced ​​images, increasing the number of channels in the input image. The extracted feature maps are then processed through three sets of BottleneckCSP1 and Conv convolution operations. Finally, the SSP module is applied to the feature maps to improve model accuracy. The SSP module extracts features from multiple aspects of the image and uses Concat to aggregate the four sets of pooled feature maps.

[0064] (3) Neck

[0065] The Neck section consists of a CSP2_X structure, an FPN (Feature Pyramid Networks) + PAN (Path Aggregation Network) structure, and two combined feature pyramid structures. The feature pyramid structure improves the network's detection performance for targets of different scaling sizes. Specifically, the FPN uses a top-down feature pyramid, upsampling and concatenating high-level semantic information to obtain the predicted feature map; the PAN uses a bottom-up feature pyramid, downsampling low-level localization features to enhance positional information.

[0066] (4) Output

[0067] The prediction section includes a bounding box loss function and non-maximum suppression (NMS). The bounding box loss uses the GIOU_Loss function, which, compared to IOU_Loss, adds an intersection scale measure to address the issue of non-overlapping bounding boxes and predicted boxes. Candidate boxes are generated on feature maps at three different scales, and predictions are performed using these candidate boxes of varying sizes. Finally, a weighted non-maximum method is used to filter the generated candidate boxes, outputting object classification and bounding box regression.

[0068] This invention addresses the problems of poor accuracy and robustness of existing network methods in fruit identification under complex orchard environments. It proposes a thin-shelled pecan fruit detection method based on an improved YOLOv5s. On the basis of the basic structure of YOLOv5s, an attention mechanism and a StageBlock module are added to enhance the semantic and positional information of the network, reduce information loss, and improve the detection accuracy of fruit targets.

[0069] Compared to the shortcomings and deficiencies of existing technologies, this invention has the following beneficial effects: Based on the basic structure of YOLOv5s, this invention adds an attention mechanism and a StageBlock module, thereby enhancing the semantic and positional information of the network, reducing information loss, and improving the detection accuracy of fruit targets. To further verify the effectiveness and feasibility of the algorithm of this invention, a comparative experiment on model performance under different scenarios and lighting conditions was designed. Through experimental comparison of the performance evaluation metrics of SSD, RetainNet, YOLOv5s, Yolov7, and Faster RCNN models, the results show that when there are few fruit targets, the algorithm of this invention improves the accuracy by 5.4% compared with the original algorithm, reaching 97.2%. When the detection density is high, the improved algorithm of this invention performs better in detecting small targets, verifying the robustness of the target recognition and detection performance of the improved model of this invention. The detection speed can reach 54.3 fps / s, which can meet the real-time requirements of automatic picking and recognition. Attached Figure Description

[0070] Figure 1 This is the basic process of convolution calculation;

[0071] Figure 2 Figure A shows the Sigmoid function and its gradient curve; Figure B shows the function curve and its gradient curve.

[0072] Figure 3 Figure A shows the Tanh function and its gradient curve; Figure B shows the function curve and the function gradient curve.

[0073] Figure 4Figure A shows the ReLU function and its gradient curve, where Figure A is the function curve and Figure B is the function gradient curve.

[0074] Figure 5 This is a schematic diagram of the pooling process;

[0075] Figure 6 It is a fully connected network diagram;

[0076] Figure 7 This is a diagram of the SE attention mechanism;

[0077] Figure 8 This is a diagram of the StageBlock module structure;

[0078] Figure 9 This is a diagram of the improved YOLOv5s network structure model;

[0079] Figure 10 This is a comparison chart of mAP between the improved YOLOv5s and other models;

[0080] Figure 11 This is a schematic diagram of overlapping fruit clustering;

[0081] Figure 12 These are test results of the robustness of the SSD algorithm under different scenarios;

[0082] Figure 13 These are test results of the robustness of the RetinaNet algorithm under different scenarios;

[0083] Figure 14 These are test results of the robustness of the YOLOv5s algorithm under different scenarios;

[0084] Figure 15 These are test results of the robustness of the YOLOv7 algorithm under different scenarios;

[0085] Figure 16 These are test results of the robustness of the Faster R-CNN algorithm under different scenarios;

[0086] Figure 17 These are the test results of the robustness of the improved YOLOv5s algorithm under different scenarios. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0088] 1. Data Acquisition and Preprocessing

[0089] (1) Collection of thin-shelled pecan fruit samples

[0090] A review of relevant literature revealed that existing thin-shelled pecan datasets generally suffer from problems such as small sample sizes, poor image quality, and limited scene types. Therefore, this invention utilizes self-collected thin-shelled pecan fruit data to create a custom thin-shelled pecan dataset for training object detection models.

[0091] The images of thin-shelled pecans used in this invention were collected at the Thin-shelled Pecan Experimental Base in Zhangmiao Village, Houbai Town, Jurong City, Zhenjiang City, Jiangsu Province (119°9'6"E", 31°52'45"N). This area belongs to the mid-subtropical monsoon climate zone, with an average annual temperature of 15.6℃, an average annual sunshine duration of 2157 hours, and an average annual precipitation of 1018.6 mm. It is a typical low-mountain and hilly area with distinct seasons and abundant rainfall. The soil is yellow-brown loam, slightly acidic to neutral.

[0092] The photos were taken in August-September 2021 and September 2022 using a Nikon D7000. The image size is 2464×1632 pixels with a resolution of 300 dpi. The distance between the camera and the pecan fruits ranged from approximately 0.5 to 3.5 meters. By changing the focal length of the camera, different image sizes could be output, thus increasing the diversity of the output images. To prevent poor model performance due to insufficient training sample diversity and to maximize the acquisition of fruit feature information, the network was designed to be robust to complex environments. During image acquisition, considering the differences in imaging effects caused by different lighting conditions, images were captured on both sunny and cloudy days. Furthermore, multiple angles were taken, including both front-lit and back-lit shots, to increase sample diversity.

[0093] (2) Dataset sample labeling

[0094] Dataset annotation provides target references for training algorithms. Generally, data annotation is done manually before training. During training, the algorithm performs linear regression based on the labeled bounding boxes to continuously update parameters. This invention uses the image annotation tool LabelImg. Sample annotation mainly includes three parts: The Annotations folder stores information about target objects in the original images. The files are typically stored in XML format, containing the target category, the region of interest (ROI) where the target object is located, and a four-tuple (Xmin, Ymin, Xmax, Ymax) representing the coordinates of the upper left and lower right corners of the ROI's bounding box. This coordinate is used to determine the matching score based on the overlap rate between the output box and the labeled box when evaluating model accuracy. A bounding box is used to select objects of interest in the image (in this invention, thin-shelled pecans). The corresponding category label is selected from the pop-up options. The file content includes the image file name and path, the bounding box label, and location information. The ImageSets folder stores the paths and category annotation information for the images corresponding to the training, validation, and test sets. The JPEGImages folder stores the original images used for training. Preserve the recorded data format after completion. In the label file of this thin-shelled pecan image, you can see the number of thin-shelled pecans and the coordinates of each thin-shelled pecan's location in the image.

[0095] In supervised machine learning, datasets are often divided into training sets, validation sets, and test sets. Some algorithms require training parameters to control their behavior, but some parameters, such as network depth, weight dimensions, and the dimension of each neuron in the network layer, cannot be learned from the algorithm; these are called hyperparameters. To test the model's generalization ability in real-world environments, it needs to be tested on a test set that is independent and identically distributed with the training samples. A common method is to construct a validation set from the training data, dividing the training data into disjoint subsets. The training set is used for learning model parameters, while the validation set is used to measure generalization error during or after training and to update hyperparameters. This invention randomly divides 1307 labeled samples in the dataset into three parts: 80% training set, 10% validation set, and 10% test set, which are used for training, validation, and testing tasks of an object detection network model, respectively.

[0096] (3) Lighting correction for fruit photos

[0097] During the shooting process, rapid and uneven changes in sunlight and errors in automatic metering by the camera may lead to overexposure and underexposure of photos. To reduce interference with subsequent detection of pecans, it is necessary to correct photos with exposure problems. Due to the rapid changes in sunlight during shooting and metering errors within the camera, overexposed photos may be taken. In such photos, prominent details are lost in overexposed areas, resulting in insufficient detail in the image. Therefore, it is easy to cause insufficient feature learning of thin-shelled pecans in the network, leading to a decrease in subsequent recognition accuracy. To solve the problem of overexposure in some photos during shooting, this invention adopts the Dark Channel Image Dehazing (DCID) algorithm (KAIMINGHE, JIANSUN, XIAOOUTANG. Single Image Haze Removal Using Dark Channel Prior[J].IEEE,2009:1956–1963.), which uses dynamic thresholding to detect white points in the image. The processing includes two steps: white point detection and white point adjustment, which suppresses the exposure of backlit or overexposed photos in the dataset, making the details in the image clearer and facilitating feature extraction.

[0098] Some photos may be underexposed due to insufficient light and changes in shooting angle. Because underexposed areas lack fruit details, such images have relatively low contrast and dull colors. Therefore, they can easily interfere with the detection and identification of pecans. This invention uses Zero-Reference Deep Curve Estimation (Z-RDCE) (Guo Chunle, Li Chongyi, Guo Jichang, et al. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement[C] / / .2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2020-2020:1777–1786.) to enhance low-light images and compensate for exposure in underexposed images in the dataset. The average time for uneven illumination correction of all underexposed and overexposed images is 0.014 seconds. After illumination correction, the images are input into the network structure.

[0099] This invention provides a detailed introduction to the source of the thin-shelled pecan fruit dataset and the data preprocessing steps. It also describes the use of LabelImg for supervised learning of the fruit dataset sample annotation, demonstrates the organization and content of the VOC format thin-shelled pecan fruit dataset, and finally, performs illumination correction on the abnormally exposed fruit photos during the shooting process. This includes using the Dark Channel Image Dehazing (DCID) algorithm for overexposed fruit photos and the Zero-Reference Deep Curve Estimation (Z-RDCE) method for underexposed fruit photos, thus correcting the thin-shelled pecan fruit photos to the normal exposure range and preparing them for image input into the network.

[0100] 2. Research on the Improved YOLOv5s Algorithm

[0101] Since the target detected in this invention is only thin-shelled pecans, which are a single type of target and similar in size and shape, the YOLOv5s network is mainly optimized from two aspects: the construction of the basic network and improving the recognition accuracy of small targets and fruits in complex environments.

[0102] Since the design of this invention focuses on the harvesting of thin-shelled pecan fruits using a robotic arm, and visual detection and target localization are necessary to provide information for the harvesting process, it is important to pay more attention to the real-time extraction and detection of the global feature information of the fruit.

[0103] For the reasons mentioned above, this invention will improve the structure of the YOLOv5s target detection algorithm based on the YOLOv5s algorithm, while minimizing the impact on detection speed. By introducing the SE attention module and the StageBlock module, the detection accuracy of the algorithm for target fruits in specific environments will be increased, thereby improving the detection capability of the algorithm.

[0104] (1) SE attention mechanism

[0105] SE attention mechanisms are characterized by their simplicity, effectiveness, and wide applicability, such as... Figure 7 As shown, global max pooling is performed on the input feature map to obtain the global features of each channel, resulting in a 1*1*C vector. This vector is then processed using a fully connected layer to learn the non-linear relationships between different channels. Subsequently, the sigmoid activation function is used as weights on the input feature map, assigning different weights to different channels. The calculated result is then added to the original feature map.

[0106] (2) StageBlock module

[0107] The StageBlock module is an important structure in HRNet, such as... Figure 8 As shown. For position-sensitive tasks like object detection, a crucial method to improve the accuracy of target localization is to maintain high image resolution during neural network processing. To reduce information loss during feature map upsampling and downsampling, HRNet processes multiple feature maps with different channels and scaling rates in parallel. Through mutual upsampling and downsampling between different branches and feature map fusion, information exchange between different branches is achieved, thereby enhancing semantic and positional information and reducing information loss.

[0108] The input feature maps are processed using different convolutions, resulting in output channels that are 1 / 8 and 1 / 4 of the input channel count, respectively. Then, two basic blocks are used for processing. Next, the feature map with 1 / 8 of the channels is downsampled by a factor of 2, and the feature map with 1 / 4 of the channels is upsampled by a factor of 2. After fusion, ReLU activation functions are applied to each feature map. Then, two basic blocks are used to process each feature map separately: the feature map with 1 / 4 of the channels is upsampled by a factor of 2, reducing the channel count to 1 / 8 of the original, and then fused with the feature map with 1 / 8 of the channels. Finally, convolution is used to restore the channel count to its original size.

[0109] The improved YOLOv5s network model is as follows: Figure 9 As shown.

[0110] (3) Performance evaluation index of target detection model

[0111] Commonly used performance evaluation metrics for object detection models include precision, recall, F1 score, mAP, and inference time. The calculation of precision and recall can be explained using a confusion matrix. Table 1 shows a binary classification confusion matrix, where TP represents positive samples predicted as positive, TN represents negative samples predicted as negative, FP represents negative samples predicted as positive, and FN represents positive samples predicted as negative.

[0112] Table 1. Binary Classification Confusion Matrix

[0113]

[0114] The Precision metric reflects the proportion of correctly predicted positive objects out of the total number of predicted positive objects (i.e., how many of the binary classifier's positive predictions are accurate). The calculation formula is:

[0115]

[0116] The recall metric reflects the proportion of correctly predicted positive objects out of all labeled objects (i.e., how many true positive examples are recalled by the binary classifier). The calculation formula is:

[0117]

[0118] A high-performance object detection model needs to balance precision and recall values. As shown in equations (8) and (9), precision and recall values ​​are usually contradictory and cannot both reach their maximum values ​​simultaneously. To comprehensively reflect the overall performance of the object detection model, the harmonic mean of the two values, F1-score, is introduced as a comprehensive evaluation metric. F1-score, using the harmonic mean, tends to favor smaller values ​​when aiming for the highest possible precision and recall values, while also penalizing extreme cases where the difference between precision and recall values ​​is too large, thus reducing the discrepancy. The F1-score calculation formula is:

[0119]

[0120] AP (Average Precision) calculates the average precision of a single class in an object detection task. Its value is the area under the curve formed by multiple precision and recall values ​​for that class. A higher AP value for a class indicates higher overall precision and recall. mAP (mean Average Precision) sums the AP values ​​of all classes and then averages them to achieve a comprehensive evaluation of the detection results for each class. 0.5 refers to a 50% overlap rate between the predicted bounding box and the actual bounding box in an object detection task. Under the VOC evaluation standard, an overlap rate greater than or equal to 50% is generally considered a correct prediction, and less than 50% is considered an incorrect prediction. mAP@0.5 refers to the mAP value under the condition that the overlap rate threshold between the predicted bounding box and the actual bounding box is 0.5. The formula for calculating mAP is:

[0121]

[0122] In equation (11), N represents the number of target categories in the target detection task, AP i The AP value for the i-th category.

[0123] Inference time is a crucial metric for deploying deep learning object detection models in real-world production environments. Whether the model can meet the time requirements of the task directly impacts its final production performance. Unlike the model training phase, which focuses on improving object detection accuracy, inference time examines the model's time cost in practical application scenarios such as video image processing and IoT signal transmission. Inference time is typically in the millisecond (ms) range.

[0124] (4) Performance analysis of the improved YOLOv5s model for detecting thin-shelled pecans

[0125] To further explore the performance of the improved YOLOv5s model in the thin-shelled pecan target detection task, several classic target detection models were selected as comparison models. Using the thin-shelled pecan dataset independently collected in this invention, the models were trained under a unified experimental environment and training parameters. The performance metrics obtained from different models were analyzed. Simultaneously, the robustness of the improved YOLOv5s model was verified to ensure that it meets reliability requirements in unstructured environment recognition and detection tasks.

[0126] A. Introduction to Experimental Environment and Parameter Settings

[0127] All experiments conducted in this invention were performed on the Ubuntu (20.04) operating system, using the open-source deep learning platform PyTorch as the deep learning framework. The relevant experimental environment configuration is shown in Table 2. CUDA, mentioned in the table, refers to a general-purpose parallel computing architecture introduced by NVIDIA. By installing the deep learning network library cuDNN, the powerful parallel computing capabilities of NVIDIA graphics cards can be utilized to accelerate the training of deep learning neural networks on NVIDIA graphics cards.

[0128] Table 2 Experimental Environment Configuration Table

[0129]

[0130]

[0131] All object detection models used in this invention underwent training, validation, and testing within the aforementioned experimental environment. To quantify the detection performance of the improved YOLOv5s model, this invention selected five advanced object detection models—SSD, RetinaNet, YOLOv5s, YOLOv7, and Faster R-CNN—as comparative models. To more accurately compare the performance differences of different models in the thin-shelled pecan detection task, the training hyperparameter settings for all models were kept consistent. The training hyperparameter settings for all models are shown in Table 3. Epoch refers to the process by which each model completes one forward and backward propagation loop, inputting the entire training set into the network for training; Batchsize refers to the number of training samples in each iteration of each model, adjusted according to actual conditions; Learningrate refers to the hyperparameter controlling the speed of training weight updates and convergence speed for each model; the Learningrate exponential decay is set to 0.003; Momentum refers to a hyperparameter set by each model to accelerate convergence speed and improve training accuracy, generally set to 0.9.

[0132] Table 3. Hyperparameter settings for experimental training

[0133]

[0134] B. Performance Comparison Analysis of Different Models in Thin-Shelled Pecan Target Detection Task

[0135] Generally, a high-performance object detection model needs both high precision and high recall. However, in the PR curve, recall tends to decrease as precision increases. This invention sets the IoU threshold to a uniform 0.5. Table 4 shows the performance evaluation metrics of different models in the thin-shelled pecan fruit object detection experiment, where Inferencetime is the detection time (including illumination correction time), and Fps is the detection frame rate per second.

[0136] Table 4 Comparison of performance indicators of different models

[0137]

[0138]

[0139] To more intuitively illustrate the performance differences between different models, the data for all model evaluation metrics in Table 4 are summarized. Figure 10 The experimental results of the improved YOLOv5s model are compared with those of five object detection models: SSD, RetinaNet, YOLOv5, YOLOv7, and Faster R-CNN.

[0140] from Figure 10 The results show that in the thin-shelled pecan fruit detection experiment, the improved YOLOv5s achieved a 7.6% improvement in mAP compared to the SSD network and a 5.8% improvement compared to the RetinaNet network. Compared to the original YOLOv5s, it improved by 5.4%, and compared to the Faster R-CNN network by 3.7%. Compared to YOLOv7, the improved YOLOv5s had a slightly higher mAP of 0.4%. Regarding detection speed, due to the increased number of model parameters, the improved YOLOv5s network achieved a slightly lower detection speed than the original YOLOv5s network, but still reached 54.3 fps / s, which meets the real-time requirements for fruit detection.

[0141] 3. Overlapping fruit clustering

[0142] Because thin-shelled pecans often grow in clusters, the previous fruit detection methods primarily focused on individual fruits. To reduce unnecessary harvesting actions and improve efficiency when using a robotic arm, this invention proposes a method for calculating the overlapping area of ​​multiple thin-shelled pecan fruits, providing better harvesting information. Thin-shelled pecan detection is as follows: Figure 11 As shown. The coordinates of the rectangular detection box at the top left and bottom right corners of the thin-shelled pecan are (x1, y1). 21 The detection boxes for another thin-shelled pecan are (x2, y2) and (x3, y3) and (x4, y4). Assumptions:

[0143]

[0144] Where, x min y min x max y max These are the coordinates of the top left and bottom right corners of the overlapping area, respectively.

[0145] If the two walnut test frames overlap, the overlap area S is:

[0146] S=(x max -x min )×(y max -y min (19)

[0147] Combining the method described above for calculating the overlap area S, when the overlap area exceeds a certain threshold, the two target detection boxes are clustered into a larger new detection box. This process is repeated until no new detection boxes can be generated, ultimately achieving the clustering calculation for the entire image. Through these steps, two or more thin-shelled pecans can be clustered together, allowing the robotic arm to reduce the number of grasping operations and improve harvesting efficiency.

[0148] 4. Improve network robustness analysis

[0149] (1) Comparison of robustness of fruit recognition in different scenarios

[0150] To verify the robustness of the proposed network, images under different lighting conditions and growth states were selected, and the proposed network was used to detect pecans. Figures 12-17 As shown, Figure 17 The results are part of the detection results of the algorithm of this invention. By comparing with other detection results, it can be seen that the algorithm of this invention can detect thin-shelled pecan fruits under various complex environments such as different lighting, different degrees of occlusion, and large field of view. The position and category of its prediction box are relatively accurate and there are few missed detections and false detections, which shows strong robustness.

[0151] During actual harvesting, the number and size of pecans change depending on the distance between the camera and the tree. When the camera is close to the tree, the number of pecans is relatively small, their size is relatively large, and the objects to be detected are relatively clear and complete, making them easier to detect. However, when the camera is far from the tree, as the size of the pecans decreases and the number of pecans increases, there will be more obstruction and overlap, making detection more difficult.

[0152] Therefore, comparative experiments were established under different scenarios, divided into 6 categories according to different scenarios, to compare the detection performance of the two methods before and after the improvement under different fruit quantities. Figures 12-17 The results are partial. It can be seen that all six algorithms perform well in detecting a small number of fruits; however, when detecting multiple fruits or densely packed fruits, SSD, RetinaNet, the original YOLOv5s, YOLOv7, and Faster R-CNN exhibit some missed detections and inaccurate localization. In a large field-of-view environment, SSD, RetinaNet, and Faster R-CNN significantly miss or misdetect more fruits than the algorithm presented in this invention.

[0153] The detection results of the six networks show that for a small number of fruits without overlap or occlusion, the detection performance of the six networks is comparable, and all can accurately identify thin-shelled pecan fruits. However, for a large number of fruits with overlap, SSD, RetinaNet, the original YOLOv5s, and Faster RCNN networks showed varying degrees of missed detections and false detections. For thin-shelled pecans with high overlap, severe occlusion, and large field of view, the false detection rate was high and the recognition accuracy was low.

[0154] The improved YOLOv5s network of this invention performs better than previous networks, without serious omissions of fruits, and also shows relatively good detection results for overlapping and occluded fruits, ensuring the network's detection accuracy. The method of this invention can be well applied to fruit detection under different quantities, sizes, and overlapping / occlusion environments, and its performance is superior to the previous YOLOv5s algorithm, especially for fruit detection in dense and large field-of-view environments, where performance is significantly improved, verifying the effectiveness of the proposed improved method.

[0155] (2) Evaluation of fruit sample count

[0156] The overlap coefficient (OC) is used to evaluate the accuracy of the detection results, correctly classifying the proportion of objects within the bounding box that belong to either the target or the background. Since thin-shelled pecans exhibit cluster overlap, this experiment primarily employs algorithms to statistically analyze the fruit area and individual fruit area to accurately count the number of pecans. Clustering detection using the obtained bounding box coordinates, the algorithm's automatic counting accuracy reached 93.5%, with high precision in identifying multiple fruit targets into a single cluster, as shown in Table 5. However, the lack of depth information in the images may lead to false detections. Therefore, a depth camera will be used in the subsequent robotic arm visual harvesting process to obtain the depth information of the fruit targets, thereby reducing the error rate in fruit clustering.

[0157] Table 5. Number of test sets and error rate for fruit samples.

[0158]

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting thin-shelled pecan fruits based on an improved YOLOv5, characterized in that, The method includes the following steps: S1. Based on the physical characteristics and morphological features of thin-shelled pecans, collect sample images of thin-shelled pecans under different times and different light conditions. Based on the sample features, preprocess the collected images to establish a target fruit recognition and detection dataset. S2. The improved YOLOv5s network is used to train the recognition and detection dataset, and the robustness and real-time performance of the algorithm are verified. The improved YOLOv5s network is an improvement on the original YOLOv5s object detection algorithm structure. By embedding SE attention modules after the second and third CBS modules, and embedding StageBlock modules after the fourth CBS module, the algorithm's detection accuracy for target fruits in specific environments is increased. Step S2 is followed by the following steps: S3. Calculate the area of ​​the overlapping fruit region and the number of overlapping fruits to cluster the thin-shelled pecan fruits; In step S2, the StageBlock module is specifically used to: process the input feature maps using different convolutions, with the number of output channels being 1 / 8 and 1 / 4 of the number of input channels, respectively, and process them using two basic blocks; then, downsample the feature map with 1 / 8 of the channels by a factor of 2, and upsample the feature map with 1 / 4 of the channels by a factor of 2, and after fusion, process it using the ReLU activation function, and then process it using two basic blocks, upsample the feature map with 1 / 4 of the channels by a factor of 2, reduce the number of channels to 1 / 8 of the original, fuse it with the feature map with 1 / 8 of the channels, and use convolution to restore the number of channels to the original size.

2. The method as described in claim 1, characterized in that, In step S1, the preprocessing includes dataset sample labeling and illumination correction for unevenly exposed photos.

3. The method as described in claim 1, characterized in that, In step S2, the SE attention module is specifically used to: obtain the global features of each channel by performing global max pooling on the input feature map, obtain a 1*1*C vector, process the vector using a fully connected layer to learn the nonlinear relationship between different channels, apply the Sigmoid activation function as weights to the input feature map, that is, assign different weights to different channels, and add the calculated result to the original feature map.

4. The method as described in claim 1, characterized in that, Step S3 includes the following specific process: determining whether the overlap area of ​​the two walnut test boxes is greater than a threshold. If so, cluster the two target detection boxes into a larger new detection box. Repeat the above process until no new detection boxes can be generated, and finally obtain the clustering of the entire image.

Images