A litchi image detection method for picking robots based on improved YOLOv8

Abstract

The present invention relates to the field of automated agricultural technology, specifically a lychee image detection method for a picking robot based on an improved YOLOv8. Lychee image data is collected in a lychee orchard to construct an orchard lychee dataset containing lychee fruit and picking point annotation information. The method then trains a YOLOv8-iGR model, formed by improving the YOLOv8-pose model using an iSE architecture, a GELAN module, and an RFAHead detection head module, to achieve simultaneous recognition of lychee fruit and picking points. The present invention proposes an iSE architecture, combines the GELEN and RFAHead detection head modules, and improves the YOLOv8-pose model to form a YOLOv8-iGR model. This model achieves simultaneous recognition of lychee fruit and picking points, significantly improving the accuracy and efficiency of simultaneous detection of lychee fruit and picking points, thereby resolving the issues of slow multi-step detection speed and difficulty in coping with complex and changing environments.

Description

Technical Field

[0001] The present invention relates to the technical field of automated agriculture, and in particular to a litchi image detection method for a picking robot based on an improved YOLOv8. Background Art

[0002] Lychee picking is the most time-consuming and labor-intensive part of the fruit production cycle, characterized by strong seasonality, high labor intensity, and high costs. To reduce production costs and increase farmers' incomes, the development of intelligent lychee-picking robots is urgent. The key to lychee picking lies in accurately identifying and locating the picking point on the fruit stem to avoid fruit damage. However, in natural environments, accurately detecting the picking point on the lychee stem faces many challenges, such as interference from non-main lychee branches, obstruction by leaves and branches, and the impact of varying natural lighting and background factors on visual recognition. Therefore, using machine vision to accurately identify and locate the picking point is a key issue for achieving intelligent operation of lychee-picking robots.

[0003] In a natural environment, the difficulties in accurately detecting the picking point of the litchi stalk lie in: interference from non-litchi main fruit branches; obstruction by leaves or branches; and the impact of changes in natural light and background factors on visual recognition.

[0004] To date, many researchers have contributed to visual systems for lychee recognition. However, the primary methods used in these studies are traditional image processing methods, such as color space transformation, threshold segmentation, and clustering to segment fruit and main branches. However, these methods can only handle simple backgrounds, single clusters of lychee fruit, and fully visible main branches. In natural environments, these methods are susceptible to factors such as lighting variations and leaf occlusion. Furthermore, the distribution of lychee clusters is random, and captured images often contain multiple clusters of lychee fruit. These factors limit the performance and robustness of traditional image processing methods in natural environments.

[0005] With the rapid development of deep convolutional neural networks in object recognition, some researchers have applied deep learning to litchi picking recognition.

[0006] However, in actual use, there are still some difficulties and challenges. Due to the large number of operation steps, the detection speed is low, and the method combining deep learning and image processing has poor robustness in complex and changing environments. Summary of the Invention

[0007] The purpose of the present invention is to provide a litchi image detection method for a picking robot based on an improved YOLOv8, so as to solve the problems of many operation steps, low detection speed, and poor robustness of the method combining deep learning and image processing in complex and changing environments.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a litchi image detection method for a picking robot based on an improved YOLOv8, comprising the following steps:

[0009] S1. Dataset collection and construction: In a litchi orchard, a camera mounted on a robotic arm is used to collect litchi image data and construct an orchard litchi dataset containing the annotation information of litchi fruits and picking points.

[0010] S2. Model training: Based on the orchard litchi dataset in step S1, the YOLOv8-iGR model is trained by improving the YOLOv8-pose model through the iSE architecture, GELAN module, and RFAHead detection head module to achieve simultaneous recognition of litchi fruits and picking points;

[0011] The iSE architecture combines convolutional neural networks with spatial attention mechanisms to improve the model's ability to capture global and local features.

[0012] The GELAN module is used to achieve richer gradient combinations by splitting the gradient flow, reducing the model calculation amount while improving performance;

[0013] The RFAHead detection head module is designed based on RFAConv to solve the problem of large convolution kernel parameter sharing and more effectively extract feature information;

[0014] S3, feature extraction and classification regression: The litchi dataset is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points;

[0015] S4. Analysis and evaluation of detection results: The test set is tested using the divided data set to detect the key points of picking litchi in the orchard and evaluate the detection effect of the YOLOv8-iGR model.

[0016] Preferably, the orchard litchi dataset collected in step S1 includes litchi images of different varieties, different time periods and different shooting angles, covering various growth conditions of litchi in the natural environment.

[0017] Preferably, in step S1, the collected images are saved as RGB images and divided into a training set, a validation set and a test set.

[0018] Preferably, the step S2 of training the YOLOv8-iGR model further includes constructing a model training environment, specifically:

[0019] Network training based on Ubuntu system and PyTorch framework;

[0020] Set training parameters, including batch size, maximum number of iterations, learning rate, momentum parameter, and weight decay coefficient;

[0021] After non-maximum suppression processing, positive and negative samples are screened according to the confidence threshold.

[0022] Preferably, in step S4, when evaluating the detection effect of the YOLOv8-iGR model, an ablation experiment and a comparative experiment are used to verify the performance of the YOLOv8-iGR model. The ablation experiment is used to evaluate the impact of different modules on the model performance, and the comparative experiment is used to compare the YOLOv8-iGR model with mainstream target detection algorithms.

[0023] Preferably, in step S4, when evaluating the detection effect of the YOLOv8-iGR model, OKS is used as the detection performance evaluation index of the key point, and the pixel Euclidean distance is used as the evaluation index of the picking point position prediction error to evaluate the detection effect of the model.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. This paper proposes the iSE architecture, combines the GELEN and RFAHead detection head modules, and improves the YOLOv8-pose model to form the YOLOv8-iGR model, which realizes the synchronous recognition of litchi fruits and picking points, significantly improving the accuracy and efficiency of synchronous detection of litchi fruits and picking points, thereby solving the problems of slow multi-step operation detection and difficulty in coping with complex and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of an image dataset of a litchi picking robot detection method based on improved YOLOv8 in the present invention;

[0027] Figure 2 This is a schematic diagram of a specific application technology route of a litchi image detection method for a picking robot based on an improved YOLOv8 in the present invention;

[0028] Figure 3 This is a schematic diagram of the overall framework of the YOLOv8-iGR model and its component modules for a litchi image detection method for a picking robot based on improved YOLOv8 in the present invention;

[0029] Figure 4 Schematic diagram of the working principle of SaE of the present invention;

[0030] Figure 5 Schematic diagram of the structure of the iSE of the present invention;

[0031] Figure 6Schematic diagram of the CSPNet and ELAN modules of the present invention;

[0032] Figure 7 This is a schematic diagram of the GELAN module structure of the present invention;

[0033] Figure 8 Schematic diagram of the RFAConv structure of the present invention;

[0034] Figure 9 Schematic diagram of the RFAHead structure of the present invention;

[0035] Figure 10 mAP in the ablation experiment of this invention kp Schematic diagram of curve comparison;

[0036] Figure 11 Schematic diagram of the comparison results of ablation experiments on different modules of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] To date, many researchers have contributed to the visual system for lychee recognition. Li et al. proposed an improved YOLOv7-Litchi detection algorithm, which has good detection capabilities for dense distribution and fruit occlusion in complex scenes. Yu et al. proposed a lychee recognition method based on a red, green, and blue depth camera. The depth image of the lychee acquired by the depth camera is segmented to remove redundant information other than the lychee. Then, a random forest binary classification model is trained using color and texture features to identify the lychee fruit. Jiao et al. proposed a lychee detection method based on YOLOx, which uses various compression operations to achieve a compact model deployed on edge devices. However, the above method is only applicable to the fields of lychee yield estimation or maturity estimation. For lychee picking, in addition to fruit identification, it is also necessary to locate the picking point.

[0039] Initially, the main method for determining litchi picking points was based on a combination of machine learning algorithms and machine vision. Zhuang et al. proposed a retinex algorithm to compensate for the illumination distribution of litchi images in low-light conditions. They then combined red / green saturation mapping and Otsu thresholding to segment the main branches. Finally, they analyzed the angular structure distribution generated by Harris corner detection to locate the picking point, achieving an accuracy of 83%. Xiong et al. proposed a method for identifying litchi at night and calculating the picking point. They first used an improved fuzzy clustering algorithm (FCM) to remove background information from litchi images, then used the Otsu algorithm to segment the litchi fruit. Finally, they used Harris corner detection to detect the picking point. This method achieved an accuracy of 93.75% for litchi identification and a maximum accuracy of 97.5% for calculating the picking point. Shu et al. developed a stereo vision system for a litchi picking robot. They used an image color algorithm to calculate the hue and color layers of the litchi fruit. They then analyzed the appearance characteristics and position data of the litchi fruit to obtain the three-dimensional coordinates of the litchi fruit.

[0040] The primary methods used in these studies are traditional image processing approaches, such as color space transformation, threshold segmentation, and clustering to segment fruit and main branches. However, these methods can only handle simple backgrounds, single clusters of lychee fruit, and fully visible main branches. In natural environments, these methods are susceptible to factors such as lighting variations and leaf occlusion. Furthermore, the distribution of lychee clusters is random, and captured images often contain multiple clusters of fruit. These factors limit the performance and robustness of traditional image processing methods in natural environments.

[0041] With the rapid development of deep convolutional neural networks in object recognition, some researchers have applied deep learning to lychee picking and recognition. Peng et al. constructed a ResDense-focal-DeepLabV3+ network for accurate segmentation of lychee stems in orchard environments, and achieved improved mIoU compared to other models in simple, medium, and complex images. Xie et al. proposed an improved DeepLabv3+ lychee stem segmentation method to address the low accuracy of lychee stem segmentation in natural environments. The model achieved an average intersection of union and average pixel accuracy of 90.28% and 94.95%, respectively, and a frame rate of 19.83 per second (FPS). The above method only segments the lychee stems, while robotic lychee picking requires knowing the precise location of the picking points. Zhong et al. proposed a YOLACT-based method for detecting the main branches of litchi fruits. They first annotated the litchi fruits and main branches as a cluster, then used the YOLACT model to segment the main branches and litchis. Finally, the center of the main branch was used as the picking point. Skeleton extraction and least-squares fitting were used to determine the angle of the main branch. This method achieved an accuracy of 89.7% and an F1 score of 83.8% for the picking point. Qi et al. used YOLOv5 to detect stems in litchi images, extracted the region of interest (ROI) of the main stem, and segmented it using PSPNet. After performing image processing on the segmented image, they obtained the pixel coordinates of the picking point on the main stem. This method achieved an accuracy of 92.50% and a recall of 76.29%, respectively. These methods further ensured the accuracy of picking point localization, but they also faced challenges. Due to their multi-step operation, detection speed was low. Furthermore, the combination of deep learning and image processing methods was less robust in complex and changing environments.

[0042] The present invention provides a technical solution: a litchi image detection method for a picking robot based on an improved YOLOv8, comprising the following steps:

[0043] S1. Dataset collection and construction: In a litchi orchard, a camera mounted on a robotic arm is used to collect litchi image data. The collected litchi dataset includes litchi images of different varieties, different time periods, and different shooting angles, covering various growth conditions of litchi in the natural environment. In addition, an orchard litchi dataset containing litchi fruit and picking point annotation information is constructed. The dataset is used for model training, and the collected images are saved as RGB images and divided into a training set, a validation set, and a test set.

[0044] S2. Model training: First, build a model training environment, specifically:

[0045] Network training based on Ubuntu system and PyTorch framework;

[0046] Set training parameters, including batch size, maximum number of iterations, learning rate, momentum parameter, and weight decay coefficient;

[0047] After non-maximum suppression processing, positive and negative samples are screened according to the confidence threshold;

[0048] Then, based on the orchard litchi dataset in step S1, the YOLOv8-iGR model, which is an improvement of the YOLOv8 model using the iSE (IRB-SaE) architecture, the GELAN module, and the RFAHead detection head module, is trained to achieve simultaneous recognition of litchi fruits and picking points.

[0049] The iSE (IRB-SaE) architecture combines convolutional neural networks with spatial attention mechanisms to improve the model's ability to capture global and local features.

[0050] The GELAN module is used to achieve richer gradient combinations by splitting the gradient flow, reducing the model calculation amount while improving performance;

[0051] The RFAHead detection head module is designed based on RFAConv to solve the problem of large convolution kernel parameter sharing and more effectively extract feature information;

[0052] S3, feature extraction and classification regression: The litchi dataset is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points;

[0053] S4. Analysis and evaluation of detection results: The test set is tested using the divided data set to detect the key points of litchi picking in the orchard, and the detection effect of the YOLOv8-iGR model is evaluated. OKS is used as the key point detection performance evaluation indicator, and the pixel Euclidean distance is used as the evaluation indicator of the picking point position prediction error to evaluate the detection effect of the model. Ablation experiments and comparative experiments are used to verify the performance of the YOLOv8-iGR model. The ablation experiment is used to evaluate the impact of different modules on the model performance, and the comparative experiment is used to compare the YOLOv8-iGR model with mainstream target detection algorithms.

[0054] See also Figure 1-11 The present invention also provides a specific embodiment of a litchi image detection method using a picking robot based on an improved YOLOv8-pose, comprising the following steps:

[0055] S1. Dataset Collection and Construction: In a litchi orchard, images of red-ripe litchi trees for training and testing were captured using a Realsense D435i camera and a mobile phone. The camera was 30 to 100 cm away from the target. The litchi varieties included "Fei Zi Xiao", "Gui Wei", and "Nuomi Ci". The acquired images were saved as 1000x750 pixel RGB images. To ensure that the dataset reflects the characteristics of the orchard in a natural environment, a total of 1281 images of different varieties of litchi trees were captured in the orchard at different time periods (9:00 am, 2:00 pm, and 6:00 pm). Figure 1 . The shooting angles are frontal, upward and downward, and the lychee calyx photographed from the upward angle is blocked; due to different lighting conditions in the natural environment, some images have sufficient lighting, and the outlines of the lychees and main fruit branches are clear, while some images are dark overall, which can easily lead to misjudgment in the identification of the picking points on the main fruit branches of lychees; in a complex natural environment, there will be occlusions from leaves and branches, which will increase the difficulty of the model in identifying lychees and main fruit branches. The collected orchard lychee dataset includes lychee images of different varieties, different time periods and different shooting angles, covering various growth conditions of lychees in the natural environment, and constructing an orchard lychee dataset containing lychee fruit and picking point annotation information. The 1000x750 pixel RGB image dataset is divided into a ratio of 8:1:1. 1025 images are randomly selected as the training set, 128 images as the validation set, and 128 images as the test set;

[0056] S2. Model training: First, build a model training environment, specifically:

[0057] Network training is based on the Ubuntu system and the PyTorch framework. The main configuration of the computer hardware is i9-9900k CPU and GeForce RTX 3090 GPU. CUDA 11.0, CUDNN 8.0.1 and python 3.8 are installed. The training environment of different algorithms is the same, and the training parameters are set as follows: the batch size of the model is set to 32, the maximum number of iterations is set to 300, and if the model performance does not improve within 50 consecutive epochs, the training is stopped. The image input size is set to 640x 640 pixels, the initial learning rate is 0.01, the cyclic learning rate is 0.2, the momentum parameter is 0.937, and the weight decay coefficient is 0.0005. In addition, in the prediction box classification obtained after non-maximum suppression processing, the prediction box with a confidence value greater than the 0.5 threshold is defined as a positive sample, otherwise it is defined as a negative sample;

[0058] The efficient spatial feature extraction capability of convolution kernels and the effectiveness of attention mechanisms have been well established in the field of computer vision. In this paper, we are committed to integrating the capabilities of convolution and attention mechanisms to improve the positioning capability of picking points. YOLOv8n is a single-stage target detection model, such as Figure 3 As shown in the figure, to improve the model's performance in lychee recognition and detection of key points for picking fruit from main branches, the iSE architecture was proposed. This fused module significantly improves model performance compared to using convolution or attention mechanisms alone. Furthermore, the model needs to be deployed on mobile devices. The introduction of GELAN reduces computational overhead without sacrificing performance. The RFAHead detection head module is used to simultaneously detect key points for picking fruit and main branches. Built on RFAConv, it effectively addresses the issue of parameter sharing during feature extraction, improving detection accuracy.

[0059] Then, based on the orchard litchi dataset in step S1, the YOLOv8-iGR model, which is an improvement of the YOLOv8 model using the iSE (IRB-SaE) architecture, the GELAN module, and the RFAHead detection head module, is trained to achieve simultaneous recognition of litchi fruits and picking points.

[0060] in:

[0061] The iSE architecture is as follows: In recent years, the effectiveness of attention mechanisms in improving model expressiveness and precise object localization has been well-established in the field of computer vision. The Squeeze Aggregate Excitation (SaE) module is a novel aggregated multilayer perceptron proposed in SENetv2 (SENetv2), designed to surpass the performance of existing architectures. The SaE module combines the Squeeze-and-Excitation operation, fusing the squeeze and excitation module with a multi-branch DenseLayer. Compared to SE (SENet), this fusion approach enhances the network's ability to capture channel information and global knowledge, resulting in better feature representation.

[0062] The working principle of SaE is as follows Figure 4 As shown in the figure, after the input features are resized and the number of channels is adjusted by the convolutional layer, the global average pooling layer is used to count the channel information. These channel information are then collected in the FC layer for squeezing and compressing the feature size. Then the outputs of all FC branches are concatenated and excitation is performed to restore the size to its original shape. Finally, the squeezing and excitation results are connected to the input of the residual module. The SaE module formula can be expressed as: SaE = x + F (x·E X (∑S q (x))), where:

[0063] Sq The function represents the squeeze operation, including the FC layer;

[0064] The F function represents feature extraction, which performs preliminary processing on the input features for subsequent squeezing and excitation;

[0065] The x function represents the input feature;

[0066] E x Functions represent stimulus operations.

[0067] In order to meet the requirements of mobile application design, the present invention designs a module that combines lightweight CNN with attention mechanism, called iSE, which absorbs the efficient feature extraction capability of the inverted residual block (IRB) of the CNN architecture to model local features, and the comprehensive information capture capability of the SaE architecture to model global features. The structure diagram of iSaE is shown in the figure. Figure 5 As shown in the figure, since SaE uses fixed weight parameters when calculating attention weights, it cannot adapt to changes in different scenarios and is easily interfered with by background information, seriously affecting the location of the picking point. iSE combines the advantages of both methods to accurately predict the location of the picking point i. The SE module effectively addresses the accuracy degradation caused by lightweight CNNs and the shortcomings of SaE's attention mechanism.

[0068] The GELAN module is specifically: when the existing method extracts features layer by layer from the input data, a lot of important information will be lost, resulting in an information bottleneck. This requires the design of a structure that can obtain sufficient information. To solve the above problem, (yolov9) et al. designed a generalized efficient layer aggregation network (GELAN) by combining CSPNet and ELAN. Figure 6 As shown in the figure, (a) is a schematic diagram of the CSPNet module structure, and (b) is a schematic diagram of the ELAN module structure. The main concept of CSPNet is to split the gradient flow so that the gradient flow information is transmitted through different paths. The main purpose of this design is to reduce the amount of model calculation while achieving richer gradient combinations, so that the model can be deployed on mobile devices without sacrificing performance. The structure diagram of CSPNet is shown in the figure. Figure 6 (a), the implementation details are as follows: y = C(x1, T(B(x2))) (2)

[0069] The input feature X is divided into two parts along the channel, represented as [x1, x2];

[0070] T is the transition function of the stage gradient flow;

[0071] C is a function used to merge two parts;

[0072] B is the BottleNeck module function;

[0073] The feature map of the input layer is divided into two parts, and then they are merged through a cross-stage hierarchy. However, as the number of CSPNet stacked modules increases, the performance of the model will decrease. This is because each stacked module will only increase the path for the gradient flow to propagate. To solve the above problems, ELAN was designed. The architecture is based on the gradient propagation path to design the network architecture. The input features are evenly divided into two branches in the channel dimension. The second branch is extracted through 4 3x3 convolution kernels. Each convolution kernel adds a path, and finally the output of the first branch is fused with the output of all paths in the second branch. The starting point is to increase the shortest gradient path of the model to achieve faster convergence, which helps to reduce feature redundancy and enhance the representation ability of features. The structural diagram of ELAN is shown below. Figure 6 (b) GELAN is a new architecture based on the capabilities of ELAN, which can use any computing blocks. Figure 8 shown.

[0074] The RFAHead detection head module is specifically as follows: for the recognition and positioning tasks of litchi fruits and picking points, the shape and distribution of the target at different positions in the image are variable, and in the convolution operation, the convolution kernel uses the same weight to extract features in different receptive fields, without considering the difference information from different positions. On the other hand, the spatial attention mechanism cannot completely solve the parameter sharing problem of large convolution kernels. (RFAConv: Innovating Spatial Attention and Standard Convolutional Operation) proposes a novel receptive field attention (RFA) to solve the above problems. Its main idea is to combine the spatial attention mechanism with the convolution operation to improve the performance of the convolutional neural network. Based on RFAConv, the present invention designs a detection head suitable for litchi fruit and picking point recognition tasks. When the spatial attention mechanism is added in front of the 3x3 convolution operation, it is specifically expressed by the following formula:

[0075] F1=X 11 ×A 11 ×K1+X 12 ×A 12 ×K2+...+X 19 ×A 19 ×K9

[0076] F2=X 21 ×A 12 ×K1+X 22 ×A 13 ×K2+...+X 29 ×A29 ×K9 ....

[0078] F N =X N1 ×A (N-1)1 ×K1+X N2 ×A (N-1)2 ×K2+…+X N9 ×A N9 ×K9 (4)

[0079] By analyzing formula (4), we can find that the convolution kernel shares some parameters when extracting the features of each receptive field. In other words, there is an overlapping part in each receptive field slider. In this case, each sliding window shares the weight of the spatial attention map, so the spatial attention mechanism does not take into account the spatial features at different positions in the receptive field and cannot effectively solve the problem of large convolution kernel parameter sharing. RFAConv uses the receptive field attention mechanism to dynamically adjust the weight of the convolution kernel, thereby providing customized attention for the feature extraction of each region. In this way, even large-sized convolution kernels can effectively capture and process important spatial features. Its structural diagram is as follows Figure 9 As shown. RFAConv inputs the input features into two paths respectively, one of which uses grouped convolutions of corresponding sizes to dynamically generate feature information based on the receptive field, mapping the original features to new features. The other path uses Avgpool to aggregate the global information of each receptive field feature, and then uses 1x1 convolution to interact with the feature information. In this way, learning the attention map by interacting with the receptive field feature information can improve the performance of the network. Finally, the feature information of the two paths is combined to generate spatial feature information at different positions in the receptive field. The spatial feature information of the receptive field is dynamically generated according to the size of the convolution kernel, thus completely solving the problem of sharing the convolution kernel parameters. In the present invention, RFAConv is integrated into the detection head of YOLOv8-pose to form the RFAHead detection head module, as shown Figure 10 As shown in the figure, the standard convolution is replaced. The RFAHead detection head module allows the model to adapt to changes in the natural environment and improve the robustness of the model.

[0080] S3, feature extraction and classification regression: The litchi dataset is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points;

[0081] S4. Detection result analysis and evaluation: The test set was tested using the partitioned data set to detect the key points of litchi picking in the orchard, and the detection effect of the YOLOv8-iGR model was evaluated. OKS was used as the key point detection performance evaluation indicator, and the pixel Euclidean distance was used as the evaluation indicator of the picking point position prediction error to evaluate the detection effect of the model. Ablation experiments and comparative experiments were used to verify the performance of the YOLOv8-iGR model. The ablation experiment was used to evaluate the impact of different modules on the model performance, and the comparative experiment was used to compare the YOLOv8-iGR model with mainstream target detection algorithms.

[0082] Specifically:

[0083] Ablation experiment: Figure 11 The comparison results of the ablation experiments of different modules are shown in Figure 2. By adding the invented iSaE module to the basic model, P box and P kp The results are improved by 4.36% and 3.20%, respectively, indicating that the addition of the iSaE module improves the network's ability to capture feature information while suppressing the interference of useless information, and the impact on the amount of computation is negligible;

[0084] Replacing the C2f of the base model with GELAN reduces GFLOPs by 15.48% and AP box and mAP kp They increased by 2.58% and 4.76% respectively, which shows that the GELAN module can effectively reduce the complexity of the model; after replacing the detection head with the RFAHead detection head module, AP box and mAP kp An increase of 1.61% and 4.32% respectively.

[0085] The combination of iSaE, GELAN, and RFAHead detection head modules improves the model's detection performance for litchi fruit and picking key points, which reduces GFLOPs by 10.71% and reduces P box 、AP box 、P kp 、mAP kp Increased by 5.51%, 2.79%, 5.61% and 5.87%. Figure 10 Shows the mAP on the validation set kp YOLOv8-iGR and YOLOv8-pose+iSaE show faster convergence speed. This observation indicates that the iSaE module accelerates the convergence of the original model.

[0086] Comparative experiment: In order to demonstrate the comprehensive performance of the proposed YOLOv8-iGR algorithm in litchi fruit detection and key point detection, several mainstream target detection algorithms are compared with key point detection algorithms. Since RT-DETR, YOLOv6n and YOLOv3n algorithms cannot directly identify key points, the detection heads of these three algorithms are replaced with the detection head of YOLOv8n-pose in this experiment. Except for RT-DETR-pose, all models are lightweight detection models. It is worth noting that the YOLOv8-iGR model has the best performance in target detection and key point detection when the number of parameters is low. Compared with YOLOv3n-pose, YOLOv5n-pose, YOLOv6n-pose, and RT-DETR-pose, in terms of target detection, YOLOv8-iGR has an AP of 0. box The highest, respectively, increased by 3.0%, 2.5%, 1.17%, and 1.9%. In terms of key point detection, the mAP of YOLOv8-iGR kp Improvements of 3.24%, 4.36%, 1.81%, and 3.24%, respectively, were achieved. YOLOv8n-pose achieved the highest detection speed of 109.3 f / s, boasting the fastest speed and best detection performance. Therefore, the YOLOv8n-pose model was selected as the baseline model. Although YOLOv8-iGR's detection speed decreased by 8.5% compared to its predecessor, its object detection and keypoint detection accuracy increased by 5.5% and 5.6%, respectively. The decrease in YOLOv8-iGR's detection speed is due to the additional computational operations required by the attention mechanism, as well as the increased memory consumption associated with storing additional attention weights. However, this is acceptable given the robustness and improved detection performance provided by the iSE module. A detection speed of 90.9 f / s still meets the requirements of real-time detection.

[0087] from Figure 11 Experimental results show that the improved YOLOv8-pose model improves the detection accuracy of key lychee picking points from 90.3% to 95.6%, while reducing computational complexity from 8.4G to 7.5G. The average pixel-wise Euclidean distance error between the predicted and actual picking point locations is within 8 pixels. Compared with various existing mainstream detection algorithms, the improved YOLOv8-pose model demonstrates significant advantages in detection under complex and changing environmental conditions. These results demonstrate that the proposed improved YOLOv8-pose model can provide technical support for the vision system of picking robots.

[0088] This paper proposes the iSE architecture, combining it with the GELEN and RFAHead detection head modules to improve the YOLOv8-pose model to form the YOLOv8-iGR model. This model achieves simultaneous recognition of litchi fruit and picking points, significantly improving the accuracy and efficiency of simultaneous detection of litchi fruit and picking points, thereby addressing the slow detection speed of multi-step operations and the difficulty in adapting to complex and changing environments. Compared with traditional methods and existing technologies, the present invention can still maintain high-performance detection capabilities in complex and changing orchard environments, such as those with varying lighting conditions, background factors, and occlusion by leaves and branches. This method achieves a more lightweight model without sacrificing detection performance, meeting the real-time requirements of IoT devices such as robotic arms in crop management. By training and testing with a high-resolution, wide-field-of-view litchi orchard image dataset captured by a camera on a robotic arm, not only is detection accuracy improved, but the optimization of the model structure and detection process also increases detection speed, reduces the complexity of multi-step operations, and enhances the model's robustness in natural environments.

[0089] Furthermore, applying deep learning technology to the identification of lychee picking points expands the application scenarios of key point detection models and lays a solid foundation for future deployment of models on mobile devices for real-time detection. This not only improves the automation level of lychee picking and reduces the labor intensity of fruit farmers, but also has the potential to be adopted in a wider range of agricultural sectors, promoting agricultural modernization and improving agricultural production efficiency and quality.

[0090] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0091] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A litchi image detection method for a picking robot based on an improved YOLOv8, characterized by: The following steps are involved: S1. Dataset collection and construction: In a litchi orchard, a camera mounted on a robotic arm is used to collect litchi image data and construct an orchard litchi dataset containing the annotation information of litchi fruits and picking points. S2. Model training: Based on the orchard litchi dataset in step S1, the YOLOv8-iGR model is trained by improving the YOLOv8-pose model through the iSE architecture, GELAN module, and RFAHead detection head module to achieve simultaneous recognition of litchi fruits and picking points; The iSE architecture combines convolutional neural networks with spatial attention mechanisms to improve the model's ability to capture global and local features. The GELAN module is used to achieve richer gradient combinations by splitting the gradient flow, reducing the model calculation amount while improving performance; The RFAHead detection head module is designed based on RFAConv to solve the problem of large convolution kernel parameter sharing and more effectively extract feature information; Training the YOLOv8-iGR model also includes building a model training environment, specifically: Network training based on Ubuntu system and PyTorch framework; Set training parameters, including batch size, maximum number of iterations, learning rate, momentum parameter, and weight decay coefficient; After non-maximum suppression processing, positive and negative samples are screened according to the confidence threshold; S3, feature extraction and classification regression: The litchi dataset is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points; S4. Detection result analysis and evaluation: The test set is tested using the divided data set to detect the key points of litchi picking in the orchard and evaluate the detection effect of the YOLOv8-iGR model. Specifically: Ablation experiments and comparative experiments are used to verify the performance of the YOLOv8-iGR model. Ablation experiments are used to evaluate the impact of different modules on model performance, and comparative experiments are used to compare the YOLOv8-iGR model with mainstream object detection algorithms. OKS is used as the key point detection performance evaluation indicator, and pixel Euclidean distance is used as the evaluation indicator of the picking point position prediction error to evaluate the detection effect of the model.

2. The litchi image detection method for a picking robot based on an improved YOLOv8 according to claim 1, characterized in that: The orchard litchi dataset collected in step S1 includes litchi images of different varieties, different time periods, and different shooting angles, covering various growth conditions of litchi in the natural environment.

3. The litchi image detection method for a picking robot based on an improved YOLOv8 according to claim 1, wherein: In step S1, the collected images are saved as RGB images and divided into a training set, a validation set, and a test set.

Images