Litchi image detection method of picking robot based on improved YOLOv8

Abstract

The invention relates to the technical field of automatic agriculture, in particular to a picking robot litchi image detection method based on improved YOLOv8, and the method comprises the steps: collecting litchi image data in a litchi orchard, and constructing an orchard litchi data set containing the labeling information of litchi fruits and picking points; a YOLOv8-iGR model formed by improving a YOLOv8-pose model through an iSE framework, a GELAN module and an RFAHead detection head module is trained, and synchronous identification of litchi fruits and picking points is achieved. According to the method, an iSE framework is provided, GELN and RFAHead detection head modules are combined, a YOLOv8-pose model is improved to form a YOLOv8-iGR model, synchronous recognition of the litchi fruits and the picking points is achieved, synchronous detection precision and efficiency of the litchi fruits and the picking points are remarkably improved, and therefore the problems that the multi-step operation detection speed is low, and complex and changeable environments are difficult to deal with are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of automated agriculture, and in particular to a litchi image detection method for a picking robot based on improved YOLOv8. Background Art

[0002] Litchi picking is the most time-consuming and labor-intensive part of the fruit production cycle, with strong seasonality, high labor intensity and high costs. In order to reduce production costs and increase farmers' income, it is particularly urgent to develop an intelligent picking robot suitable for lychees. The key to lychee picking is to accurately identify and locate the picking point of the fruit stalk to avoid damage to the fruit. However, in the natural environment, the accurate detection of the picking point of the lychee fruit stalk faces many challenges, such as interference from non-lychee main fruit branches, occlusion from leaves or branches, and the impact of changes in natural light and background factors on visual recognition. Therefore, using machine vision to accurately identify and locate the picking point is one of the key issues for the lychee picking robot to achieve intelligent operation.

[0003] In the natural environment, the difficulty in accurately detecting the picking point of the litchi stalk lies in: the interference of non-litchi main fruit branches; the obstruction of leaves or branches; and the impact of changes in natural light and background factors on visual recognition.

[0004] To date, many researchers have contributed to the visual system of litchi recognition, but the main methods used in the research are traditional image processing methods, such as color space transformation, threshold segmentation, clustering segmentation of fruits and main fruit branches, but these methods can only handle simple backgrounds, single clusters of litchi fruits, and fully visible main fruit branches. In natural environments, this method is easily affected by factors such as lighting changes and leaf occlusion. At the same time, the distribution of litchi clusters is random, and the captured images usually contain multiple clusters of litchi fruits, which limits the algorithm detection performance and robustness of traditional image processing methods in natural environments.

[0005] With the rapid development of deep convolutional neural networks in object recognition, some researchers have applied deep learning to litchi picking recognition.

[0006] However, there are still some difficulties and challenges in actual use, due to the large number of operation steps, low detection speed, and poor robustness of the method combining deep learning and image processing in complex and changing environments. Summary of the invention

[0007] The purpose of the present invention is to provide a lychee image detection method for a picking robot based on improved YOLOv8 to solve the problems of more operation steps, lower detection speed and poor robustness of the method combining deep learning and image processing in a complex and changeable environment.

[0008] To achieve the above object, the present invention provides the following technical solution: a litchi image detection method for a picking robot based on improved YOLOv8, comprising the following steps:

[0009] S1. Dataset collection and construction: In a litchi orchard, a camera mounted on a robotic arm is used to collect litchi image data and construct an orchard litchi dataset containing litchi fruit and picking point annotation information;

[0010] S2. Model training: Based on the orchard litchi dataset in step S1, the YOLOv8-iGR model formed by improving the YOLOv8-pose model through the iSE architecture, GELAN module and RFAHead detection head module is trained to achieve synchronous recognition of litchi fruits and picking points;

[0011] The iSE architecture combines convolutional neural networks with spatial attention mechanisms to improve the model's ability to capture global and local features;

[0012] The GELAN module is used to achieve richer gradient combinations by splitting the gradient flow, thereby reducing the amount of model calculation and improving performance;

[0013] The RFAHead detection head module is designed based on RFAConv to solve the problem of large convolution kernel parameter sharing and extract feature information more effectively;

[0014] S3, feature extraction and classification regression: The litchi data set is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points;

[0015] S4. Analysis and evaluation of detection results: The test set is tested using the divided data set to detect the key points of picking lychees in the orchard and evaluate the detection effect of the YOLOv8-iGR model.

[0016] Preferably, the orchard litchi dataset collected in step S1 includes litchi images of different varieties, different time periods and different shooting angles, covering various growth conditions of litchi in the natural environment.

[0017] Preferably, in step S1, the collected images are saved as RGB images and divided into a training set, a validation set and a test set.

[0018] Preferably, the step S2 of training the YOLOv8-iGR model further includes constructing a model training environment, specifically:

[0019] Network training based on Ubuntu system and PyTorch framework;

[0020] Set training parameters, including batch size, maximum number of iterations, learning rate, momentum parameter, and weight decay coefficient;

[0021] After non-maximum suppression processing, positive and negative samples are screened according to the confidence threshold.

[0022] Preferably, in step S4, when evaluating the detection effect of the YOLOv8-iGR model, an ablation experiment and a comparative experiment are used to verify the performance of the YOLOv8-iGR model. The ablation experiment is used to evaluate the impact of different modules on the model performance, and the comparative experiment is used to compare the YOLOv8-iGR model with the mainstream target detection algorithm.

[0023] Preferably, in step S4, in evaluating the detection effect of the YOLOv8-iGR model, OKS is used as the detection performance evaluation index of the key point, and the pixel Euclidean distance is used as the evaluation index of the picking point position prediction error to evaluate the detection effect of the model.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention proposes an iSE architecture, combines the GELEN and RFAHead detection head modules, improves the YOLOv8-pose model to form the YOLOv8-iGR model, realizes the synchronous recognition of litchi fruits and picking points, and significantly improves the synchronous detection accuracy and efficiency of litchi fruits and picking points, thereby solving the problems of slow multi-step operation detection speed and difficulty in coping with complex and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of an image data set of a litchi image detection method for a picking robot based on improved YOLOv8 in the present invention;

[0027] Figure 2 A schematic diagram of a specific application technology route of a litchi image detection method for a picking robot based on improved YOLOv8 of the present invention;

[0028] Figure 3 A schematic diagram of the overall framework of the YOLOv8-iGR model and its component modules of a litchi image detection method for a picking robot based on improved YOLOv8 of the present invention;

[0029] Figure 4 It is a schematic diagram of the working principle of SaE of the present invention;

[0030] Figure 5 is a schematic diagram of the structure of the iSE of the present invention;

[0031] Figure 6It is a schematic diagram of the structure of CSPNet and ELAN modules of the present invention;

[0032] Figure 7 This is a schematic diagram of the structure of the GELAN module of the present invention;

[0033] Figure 8 This is a schematic diagram of the RFAConv structure of the present invention;

[0034] Fig. 9 This is a schematic diagram of the structure of RFAHead of the present invention;

[0035] Fig.10 is the mAP in the ablation experiment of this invention kp Schematic diagram of curve comparison;

[0036] Fig.11 Schematic diagram of comparison results of ablation experiments on different modules of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] To date, many researchers have contributed to the visual system for litchi recognition. Li et al. proposed an improved YOLOv7-Litchi detection algorithm, which has good detection capabilities for dense distribution and fruit occlusion in complex scenes. Yu et al. proposed a litchi recognition method based on a red, green, and blue depth camera, which segmented the litchi depth image obtained by the depth camera to remove redundant information other than litchi, and then used color and texture features to train a random forest binary classification model to identify litchi fruits. Jiao et al. proposed a litchi detection method based on YOLOx, which uses various compression operations to achieve a compact model deployed on edge devices. However, the above method is only applicable to the field of litchi yield estimation or maturity estimation. For the field of litchi picking, in addition to fruit recognition, it is also necessary to locate the picking point.

[0039] Initially, the main method for determining the picking point of litchi was based on the combination of machine learning algorithms and machine vision. Zhuang et al. proposed a retinex algorithm to compensate for the illumination distribution of low-light images of litchi, then combined red / green saturation mapping and Otsu thresholding to segment the main fruit branch area of ​​litchi, and finally analyzed the angular structure distribution generated by Harris corner detection to locate the picking point, with an accuracy of 83%. Xiong et al. proposed a method for identifying litchi at night and a method for calculating the picking point. First, the improved fuzzy clustering algorithm (FCM) was used to remove the background information of the litchi image, then the Otsu algorithm was used to segment the litchi fruit, and finally the Harris corner detection method was used for picking point detection. The method had an accuracy of 93.75% for identifying litchi and a maximum accuracy of 97.5% for calculating the picking point. Shu et al. developed a stereo vision system suitable for litchi picking robots, using the corresponding image color algorithm to calculate the hue color layer of litchi, and then analyzed the appearance characteristics and position data of litchi to obtain the three-dimensional coordinates of litchi fruit.

[0040] The main methods used in the above studies are traditional image processing methods, such as color space transformation, threshold segmentation, clustering segmentation of fruits and main fruit branches, but these methods can only handle simple backgrounds, single clusters of litchi fruits, and fully visible main fruit branches. In natural environments, this method is easily affected by factors such as lighting changes and leaf occlusion. At the same time, the distribution of litchi clusters is random, and the captured images usually contain multiple clusters of litchi fruits, which limits the algorithm detection performance and robustness of traditional image processing methods in natural environments.

[0041] With the rapid development of deep convolutional neural networks in object recognition, some researchers have applied deep learning to litchi picking recognition. Peng et al. constructed a ResDense-focal-DeepLabV3+ network for accurate segmentation of litchi branches in an orchard environment, and the mIoU was improved compared with other models in simple, medium, and complex images. Xie et al. proposed an improved DeepLabv3+ litchi stem segmentation method to solve the problem of low accuracy of litchi stem segmentation in natural environments. The model has an average intersection of 90.28% and 94.95% in union and average pixel accuracy, and a frame rate of 19.83 per second (FPS). The above method only segments the branches of litchi, while the robot needs to know the precise location of the picking point to pick litchi. Zhong et al. proposed a YOLACT-based method for detecting the main fruit branches of litchi. First, the litchi fruit and the main fruit branches were annotated as a litchi cluster. Then, the main fruit branches and litchi were segmented using the YOLACT model. Finally, the center point of the main fruit branch was used as the picking point. The angle of the main fruit branch was determined by skeleton extraction and least squares fitting. The picking point accuracy calculated by this method was 89.7% and F1 was 83.8%. Qi et al. used YOLOv5 to detect the stems in the litchi image, extracted the region of interest (ROI) of the main stem, and segmented it using PSPNet. The segmented image was processed to obtain the pixel coordinates of the picking point on the main stem. The accuracy and recall of this method were 92.50% and 76.29%, respectively. The above methods further ensure the accuracy of picking point positioning, but these methods will have some difficulties and challenges. Due to the multi-step operation, the detection speed is low, and in complex and changing environments, the method combining deep learning and image processing has poor robustness.

[0042] The present invention provides a technical solution: a litchi image detection method for a picking robot based on improved YOLOv8, comprising the following steps:

[0043] S1. Dataset collection and construction: In a litchi orchard, a camera installed on a robotic arm is used to collect litchi image data. The collected orchard litchi dataset includes litchi images of different varieties, different time periods, and different shooting angles, covering various growth conditions of litchi in the natural environment, and constructs an orchard litchi dataset containing litchi fruit and picking point annotation information. The dataset is used for model training, and the collected images are saved as RGB images and divided into training set, validation set, and test set;

[0044] S2. Model training: First, build a model training environment, specifically:

[0045] Network training based on Ubuntu system and PyTorch framework;

[0046] Set training parameters, including batch size, maximum number of iterations, learning rate, momentum parameter, and weight decay coefficient;

[0047] After non-maximum suppression processing, positive and negative samples are screened according to the confidence threshold;

[0048] Then, based on the orchard litchi dataset in step S1, the YOLOv8-iGR model formed by improving the YOLOv8 model through the iSE (IRB-SaE) architecture, GELAN module and RFAHead detection head module is trained to achieve simultaneous recognition of litchi fruits and picking points;

[0049] The iSE (IRB-SaE) architecture combines convolutional neural networks with spatial attention mechanisms to improve the model's ability to capture global and local features;

[0050] The GELAN module is used to achieve richer gradient combinations by splitting the gradient flow, thereby reducing the amount of model calculation and improving performance;

[0051] The RFAHead detection head module is designed based on RFAConv to solve the problem of large convolution kernel parameter sharing and extract feature information more effectively;

[0052] S3, feature extraction and classification regression: The litchi data set is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points;

[0053] S4. Analysis and evaluation of detection results: The test set is tested using the divided data set to realize the detection of key points for picking lychees in the orchard, and the detection effect of the YOLOv8-iGR model is evaluated. OKS is used as the key point detection performance evaluation indicator, and the pixel Euclidean distance is used as the evaluation indicator for the prediction error of the picking point position to evaluate the detection effect of the model. Ablation experiments and comparative experiments are used to verify the performance of the YOLOv8-iGR model. The ablation experiment is used to evaluate the impact of different modules on the model performance, and the comparative experiment is used to compare the YOLOv8-iGR model with mainstream target detection algorithms.

[0054] See also Figure 1-11 The present invention also provides a specific embodiment of a litchi image detection method using a picking robot based on an improved YOLOv8-pose, comprising the following steps:

[0055] S1. Dataset collection and construction: In the litchi orchard, the Realsense D435i camera and mobile phone were used to capture images of red-ripe litchi for training and testing. The camera was 30-100 cm away from the target. The varieties of litchi included "Fei Zi Xiao", "Gui Wei", and "Nuomi Ci". The acquired images were saved as RGB images of 1000x750 pixels. In order to ensure that the dataset reflects the characteristics of the orchard in the natural environment, a total of 1281 images of different varieties of litchi trees were taken in the orchard at different time periods (9:00am, 2:00pm, 6:00pm), such as Figure 1 . The shooting angles are front view, upward view and downward view. The calyx of litchi taken from the upward view is blocked. Due to different lighting conditions in the natural environment, some images have sufficient lighting, and the outlines of litchi and main fruit branches are clear, while some images are dark as a whole, which can easily lead to misjudgment of the picking points on the main fruit branches of litchi. In a complex natural environment, there will be occlusions from leaves and branches, which will increase the difficulty of the model in identifying litchi and main fruit branches. The collected orchard litchi dataset includes litchi images of different varieties, different time periods and different shooting angles, covering various growth conditions of litchi in the natural environment, and constructing an orchard litchi dataset containing litchi fruit and picking point annotation information. The 1000x750 pixel RGB image dataset is divided into a ratio of 8:1:1. 1025 images are randomly selected as the training set, 128 images as the validation set, and 128 images as the test set;

[0056] S2. Model training: First, build a model training environment, specifically:

[0057] Network training is based on the Ubuntu system and the PyTorch framework. The main configuration of the computer hardware is i9-9900kCPU and GeForce RTX 3090GPU. CUDA 11.0, CUDNN 8.0.1, and python 3.8 are installed. The training environment for different algorithms is the same, and the training parameters are set as follows: the batch size of the model is set to 32, the maximum number of iterations is set to 300, and if the model performance does not improve within 50 consecutive epochs, the training is stopped. The image input size is set to 640x 640 pixels, the initial learning rate is 0.01, the cyclic learning rate is 0.2, the momentum parameter is 0.937, and the weight decay coefficient is 0.0005. In addition, in the classification of the predicted boxes obtained after non-maximum suppression processing, the predicted boxes with a confidence value greater than the 0.5 threshold are defined as positive samples, otherwise they are defined as negative samples;

[0058] The efficient spatial feature extraction capability of convolution kernels and the effectiveness of attention mechanisms have been well established in the field of computer vision. In this paper, we are committed to integrating the capabilities of convolution and attention mechanisms to improve the localization capability of picking points. YOLOv8n is a single-stage target detection model, such as Figure 3 As shown in the figure, in order to improve the model's performance in lychee recognition and detection of key points for picking the main fruit branches, the iSE structure is proposed. The fused module improves the model performance more significantly than using convolution or attention mechanism alone. In addition, the model needs to be deployed on mobile devices. The introduction of GELAN reduces computational overhead without sacrificing performance. The RFAHead detection head module is used to simultaneously detect the key points for picking lychee fruits and main fruit branches. It is based on RFAConv and effectively solves the problem of parameter sharing during feature extraction, improving detection accuracy.

[0059] Then, based on the orchard litchi dataset in step S1, the YOLOv8-iGR model formed by improving the YOLOv8 model through the iSE (IRB-SaE) architecture, GELAN module and RFAHead detection head module is trained to achieve simultaneous recognition of litchi fruits and picking points;

[0060] in:

[0061] The iSE architecture is as follows: In recent years, the effectiveness of attention mechanisms in improving the expressive power of models and accurate target positioning has been well verified in the field of computer vision. The Squeeze aggregated excitation (SaE) module is a novel aggregated multi-layer perceptron proposed by SENetv2 (SENetv2), which aims to surpass the performance of existing architectures. The SaE module combines the operation of Squeeze-and-Excitation, and uses the squeeze and excitation modules to fuse with multi-branch DenseLayer. Compared with SE (SENet), this fusion method enhances the network's ability to capture channel information and global knowledge, resulting in better feature representation.

[0062] The working principle of SaE is as follows Figure 4 As shown in the figure, after the input features are resized and the number of channels is adjusted by the convolution layer, the global average pooling layer is used to count the channel information, which is then collected in the FC layer for squeezing and compressing the feature size. Then the outputs of all FC branches are concatenated, and the excitation operation is performed to restore the size to its initial shape. Finally, the squeezing and excitation results are connected to the input in the residual module. The SaE module formula can be expressed as: SaE = x + F (x·E X (∑S q (x))), where:

[0063] Sq The function represents the squeeze operation, including the FC layer;

[0064] The F function represents feature extraction, which performs preliminary processing on the input features for subsequent squeezing and excitation;

[0065] The x function represents the input features;

[0066] E x Functions represent stimulus operations.

[0067] In order to meet the requirements of mobile application design, the present invention designs a module combining lightweight CNN and attention mechanism, called iSE, which absorbs the efficient feature extraction capability of the inverted residual block (IRB) of the CNN architecture to model local features, and the comprehensive information capture capability of the SaE architecture to model global features. The structure diagram of iSaE is shown in the figure Figure 5 As shown in the figure. Since SaE uses fixed weight parameters when calculating attention weights and cannot adapt to changes in different scenes, it is easily disturbed by background information, which seriously affects the positioning of the picking point. iSE combines the advantages of both and accurately predicts the location of the picking point i. The SE module effectively solves the problem of accuracy degradation caused by lightweight CNN and the defects of SaE attention mechanism.

[0068] The GELAN module is as follows: When the existing method extracts features layer by layer from the input data, a lot of important information will be lost, resulting in an information bottleneck. This requires the design of a structure that can obtain sufficient information. In order to solve the above problems, (yolov9) et al. designed a generalized efficient layer aggregation network (GELAN) by combining CSPNet and ELAN. Figure 6 As shown in the figure, (a) is a schematic diagram of the CSPNet module structure, and (b) is a schematic diagram of the ELAN module structure. The main concept of CSPNet is to split the gradient flow so that the gradient flow information can be propagated through different paths. The main purpose of this design is to reduce the amount of model calculation while achieving richer gradient combinations, so that the model can be deployed on mobile devices without sacrificing performance. The structure diagram of CSPNet is shown in Figure 6 (a), the implementation details are as follows: y = C (x1, T (B (x2))) (2)

[0069] The input feature X is divided into two parts along the channel, represented as [x1, x2];

[0070] T is the transition function of the stage gradient flow;

[0071] C is a function used to merge two parts;

[0072] B is the BottleNeck module function;

[0073] The feature map of the input layer is divided into two parts, and then they are merged through a cross-stage hierarchy. However, as the number of CSPNet stacked modules increases, the performance of the model will decrease. This is because each stacked module will only increase the path for the gradient flow to propagate. To solve the above problems, ELAN was designed. The architecture is based on the gradient propagation path to design the network architecture. The input features are evenly divided into two branches in the channel dimension. The second branch is extracted through 4 3x3 convolution kernels. Each convolution kernel adds a path, and finally the first branch is fused with the output of all paths in the second branch. The starting point is to increase the shortest gradient path of the model to achieve faster convergence, which helps to reduce feature redundancy and enhance the representation ability of features. The structure diagram of ELAN is shown in the figure. Figure 6 (b) shows that GELAN is a new architecture based on the capabilities of ELAN, which can use any computing blocks. Figure 8 shown.

[0074] The RFAHead detection head module is specifically as follows: For the identification and positioning tasks of litchi fruits and picking points, the shape and distribution of the target at different positions in the image are variable, and in the convolution operation, the convolution kernel uses the same weights to extract features in different receptive fields, without considering the difference information from different positions. On the other hand, the spatial attention mechanism cannot completely solve the parameter sharing problem of large convolution kernels. (RFAConv: Innovating Spatial Attention and Standard Convolutional Operation) A novel receptive field attention (RFA) is proposed to solve the above problems. Its main idea is to combine the spatial attention mechanism with the convolution operation to improve the performance of the convolutional neural network. Based on RFAConv, the present invention designs a detection head suitable for litchi fruit and picking point recognition tasks. When the spatial attention mechanism is added in front of the 3x3 convolution operation, it is specifically expressed by the following formula:

[0075] F1=X 11 ×A 11 ×K1+X 12 ×A 12 ×K2+...+X 19 ×A 19 ×K9

[0076] F2=X 21 ×A 12 ×K1+X 22 ×A 13 ×K2+...+X 29 ×A29 ×K9 ....

[0078] F N =X N1 ×A (N-1)1 ×K1+X N2 ×A (N-1)2 ×K2+…+X N9 ×A N9 ×K9 (4)

[0079] By analyzing formula (4), we can find that the convolution kernel shares some parameters when extracting the features of each receptive field. In other words, there is an overlapping part in each receptive field slider. In this case, each sliding window shares the weight of the spatial attention map, so the spatial attention mechanism does not take into account the spatial features at different positions in the receptive field and cannot effectively solve the problem of large convolution kernel parameter sharing. RFAConv uses the receptive field attention mechanism to dynamically adjust the weight of the convolution kernel, thereby providing customized attention for feature extraction in each region. In this way, even large-sized convolution kernels can effectively capture and process important spatial features. Its structural diagram is shown in the figure. Fig. 9 As shown. RFAConv inputs the input features into two paths respectively, one of which uses grouped convolutions of corresponding sizes to dynamically generate feature information based on the receptive field, mapping the original features to new features. The other path uses Avgpool to aggregate the global information of each receptive field feature, and then uses 1x1 convolution to interact with the feature information. In this way, learning the attention map through interactive receptive field feature information can improve the performance of the network. Finally, the feature information of the two paths is combined to generate spatial feature information at different positions in the receptive field. The spatial feature information of the receptive field is dynamically generated according to the size of the convolution kernel, thus completely solving the problem of sharing convolution kernel parameters. In the present invention, RFAConv is integrated into the detection head of YOLOv8-pose to form the RFAHead detection head module, as shown Fig.10 As shown in the figure, the standard convolution is replaced. The RFAHead detection head module allows the model to adapt to changes in the natural environment and improve the robustness of the model.

[0080] S3, feature extraction and classification regression: The litchi data set is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points;

[0081] S4. Analysis and evaluation of detection results: The test set is tested using the divided data set to detect the key points of picking lychees in the orchard, and the detection effect of the YOLOv8-iGR model is evaluated. OKS is used as the key point detection performance evaluation indicator, and the pixel Euclidean distance is used as the evaluation indicator of the picking point position prediction error to evaluate the detection effect of the model. The performance of the YOLOv8-iGR model is verified by ablation experiments and comparative experiments. The ablation experiment is used to evaluate the impact of different modules on the model performance, and the comparative experiment is used to compare the YOLOv8-iGR model with the mainstream target detection algorithm.

[0082] Specifically:

[0083] Ablation experiment: Fig.11 The comparison results of ablation experiments with different modules are shown in Figure 2. By adding the invented iSaE module to the basic model, P box and P kp The improvement is 4.36% and 3.20%, indicating that the addition of the iSaE module improves the network's ability to capture feature information while suppressing the interference of useless information, and the impact on the amount of calculation is negligible;

[0084] Replacing the C2f of the base model with GELAN reduces GFLOPs by 15.48% and AP box and mAP kp They increased by 2.58% and 4.76% respectively, which shows that the GELAN module can effectively reduce the complexity of the model; after replacing the detection head with the RFAHead detection head module, AP box and mAP kp An increase of 1.61% and 4.32% respectively.

[0085] The combination of iSaE, GELAN, and RFAHead detection head modules improves the model’s detection performance for litchi fruit and picking key points, which reduces GFLOPs by 10.71% and reduces P box 、AP box , P kp 、mAP kp Increased by 5.51%, 2.79%, 5.61% and 5.87%. Fig.10 The mAP on the validation set is shown kp Curve. YOLOv8-iGR and YOLOv8-pose+iSaE show faster convergence speed, and this observation indicates that the iSaE module accelerates the convergence of the original model.

[0086] Comparative experiment: In order to demonstrate the comprehensive performance of the proposed YOLOv8-iGR algorithm in litchi fruit detection and key point detection, several mainstream object detection algorithms are compared with the key point detection algorithms. Since the RT-DETR, YOLOv6n and YOLOv3n algorithms cannot directly identify key points, the detection heads of these three algorithms are replaced with the detection heads of YOLOv8n-pose in this experiment. Except for RT-DETR-pose, all models are lightweight detection models. It is worth noting that the YOLOv8-iGR model has the best performance in object detection and key point detection when the number of parameters is low. Compared with YOLOv3n-pose, YOLOv5n-pose, YOLOv6n-pose, and RT-DETR-pose, in terms of object detection, YOLOv8-iGR has an AP of . box The highest, respectively increased by 3.0%, 2.5%, 1.17%, and 1.9%. In terms of key point detection, YOLOv8-iGR's mAP kp The improvements were 3.24%, 4.36%, 1.81%, and 3.24%, respectively. YOLOv8n-pose achieved the highest value of 109.3f / s in detection speed, with the fastest speed and good detection performance, so the YOLOv8n-pose model was selected as the baseline model. Although the detection speed of YOLOv8-iGR decreased by 8.5% compared with the original model, the accuracy of target detection and key point detection increased by 5.5% and 5.6%, respectively. The decrease in the detection speed of YOLOv8-iGR was due to the addition of the attention mechanism, which required additional computing operations, and the storage of additional attention weights, which increased memory consumption. However, this is acceptable considering the robustness and detection performance improvement provided by the iSE module. The detection speed of 90.9f / s still meets the needs of real-time detection.

[0087] from Fig.11 The experimental results in show that the improved YOLOv8-pose improves the detection accuracy of litchi picking key points from 90.3% to 95.6%, and reduces the computational complexity from 8.4G to 7.5G. The average pixel Euclidean distance error between the predicted picking point position and the actual position is within 8 pixels. Compared with various existing mainstream detection algorithms, the improved YOLOv8-pose has great advantages in detection performance under complex and changing environmental factors. These results show that the proposed improved YOLOv8-pose model can provide technical support for the visual system of the picking robot.

[0088] The present invention proposes an iSE architecture, combines GELEN and RFAHead detection head modules, improves the YOLOv8-pose model to form a YOLOv8-iGR model, realizes the synchronous recognition of litchi fruits and picking points, and significantly improves the synchronous detection accuracy and efficiency of litchi fruits and picking points, thereby solving the problem of slow detection speed of multi-step operation and difficulty in coping with complex and changeable environments. Compared with traditional methods and prior art, the present invention can still maintain high-performance detection capabilities in complex and changeable orchard environments, such as scenes with illumination differences, background factor differences, leaves and branches blocking, etc. The present invention realizes that the model is more lightweight without sacrificing detection performance, and meets the real-time requirements of IoT devices such as robotic arms in crop management. By using the high-resolution, wide-viewing litchi orchard image data set captured by the camera on the robotic arm for training and testing, not only the detection accuracy is improved, but also the detection speed is improved by optimizing the model structure and detection process, the cumbersomeness caused by multi-step operation is reduced, and the robustness of the model in the natural environment is enhanced.

[0089] In addition, the application of deep learning technology to the identification of litchi picking points has expanded the application scenarios of key point detection models, laying a solid foundation for deploying models on mobile terminals for real-time detection in the future. It not only improves the automation level of litchi picking and reduces the labor intensity of fruit farmers, but is also expected to be promoted in a wider range of agricultural fields, promote the process of agricultural modernization, and improve agricultural production efficiency and quality.

[0090] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0091] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A litchi image detection method for a picking robot based on improved YOLOv8, characterized in that: The following steps are involved: S1. Dataset collection and construction: In a litchi orchard, a camera mounted on a robotic arm is used to collect litchi image data and construct an orchard litchi dataset containing litchi fruit and picking point annotation information; S2. Model training: Based on the orchard litchi dataset in step S1, the YOLOv8-iGR model formed by improving the YOLOv8-pose model through the iSE architecture, GELAN module and RFAHead detection head module is trained to achieve synchronous recognition of litchi fruits and picking points; The iSE architecture combines convolutional neural networks with spatial attention mechanisms to improve the model's ability to capture global and local features; The GELAN module is used to achieve richer gradient combinations by splitting the gradient flow, thereby reducing the amount of model calculation and improving performance; The RFAHead detection head module is designed based on RFAConv to solve the problem of large convolution kernel parameter sharing and extract feature information more effectively; S3, feature extraction and classification regression: The litchi data set is input into the YOLOv8-iGR feature extraction network in step S2 for feature extraction to obtain feature maps of different scales, and then classification and regression operations are performed to obtain the detection results of litchi fruits and picking points; S4. Analysis and evaluation of detection results: The test set is tested using the divided data set to detect the key points of picking lychees in the orchard and evaluate the detection effect of the YOLOv8-iGR model.

2. The method for detecting litchi images by a picking robot based on improved YOLOv8 according to claim 1, characterized in that: The orchard litchi dataset collected in step S1 includes litchi images of different varieties, different time periods and different shooting angles, covering various growth conditions of litchi in the natural environment.

3. The litchi image detection method of a picking robot based on improved YOLOv8 according to claim 1, characterized in that: In step S1, the collected images are saved as RGB images and divided into a training set, a validation set and a test set.

4. The litchi image detection method of a picking robot based on improved YOLOv8 according to claim 1, characterized in that: The step S2 of training the YOLOv8-iGR model also includes building a model training environment, specifically: Network training based on Ubuntu system and PyTorch framework; Set training parameters, including batch size, maximum number of iterations, learning rate, momentum parameter, and weight decay coefficient; After non-maximum suppression processing, positive and negative samples are screened according to the confidence threshold.

5. The litchi image detection method of a picking robot based on improved YOLOv8 according to claim 1, characterized in that: In step S4, the detection effect of the YOLOv8-iGR model is evaluated by using an ablation experiment and a comparative experiment to verify the performance of the YOLOv8-iGR model. The ablation experiment is used to evaluate the impact of different modules on the model performance, and the comparative experiment is used to compare the YOLOv8-iGR model with mainstream target detection algorithms.

6. The litchi image detection method of a picking robot based on improved YOLOv8 according to claim 1, characterized in that: In step S4, the detection effect of the YOLOv8-iGR model is evaluated by using OKS as the key point detection performance evaluation index, and using the pixel Euclidean distance as the evaluation index of the picking point position prediction error to evaluate the detection effect of the model.

Images