Litchi fruit dynamic identification and counting method based on improved YOLOv8s

Abstract

The invention discloses a litchi fruit dynamic identification and counting method based on improved YOLOv8s. The method comprises the following steps: acquiring a to-be-identified litchi image; inputting the to-be-identified litchi image into an improved YOLOv8s model to obtain a litchi fruit counting result; wherein the improved YOLOv8s model completes training through a training set, and the training set is marked litchi image data. According to the method, the model is subjected to lightweight improvement while the recognition precision and speed are guaranteed, the parameter quantity and the calculation quantity are reduced, equipment resources are saved, the requirement for real-time detection is met, and a foundation is laid for predicting the yield of litchi fruits.

Description

Technical Field

[0001] The invention belongs to the field of image recognition, and in particular relates to a litchi fruit dynamic recognition and counting method based on improved YOLOv8s. Background Art

[0002] Litchi is an important subtropical cash crop and one of the specialty fruits in southern my country. Litchi exposed to high temperatures for a long time will lead to the production of litchi acid, which causes deterioration and affects the overall flavor of the fruit. Therefore, it is crucial to predict the lychee yield of an orchard in advance. Such predictions help fruit farmers to effectively plan picking and storage activities and enable them to formulate supply strategies early. Traditionally, the estimation of lychee yield is carried out by counting by agroforestry scientists, which is costly and has low accuracy of results because it requires careful observation by experts and a long estimation time. Manual estimation of yield is a challenging task. Therefore, it is necessary to establish an intelligent yield estimation system that can overcome the above problems.

[0003] In recent years, due to the rapid development of computer technology, significant progress has been made in the field of computer vision. Computer vision technology has been widely used in agricultural research, including defect detection, pest and disease detection, plant phenotyping, weed control management, etc., aiming to solve challenges such as high labor intensity and low productivity, and ultimately improve the quality of agricultural production. At present, the detection of litchi mainly includes traditional machine vision and deep learning methods based on convolutional neural networks (CNN). Traditional litchi target recognition methods rely on multiple features such as color, geometry, texture, etc. of litchi fruit to perform target classification, detection and segmentation tasks. Xiong Juntao et al. used the Cr component of the YCbCr color model for threshold segmentation to remove the image background, applied morphological operations and connected region annotation to eliminate noise, and clustered and segmented litchi images through one-dimensional random signal histogram analysis and fuzzy C-means clustering. The experimental results show that the comprehensive recognition rate of this method is 95.5%. Wang et al. used wavelet transform to normalize the color image of litchi for lighting, and used K-means clustering algorithm to segment the color of litchi and separate litchi from leaves, branches and background. The experimental results showed that the average recognition rates of this method were 98.8% and 97.5% respectively when the image was not occluded and partially occluded. Guo et al. used Hough circle transform and equivalent foreground area circle method to determine the position of litchi, used support vector machine to filter background impurities, and used three-point circle theorem to extract single litchi fruit from overlapping litchi foreground area. The detection results showed that the recall rate of this method was 86.66%, the accuracy rate was 87%, and the F1 value was 0.87. Yu et al. used Kinect V2 red, green and blue depth camera to capture various types of litchi image data, and used color and texture features to train a random forest binary classification model. The experimental results showed that the recognition accuracy of this method for red litchi was 94.50%, and the recognition accuracy for green litchi was 89.92%. The above research methods mainly use traditional image processing methods. Traditional recognition algorithms are constantly improving, and algorithms based on manually designed features are approaching their performance limits. However, they still have some limitations, including the introduction of redundant regions and difficulty in processing semantic information in complex backgrounds. Traditional target recognition algorithms have poor generalization and robustness, and are more suitable for situations with obvious features and simple backgrounds, but not for litchi recognition in complex environments.

[0004] With the rapid development of computer vision and digital image processing, deep learning technology has been widely used in agriculture, which provides a direction for the upgrading of the litchi industry. Litchi fruit target detection can provide necessary technical support and intelligent solutions for the harvesting process. Wang et al. used the Mask R-CNN model to segment the more obvious bifurcated stems in the field of view. After processing the segmentation mask, they proposed a "point + line" dual reference model to guide the robot arm to pick. The experimental results show that the success rate of the system in locating litchi branches is 88.46%. Ye Jin et al. used Faster R-CNN as the basic network, introduced the aggregation loss term in the loss function to force the proposed box to locate the object close and compactly, and added the bounding box exclusion loss designed specifically for dense crop scenes to improve the robustness of detecting a large number of flowers. The experimental results show that the average precision of this method for dense litchi flower detection is 87.94%, the F1 value is 0.8707, the missing rate is 13.29%, and the recall rate is 86.21%. Although a two-stage algorithm similar to RCNN has achieved high detection accuracy in fruit recognition tasks in orchard environments, its network structure is complex, consumes a lot of computing resources, and has a slow inference speed, making it challenging to apply in real-time scenarios.

[0005] The accuracy of the visual recognition system is an important prerequisite for achieving yield estimation and automated harvesting. Lychee recognition not only faces the difficulties of changing colors, close-up views, and small targets, but also presents unstructured characteristics such as diverse shapes, random distribution, overlap, and occlusion in the natural environment. Solving the problem of small target detection is the key to achieving accurate yield estimation of lychee fruits in large scenes. Taking these influencing factors into consideration, how to improve the recognition effect of lychees in orchards and meet the lightweight requirements of network models to facilitate deployment is the primary focus of the present invention. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a dynamic recognition and counting method for litchi fruits based on improved YOLOv8s, which improves the model lightweight while ensuring recognition accuracy and speed, reduces the number of parameters and calculations, saves equipment resources and meets the needs of real-time detection.

[0007] To achieve the above object, the present invention provides a litchi fruit dynamic recognition and counting method based on improved YOLOv8s, comprising:

[0008] Obtaining a litchi image to be identified;

[0009] The litchi image to be identified is input into the improved YOLOv8s model to obtain the litchi fruit counting result; wherein, the improved YOLOv8s model is trained through a training set, and the training set is labeled litchi image data.

[0010] Optionally, before labeling the training set data, the method further includes: collecting original litchi image data and preprocessing the original litchi image data;

[0011] Preprocessing the original litchi image data includes:

[0012] The original litchi image data is processed by a combination of mirroring, rotating, adding noise and adjusting brightness to obtain the litchi image data.

[0013] Optionally, the improved YOLOv8s model is:

[0014] Introduce partial convolution to replace some conventional convolutions in the C2f module for lightweight improvement, which is used to reduce the number of model parameters and calculations;

[0015] Incorporate an efficient multi-scale attention mechanism to improve the feature extraction capability of the backbone network;

[0016] The weighted bidirectional feature pyramid network of the P2 feature layer is introduced to replace the path aggregation network to enhance the model's cross-scale feature fusion of different scales;

[0017] The normalized Gauss-Wasserstein distance metric is introduced into the regression loss function to improve the model's learning ability for small litchi objects;

[0018] Based on the above improvements, the improved YOLOv8s model is obtained.

[0019] Optionally, introducing partial convolution to replace part of the conventional convolution in the C2f module for lightweight improvement includes: using the partial convolution to replace part of the conventional convolution, and using conventional Conv on a part of the input channel to extract spatial features, thereby completing the lightweight improvement.

[0020] Optionally, using the efficient multi-scale attention mechanism to improve the feature extraction capability of the backbone network includes:

[0021] Reshape some channels into batch channel dimensions, and group the channel dimensions into multiple sub-features;

[0022] The sub-features are respectively input into parallel 3x3 and 1x1 convolutions to obtain intermediate features;

[0023] Based on the intermediate features and the context information, obtaining output features for output parallelism;

[0024] The output features are aggregated to obtain image features.

[0025] Optionally, obtaining the weighted bidirectional feature pyramid network includes:

[0026] Based on bidirectional span connection and fast normalization fusion, the weighted bidirectional feature pyramid network is obtained.

[0027] Optionally, the method for performing cross-scale fusion using the weighted bidirectional feature pyramid network is:

[0028]

[0029] in, is the intermediate feature of the i-th level of the top-down path, is the i-th layer feature extracted from the backbone, is the output feature of the i-th level of the bottom-up path, Resize is an upsampling or downsampling operation, w1 is the learning weight of the input feature I1, w2 is the learning weight of the input feature I2, is the i+1th layer feature extracted from the backbone, w'1 is the first-order learning weight of the input feature I1, w'2 is the first-order learning weight of the input feature I2, w'3 is the first-order learning weight of the input feature I3, ε is the initial learning rate, is the output feature of the i-1th level of the bottom-up path.

[0030] Optionally, the normalized Gauss-Wasserstein distance metric is:

[0031]

[0032] Among them, NWD (N a ,N b ) is the Gauss-Wasserstein distance normalized in exponential form, is a distance measurement, N a and N b A=(cx a ,cy a ,w a ,h a ) and B=(cx b ,cy b ,w b ,h b ) is a Gaussian distribution modeled by , where cx, cy, w, h are the coordinates, width, and height of the bounding box, respectively, and C is a constant that is closely related to the dataset.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] The present invention improves the model to be lightweight while ensuring recognition accuracy and speed, reduces the number of parameters and calculations, saves equipment resources and meets the needs of real-time detection. In addition, the improved model is applied to the actual scene shot by the drone to count the detected lychee fruits, and the detection results of the dynamic video are further visualized, paving the way for predicting the yield of lychee fruits.

[0035] The improved YOLOv8s model of the present invention introduces PConv to replace part of the conventional convolution in the C2f module for lightweight improvement, and then integrates the EMA attention mechanism to improve the model's attention to the features of litchi and reduce the influence of complex factors in the background on the detection effect; the neck network is replaced by BiFPN that integrates the P2 feature layer, and the feature information between different network layers is bidirectionally integrated to enhance the feature fusion ability of the model; NWD is integrated into the regression loss function to improve the problems of false detection and missed detection of small targets in complex environments without increasing the calculation complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 1 is an example of a litchi fruit collection image according to an embodiment of the present invention, wherein (a) is a ripe fruit image, (b) is an immature fruit image, (c) is a leaf occlusion image, (d) is a fruit occlusion image, (e) is a close-up image, and (f) is a distant image;

[0038] Figure 2 1. The data expansion example diagram of the embodiment of the present invention, (a) is the original data image, (b) is the mirror image + rotation, (c) is the rotation + noise, (d) is the rotation + brightness adjustment;

[0039] Figure 3 This is a network structure diagram of an improved YOLOv8s model according to an embodiment of the present invention;

[0040] Figure 4 This is a structural diagram of a C2f module according to an embodiment of the present invention;

[0041] Figure 5 A C2f-Faster structure diagram of an embodiment of the present invention;

[0042] Figure 6 It is a structural diagram of C2f-Faster-EMA according to an embodiment of the present invention;

[0043] Figure 7 1 is a BiFPN structure diagram of an embodiment of the present invention, (a) is an original BiFPN structure diagram, (b) is an improved BiFPN structure diagram;

[0044] Figure 8 This is a flow chart of a method for dynamic identification and counting of litchi fruits based on improved YOLOv8s according to an embodiment of the present invention;

[0045] Fig. 9 It is a loss convergence comparison curve diagram of YOLOv8s and YOLOv8s-FEB training in an embodiment of the present invention;

[0046] Fig.10 It is a model heat map visualization of an embodiment of the present invention;

[0047] Fig.11 These are example diagrams of partial tracking results of dynamic videos according to an embodiment of the present invention, where (a) is the result of 121 frames, (b) is the result of 124 frames, (c) is the result of 129 frames, (d) is the result of 552 frames, (e) is the result of 556 frames, and (f) is the result of 560 frames. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] This embodiment proposes a dynamic identification and counting method for litchi fruits based on improved YOLOv8s. Figure 8 As shown, the specific steps include:

[0051] Obtaining a litchi image to be identified;

[0052] The litchi image to be identified is input into the improved YOLOv8s model to obtain the litchi fruit counting result; wherein, the improved YOLOv8s model is trained through a training set, and the training set is the labeled litchi image data.

[0053] Specifically, this embodiment improves the model to be lightweight while ensuring recognition accuracy and speed, reduces the number of parameters and calculations, saves equipment resources and meets the needs of real-time detection. In addition, the improved model is applied to the actual scene shot by the drone to count the detected lychee fruits, and the detection results of the dynamic video are further visualized, paving the way for predicting the yield of lychee fruits.

[0054] Furthermore, before labeling the training set data, the method further includes: collecting original litchi image data and preprocessing the original litchi image data;

[0055] The preprocessing of the original litchi image data includes:

[0056] The original litchi image data is processed by a combination of mirroring, rotating, adding noise and adjusting brightness to obtain litchi image data.

[0057] Specifically, the collection equipment used was a handheld smartphone and a DJI mavic3 classic drone. Due to the complexity of the natural environment background and the diversity of the target shapes of litchi fruits, different litchi colors, light conditions, shooting angles, occlusion categories, shooting distances, etc. were selected during the collection process to further increase the diversity of target data and the generalization ability of the model. A total of 837 litchi images were collected and saved in JPG format. Figure 1 (a)-(f) show some of the collected litchi images.

[0058] First, the obtained litchi image sample data is labeled by drawing the minimum bounding rectangle of the litchi fruit in the image using LabelImg software to ensure that the background area contained in the rectangle is as small as possible. Set the category attribute of the rectangle to "Litchi", and create a TXT format label file according to the VOC2007 data format, including the height, width and category information of the rectangle. The annotated XML file name is consistent with the image name.

[0059] In order to enhance the diversity of the dataset and improve the detection performance and generalization ability of the target detection model, data enhancement operations are performed on the collected litchi image data. The OpenCV library in Python is used to expand the original dataset, and a combination of mirroring, rotation, noise addition, and brightness adjustment is used to cover as many litchi images as possible in various complex backgrounds in natural orchard environments. Some examples are as follows: Figure 2 (a)-(d) are shown. Adding Gaussian noise to the image can simulate the image blur caused by motion jitter; mirroring and rotating the original data can simulate the image taken by the camera at multiple angles; adjusting the brightness value of the image pixels can simulate the image generated under different lighting conditions. After amplification, the number of litchi images obtained is 3348, and the corresponding TXT tag files are converted accordingly. The data set is divided into training set, validation set and test set in a ratio of 8:1:1.

[0060] Furthermore, the improved YOLOv8s model is:

[0061] Introduce partial convolution to replace some conventional convolutions in the C2f module for lightweight improvement, which is used to reduce the number of model parameters and calculations;

[0062] Incorporate an efficient multi-scale attention mechanism to improve the feature extraction capability of the backbone network;

[0063] The weighted bidirectional feature pyramid network of the P2 feature layer is introduced to replace the path aggregation network to enhance the model's cross-scale feature fusion of different scales;

[0064] The normalized Gauss-Wasserstein distance metric is introduced into the regression loss function to improve the model's learning ability for small litchi objects;

[0065] Based on the above improvements, an improved YOLOv8s model is obtained.

[0066] Specifically, as a product of comprehensive performance optimization, YOLOv8 not only supports multiple visual tasks such as target classification, target detection, target tracking and pose estimation, but also provides efficient processing capabilities for these tasks. Based on YOLOv5, YOLOv8 introduces new functions and improvements. In the backbone network and Neck part, the C3 structure of YOLOv5 is replaced with the C2f structure with richer gradient flow, and the number of channels is adjusted for different scale models, which greatly improves the model performance. At the same time, the convolution structure in the upsampling stage is deleted, so that the model has a faster computing speed. In the Head part, the current mainstream decoupled head structure is replaced, and the classification and detection heads are separated, further reducing the complexity of the model. At the same time, the anchor-based detection method is replaced with the anchor-free detection method. This design reduces the number of bounding boxes and speeds up the speed of non-maximum suppression (NMS), ensuring that it can detect and identify objects in the image in a shorter time. The loss calculation consists of two branches: classification and regression. The previous confidence branch is deleted. The classification loss still uses the binary cross-entropy loss function (BCE Loss), while the regression loss uses the distribution focal loss function (Distribution Focal Loss) and the complete intersection over union loss (CIoU Loss). The two types of losses are weighted with certain weights, and the total loss is finally obtained by summing. In terms of loss calculation, a positive and negative sample dynamic allocation strategy (TaskAlignedAssigner) is adopted. The weighted scores of classification and regression can be adjusted according to the training loss or other indicators to distinguish samples to better adapt to different data sets and models. In addition, YOLOv8 also provides customizable model options, allowing users to adjust the depth and width of the network by controlling the scaling factor, so that YOLOv8 can flexibly adapt to various application scenarios from edge devices to high-performance servers. Through this multi-restricted size measurement model design, users can customize the model configuration that best suits their usage scenarios according to actual needs and resources, thereby improving the effective utilization of resources and the efficiency of task execution.

[0067] The main process of the YOLOv8 algorithm is as follows: First, the image is resized to 640×640×3 through image preprocessing, and then the image is input into the backbone network to extract features. The result of feature extraction is a feature layer with a size of 80×80×256×α, 40×40×512×α, and 20×20×1024×α. Subsequently, the network fuses the three feature layers in the middle layer to form three new fused feature layers. Finally, all fused feature layers are sent to the decoupled detection head to complete the target detection task.

[0068] Improve the structural design of the YOLOv8 litchi detection model:

[0069] This embodiment combines the model size, inference speed and detection accuracy of the five versions of YOLOv8, and selects the YOLOv8s model as the basic model for litchi fruit detection in the orchard scene. In order to make the YOLOv8s model more suitable for litchi detection in the orchard, especially for the problem of litchi small target detection, this embodiment improves the YOLOv8s model, and its network structure is as follows Figure 3 In the improved model, PConv is introduced into the C2f module for lightweight improvement to design a fast and lightweight network model. At the same time, the EMA attention mechanism is integrated to improve the feature extraction capability of the backbone network. In order to compensate for the receptive field loss caused by downsampling and enhance the feature fusion capability of the neck, the original PANet is changed to BiFPN fused with the P2 feature layer. NWD is integrated in the loss function to make the model more sensitive to small target detection and reduce missed detection and false detection.

[0070] Furthermore, introducing partial convolution to replace part of the conventional convolution in the C2f module for lightweight improvement includes: using partial convolution to replace part of the conventional convolution, and using conventional Conv on part of the input channel to extract spatial features, thereby completing the lightweight improvement.

[0071] Specifically, the C2f module in YOLOv8 is composed of multiple Bottleneck blocks, which causes YOLOv8 to contain a large number of Bottleneck structures, making the model computationally complex and requiring more storage space and computing resources. At the same time, it increases the time cost when training and reasoning the model. Figure 4As shown. Currently, many mainstream lightweight networks such as EfficientViT, FasterNet, ConvNeXt V2, EfficientFormer, etc. use deep convolution or group convolution to extract spatial features. These are all focused on reducing the number of floating-point operations. However, this reduction in floating-point operations does not necessarily lead to a similar degree of reduction in latency. This is mainly due to the frequent operator memory access, which leads to low floating-point operations per second. While reducing the amount of calculation, the operator is affected by the side effect of increased memory access, and the fragmented calculation efficiency is low. The partial convolution (PConv) in FasterNet only applies regular convolution on part of the input channel. By reducing redundant calculations and memory access at the same time, spatial features can be extracted more efficiently, which helps to improve the performance of the model for small target detection. PConv only needs to apply regular Conv on part of the input channel for spatial feature extraction, and keep the remaining channels unchanged. Therefore, PConv has a lower number of floating-point operations and a higher floating-point calculation speed than regular convolution, deep convolution, and group convolution. The FLOPs floating-point operation of PConv is calculated as follows:

[0072]

[0073] In the formula, h and w represent the width and height of the feature map respectively; k is the size of the convolution kernel; c p is the number of channels involved in the convolution operation. Usually, c p The number of channels used is 1 / 4 of that of regular convolution, so the FLOPs of PConv is only 1 / 16 of that of regular convolution.

[0074] This embodiment refers to the design idea of ​​the FasterNet Block module in FasterNet, and uses the class inheritance method to replace Bottleneck with FasterNet Block in the feature extraction of the C2f module to design the C2f-Faster module. The introduction of PConv replaces part of the conventional convolution, reduces the floating point number and calculation amount in the convolution process, and realizes the lightweight design and fast detection of the model. The C2f-Faster module structure is as follows Figure 5 shown.

[0075] Furthermore, the efficient multi-scale attention mechanism is used to improve the feature extraction capability of the backbone network, including:

[0076] Reshape some channels into batch channel dimensions and group the channel dimensions into multiple sub-features;

[0077] Input the sub-features into parallel 3x3 and 1x1 convolutions respectively to obtain intermediate features;

[0078] Based on the intermediate features and context information, output features in parallel are obtained;

[0079] Aggregate the output features to obtain image features.

[0080] Specifically, due to the small size of litchi fruits, the background accounts for a large proportion, and the sizes of litchis in different maturity states in the image are different. When litchi is immature, the skin of the litchi is relatively green or has uneven color, and the color of the fruit skin is extremely similar to that of the leaves. The accuracy of the recognition of the appearance features of litchi affects the accuracy of litchi yield estimation. The Efficient Multi-Scale Attention Mechanism (EMA) aims to retain the information on each channel and reduce the computational overhead. It reshapes some channels into batch dimensions and groups the channel dimensions into multiple sub-features so that the spatial semantic features are evenly distributed in each feature group. A new cross-space learning method is used to aggregate the output features of the two parallel branches to capture the pixel-level pairwise relationship to produce better pixel-level attention. In contrast to the progressive behavior formed by the limited receptive field, parallel 3x3 and 1x1 convolutions are used to utilize more contextual information in the intermediate features. In this embodiment, in order to reduce the interference of redundant information and improve the network's attention to litchi fruit features, the EMA module is integrated into the forward propagation process of FasterNet Block, and then the C2f-Faster-EMA module is constructed to achieve accurate extraction of litchi spatial features. The C2f-Faster-EMA module structure is as follows Figure 6 shown.

[0081] Based on the original YOLOv8 network, this embodiment uses the C2f-Faster-EMA module to replace the C2f module in the backbone network. This improvement adds an attention mechanism in the forward propagation process, so that the feature extraction process focuses on improving the accuracy of small target detection, while reducing the number of floating-point operations in the backbone network feature extraction process. The addition of the attention mechanism increases the computational complexity of the model, but effectively improves the extraction of image features. Therefore, for the C2f module in the neck of YOLOv8, the EMA module is not added, but the C2f-Faster module is used for replacement.

[0082] Furthermore, obtaining a weighted bidirectional feature pyramid network includes:

[0083] Based on bidirectional span connections and fast normalized fusion, a weighted bidirectional feature pyramid network is obtained.

[0084] Specifically, in order to solve the problem of insufficient feature fusion in PANet, cross-layer multi-scale fusion is proposed. At the same time, in order to fuse more features, an additional path is added to connect the input nodes and output nodes of the same-level network. Each direction (top-down and bottom-up) is regarded as a feature network layer, which is reused multiple times to obtain higher-level feature fusion. BiFPN can effectively enhance the model's detection ability for small target objects. The structure of the original BiFPN is as follows Figure 7 (a) shown.

[0085] During the feature fusion process, since different features have different resolutions, their contributions to the output are usually different. BiFPN uses a fast normalized fusion module to balance the weights of different features and add an additional weight to each input to allow the network to learn the importance of each input feature. The relationship between input and output is:

[0086]

[0087] Where w i The input feature I i The corresponding learning weights, where in each w i Adding the ReLU activation function later ensures that w i ≥0, set ε=0.0001 to avoid numerical instability.

[0088] Based on bidirectional cross-scale connection and fast normalization fusion, the final weighted bidirectional feature pyramid network is obtained. Take feature layer P4 as an example:

[0089]

[0090] In the formula is the intermediate feature of level 4 of the top-down path, is the 4th layer feature extracted from the backbone, It is the output feature of the fourth level of the bottom-up path, and Resize is an upsampling or downsampling operation.

[0091] At the same time, in order to further improve the efficiency of feature fusion, the Conv here uses depthwise separable convolution, and a BN layer and activation function are added after each convolution. All other feature layers are constructed in a similar way.

[0092] Usually, the yield of lychee fruit is estimated when the lychee fruit is not yet fully ripe or has already ripened. At the same time, in the large-scale lychee fruit target detection, there are many lychee targets in the image field of view and the targets themselves are small, usually including immature lychees and ripe lychees. In order to identify these two targets, without significantly increasing the number of parameters, this embodiment adds the P2 shallow feature layer to BiFPN, so that the model obtains higher resolution receptive field feature fusion to obtain more accurate lychee location information. Improved BiFPN structure such as Figure 7 (b) as shown.

[0093] Furthermore, the method of cross-scale fusion using weighted bidirectional feature pyramid network is as follows:

[0094]

[0095] in, is the intermediate feature of the i-th level of the top-down path, is the i-th layer feature extracted from the backbone, is the output feature of the i-th level of the bottom-up path, Resize is an upsampling or downsampling operation, w1 is the learning weight of the input feature I1, w2 is the learning weight of the input feature I2, is the i+1th layer feature extracted from the backbone, w'1 is the first-order learning weight of the input feature I1, w'2 is the first-order learning weight of the input feature I2, w'3 is the first-order learning weight of the input feature I3, ε is the initial learning rate, is the output feature of the i-1th level of the bottom-up path.

[0096] Furthermore, when normalizing the Gauss-Wasserstein distance detection, first, it models the bounding box as a two-dimensional Gaussian distribution, where the central pixel of the bounding box has a higher weight and the importance of the pixel decreases from the center to the boundary. Then, the similarity between the bounding box and the ground truth is calculated and normalized to obtain NWD, as shown in Equations (7) and (8).

[0097]

[0098] In the formula is the Gauss-Wasserstein distance between two bounding boxes, C is a constant that is closely related to the dataset, is a distance measurement. N a and N b A=(cx a ,cy a ,w a ,h a ) and B=(cx b ,cy b ,w b,h b ) modeled by a Gaussian distribution, where cx, cy, w, and h are the coordinates, width, and height of the bounding box, respectively. a ,N b ) is the Gauss-Wasserstein distance normalized in exponential form.

[0099] The NWD metric can also be easily embedded into the loss functions of label assignment, non-maximum suppression, and anchor-based detectors to replace the original IoU metric. Some studies applied the NWD metric to non-maximum suppression and loss functions, thus achieving better model performance.

[0100] This embodiment enhances the model's ability to detect small objects by introducing the NWD metric into the regression loss function of YOLOv8, as shown in formulas (9) and (10).

[0101] Loss = w1·Loss Objectness +w2·Loss Classification +w3·Loss Regression (7)

[0102] Loss Regression =α1·CIoU+α2·NWD (8)

[0103] Where w is the weight of each loss, Loss Regression It consists of CIoU and NWD with different scaling factors α.

[0104] At present, most multi-target tracking algorithms use the strategy of high-score bounding box matching to identify targets, but low-score bounding boxes are directly discarded, resulting in a large amount of target information missing. In the BoT-SORT tracking algorithm, multiple targets are tracked by associating each detection bounding box, including low-score detection bounding boxes, which can greatly improve the overall performance of the tracking process. The overall process of the BoT-SORT algorithm is similar to Byte-Track, but there are 4 improvements: ①BoT-SORT not only uses the appearance information of the target for tracking, but also considers the motion information of the target. The tracking accuracy is improved by jointly modeling the motion trajectory and appearance features. ②BoT-SORT uses the camera motion compensation method to improve the prediction ability of the bounding box and reduce the impact of camera motion on target tracking. ③BoT-SORT uses a more accurate Kalman filter state vector to predict and update the target position, reduce tracking errors, and improve target positioning accuracy. ④BoT-SORT uses the cosine distance fusion method of IoU and ReID to improve the robustness between detection and trajectory.

[0105] The tracker combined with the target detection network can realize the function of multi-target tracking. In order to enable YOLOv8s-FEB to accurately count the yield of litchi orchards and to adapt YOLOv8s-FEB to more application scenarios, this embodiment combines the proposed YOLOv8s-FEB with BoT-SORT to realize the dynamic recognition and counting of litchi fruits. Figure 8 The figure shows the workflow of combining YOLOv8s-FEB and BoT-SORT for litchi fruit detection and tracking. First, the lychees on the whole tree are photographed in 3D by a drone to collect video data. Then, the collected video sequence is transmitted to YOLOv8s-FEB for litchi target detection. Finally, the recognition output of YOLOv8s-FEB is used as the input of BoT-SORT to complete the tracking and counting of litchi fruits.

[0106] The hardware configuration of the training model platform in this embodiment is as follows: CPU is Intel(R) Xeon(R) Platinum8383C CPU@2.70GHz 3.60GHz, 128GB running memory; GPU is NVIDIA GeForce RTX 4090, 24GB video memory. Software environment: Windows 10 operating system, Python 3.8.17 version, Pytorch 1.12.1 framework, CUDA 12.2 version. The training parameter settings are shown in Table 1.

[0107] Table 1 Training hyperparameter settings

[0108]

[0109] In order to evaluate the detection performance of the model on the litchi data set, this embodiment uses the evaluation indicators commonly used in the target detection algorithm, including precision (Precision, P), recall (Recall, R), mean average precision (MeanAveragePrecision, mAP), F1 value (F1Score), frames per second (Frames Per Second, FPS), parameter amount (Params) and floating point operations (FLOPs). Among them, mAP when IoU is 0.5 is selected as the performance evaluation index for identifying litchi fruits. The higher the value, the better the recognition effect of the improved model. The higher the FPS of the model detection speed, the better the real-time performance. The smaller the value of the parameter amount and the floating point operation amount, the lower the complexity of the model.

[0110] In order to verify the effectiveness of the proposed improved methods, the improved methods were introduced one by one on the basis of the YOLOv8s model. All models used the same data set and carried out ablation tests under the same parameter environment. In Experiment 2, the C2f-Faster-EMA module was used to replace the original C2f module in the backbone network of YOLOv8s. Compared with the original model, the recognition accuracy and average precision increased by 0.8 and 1.0 percentage points respectively, indicating that EMA can enhance the feature extraction ability of the network model and pay attention to more easily submerged feature information of the target to be detected. The model recall rate decreased slightly. The reason is that only part of the channels are convolutional, which effectively reduces the model weight and computational redundancy, while losing a small amount of features that may be contained in the remaining channels, resulting in a slight decrease in recall rate. After replacing it with BiFPN that integrates the P2 feature layer in Experiment 3, the model accuracy and average precision increased by 1.1 and 0.8 percentage points respectively, indicating that BiFPN can fuse more features and provide target features through a weighted feature fusion mechanism. The features are assigned more weights; after adding NWD to the regression loss function in experiment 4, the recall rate of the model increased by 1.9 percentage points, indicating that NWD can improve the fitting degree of the model and enhance the model's ability to detect small targets; after replacing C2f-Faster-EMA and BiFPN in experiment 5, compared with the original model, the model precision, recall rate, and average precision mean values ​​are all improved, while the model parameter volume and calculation amount are reduced by 47.3% and 27.1% respectively, and the FPS is increased by 4.5, proving the effectiveness of lightweight improvement; experiment 6 adds NWD on the basis of experiment 4, and combines the advantages of multiple modules to achieve the final model recognition accuracy of 89.3%, recall rate of 85.5%, and average precision of 91.9%, which are 1.4, 1.6, and 1.7 percentage points higher than the original YOLOv8s, respectively. The detection speed FPS of the final model is 38.9, which is greater than the FPS value of 24 that meets the real-time requirements. While improving the detection speed, the improved model has higher detection accuracy, and the model parameter volume and calculation amount are lower, indicating that the improved model has better model performance advantages.

[0111] Next, we compare the convergence of the position loss values ​​of the YOLOv8s original model and the YOLOv8s-FEB improved model, and perform 200 iterations in total. The result curve is: Fig. 9 As shown. Fig. 9 It can be seen that the convergence speed of the improved model YOLOv8s-FEB is faster than that of the original model YOLOv8s, which shows that the improved method proposed in this embodiment can not only improve the model detection performance and reduce the model complexity, but also has obvious advantages in terms of convergence speed.

[0112] In order to verify the effectiveness of the improved lightweight network in this embodiment, the YOLOv8s target detection network is used as the basic network to replace the mainstream lightweight backbone feature extraction network, such as EfficientViT, FasterNet, ConvNeXt V2, and EfficientFormer. On the basis of keeping other parameters consistent except the backbone network, the recognition effects of different backbone networks on litchi targets are compared.

[0113] EfficientViT has the least number of parameters and the smallest amount of computation. In terms of detection, its precision is 0.7 percentage points higher than that of YOLOv8s, but its recall rate and average precision are 3.0 and 1.2 percentage points lower than those of YOLOv8s, respectively. ConvNeXtV2 has a smaller amount of computation, and its precision and average precision are 1.6 and 0.3 percentage points higher than those of YOLOv8s, respectively, but its recall rate is 1.4 percentage points lower than that of YOLOv8s, and its parameters are larger. EfficientFormer has improved precision, recall, and average precision compared to YOLOv8s, increasing by 0.2, 0.8, and 0.4 percentage points, respectively, but its parameters and computation are relatively large. C2f-faster-EMA has fewer parameters and a relatively small amount of floating-point operations, and its performance in precision and average precision is higher than that of YOLOv8s, increasing by 0.8 percentage points and 1.0 percentage points, respectively, and its recall rate is slightly lower than that of YOLOv8s. In summary, C2f-faster-EMA has better performance. While reducing the complexity of the model, it can improve the recognition effect of the model on litchi targets. Therefore, it is reasonable to use C2f-Faster-EMA to replace C2f in YOLOv8s.

[0114] In order to explore the contribution of CIoU and NWD to the regression loss function, this embodiment explores their influence on small target detection by adjusting the ratio between these two indicators. Using CIoU to completely replace NWD can improve the detection accuracy, but the low recall rate can easily cause missed detection and false detection, which is not conducive to the recognition of small target lychees; setting the NWD ratio coefficient to 1 is the most effective in improving the recall rate of small target recognition, but the recognition accuracy is reduced; compared with the model with the NWD ratio coefficient set to 1, setting the CIoU ratio coefficient to 0.3 and the NWD ratio coefficient to 0.7 can improve the recognition accuracy and average precision of the model, but the recall rate is slightly reduced; and setting their respective ratio coefficients to 0.5 can improve the overall recognition performance of the model. Although its accuracy is slightly lower than the model with the CIoU ratio coefficient set to 1, its recall rate is slightly lower than the model with the NWD ratio coefficient set to 1, but its F1 score reaches the highest. The accuracy and recall rate of model recognition are equally important for the yield estimation of lychee fruits. Therefore, the ratio coefficients of NWD and CIoU in this embodiment are 0.5 respectively, which is the best combination.

[0115] However, the performance of NWD on the original YOLOv8s model is not ideal, and the accuracy rate has decreased. When the EMA attention mechanism is used to strengthen feature extraction and BiFPN is used for feature fusion, NWD shows good performance, which may be because EMA reduces the interference of redundant information and the cross-scale fusion and weighted feature fusion of BiFPN make the fused feature information richer and more accurate, so that NWD can play a better role. More specifically, after using NWD, the accuracy rate increased from 88.7% to 89.3%, while the number of parameters and computational complexity of the model did not change, which well proves that NWD has a positive effect on the detection of small objects.

[0116] The above analysis shows that the improved model YOLOv8s-FEB of this embodiment has better detection effect than the original YOLOv8s. In order to further explore the performance advantages and disadvantages of the YOLOv8s-FEB model and the current excellent target detection models, this embodiment trained the commonly used Faster R-CNN, SSD, CenterNet, RetinaNet, YOLOv5m, YOLOv7, YOLOv8m models and the lightweight models YOLOv3-tiny, YOLOv5s, and YOLOv7-tiny under the same experimental environment and data set.

[0117] For the litchi target detection scenario, the general convolutional neural networks Faster R-CNN, SSD, CenterNet and RetinaNet have poor detection effects. Although RetinaNet has the highest detection accuracy, its recall rate and average precision are low, and the model complexity is high. Compared with the YOLOv5m model, the accuracy, recall rate and average precision of YOLOv8s-FEB are 2.0, 2.7 and 3.4 percentage points higher, respectively, and the number of parameters and the amount of calculation are reduced by 71.9% and 56.8, respectively. Compared with the YOLOv7 model, the average precision of the YOLOv8s-FEB model is slightly lower, because the network of YOLOv7 is more complex and can extract features deeply, but the inference time of YOLOv7 on the validation set is longer, which is not conducive to the deployment of the model and real-time detection. Compared with the YOLOv8m model, the recall rate of the improved model is increased by 0.6 percentage points, and its number of parameters and the amount of calculation are reduced by 77.3% and 73.7%, respectively.

[0118] Compared with the YOLOv3-tiny, YOLOv5s, and YOLOv7-tiny lightweight models, although the YOLOv8s-FEB model has a slightly higher computational complexity, it has the smallest number of parameters, and its precision, recall, and average precision are all greater than those of the three lightweight models. The average precision is 10.0, 2.4, and 0.9 percentage points higher than that of YOLOv3-tiny, YOLOv5s, and YOLOv7-tiny, respectively. At the same time, the model parameters are reduced by 32.4%, 16.4%, and 2.5% compared to YOLOv3-tiny, YOLOv5s, and YOLOv7-tiny, respectively.

[0119] The results show that the improved YOLOv8s-FEB has obvious advantages over other mainstream networks. While having the highest F1 value, it reduces a certain amount of calculation and parameters, consumes less memory resources, and is more suitable for litchi fruit target detection tasks in litchi yield estimation scenarios.

[0120] In order to more intuitively observe the effectiveness of the improved model in solving the problem of overlapping and occlusion of litchi fruits, this example extracts three images with overlapping and occlusion problems and uses Grad-CAM

[27] to draw a heat map for visualization. The brightness of a certain area in the heat map can show the weight assigned by the detection network in the image. The results are shown in Figure 2. Fig.10 As shown in the figure, this further shows from a visualization perspective that YOLOv8s-FEB can pay attention to more details of relevant areas and focus on key parts that are not blocked. The improved model focuses on more prominent litchi target features and has richer feature extraction and semantic understanding capabilities.

[0121] In order to verify the detection effect of the improved model based on YOLOv8s in the actual litchi orchard scene, the dynamic video actually collected by the drone in the orchard without model training is used as the input of YOLOv8s-FEB, and the obtained litchi fruit information is used as the input of BoT-SORT. Each detected litchi will be assigned a unique ID number, and the tracker counts the unique IDs in the video sequence and outputs the tracking results of each frame, and finally returns the counting results. Fig.11 (a)-(f) show the tracking results of BoT-SORT in the video frame. The red box represents each litchi target tracked, and the number in the upper left corner represents the ID matched by the tracker to each fruit. The video was shot by a DJI mavic3 classic drone at a distance of 1-2 meters from the tree canopy, with the camera facing vertically to the tree canopy, and at a speed of about 0.15m / s around the fruit tree. The video frame rate is 30 frames / second, the original resolution is 1080P, and H.265 encoding is used.

[0122] from Fig.11 Four different counting situations can be observed in the figure. The first category is correct counting. For example, the lychee fruits with IDs 221 and 263 can be accurately tracked by the tracker throughout the process from frame 121 to frame 129. The lychee fruit with ID 325 is confirmed at frame 121. Although it is temporarily lost due to occlusion when it moves to frame 124, it is identified as the same fruit by the detection model association at frame 129 and is assigned the same ID number again. The second category is repeated counting. For example, the lychee fruit with ID 229 is confirmed at frame 121, but when it moves to frame 124, it is lost due to occlusion of the fruit and branches. Subsequently, at frame 129, the fruit with ID 229 is detected again, but because the disappearance time exceeds the tracking time threshold, the model identifies it as a newly appeared fruit and its ID is changed to 405, resulting in an ID switching phenomenon, which causes the fruit to be counted repeatedly. The phenomenon of ID switching of litchi fruits occurs in the video sequence. On the one hand, the target tracker may make association errors and mistake the same litchi target as two or more different litchis, causing the target to jump from a lower ID to a higher ID. On the other hand, since some litchis are very similar in appearance or movement, the tracker may be confused, resulting in ID switching and a high final count. The third type of situation is missed counts, e.g. Fig.11The lychee fruit in the blue circle in (d), (e), and (f) is small in size due to the far field of view. From the beginning to the end of the entire video, the detector and tracker failed to detect and track the fruit. The fourth type of situation is wrong counting. For example, at frame 552, the lychee fruit with ID 1222 is confirmed, and the two adjacent lychee fruits in the red frame with ID 1211 and 1212 are mistakenly identified as one fruit. Subsequently, at frame 556, the fruit with ID 1222 disappears, and the detection model identifies it as the same fruit as the fruit with ID 1211. The three overlapping fruits generate a fruit frame with ID 1211. When faced with adjacent or overlapping fruits, the count may be missed due to the tracker's missed recognition. When the tracker mistakenly identifies two or more different lychee targets as the same lychee, a counting error occurs, resulting in the count value in the video sequence being lower than the actual value.

[0123] The maximum ID of litchi detected in the last frame of the video is 1450, and the total number of litchi in the video is about 1714 after manual counting 3 times and taking the average, that is, the target detection and tracking accuracy is about 84.6%. The maximum ID of litchi obtained by tracking with the original YOLO v8s+BoT-SORT is 1203, and the target detection and tracking accuracy is about 70.2%. Therefore, the improved YOLOv8s-FEB combined with BoT-SORT can better achieve the tracking and counting of litchi.

[0124] In this embodiment, the detection of litchi in large scenes has problems such as small targets, overlaps and occlusions. A lightweight litchi target detection algorithm YOLOv8s-FEB based on the improvement of YOLOv8 is proposed. At the same time, in order to realize the yield estimation of litchi fruits in the actual orchard environment, a dynamic identification and counting method of litchi fruits based on YOLOv8s-FEB+BoT-SORT is designed. The main research conclusions are as follows:

[0125] (1) We independently photographed litchi images in natural orchards and established a litchi image dataset through data annotation and data enhancement to ensure data integrity and reliability for model training, testing, and verification.

[0126] (2) PConv is introduced to replace some conventional convolutions in the C2f module for lightweight improvement, and then the EMA attention mechanism is integrated to improve the model's attention to litchi features and reduce the impact of complex factors in the background on the detection effect; the neck network is replaced by BiFPN that integrates the P2 feature layer, bidirectionally integrating feature information between different network layers to enhance the model's feature fusion capability; NWD is integrated into the regression loss function to improve the problem of misdetection and missed detection of small targets in complex environments without increasing the computational complexity of the model. Combined with the remaining core modules of the original YOLOv8s, the YOLOv8s-FEB lightweight litchi detection model is proposed to complete the accurate detection of litchi fruits.

[0127] (3) Analysis of the comparative results of the ablation test shows that the precision, recall, and average precision of the improved YOLOv8s-FEB model on the test set are 89.3%, 85.5%, and 91.9%, respectively, which are 1.4, 1.6, and 1.7 percentage points higher than the original YOLOv8s model. At the same time, the parameters and computational complexity of the improved model are relatively small, which are 5.86M and 20.7GB, respectively, which are 47.3% and 27.1% lower than the original model. This shows that the improved model proposed in this embodiment occupies less memory resources and has better detection performance.

[0128] (4) Analysis of the comparison results of different target detection algorithms shows that the improved model proposed in this embodiment has certain advantages over the mainstream target detection network models of FasterR-CNN, SSD, CenterNet, RetinaNet, YOLOv5m, YOLOv7 and YOLOv8m, and the average precision can be improved by up to 46.6%. Compared with the lightweight models YOLOv3-tiny, YOLOv5s and YOLOv7-tiny, YOLOv8s-FEB has the smallest number of parameters, and the precision, recall and average precision are all greater than those of the three lightweight models, with the average precision being 10.0, 2.4 and 0.9 percentage points higher, respectively.

[0129] (5) Combining the YOLOv8s-FEB target detection algorithm with the BoT-SORT multi-target tracking algorithm, the lychee targets on the fruit trees are detected and tracked, and the fruit counting based on the 3D video sequence is realized. The final missed detection rate of detection and tracking is about 0.154 percentage points, which is 14.4 percentage points lower than that of the original YOLO v8s+BoT-SORT, and is more suitable for real-time detection and counting in the orchard environment. The fruit dynamic recognition and counting method proposed in this embodiment is low-cost and easy to implement, and can help to realize the future yield estimation of orchards.

[0130] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A dynamic identification and counting method for litchi fruits based on improved YOLOv8s, characterized in that: include: Obtaining a litchi image to be identified; The litchi image to be identified is input into the improved YOLOv8s model to obtain the litchi fruit counting result; wherein, the improved YOLOv8s model is trained through a training set, and the training set is labeled litchi image data.

2. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 1, characterized in that, Before labeling the training set data, the method further includes: collecting original litchi image data and preprocessing the original litchi image data; Preprocessing the original litchi image data includes: The original litchi image data is processed by a combination of mirroring, rotating, adding noise and adjusting brightness to obtain the litchi image data.

3. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 1, characterized in that, The improved YOLOv8s model is: The partial convolution PConv is introduced to replace some conventional convolutions in the C2f module for lightweight improvement, which is used to reduce the number of parameters and calculations of the model; Incorporate an efficient multi-scale attention mechanism to improve the feature extraction capability of the backbone network; The weighted bidirectional feature pyramid network of the P2 feature layer is introduced to replace the path aggregation network to enhance the model's cross-scale feature fusion of different scales; The normalized Gauss-Wasserstein distance metric is introduced into the regression loss function to improve the model's learning ability for small litchi objects; Based on the above improvements, the improved YOLOv8s model is obtained.

4. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 3, characterized in that, Introducing partial convolution to replace part of the conventional convolution in the C2f module for lightweight improvement includes: using the partial convolution to replace part of the conventional convolution, and using conventional Conv on part of the input channel to extract spatial features, thereby completing the lightweight improvement.

5. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 3, characterized in that, Using the efficient multi-scale attention mechanism, the feature extraction capability of the backbone network is improved, including: Reshape some channels into batch channel dimensions, and group the channel dimensions into multiple sub-features; The sub-features are respectively input into parallel 3x3 and 1x1 convolutions to obtain intermediate features; Based on the intermediate features and the context information, obtaining output features for output parallelism; The output features are aggregated to obtain image features.

6. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 3, characterized in that, Acquiring the weighted bidirectional feature pyramid network includes: Based on bidirectional span connection and fast normalization fusion, the weighted bidirectional feature pyramid network is obtained.

7. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 6, characterized in that: The method for cross-scale fusion using the weighted bidirectional feature pyramid network is: in, is the intermediate feature of the i-th level of the top-down path, is the i-th layer feature extracted from the backbone, is the output feature of the i-th level of the bottom-up path, Resize is an upsampling or downsampling operation, w1 is the learning weight of the input feature I1, w2 is the learning weight of the input feature I2, is the i+1th layer feature extracted from the backbone, w'1 is the first-order learning weight of the input feature I1, w'2 is the first-order learning weight of the input feature I2, w'3 is the first-order learning weight of the input feature I3, ε is the initial learning rate, is the output feature of the i-1th level of the bottom-up path.

8. The litchi fruit dynamic recognition and counting method based on improved YOLOv8s according to claim 3, characterized in that: The method for obtaining the normalized Gauss-Wasserstein distance metric is: Among them, NWD (N a ,N b ) is the Gauss-Wasserstein distance normalized in exponential form, is a distance measurement value, Na and Nb are given by A = (cx a ,cy a ,w a ,h a ) and B=(cx b ,cy b ,w b ,h b ) is a Gaussian distribution modeled by , where cx, cy, w, h are the coordinates, width, and height of the bounding box, respectively, and C is a constant that is closely related to the dataset.

Images