A winter jujube detection and counting method based on improved YOLOv5n deployed in an Android system

Abstract

The application designs an improved YOLOv5n model for detecting and counting winter jujubes in an Android system. The improvements made to the model include using SPD-Conv to replace the stride convolution layer to better detect small targets and low-resolution images. The lightweight upsampling operator Content-Aware ReAssembly of Features (CARAFE) is incorporated, which can use adaptive and optimized recombination kernels at different positions to improve performance. GSConv is used to replace some Conv in the Neck to improve accuracy while making the model more lightweight. Experimental results show that compared with the original YOLOv5n model, the improved model has an accuracy, recall rate and mAP that is 2.40%, 1.80% and 3.00% higher respectively, and RMSE and MAPE that is 9.11% and 5.30% lower respectively. The improved model is deployed in an Android system, and the camera is called for testing to detect and count winter jujubes. The detection interface picture reasoning time is 30-90 milliseconds. This method can effectively meet the needs of real-time detection and counting of winter jujubes, and provides a theoretical basis and technical support for visual inspection of winter jujube yield.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a winter jujube detection and counting method based on improved YOLOv5n deployed on an Android system. BACKGROUND

[0002] Winter jujube is a unique fresh jujube variety in China, which is popular among consumers due to its crisp taste and rich nutrition. In recent years, with the increase of winter jujube planting area in China, winter jujube yield estimation has become increasingly important, and yield estimation plays a guiding role in agronomic treatment, labor resource planning and market planning.

[0003] Fruit recognition, as a key technology for fine orchard management such as fruit yield prediction, has become a research hotspot in recent years. Traditional machine vision methods have poor robustness and real-time performance in different natural environments, and it is difficult to meet the real-time detection requirements of fruits. Convolutional neural networks have good application effect in fruit recognition. Convolutional neural network methods are divided into two categories: one is a two-stage detection model based on region proposal method, which generates a proposal box first and then classifies it, represented by R-CNN and Fast R-CNN. This model has high detection accuracy and strong generalization ability, but the detection time is long and cannot meet the real-time requirements. The other is an end-to-end one-stage detection model that directly obtains the predicted target position and classification using a single convolutional network, represented by the YOLO series model. This model is widely used due to its real-time, high-precision and other advantages.

[0004] Estimating fruit yield based on counting fruit number is an important task during the fruit harvest period; Qiao et al. (Qiao, Y., Hu, Y., Zheng, Z., Yang, H., Zhang, K., Hou, J., Guo, J., 2022. A Counting Method of Red Jujube Based on Improved YOLOv5s. Agriculture, 12(12), 2071.) proposed a red jujube counting method based on improved YOLOv5s, using ShuffleNetV2 as the model backbone, a new data loading module Stem was proposed to reduce information loss during the sampling process, BiFPN was used instead of PANet to enhance the model's feature fusion ability and improve the model's precision. Finally, the improved YOLOv5s detection model was used to count the red jujubes. Compared with YOLOv5s, the improved model parameters and model size were reduced by 93.75% and 91.67%, respectively, and Precision, Recall, and Fps were increased by 4.30%, 2.00%, and 3.60%, respectively. This method achieved fast and accurate detection of red jujubes while reducing the model size, providing a theoretical basis for visual estimation of red jujube yield; Wu et al. (Wu, D., Lv, S., Jiang, M., Song, H., 2020. Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments. Computers and Electronics in Agriculture, 178, 105742.) proposed a real-time detection method for apple flowers based on channel pruning YOLOv4 deep learning algorithm. The YOLOv4 model was constructed under the CSPDarknet53 framework, and channel pruning algorithm was used to prune the model, which simplified the apple flower detection model and achieved fast and accurate apple flower detection. The experimental results showed that the pruned apple flower detection model reduced the number of parameters by 96.74%, the model size by 231.51MB, and the inference time by 39.47%. The mAP was 97.31%, only 0.24% lower than the pruned model. This study provides technical reference for apple yield estimation and development of apple flower thinning robot; Mirhaji et al. (Mirhaji, H., Soleymani, M., Asakereh, A., Mehdizadeh, S. A., 2021.(Fruit detection and load estimation of anorange orchard using the YOLO models through simple approaches in different imaging and illumination conditions. Computers and Electronics in Agriculture, 191, 106533.) This study applies the YOLO detection model to the detection and counting of anoranges in Iran. The model was evaluated on 100 sample trees. After detecting and counting anoranges on images of all trees in the orchard, a yield change map of the orchard was extracted. The YOLOv4 model was used for orchard yield estimation. The model's precision, recall, F1-score, and mAP were 91.23%, 92.80%, 92.00%, and 90.80%, respectively. The results indicate that the YOLO model can effectively provide a rapid method for citrus fruit yield detection and estimation.

[0005] Rapid detection and counting of jujube fruits is one of the key technologies for estimating jujube yield. In machine learning research, small-sized jujube fruits cannot be detected with high accuracy; while in deep learning research, due to the large size of the model, the detection speed is slow, which limits the deployment on mobile devices.

[0006] This paper proposes a winter jujube detection and counting method based on an improved YOLOv5n, which can effectively identify small-sized winter jujube fruits. While ensuring sufficient detection speed after the model is deployed on mobile devices, it improves the accuracy of the model in detecting winter jujubes, so as to meet the requirements of accurate and real-time detection and counting of winter jujubes. Summary of the Invention

[0007] The purpose of this invention is to provide a method for detecting and counting jujubes based on an improved YOLOv5n, which is deployed on the Android system, thereby solving the aforementioned problems in the prior art.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for detecting and counting jujubes based on an improved Yolov5n, deployed on the Android system, includes the following steps:

[0010] S1. Construct an image recognition network model based on the improved Yolov5n model, and replace all Conv modules in the original Yolov5n Backbone and some Conv modules in the Neck with the SPD-Conv module;

[0011] S2, an image recognition network model based on an improved Yolov5n model is constructed, and a CARAFE upsampling module and a GSConv module are used to improve the Neck of the original Yolov5n;

[0012] S3, a winter jujube picture dataset is obtained, and the dataset is divided into a training set and a test set;

[0013] S4, the training set and the test set are respectively expanded through image enhancement processing;

[0014] S5, the improved Yolov5n model is trained using the expanded dataset, and a trained image recognition network model is obtained;

[0015] S6, the optimized image recognition network model is tested using the test dataset, and various indicators of the model test results are evaluated;

[0016] S7, the weight file of the improved Yolov5n model is quantized using a float16 quantization method;

[0017] S8, the model is deployed on an Android system, and a winter jujube detection and counting app is obtained;

[0018] S9, the winter jujube detection and counting app is tested using winter jujube images, and the accuracy and counting accuracy of the test results are evaluated.

[0019] Preferably, in the network model constructed in step S1, all Conv modules in the Backbone of Yolov5n and part of the Conv modules in the Neck are replaced with SPD-Conv modules, which specifically include the following steps:

[0020] S11, 6 SPD-Conv modules are added, and the SPD module is used to downsample the input feature map;

[0021] S12, the added module name is added to the import list, and the used module name is added to the subsequent module calling code.

[0022] Preferably, in the network model constructed in step S2, the upsampling operator CARAFE is used instead of the original Yolov5n upsampling, and the GSConv is used instead of the two Conv modules in the original Yolov5n, which specifically includes the following steps:

[0023] S21, the upsampling module of the Neck in the original Yolov5n is replaced with a CARAFE module;

[0024] S22, the two Conv modules of the Neck in the original Yolov5n are replaced with GSConv modules;

[0025] S23, add the added module name to the import list, and add the used module name in the subsequent module calling code.

[0026] Preferably, the method for obtaining the winter jujube image dataset in step S3 comprises the following steps:

[0027] S31, the sample image acquisition site is selected from the winter jujube test demonstration station of Northwest A&F University in Dali County, Weinan City, Shaanxi Province, and the winter jujube planting row spacing is 4m and the row spacing is 2m;

[0028] S32, in the image acquisition process, images under different light conditions, different fruit quantities, and different leaf shielding conditions are collected;

[0029] S33, the winter jujube image is preprocessed using Open cv, the resolution of the processed image is 640*480, and the black edges are automatically filled;

[0030] S34, divide the processed winter jujube sample dataset into a training set and a validation set, and the ratio is 9:1.

[0031] Preferably, the data augmentation method shown in step S4 includes but is not limited to using Open cv to perform data augmentation on images through image processing methods such as mirroring, adding noise, adjusting brightness, flipping, translation, and random erasing in a python environment.

[0032] Preferably, the model evaluation indicators of step S6 include precision, recall, mAP, root mean square error (RMSE), mean absolute percentage error (MAPE), model parameter quantity (Parameters), model size (MB), inference time (ms), wherein the definitions of precision, recall, mAP, root mean square error, and mean absolute percentage error are as follows:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] Preferably, step S7 selects a float16 quantization method to quantize the weight file of the improved Yolov5n model, which comprises the following steps:

[0039] S71, import the weight file of the improved detection model into export.py;

[0040] S72, set batchsize to 1 in export.py, the image output size to 320*320, and the model file output format to.tflite.

[0041] Preferably, step S8 deploys the model on an Android system, specifically including the following steps:

[0042] S81, import the quantized best5n-fp16.tflite model file into yolov5-tflite-android-master, and set the input and output parameters;

[0043] S82, modify the original code of FullimageAnalyse and FullScreenAnalyse files to realize the counting of winter jujube fruits;

[0044] S83, use the Build apk function of Android Studio to export the apk file of the model file, and obtain the winter jujube detection and counting app after downloading and installing on an Android mobile phone.

[0045] Preferably, step S9 tests the winter jujube detection and counting app using winter jujube images, and evaluates the accuracy and counting accuracy of the test results, the evaluation indexes including precision, inference time and detection accuracy, wherein the detection accuracy is defined as follows:

[0046]

[0047] The present application has the following beneficial effects:

[0048] The application provides a winter jujube detection and counting method based on an improved Yolov5n deployed in an Android system; the improvements made by the model include using SPD-Conv to replace the stride convolution layer to better detect small targets and low-resolution images, incorporating content-aware feature reorganization (CARAFE), which is a lightweight up-sampling operator that can use adaptively and optimized recombination kernels at different positions to improve performance, and replacing some Conv in the Neck with GSConv to improve accuracy while making the model more lightweight; the experimental results show that compared with the original YOLOv5n model, the improved model has higher counting accuracy, with an accuracy, recall rate and mAP increase of 2.40%, 1.80% and 3.00% respectively, and a RMSE and MAPE decrease of 9.11% and 5.30% respectively; we deploy the improved model on an Android device, call the camera to detect and count winter jujubes in real time, and the picture inference time of the detection interface is 30-90 milliseconds, so that our method can effectively meet the needs of real-time detection and counting of winter jujubes and provide a theoretical basis and technical support for visual inspection of winter jujube yield. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the network model structure diagram of the improved Yolov5n used in example 1;

[0050] Figure 2 is the structure diagram of the SPD-Conv module used in example 1;

[0051] Figure 3 is the feature pyramid structure diagram with CARAFE used in example 1;

[0052] Figure 4 is the structure diagram of the GSConv module used in example 1;

[0053] Figure 5 is the detection effect diagram of different algorithm models for winter jujube detection and counting in example 1;

[0054] Figure 6 is the flowchart of the model weight file quantization used in example 1;

[0055] Figure 7 is the winter jujube detection and counting flowchart in example 1, (a) is the winter jujube detection and counting main flowchart, and (b) is the winter jujube detection and counting program sub-flowchart;

[0056] Figure 8 is the winter jujube detection and counting effect diagram of the Android device in example 1, (a) is the detection of 3 winter jujubes, (b) is the detection of 6 winter jujubes partially covered by leaves, and (c) is the detection of 18 winter jujubes. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings; it should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] Example 1

[0059] The present example provides an improved Yolov5n-based winter jujube detection and counting method deployed in an Android system, as shown in Figure 1 The method comprises the following steps:

[0060] S1, construct an image recognition network model based on an improved Yolov5n model, replace all Conv modules in the Backbone and part of the Conv modules in the Neck of the original Yolov5n with SPD-Conv modules.

[0061] Convolutional neural networks are an important part of target detection, and winter jujube detection belongs to small target detection. In complex natural environments, it is difficult to ensure that the camera provides sufficient clarity; in the Backbone and Neck of YOLOv5n, SPD-Conv modules are used, which are composed of SPD layers and Conv layers. Replace all stride convolution layers in the YOLOv5n model architecture with SPD-Conv modules, which can effectively improve the accuracy of low-resolution image detection and small target detection.

[0062] In the Backbone of the original Yolov5n model, SPD-Conv modules are used, which comprise the following steps:

[0063] S11, replace all Conv modules in the Backbone of the original Yolov5n model with SPD-Conv modules, wherein the original Backbone code part includes:

[0064]

[0065] The improved Backbone code part includes:

[0066]

[0067] S12, modify the models / common.py file, add a space_to_depth module defined as a new class, and add the required module code to the common.py file.

[0068] S13, add 6 SPD_Conv modules, SPD-Conv in Backbone down-samples the input feature map.

[0069] Any intermediate feature map X of size WxWxC, cut out a series of sub-feature maps as follows:

[0070] f 0,0 =X[0:W:scale,0:W:scale],f 1,0 =X[1:W:scale,0:W:scale],...,

[0071] f scale-1,0 = X [scale-1 : W : scale, 0: W : scale]; (1)

[0072] f 0,1 =X[0:W:scale,1:W:scale],f 1,1 ,...,

[0073] f scale-1,1 = X [scale-1 : W : scale, 1 : W : scale]; (2) ...

[0074] f 0,scale-1 =X[0:W:scale,scale-1:W:scale],f 1,scale-1 ,...,

[0075] f scale-1,scale-1 = X [scale-1 : W : scale, scale-1 : W : scale]. (3)

[0076] Wherein, X is the intermediate feature map, W is the height and width of the intermediate feature map, C is the channel size of the intermediate feature map, f 0,0 , f 1,0 , f 0,1 , f 1,1 4 sub-maps, scale is the scale factor; given the original feature map X, each sub-map down-samples X by the scale factor to obtain four sub-maps f 0,0 , f 1,0 , f 0,1 , f 1,1 , connect the sub-feature maps along the channel dimension to obtain the feature map X'; after the SPD feature transformation layer, a non-stride convolution layer with a filter is used to further transform X', which can preserve as much image feature information as possible; the structure diagram of the SPD-Conv module is as followsFigure 2 As shown.

[0077] S14, add the added module name to the import list, and add the used module name in the subsequent module calling code.

[0078] S2, build an image recognition network model based on the improved Yolov5n model, and use the up-sampling module CARAFE and the GSConv module to improve the Neck of the original Yolov5n.

[0079] The Neck of the Yolov5n model is improved, which includes the following steps:

[0080] S21, modify the Neck part in the Yolov5n model yaml file, wherein the code part of the original Neck includes:

[0081]

[0082]

[0083] The code part improved by using the CARAFE module and the GSConv module includes:

[0084]

[0085]

[0086] S22, replace the up-sampling module in the Neck with the CARAFE module, and replace the two Conv modules with the GSConv module, modify the models / common.py file, define each module added as a new class, and add the required module code to the common.py file.

[0087] Feature up-sampling is a key operation in the feature pyramid architecture; this study uses a lightweight content-aware feature reorganization method CARAFE, and the feature pyramid structure with CARAFE is as follows Figure 3As shown; CARAFE acts as a reorganization operator of the content-aware kernel, which consists of two steps: the first step is to predict a reorganization kernel according to the content of each target position, and the second step is to reorganize the features with the predicted kernel, the use of CARAFE will amplify the feature map up-sampling by 2 times, which is integrated into the FPN by seamlessly replacing the nearest neighbor interpolation, the use of CARAFE can effectively improve the model precision without increasing the parameter amount, which is suitable for lightweight network architecture; in the Backbone, spatial information is transferred to the channel with down-sampling, and each time the height and width of the feature map are compressed and the channel size is expanded, which will cause the loss of part of the feature information, GSConv as a kind of channel dense convolution can keep the connection between channels as much as possible, reduce the loss of feature information, and can effectively reduce the time complexity, in the Neck, the feature map is down-sampled by the Backbone, the width and height reach the minimum value, and the channel size reaches the maximum value, its repetitive information is less, and it is no longer compressed, this study replaces the Conv in the Neck with GSConv, compared with Conv convolution, GSConv provides the model with sufficient precision and inference speed while effectively reducing the parameters of the model, making the model more lightweight; the structure diagram of the GSConv module is as shown in Figure 4 .

[0088] S23, add the added module name to the import list, and add the used module name in the subsequent module calling code.

[0089] S3, the method for obtaining the winter jujube image dataset, specifically comprising the following steps:

[0090] S31, the sample image acquisition site is selected from the winter jujube test demonstration station of Northwest A&F University in Dali County, Weinan City, Shaanxi Province, and the winter jujube planting row spacing is 4m and the row spacing is 2m.

[0091] S32, in the image acquisition process, images under different light conditions, different fruit quantities, and different leaf shielding conditions are collected.

[0092] S33, the winter jujube image is preprocessed using Open cv, the resolution of the processed image is 640x480, and the black edge is automatically filled.

[0093] S34, divide the processed winter jujube sample dataset into a training set and a validation set, and the ratio is 9:1.

[0094] After obtaining the above training set and validation set images, data augmentation is performed on the training set and the validation set respectively to increase the winter jujube dataset, so that the trained model has stronger generalization ability.

[0095] S4, the method for data expansion includes but is not limited to using Open cv in a python environment to expand data through image processing methods such as mirroring, adding noise, adjusting brightness, flipping, translation, random erasing, etc., specifically including the following steps:

[0096] S41, horizontal mirroring refers to rotating the original image by 10% probability horizontally.

[0097] S42, adding noise refers to adding salt and pepper noise to the image randomly to improve model stability.

[0098] S43, adjusting brightness refers to randomly adjusting the brightness value of the jujube image, and the random adjustment factor is 0.4.

[0099] S44, flipping refers to rotating the image by a random angle of -180° to 180°.

[0100] S45, translation refers to moving the image along the X or Y axis or in both directions.

[0101] S46, random erasing is to erase the jujube image randomly, and the erasing area is 2% to 15% of the whole image, so as to improve the generalization ability of the network to the occluded part.

[0102] S5, the improved Yolov5n model is trained using the expanded data set, the batch-size of the training is adjusted to 64, and the number of training rounds is set to 300. The specific experimental environment settings are shown in Table 1.

[0103] Table 1 Experimental environment

[0104]

[0105] S6, the optimized image recognition network model is tested using the test data set, and the accuracy of the test results is evaluated; the model evaluation indicators include precision, recall, mAP, RMSE, MAPE, model parameter quantity, model size (MB), inference time (ms), wherein the definitions of precision, recall, average precision, root mean square error, and mean absolute percentage error are as follows:

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] In order to verify the effectiveness of the improved YOLOv5n model for winter jujube detection, we select the batch size as 16, the experimental round as 300 rounds, and compare the evaluation indexes of YOLOv5n, YOLOv3-tiny, YOLOv4-tiny, YOLOv7-tiny, SSD, Fasterrcnn model and the improved YOLOv5n model. The comparison results are shown in Table 2.

[0112] Table 2 Performance of different algorithm models

[0113]

[0114] According to Table 2, compared with the improved model and YOLOv5n model, the improved model has higher precision than YOLOv5n model, and the precision, recall rate and mAP are increased by 2.40%, 1.80% and 3.00% respectively. In the model regression evaluation index, the RMSE of the improved model is reduced by 0.39, and the MAPE is reduced by 5.30% compared with YOLOv5n model. The model parameters of the improved model are increased by 317624, the model size is increased by 0.62MB, and the inference time is increased by 2.60ms. The improved model still meets the requirements of real-time detection and counting of winter jujube; compared with the lightweight models YOLOv3-tiny, YOLOv4-tiny and YOLOv7-tiny which are also YOLO series, in terms of model precision, the precision of the improved model is the highest among the above models, which is 4.18%, 5.32% and 1.21% higher than the above models respectively. The recall rate is also the highest among the above models, which is 4.19%, 3.16% and 1.11% higher than the above models respectively. The mAP is 4.61%, 6.44% and 3.78% higher than the above models respectively; in terms of model size, since the Depth_multiple and Width_multiple of YOLOv5n model in YOLOv5 series are the smallest, the model parameter amount is 76.02%, 65.69% and 65.41% less than the above three lightweight models respectively, and the model size is 75.00%, 81.50% and 64.52% less than the above three lightweight models respectively. The improved model has the smallest model parameters and model size, which is conducive to the deployment of the model in the Android system; in terms of model regression evaluation index, the improved model has the lowest RMSE and MAPE, which are 3.89 and 20.33% respectively, indicating that the model has the highest accuracy in counting winter jujube; the SSD model and Fasterrcnn model are not lightweight, and the model parameters are 13 times and 66 times of the improved model respectively. Moreover, the model is too large and the model inference time is too long, which cannot meet the real-time detection requirements, and the detection accuracy and model regression evaluation index are also lower than the improved model.

[0115] The detection effect of winter jujube using the above algorithm model is shown in Figure 5 Table 3 shows the number of winter jujube detected by each model and the detection accuracy (Accuracy).

[0116]

[0117] Table 3 shows the number of winter jujube detected by each model and the detection accuracy (Accuracy).

[0118]

[0119] Figure 5 Table 3 shows the results of evaluating the performance of various model detection and counting of winter jujube, showing the effectiveness of each model in detecting and counting winter jujube; Figure 5 Image 1 shows the detection of winter jujube images partially obscured by leaves. Among the 12 winter jujube detections, all models have missed some winter jujubes, with our model missing the fewest, missing 1 winter jujube. The Faster R-CNN model incorrectly detects leaves as winter jujubes because it does not have a feature pyramid architecture, which cannot fully extract image information, leading to detection errors; Figure 5 Image 2 shows the detection of images with a small number of winter jujubes and some winter jujubes obscured by leaves. Our model is the only one that can accurately detect all winter jujubes (7 in total); in images with a large number of winter jujubes obscured by leaves and / or other winter jujubes (such as Figure 5 Image 3), all models have missed some winter jujubes, with our model performing the best, missing 2 out of 25 winter jujubes; in images with weaker environmental light and more winter jujubes obscured by leaves (such as Figure 5 Image 4), all models cannot accurately detect all winter jujubes (16 in total), with our model performing the best, missing only 1 winter jujube; in terms of detection accuracy, our model has a detection accuracy of 93.33% in the above images, which is the highest among all models.

[0120] S7, quantize the weight file of the improved Yolov5n model using float16 quantization method, which includes the following steps:

[0121] S71, the quantization method is based on tensorflow under python environment, the version of tensorflow is 2.11.0, and the weight file of the improved detection model is imported into export.py.

[0122] S72, set batchsize to 1 in export.py, the image output size to 320x320, and the model file output format to.tflite.

[0123] The float16 quantization method quantizes the weight and activation function of the convolutional neural network from 32-bit floating point to 16-bit floating point; the model quantization process is as follows Figure 6 The resulting model file after quantization is best5n-fp16.tflite, with a size of 3.64MB.

[0124] S8, the model is deployed on the Android system to obtain the jujube detection and counting app, specifically including the following steps:

[0125] S81, select Android Studio based on java environment to realize the Android system deployment of the model, set the experimental environment to Android Studio 2021.3.1 under Windows 11 system, Android SDK 29.

[0126] S82, import the quantized best5n-fp16.tflite model file into the assets under the yolov5-tflite-android-master file, set INPNUT_SIZE=(320, 320), OUTPUT_SIZE={1, 6300, 6}, MODEL=

[0127] “best5n-fp16.tflite”.

[0128] S83, modify the original code of FullimageAnalyse and FullScreenAnalyse file part to realize the function of counting the detected jujubes, wherein the original code part to be modified includes:

[0129]

[0130] The modified code part is:

[0131]

[0132]

[0133] S84, use the Build apk function of Android Studio to export the apk file of the model file, and obtain the jujube detection and counting app after downloading and installing the apk file on the Android mobile phone.

[0134] The app occupies less memory and has lower requirements for mobile phone performance, and can be installed and used on most Android mobile phones. This jujube detection and counting app can call the mobile phone camera to detect and count jujubes in real time. The app visual interface provides functions including model selection, switching full screen model, recording the number of identified jujubes, recording the current picture inference time, etc. Real-time detection and counting of jujubes are realized, which provides a theoretical basis for jujube yield estimation; the Android system deployment and detection flowchart is as shown in Figure 7 .

[0135] S9, the winter jujube image is used to test the winter jujube detection and counting app, and the accuracy and counting accuracy of the test results are evaluated, which includes the following steps:

[0136] The specific experimental environment of the mobile terminal in this study is shown in Table 4.

[0137] Table 4 Mobile terminal experimental environment

[0138]

[0139] The app is used to detect and count winter jujubes in natural environment, and the app interface provides model selection, full screen switching, winter jujube counting, and display inference time functions.

[0140] The performance of the app winter jujube detection and counting is evaluated, and the evaluation indicators include precision, inference time, and detection accuracy; the winter jujube detection effect is shown in the following Figure 8

[0141] Figure 8 In (a), 3 winter jujubes were detected, the winter jujube detection accuracy was more than 90%, the winter jujube count was 3, the detection accuracy was 100%, no missed detection occurred, and the inference time was 35ms, which met the requirements of real-time detection and counting of winter jujubes; Figure 8 In (b), 6 winter jujubes were detected, the winter jujube detection accuracy was more than 85%, the winter jujube count was 6, the detection accuracy was 100%, no missed detection occurred, and the inference time was 57ms, which met the requirements of real-time detection and counting of winter jujubes; Figure 8 In (c), 18 winter jujubes were detected, the winter jujube count was 17, the detection accuracy was 94.44%, and 1 winter jujube was missed, which was caused by insufficient environmental light, resulting in a dark image, and the undetected winter jujube was similar in color to the leaves, the model did not extract enough information, resulting in lower detection accuracy for some winter jujubes. Due to the large number of detected winter jujubes, the inference time was 89ms, which met the requirements of real-time detection and counting of winter jujubes.

[0142] In summary, the improved model improves the detection ability of small target fruits and maintains high detection accuracy, and has good performance in winter jujube target detection and is suitable for other jujube and other small target detection practical applications; in the deployment of the model in the Android system, the apk is generated through Android Studio, and the app is installed and used on the Android phone, realizing the detection and counting of winter jujubes on the mobile terminal; this method meets the requirements of real-time detection and counting of winter jujubes, and provides theoretical basis and technical support for visual yield estimation of winter jujubes.

[0143] ​The above merely describes the preferred embodiments of the present application, but not for limiting the protection scope of the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A winter jujube detection and counting method based on improved YOLOv5n deployed in an Android system, characterized in that, The method comprises the following steps: S1, constructing an image recognition network model based on an improved Yolov5n model, replacing the Conv modules of the 1st, 3rd, 5th and 7th layers in the Backbone of the original Yolov5n and the Conv modules of the 18th and 21st layers in the Neck of the original Yolov5n with SPD-Conv modules; S2, constructing an image recognition network model based on an improved Yolov5n model, replacing the Upsample modules of the 11th and 15th layers in the Neck of the original Yolov5n with a CARAFE upsampling module, and replacing the Conv modules of the 18th and 21st layers in the Neck of the original Yolov5n improved by the SPD-Conv module with GSConv modules; S3, obtaining a winter jujube picture data set, and dividing the data set into a training set and a test set; S4, respectively expanding the training set and the test set through image enhancement processing; S5, training the improved Yolov5n model using the expanded data set to obtain a trained image recognition network model; S6, testing the optimized image recognition network model using the test data set, and evaluating the indicators of the model test results; S7, quantizing the weight file of the improved Yolov5n model in a float16 quantization manner; S8, deploying the model to a device running an Android system to obtain a winter jujube detection and counting app; S9, testing the winter jujube detection and counting app using winter jujube images, and evaluating the accuracy and counting accuracy of the test results.

2. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, In the network model constructed in step S1, the Conv modules of the 1st, 3rd, 5th and 7th layers in the Backbone of the original Yolov5n and the Conv modules of the 18th and 21st layers in the Neck of the original Yolov5n are replaced with SPD-Conv modules, which specifically comprises the following steps: S11, adding six SPD-Conv modules to the corresponding six Conv modules, and the SPD module is used to downsample the input feature map; S12, adding the module name to the import list, and adding the used module name in the subsequent module calling code.

3. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, In the network model constructed in step S2, the Upsample modules of the 11th and 15th layers in the Neck of the original Yolov5n are replaced with CARAFE modules, and the Conv modules of the 18th and 21st layers in the Neck of the original Yolov5n improved by the SPD-Conv module are replaced with GSConv modules, which specifically comprises the following steps: S21, replacing the Upsample modules of the Neck of the original Yolov5n with CARAFE modules; S22, replacing the two Conv modules of the Neck of the original Yolov5n improved by the SPD-Conv module with GSConv modules; S23, adding the module name to the import list, and adding the used module name in the subsequent module calling code.

4. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, The method for obtaining the winter jujube image dataset in step S3 specifically comprises the following steps: S31, the sample image collection site is selected from the winter jujube experimental demonstration station of Northwest A&F University in Dali County, Weinan City, Shaanxi Province, the winter jujube planting row spacing is 4 m, and the row spacing is 2 m; S32, in the image collection process, images under different light conditions, different fruit quantities and different leaf shielding conditions are collected; S33, Opencv is used to pre-process the winter jujube image, the resolution of the processed image is 640*480, and black edges are automatically filled; S34, the processed winter jujube sample dataset is divided into a training set and a verification set, and the proportion is 9:

1.

5. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, The data augmentation method shown in step S4 includes but is not limited to using Open cv to perform data augmentation on images through image processing methods such as mirroring, adding noise, adjusting brightness, flipping, translation, random erasing and the like based on a python environment.

6. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, The model evaluation indexes of step S6 include precision, recall, mAP (mean average precision), RMSE (root mean square error), MAPE (mean absolute percentage error), parameter quantity, model size and inference time, wherein the definitions of precision, recall, average precision, RMSE and MAPE are shown in formulas 1-5: Wherein, TP represents the number of true positive samples, FP represents the number of false positive samples, FN represents the number of false negative samples, P(i) represents the accuracy when the i-th picture is identified, ΔR(i) represents the change of recall rate when the number of identified pictures is from i-1 to i, C represents the number of detection classes, N represents the total number of image samples, y i represents the true value of the number of winter jujubes in each image, represents the predicted value of the number of winter jujubes in each image under the current model; Precision, recall and mAP are important indexes for evaluating the model, the higher the precision and recall, the higher the accuracy of the model; the larger the mAP value, the better the comprehensive performance of the model; the smaller the RMSE and MAPE, the higher the accuracy of the model in counting winter jujubes; when the RMSE and MAPE are 0, the model is a perfect model; the inference time represents the calculation performance of the detection model, the smaller the inference time, the better the real-time performance of the detection model.

7. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, Step S7 selects a float16 quantization method to quantize the weight file of the trained YOLOv5n model, which specifically comprises the following steps: S71, the weight file of the improved detection model is imported into export.py; S72, set batchsize to 1 in export.py, the image output size to 320*320, and the model file output format to.tflite.

8. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, Step S8 deploys the model on an Android system, which specifically comprises the following steps: S81, import the quantized best5n-fp16.tflite model file into yolov5-tflite-android-master and set the input and output parameters; S82, modify part of the original code of FullimageAnalyse and FullScreenAnalyse files to realize counting of winter jujube fruits; S83, use the Build apk function of Android Studio to export the apk file of the model file, and download and install it on an Android mobile phone to obtain a winter jujube detection and counting app.

9. The improved YOLOv5n-based jujube detection and counting method deployed in an Android system according to claim 1, characterized in that, The app is used in step S9 to detect and count the winter jujubes in the natural environment. The app interface provides functions such as model selection, switching to full screen, winter jujube counting, and display of inference time. The accuracy and counting precision of the test results are evaluated. The evaluation indexes include precision, inference time, and detection accuracy. The definition of detection accuracy is shown in formula 6:

Images