Lightweight fruit maturity detection method and system suitable for embedded equipment

Abstract

The invention discloses a lightweight fruit maturity detection method and system suitable for embedded equipment, and the method comprises the steps: collecting a multi-scene fruit plantation video, and obtaining a plurality of fruit images and a crop fruit image data set; the method comprises the following steps: performing improvement based on a YOLOv81 model, taking a CSPGhostNet network as a backbone network, taking a feature fusion network PANet as a neck network, and setting a classifier and a regression device as a head network so as to establish a fruit detection model; taking the fruit image data set as input, performing multi-layer network channel pruning on the fruit detection model based on a model pruning method, and obtaining a compressed fruit detection model; performing fine tuning training on the fruit detection model and the compressed fruit detection model by using a knowledge distillation method to obtain a lightweight fruit detection model; and detecting the fruit maturity by using the light-weight fruit detection model. According to the invention, rapid and accurate detection of fruit maturity can be realized.

Description

Technical Field

[0001] The invention relates to the technical field of machine vision, and in particular to a lightweight fruit maturity detection method and system suitable for embedded devices. Background Art

[0002] Blueberry fruit is favored by consumers for its rich nutritional value and unique taste. The maturity of blueberries directly affects its taste, nutritional value and market value. Blueberries usually grow in clusters, and their ripening process is carried out in stages. In each cluster, three different ripening states can be observed: fully ripe fruits, partially ripe fruits, and unripe fruits. This diversity of maturity provides flexibility for picking, allowing selective harvesting according to maturity at different time points. Therefore, rapid and accurate detection of the maturity of blueberry fruits is crucial for the picking, sorting, packaging and sales of blueberries. Traditional blueberry maturity detection methods mainly rely on manual visual judgment, which is inefficient, subjective, and difficult to meet the needs of large-scale production. With the development of computer vision technology, machine vision detection methods have gradually been applied to blueberry maturity detection, but the existing technology still has shortcomings in accuracy, speed and adaptability.

[0003] In recent years, deep learning technology has made significant progress in the field of image recognition and object detection. As a representative of real-time object detection frameworks, the YOLO (You Only Look Once) series of algorithms have attracted widespread attention for their fast and efficient performance. However, the YOLO series of algorithms still face challenges in dealing with small target detection, complex background and blueberry fruit maturity detection under changing illumination. As an updated version of the YOLO series of algorithms, YOLOv8 has improved the speed and accuracy of target detection, but still needs to be further optimized in blueberry fruit maturity detection. YOLOv8 has made significant progress in the speed and accuracy of target detection, but when applied on embedded devices, it still faces challenges such as small target detection, complex background and changing illumination, which require further optimization of the algorithm to reduce model complexity and the number of parameters. Faced with the dense arrangement, color gradient and occlusion problems of blueberry fruit detection in natural environments, the algorithm needs not only accuracy but also robustness, which may involve specific data preprocessing, model adjustment or the use of reinforcement learning strategies to deal with it.

[0004] Existing methods for detecting blueberry fruit maturity have the following common disadvantages:

[0005] 1. Traditional machine learning methods are efficient and fast in extracting low-dimensional image features. However, when faced with complex high-dimensional features, it takes a lot of time to manually extract the corresponding features, which is not conducive to large-scale detection of blueberry fruit maturity.

[0006] 2. Object detection methods based on deep learning are updated quickly and have high detection accuracy, but most of them are optimized for general data sets. They need to be retrained in blueberry detection data, build an efficient feature extraction network based on blueberry category characteristics, and verify the feasibility of the model in plantations;

[0007] 3. The processors of embedded devices usually have low processing power and memory limitations and cannot run computationally intensive models. It is necessary to reduce the parameters and computational requirements by lightweight models so that they can run efficiently on these devices;

[0008] 4. Lightweight models need to find a balance between reducing model size and maintaining detection accuracy. Over-reliance on lightweight architecture design will lead to a significant drop in accuracy and make it difficult to meet the requirements of specific applications.

[0009] In view of the limitations of the existing YOLOv8 algorithm in blueberry maturity detection, it is necessary to improve the algorithm to improve the detection accuracy and speed of blueberry fruits of different maturity and provide effective support for embedded device deployment.

[0010] In the prior art, Tian Youwen et al. published a paper titled "Blueberry Maturity Detection in Complex Field Environments Based on Improved YOLOv8". In order to effectively reduce the number of parameters and improve the running speed of the model, the paper uses MobileNetV3 as the backbone feature extraction network to extract feature information, which is beneficial for detection in complex field environments. In the prior art, in order to improve the running speed of the model, only one module in the network design is lightweight. Summary of the invention

[0011] The technical problem to be solved by the present invention is to provide a lightweight fruit maturity detection method suitable for embedded devices.

[0012] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0013] A lightweight fruit maturity detection method suitable for embedded devices, comprising:

[0014] Collect multi-scene fruit plantation videos, obtain multiple fruit images, and crop fruit image datasets;

[0015] Based on the YOLOv8l model, the CSPGhostNet network is used as the backbone network, the feature fusion network PANet is used as the neck network, and the classifier and regressor are set as the head network to establish a fruit detection model;

[0016] Taking the fruit image dataset as input, the multi-layer network channel pruning of the fruit detection model is performed based on the model pruning method to obtain the compressed fruit detection model;

[0017] Use knowledge distillation to fine-tune the fruit detection model and compressed fruit detection model to obtain a lightweight fruit detection model.

[0018] A lightweight fruit detection model is used to detect fruit maturity.

[0019] Advantages: First, blueberry data samples are collected to construct a blueberry maturity dataset; then, the deep learning-based target detection algorithm YOLOv8 is improved to have higher accuracy and faster detection speed in the blueberry dataset detection task; then, the model lightweight technology is used to prune the network of the improved model, and the pruned model is fine-tuned and trained using the knowledge distillation method to ensure the detection accuracy while increasing the detection speed as much as possible, so as to facilitate deployment on an embedded platform for real-time calculation; finally, by detecting the distribution of blueberry maturity in the image, the blueberry maturity calculation formula is defined, and the maturity status of blueberries is accurately judged, providing a reliable basis for the establishment of the subsequent blueberry automatic picking system.

[0020] In general, the idea of ​​this patent invention is to use target detection, model lightweighting and other technologies to detect and analyze blueberry images, calculate the maturity of blueberries, improve the automation of blueberry picking processes, reduce labor costs, and improve the economic benefits of the blueberry industry. This idea helps to improve the automation of blueberry plantations, improve the stability, reliability and efficiency of blueberry picking, and reduce quality risks and maintenance costs, and has certain practical application potential.

[0021] In one embodiment of the present invention, the CSPGhostNet network includes a convolution module Conv, a first Ghost module, three first composite residual blocks G2f, a second Ghost module, six second composite residual blocks G2f, a third Ghost module, six third composite residual blocks G2f, a fourth Ghost module, three third composite residual blocks G2f and a fast spatial pyramid pooling SPPF, which are arranged in sequence to reduce the number of parameters and computational complexity of the fruit detection model.

[0022] In one embodiment of the present invention, the first Ghost module, the second Ghost module, the third Ghost module and the fourth Ghost module have the same structure, and all include a standard convolution layer, a linear transformation layer and a feature map splicing layer;

[0023] The fruit images in the fruit image dataset are used as feature maps. Assume that the input feature map F i The size is H×W×c, and the size of the output feature map F1 after the standard convolution layer is H o ×W o ×m, the channels in the feature map F1 are linearly transformed using the linear transformation layer Φi , output feature map F c , multiple identically transformed feature maps F c Input to the feature map concatenation layer and input feature map F i Splice and output feature map F o Size H o ×W o ×u; where H, H o It is represented by the length of the feature map at different layers in the Ghost module, W, W o represents the width of the feature map at different layers in the Ghost module, c, m, u represent the number of pixels of the feature map at different layers in the Ghost module; and H o <H;W o <W;u<m<c。

[0024] In one embodiment of the present invention, the structures of three first composite residual blocks G2f, six second composite residual blocks G2f, six third composite residual blocks G2f and three third composite residual blocks G2f are the same, and each composite residual block G2f includes two Ghost modules with a step size of 1, and the activation function is LeakyReLU; the first 1x1 Ghost module divides the output into two parts, and each channel remains consistent. The latest output is passed to the corresponding number of residual modules through expansion iteration, and finally all the outputs are stacked together, and the number of output channels is adjusted through another 1x1 Ghost module.

[0025] In one embodiment of the present invention, the fast spatial pyramid pooling SPPF includes a bottom feature layer feat3, a lower feature layer feat2, and a middle feature layer feat1;

[0026] The bottom feature layer feat3 has a feature map size of 20×20×1024;

[0027] The feature fusion network PANetde upsamples the underlying feature layer feat3, and the feature map size becomes 40×40×512;

[0028] Then it is connected with the lower feature layer feat2, and the combined features are processed by the composite residual block G2f to obtain the new feature P4, and the feature map is kept at 40×40×512;

[0029] Feature P4 is upsampled again, and the feature map size becomes 80×80×512. It is fused with the middle feature layer feat1. The fused features are processed by the composite residual block G2f to obtain feature P3, and the feature map size is 80×80×256.

[0030] Feature P3 is downsampled through a 3x3 convolution, and the feature map size becomes 40×40×256. It is then stacked with feature P4 and processed again through the composite residual block G2f to obtain the updated feature P4, with the size restored to 40×40×512.

[0031] Finally, the updated feature P4 is downsampled by 3x3 convolution to a size of 20×20×512, stacked with the feature P5 of the underlying feature layer feat3, and processed by the G2f module to obtain the updated P5 with a size of 20×20×1024.

[0032] In one embodiment of the present invention, the loss function of the fruit detection model includes: the classification loss adopts the VFL Loss loss function, and the regression loss introduces the Alpha-IoU; wherein the regression loss is obtained by the following formula:

[0033]

[0034] Where, L α-CIoU is represented as the regression loss, Y is represented as the intersection-over-union ratio between the predicted box and the true box, b and b gt They are respectively represented as the coordinates of the center points of the predicted box and the true box, C is represented as the diagonal length of the circumscribed rectangle, ρ is represented as the distance between the center points of the predicted box and the true box, β is represented as the trade-off parameter, v is represented as the metric parameter, and α is represented as a power parameter greater than 0.

[0035] In one embodiment of the present invention, obtaining a compressed fruit detection model comprises the following steps:

[0036] S31, loading the fruit detection model and the fruit image dataset, and setting the number of iterations and the precision value;

[0037] S32, determining the global pruning threshold, and sorting the global thresholds according to size;

[0038] S33, determining the pruning channel index;

[0039] S33, judging whether pruning is required according to the channel index;

[0040] S34, if yes, prune the network channel, if no, keep the channel and assign values ​​directly;

[0041] S35, repeating steps S33 and S34 to determine whether the maximum number of iterations is reached or whether the accuracy meets the requirements;

[0042] S36, if yes, then output and obtain the compressed fruit detection model, if no, then return to step S33.

[0043] In one embodiment of the present invention, a scale factor S is introduced into the BN layer of the convolution module Conv and L1 regularization is added. The scale factor S enters the channel in the composite residual block G2f to obtain a sparse scale factor; wherein the scale factor S is multiplied by each channel and output; then the loss function calculation formula for the training of the compressed fruit detection model is:

[0044]

[0045] Where L is the total loss function of the compressed fruit detection model, (x, y) is the input and output of the training, f(x, W) is the prediction function, which accepts the input data x and the training weight W and outputs the prediction result, λ is the regularization coefficient, g(s) is the regularization function, S is a function of the training weight W, Γ is the set of all possible regularization terms, and l is the loss function.

[0046] In one embodiment of the present invention, the lightweight fruit detection model is obtained by:

[0047] The knowledge distillation method includes a student network and a teacher network;

[0048] The weight file of the pruned compressed fruit detection model is placed in the corresponding path of the student network, and the weight file of the unpruned fruit detection model is placed in the corresponding path of the teacher network, and the target ratio is set for training; when the accuracy reaches the target ratio, the lightweight fruit detection model is obtained.

[0049] The present invention also provides a lightweight fruit maturity detection system applicable to embedded devices, which executes the lightweight fruit maturity detection method applicable to embedded devices as described above, including:

[0050] The data module is used to collect videos of fruit plantations in multiple scenes, obtain multiple fruit images, crop fruit image datasets, and label multiple fruit images into mature labels and immature labels;

[0051] The low-parameter model module is used to improve the YOLOv8l model, using the CSPGhostNet network as the backbone network, the feature fusion network PANet as the neck network, and the classifier and regressor as the head network to build a fruit detection model;

[0052] A simplified model module is used to take the fruit image dataset as input, perform multi-layer network channel pruning on the fruit detection model based on the model pruning method, and obtain a compressed fruit detection model;

[0053] The optimization model module is used to fine-tune the fruit detection model and the compressed fruit detection model using the knowledge distillation method to obtain a lightweight fruit detection model;

[0054] The detection module is used to detect the maturity of fruits using a lightweight fruit detection model.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention constructs a CSPGhostNet backbone network based on the Ghost module, which effectively reduces the number of parameters and computational complexity of the model. Secondly, the Alpha-IoU loss function is introduced to guide the regression of the target box, which not only speeds up the convergence speed of the model, but also improves the detection accuracy. Finally, through the network pruning technology, the model is further streamlined, redundant parameters are removed, and then the knowledge distillation algorithm is used to fine-tune the pruned model, thereby optimizing the efficiency and performance of the model, which is conducive to real-time computing on embedded devices. Combined with the blueberry maturity calculation formula, the maturity of blueberries can be accurately judged. Through the above algorithm optimization and technological innovation, the rapid and accurate detection of the maturity of blueberry fruits can be achieved to meet the needs of modern agricultural automation and intelligence.

[0056] Lightweight models need to find a balance between reducing model size and maintaining detection accuracy. Over-reliance on lightweight architecture design will lead to a significant drop in accuracy and make it difficult to meet the requirements of specific applications. Therefore, based on lightweight architecture design, parameter pruning and knowledge distillation are used to optimize specific tasks to achieve the best lightweight effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The present invention is a flowchart of a lightweight fruit maturity detection method applicable to an embedded device according to an embodiment of the present invention.

[0058] Figure 2 Schematic diagram of a fruit maturity calculation framework according to an embodiment of the present invention.

[0059] Figure 3 Schematic diagram of a Ghost module according to an embodiment of the present invention.

[0060] Figure 4 This is a network structure diagram of the fruit detection model according to an embodiment of the present invention.

[0061] Figure 5 Schematic diagram of the network structure of the composite residual block G2f according to an embodiment of the present invention.

[0062] Figure 6 This is a channel pruning graph of an embodiment of the present invention.

[0063] Figure 7 This is a target ratio distillation framework diagram of an embodiment of the present invention.

[0064] Figure 8 This is a graph showing the blueberry maturity detection results according to an embodiment of the present invention.

[0065] Fig. 9 The present invention is a block diagram of a lightweight fruit maturity detection system applicable to an embedded device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] To facilitate those skilled in the art to understand the technical solution of the present invention, the technical solution of the present invention is further described in conjunction with the accompanying drawings.

[0067] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0068] See also Figure 1 and Figure 2 As shown, the present invention provides a lightweight fruit maturity detection method applicable to embedded devices, comprising:

[0069] S10, collect multi-scene fruit plantation videos, obtain multiple fruit images, crop fruit image datasets, and label the multiple fruit images into mature labels and immature labels.

[0070] In this embodiment, blueberry is used as an example. Multi-scene blueberry plantation videos are collected, and the videos are split frame by frame to obtain 2,000 image data of the original blueberry plants. The data set is expanded to 20,000 images by adding noise, rotating, translating, adjusting contrast and other data enhancement methods as a fruit image data set. The blueberry image data set is annotated using labellmg image annotation software, and the fruit image data set is randomly divided into a training set, a test set and a validation set in a ratio of 8:1:1, with the immature label being 0 and the mature label being 1.

[0071] S20, based on the improvement of YOLOv8l model, uses the CSPGhostNet network as the backbone network, the feature fusion network PANet as the neck network, and sets the classifier and regressor as the head network to establish a fruit detection model.

[0072] See also Figures 1 to 5 As shown, in one embodiment of the present invention, the YOLOv8l model includes a backbone network, a neck network and a head network. The present invention is improved on the basis of the YOLOv8l model, using the CSPGhostNet network as the backbone network, the feature fusion network PANet as the neck network, and setting the classifier and the regressor as the head network to establish a fruit detection model, aiming to solve the problems of large amount of calculation and many weight parameters of traditional networks.

[0073] In this embodiment, the CSPGhostNet network includes a convolution module Conv, a first Ghost module, three first composite residual blocks G2f, a second Ghost module, six second composite residual blocks G2f, a third Ghost module, six third composite residual blocks G2f, a fourth Ghost module, three third composite residual blocks G2f and fast spatial pyramid pooling SPPF, which are arranged in sequence to reduce the number of parameters and computational complexity of the fruit detection model.

[0074] The first Ghost module, the second Ghost module, the third Ghost module and the fourth Ghost module have the same structure, and all include a standard convolution layer, a linear transformation layer and a feature map splicing layer.

[0075] The fruit images in the fruit image dataset are used as feature maps. Assume that the input feature map F i The size is H×W×c, and the size of the output feature map F1 after the standard convolution layer is H o ×W o ×m, the channels in the feature map F1 are linearly transformed using the linear transformation layer Φ i , output feature map F c , multiple identically transformed feature maps F c Input to the feature map concatenation layer and input feature map F i Splice and output feature map F o Size H o ×W o ×u, the convolution kernel size is k×k; where H, H o It is represented by the length of the feature map at different layers in the Ghost module, W, W o represents the width of the feature map at different layers in the Ghost module, c, m, u represent the number of pixels of the feature map at different layers in the Ghost module; and H o <H;W o <W;u<m<c。

[0076] The CSPGhostNet network adopts the Ghost module, which first generates a certain number of feature maps through a small number of standard convolutions, then processes these feature maps through linear operations to generate a set of similar new feature maps, and finally merges the original and newly generated feature maps to improve network performance.

[0077] Since the Ghost module has an identical transformation in the transformation process, the actual number of transformations is s-1. cAfter the identity transformation, the original feature map is concatenated to obtain the final output feature map. The feature map sizes output by the standard convolutional layer and the Ghost module are compared, and the comparison formula is as follows:

[0078]

[0079] u = m × s;

[0080] In the formula, r s It is represented as the scaling coefficient, k is represented as the size of the convolution kernel in the feature map concatenation layer, d is represented as the convolution kernel size of the feature map of the identity transformation, which is approximately equal to k, and s is represented as the transformation coefficient.

[0081] Get r s The numerator of the formula is the amount of calculation of the standard convolution, and the denominator is the amount of calculation of the Ghost module. The calculation amounts of the two are compared under the same output premise. The approximate calculation shows that the transformation coefficient is s, then the amount of calculation of the standard convolution is s times the amount of calculation of the Ghost module.

[0082] Compared with standard convolution, the computational complexity of the Ghost module is greatly reduced, which is about 1 / s of the computational complexity of standard convolution. It uses simple linear transformation to generate similar feature maps, which not only reduces the redundant information generated by the convolution calculation process, but also ensures network performance.

[0083] In this embodiment, the model used for blueberry detection is based on the improved YOLOv8l target detection model, the backbone network is the lightweight network CSPGhostNet, the feature fusion network is PANet, the Ghost module is introduced into the residual block structure in the original CSPDarkNet, and the sparse distribution of the residual connection is optimized to improve the significance of target detection in complex environments. Then, the target situation corresponding to the prior frame is predicted by the Head classifier and the regressor, and the optimal detection frame is screened out by non-maximum suppression to obtain the final fruit detection model for blueberry detection.

[0084] In this embodiment, the specific network structure parameters of the lightweight network CSPGhostNet are shown in Table 1.

[0085] Table 1 CSPGhostNet structure

[0086]

[0087]

[0088] In this embodiment, the structures of the three first composite residual blocks G2f, the six second composite residual blocks G2f, the six third composite residual blocks G2f and the three third composite residual blocks G2f are the same, and each composite residual block G2f includes two Ghost modules with a step size of 1, and the activation function is LeakyReLU. The first 1x1 Ghost module divides the output into two parts, and each channel remains consistent. The latest output is passed to the corresponding number of residual modules through extended iterations, and finally all the outputs are stacked together, and the number of output channels is adjusted through another 1x1 Ghost module.

[0089] Fast Spatial Pyramid Pooling SPPF extracts multiple feature layers for object detection. In the backbone of the convolutional neural network, the feature layer experiences a decrease in size and an increase in the number of channels as the level decreases. Specifically, when the input image size is 640x640 pixels with 3 color channels, the feature layers of three different levels have the following sizes and numbers of channels. Specifically, Fast Spatial Pyramid Pooling SPPF includes the bottom feature layer feat3, the lower feature layer feat2, and the middle feature layer feat1.

[0090] in:

[0091] The feature map size of the middle feature layer feat1 is 80x80 pixels and has 256 channels;

[0092] The feature map size of the lower feature layer feat2 is 40x40 pixels and has 512 channels;

[0093] The feature map size of the bottom feature layer feat3 is 20x20 pixels with 1024 channels.

[0094] This design allows the network to capture features of different scales at different levels, thereby improving the model's ability to understand and process input data.

[0095] In this embodiment, after obtaining the above three effective feature layers, the feature fusion network PANetde fuses the feature layers of different scales, which is conducive to extracting better features. The underlying feature layer feat3 has a feature map size of 20×20×1024; the feature fusion network PANetde upsamples the underlying feature layer feat3, and the feature map size becomes 40×40×512;

[0096] Then it is connected with the lower feature layer feat2, and the combined features are processed by the composite residual block G2f to obtain the new feature P4, and the feature map is kept at 40×40×512;

[0097] Feature P4 is upsampled again, and the feature map size becomes 80×80×512. It is fused with the middle feature layer feat1. The fused features are processed by the composite residual block G2f to obtain feature P3, and the feature map size is 80×80×256.

[0098] Feature P3 is downsampled through a 3x3 convolution, and the feature map size becomes 40×40×256. It is then stacked with feature P4 and processed again through the composite residual block G2f to obtain the updated feature P4, with the size restored to 40×40×512.

[0099] Finally, the updated feature P4 is downsampled by 3x3 convolution to a size of 20×20×512, stacked with the feature P5 of the underlying feature layer feat3, and processed by the G2f module to obtain the updated P5 with a size of 20×20×1024.

[0100] In this embodiment, the classifier and regressor outputs in the head network contain four key parameters, which together define the prediction box corresponding to each feature point. The first two parameters specify the horizontal and vertical distances of the upper left corner of the prediction box relative to the feature point, and the last two parameters determine the corresponding offsets of the lower right corner of the prediction box. By adjusting these regression parameters, the position of the prediction box can be accurately located. At the same time, the category output consists of 2 parameters, which are used to identify and classify the target objects contained in each feature point. Among them, the classifier and regressor are not modified in this application.

[0101] In this embodiment, the loss function of the traditional YOLOv8 model is composed of multiple components, including VFL Loss for classification loss and CIoU Loss for regression loss. In addition, the traditional YOLOv8 model also uses DFL to obtain the regression value through probability calculation. Although CIoU Loss performs well in describing the relative distance of the aspect ratio of the object, it does not fully consider the sample balance problem. To solve this problem, the fruit detection model introduces Alpha-IoU to replace CIoU as the regression loss function. This improvement not only improves the loss value of the target confidence, speeds up the calculation of the gradient, but also makes the model more accurate in predicting abnormal objects. Specifically, the regression loss is obtained by the following formula:

[0102]

[0103] Where, L α-CIoU is represented as the regression loss, Y is represented as the intersection-over-union ratio between the predicted box and the true box, b and b gtThey are respectively represented as the center point coordinates of the predicted box and the true box, C is represented as the diagonal length of the circumscribed rectangle, or the diagonal length of the minimum closure area containing the predicted box and the true box, ρ is represented as the distance between the center points of the predicted box and the true box, β is represented as a trade-off parameter, which is used to balance the influence of the aspect ratio similarity term v in the loss function, v is represented as a metric parameter, which is used to measure the similarity of the aspect ratio of the predicted box and the true box, and α is represented as a power parameter greater than 0, which is used to adjust the sensitivity of the loss function. In this embodiment, α=3 can achieve better results. These parameters work together to enable the Alpha-CIoU loss function to more comprehensively consider factors such as the degree of overlap between the predicted box and the true box, the center point distance, and the aspect ratio similarity, thereby guiding the model to more accurately predict the bounding box during training.

[0104] In this embodiment, non-maximum suppression filters out high-quality prediction results. First, all prediction boxes with scores higher than the set confidence threshold are filtered out. This step can significantly reduce the number of prediction boxes that need further processing. Next, non-maximum suppression is performed on each category separately, that is, the prediction box with the highest score is found and retained in each category. The prediction boxes of each category are sorted in descending order according to the score to ensure that the prediction box with the highest score is processed first. Then, starting with the prediction box with the highest score, calculate its overlap with all other prediction boxes (usually measured by calculating the intersection over union (IoU)). If it is found that the overlap exceeds the preset threshold, it is considered that these prediction boxes are detecting the same object. At this time, the prediction boxes with lower scores will be eliminated and only the one with the highest score will be retained.

[0105] S30, taking the fruit image dataset as input, performing multi-layer network channel pruning on the fruit detection model based on the model pruning method, and obtaining a compressed fruit detection model;

[0106] See also Figure 1 and Figure 6 As shown, in one embodiment of the present invention, in order to enable the fruit detection model to be calculated in real time on an embedded platform, the multi-layer network channel is pruned based on the Network Slimming method to achieve model compression. Obtaining the compressed fruit detection model includes the following steps:

[0107] S31, loading the fruit detection model and the fruit image dataset, and setting the number of iterations and the precision value;

[0108] S32, determining the global pruning threshold, and sorting the global thresholds according to size;

[0109] S33, determining the pruning channel index;

[0110] S33, judging whether pruning is required according to the channel index;

[0111] S34, if yes, prune the network channel, if no, keep the channel and assign values ​​directly;

[0112] S35, repeating steps S33 and S34 to determine whether the maximum number of iterations is reached or whether the accuracy meets the requirements;

[0113] S36, if yes, then output and obtain the compressed fruit detection model, if no, then return to step S33.

[0114] In this embodiment, during network training, a scale factor S is introduced into the BN layer of each convolution module Conv channel and L1 regularization is added to obtain a sparse scale factor, where the scale factor S can be multiplied by each channel and output, and the training loss function calculation formula is:

[0115]

[0116] Where L is the total loss function of the compressed fruit detection model, which is used to evaluate the performance of the model on the entire training set. (x, y) is the input and output of the training, corresponding to the sample points in the training set and the corresponding true labels. f(x, W) is the prediction function, which accepts the input data x and the training weight W and outputs the prediction result. λ is the regularization coefficient, which is used to control the weight of the regularization term in the total loss. g(s) is the regularization function, which is used to penalize the complexity of the model, such as the L1 or L2 norm of the parameter. L1 regularization will cause the model weight to be sparse, while L2 regularization tends to make the weight value close to zero but not completely zero. S is a function of the training weight W, which is the absolute value of the weight (L1 regularization). Γ is the set of all possible regularization terms. l is the loss function, which is used to calculate the difference between the predicted value and the true value of a single sample. Common loss functions include mean square error (MSE), cross entropy loss, etc.

[0117] This loss function combines two parts: the first part is the sum of the model's prediction errors for all training samples, and the second part is the regularization term of the model complexity. By adjusting the value of λ, a trade-off can be made between the model's fitting performance and complexity to prevent overfitting.

[0118] Comparison of the network before and after channel pruning Figure 6 As shown in the figure, Network Slimming associates the scale factor with each channel in the convolutional layer during the pruning process, identifies the secondary channels through regularization, and uses the set threshold to prune the small scale factor channels to prune the initial network into a lightweight network.

[0119] This loss function combines two parts: the first part is the sum of the model's prediction errors for all training samples, and the second part is the regularization term of the model complexity. By adjusting the value of λ, a trade-off can be made between the model's fitting performance and complexity to prevent overfitting.

[0120] S40, fine-tuning the fruit detection model and the compressed fruit detection model using a knowledge distillation method to obtain a lightweight fruit detection model;

[0121] See also Figure 1 and Figure 7 As shown, in one embodiment of the present invention, the knowledge distillation method (Distilling theKnowledge in a Neural Network) adopts the Teacher-Student model: a complex and large model is used as a Teacher, and the Student model structure is relatively simple. The Teacher is used to assist the training of the Student model. The Teacher has strong learning ability and can transfer the knowledge it has learned to the Student model with relatively weak learning ability, thereby enhancing the generalization ability of the Student model.

[0122] In this embodiment, the lightweight fruit detection model is obtained by:

[0123] The knowledge distillation method includes a student network and a teacher network;

[0124] The weight file of the pruned compressed fruit detection model is placed in the corresponding path of the student network, and the weight file of the unpruned fruit detection model is placed in the corresponding path of the teacher network, and the target ratio is set for training; when the accuracy reaches the target ratio, the lightweight fruit detection model is obtained.

[0125] In this embodiment, in the knowledge distillation method, the teacher network is usually a pre-trained model with better performance, while the student network is a model with a simpler structure or fewer parameters. The goal is to let the student network learn the knowledge of the teacher network. In target ratio distillation, "target ratio" usually refers to the ratio between the performance indicators (such as accuracy, recall rate, F1 score, etc.) of the teacher model and the student model on a specific task. The core idea of ​​this method is that even if the student model cannot fully achieve the performance of the teacher model, the training process can be adjusted to ensure that the two maintain a certain proportional relationship in performance, thereby maintaining the key characteristics of the teacher model to a certain extent.

[0126] S50, using a lightweight fruit detection model to detect fruit maturity.

[0127] See also Figures 1 to 8As shown, in one embodiment of the present invention, a video image is input for calculation to obtain the feature layer prediction results calculated by the lightweight fruit detection model, corresponding to two outputs, and the three regression output layers have shapes of (20,20,4), (40,40,4), and (80,80,4), which are used to calculate the scale of the prediction box. The shapes of the three category output layers are (20,20,2), (40,40,2), and (80,80,2), which are used to calculate the target types contained in each feature point. The prediction results are processed using score screening and non-maximum suppression to filter out the best prediction box for the current detection object.

[0128] In this embodiment, the number of mature and immature detection frames is obtained respectively, and the maturity of blueberries in the current image is calculated:

[0129] Get the number of target boxes calculated by the lightweight fruit detection model;

[0130] The immature category is labeled 0, and the number of detection boxes is recorded as sum0;

[0131] The mature category is label 1, and the number of detection boxes is recorded as sum1;

[0132] The calculation of blueberry maturity is as follows:

[0133]

[0134] Where B is the degree of fruit maturity, sum0 is the number of unripe category boxes, and sum1 is the number of ripe category boxes.

[0135] The maturity of the blueberries in the current image can be determined based on the B value: less than 50% is unripe, 50% to 80% is partially ripe, and greater than 80% is ripe.

[0136] See also Figures 1 to 9 As shown, the present invention also provides a lightweight fruit maturity detection system applicable to embedded devices, which executes the lightweight fruit maturity detection method applicable to embedded devices as described above, including:

[0137] The data module is used to collect videos of fruit plantations in multiple scenes, obtain multiple fruit images, crop fruit image datasets, and label multiple fruit images into mature labels and immature labels;

[0138] The low-parameter model module is used to improve the YOLOv8l model, using the CSPGhostNet network as the backbone network, the feature fusion network PANet as the neck network, and the classifier and regressor as the head network to build a fruit detection model;

[0139] A simplified model module is used to take the fruit image dataset as input, perform multi-layer network channel pruning on the fruit detection model based on the model pruning method, and obtain a compressed fruit detection model;

[0140] The optimization model module is used to fine-tune the fruit detection model and the compressed fruit detection model using the knowledge distillation method to obtain a lightweight fruit detection model;

[0141] The detection module is used to detect the maturity of fruits using a lightweight fruit detection model.

[0142] In this embodiment, the system environment of the fruit detection model training platform is Windows 10, the hardware CPU is Inteli7-10875H CPU @ 2.30GHZ, the GPU is Nvidia GeForce GTX2060, the memory is dual-channel 16GB, and the programming framework is the Pytorch open source framework. The model inference platform is NVIDIA Jetson Xavier NX, the system environment is Ubuntu 18.04, the CPU is 6-core ARMv8.2 @ 1.4GHz, the GPU is 384-core Volta @ 1100Mhz, and the memory is single-channel 8GB. The experimental data is selected from an actual blueberry plantation, and 2000 blueberry fruit images are selected.

[0143] See also Figures 1 to 9 As shown, in one embodiment of the present invention, multi-layer network channel pruning is performed on the fruit detection model, and the unpruned fruit detection model is used as the basic model for comparative experiments. The experimental comparison data is shown in Table 1. By pruning the network channel of the fruit detection model, compressing the model and fine-tuning the training, the accuracy of the network is made as close as possible to the value before pruning. Among them, Pruning-1, Pruning-2, and Pruning-3 respectively represent pruning models with different pruning rates. Among them, YOLOv8l_GC is represented as a fruit detection model, and YOLOv8l is a traditional model.

[0144] Shows the comparison of 2 fruit detection models before and after pruning

[0145]

[0146] As shown in Table 2, YOLOv8l_GC improves mAP by 1.1% compared with YOLOv8l model, reduces parameters by 40.6%, GFLOPs by 50%, and weight file size by 39.9%; when the pruning rate is 50%, the pruning model Pruning-1 reduces mAP by 2.3%, reduces parameters by 40.9%, GFLOPs by 28.6%, and weight file size by 33.3% compared with YOLOv8l_GC model; when the pruning rate is 60%, the pruning model Pruning-2 reduces mAP by 3.1%, reduces parameters by 72.2%, GFLOPs by 74.1%, and weight file size by 67.1% compared with YOLOv8l_GC model. The space occupied by the blueberry detection model is greatly reduced, while also retaining a high accuracy value. When the pruning rate is 70%, the pruning model Pruning-3 reduces mAP by 7.9%, the number of parameters by 87.6%, GFLOPs by 88.1%, and the weight file size by 86.9% compared to the YOLOv8l_GC model. Although the Pruning-3 method achieves the lowest resource usage, the high parameter pruning rate results in a mAP of only 85.8%, which cannot meet the application requirements of blueberry recognition. Therefore, the Pruning-2 model with a pruning rate of 60% is selected as the best solution for subsequent research.

[0147] The compressed fruit detection model before and after distillation was deployed on the NVIDIA Jetson Zavier NX embedded platform for experiments. The experimental comparison data is shown in Table 3.

[0148] Table 3 Comparison of distillation of compressed fruit detection models

[0149]

[0150] In Table 3, YOLOv8l_GC is used as the teacher network, and the small network pruned by the Pruning-2 method is used as the student network. After distillation training, the student network has a mAP of 92.8%, a model weight file size of 16.5MB, and an FPS of 22.5 frames / s. Compared with YOLOv8l_GC, the mAP is reduced by 0.9%, but the FPS is increased by 153%. Finally, a speed of 22.5 frames / s is achieved on the NVIDIA Jetson Zavier NX embedded platform, indicating the effectiveness of the knowledge distillation algorithm based on the pruned model in model compression, ensuring the real-time performance of blueberry detection on embedded devices with limited computing power.

[0151] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0152] The above-described embodiments merely represent implementation methods of the invention. The protection scope of the present invention is not limited to the above-described embodiments. For those skilled in the art, several modifications and improvements may be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A lightweight fruit maturity detection method suitable for embedded devices, characterized in that: include: Collect videos of fruit plantations in multiple scenes, obtain multiple fruit images, and create a crop fruit image dataset; Based on the YOLOv8l model, the CSPGhostNet network is used as the backbone network, the feature fusion network PANet is used as the neck network, and the classifier and regressor are set as the head network to establish a fruit detection model; Taking the fruit image dataset as input, the multi-layer network channel pruning of the fruit detection model is performed based on the model pruning method to obtain the compressed fruit detection model; Use knowledge distillation to fine-tune the fruit detection model and compressed fruit detection model to obtain a lightweight fruit detection model. A lightweight fruit detection model is used to detect fruit maturity.

2. The lightweight fruit maturity detection method applicable to embedded devices according to claim 1, characterized in that: The CSPGhostNet network includes a convolution module Conv, a first Ghost module, three first composite residual blocks G2f, a second Ghost module, six second composite residual blocks G2f, a third Ghost module, six third composite residual blocks G2f, a fourth Ghost module, three third composite residual blocks G2f and fast spatial pyramid pooling SPPF, which are arranged in sequence to reduce the number of parameters and computational complexity of the fruit detection model.

3. The lightweight fruit maturity detection method applicable to embedded devices according to claim 2, characterized in that: The first Ghost module, the second Ghost module, the third Ghost module and the fourth Ghost module have the same structure, and all include a standard convolution layer, a linear transformation layer and a feature map concatenation layer; The fruit images in the fruit image dataset are used as feature maps. Assume that the input feature map F i The size is H×W×c, and the size of the output feature map F1 after the standard convolution layer is H o ×W o ×m, the channels in the feature map F1 are linearly transformed using the linear transformation layer Φ i , output feature map F c , multiple identically transformed feature maps F c Input to the feature map concatenation layer and input feature map F i Splice and output feature map F o Size H o ×W o ×u; where H, H o It is represented by the length of the feature map at different layers in the Ghost module, W, W o represents the width of the feature map at different layers in the Ghost module, c, m, u represent the number of pixels of the feature map at different layers in the Ghost module; and H o <H;W o <W;u<m<c。 4. The lightweight fruit maturity detection method applicable to embedded devices according to claim 3, characterized in that: The structures of the three first composite residual blocks G2f, the six second composite residual blocks G2f, the six third composite residual blocks G2f and the three third composite residual blocks G2f are the same. Each composite residual block G2f includes two Ghost modules with a step size of 1, and the activation function is LeakyReLU. The first 1x1 Ghost module divides the output into two parts, and each channel remains consistent. The latest output is passed to the corresponding number of residual modules through extended iterations. Finally, all the outputs are stacked together, and the number of output channels is adjusted through another 1x1 Ghost module.

5. The lightweight fruit maturity detection method applicable to embedded devices according to claim 3, characterized in that: Fast spatial pyramid pooling SPPF includes the bottom feature layer feat3, the lower feature layer feat2, and the middle feature layer feat1; The bottom feature layer feat3 has a feature map size of 20×20×1024; The feature fusion network PANetde upsamples the underlying feature layer feat3, and the feature map size becomes 40×40×512; Then it is connected with the lower feature layer feat2, and the combined features are processed by the composite residual block G2f to obtain the new feature P4, and the feature map is kept at 40×40×512; Feature P4 is upsampled again, and the feature map size becomes 80×80×512. It is fused with the middle feature layer feat1. The fused features are processed by the composite residual block G2f to obtain feature P3, and the feature map size is 80×80×256. Feature P3 is downsampled through a 3x3 convolution, and the feature map size becomes 40×40×256. It is then stacked with feature P4 and processed again through the composite residual block G2f to obtain the updated feature P4, with the size restored to 40×40×512. Finally, the updated feature P4 is downsampled by 3x3 convolution to a size of 20×20×512, stacked with the feature P5 of the underlying feature layer feat3, and processed by the G2f module to obtain the updated P5 with a size of 20×20×1024.

6. The lightweight fruit maturity detection method applicable to embedded devices according to claim 1, characterized in that: The loss function of the fruit detection model includes: the classification loss uses the VFL Loss loss function, and the regression loss introduces Alpha-IoU; the regression loss is obtained by the following formula: Where, L α-CIoU is represented as the regression loss, Y is represented as the intersection-over-union ratio between the predicted box and the true box, b and b gt They are respectively represented as the coordinates of the center points of the predicted box and the true box, C is represented as the diagonal length of the circumscribed rectangle, ρ is represented as the distance between the center points of the predicted box and the true box, β is represented as the trade-off parameter, v is represented as the metric parameter, and α is represented as a power parameter greater than 0.

7. The lightweight fruit maturity detection method applicable to embedded devices according to claim 2, characterized in that: Obtaining the compressed fruit detection model includes the following steps: S31, loading the fruit detection model and the fruit image dataset, and setting the number of iterations and the precision value; S32, determining the global pruning threshold, and sorting the global thresholds according to size; S33, determining the pruning channel index; S33, judging whether pruning is required according to the channel index; S34, if yes, prune the network channel, if no, keep the channel and assign values ​​directly; S35, repeating steps S33 and S34 to determine whether the maximum number of iterations is reached or whether the accuracy meets the requirements; S36, if yes, then output and obtain the compressed fruit detection model, if no, then return to step S33.

8. The lightweight fruit maturity detection method applicable to embedded devices according to claim 7, characterized in that: A scale factor S is introduced into the BN layer of the convolution module Conv and L1 regularization is added. The scale factor S enters the channel in the composite residual block G2f to obtain the sparse scale factor. The scale factor S is multiplied by each channel to produce the output; the loss function calculation formula for the training of the compressed fruit detection model is: Where L is the total loss function of the compressed fruit detection model, (x, y) is the input and output of the training, f(x, W) is the prediction function, which accepts the input data x and the training weight W and outputs the prediction result, λ is the regularization coefficient, g(s) is the regularization function, S is a function of the training weight W, Γ is the set of all possible regularization terms, and l is the loss function.

9. The lightweight fruit maturity detection method applicable to embedded devices according to claim 1, characterized in that: Obtain the lightweight fruit detection model in the following ways: The knowledge distillation method includes a student network and a teacher network; The weight file of the pruned compressed fruit detection model is placed in the corresponding path of the student network, and the weight file of the unpruned fruit detection model is placed in the corresponding path of the teacher network, and the target ratio is set for training; when the accuracy reaches the target ratio, the lightweight fruit detection model is obtained.

10. A lightweight fruit maturity detection system suitable for embedded devices, characterized in that: The lightweight fruit maturity detection method applicable to an embedded device according to any one of claims 1 to 9 comprises: The data module is used to collect videos of fruit plantations in multiple scenes, obtain multiple fruit images, crop fruit image datasets, and label multiple fruit images into mature labels and immature labels; The low-parameter model module is used to improve the YOLOv8l model, using the CSPGhostNet network as the backbone network, the feature fusion network PANet as the neck network, and the classifier and regressor as the head network to build a fruit detection model; A simplified model module is used to take the fruit image dataset as input, perform multi-layer network channel pruning on the fruit detection model based on the model pruning method, and obtain a compressed fruit detection model; The optimization model module is used to fine-tune the fruit detection model and the compressed fruit detection model using the knowledge distillation method to obtain a lightweight fruit detection model; The detection module is used to detect the maturity of fruits using a lightweight fruit detection model.

Images