Lightweight strawberry fruit identification method based on deep learning

Abstract

The invention discloses a lightweight strawberry fruit identification method based on deep learning, and belongs to the technical field of strawberry identification, and the method comprises the steps: constructing a strawberry data set, collecting strawberry fruit images under the conditions of different distances, angles and light rays, screening effective samples, marking the data set according to mature and immature categories, and carrying out the recognition of the strawberry fruits. The method comprises the following steps: constructing a strawberry fruit identification model GGWL, improving and constructing a GGWL model based on a YOLOv11n architecture, constructing a GhostHGNetV2 module as a backbone network, adopting a GDFPN network as a neck network, optimizing detection performance by using a WIoUv3 loss function, realizing model lightweight by adopting a Lamp pruning algorithm, inputting a labeled strawberry data set into the GGWL model for processing, and completing classification and detection of strawberry fruits; according to the lightweight strawberry fruit recognition method based on deep learning, model lightweight is realized, the calculation amount is reduced, the requirement of mobile deployment equipment is met, and the strawberry fruit detection precision and speed under a complex background are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of strawberry recognition, and in particular to a lightweight strawberry fruit recognition method based on deep learning. Background Art

[0002] With the recent increase in strawberry orchard cultivation and the loss of rural labor, there is a need to accelerate research on mechanized strawberry harvesting. Efficient recognition is crucial for mechanized strawberry harvesting. Issues such as overlapping strawberries, occlusion of fruit by stems and leaves, and varying growth stages pose significant challenges to efficient and accurate mechanized strawberry harvesting. Currently, most domestic harvesting robots still rely on traditional machine vision methods to detect and recognize the complex and diverse strawberry harvesting process. These methods suffer from slow detection speed, low accuracy, missed detections, and false detections, severely impacting production efficiency. Strawberry fruit recognition is a key factor in determining harvesting efficiency. GPUs have seen significant development in hardware, and in recent years, GPU-based deep learning object detection algorithms have significantly improved detection speed and accuracy. Furthermore, traditional image processing algorithms can support harvesting robots in structured environments. In recent years, deep learning algorithms have made significant progress in object detection, enabling robots to possess stronger autonomous perception capabilities and harvest strawberries in unstructured environments. Deep learning uses neural networks to extract features from input images, then uses a multi-layered neural network model to learn a large amount of prior information, ultimately enabling the ability to recognize new objects in unstructured environments. Therefore, incorporating deep learning into strawberry picking visual recognition can significantly improve the success rate of strawberry picking robots and significantly enhance picking efficiency.

[0003] In recent years, researchers at home and abroad have conducted extensive research on fruit detection and achieved significant progress, primarily through traditional machine learning algorithms and deep learning algorithms. Traditional machine learning detection methods typically rely on appearance features such as color, shape, and texture. These methods are easily affected by lighting and environmental factors, hindering further improvement in fruit detection accuracy. With the development of artificial intelligence (AI), deep learning has been widely applied in agricultural scenarios, significantly improving fruit detection accuracy. Compared with traditional machine learning algorithms, deep learning object detection algorithms offer advantages such as good adaptability, fast detection speed, and high accuracy. Chen et al. proposed a tomato fruit detection model based on MTD-YOLOv7, which simultaneously detects tomato clusters, tomato fruit, and fruit maturity, demonstrating excellent detection performance and strong generalization. Sun Jun proposed a grape detection model, MRW-YOLOv5s, for complex orchard environments. The results showed that the model further improved grape fruit recognition under occlusion and illumination. Liu et al. proposed an improved model based on the Swin-B transformer, focusing on identifying the ripeness of small strawberry fruit. The model achieved COCO detection metrics with average precision (AP), AP50, and AP75 of 74.1%, 93.9%, and 84.2%, respectively. Wang Yuanhao et al. proposed an improved algorithm based on YOLOv8-SAM to address the occlusion and localization issues of citrus fruit. Using edge detection, least squares, and a three-dimensional dense depth point cloud from a binocular stereo camera, the model achieved an average recognition accuracy of 91.1%. These studies demonstrate the strong feasibility of deep learning for fruit recognition and have accumulated valuable research foundation and experience for strawberry recognition.

[0004] With the continuous advancement of deep learning research in the agricultural field, strawberry recognition and detection technology has also been further improved. Du et al. proposed an improved network model based on YOLOv7. This model quickly and accurately detects ripe strawberries and identifies occlusion levels in complex environments, but has not yet addressed the impact of lighting on strawberry recognition. Huang Jiacai et al. proposed a lightweight Mobile-YOLOv5s strawberry recognition and detection algorithm based on an improvement of YOLOv5s. This algorithm alleviates the problem of missed detection in occluded and small targets during strawberry picking, while also improving strawberry recognition accuracy and computational speed. Yang et al. proposed an improved LS-YOLOv8s strawberry maturity detection model based on YOLOv8s. This model mitigates the complex growing environment, interference from light intensity, and the influence of clustered shadows on strawberries. Yang Zhenyu et al. proposed a strawberry recognition and localization method that combines an improved YOLOv8 algorithm with Pose keypoint detection technology. This method further addresses the low localization accuracy of the stalk picking point and the difficulty in identifying occluded strawberries during the operation of a strawberry picking robot.

[0005] In summary, strawberry fruit recognition faces two major challenges: 1. The accuracy and speed of ripe strawberry fruit recognition still need to be improved. 2. Due to occlusion by the strawberry fruit, branches, and leaves, as well as the influence of complex backgrounds such as lighting, strawberry fruit recognition accuracy is low. To address these issues, this paper proposes a lightweight strawberry fruit recognition method based on machine vision, which achieves more accurate and rapid strawberry fruit detection in complex backgrounds. Summary of the Invention

[0006] The purpose of the present invention is to provide a lightweight strawberry fruit recognition method based on deep learning to solve the problem that complex background affects strawberry fruit recognition, and to accurately and quickly predict ripe strawberry fruits.

[0007] To achieve the above objectives, the present invention provides a lightweight strawberry fruit recognition method based on deep learning, comprising the following steps:

[0008] S1. Construct a strawberry dataset, collect strawberry fruit images at different distances, angles, and lighting conditions, screen valid samples, and annotate the dataset into two categories: ripe and unripe.

[0009] S2. Build a strawberry fruit recognition model GGWL. Improve the GGWL model based on the YOLOv11n architecture. Build a GhostHGNetV2 module network to replace the original YOLOv11 backbone network. Use the GDFPN network as the neck network. Use the WIoUv3 loss function to optimize detection performance. Use the Lamp pruning algorithm to achieve model lightweighting.

[0010] S3. Input the labeled strawberry dataset into the GGWL model for processing. It goes through the backbone network for multi-level feature extraction, the neck network for cross-scale feature fusion, and the head network for classification and detection to complete the classification and detection of strawberry fruits.

[0011] Preferably, the specific steps of S1 are as follows:

[0012] S11, using a handheld device to collect strawberry fruit image data at different distances, angles, and lighting conditions, and saving the images in PNG format;

[0013] S12, screening the collected strawberry image data to remove low-quality, repeated, and redundant images;

[0014] S13. Determine the maturity of the strawberries based on the proportion of the red colored area, and classify and label the dataset into two categories: mature and unripe.

[0015] S14. Labeling is performed in the order from the top to the end of the strawberry fruit. The label box includes the fruit and part of the calyx or nearby pixels. The label is saved in txt format.

[0016] Preferably, the GhostHGNetV2 module network structure in S2 includes an HGStem module, multiple GhostHGBlock modules and a DWConv module, and introduces an SPPF module and a C2PSA module;

[0017] The HGStem module is a preprocessing layer used for preliminary feature extraction. After 3×3 convolution, it is divided into two branches for processing, performing maximum pooling and two 2×2 convolutions respectively. The final concatenation is output after 3×3 and 1×1 convolutions.

[0018] The GhostHGBlock module draws on the OSA module concept and combines multiple GhostConv lightweight convolutions to concatenate the output of each convolution layer with the input channel, and then identifies and amplifies key features through squeezing and excitation operations.

[0019] The GhostConv module is used to replace the lightweight convolution composed of 1×1 convolution and DWConv in the HGBlock module. The purpose is to reduce the amount of computation while enhancing the strawberry feature extraction effect. It not only keeps the model lightweight, but also retains the original feature information.

[0020] The SPPF module improves the traditional SPP parallel structure into a serial structure and uses multi-scale pooling technology to fuse spatial features of different granularities.

[0021] The C2PSA module combines multi-head self-attention mechanism, multi-scale convolution and channel weighted operations to improve the detection ability of complex occluded objects, and selectively incorporates residual structure to optimize gradient propagation and improve network training efficiency.

[0022] Preferably, the convolution in S2 is composed of a convolution layer, a BN layer, and a SiLU activation function. The BN layer normalizes each batch of input data. The calculation formula of the SiLU activation function is as follows:

[0023] SiLU=x*Sigmoid(βx) (1)

[0024] Where x is the input value, β is an adjustable parameter, and Sigmoid() represents the Sigmoid function.

[0025] Preferably, in S2, the specific operation of the GhostConv module is as follows:

[0026] a. Perform 1×1 convolution operation on the input feature map to aggregate information features between channels.

[0027] b. Perform group convolution to generate a new feature map. First, reduce the number of channels through conventional convolution. The calculation formula is as follows:

[0028] Y'=X*f' (2)

[0029] Among them, * represents the convolution operation, X represents the input feature map, f' represents the convolution kernel, and Y' represents the output feature map;

[0030] Based on the obtained output feature map, DWConv convolution is used to further reduce the amount of calculation. The calculation formula is as follows:

[0031]

[0032] Among them, y' i represents the i-th intrinsic feature map, Φ i,j Represents the linear operation responsible for generating the jth ghost map, y ij represents the jth ghost map generated after processing the i-th intrinsic feature map;

[0033] c. Concatenate the output feature map and the ghost map to obtain the final output feature map.

[0034] Preferably, the GDFPN network in S2 is improved based on the efficient-RepGFPN module, and the Upsample upsampling module is improved to a Dysample module.

[0035] Preferably, the specific workflow of the Dysample module in S2 is as follows:

[0036] a. Given a feature size of C×H1×W1 Figure X , where C represents the number of channels, H represents the height, W represents the width, and an upsampling scale factor s;

[0037] b. Use the linear layer to convert the number of input channels C to 2s 2 , the generation size is 2s 2 The offset O of ×H×W is reshaped into 2×sH×sW size through pixel shuffling operation, and the offset O and the original sampling grid G are added to obtain the sampling set S of size 2×sH×sW. The specific operation process is as follows

[0038] O=linear(X) (4)

[0039] S=H+O (5)

[0040] c. To alleviate the problem of local sampling position overlap caused by the normalization layer, a static range factor of 0.25 is introduced and the offset is multiplied by this factor to constrain the movement range of the sampling position. The calculation formula is as follows:

[0041] O=0.25lineat(X) (6)

[0042] d. Through the grid sample function and sampling set S, the input features Figure X Perform bilinear interpolation resampling to generate upsampled features of size 2×sH×sW Figure X ',

[0043] X' = gridsample(X,S) (7).

[0044] Preferably, the calculation formula of the WIoUv3 loss function in S2 is as follows:

[0045] L WIoUv3 =rL WIoUV1 (8)

[0046]

[0047] in, is the monotonic focusing factor, β is the outlier of the anchor box, r is the non-monotonic focusing factor, δ and α are the hyperparameters in the model, L IoU The loss function of the intersection-over-union (IoU) is calculated as follows:

[0048] L IoU =1-IoU (11)

[0049] LWIoUV1 is the WIoUv1 loss function, and the calculation formula is as follows:

[0050] L WIoUv1 =R WIoULIoU (12)

[0051]

[0052] Among them, R WIoU is the weight factor, W g and H g is the width and height of the minimum bounding rectangle of the anchor box and the target box, x and y are the coordinates of the center point of the anchor box, and x gt 、y gt is the coordinate of the center point of the target box.

[0053] Preferably, the specific workflow of Lamp pruning in S2 is as follows:

[0054] a. Calculate the weight of each layer, flatten the weight tensor of each layer into a one-dimensional vector W, calculate the magnitude |W| of each vector W, and sort |W| in ascending order;

[0055] b. Calculate the Lamp score for each weight using the following formula:

[0056]

[0057] Among them, u and v represent the index of weight size |W|, W[u] and W[v] represent the weight items mapped by index u and v, and u<v;

[0058] c. Input the weights of all layers and their corresponding Lamp scores, and prune the weight items with lower scores. The process of weight pruning matrix is as follows:

[0059]

[0060] Among them, i and j represent different weights, and φ represents the deleted small weight;

[0061] d. Update the weight matrix and repeat the above steps until the required global sparsity constraint is met. Stop pruning when the pre-set sparsity is met.

[0062] Therefore, the present invention adopts the above-mentioned lightweight strawberry fruit recognition method based on deep learning, which has the following beneficial effects:

[0063] (1) By introducing HGNetV2 to construct the GhostHGNetV2 backbone network, feature maps of different sizes are extracted, which is suitable for detecting multi-scale targets. Compared with the Mobilenetv4, Repvit, EfficientViT and SwinTransformer backbone networks, the accuracy, recall rate, average precision and other evaluation indicators are improved, which effectively improves the occlusion problem of strawberry fruit recognition.

[0064] (2) The neck network introduces the GFPN network and uses DySample to improve the upsampling module to achieve cross-scale multi-level feature fusion, further improving the accuracy of strawberry fruit detection. After the improvement, the precision is improved by 0.7%, MAP0.5 is improved by 0.2%, and MAP0.5:0.95 is improved by 0.8%;

[0065] (3) Using the WIoUv3 loss function, the aspect ratio penalty term in CIoU is removed, while balancing the impact of high-quality and low-quality anchor frames on model regression, enhancing the generalization ability of the model. After the improvement, the Precision is improved by 0.9%, MAP0.5 is improved by 0.1%, and MAP0.5:0.95 is improved by 0.8%, further alleviating the impact of light on the low accuracy of strawberry fruit recognition;

[0066] (4) The Lamp method is used to perform channel pruning on the improved overall model, reducing the number of redundant channels and retaining key channel information. The Precision is improved by 0.9%, MAP0.5 is improved by 0.1%, and MAP0.5:0.95 is improved by 0.5%.

[0067] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flowchart of a lightweight strawberry fruit recognition method based on deep learning of the present invention;

[0069] Figure 2 This is a lightweight strawberry fruit recognition method based on deep learning in the present invention, and an environmental map of the elevated greenhouse strawberry dataset is collected;

[0070] Figure 3 This is a sample image of a strawberry fruit dataset for a lightweight strawberry fruit recognition method based on deep learning in the present invention;

[0071] Figure 4 This is an overall framework diagram of a lightweight strawberry fruit recognition method model based on deep learning in the present invention;

[0072] Figure 5This is a structural diagram of the GhostHGNetV2 backbone network of a lightweight strawberry fruit recognition method based on deep learning in the present invention;

[0073] Figure 6 This is a structural diagram of the neck network of the GDFPN, a lightweight strawberry fruit recognition method based on deep learning in the present invention;

[0074] Figure 7 This is a structural diagram of the dynamic upsampling of the DySample module of the lightweight strawberry fruit recognition method based on deep learning in the present invention.

[0075] Figure 8 This is a point sampling structure diagram based on static range factors of the DySample module of the lightweight strawberry fruit recognition method based on deep learning of the present invention;

[0076] Figure 9 Schematic diagram of anchor boxes and true bounding boxes for a lightweight strawberry fruit recognition method based on deep learning in the present invention;

[0077] Figure 10 This is a loss update curve diagram of different loss functions in each round of iteration of a lightweight strawberry fruit recognition method based on deep learning in the present invention. DETAILED DESCRIPTION

[0078] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0079] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0080] Example

[0081] like Figure 1 As shown, the present invention provides a lightweight strawberry fruit recognition method based on deep learning, comprising the following steps:

[0082] S1. Construct a strawberry dataset, collect strawberry fruit images at different distances, angles, and lighting conditions, screen valid samples, and annotate the dataset into two categories: ripe and unripe.

[0083] S11, use a handheld device to collect strawberry fruit image data at different distances, angles, and light conditions. The environment for collecting strawberry image data in this embodiment is as follows: Figure 2As shown in the figure, shooting at 45° to the left, front, and right can increase the diversity of samples and enhance the robustness of the model; the machine vision of the picking robot at the far and near ends is simulated at two positions 0.3 meters and 0.6 meters away from the strawberry rack; the picking operation of the picking robot is simulated at different time periods under various lighting conditions such as front light and back light, and the images are saved in PNG format to facilitate the model to process RGB images. The collected strawberry dataset image samples are as follows: Figure 3 shown.

[0084] S12. Screen the collected strawberry image data to remove low-quality, repeated, and redundant images, such as those with out-of-focus or underexposure, to improve the effectiveness of the sample.

[0085] S13. Determine the ripeness of the strawberries based on the proportion of the red-colored area. The standard for determining whether strawberries are ripe for picking is: if the red-colored area of the strawberries reaches 80% or more from the RGB image, the strawberries meet the picking standard. Considering the time spent on strawberry picking, transportation, storage, and processing, the strawberries should be fully ripe and edible when they are sold. According to this standard, the dataset is classified and labeled into two categories: ripe (ripe) and unripe (unripe).

[0086] S14. Strawberry fruits grown on elevated or ridges have the characteristic of growing vertically downward. When labeling, the strawberry fruits should be labeled in order from top to bottom. The unobstructed fruits should be completely contained in the label frame and contain a small part of the calyx. The obscured fruits should have the unobstructed part of the fruit contained in the label frame and a small number of adjacent pixels. The labels should be saved in txt format to facilitate the model to process the text labels.

[0087] S2. Construct a strawberry fruit recognition model GGWL. The structure of the GGWL model is shown in the following figure. Figure 4 As shown in the figure, the GGWL model is improved based on the YOLOv11n architecture, the HGNetV2 network is introduced, and the lightweight convolution in the HGBlock module is replaced with GhostConv. The GhostHGNetV2 module network is constructed to replace the original YOLOv11 backbone network. The GhostHGNetV2 module extracts features in a multi-level manner, which is more suitable for small target strawberry fruit detection. It further extracts the unobstructed area features of the strawberry fruit, and uses the GDFPN network as the neck network to achieve cross-scale multi-level feature fusion. In order to solve the problem that strawberry fruit is difficult to identify and has low recognition accuracy due to the influence of illumination, the WIoUv3 loss function is used to optimize the detection performance, reduce the influence of low-quality samples of strawberry images, and use the Lamp pruning algorithm to achieve model lightweighting. The improved overall model is pruned to reduce the number of redundant channels and retain key channel information.

[0088] The network structure of GhostHGNetV2 module includes a HGStem module, multiple GhostHGBlock modules and DWConv modules, and introduces SPPF module and C2PSA module; the network structure of GhostHGNetV2 module is as follows Figure 5 shown.

[0089] The HGStem module is a preprocessing layer used to perform preliminary feature extraction on the original strawberry image. The input feature map is divided into two branches after 3×3 convolution, which perform maximum pooling and two 2×2 convolutions respectively. The first convolution reduces the number of feature map channels and fills the size, while the second convolution restores the number of feature map channels and halves the size. Finally, the two branches perform channel splicing and output through 3×3 and 1×1 convolutions. The output feature map size is 1 / 4 of the original image, reducing the complexity of the subsequent calculation process.

[0090] The GhostHGBlock module draws on the principles of the OSA module, combining multiple GhostConv lightweight convolutions. It concatenates the output of each convolution layer with the input channel by channel, then identifies and amplifies key features through squeezing and excitation operations. The OSA module aggregates its features to the final layer, with each convolution layer employing bidirectional connections. One approach involves directly connecting to the next convolution layer to generate features with a larger receptive field. Another approach involves only a single aggregation of each convolution layer with the final convolution layer, reducing memory access costs and making GPU computation more efficient. Because deep features are more critical for object detection tasks, the ratio of deep features to low-level features is increased by increasing the number of output channels at each stage.

[0091] The GhostConv module is used to replace the lightweight convolution composed of 1×1 convolution and DWConv in the HGBlock module. The purpose is to reduce the amount of computation while enhancing the strawberry feature extraction effect. It not only keeps the model lightweight, but also retains the original feature information.

[0092] The specific operations of the GhostConv module are as follows:

[0093] a. Perform 1×1 convolution operation on the input feature map to aggregate information features between channels.

[0094] b. In order to reduce the amount of network calculation, group convolution is performed to generate new feature maps. First, the number of channels is reduced through conventional convolution to reduce the amount of calculation. The calculation formula is as follows:

[0095] Y'=X*f' (2)

[0096] Among them, * represents the convolution operation, X represents the input feature map, f' represents the convolution kernel, and Y' represents the output feature map;

[0097] Based on the obtained output feature map, DWConv convolution is used to further reduce the amount of calculation. The calculation formula is as follows:

[0098]

[0099] Among them, y' i represents the i-th intrinsic feature map, Φ i,j Represents the linear operation responsible for generating the jth ghost map, y ij represents the jth ghost map generated after processing the i-th intrinsic feature map;

[0100] c. Concatenate the output feature map and ghostmap channels to obtain the final output feature map.

[0101] The SPPF module uses multi-scale pooling technology to fuse spatial features of different granularities, improving the traditional SPP (Spatial Pyramid Pooling) parallel structure into a serial structure. SPP is a technology widely used in image recognition and target detection. Its core function is to extract and encode features from images at different scales, thereby scaling input images of any size to a fixed size and generating feature vectors of fixed length. Through this serial structure transformation, SPPF significantly reduces the complexity of the module while maintaining its original performance, thereby accelerating the model training and inference process.

[0102] The C2PSA module is an advanced network structure that combines a multi-head self-attention mechanism, multi-scale convolution, and channel weighting operations, greatly enhancing the model's ability to focus on complex occluded objects and important areas, demonstrating outstanding performance in static target detection tasks in particular. To optimize gradient propagation and improve network training efficiency, C2PSA selectively incorporates a residual structure (shortcut). In addition, the architecture utilizes multi-scale convolution technology to map input features to a high-dimensional space, thereby accurately capturing the complex nonlinear relationships of the input features and enabling the model to learn richer and more detailed feature representations.

[0103] The convolution in this embodiment consists of a convolution layer, a BN layer, and a SiLU activation function. The BN layer normalizes each batch of input data. The calculation formula of the SiLU activation function is as follows:

[0104] SiLU=x*Sigmoid(βx) (1)

[0105] Where x is the input value, β is an adjustable parameter, and Sigmoid() represents the Sigmoid function.

[0106] The neck network GDFPN network is based on the efficient-RepGFPN module. The Upsample upsampling module is improved to the Dysample module. It is a path fusion containing skip layers and cross-scale connections. It can fully exchange deep feature information and shallow feature information to prevent the loss of up-down relationships in feature fusion. The network structure is as follows Figure 6 As shown in the figure, different scale features use different numbers of channels, flexibly controlling the expressive power of deep and shallow features under lightweight computational constraints. The original convolution-based feature fusion is improved to a CSPStage connection, while introducing reparameterization and ELAN connections, improving model accuracy without increasing computational overhead.

[0107] The Dysample module is an ultra-lightweight and effective dynamic upsampler, which is a simpler and more efficient way to increase image resolution. The network structure is as follows: Figure 7 As shown in the figure, compared to the traditional CARAFE upsampling method, DySample requires fewer computing resources and can achieve image resolution improvement without adding additional burden. DySample uses a simple and efficient method to generate content-aware upsampling results without the need for additional high-resolution feature input. This allows DySample to maintain high performance while reducing model complexity and computational cost.

[0108] The specific workflow of the Dysample module is as follows:

[0109] a. Given a feature size of C×H1×W1 Figure X , where C represents the number of channels, H represents the height, W represents the width, and an upsampling scale factor s;

[0110] b. Use the linear layer to convert the number of input channels C to 2s 2 , the generation size is 2s 2 The offset O of ×H×W is reshaped into 2×sH×sW size through pixel shuffling operation, and the offset O and the original sampling grid G are added to obtain the sampling set S of size 2×sH×sW. The specific operation process is shown as follows:

[0111] O=linear(X) (4)

[0112] S=G+O (5)

[0113] Due to the existence of the normalization layer, the value of an output feature is usually in the range of [-1, 1], centered on 0. Therefore, the moving range of the local sampling positions may overlap significantly, which can easily affect the prediction near the boundary, causing stage-by-stage propagation and resulting in a deviation from the actual output.

[0114] c. In order to alleviate the problem of local sampling position overlap caused by the normalization layer, a static range factor of 0.25 is introduced and the offset is multiplied by this factor to constrain the movement range of the sampling position, such as Figure 8 As shown, the calculation formula is as follows:

[0115] O=0.25linear(X) (6)

[0116] d. Through the grid sample function and sampling set S, the input features Figure X Perform bilinear interpolation resampling to generate upsampled features of size 2×sH×sW Figure X '.

[0117] X'=gridsample(X,S) (7)

[0118] This embodiment uses the WIoUv3 loss function to effectively improve the problem of strawberry fruit being difficult to identify and having low recognition accuracy due to the influence of light. WIoUv3 introduces a dynamic non-monotonic mechanism. Figure 9 As shown in the figure, the upper left rectangle represents the ground truth bounding box, and the lower right rectangle represents the anchor box. This loss function uses dynamic non-monotonic frequency modulation to increase the focus on high-quality anchor boxes and reduce the influence of low-quality samples. WIoU has three versions: WIoUv1, WIoUv2, and WIoUv3. WIoUv3 is obtained by adding a non-monotonic focusing coefficient to WIoUv1.

[0119] The calculation formula of WIoUv3 loss function is as follows:

[0120] L WIoUv3 =rL WIoUV1 (8)

[0121]

[0122] in, is the monotonic focusing factor; β is the outlier of the anchor box. The higher the outlier, the worse the quality of the anchor box. r is the non-monotonic focusing factor, which is constructed by the outlier β and δ, α. δ and α are hyperparameters in the model, which can effectively prevent low-quality samples from generating large harmful gradients, increase attention to correct samples, and improve the accuracy of the network model. IoU The loss function of the intersection-over-union (IoU) is calculated as follows:

[0123] L IoU =1-IoU (11)

[0124] L WIoUV1 is the WIoUv1 loss function, and the calculation formula is as follows:

[0125] LWIoUv1 =R WIoU L IoU (12)

[0126]

[0127] Among them, R WIoU is a weight factor, which focuses on the center distance between the anchor box and the target box. The lower the overlap between the anchor box and the target box, the better the R WIoU The larger the W g and H g is the width and height of the minimum bounding rectangle of the anchor box and the target box, x and y are the coordinates of the center point of the anchor box, and x gt 、y gt is the coordinate of the center point of the target box.

[0128] The GGWL model uses the Lamp pruning algorithm. Model pruning algorithms can be mainly divided into model training, model pruning, and model fine-tuning. Model training generally refers to training a model with good detection performance. This model generally has characteristics such as a large number of parameters and a large amount of computation. This model can be used as a baseline model for pruning operations. The pruning algorithm uses a specific pruning method to remove redundant information in the model to obtain a relatively compact network model. After pruning, the compact model has a significant reduction in parameters and computation, but often has a loss in accuracy. Fine-tuning is to improve the problem of reduced accuracy. After model pruning is completed, fine-tuning and retraining can improve the detection performance of the pruned model and even surpass the detection performance of the original model. In order to reduce the number of parameters of the improved precision model and improve sparsity, the complexity of the model can be reduced while maintaining model accuracy.

[0129] The specific workflow of Lamp pruning is as follows:

[0130] a. Calculate the weight size of each layer. Because the dimensions of the weights of the fully connected layer and the convolutional layer are not uniform, flatten the weight tensor of each layer into a one-dimensional vector W, calculate the magnitude |W| of each vector W, and sort |W| in ascending order;

[0131] b. Calculate the Lamp score for each weight using the following formula:

[0132]

[0133] Here, u and v represent the indices of the weight magnitude |W|, W[u] and W[v] represent the weight terms mapped to the indices u and v, with u < v. The numerator represents the square of the weight magnitude of the target connection, and the denominator represents the sum of the squares of the weight magnitudes of the remaining connections in the same layer. This metric measures the importance of the current layer connection relative to the other connections in the same layer. As the formula shows, larger weight terms correspond to higher Lamp scores, while lower-scoring weight terms are considered relatively unimportant connections and can be pruned. The Lamp score ensures that each layer has at least one optimal channel with a score of 1, effectively preventing layer failure.

[0134] c. Input the weights of all layers and their corresponding Lamp scores, and prune the weight items with lower scores. The process of weight pruning matrix is as follows:

[0135]

[0136] Among them, i and j represent different weights, and φ represents the deleted small weight;

[0137] d. Update the weight matrix and repeat the above steps until the required global sparsity constraint is met. Stop pruning when the pre-set sparsity is met.

[0138] The computational design of this embodiment enables the Lamp score to better measure the relative importance of channels, and can achieve adaptive layer-by-layer sparsification, combining the advantages of global pruning and local pruning. When the pruning amplitude is small, the accuracy of the model may be improved due to reduced complexity, reduced overfitting, and other reasons; however, when the pruning amplitude is too large, the accuracy of the model will be greatly reduced due to problems such as the loss of important features, reduced network capacity, and damage to generalization ability. Therefore, in order to balance the model size and accuracy, it is necessary to carefully select the pruning amplitude and perform appropriate fine-tuning after pruning to restore the performance of the model. In the model of this embodiment, operations such as convolution have different numbers of channels, but not all channels play an active role in model prediction, which creates conditions for the lightweight model. By calculating the Lamp score of each channel in the convolution to perform channel pruning, the number of model parameters and the amount of calculation are reduced as much as possible while retaining the key information of the model.

[0139] S3. Input the labeled strawberry dataset into the GGWL model for processing. The GGWL model first performs preprocessing operations to compress the image into a 640×640 pixel RGB image. It then passes through the backbone network for multi-level feature extraction, the neck network for cross-scale feature fusion, and the head network for classification and detection to complete the classification and detection of strawberry fruits.

[0140] This example verifies the effectiveness and superiority of the GGWL model through ablation experiments, target detection model comparison experiments, backbone network comparison experiments, and loss function comparison experiments.

[0141] This embodiment uses the precision (Precision, P), recall (Recall, R) and mean average precision (MAP) of image detection to evaluate the final effect, where MAP0.5 and MAP0.5:0.95 are both indicators for measuring the performance of the target detection model. MAP0.5 refers to the mean of the average precision and average value of each category when the I0U threshold is 0.5, and MAP 0.5:0.95 refers to the mean of the average precision and average value of each category when the IOU threshold is from 0.5 to 0.95. At the same time, in order to measure whether the network can be further deployed to the mobile terminal, it is necessary to comprehensively consider the number of parameters and the detection speed FPS (frames per second) of the picture. The F1 score is the harmonic mean of the precision and recall rate, which can assist in evaluating the performance of the model. In the experiment of this embodiment, the calculation formulas for the detection rate FPS, mAP, and AP (average precision) are as follows:

[0142]

[0143] Among them, P reTIme is the image preprocessing time, in ms, I nferTime The time required for network inference, in ms, N MSTime The time for network prediction box optimization, in milliseconds, where n is the number of detected categories.

[0144] Ablation experiments were conducted using the YOLOv11n object detection model as the baseline model. GhostHGNetV2, GDFPN, WIoUv3, and Lamp modules were sequentially superimposed and the results analyzed. The results are shown in Table 1. √ indicates the use of the algorithm, × indicates the absence of the algorithm, and Params represents the number of parameters. As shown in the table, adding GhostHGNetV2, GDFPN, and WIoUv3 individually did not improve the performance, or even slightly worsened it. Combining the GhostHGNetV2 backbone network with the GDFPN neck network improved both Precision and MAP0.5:0.95 by 0.5% compared to the baseline model. The GhostHGNetV2 feature extraction method is more compatible with the GDFPN cross-scale, multi-level feature fusion approach. However, the increased number of Params resulted in a larger model and reduced detection speed. Adding the WIoUv3 loss function provides a reasonable gradient gain distribution strategy and balances high and low sample quality, resulting in improvements of 1.5% and 1.7% in Precision and MAP0.5:0.95, respectively, compared to the baseline model. On this basis, the Lamp pruning method was added. Lamp fine-tunes model parameters by removing redundant channel information from the model. Compared to the baseline model, Precision, Recall, MAP0.5, MAP0.5:0.95, and FPS increased by 2.2%, 0.2%, 0.2%, 2.1%, and 0.9 frames per second, respectively. Params, GFLOPs, and Weight Size increased by 25%, 2%, and 30%, respectively. The above analysis shows that maintaining the model's lightweight within an acceptable range improves model detection accuracy and speed. Observing the superposition effect shows that the improved model achieves optimal detection results.

[0145] Table 1 Ablation test

[0146]

[0147]

[0148] The backbone network comparison test also uses the YOLOv11n target detection network as the base network, replacing the mainstream feature extraction backbone network. On the basis of keeping other parameters except the backbone network consistent, the target training effects of different backbone networks are compared, as shown in Table 2.

[0149] Table 2 Backbone network comparison test

[0150] backbone network Precision / % Recall / % MAP0.5 / % MAP0.5:95 / % GFLOPs Starnet 89.5 88.2 93.3 66.6 5.0 Mobilenetv4 89.0 87.7 93.4 66.9 21.0 Repvit 89.7 89.7 93.5 67.4 17.0 EfficientViT 88.8 89.3 93.1 67.5 7.9 SwinTransformer 89.1 90.2 93.5 67.9 77.6 GhostHGNetV2 90.1 89.1 93.9 68.3 7.7

[0151] As shown in the table, using GhostHGNetV2 to improve the backbone network improves Precision by 0.6%, 1.1%, 0.4%, and 1.3%, respectively, compared to the Starnet, Mobilenetv4, Repvit, EfficientViT, and SwinTransformer backbone networks. MAP0.5 also improves by 0.6%, 0.5%, 0.4%, and 0.8%, respectively. It also achieves an advantage in MAP0.5:95, and is second only to Starnet in Params and GFLOPs. The above analysis shows that the GhostHGNetV2 backbone network achieves the best detection results. Its multi-layer feature extraction approach allows it to better capture subtle features in strawberry images, further improving occlusion issues in strawberry fruit recognition.

[0152] Loss function comparison test: The YOLOv11 target detection model before improvement uses CIoU as the bounding box regression loss function, which has a strong fitting ability during training. However, since the aspect ratio of the predicted box is described as a relative value, there is uncertainty in the calculation. At the same time, the quality of the anchor box will also affect the regression loss. This embodiment test also compares the convergence of WIoUv3, PIoU, ShapeIoU, SIoU, EIoU, CIoU, and DIoU loss functions. The test results are as follows: Figure 10 As shown in the figure, the curve with the lowest loss value at the bottom is the WIoUv3 loss function curve. When using EIoU, the convergence speed is the slowest, and the loss value after convergence is the highest. PIoU, ShapeIoU, SIoU, CIoU, and DIoU have similar convergence effects, with slightly higher convergence speeds than EIoU and slightly lower loss values after convergence. WIoUV3 has the best convergence effect, with the model's gradient descent speed during training being the fastest, and the loss value after convergence being much smaller than the other loss functions. Evaluation indicators such as accuracy, recall, and average precision have all improved.

[0153] Target detection model comparison test: The GGWL-YOLO model improved based on YOLOv11n is compared with mainstream target detection models such as YOLOv7tiny, YOLOv8n, YOLOv8s, YOLOv11n, YOLO11s, RT-DETR-r18, SSD, and Fast-RCNN, as well as excellent strawberry fruit recognition models such as LS-YOLOv8s, DSW-YOLO, and YOLOv8pose. The improvement effect of the same improvement method in this embodiment on the YOLOv8s and YOLOv11s base models is not as good as that on the YOLOv11n base model, indicating that the improvement method in this embodiment is not suitable for large models, but is more suitable for lightweight models. The test results are shown in Table 3.

[0154] Table 3 Comparative test of different target detection models

[0155]

[0156]

[0157] As shown in the table, the GGWL-YOLO model's Precision, MAP0.5, and MAP0.5:0.95 metrics are all higher than those of the other models. Six models were selected for strawberry fruit detection analysis. YOLOv11n served as the baseline model for this experiment, while RT-DETR-r18 was a mainstream model comparable to the YOLO model. LS-YOLOv8s, DSW-YOLO, and YOLOv8pose were other excellent strawberry fruit recognition models that provided a more comparable comparison to the improved GGWL-YOLO model proposed in this example. An analysis of experimental metrics shows that Precision is 2.2%, 2.4%, 0.7%, 0.8%, and 4.4% higher than YOLOv11n, RT-DETR-r18, LS-YOLOv8s, DSW-YOLO, and YOLOv8pose, respectively. MAP0.5 is 0.2%, 0.1%, 1.5%, 0.2%, 0%, and 3.6% higher, respectively. Meanwhile, GFLOPs are reduced by -12.1% (increased), 86.9%, 80.2%, and 36.3%, respectively. Compared to YOLOv11n, GGWL-YOLO sacrifices a small amount of detection speed but achieves a significant performance improvement. It boasts higher detection accuracy, saves memory and computing resources, and is more suitable for real-time detection of ripe strawberries on mobile devices.

[0158] The GGWL model proposed in this embodiment has various indicators on the validation set: Precision of 92.5%, MAP0.5 of 94.3%, MAP0.5:0.95 of 70.9, Params of 2.38M, GFLOPs of 7.4, and FPS of 91.5 frames / s. Compared with the original model YOLOv11n, the improved model has a Precision increase of 2.2%, a MAP0.5 increase of 0.2%, and a MAP0.5:0.95 increase of 2.3%. The remaining indicators also have certain advantages over other mainstream object detection network models.

[0159] Therefore, the present invention adopts the above-mentioned lightweight strawberry fruit recognition method based on deep learning, which improves the problem of low strawberry recognition accuracy and slow detection speed caused by complex backgrounds such as occlusion of strawberry fruits and branches and leaves and light, and improves the strawberry recognition accuracy and speed.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A lightweight strawberry fruit recognition method based on deep learning, characterized in that: The following steps are involved: S1. Construct a strawberry dataset, collect strawberry fruit images at different distances, angles, and lighting conditions, screen valid samples, and annotate the dataset into two categories: ripe and unripe. S2. Build a strawberry fruit recognition model GGWL. Improve the GGWL model based on the YOLOv11n architecture. Build a GhostHGNetV2 module network to replace the original YOLOv11 backbone network. Use the GDFPN network as the neck network. Use the WIoUv3 loss function to optimize detection performance. Use the Lamp pruning algorithm to achieve model lightweighting. S3. Input the labeled strawberry dataset into the GGWL model for processing. It goes through the backbone network for multi-level feature extraction, the neck network for cross-scale feature fusion, and the head network for classification and detection to complete the classification and detection of strawberry fruits.

2. A lightweight strawberry fruit recognition method based on deep learning according to claim 1, characterized in that, The specific steps of S1 are as follows: S11, using a handheld device to collect strawberry fruit image data at different distances, angles, and lighting conditions, and saving the images in PNG format; S12, screening the collected strawberry image data to remove low-quality, repeated, and redundant images; S13. Determine the maturity of the strawberries based on the proportion of the red colored area, and classify and label the dataset into two categories: mature and unripe. S14. Labeling is performed in the order from the top to the end of the strawberry fruit. The label box includes the fruit and part of the calyx or nearby pixels. The label is saved in txt format.

3. A lightweight strawberry fruit recognition method based on deep learning according to claim 1, characterized in that: The network structure of the GhostHGNetV2 module in S2 includes an HGStem module, multiple GhostHGBlock modules and a DWConv module, and introduces the SPPF module and the C2PSA module; The HGStem module is a preprocessing layer used for preliminary feature extraction. After 3×3 convolution, it is divided into two branches for processing, performing maximum pooling and two 2×2 convolutions respectively. The final concatenation is output after 3×3 and 1×1 convolutions. The GhostHGBlock module draws on the OSA module concept and combines multiple GhostConv lightweight convolutions to concatenate the output of each convolution layer with the input channel, and then identifies and amplifies key features through squeezing and excitation operations. The GhostConv module is used to replace the lightweight convolution consisting of 1×1 convolution and DWConv in the HGBlock module; The SPPF module improves the traditional SPP parallel structure into a serial structure and uses multi-scale pooling technology to fuse spatial features of different granularities. The C2PSA module combines multi-head self-attention mechanism, multi-scale convolution and channel weighted operations to improve the detection ability of complex occluded objects, and selectively incorporates residual structure to optimize gradient propagation and improve network training efficiency.

4. A lightweight strawberry fruit recognition method based on deep learning according to claim 3, characterized in that: The convolution in S2 consists of a convolution layer, a BN layer, and a SiLU activation function. The BN layer normalizes each batch of input data. The calculation formula of the SiLU activation function is as follows: SiLU=x*Sigmoid(βx) (1) Where x is the input value, β is an adjustable parameter, and Sigmoid() represents the Sigmoid function.

5. A lightweight strawberry fruit recognition method based on deep learning according to claim 3, characterized in that, In S2, the specific operations of the GhostConv module are as follows: a. Perform 1×1 convolution operation on the input feature map to aggregate information features between channels. b. Perform group convolution to generate a new feature map and reduce the number of channels through conventional convolution. The calculation formula is as follows: Y'=X*f' (2) Among them, * represents the convolution operation, X represents the input feature map, f' represents the convolution kernel, and Y' represents the output feature map; Based on the obtained output feature map, DWConv convolution is used to further reduce the amount of calculation. The calculation formula is as follows: Among them, y' i represents the i-th intrinsic feature map, Φ i,j Represents the linear operation responsible for generating the jth ghost map, y ij represents the jth ghost map generated after processing the i-th intrinsic feature map; c. Concatenate the output feature map and the ghost map to obtain the final output feature map.

6. A lightweight strawberry fruit recognition method based on deep learning according to claim 1, characterized in that: The GDFPN network described in S2 is improved based on the efficient-RepGFPN module, and the Upsample upsampling module is improved to the Dysample module.

7. A lightweight strawberry fruit recognition method based on deep learning according to claim 6, characterized in that, The specific workflow of the Dysample module described in S2 is as follows: a. Given a feature map X of size C×H1×W1, where C represents the number of channels, H represents the height, W represents the width, and an upsampling scale factor s; b. Use the linear layer to convert the number of input channels C to 2s 2 , the generation size is 2s 2 The offset O of ×H×W is reshaped into 2×sH×sW size through pixel shuffling operation, and the offset O and the original sampling grid G are added to obtain the sampling set S of size 2×sH×sW. The specific operation process is as follows O=linear(X) (4) S=H+O (5) c. To alleviate the problem of local sampling position overlap caused by the normalization layer, a static range factor of 0.25 is introduced and the offset is multiplied by this factor to constrain the movement range of the sampling position. The calculation formula is as follows: O=0.25lineat(X) (6) d. Through the grid sample function and sampling set S, the input feature map X is bilinearly interpolated and resampled to generate an upsampled feature map X' of size 2×sH×sW X' = gridsample(X,S) (7).

8. A lightweight strawberry fruit recognition method based on deep learning according to claim 1, characterized in that, The calculation formula of WIoUv3 loss function in S2 is as follows: L WIoUv3 =rL WIoUV1 (8) in, is the monotonic focusing factor, β is the outlier of the anchor box, r is the non-monotonic focusing factor, δ and α are the hyperparameters in the model, L IoU The loss function of the intersection-over-union (IoU) is calculated as follows: L IoU =1-IoU (11) L WIoUV1 is the WIoUv1 loss function, and the calculation formula is as follows: L WIoUv1 =R WIoU L IoU (12) Among them, R WIoU is the weight factor, W g and H g is the width and height of the minimum bounding rectangle of the anchor box and the target box, x and y are the coordinates of the center point of the anchor box, and x gt 、y gt is the coordinate of the center point of the target box.

9. A lightweight strawberry fruit recognition method based on deep learning according to claim 1, characterized in that, The specific workflow of Lamp pruning in S2 is as follows: a. Calculate the weight of each layer, flatten the weight tensor of each layer into a one-dimensional vector W, calculate the magnitude |W| of each vector W, and sort |W| in ascending order; b. Calculate the Lamp score for each weight using the following formula: Among them, u and v represent the index of weight size |W|, W[u] and W[v] represent the weight items mapped by index u and v, and u<v; c. Input the weights of all layers and their corresponding Lamp scores, and prune the weight items with lower scores. The process of weight pruning matrix is as follows: Among them, i and j represent different weights, and φ represents the deleted small weight; d. Update the weight matrix and repeat the above steps until the required global sparsity constraint is met. Stop pruning when the pre-set sparsity is met.

Images