Strawberry binocular detection positioning method based on YOLOv8 fused CGFM network model

Abstract

The invention discloses a strawberry binocular detection positioning method based on YOLOv8 fused with a CGFM network model, images of strawberry flower buds, flowers, immature fruits and mature fruits in four growth stages are collected, the whole growth process of strawberries is automatically monitored and managed based on a WCS-YOLOv8s model, and firstly, the initial process of model training is stabilized based on a data enhancement strategy of a Warmup learning rate; secondly, a CGFM module is introduced to fuse different information through a multi-head self-attention mechanism, and the recognition capability in a complex scene is improved. And finally, the SE-MSDWA module is combined with multi-scale convolution and an SE attention mechanism, so that the model feature extraction capability is enhanced, and redundant information is inhibited at the same time. According to the method, the identification problem caused by branch and leaf shielding, small target and maturity difference is effectively solved, the detection precision and generalization ability are remarkably improved, and technical support is provided for automatic monitoring, management and picking in the strawberry growth process.

Description

Technical Field

[0001] The present invention relates to the technical field of picture classification, and in particular to a strawberry binocular detection and positioning method based on a YOLOv8 fused CGFM network model. Background Art

[0002] With the development of navigation technology and vision technology, the objects of picking have expanded from tomatoes to a variety of fruit and vegetable crops. However, due to the complexity of the strawberry picking process, the current intelligence and mechanization of strawberries are still low, and they mainly rely on manual picking management. Reasonable management of the growth process of strawberry seedlings is the key to high-quality cultivation. As an important management measure, flower and fruit thinning helps to regulate nutrient supply, reduce crop load, and promote fruit development. Correct flower and fruit thinning is very important for the yield and quality of strawberries. When thinning fruits, first remove deformed and diseased fruits, which can increase the yield by 20%-30% compared with non-flower and fruit thinning. It is beneficial to reduce plant nutrient consumption, concentrate nutrients, concentrate fruit ripening period, reduce the number of harvests, improve fruit quality and commercial fruit rate, and prevent premature aging of plants. In outdoor orchards, soil background brings challenges to fruit detection. Various interference factors, such as object direction, occlusion, light intensity, lighting conditions, fruit distance, fruit aggregation and camera angle, will cause target deformation and increase the difficulty of detection. At present, the strawberry cultivation mode is gradually changing from open-air cultivation to greenhouse cultivation. In greenhouse environments, rapid detection of flowers and fruits is crucial for the development of automated flower and fruit management robots. Currently, no research on binocular recognition and positioning for the strawberry growth process has been found.

[0003] Traditional machine vision methods mainly use manually extracted features (such as color, geometric features, texture, and spectral information) to detect flowers and fruits. The limited representation ability of manual features makes traditional machine vision less robust and accurate in complex environments. Deep learning has significantly improved the detection of flowers and fruits of crops, but strawberry research has mainly focused on fruit recognition and ignored seedling detection. In addition, there is limited research on visually assisted sparse flower and fruit detection, and it is still difficult to accurately identify flowers and fruits in greenhouse environments. First, the size of different targets varies greatly, making it difficult to accurately detect smaller targets. Second, the dense arrangement of strawberry seedlings leads to overlapping and shading of flowers and leaves, making it difficult to distinguish flower buds and immature fruits from the background. Summary of the invention

[0004] In order to overcome the above problems existing in the prior art, the present invention proposes a strawberry binocular detection and positioning method based on YOLOv8 fused with CGFM network model.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a strawberry binocular detection and positioning method based on YOLOv8 fused with CGFM network model, comprising the following steps: Step 1: Collect photos of immature, mature, buds and flowers, annotate the photos, and construct a dataset of strawberry with different maturity and flower and fruit thinning detection; Step 2: randomly divide the data set obtained in step 1 into a training set, a validation set, and a test set; Step 3, build a WCS-YOLOv8s model, use the training set obtained in step 1 to train the model, use the validation set to test the performance of the WCS-YOLOv8s model based on the cross-validation method, and finally obtain a trained WCS-YOLOv8s model; Step 4: Collect the photos to be detected and input them into the WCS-YOLOv8s model trained in step 3 to detect and locate the strawberries; The WCS-YOLOv8s model in step 3 includes an input layer, a backbone network, a neck network, and a prediction layer. The input layer establishes an image processing strategy and uses a Warmup data enhancement module for data enhancement. The SE-MSDWA module is applied in the backbone network for efficient feature extraction. The CGFM module is used in the neck network to improve the feature fusion performance of the network. The prediction layer uses three different detection heads to predict the target category probability and position in three feature maps of different scales.

[0006] In the above-mentioned strawberry binocular detection and positioning method based on the YOLOv8 fusion CGFM network model, photos are collected by binocular ranging technology in step 1.

[0007] In the above strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model, the Warmup data enhancement module gradually increases the probability of data enhancement as the epoch changes. In the early stage of model training, the Warmup data enhancement module selects a smaller learning rate to help the model gradually stabilize at a smaller learning rate. When the training reaches 1 / 5 of the total number of rounds, data enhancement is performed in the default mode of YOLOv8, and the formula for the change of data enhancement probability is: Among them, Current_epoch is the current epoch number of the training process, and Total_epochs is the total number of epochs of the training process.

[0008] In the above-mentioned strawberry binocular detection and positioning method based on the YOLOv8 fusion CGFM network model, the CGFM module splices two different feature maps in the channel dimension, inputs them into a multi-head self-attention mechanism module containing convolution, generates attention weights, and applies the generated attention weights to the original features through element-level multiplication to enhance the representation ability of the features.

[0009] In the above-mentioned strawberry binocular detection and positioning method based on the YOLOv8 fusion CGFM network model, the SE-MSDWA module combines a multi-scale convolutional layer, a depth-separable convolutional layer, and a SE module. The multi-scale convolutional layer is used to capture feature information of different scales. The depth-separable convolutional layer reduces the amount of calculation and parameters while improving the feature extraction capability. The SE module is used to dynamically adjust the weights of each channel, highlight important features, and enhance the effectiveness of feature representation.

[0010] The above-mentioned strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model, the SE-MSDWA module processing process includes: independently performing convolution operations on each channel through a multi-scale convolution layer to extract important spatial information; performing information interaction between channels through point-by-point convolution; using three sets of convolution kernels of different scales for multi-scale convolution processing. After convolution processing, the depth-separable convolution layer decomposes the n*n convolution into n*1 convolution and 1*n convolution. The feature map is fused with multi-scale information through an additional convolution layer and finally enters the SE module. The SE module first performs adaptive average pooling to reduce the feature map of each channel to 1x1, and then calculates the weight of each channel through two fully connected layers and activation functions, and reapplies these weights to the original feature map. The beneficial effect of the present invention is that the present invention first detects the target position in the two-dimensional image by YOLO, obtains the coordinate information of the target, and then calculates the depth information of the strawberry by the depth map, thereby measuring the (x, y, z) coordinates of the strawberry relative to the camera to support the management and picking of strawberries. The final detection result is visualized in the figure, and the detected target is framed by a rectangular frame, and the target category, confidence and measurement distance are marked above. WCS-YOLOv8s performs well in various complex scenes, which fully proves the effectiveness of the method of this embodiment. The experimental results show that the WCS-YOLOv8s model performs well in improving strawberry picking efficiency and reducing damage, with a detection speed of 45.9fps. Although the speed has dropped slightly, it still meets the actual application requirements and has increased by 1.94% and 1.61% on the map@0.5 and map@0.5:0.95 indicators respectively. The model proposed in the present invention provides a reliable detection and positioning method for the automated management and picking of strawberries, and also provides technical support for the automated management of orchards. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The photos and annotations in the embodiments of the present invention are as follows; Figure 2 It is a schematic diagram of the WCS-YOLOv8s network structure of the present invention; Figure 3 This is a schematic diagram of the CGFM module proposed in the present invention; Figure 4Schematic diagram of the SE-MSDWA module proposed by the present invention Figure 5 It is a heat map of part of the data in the embodiment of the present invention; among them, (a-1), (b-1), and (c-1) are original image data; (a-2), (b-2), and (c-2) are heat maps generated by the Grad-CAM algorithm. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0013] like Figure 2 As shown, this embodiment discloses a strawberry binocular detection and positioning method based on the YOLOv8 fusion CGFM network model, which specifically includes the following steps: Step 1, the data set used in this embodiment covers different varieties of strawberries. Mobile phones and Intel D435i cameras were used during the collection process, and the shooting distance was kept between 0.3-0.8 meters to ensure image quality and the diversity of different distances and angles. The collection time period of the data set is 07:00-19:00, including various weather conditions. A total of 1957 pictures were collected, and each image was adjusted to 640x640 pixels to construct a data set for strawberry with different maturity and thinning of flowers, fruits and vegetables. The depth information of the binocular camera can not only obtain the category and position of the target, but also capture the depth information of the object, providing a reliable solution for automated management and picking. The data set is randomly divided into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0014] The self-built data set of this embodiment was collected at the Hongshiya Strawberry Planting Base in Huangdao District, Qingdao City, Shandong Province (121.49°E, 36.77°N). The base was established in 1991 and has more than 360 planting sheds covering an area of ​​more than 500 hectares. The data set was taken in the planting sheds of different farmers, eliminating the data bias caused by geographical location and variety.

[0015] The dataset includes four categories: immature, mature, bud, and flower. Each image is labeled under the guidance of the farmer, such as Figure 1 As shown in the figure, two rounds of confirmation were performed to ensure the accuracy of the dataset. These images were taken in a variety of environments and lighting conditions, including bright and dim light, complex environments and backgrounds, so the dataset is versatile and can be used for strawberry maturity and flower detection and recognition tasks in various environments.

[0016] This embodiment uses binocular ranging technology in thinning flowers, vegetables and fruits and automatic picking. It is similar to the principle of human eyes. It observes the same object from two positions and calculates the distance using the parallax angle and parallax baseline, thereby realizing early warning and braking of obstacles without judging the type of obstacles. The advantages of binocular cameras include: moderate cost, lower than laser radar; no viewing angle limitation, directly measuring the distance of all objects; higher accuracy than monocular; no need to maintain a sample database.

[0017] Step 2: Build the WCS-YOLOv8s model, use the training set obtained in step 1 to train the model, and use the validation set to test the performance of the WCS-YOLOv8s model based on the cross-validation method.

[0018] YOLOv8 uses an end-to-end architecture to extract and fuse features of the input image through a convolutional neural network for subsequent regression and prediction of target categories and locations. The YOLOv8 framework is divided into four main components: input layer, backbone network, neck network, and prediction layer.

[0019] The input layer establishes an image processing strategy, including reading images and labels, and performing corresponding data enhancement. The backbone network uses convolution operations to extract feature maps at various levels of abstraction, and adopts cross-stage feature fusion to minimize parameter redundancy and improve model accuracy. The neck network inherits the feature pyramid network, enriches feature maps in a top-down approach, and integrates a path aggregation network to achieve bottom-up feature enhancement. The prediction layer uses three different detection heads to predict the target category probability and position in three different scale feature maps, thereby effectively meeting the detection needs of small, medium and large objects in the image.

[0020] This embodiment has been improved on the basis of YOLOv8s. First, Warmup data enhancement is proposed, that is, the original data enhancement strategy is changed to gradually increase the probability of data enhancement as the epoch changes. Secondly, the self-developed SE-MSDWA block is applied at the end of the backbone network to achieve efficient feature extraction, so that the model focuses on the strawberry area. Finally, the neck network is improved and the CGFM module proposed in this embodiment is used to improve the feature fusion performance of the network. Based on the above content, a model for detecting strawberries of different maturity and thinning flowers and fruits is proposed, called WCS-YOLOv8s. The network framework of WCS-YOLOv8s is as follows: Figure 2 shown.

[0021] Data enhancement reduces the need for raw data and annotations, while improving the adaptability and robustness of the model to changes in input data (such as noise, size differences, and lighting conditions). In order to improve the robustness and generalization ability of the model under individual strawberry posture changes and lighting conditions, this example uses five image processing techniques for data enhancement, namely rotation, brightness adjustment, adding Gaussian noise, contrast adjustment, and image occlusion. These methods are flexibly combined and applied to enhance the data set online.

[0022] The learning rate is one of the most important hyperparameters in neural networks. There are many ways to optimize the learning rate, and Warmup is one of them. Warmup was first mentioned in ResNet. It uses a smaller learning rate at the beginning of training, and then switches to a pre-set learning rate for training after a few epochs of training. Since the weights of the model are randomly initialized at the beginning of training, if a larger learning rate is selected at this time, it may cause model instability (oscillation). Using warmup to warm up the learning rate can make the model gradually stabilize at a smaller learning rate within the first few epochs of training. After the model is relatively stable, it can be trained using the pre-set learning rate, thereby accelerating the convergence of the model and improving the model effect.

[0023] According to the idea of ​​setting the learning rate, this embodiment uses different probabilities for data enhancement throughout the training process, which can bring many benefits. The use of lower probability data transformation in the early stage helps the model learn the relationship between samples, improves its adaptability to different data distributions, and allows the model to smoothly enter the training process to avoid falling into a local optimal solution. Gradually increasing the probability of sample transformation as training progresses can further enhance the generalization ability of the model, promote more robust feature representation, speed up the training process, and improve efficiency. Through this strategy, the model can effectively learn and generalize while improving its robustness and training efficiency. The method of this embodiment is to use a smaller probability for data enhancement in the early stage of training. When the training reaches 1 / 5 of the total number of rounds, all data enhancements will be performed in the default mode of YOLOv8. The formula for the change of data enhancement probability is as follows:

[0024] Where: Current_epoch is the current round number of the training process; Total_epochs is the total number of rounds of the training process.

[0025] Concat has its limitations and shortcomings in deep learning. It concatenates features from different sources without actually fusing them, and may not fully utilize feature information from different layers. Therefore, the Concat operation may not fully utilize the complementary effects of different features. To overcome these shortcomings, this embodiment proposes a feature fusion structure CGFM based on a self-attention mechanism to better fuse features from different sources and improve the performance and efficiency of the model.

[0026] Context Guide Fusion Module (CGFM) is an innovative feature fusion module designed to improve the Feature Pyramid Network (FPN) in YOLOv8. The module design takes into account the guidance and adaptive adjustment of contextual information in the process of multi-scale feature fusion. Through the multi-head self-attention mechanism, CGFM can capture and utilize important contextual information in the process of feature fusion through weighted feature reorganization operations, selectively emphasize informative features, and suppress irrelevant features, thereby improving the effectiveness of feature representation and the detection accuracy of the model.

[0027] The CGFM module generates attention weights by concatenating two different feature maps in the channel dimension and then inputting them into a multi-head self-attention mechanism module containing convolution. These weights are applied to the original features through element-wise multiplication to enhance the representation of the features. CGFM uses the multi-head self-attention mechanism to enhance important features and suppress unimportant features, thereby improving the discriminability and visual performance of the fused features through detail enhancement.

[0028] This embodiment generates query, key and value matrices through 1x1 convolutional layers, effectively processes local image features, retains spatial information, and improves computational efficiency through a shared weight mechanism. Compared with the traditional Transformer method of flattening image sequences, the module in this embodiment directly processes two-dimensional feature maps, reducing the number of parameters and computational complexity, and improving model efficiency. The detailed structure of CGFM is shown in Figure 2. Figure 3 shown.

[0029] The SE-Enhanced Multi-Scale Depthwise Attention (SE-MSDWA) module aims to improve the feature extraction capability and overall performance of convolutional neural networks by combining deep separable convolution, multi-scale convolution, and Squeeze-and-Excitation (SE) blocks. The purpose of designing this module is to address the shortcomings of traditional convolutional layers in capturing multi-scale features, while dynamically adjusting the channel weights in the feature map through the SE block to highlight important features. The advantage of this module is that it can effectively capture feature information of different scales, and through deep separable convolution, it reduces the computational complexity and the number of parameters, while enhancing the feature representation capability.

[0030] Specifically, the SE-MSDWA module first performs convolution operations on each channel independently through deep convolution to extract important spatial information, and then exchanges information between channels through point-by-point convolution. Secondly, the module uses three sets of convolution kernels of different scales to perform multi-scale convolution processing: (1, 5) and (5, 1) convolution kernels capture smaller range features, (1, 9) and (9, 1) convolution kernels capture medium range features, and (1, 17) and (17, 1) convolution kernels capture larger range features. After these convolution processes, the feature map is fused with multi-scale information through an additional convolution layer and finally enters the SE block. The SE block first performs adaptive average pooling to reduce the feature map of each channel to 1x1, and then calculates the weight of each channel through two fully connected layers and activation functions, and reapplies these weights to the original feature map. The specific implementation is as follows: Figure 4 shown.

[0031] The SE-MSDWA module effectively solves the shortcomings of traditional convolutional layers in capturing multi-scale features and processing feature redundancy. By dynamically adjusting channel weights, it significantly enhances the network's feature representation capabilities, thereby improving the model's performance in various computer vision tasks.

[0032] The SE-MSDWA module solves the key problems of traditional convolutional neural networks in feature extraction and representation through innovative design and functional improvement in many aspects. The module introduces multi-scale convolution kernels, which can effectively capture feature information of different scales and improve the network's processing capabilities for objects of various sizes and complex scenes; it adopts deep separable convolution to improve feature extraction capabilities while reducing the amount of calculation and parameters; it dynamically adjusts the weights of each channel through the SE block to highlight important features, suppress irrelevant information, and enhance the effectiveness of feature representation; it combines multi-scale convolution and SE blocks to generate rich and meaningful feature representations, improving the recognition and classification accuracy of the network; and it enhances the adaptability of the network in different scenarios and tasks through multi-scale feature extraction and channel attention mechanisms.

[0033] Based on the above classification method, a series of experiments were conducted, and the model proposed in this embodiment was compared with the model in the prior art to obtain a data graph. The specific analysis is as follows: Figure 5The heat map of some data; (a-1, b-1, c-1) is the original image data, and (a-2, b-2, c-2) is the heat map generated by the Grad-CAM algorithm. By comparing the original image and the model heat map, we can see that it makes the model's decision-making process more transparent and improves interpretability by highlighting the image areas that the model considers most important in the image classification task. This visualization can not only help researchers identify erroneous or irrelevant features that the model may rely on, but also provide guidance for model improvement, thereby designing better architectures or data enhancement methods.

[0034] In order to verify the advantages of the algorithm in this embodiment, this embodiment compares the optimization scheme with the current mainstream algorithm, and the results are shown in Table 1. Compared with the Rtmdet-tiny algorithm, the average precision of the WCS-YOLOv8s model is improved by 8.13. Although the detection frame rate (fps) of the WCS-YOLOv8s model is slightly lower than that of other models, its model parameters are the smallest and suitable for actual strawberry picking. Considering all parameters, the overall effect of the WCS-YOLOv8s model is superior.

[0035] Table 1 Comparison of rotten strawberry classification performance of different methods This example uses the proposed Warmup data enhancement method and uses the CGFM module and SE-MSDWA block to set up four groups of experiments to ensure the feasibility of the optimization scheme. The results are shown in Table 2.

[0036] First, the Warmup data enhancement strategy was added to the baseline model. While keeping the model structure unchanged, this strategy achieved an increase of 1.02% in mAP@0.5:0.95 and 2.3% in recall without increasing the amount of computation. Secondly, the CGFM module was introduced into the Neck structure, which increased mAP@0.5:0.95 by 0.9%. Subsequently, the SE-MSDWA module was added to the backbone network, which brought an increase of 0.86% in mAP@0.5 and 1.1% in Precision. Finally, WCS-YOLOv8 improved mAP@0.5 by 1.94%, mAP@0.5:0.95 by 1.61%, and recall by 2.4%. These performance improvements were intuitively reflected in subsequent tests, effectively reducing the missed detection and false detection of the baseline model in complex situations, and providing theoretical support for the automated management and picking of strawberries. As can be seen from Table 1, the WCS-YOLOv8s model uses only 42.78% of the YOLOv8l parameters, but achieves a higher mAP@0.5 index, proving the effectiveness of the improved strategy.

[0037] Table 2 Ablation experiment The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A strawberry binocular detection and positioning method based on YOLOv8 fused with CGFM network model is characterized in that: The steps include: Step 1: Collect photos of immature, mature, buds and flowers, annotate the photos, and construct a dataset of strawberry with different maturity and flower and fruit thinning detection; Step 2: randomly divide the data set obtained in step 1 into a training set, a validation set, and a test set; Step 3, build a WCS-YOLOv8s model, use the training set obtained in step 1 to train the model, use the validation set to test the performance of the WCS-YOLOv8s model based on the cross-validation method, and finally obtain a trained WCS-YOLOv8s model; Step 4: Collect the photos to be detected and input them into the WCS-YOLOv8s model trained in step 3 to detect and locate the strawberries; The WCS-YOLOv8s model in step 3 includes an input layer, a backbone network, a neck network, and a prediction layer. The input layer establishes an image processing strategy and uses a Warmup data enhancement module for data enhancement. The SE-MSDWA module is applied in the backbone network for efficient feature extraction. The CGFM module is used in the neck network to improve the feature fusion performance of the network. The prediction layer uses three different detection heads to predict the target category probability and position in three feature maps of different scales.

2. The strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model according to claim 1, characterized in that, In the step 1, photos are collected by binocular ranging technology.

3. The strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model according to claim 1, characterized in that, The Warmup data enhancement module gradually increases the probability of data enhancement as the number of rounds changes. In the early stage of model training, the Warmup data enhancement module selects a smaller learning rate to help the model gradually stabilize at a smaller learning rate. When the training reaches 1 / 5 of the total number of rounds, data enhancement is performed in the default mode of YOLOv8. The formula for the change of data enhancement probability is: Among them, Current_epoch is the current epoch number of the training process, and Total_epochs is the total number of epochs of the training process.

4. The strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model according to claim 1, characterized in that, The CGFM module concatenates two different feature maps in the channel dimension and inputs them into a multi-head self-attention mechanism module containing convolution to generate attention weights. The generated attention weights are applied to the original features through element-level multiplication to enhance the representation ability of the features.

5. The strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model according to claim 1, characterized in that, The SE-MSDWA module combines a multi-scale convolutional layer, a depth-wise separable convolutional layer, and an SE module. The multi-scale convolutional layer is used to capture feature information of different scales. The depth-wise separable convolutional layer reduces the amount of calculation and parameters while improving the feature extraction capability. The SE module is used to dynamically adjust the weights of each channel, highlight important features, and enhance the effectiveness of feature representation.

6. The strawberry binocular detection and positioning method based on YOLOv8 fusion CGFM network model according to claim 5 is characterized in that, The processing process of the SE-MSDWA module includes: performing convolution operations on each channel independently through a multi-scale convolution layer to extract important spatial information; performing information interaction between channels through point-by-point convolution; using three sets of convolution kernels of different scales for multi-scale convolution processing. After convolution processing, the depth-separable convolution layer decomposes the n*n convolution into n*1 convolution and 1*n convolution. The feature map is fused with multi-scale information through an additional convolution layer and finally enters the SE module. The SE module first performs adaptive average pooling to reduce the feature map of each channel to 1x1, and then calculates the weight of each channel through two fully connected layers and activation functions, and reapplies these weights to the original feature map.

Images