Strawberry target detection method based on convolutional neural network

Abstract

The invention discloses a strawberry target detection method based on a convolutional neural network, and belongs to the technical field of image processing, and the method comprises the steps: constructing a strawberry data set through strawberry image collection, and carrying out the preprocessing operation of data annotation, data enhancement, data division and the like, so as to improve the quality of the strawberry data set; a strawberry target detection model based on the improved YOLOv5 is constructed, a C3 module in a backbone network is replaced by a CSP structure based on SENetV2, a conventional up-sampling module in a neck network is replaced by a lightweight dynamic up-sampler DySample, and a Focaler-IOU loss function is introduced; network weight parameters are obtained through model training, and target prediction is achieved. The strawberry shape can be accurately judged in a complex environment, and automatic strawberry picking and intelligent monitoring are assisted.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a strawberry target detection technology based on a convolutional neural network. Background Technique

[0002] With the rapid development of Internet of Things and artificial intelligence technologies, the strawberry facility cultivation industry is transforming towards the intelligent direction, including fields such as intelligent control of strawberry greenhouses, automatic strawberry picking, strawberry pest and disease monitoring, and intelligent thinning of strawberry flowers and fruits. Currently, in strawberry cultivation management in China, it is mainly manual, with high labor costs and being greatly affected by human subjectivity. Strawberry target detection refers to the use of image processing and deep learning technologies to achieve the classification and state recognition of targets such as flowers, fruits, inflorescences, and calyces at different growth stages of strawberries, which can provide accurate target state parameters for fields such as strawberry maturity detection, automatic picking, and intelligent thinning of flowers and fruits, and is of great significance for intelligent detection and automatic control of strawberries.

[0003] With the rapid development of deep learning technologies, in order to better meet the needs of intelligent strawberry monitoring, many scholars have studied strawberry target detection technologies. Liu Mochen et al. studied the strawberry target detection technology for elevated greenhouse strawberries, analyzed the characteristics of red-ripe strawberries and fruit stalks, and proposed an improved YOLOv8-Pose model recognition method based on an attention mechanism. By integrating the Slim-neck module and the CBAM attention mechanism, the detection ability of small targets was improved, and the detection accuracy of strawberries and the detection accuracy of fruit stalk key points in complex environments such as occlusion and light changes were effectively improved. Zhang Huiyao et al. studied the strawberry maturity detection technology using a convolutional neural network, proposed an improved algorithm of the YOLOv5s model based on the Swin Transformer module, and integrated the CA attention mechanism and the WIoU loss function to improve the feature extraction ability of the model. Huang Jiacai et al. studied the strawberry target detection technology in complex environments, proposed a YOLOv5s model recognition method based on the MobileNetV3 backbone network and the Alpha-IoU loss function, and optimized the prior box size by clustering to improve the detection ability of small targets. The improved model significantly improved the speed and accuracy of strawberry target detection. Li Hongdan et al. proposed an improved YOLOv7 model strawberry maturity detection method based on PConv head convolution and an attention mechanism. By using the CARAFE operator, the detail perception ability of the model's upsampling was improved, and the method used could better balance the speed and accuracy of target detection. Wang Ruibin et al. studied the mature strawberry fruit target recognition technology, proposed an improved algorithm of the SSD model based on the deep residual network and the CBAM attention mechanism, and optimized the calculation efficiency of the loss function by combining the predicted box position loss and the confidence loss, effectively improving the target detection accuracy of the model.

[0004] In summary, the main problems of existing strawberry object detection are as follows: current strawberry object detection mainly focuses on maturity detection and cannot cover multi-object detection in the entire growth cycle of strawberries. At the same time, there are still problems such as low detection accuracy of the model in complex environments in current research. Summary of the Invention

[0005] Object of the Invention: The object of the present invention is to provide a strawberry object detection method based on a convolutional neural network, which solves the problem of high-precision detection of multiple types of strawberry objects in existing complex environments.

[0006] Technical Solution: To achieve the above object, the present invention provides the following technical solution:

[0007] A strawberry object detection method based on a convolutional neural network, characterized by comprising the following steps:

[0008] S1. Obtain images collected at different angles, different distances, and different light intensities through a collection device, and construct a strawberry data set.

[0009] S2. Preprocess the data set, including processes such as data annotation, data augmentation, and data cleaning, and randomly divide the processed strawberry data set into a training set, a validation set, and a test set according to a certain ratio.

[0010] S3. Construct a strawberry object detection model based on a convolutional neural network, including improving the original YOLOv5 model, and the improvements include:

[0011] 1) Based on the YOLOv5 as the basic model, replace the C3 module in the backbone network with a CSP structure based on SENetV2 to enhance the feature expression ability of the model.

[0012] 2) Introduce a lightweight dynamic upsampler DySample in the neck network to improve the running efficiency of the model.

[0013] 3) In the loss function part, introduce the Focaler-IOU loss function to effectively solve the distribution problem of difficult and simple samples and improve the detection accuracy of the bounding box.

[0014] S4. Use the data set to train the improved YOLOv5 network model.

[0015] S5. Use the test set to test and evaluate the improved YOLOv5 network model.

[0016] S6. Use the improved YOLOv5 network model to perform strawberry object detection.

[0017] Further, in step 1, the steps of constructing the strawberry dataset include strawberry image collection and web collection. Image data in the strawberry greenhouse is collected through a mobile phone, including sample data under different lighting conditions, and it is required to cover situations such as strawberry occlusion and overlap. The images obtained by the web crawler tool need to be screened and processed.

[0018] Further, the data preprocessing steps in step 2 include:

[0019] 1) Data annotation. Using the Labelimg software, considering the distribution of the number of samples of different types of strawberries, the strawberry targets are divided into 5 types, namely flowers, flower fruits, inflated fruits, white-red fruits, and red fruits. Among them, the flower fruit is the stage when the strawberry flower has not withered and the young fruit has just grown, the inflated fruit is the intermediate stage when the strawberry is in the stage from flower withering to fruit color change, and the fruit with a red coloring area greater than three-quarters is a red fruit.

[0020] 2) Data augmentation. By methods such as flipping, rotating, scaling, cropping, color transformation, Mosaic, etc. on the images, the dataset is enriched.

[0021] 3) Data cleaning. The enhanced images are checked to identify and correct errors, incomplete, inconsistent or duplicate information in the data.

[0022] Further, the YOLOv5 network model in step 3 mainly includes three parts: the backbone network, the neck network, and the head network. The model improvement steps include:

[0023] 1) Based on YOLOv5 as the base model, the C3 module in the backbone network is replaced with a CSP structure based on SENetV2 to improve the feature expression ability of the model. The SENetV2 module enhances the channel representation ability of the model through squeezing and excitation operations, and draws on the design structure of the aggregation layer. By splicing the outputs of multiple fully connected layer branches, a richer global feature representation is obtained. The calculation formula of SENetV2 is as follows:

[0024] Y = F(x · Ex∑Sq(x)) (I);

[0025] Where the input is represented as x, F(x) is the feature processing within the residual structure, and the Sq function represents the squeezing operation.

[0026] 2) Introduce a lightweight dynamic upsampler DySample in the neck network to improve the running efficiency of the model. DySample adaptively determines the initial sampling position according to the content and structure of the input feature map, sets the offset range, so as to capture important feature information. Using the generated content-aware offset, DySample performs an upsampling operation based on point sampling. This method based on point sampling can utilize feature information more effectively while reducing unnecessary computational volume. Given a feature map χ of size C×H 1 ×W 1 and a point sampling set δ of size 2g×H 2 ×W 2 , where 2g in the first dimension represents the coordinates of x and y. The grid_sample function uses the positions in the point sampling set δ to resample the map χ to χ' of size C×H 2 ×W 2 , as shown in the following formula.

[0027] χ' = grid_sample(χ,δ) (II);

[0028] Given the upsampling scale factor s and the feature map χ of size C×H×W, use a linear layer with input and output channels of C and 2gS 2 respectively to generate an offset O of size 2g 2 ×H×W. Then reshape it through pixel recombination into a high-resolution original sampling grid G of size 2g×sH×sW, and the point sampling set δ is the sum of the offset O and the original sampling G, as shown in the following formula.

[0029] δ = O + G (III);

[0030] 3) In the loss function part, introduce the Focaler-IOU loss function to effectively solve the distribution problem of difficult and easy samples and improve the bounding box detection accuracy. Focaler-IOU is defined as shown in the following formula.

[0031]

[0032] where, f IoU loss is the intersection over union of the candidate box and the ground truth box, which is 0.5, and, d and u are adjustable parameters, {d,u} ∈ [0,1], and the calculation method of the Focaler-IOU loss function is as shown in the following formula.

[0033] L Focaler-IoU = 1 - f Focaler-IoU (V);

[0034] Further, in step 4, the model adopts transfer learning method during training and uses a phased training mode. During the training process, the SGD optimizer is adopted, and the cosine annealing decay method is used for the iteration of the learning rate.

[0035] Further, the model test evaluation metrics in step 5 include precision, recall, average precision (AP), and mean average precision (mAP), etc. The precision standard represents the proportion of positive samples in all datasets determined to be positive samples. The recall rate refers to the proportion of positive samples in the actual positive sample datasets that are finally determined to be positive. AP is a comprehensive evaluation index for measuring precision and recall, and mAP is the average value of AP for each category. The following are the calculation formulas:

[0036]

[0037] Among them, TP is the case where the target in the dataset is actually a positive sample, FP is the case where negative sample data is misjudged as a positive sample, and FN is the case where positive samples are misjudged as negative samples.

[0038] Further, in step 6, the constructed strawberry target detection model and the trained weight parameters are used for online recognition of strawberry targets.

[0039] Beneficial effects: A strawberry target detection method based on convolutional neural network disclosed by the present invention is based on the existing YOLOv5 network model, introduces the CSP structure based on SENetV2, and improves the feature expression ability of the backbone network. By introducing a lightweight dynamic upsampler DySample in the neck network, the operation efficiency of the model is improved; by introducing the Focaler-IOU loss function, the distribution problem of difficult and simple samples is effectively solved, and the detection accuracy of the bounding box is improved. The present invention can realize strawberry target recognition in complex environments, which is of great significance for realizing intelligent management of strawberries. Description of the Drawings

[0040] Figure 1 is the flow chart of the detection method of the present invention;

[0041] Figure 2 is the improved YOLOv5 model architecture diagram;

[0042] Figure 3 is the SENetV2 structure diagram;

[0043] Figure 4 is the C3_SENetV2 structure diagram;

[0044] Figure 5 is the dynamic upsampler structure diagram of the DySample module;

[0045] Figure 6It is a figure generated from a point sampling set. Specific implementation manner

[0046] To better understand the content of this invention patent, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0047] As Figure 1 shown, a strawberry target detection method based on a convolutional neural network includes the following steps:

[0048] Step 1: Obtain images collected at different angles, different distances, and different light intensities through a collection device to construct a strawberry data set.

[0049] Step 2: Preprocess the data set, including data annotation, data augmentation, data cleaning, and divide the data into a training set, a validation set, and a test set.

[0050] Step 3: Construct a strawberry target detection model based on a convolutional neural network, including improving the original YOLOv5 model. The improvements include:

[0051] Step 4: Use the data set to train and evaluate the improved YOLOv5 network model.

[0052] Step 5: Use the test set to test and evaluate the improved YOLOv5 network model;

[0053] Step 6: Use the improved YOLOv5 network model to perform strawberry target detection.

[0054] In step 1, the steps of constructing the strawberry data set include strawberry image collection and network collection. Image data in the strawberry greenhouse is collected through a mobile phone, including sample data under different lighting conditions, and it is required to cover situations such as strawberry occlusion and overlap. During the process of taking pictures with the mobile phone, the distance from the strawberry ridge is about 20 cm, and the height is about 35 cm. Images of strawberry growth under different lighting environments are collected. The images obtained by the web crawler tool need to be screened and processed.

[0055] In step 2, preprocess the data set, including processes such as data annotation, data augmentation, and data cleaning, and randomly divide the processed strawberry data set into a training set, a validation set, and a test set according to a certain ratio. Specifically, it includes the following steps:

[0056] 2.1) Data annotation. Using the Labelimg software, considering the distribution of different types of strawberry samples, the strawberry shapes are divided into 5 types, namely conical, spindle-shaped, wedge-shaped, other shapes, and non-pickable types. Among them, unripe strawberries are labeled as non-pickable types, and the other types are all ripe strawberries. In the shape classification of strawberries, other shapes include all other shapes of ripe strawberries except the classified types. In addition, it also includes situations where the strawberry shape cannot be accurately judged due to overlap, occlusion, etc.

[0057] 2.2) Data augmentation. Enrich the dataset by flipping, rotating, scaling, cropping, color transformation, Mosaic, etc. on the images.

[0058] 2.3) Data cleaning. Check the augmented images and identify and correct errors, incomplete, inconsistent, or duplicate information in the data.

[0059] 2.4) To better evaluate the detection effect of the model, the processed strawberry dataset is randomly divided into a training set, a validation set, and a test set according to the ratio of 9:1:1.

[0060] In step 3, the YOLOv5 network model mainly includes three parts: the backbone network, the neck network, and the head network. As Figure 2 shown, in the backbone network part, the CSP structure based on SENetV2 is introduced to replace the original C3 module. In the neck network part, the dynamic upsampler DySample is introduced to replace the original nearest neighbor upsampler. In terms of the loss function, the Focaler-IOU loss function with higher bounding box detection accuracy is introduced. The specific model improvements include the following steps:

[0061] 3.1) In terms of the backbone network, based on YOLOv5 as the basic model, the C3 module in the backbone network is replaced with the CSP structure based on SENetV2 to improve the feature expression ability of the model. As Figure 3 shown, the SENetV2 module enhances the channel representation ability of the model through compression and excitation operations, and draws on the aggregation layer design structure. By splicing the outputs of multiple fully connected layer branches, a richer global feature representation is obtained. The calculation formula of SENetV2 is as follows:

[0062] Y = F(x · Ex∑Sq(x)) (I);

[0063] where the input is represented as x, F(x) is the feature processing within the residual structure, and the Sq function represents the squeezing operation.

[0064] As Figure 4As shown in the figure, an SENetV2 module is added to the Bottleneck structure in the C3 module of the backbone network to improve the feature extraction ability of the C3 module.

[0065] 3.2) A lightweight dynamic upsampler DySample is introduced in the neck network, which improves the running efficiency of the model. DySample adaptively determines the initial sampling position and sets the offset range according to the content and structure of the input feature map, so as to capture important feature information. Using the generated content-aware offset, DySample performs an upsampling operation based on point sampling. This point-sampling method can more effectively utilize feature information while reducing unnecessary computational volume. The dynamic sampling process of the DySample module is as Figure 5 shown. Given a feature map χ of size C×H 1 ×W 1 and a point sampling set δ of size 2g×H 2 ×W 2 , where 2g in the first dimension represents the coordinates of x and y. The mapping χ is resampled to χ' of size C×H 2 ×W 2 using the positions in the point sampling set δ by the grid_sample function, as shown in the following formula.

[0066] χ' = grid_sample(χ,δ) (II);

[0067] The calculation method of the point sampling set δ is as Figure 6 shown. Given the upsampling scale factor s and a feature map χ of size C×H×W, a linear layer with input and output channels of C and 2gS 2 respectively is used to generate an offset O of size 2g 2 ×H×W. Then, it is reshaped into a high-resolution original sampling grid G of size 2g×sH×sW through pixel recombination. The point sampling set δ is the sum of the offset O and the original sampling G, as shown in the following formula.

[0068] δ = O + G(III);

[0069] 3.3) In the loss function part, the Focaler-IOU loss function is introduced, which effectively solves the distribution problem of difficult and easy samples and improves the bounding box detection accuracy. Focaler-IOU is defined as shown in the following formula.

[0070]

[0071] where, f IoUThe loss is the intersection over union of the candidate box and the ground truth box, which is 0.5. Here, d and u are adjustable parameters, where {d, u} ∈ [0, 1]. The calculation formula of the Focaler-IOU loss function is shown as follows.

[0072] L Focaler-IoU = 1 - f Focaler-IoU (V);

[0073] In step 4, the model training environment is the Windows 10 operating system, and an NVIDIA RTX 2070 graphics card with a video memory size of 8GB is used. To improve the model training efficiency and considering that the backbone network of the model remains unchanged, a transfer learning training method is adopted. The model is trained for 350 rounds in total. In the first 50 rounds, the backbone feature network of the model is frozen, and the batch size is set to 16. In the following 300 rounds, the frozen network is unfrozen for overall training, and the batch size is set to 8. The SGD optimizer is used during the model training process, and the cosine annealing decay method is used for the iteration of the learning rate. The initial learning rate is 1e-2, and the minimum learning rate is 1e-4.

[0074] In step 5, the model test evaluation metrics include precision, recall, average precision (AP), and mean average precision (mAP), etc. The precision criterion represents the proportion of positive samples in all datasets determined to be positive samples. The recall rate refers to the proportion of positive samples in the actual positive sample datasets that are finally determined to be positive. AP is a comprehensive evaluation metric that measures precision and recall, and mAP is the mean of the APs of each category. The following are the calculation formulas:

[0075]

[0076] Among them, TP represents the situation where the target in the dataset is actually a positive sample, FP represents the situation where negative sample data is misjudged as a positive sample, and FN represents the situation where positive samples are misjudged as negative samples.

[0077] In step 6, the constructed strawberry target detection model and the trained weight parameters are used for online recognition of strawberry targets.

[0078] The above is only the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, adjustments can also be made to each implementation position, and these adjustments should also be regarded as the protection scope of the present invention.

Claims

1. A strawberry target detection method based on convolutional neural network, characterized in that, The following steps are involved: S1. Use the acquisition device to obtain images collected at different angles, distances and light intensities to build a strawberry dataset. S2. Preprocess the data set, including data labeling, data enhancement, data cleaning, and divide the data into training set, validation set, and test set. S3. Construct a strawberry target detection model based on a convolutional neural network, including improving the original YOLOv5 model. The improvements include: 1) Taking YOLOv5 as the basic model, the C3 module in the backbone network is replaced with the CSP structure based on SENetV2 to improve the feature expression ability of the model. 2) A lightweight dynamic upsampler DySample is introduced into the neck network to improve the operation efficiency of the model. 3) In the loss function part, the Focaler-IOU loss function is introduced to effectively solve the distribution problem of difficult and simple samples and improve the bounding box detection accuracy. S4. Use the dataset to train the improved YOLOv5 network model. S5. Use the test set to test and evaluate the improved YOLOv5 network model; S6. Use the improved YOLOv5 network model to detect strawberry targets.

2. A strawberry target detection method based on a convolutional neural network according to claim 1, characterized in that: The steps of constructing the strawberry dataset in step S1 include: 1) Collect image data in the strawberry greenhouse through mobile phones, including sample data under different lighting conditions, which must be able to cover situations such as strawberry occlusion and overlap. 2) Use web crawler tools to obtain strawberry data images and perform screening and processing.

3. A strawberry target detection method based on convolutional neural network according to claim 1, characterized in that: The data preprocessing step in step S2 includes: 1) Data labeling. Labelimg software was used to divide strawberry targets into five types, taking into account the distribution of the number of samples of different types of strawberries, namely flowers, flower fruits, expanded fruits, white-red fruits, and red fruits. Flower fruits refer to the stage when the strawberry flowers have not yet withered and the young fruits have just grown. Expanded fruits refer to the intermediate stage from flower withering to fruit color change. Fruits with a red coloring area greater than three-quarters are red fruits. 2) Data enhancement: Enrich the data set by flipping, rotating, scaling, cropping, color conversion, mosaicing, etc. 3) Data cleaning: Check the enhanced images to identify and correct errors, incompleteness, inconsistency or duplication in the data.

4. A strawberry target detection method based on convolutional neural network according to claim 1, characterized in that: In step S3, the YOLOv5 network model mainly includes three parts: backbone network, neck network and head network. The model improvement steps include: 1) Using YOLOv5 as the basic model, the C3 module in the backbone network is replaced with a CSP structure based on SENetV2 to improve the model's feature expression ability. The SENetV2 module uses squeezing and excitation operations to enhance the model's channel representation ability, and draws on the design structure of the aggregation layer to obtain a richer global feature representation by splicing the outputs of multiple fully connected layer branches. The calculation formula of SENetV2 is as follows: Y = F(x·Ex∑Sq(x)) (I); The input is represented as x, F(x) is the feature processing within the residual structure, and the Sq function represents the squeezing operation. 2) A lightweight dynamic upsampler DySample is introduced in the neck network to improve the operating efficiency of the model. DySample adaptively determines the initial sampling position and sets the offset range according to the content and structure of the input feature map. In order to capture important feature information. Using the generated content-aware offset, DySample performs an upsampling operation based on point sampling. This point sampling-based method can more effectively utilize feature information while reducing unnecessary computation. Given a feature map χ of size C×H1×W1 and a point sampling set δ of size 2g×H2×W2, where the first dimension 2g represents the coordinates of x and y. The grid_sample function uses the positions in the point sampling set δ to resample the map χ to χ' of size C×H2×W2, as shown in the following formula. χ'=grid_sample(χ,δ) (II); Given an upsampling factor s and a feature map χ of size C×H×W, we use C and 2gS as input and output channels respectively. 2 The linear layer generates a size of 2g 2 ×H×W offset O. Then through pixel reorganization, it is reshaped into a high-resolution original sampling grid G ​​of size 2g×sH×sW, then the point sampling set δ is the sum of the offset O and the original sampling G, as shown in the following formula. δ=O+G(III); 3) In the loss function part, the Focaler-IOU loss function is introduced to effectively solve the distribution problem of difficult and simple samples and improve the bounding box detection accuracy. The definition of Focaler-IOU is shown in the following formula. Among them, f IoU The loss is the intersection-over-union ratio between the candidate box and the real box, which is 0.5, while d and u are adjustable parameters, {d,u}∈[0,1]. The Focaler-IOU loss function is calculated as shown in the following formula. L Focaler-IoU =1-f Focaler-IoU (V)。 5. A strawberry target detection method based on convolutional neural network according to claim 1, characterized in that: In step S4, the model adopts transfer learning method in training and adopts phased training mode. The SGD optimizer is used in the training process, and the cosine annealing decay method is used to iterate the learning rate.

6. A strawberry target detection method based on convolutional neural network according to claim 1, characterized in that: In step S5, the model test evaluation indicators in step 5 include precision, recall, average precision (AP) and mean average precision (mAP). The precision standard refers to the proportion of all data sets judged as positive samples that are actually positive samples, and the recall rate refers to the proportion of data sets that are actually positive samples that are finally judged as positive. AP is a comprehensive evaluation indicator to measure precision and recall, and mAP is the average AP of each category. The following is the calculation formula: Among them, TP is the case where the target in the data set is actually a positive sample, FP is the case where the negative sample data is mistakenly judged as a positive sample, and FN is the case where the positive sample is mistakenly judged as a negative sample.

7. A strawberry target detection method based on convolutional neural network according to claim 1, characterized in that: In step S6, the constructed strawberry target detection model and the trained weight parameters are used to perform online recognition of strawberry targets.

Images