Method for detecting maturity of string tomatoes based on improved YOLOv5 network

Abstract

The present application relates to a kind of based on the detection method of string tomato maturity of improved YOLOv5 network, including the following steps: constructing string tomato data set and single small tomato data set;Based on the training of data set to improved YOLOv5 network, obtain weight model configuration;Construct string tomato detection model DetectTomato, it identifies out string tomato detection target in the image to be detected, and the string tomato detection target image and its position information are respectively stored in folder;Maturity detection model DetectMaturity is constructed, and string tomato maturity is calculated, and the position of string tomato detection target image corresponding to the highest maturity and reaching preset maturity standard value is output, determines the picking target, after each maturity detection, all empty folder.The present application can realize the accurate identification of string tomato, and further maturity detection is carried out, determines the picking target, improves string tomato maturity detection and picking accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and pattern recognition, and particularly relates to the technical field of string tomato maturity detection, and specifically to a string tomato maturity detection method based on an improved YOLOv5 network. BACKGROUND

[0002] Tomatoes are rich in nutrients, including carbohydrates, vitamin C, carotene, protein, and various nutrients. String tomatoes, also known as cluster tomatoes, are a new type of tomato variety that has gained popularity in domestic and international markets in recent years. They have a good market. The shelf life of ordinary soft fruit tomatoes is about 10 days, while string tomatoes can remain red on the cluster for more than 15 days without softening or cracking, with a shelf life of more than 20 days, and a longer shelf life. The growth cycle of string tomatoes is about three to four months, and they can be marketed throughout the year. Therefore, string tomato picking is a critical step in entering the market.

[0003] For string tomatoes grown in greenhouses, the current method is for planting personnel to conduct daily inspections and record the growth status of the fruit, including the fruit hanging situation, fruit size, fruit quantity, etc. This type of work is time-consuming and labor-intensive, and requires a large amount of labor cost. In the era of intelligent industry and agriculture, intelligent control is an important means to reduce costs and improve production efficiency.

[0004] In recent years, the rapid development of machine vision has also driven the development of intelligent production in the agricultural field. However, for the intelligent picking of string tomatoes in greenhouses, the image processing algorithm of the intelligent picking device will identify all the string tomatoes in the image, and the intelligent picking device can only pick one string tomato at a time. Therefore, it is necessary to solve the problem of efficient picking of a single string tomato in a complex scene. At the same time, for the whole string picking of string tomatoes, the quality level of the string tomatoes also needs to be considered. Only string tomatoes that meet the specified limits can be picked as a whole to reduce production losses. Therefore, the determination and identification of string tomatoes that meet the standards have become a problem that needs to be solved in the actual picking process. SUMMARY

[0005] To solve the above technical problems, the present application provides a string tomato maturity detection method based on an improved YOLOv5 network, which can accurately identify string tomatoes and further detect their maturity to determine the pickable target. Compared with traditional target detection algorithms, the present application provides a specific solution for the practical engineering of string tomato picking, with logic, completeness, and high accuracy.

[0006] The technical solution adopted by the present application is as follows:

[0007] A string tomato maturity detection method based on an improved YOLOv5 network, comprising the following steps:

[0008] Step 1: Construct a string tomato dataset and a single small tomato dataset composed of images of single small tomatoes at each growth stage in the string tomato;

[0009] Step 2: Train the string tomato dataset and the single small tomato dataset based on an improved YOLOv5 network respectively to obtain corresponding weight model configurations, wherein the improved YOLOv5 network replaces the convolution layer of the backbone layer of the YOLOv5 network model with GhostConv and replaces the Detect of the output head of the head layer with an ASFF adaptive feature fusion layer;

[0010] Step 3: Based on the improved YOLOv5 network, introduce a target position interception storage mechanism in the detect program to construct a string tomato detection model DetectTomato;

[0011] Step 4: The string tomato detection model DetectTomato calls the weight model configuration of the string tomato dataset to identify the string tomato detection target in the image to be detected, obtains the image and position information of all identified string tomato detection targets, and stores the string tomato detection target image and its position information in the exp folder and the label folder respectively;

[0012] Step 5: Based on the improved YOLOv5 network, introduce a statistical function and a target position output mechanism in the detect program to construct a maturity detection model DetectMaturity;

[0013] Step 6: The maturity detection model DetectMaturity calls the weight model configuration of the single small tomato dataset, traverses all string tomato detection target images stored in the exp folder, identifies single small tomatoes in each string tomato detection target image, calls the statistical function to count the number of small tomatoes in the same string tomato detection target image at the ripe stage, the half-ripe stage and the green-ripe stage, divides the number of ripe small tomatoes by the total number of small tomatoes to calculate the maturity of the string tomato, outputs the position of the string tomato detection target image corresponding to the label folder with the highest maturity and reaching the preset maturity standard value, thereby determining the picking target, and after completing the maturity detection of each string tomato detection target image, the label folder and the exp folder are emptied by calling the os library.

[0014] The beneficial effects of the present application include:

[0015] 1. Clearly single picking target: the application identifies the cluster tomatoes appearing in the image data through the cluster tomato detection model DetectTomato, and stores the identified single cluster tomato detection target image and position information, and then identifies and maturity detection of the single cherry tomato image through the maturity detection model DetectMaturity, clearly defines the final picking target, and provides an effective method for reducing production picking loss and picking fluency;

[0016] 2, Maturity classification of cherry tomato fruit: the growth period of cherry tomato fruit is divided into three periods, including green maturity period, half-ripe period and full-ripe period, the biggest difference among the three periods is the R and G components in RGB component, that is, red and green. The application draws R-G component statistical table of three growth periods, and the maturity detection model DetectMaturity divides the growth period based on the R-G component statistical table of three growth periods, which provides a scientific basis for maturity detection of cherry tomato fruit and improves the accuracy of cluster tomato maturity detection and picking.

[0017] 3. Light network model: in order to improve the training speed, the convolution layer of the backbone layer of YOLOv5 network model is replaced by GhostConv, and the Detect of the output head of the head layer is replaced by ASFF adaptive feature fusion layer, which reduces the training time by 13% while keeping the recognition accuracy almost unchanged, and improves the application value to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The flowchart of the improved YOLOv5 network-based cluster tomato maturity detection method of the application;

[0019] Figure 2 The improved YOLOv5 network model structure diagram in the application;

[0020] Figure 3 The flowchart of the cluster tomato detection model DetectTomato in the application;

[0021] Figure 4 The flowchart of the maturity detection model DetectMaturity in the application;

[0022] Figure 5 The cluster tomato recognition effect diagram of the cluster tomato detection model DetectTomato in the application;

[0023] Figure 6 The cluster tomato maturity detection effect diagram of the maturity detection model DetectMaturity in the application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described in detail below with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0025] Based on the recognition ability of YOLOv5, the network model in the training stage of YOLOv5 is improved, and a lightweight model is designed. The detect program of YOLOv5 is improved, and the target position interception storage mechanism, the statistical function and the target position output mechanism are introduced respectively, so as to realize twice detection of the cluster tomatoes in the greenhouse. The purpose of the first detection is to identify all the cluster tomatoes in the image data and determine the image position of the cluster tomatoes. The purpose of the second detection is to calculate the maturity of all the cluster tomatoes in the first detection, so as to determine the most suitable cluster tomato target for picking. The flow chart of the detection method of the present application is shown in Figure 1 The specific steps are as follows:

[0026] Step 1: Constructing cluster tomato data set and single small tomato data set, wherein the single small tomato data set is composed of images of single small tomatoes in each growth period of cluster tomatoes.

[0027] In this step, first, the equipment collects cluster tomato images in the greenhouse to obtain cluster tomato image data. Then, the cluster tomato image data is labeled by using the labeling tool MakeSense. After labeling, the cluster tomato data set is made, and the cluster tomato data set is divided into training set, validation set and test set according to a certain proportion. Due to the complex background of cluster tomatoes, the labeling tool MakeSense is used to label samples with short shooting distance. The number of cluster tomato images with slightly far distance is large, which is not easy to identify, and has no practical significance for actual picking. A total of 2000 images are collected and labeled. The 2000 images obtained are used as the training set of the cluster tomato data set, 500 images are taken from the training set as the validation set, and 300 images are newly shot and labeled as the test set. A total of 200 rounds of training are performed.

[0028] When constructing the single small tomato data set, the following steps are specifically included:

[0029] According to the collected images of individual cherry tomatoes at each growth stage, the R component and the G component of each image are obtained by opencv, and the R-G component statistical table of each growth stage is drawn. The difference between the green channel and the red channel is large for individual cherry tomatoes at different stages, so the growth stage of individual cherry tomatoes can be determined by the difference between the green channel and the red channel. Using 100 images of ripe cherry tomatoes, green cherry tomatoes and semi-ripe cherry tomatoes collected, the size of the green and red channel components of cherry tomatoes at each stage is obtained, the red channel component and the green channel component are subtracted, and the R-G histogram is obtained. By analyzing the images at each growth stage, the R-G component statistical table of each growth stage in Table 1 is obtained.

[0030] Table 1 R-G component statistical table of each growth stage

[0031]

[0032] Collect images of individual cherry tomatoes in a number of string tomatoes in the greenhouse. For the collected individual cherry tomato images, the corresponding R-G component map of each image is obtained by opencv. By comparing the R-G component map with the R-G component statistical table shown in Table 1, the growth stage division and classification annotation of each individual cherry tomato image is carried out using the annotation tool MakeSense. The same number of ripe cherry tomato images, green cherry tomato images and semi-ripe cherry tomato images are annotated, and all the ripe cherry tomato images, green cherry tomato images and semi-ripe cherry tomato images together form the individual cherry tomato dataset, thus the individual cherry tomato dataset is constructed. For example, 300 ripe cherry tomato images, 300 green cherry tomato images and 300 semi-ripe cherry tomato images are collected and annotated, and 900 images are obtained as the training set of the cherry tomato dataset. 50 images of each stage are taken as the validation set, and 50 images of each stage are re-collected and annotated as the test set. A total of 300 rounds of training are performed. Table 2 is the arrangement of the complete training set, validation set and test set.

[0033] Table 2 Dataset type, size and training rounds

[0034] Dataset Training set Validation set Test set Training rounds String tomato dataset 2000 500 300 200 Small tomato dataset 900 150 150 300

[0035] Step 2: Based on the improved YOLOv5 network, the string tomato dataset and the individual cherry tomato dataset are trained respectively to obtain the corresponding weight model configuration. The specific structure of the improved YOLOv5 network is as follows Figure 2As shown, in order to improve the training speed, the convolution layer of the backbone layer of the YOLOv5 network model is replaced with GhostConv, and the Detect of the output head of the head layer is replaced with the ASFF adaptive feature fusion layer. In the case of almost unchanged recognition accuracy, the training time is reduced by 13%, which improves the application value to a certain extent.

[0036] Firstly, through warpPerspective in opencv, the image matrix corresponding to the string tomato data set or single small tomato data set is processed in a series of rotation, translation, scaling and other processes. Four processed images in the data set are spliced and integrated, and the integrated large image is randomly rotated, translated, scaled and cropped to realize image data enhancement.

[0037] Secondly, feature extraction is performed based on the backbone layer in the improved YOLOv5 network to extract image features, wherein the backbone layer includes GhostConv, C3 module and SPPF module. The Bottleneck in BottleneckCSP is similar to resnet (residual model). In order to obtain better features, resnet allows the neural network to select the optimal solution by itself, skips the layers with poor training results, and uses resnet to extract features by stacking more layers to ensure that the training result is at least not worse than the original. With the subsequent update of YOLOv5, the present application uses the version 6.0 of YOLOv5. The original BottleneckCSP is updated to C3 module, which reduces a 1*1 convolution kernel in BottleneckCSP and removes a BN layer and an activation function. Its function is to simplify the network structure, reduce the calculation amount and reduce the model inference time. Through the three-layer pooling of the SPPF module, the fusion of high-level information is realized, while the size of the feature map is kept unchanged. The introduction of GhostConv reduces the calculation amount and improves the training efficiency.

[0038] Finally, the head layer of data fusion uses the head layer with FPN+PAN structure to fuse the image information from top to bottom, so that the bottom can receive high-level information, and then fuse from bottom to top, so that the top receives the bottom information. This network structure improves the generalization ability of the network, considers more comprehensive image features, and realizes a more reasonable neural network structure. The output head of the head layer adopts ASFF adaptive feature fusion, which reduces the calculation amount while keeping the training accuracy almost unchanged, and improves the training efficiency.

[0039] The weight model configuration of each of the string tomato data set and the small tomato data set is obtained by training the above multi-layer network structure. The mAP indexes (training effect evaluation indexes) of string tomato, full-ripe period fruit, semi-ripe period fruit and green-ripe period fruit are 99.5%, 99.8%, 99.8% and 99.7% respectively. The training effect is significant.

[0040] Step 3: Based on the improved YOLOv5 network, a target position interception storage mechanism is introduced in the detect program to build a string tomato detection model DetectTomato. In the prediction layer of the detect program, all rectangular frames are grouped according to different class labels through nms non-maximum suppression, and the highest scoring rectangular frame is taken out. The remaining rectangular frames are traversed, and the intersection over union of the current highest scoring rectangular frame is calculated. The frames with an IOU threshold greater than the set threshold are deleted, and the above operation is repeated for the remaining rectangular frames until all rectangular frames are processed. Finally, the optimal selected frame of the identified target is obtained.

[0041] The left upper x coordinate, left upper y coordinate, image width w and image height h of the optimal selected frame in the prediction layer are obtained, the target region of the original image is framed and output, and is stored in the preset exp folder. The storage format is jpg, and the x, y, w, h information is stored in the preset label folder. The storage format is txt. The exp folder and the label folder are placed in the runs folder. Thus, the string tomato detection model DetectTomato is completed.

[0042] Step 4: The specific process of actually running the string tomato detection model DetectTomato is as shown in Figure 3 The string tomato detection model DetectTomato first calls the weight model configuration of the string tomato data set to predict the to-be-detected image and obtain multiple string tomato detection targets.

[0043] Secondly, the position information of all recognized string tomato detection targets in the original image is obtained, including the left upper x coordinate, the left upper y coordinate, the image width w and the image height h. The position information is stored in the label folder in the runs folder, and the storage format is txt.

[0044] Finally, according to the obtained position information, the string tomato detection target image is intercepted, and the string tomato detection target image is stored in the exp folder in the runs folder, and the storage format is jpg. Thus, the target recognition and detection of string tomato are completed.

[0045] Step 5: Based on the improved YOLOv5 network, introduce statistical functions in the detect program, and also introduce the target position output mechanism to build the maturity detection model DetectMaturity.

[0046] Step 4: Traverse all single string tomato images stored in the exp folder, identify and count the number of small tomatoes in the three periods of full-ripe, half-ripe, and green-ripe in the single string tomato image in turn, divide the number of full-ripe small tomatoes by the total number of small tomatoes in the three periods, calculate the maturity of the single string tomato, output the target image position in the label folder in step 4 with the highest maturity and reaching the standard, call the os library, and clear the label folder and exp folder after the string tomato maturity detection is completed, to build the maturity detection model DetectMaturity.

[0047] Step 6: The specific process of running the maturity detection model DetectMaturity is shown in Figure 4 Step 6: The specific process of running the maturity detection model DetectMaturity is shown in

[0048] The growth period of small tomatoes on string tomatoes can be divided into three periods: full-ripe, half-ripe, and green-ripe. In order to avoid picking string tomatoes with too many half-ripe and green-ripe small tomatoes, it is stipulated that the proportion of full-ripe small tomatoes on a single string tomato should be greater than 90%, therefore the following function is added to the detect program, where Maturity represents maturity, λ1 is the number of full-ripe small tomatoes, λ2 is the number of half-ripe small tomatoes, and λ3 is the number of green-ripe small tomatoes:

[0049]

[0050] If there is a maturity degree greater than 90% and the highest string tomato image, the target position information of the string tomato image is outputted.

[0051] Figure 5 The string tomato detection model DetectTomato is shown in the string tomato recognition effect diagram, the string tomato detection model DetectTomato can accurately identify the string tomato with a short shooting distance, and give the confidence. Figure 6 The maturity detection model DetectMaturity is shown in the string tomato recognition effect diagram. Figure 5 One of the string tomato maturity detection effect diagrams, as shown in the figure, the maturity detection model DetectMaturity can accurately identify the maturity of a single cherry tomato, and provides a favorable scientific basis for determining the final picking target.

[0052] The technical features of the above-mentioned embodiments can be combined arbitrarily, in order to make the description simple, not all possible combinations of the technical features in the above-mentioned embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0053] The above-mentioned embodiments only express several embodiments of the present application, the description is more specific and detailed, but it should not be understood as the limitation of the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of variations and improvements can be made, which belong to the protection scope of the present application. Therefore, the protection scope of the present application patent should be subject to the appended claims.

Claims

1. A method for detecting the maturity of string tomatoes based on an improved YOLOv5 network, characterized in that, The method comprises the following steps: Step 1: constructing a string tomato data set and a single small tomato data set composed of images of single small tomatoes at each growth stage in the string tomato; Step 2: training the string tomato data set and the single small tomato data set based on an improved YOLOv5 network respectively to obtain corresponding weight model configurations, wherein the improved YOLOv5 network replaces the convolution layer of the backbone layer of the YOLOv5 network model with GhostConv and replaces the Detect of the head output head with an ASFF adaptive feature fusion layer; Step 3: introducing a target position interception and storage mechanism in the detect program based on the improved YOLOv5 network to construct a string tomato detection model DetectTomato; Step 4: the string tomato detection model DetectTomato calls the weight model configuration of the string tomato data set, identifies string tomato detection targets in a to-be-detected image, obtains image and position information of all identified string tomato detection targets, and stores the string tomato detection target image and the position information in the exp folder and the label folder respectively; Step 5: introducing a statistical function and a target position output mechanism in the detect program based on the improved YOLOv5 network to construct a maturity detection model DetectMaturity; Step 6: the maturity detection model DetectMaturity calls the weight model configuration of the single small tomato data set, iterates through all string tomato detection target images stored in the exp folder, identifies single small tomatoes in each string tomato detection target image, calls a statistical function to count the number of small tomatoes at the ripe stage, the half-ripe stage and the green-ripe stage in the same string tomato detection target image, divides the number of ripe small tomatoes by the total number of small tomatoes to calculate the maturity of the string tomato, outputs the position of the string tomato detection target image corresponding to the label folder with the highest maturity and reaching a preset maturity standard value, thereby determining the picking target, and calling the os library to empty the label folder and the exp folder after the maturity detection of each string tomato detection target image is completed.

2. The method for detecting the maturity of string tomatoes based on the improved YOLOv5 network according to claim 1, characterized in that, Step 2 specifically comprises the following steps: Based on warpPerspective in opencv, a series of rotation, translation and scaling processing is performed on the image matrix corresponding to the data set, four processed images in the data set are taken for splicing, and the integrated large image is subjected to random rotation, translation, scaling and cropping to realize image data enhancement; Based on the backbone layer in the improved YOLOv5 network, image features are extracted, wherein the backbone layer comprises GhostConv, a C3 module and a SPPF module; Based on the head layer in the improved YOLOv5 network, image features are fused to obtain a weight model configuration, wherein the head layer adopts the structure of FPN+PAN, and the head output head adopts ASFF adaptive feature fusion.

3. The method according to claim 1, wherein, Step 4 specifically comprises the following steps: The DetectTomato detection model predicts the to-be-detected image by using the weight model configuration of the string tomato data set, and obtains a plurality of string tomato detection targets; The position information of the string tomato detection target in the original image is obtained, including the left upper x coordinate, the left upper y coordinate, the image width w, and the image height h, and the position information is stored in the label folder in the runs folder in the txt format; According to the obtained position information, the string tomato detection target image is intercepted, and the string tomato detection target image is stored in the exp folder in the runs folder in the jpg format.

4. The method according to claim 1, wherein, The string tomato data set is constructed by the following steps: The string tomato image is obtained by the acquisition device, and the string tomato image data is obtained; The string tomato image data is labeled by using the labeling tool MakeSense, and the string tomato data set is prepared after labeling, and the string tomato data set is divided into training set, verification set and test set according to a certain proportion.

5. The method according to claim 1, wherein, The single small tomato data set is constructed by the following steps: Collect the images of single small tomatoes in each growth period in string tomatoes, obtain the R component and G component of each image by using opencv, and draw the R-G component statistical table of each growth period; Collect a plurality of images of single small tomatoes in string tomatoes in the greenhouse, and obtain the corresponding R-G component map of each image by using opencv; Compare the R-G component map with the drawn R-G component statistical table, and use the labeling tool MakeSense to label the growth period and classification of each single small tomato image, label the same number of ripe small tomato images, green ripe small tomato images and semi-ripe small tomato images, and all the ripe small tomato images, green ripe small tomato images and semi-ripe small tomato images together constitute a single small tomato data set.

Images