A weed detection method for vegetable seedling fields based on deep learning and image processing

Abstract

The present invention relates to a vegetable seedling field weed detection method based on deep learning and image processing. Collect pictures of vegetables and main weeds in vegetable seedling fields at different growth stages, establish image data sets after preprocessing, and divide them into training sets, test sets and verification sets; train a weed target detection convolutional neural network model on the collected beet and weed image data sets, and use the test set to perform accuracy testing; input the image to be detected, generate a weed target detection frame, weed category and weed target confidence score; use a non-maximum suppression algorithm to filter out a local confidence maximum detection frame of the corresponding type from the detection frame as the current weed target detection true value frame; use the super green image feature combined with the OTSU adaptive threshold segmentation algorithm in the weed target detection frame to segment the current area image, and obtain a more accurate target image centroid coordinate position and an external rectangular frame. This method can quickly and accurately identify and locate crops and weed targets in vegetable seedling fields.

Description

Technical Field

[0001] The present invention relates to the field of weed detection, and in particular to a method for detecting weeds in a vegetable seedling field based on deep learning and image processing. Background Art

[0002] Field weeds compete with crops for nutrients and water, occupy farmland growth space, affect crop photosynthesis, interfere with crop growth, reduce crop yield and quality, and indirectly contribute to the breeding and spread of pests and diseases. Organic vegetables grown in vegetable greenhouses have high economic value, but most vegetable greenhouses in my country still use traditional manual weeding and chemical weeding methods. Manual weeding is labor-intensive, time-consuming, labor-intensive, and inefficient, and chemical weeding can easily cause environmental pollution. With the development of precision agricultural technology, automated weeding machinery has gradually become the focus of weed control research. The premise for automated weeding machinery to perform weeding tasks is the accurate positioning of field weed targets. At present, the main methods for identifying weeds in vegetable seedling fields are machine vision image recognition and multispectral analysis. The machine vision image recognition method mainly uses image acquisition equipment to collect field RGB images, analyzes and extracts the morphological features of weeds and crops such as shape, color, and contour, and trains image feature classifiers to distinguish between the two. It has high requirements for image acquisition environment, acquisition equipment, and image processing. Image feature extraction and image feature classifier training are very complex and rely on human experience, and have poor adaptability to actual weeding operations. Multispectral technology uses acquisition equipment to collect the spectral image information of field crops and weeds, and uses the different spectral information of crops and weeds under different lighting conditions to distinguish between the two. This method has high requirements for analysis and acquisition equipment, and is sensitive to the lighting and weather conditions of the working environment. It is difficult to promote on a large scale and is not conducive to actual agricultural production operations. Summary of the invention

[0003] The purpose of the present invention is to provide a vegetable seedling field weed detection method based on deep learning and image processing, which can quickly and accurately detect weed targets in vegetable seedling fields, has low requirements on acquisition equipment and image data quality, and can be applied to weed detection tasks in non-structured complex environments.

[0004] To achieve the above object, the technical solution of the present invention is: a method for detecting weeds in vegetable seedling fields based on deep learning and image processing, comprising the following steps:

[0005] S1, collect a large number of RGB three-channel images of vegetables and main weeds in vegetable seedling fields at different growth stages in the vegetable greenhouse;

[0006] S2. After data annotation, image preprocessing and data enhancement, an image dataset is established, and the samples in the image dataset are divided into a training set, a test set and a validation set in a ratio of 8:1:1;

[0007] S3, training a convolutional neural network on the collected beet and weed image dataset to obtain network pre-weights, then training a weed target detection convolutional neural network model on the training set by transfer learning and network freezing, using the test set to perform accuracy testing on the trained weed target detection convolutional neural network model to obtain detection results and perform network fine-tuning;

[0008] S4, inputting the image to be detected into the network model for inference, generating a weed target detection frame, a weed category and a weed target confidence score in the image to be detected;

[0009] S5, using the non-maximum suppression algorithm to select the local confidence maximum detection frame of the corresponding type from all detection frames as the current weed target detection true value frame;

[0010] S6. In all weed target detection frames, the super green image features are combined with the OTSU adaptive threshold segmentation algorithm to segment the current area image, and output more accurate weed target image centroid coordinates and external rectangular frames.

[0011] In one embodiment of the present invention, in step S3, the collected beet and weed image dataset is a beet and weed image dataset collected by the University of Bonn in Germany.

[0012] In one embodiment of the present invention, all images are RGB three-channel images collected from vegetable seedling fields in vegetable greenhouses.

[0013] In one embodiment of the present invention, in step S2, the image preprocessing method is Gaussian filtering and Gamma correction.

[0014] In one embodiment of the present invention, in step S2, the data enhancement method is: randomly scaling the sample image between 70% and 120%; randomly cropping the image; horizontally flipping the image relative to the center line; adding random noise that satisfies the Gaussian distribution to the image; adding a random black mask to the weed target position; copying the weed target on the image and pasting it to a random position of the image; applying color changes to the image on the RGB, HSV, and LUV color channels.

[0015] In one embodiment of the present invention, in step S3, the convolutional neural network model pre-weight is a CSPDarknet-53 backbone feature extraction network model trained on a beet and weed image dataset, comprising 16 convolutional layers with convolution kernels in the form of 1×1 and 3×3.

[0016] In one embodiment of the present invention, in step S6, the image segmentation adopts the super green image feature combined with the OTSU adaptive threshold segmentation algorithm, and then the connected domain and the target centroid coordinates are calculated on the foreground image, and finally the detection result including the weed category, the weed target detection frame and the weed centroid coordinates is output.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] For the task of field weed detection, traditional machine vision methods require complex image preprocessing of images, and then manually extract their image features (color, shape, texture, contour and artificially designed features, etc.) and use the extracted image features to train classifiers for image recognition. Machine vision methods have high requirements for image quality and image preprocessing. They need to perform corresponding feature analysis for each type of crop and weed and rely on human experience. The processing process is complex and has poor adaptability to complex environments. Multispectral image recognition methods have high requirements for ambient light and weather when spectral images are collected. When it comes to multiple crops and weeds, it is impossible to ensure high accuracy, and it cannot be widely promoted due to high requirements for collection and analysis equipment. This method combines deep learning and image processing to give full play to the high accuracy and robustness of deep learning algorithms and the strong versatility of traditional image processing algorithms. It only requires an honest RGB camera to collect vegetable field crop and weed images to form a data set. Through deep learning training, a weed detector based on a deep convolutional neural network is obtained. The image processing method is used to enhance the positioning accuracy of the detector for weed targets, and to achieve rapid and accurate detection of weeds in vegetable seedling fields. Experiments on validation set samples show that this method can achieve extremely high detection accuracy through data enhancement and transfer learning when the number of data set samples is small. In this method, the detection accuracy of crops and weeds is high, the speed is faster, the robustness is stronger, the cost of acquisition equipment and the quality of image data are low, and it has strong applicability for vegetable seedling field weed detection tasks and can be well promoted. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of constructing a deep convolutional neural network weed detector in the present invention;

[0020] Figure 2 is a flow chart of detecting weeds in vegetable seedling fields in the present invention;

[0021] Figure 3 It is a structural diagram of the deep convolutional neural network model in the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0023] The present invention provides a vegetable seedling field weed detection method based on deep learning and image processing, comprising the following steps:

[0024] S1, collect a large number of RGB three-channel images of vegetables and main weeds in vegetable seedling fields at different growth stages in the vegetable greenhouse;

[0025] S2. After data annotation, image preprocessing and data enhancement, an image dataset is established, and the samples in the image dataset are divided into a training set, a test set and a validation set in a ratio of 8:1:1;

[0026] S3, training a convolutional neural network on the collected beet and weed image dataset to obtain network pre-weights, then training a weed target detection convolutional neural network model on the training set by transfer learning and network freezing, using the test set to perform accuracy testing on the trained weed target detection convolutional neural network model to obtain detection results and perform network fine-tuning;

[0027] S4, inputting the image to be detected into the network model for inference, generating a weed target detection frame, a weed category and a weed target confidence score in the image to be detected;

[0028] S5, using the non-maximum suppression algorithm to select the local confidence maximum detection frame of the corresponding type from all detection frames as the current weed target detection true value frame;

[0029] S6. In all weed target detection frames, the super green image features are combined with the OTSU adaptive threshold segmentation algorithm to segment the current area image, and output more accurate weed target image centroid coordinates and external rectangular frames.

[0030] The following is the specific implementation process of the present invention.

[0031] like Figure 1 As shown, building a deep convolutional neural network weed detector includes the following steps:

[0032] S1. Use a camera to collect 1500 RGB three-channel images in the vegetable greenhouse, use the Labelimg annotation tool to annotate the crop and weed targets, then use the image preprocessing method (Gamma correction and Gaussian filtering) to process the images and use the data enhancement method (random scaling, random cropping, horizontal flipping, adding Gaussian noise, random occlusion, etc.) to expand the number of data set samples to form a data set, and divide the data set into training set, test set and validation set according to 8:1:1.

[0033] S2. Use the pytorch deep learning framework to build a deep neural network model. The network model structure is as follows Figure 3 As shown, the network pre-weights are trained on a dataset of beets and weed images collected by the University of Bonn in Germany.

[0034] S3. Use transfer learning and network freezing methods to train the weed detector on the training set, and then test and fine-tune the detector on the test set.

[0035] S4. Verify the detection accuracy of the detector on the verification machine. If the detector accuracy is high (e.g., above 90%) and can meet the usage requirements, the detector training is completed. Otherwise, if the detection accuracy is poor, return to S3 for further training and tuning.

[0036] like Figure 2 As shown, a vegetable seedling field weed detection method based on deep learning and image processing includes the following steps:

[0037] S1. Use a camera to take pictures of the vegetable seedling field or upload images from the local terminal as the image input to be inspected

[0038] S2. Use the image scaling method to scale the image to the image input pixel size (608×608) required by the preset convolutional neural network model.

[0039] S3. The input image that meets the preset input requirements of the convolutional neural network model is passed into the network model. The model generates an output feature map through the previous reasoning. Then the post-processing program calculates the image target based on the output feature map and generates the detection box, target category and classification confidence information.

[0040] The structure diagram of the convolutional neural network is as follows Figure 3 As shown in the figure, it mainly includes the backbone feature network CSPDarknet53, the spatial pyramid pooling network SPP, the path aggregation network PANet and three output structures. The backbone feature network chooses CSPDarknet53 improved by using the CSP structure on Darknet53, and uses the Mish function for network activation, which can fully extract the features of the input image. The spatial pyramid pooling network SPP uses three maximum pooling layers with different kernels and a residual connection to fuse image features and increase the receptive field. The path aggregation network PANet consists of two paths from top to bottom and bottom to top, which fully fuses image features of different resolutions and enhances the detection ability of objects of different sizes. The three feature maps of different resolutions output from PANet output output feature maps of different sizes through the output structure. After the output feature map is processed by the post-processing program, the feature map parameters are parsed into the target detection box size, detection box coordinates, target category and classification confidence score.

[0041] S4. Use the non-maximum suppression algorithm to deduplicate the detection frame obtained in S3 to obtain a high-quality detection frame as the final detection result.

[0042] S5. Split the RGB image into R, G, and B three-channel color images, and calculate the super green feature channel image according to the pixel values ​​of the R, G, and B three-channel images.

[0043] Since the vegetation in the vegetable seedling field is usually green, which is quite different from the soil background, the soil background and vegetation can be efficiently separated by using the color feature difference. The super green feature is in the RGB color space. By strengthening the green component in the color space and enhancing the difference between the green feature and other features, the green feature component in the color space is better separated to achieve the purpose of segmenting the foreground and soil background.

[0044] The calculation formula for calculating the super green feature channel image using the R, G, and B three-channel images is:

[0045] ;

[0046] S6. According to the position and size of the detection frame obtained in S3, the OTSU adaptive threshold segmentation algorithm is used to segment the image at the corresponding position on the super green feature channel image, and the connected domain of the detection frame position image and the centroid coordinates of the corresponding connected domain are calculated. The centroid coordinates are the centroid coordinates of the target to be detected.

[0047] S7: Output the final weed target detection result.

[0048] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for detecting weeds in vegetable seedling fields based on deep learning and image processing, characterized in that: The steps include: S1, collect a large number of RGB three-channel images of vegetables and main weeds in vegetable seedling fields at different growth stages in the vegetable greenhouse; S2. After data annotation, image preprocessing and data enhancement, an image dataset is established, and the samples in the image dataset are divided into a training set, a test set and a validation set in a ratio of 8:1:1; S3, training a convolutional neural network on the collected beet and weed image dataset to obtain network pre-weights, then training a weed target detection convolutional neural network model on the training set by transfer learning and network freezing, using the test set to perform accuracy testing on the trained weed target detection convolutional neural network model to obtain detection results and perform network fine-tuning; S4, inputting the image to be detected into the network model for inference, generating a weed target detection frame, a weed category and a weed target confidence score in the image to be detected; S5, using the non-maximum suppression algorithm to select the local confidence maximum detection frame of the corresponding type from all detection frames as the current weed target detection true value frame; S6. In all weed target detection frames, the super green image features are combined with the OTSU adaptive threshold segmentation algorithm to segment the current area image, and output more accurate centroid coordinates and external rectangular frames of the weed target images; In step S3, the convolutional neural network model pre-weight is a CSPDarknet-53 backbone feature extraction network model trained on the beet and weed image dataset, which contains 16 convolutional layers with convolution kernels of 1×1 and 3×3; In step S6, the image segmentation uses the super green image feature combined with the OTSU adaptive threshold segmentation algorithm, and then calculates the connected domain and the target centroid coordinates on the foreground image, and finally outputs the detection result including the weed category, the weed target detection box and the weed centroid coordinates.

2. The vegetable seedling field weed detection method based on deep learning and image processing according to claim 1, characterized in that: In step S3, the collected beet and weed image dataset is a beet and weed image dataset collected by the University of Bonn in Germany.

3. The vegetable seedling field weed detection method based on deep learning and image processing according to claim 1, characterized in that: All images are RGB three-channel images collected from vegetable seedling fields in vegetable greenhouses.

4. The vegetable seedling field weed detection method based on deep learning and image processing according to claim 1, characterized in that: In step S2, the image preprocessing method is Gaussian filtering and Gamma correction.

5. The vegetable seedling field weed detection method based on deep learning and image processing according to claim 1, characterized in that: In step S2, the data enhancement method is as follows: randomly scale the sample image between 70% and 120%; randomly crop the image; horizontally flip the image relative to the center line; add random noise that satisfies the Gaussian distribution to the image; add a random black mask to the weed target position; copy the weed target on the image and paste it to a random position in the image; and apply color changes to the image on the RGB, HSV, and LUV color channels.

Images