A weed identification method integrating deep learning and image processing

Abstract

This invention discloses a weed identification method integrating deep learning and image processing, comprising: acquiring field images; dividing the field images into several grid images; identifying the grid images using a classification neural network model, and labeling each grid image as either crop or background; using color factors in image processing to perform image segmentation, area filtering, and connected component labeling on the green pixels in the two types of grid images, counting the number of connected components in the two types of grid images, and identifying the weed distribution in the grid images based on the number of connected components. In this invention, the neural network model only needs to focus on whether there are crops in the image, and the crop varieties are single or limited. Even if weed species not appearing in the training set are encountered, it does not affect the identification result. Therefore, this invention can effectively reduce the complexity of weed identification and the cost of building the training set images, and can improve the robustness and generalization ability of the model.

Description

Technical Field

[0001] This invention relates to the field of weed identification technology, specifically to a weed identification method that integrates deep learning and image processing. Background Technology

[0002] Weeds compete with crops for water, nutrients, and sunlight, breed pests and diseases, and lead to reduced grain yields. There are over 140 common weed species in fields. Weed control methods include manual weeding, chemical weeding, biological weeding, and mechanical weeding. Manual weeding is labor-intensive and inefficient. Chemical weeding is fast-acting, but excessive use can cause environmental pollution. Biological weeding, while environmentally friendly, is unsuitable for sudden weed outbreaks. Mechanical weeding, as a green and pollution-free method, has gained widespread attention, but if weeds are not accurately identified, it often damages crops. Currently, manual weeding is still the primary method in fields. Labor shortages and rising labor costs in rural areas not only increase planting costs but also limit the development of the agricultural industry. Therefore, the development of efficient and intelligent weeding equipment is imperative. To achieve intelligent weeding, accurate weed identification is the first step.

[0003] In large-scale production bases, vegetable crops (such as beets, cabbage, tomatoes, and peppers) are typically grown using mechanized methods. The row and plant spacing of vegetable seedlings is more standardized and larger than that of manually cultivated vegetable plots, making them suitable for mechanized and intelligent weeding. At the same time, existing and traditional manual weeding methods are clearly inadequate for the needs of large-scale production.

[0004] With the development of machine learning technology, especially deep learning, convolutional neural networks have been widely used. Currently, commonly used deep learning models are divided into object detection neural networks and image classification neural networks. Object detection networks can identify weeds and label their bounding boxes, but the size of these bounding boxes varies, while the unit operating range of weeding actuators is often fixed, thus these bounding boxes cannot be directly used for precise weeding. On the other hand, building a training set for object detection models is costly, requiring the collection of numerous and diverse weed images and the labeling of bounding boxes for each. Compared to object detection networks, image classification networks typically have higher recognition accuracy; however, they can only identify whether an image contains weeds, not their location within the image.

[0005] To address the aforementioned problems, this invention proposes a weed identification method that integrates image classification networks (classification neural network models) and image processing. Summary of the Invention

[0006] The purpose of this invention is to address the problems existing in the prior art by providing a weed recognition method that integrates deep learning and image processing.

[0007] This invention is achieved through the following technical solution:

[0008] This invention provides a weed identification method that integrates deep learning and image processing, comprising the following steps:

[0009] S1. Collect field images;

[0010] S2. Divide the field image into several grid images evenly;

[0011] S3. The trained classification neural network model is used to identify grid images and each grid image is labeled as either crop or background. Grid images labeled as crops contain crops, while grid images labeled as background do not contain crops.

[0012] S4. Take the grid images labeled as background and the grid images labeled as crops. Use color factors in image processing to perform image segmentation, area filtering, and connected component labeling on the green pixels (plants, crops / weeds) in both grid images. Count the number of connected components in both grid images and identify the distribution of weeds in the grid images by counting the number of connected components. Since the location of each grid image in the field image is known, the region of weeds in the field image can be determined by using a classification neural network model to identify all grid images containing weeds.

[0013] Furthermore, in S4, for a grid image marked as background, if the number of connected regions is 0, then the grid image is a soil area without crops or weeds; if the number of connected regions is N, and N > 0, then the grid image contains N weeds.

[0014] Furthermore, in S4, for a grid image labeled as a crop, if the number of connected components is 1, then the grid image contains only crops; if the number of connected components is N, denoted as R1,…,R N If N > 1, then the following processing is performed: In the grid image, only the pixels at the R1 connected region are retained, and the pixels at the other connected regions are hidden (the transparency is set to 1, or the pixel value is directly set to 0 or 255). Then, the grid image is input into a classification neural network model for recognition (the positions of the connected regions are mapped to the original grid image for pixel processing; here, the original grid image with the pixels at the specified connected regions hidden is input into the classification neural network model for recognition, not the binary image after image processing). If the recognition result is crops, then the pixels in the R1 connected region are crop pixels; otherwise, they are weed pixels. The same operation is performed, retaining only R2... up to R NThe pixels at the connected regions are then input into a classification neural network model for identification, from R2 to R... N By identifying each connected region, all crops and weeds within the grid image can be identified.

[0015] Furthermore, in S2, among the divided grid images, grid images containing crops are positive samples, and grid images not containing crops are negative samples; the positive sample label is crop, and the negative sample label is background.

[0016] Furthermore, the positive sample grid image labeled as a crop is an image containing crops. That is, as long as the grid image contains crops, it is considered a crop. Specifically, there are two scenarios: the grid image contains only crops, and the image contains both crops and weeds.

[0017] Furthermore, the negative sample grid image with the label as background is an image that does not contain crops, specifically in two scenarios: the grid image contains only weeds, and the image contains neither crops nor weeds (i.e., only soil or other background exists).

[0018] Furthermore, the size of the grid image is consistent with the size of the training set images of the neural network model.

[0019] Furthermore, the size of the grid image is set according to the working range of the weeding actuator, so that the working range of the weeding actuator covers each grid area.

[0020] Furthermore, in step S4, when performing area filtering, an area threshold needs to be set to filter out crop or weed pixels with excessively small areas in the grid image, in order to avoid misjudgment by the classification neural network model during recognition. When dividing the grid image, it is possible that only a very small portion of the grid image contains weeds or crops. Because the target size in these grid images is extremely small, the neural network model is highly prone to misidentification. Therefore, scene optimization is performed using an area threshold, specifically by ignoring and filtering out crop or weed pixels smaller than the specified area threshold. The target of filtering may be image noise, or a small patch of weeds or crops. The latter will not affect the actual weed recognition, because the main part of the small patch of crop or weed will appear in other grid images and be correctly recognized.

[0021] The beneficial effects of this invention are:

[0022] In this invention, the neural network model classifies the recognition targets into positive samples (crops) and negative samples (non-crops, background). Non-crop images are those containing weeds or soil background. The neural network model only needs to check if crops are present in the image; if no crops are present, it is considered a background image. Since there are many types of weeds, it is difficult to identify them all and maintain a high recognition rate. According to the method of this invention, the neural network model only needs to focus on whether crops are present in the image, and the types of crops are singular or limited. Even if a weed type not present in the training set is encountered, it does not affect the recognition result. Therefore, the method of this invention can effectively reduce the complexity of weed recognition and the cost of building a training set of images, and can improve the robustness and generalization ability of the model. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a field image divided into several grid images according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of weed identification in grid image No. 2 in this embodiment of the invention;

[0025] Figure 3 This is a schematic diagram of weed identification in grid image No. 6 of this invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1

[0028] This invention provides a weed identification method that integrates deep learning and image processing, specifically including the following steps:

[0029] Step 1: Train the classification neural network model

[0030] The original image containing crops or weeds is uniformly divided into several grid images. The acquired grid images are manually classified, with grid images containing crops designated as positive samples and those not containing crops as negative samples. These positive and negative samples are then used to train a classification neural network model.

[0031] Positive sample grid images contain two types of scenes:

[0032] 1) The image only contains crops;

[0033] 2) The image contains both crops and weeds.

[0034] The negative sample grid images contain two types of scenes:

[0035] 1) The image only contains weeds;

[0036] 2) There are no weeds or crops in the image (i.e., only soil or other background exists).

[0037] When training the classification neural network model, positive samples are labeled as crops and negative samples are labeled as background. The classification neural network model is trained until the network converges or reaches the maximum number of iterations, and the optimal network model is saved.

[0038] Step 2: Acquire field images

[0039] Field images of the area to be treated are captured using cameras or the vision system of smart weeding equipment.

[0040] Step 3: Divide the field image into several grid images evenly.

[0041] The field image is uniformly divided into several grid images, the size of which is consistent with the size of the images in the training set of the classification neural network model. (Refer to...) Figure 1 The field images were divided into 12 grid images.

[0042] Depend on Figure 1 As shown, the divided grid images can be classified into the following scenarios:

[0043] Scenario 1: The grid image contains neither crops nor weeds, that is, it only contains soil (grid image number 8);

[0044] Scenario 2: The grid images contain only a single crop target (grid images 1, 5, 7, 9, and 11);

[0045] Scenario 3: A grid image contains multiple crop targets (grid image number 2);

[0046] Scenario 4: A grid image contains only a single weed target (grid image number 12);

[0047] Scene 5: A grid image contains multiple weed targets (grid image number 4);

[0048] Scenario 6: The grid images contain both weeds and crop targets (grid images 3, 6, and 10).

[0049] Step 4: Recognition of Grid Images

[0050] The trained classification neural network model identifies the 12 grid images and labels each grid image as either a crop or background. Grid images labeled as crops are those containing crops; those labeled as background are those not containing crops.

[0051] In this embodiment, 12 grid images are input into the classification neural network model for recognition. Scenes 2, 3, and 6 containing crops, i.e., grid images 1, 2, 3, 5, 6, 7, 9, 10, and 11, are positive samples and are identified as crops. Scenes 1, 4, and 5 not containing crops, i.e., grid images 4, 8, and 21, are negative samples and are identified as background.

[0052] Step 5: Weed Identification

[0053] (i) For the grid images marked as background (grid images No. 4, 8, and 21), the above grid images are processed using color factors (such as the super green factor, 2G-RB) in image processing. After segmentation, the binary images are obtained by automatic thresholding (such as the OTSU algorithm). Then, the area filtering method is used to filter out noise pixels and pixels of extremely small targets. Finally, the connected regions are marked and the number of connected regions is counted.

[0054] If the number of connected regions is 0, the grid image is soil (grid 8); if the number of connected regions is greater than 0, the grid image is weed grid (grids 4 and 12), and the number of connected regions is the number of weeds (grid 12 contains 1 weed and grid 4 contains 2 weeds).

[0055] (ii) For the grid images marked as crops (grid images No. 1, 2, 3, 5, 6, 7, 9, 10, and 11), the above grid images are processed using color factors (such as the super green factor, 2G-RB) in image processing. After segmentation, an automatic thresholding method (such as the OTSU algorithm) is used to obtain binary images. Then, the area filtering method is used to filter out noise pixels and pixels of extremely small targets. Finally, connected regions are marked and the number of connected regions is counted.

[0056] If the number of connected components is 1, then the grid image is a crop image (and there is only one crop, such as grids 1, 5, 7, 9, and 11); if the number of connected components is greater than 1, then the following processing is performed:

[0057] Grid #2:

[0058] Reference Figure 2The image contains two connected regions. First, the first connected region is mapped onto the original grid image. The pixel opacity of this region is set to 1, or the pixel value is set to 0 or 255. The grid image is then input into a neural network model for identification. The identification result is "crop," indicating that the second connected region is also a crop. Similarly, the second connected region is mapped onto the original grid image. The pixel opacity of this region is set to 1, or the pixel value is set to 0 or 255. The grid image is then input into a neural network model for identification. The identification result is also "crop," indicating that the first connected region is also a crop. Therefore, it can be inferred that both connected regions in grid image number 2 are crops.

[0059] Grid 3:

[0060] There are two connected regions. First, the first connected region is mapped onto the original grid image, and the pixel opacity of this region is set to 1, or the pixel value is set to 0 or 255. Then, the grid image is input into the neural network model for recognition. The recognition result is background, indicating that the second connected region is weeds. Similarly, the second connected region is mapped onto the original grid image, and the pixel opacity of this region is set to 1, or the pixel value is set to 0 or 255. Then, the grid image is input into the neural network model for recognition. The recognition result is crops, indicating that the first connected region is crops. Thus, it can be inferred that in grid image number 3, the two connected regions are crops and weeds, respectively.

[0061] Grid 6:

[0062] Reference Figure 3 There are four connected regions. First, retain the first connected region. Map the remaining connected regions to the original grid image, set the opacity to 1, or set the pixel value to 0 or 255. Then, input the grid image into the neural network model for recognition. The recognition result is background, indicating that the first connected region is weeds. Similarly, retain the second connected region, map the remaining connected regions to the original grid image, set the opacity to 1, or set the pixel value to 0 or 255. Then, input the grid image into the neural network model for recognition. The recognition result is crops, indicating that the second connected region is crops. Process all connected regions in this way to identify the category of all connected regions (2 connected regions are weeds, and 2 connected regions are crops).

[0063] Grid 10:

[0064] There are three connected regions. First, the first connected region is retained, and the remaining connected regions are mapped to the original grid image. The opacity is set to 1, or the pixel value is set to 0 or 255. Then, the grid image is input into the neural network model for recognition. The recognition result is "crop," indicating that the first connected region is a crop. Similarly, the second connected region is retained, and the remaining connected regions are mapped to the original grid image. The opacity is set to 1, or the pixel value is set to 0 or 255. Then, the grid image is input into the neural network model for recognition. The recognition result is "background," indicating that the second connected region is weeds. Processing the third connected region in this way allows for the identification of the category of all connected regions (2 connected regions are crops, and 1 connected region is weeds).

[0065] Therefore, according to the above method of the present invention, all grid images containing weeds can be identified. Since the position of each grid image in the field image is known information, the area of ​​weeds in the field image can be determined by using the classification neural network model to identify all grid images containing weeds.

[0066] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions that fall within the scope of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for weed recognition by fusing deep learning and image processing, characterized in that, The method comprises the following steps: S1, collecting a field image; S2, dividing the field image into a plurality of grid images; S3, identifying the grid images through a completed classification neural network model, and marking each grid image as a crop or a background, the grid image marked as the crop containing the crop, and the grid image marked as the background containing no crop; S4, taking the grid image marked as the background and the grid image marked as the crop, performing image segmentation, area filtering and connected region marking on green pixels in the two kinds of grid images respectively through a color factor in image processing, and counting the number of connected regions in the two kinds of grid images to identify the weed distribution in the grid images; For the grid image marked as the background, if the number of connected regions is 0, the grid image is a soil region containing no crop and no weed; if the number of connected regions is N and N>0, the grid image contains N weeds. For the grid image marked as crop, if the number of connected regions is 1, the grid image only contains crops; if the number of connected regions is N, set as R 1,…, R N , and N > 1, the following processing is performed: in the grid image, only the pixels at the positions of the R 1 connected regions are retained, the pixels at the positions of the remaining connected regions are hidden, and the grid image is input into a classification neural network model for recognition. If the recognition result is crop, the pixels at the positions of the R 1 connected regions are crop pixels, and otherwise are weed pixels. The same operation is performed on the R 2,…, until R N connected regions to identify all crops and weeds in the grid image.

2. The weed identification method fusing deep learning and image processing according to claim 1, wherein in the S2, the grid image containing the crop is a positive sample, and the grid image containing no crop is a negative sample; the positive sample is labeled as the crop, and the negative sample is labeled as the background.

3. The weed identification method fusing deep learning and image processing according to claim 2, wherein the positive sample grid image contains two scenes: only the crop in the grid image, and both the crop and the weed in the image; and the negative sample grid image contains two scenes: only the weed in the grid image, and no crop and no weed in the image.

4. The weed identification method fusing deep learning and image processing according to claim 1, wherein the size of the grid image is consistent with the size of the neural network model training set image.

5. The weed identification method fusing deep learning and image processing according to claim 1, wherein the size of the grid image is consistent with the size of the working range of the weeding executor.

6. The weed identification method fusing deep learning and image processing according to claim 1, wherein in the S4, an area threshold is set to filter out the crop or weed pixels smaller than the area threshold in the grid image during the area filtering. ​ ​ ​ ​ ​ ​

Images