DeeplabV3 +-based unmanned aerial vehicle image multi-crop plot identification and productivity prediction method

Abstract

The invention discloses an unmanned aerial vehicle image multi-crop plot identification and productivity prediction method based on DeeplabV3 +, and the method comprises the steps: obtaining an unmanned aerial vehicle image of a research region, and carrying out the splicing processing of the image; preprocessing and labeling the images, and dividing the images into a training set and a verification set; training, verifying and testing the improved DeeplabV < 3 + > deep learning network model by using the data set to obtain a final unmanned aerial vehicle image plot recognition and crop classification model based on DeeplabV < 3 + >; carrying out land parcel identification and crop classification on the preprocessed to-be-detected image by using the trained land parcel identification and crop classification model, and carrying out splicing processing; and the spliced prediction segmentation map is further processed by using envi and ArcGis software to obtain a productivity prediction map of various crops. According to the invention, the dependence on manual field investigation is reduced, the method is less limited by topographic factors, the method is faster and more accurate, and the efficiency of agricultural asset checking is improved.

Description

Technical Field

[0001] The present invention relates to the field of agricultural technology, and in particular to a method for identifying multiple crop plots and predicting productivity based on UAV images using DeeplabV3+. Background Art

[0002] With the in-depth development and application of modern information technologies such as computers, the Internet, the Internet of Things, artificial intelligence, big data, and cloud computing, the use of high-tech and Internet technologies to develop agriculture is a smart solution to change traditional agricultural production methods and solve the future problems of "who will farm" and "how to farm", and it is also an inevitable trend in agricultural development. Smart agriculture achieves intelligent management of the entire agricultural production process through standardized, centralized, automated, and intelligent agricultural production methods, effectively improving resource utilization efficiency.

[0003] "Deep learning" has gone through a long history of development and is now widely known for its outstanding learning ability. "Deep learning" is a machine learning method based on artificial neural networks. It can automatically learn abstract and high-level feature representations from large amounts of data, thereby improving the ability of pattern recognition and prediction. In recent years, deep learning technology has been continuously innovating, showing the characteristics of wide cross-domain applications and deepening technological integration. Among them, it is combined with technologies such as the Internet of Things, big data, and cloud computing to promote the development of emerging fields such as smart agriculture. DeeplabV3+ is a high-performance image segmentation model that improves the accuracy and efficiency of image segmentation by introducing FPN and dilated convolution.

[0004] Using the data images of land objects collected by lightweight and convenient drones to identify plots, classify crops and predict productivity is an important aspect of developing smart agriculture and precision agriculture. This technical method has the characteristics of systematicity, versatility and flexibility. The relevant parameters of the neural network can be adjusted as needed to achieve the optimal effect of plot identification and crop classification model. At the same time, the trained plot identification and crop classification model can be combined with the crop productivity prediction model to obtain the productivity prediction map of various crops. This method can not only reduce the dependence on artificial survey data, but also obtain the productivity prediction map of various crops, improve work efficiency and reduce labor costs. At present, remote sensing images are mainly used to identify crop plots, but this method is relatively limited for the accurate prediction of the productivity of various crops, especially for relatively complex and irregular crop planting plots. There are almost no studies on "plot identification, crop classification and productivity prediction based on drones". Therefore, scientific, real-time and efficient plot identification, crop classification and productivity prediction are of great significance to improving the efficiency of agricultural asset inventory, the intelligent and sustainable development of agricultural production and precision agriculture.

[0005] Compared with human subjective survey data, drones have irreplaceable advantages. Drone technology has great advantages in modern technology applications due to its safety and reliability, flexibility and adaptability, economy and efficiency, and has become a new method for land identification and crop classification. Drones can collect field data quickly and efficiently, especially in the face of a wide variety of crops and irregular land shapes. It can effectively avoid the damage to crops caused by people entering the field, thereby achieving accurate land identification and crop classification, while greatly reducing labor costs and improving the efficiency of collection and analysis. Based on this land identification, crop classification and productivity prediction methods, further research can be carried out in the future. Summary of the invention

[0006] The main purpose of the present invention is to provide a method for identifying and predicting the productivity of various crop plots based on UAV images using DeeplabV3+. Through this method, it is possible to use trained plot identification and crop classification models based on UAV images, bring various productivity models into the images of various crops, obtain the required relevant productivity indicator data of various crops, realize process-based, systematized, efficient and high-quality, and realize a systematic method for predicting the productivity of various crops based on UAV images, which is of reference significance.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for identifying and predicting productivity of various crop plots based on UAV images of DeeplabV3+, which includes obtaining UAV images of the study area and splicing the images; preprocessing and annotating the images, and dividing them into training set and verification set; using the data set to train, verify and test the improved DeeplabV3+ deep learning network model, and obtaining the final UAV image plot identification and crop classification model based on DeeplabV3+; using the trained plot identification and crop classification model to perform plot identification and crop classification on the preprocessed images to be detected and splicing them; using envi and ArcGis software to further process the spliced ​​prediction segmentation map to obtain the productivity prediction map of various crops.

[0009] The specific steps include:

[0010] (1) Obtain plot image data and use drones to collect RGB images during the crop phenological period;

[0011] (2) Image preprocessing and data set construction: Preprocess the collected image data;

[0012] This includes image stitching and cropping to generate a set of images with a length and width of 640 pixels, i.e., constructing the initial dataset voc2007 required for deep learning; using labelme to label different crop plots in the image, using different label names for different crop plots;

[0013] The data set required for deep learning is preprocessed and divided into a training set and a validation set in a ratio of 9:1. The training set is used as a data sample for model fitting, and the validation set is used to adjust the model's hyperparameters and to conduct a preliminary evaluation of the model's capabilities.

[0014] (3) Train the convolutional neural network model;

[0015] The DeeplabV3+ convolutional neural network model in deep learning is used, and the parameters of the DeeplabV3+ network model are optimized. The adam optimizer is used to improve the training efficiency. The improved network model is trained, verified and tested to obtain the optimal weight file .pth of the drone image plot recognition and crop classification model based on DeeplabV3+, and the plot recognition and crop classification models of different crops are constructed.

[0016] The model learns the characteristics of the key growth periods of different crops through training data, and realizes the automatic identification of plots of different crops. And mIoU, mPA, mPrecision, and mRecall are used as model evaluation indicators;

[0017] (4) Obtaining a map combining the original map and the crop plot segmentation map;

[0018] The trained plot recognition and crop classification model is used to detect plot recognition and crop classification on a set of images with a length and width of 640 pixels. The obtained segmentation prediction map is spliced ​​to obtain a combination of the original image and the crop plot segmentation map, which is the basic map for subsequent operations.

[0019] (5) Create raster images of each crop plot;

[0020] Import the image described in step (4) into the envi software, and use its built-in Region of Interest (ROI) Tool to process it, extract the ROI areas of different crops respectively, and use the Maximum Likelihood Classification tool to obtain the raster image classification obtained by the maximum likelihood classification method, including the classification categories of different crops. Use the Classification to Vector tool in the envi software to create an independent vector layer for different crop categories, that is, obtain the .shp file of a single crop plot. Use the cropping tool in the ArcGis software, use the original image as the input raster, and use the obtained .shp files of different crops as the output range, where one of the minimum X value and the maximum X value in the rectangle is set to 0, and one of the minimum Y value and the maximum Y value in the rectangle is set to 0, so that the size of the cropped images is the same, so as to facilitate the subsequent image fusion operation, and export the cropped images as TIFF images, so as to obtain the plot images of different crops respectively;

[0021] (6) Obtain productivity prediction maps for multiple crops: Import the different crop plot maps generated in step (5) into the ENVI software to obtain yield prediction maps.

[0022] The specific method of step (6) is:

[0023] On the basis of the plot map (taking corn as an example), the yield prediction model of the crop in the key growth period is integrated, that is, the Co-occurrence Measures tool in the ENVI software is used to obtain the red band mean and blue band mean required by the yield prediction model in the key growth period, and then the Band Math tool is used to bring the obtained values ​​into the yield prediction model in the key growth period for calculation. After removing the outliers, the Edit ENVI Header tool is used to set the Data Ignore Value to 0 and export it to the crop yield prediction map in tif format;

[0024] Finally, the Band Math tool in the ENVI software was used to merge the prediction maps of other crops, and finally the yield prediction maps of multiple crops were obtained.

[0025] The beneficial effects of the present invention mainly include:

[0026] 1. The land image data obtained by drones is richer in details, which improves the efficiency of identification and analysis and effectively reduces the errors and time consumption caused by manual observation;

[0027] 2. It realizes the identification of plots of various crops and the prediction of productivity, thus reducing the cost of agricultural production. The present invention provides a method system for identifying plots of various crops in the field and predicting productivity based on drone images;

[0028] 3. It provides a method for identifying plots of various crops and predicting their productivity, which is convenient for storing plot-related information and for conducting more in-depth research on various crop indicators based on the plot information of each crop. This is conducive to improving the efficiency of planting and management of each crop. The predicted indicators can be used as a reference, and favorable measures can be taken based on these reference information to further improve the productivity and quality of each crop. This method is suitable for plots with irregular shapes and shows good crop plot identification and classification capabilities.

[0029] 4. It provides a technical solution system that can be used as a reference for plot identification and productivity prediction of many other crops.

[0030] In summary, the method system proposed in the present invention for realizing multiple crop plot identification and productivity prediction based on DeeplabV3+ UAV images has important reference value and application prospects in multiple crop plot identification and productivity prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the original image of Chongzhou Base A after splicing in the embodiment;

[0032] Figure 2 This is the original image of Chongzhou Base B after splicing in the embodiment;

[0033] Figure 3 This is the original picture of Huihe Village after splicing in the embodiment;

[0034] Figure 4 This is a graph showing the change in loss value when using DeeplabV3+ training in this example;

[0035] Figure 5 This is a graph showing the miou value changes using DeeplabV3+ training in this example;

[0036] Figure 6 The mIoU evaluation index of the plot recognition and crop classification model obtained after training with DeeplabV3+ in the embodiment;

[0037] Figure 7 The mPA evaluation index of the plot recognition and crop classification model obtained after training with DeeplabV3+ in the embodiment;

[0038] Figure 8 The mPrecision evaluation index of the plot recognition and crop classification model obtained after training with DeeplabV3+ in the embodiment;

[0039] Fig. 9 The mRecall evaluation index of the plot recognition and crop classification model obtained after training with DeeplabV3+ in the embodiment;

[0040] Fig.10 The figure is a combination of the original image of Chongzhou base A in the embodiment and the crop segmentation prediction;

[0041] Fig.11 The figure is a combination of the original image of Chongzhou base B in the embodiment and the crop segmentation prediction;

[0042] Fig.12 This is a picture showing the combination of the original image of Huihe Village and the crop segmentation prediction;

[0043] Fig.13 The rice, corn and soybean vector diagrams of Chongzhou Base A in the embodiment;

[0044] Fig.14 The rice, corn and soybean vector diagrams of Chongzhou Base B in the embodiment;

[0045] Fig.15 For example, rice, corn and soybean vector diagrams of Huihe Village;

[0046] Fig.16 This is a soybean plot map of Chongzhou Base A in the embodiment;

[0047] Fig.17 This is a corn plot map of Chongzhou base A in the embodiment;

[0048] Fig.18 This is a soybean plot map of Chongzhou Base B in the embodiment;

[0049] Fig.19 This is a corn plot map of Chongzhou base B in the embodiment;

[0050] Fig. 20 For example, the soybean plot map of Huihe Village;

[0051] Fig.21 To illustrate the corn plot map of Huihe Village;

[0052] Fig. 22 The yield prediction diagram of soybean A in Chongzhou base of Example;

[0053] Fig.23 The yield prediction diagram of corn A in Chongzhou base of Example;

[0054] Fig.24 The yield prediction diagram of soybean B in Chongzhou base of Example;

[0055] Fig.25 The yield prediction diagram of corn B in Chongzhou base of Example;

[0056] Fig.26 This is the soybean yield prediction diagram for Example Huihe Village;

[0057] Fig. 27 To illustrate the yield prediction chart of corn in Huihe Village;

[0058] Fig.28 The corn and soybean yield forecast graphs of Chongzhou Base A are integrated into the embodiment;

[0059] Fig.29 The corn and soybean yield forecast graphs of Chongzhou Base B are integrated for the example;

[0060] Fig.30 The embodiment incorporates the corn and soybean yield forecast map of Huihe Village. DETAILED DESCRIPTION

[0061] The specific technical solution of the present invention is explained in conjunction with embodiments.

[0062] This embodiment includes the following steps:

[0063] (1) Obtaining plot image data: In order to enhance the diversity of plot image data, the present invention uses DJI Mavic 3 drone to collect RGB images of crops during the phenological period in Huihe Village and Chongzhou Base, including rice, corn, soybean and other crops.

[0064] (2) Image preprocessing and data set construction: Preprocess the collected image data, including image stitching, etc. Figure 1 , Figure 2 and Figure 3 The image was cropped to generate a set of images with a length and width of 640 pixels, which is the initial dataset voc2007 required for deep learning. Labelme was used to label different crop plots in the image, and different crop plots were labeled with different names. The dataset required for deep learning was preprocessed and divided into a training set and a validation set at a ratio of 9:1. The training set was used as a data sample for model fitting, and the validation set was used to adjust the model's hyperparameters and to conduct a preliminary evaluation of the model's capabilities.

[0065] (3) Training the convolutional neural network model: The present invention uses the DeeplabV3+ convolutional neural network model in deep learning as an example, and optimizes the relevant parameters of the DeeplabV3+ network model, using the adam optimizer to improve the training efficiency. The loss value and miou value change diagram of the model training process are as follows: Figure 4 and Figure 5. Taking rice, corn, and soybean datasets as examples, the present invention trains, verifies, and tests the improved network model, obtains the optimal weight file .pth of the drone image plot recognition and crop classification model based on DeeplabV3+, and constructs the plot recognition and crop classification models of rice, corn, and soybean. The model learns the characteristics of the rice jointing stage, corn filling stage, and soybean grain filling stage through a large amount of training data to realize the automatic recognition of rice, corn, and soybean plots. And mIoU, mPA, mPrecision, and mRecall are used as model evaluation indicators, respectively. Figures 6 to 9 The trained plot recognition and crop classification models achieved 92.70% in mPrecision, with the accuracy of rice reaching 0.99, corn 0.95, and soybean 0.85 due to the small number of training samples.

[0066] (4) Obtaining a map combining the original image and the crop plot segmentation map: Using the trained plot recognition and crop classification model, perform plot recognition and crop classification detection on a set of images of 640 pixels in length and width in Huihe Village and Chongzhou Base, and splice the obtained segmentation prediction map to obtain a map combining the original image and the crop plot segmentation map, as shown in Fig.10 , Fig.11 and Fig.12 , which is the basic graph for subsequent operations.

[0067] (5) Create raster images of each crop plot: Import the image described in step (4) into the envi software, and use its built-in Region of Interest (ROI) Tool to process it, extract four ROI regions of rice, corn, soybean, and others, and use the Maximum Likelihood Classification tool to obtain the raster image classification obtained by the maximum likelihood classification method, including four classification categories of rice, corn, soybean, and others. Use the Classification to Vector tool in the envi software to create a separate vector layer for each of the three crop categories of rice, corn, and soybean, such as Fig.13 , Fig.14 and Fig.15, that is, obtain the .shp files of individual crop plots of rice, corn, and soybean. Using the clipping tool in ArcGis software, the present invention uses the original images of Huihe Village and Chongzhou Base as input grids, and obtains the .shp files of rice, corn, and soybean as output ranges, wherein one of the minimum X value and the maximum X value in the rectangle is set to 0, and one of the minimum Y value and the maximum Y value in the rectangle is set to 0, so that the size of the images of each crop after clipping is the same, so as to facilitate the subsequent image fusion operation, and export the images obtained after clipping as TIFF images, and then obtain the rice plot map, corn plot map, and soybean plot map of Huihe Village and Chongzhou Base, respectively, as shown in FIG. Figure 16 to Figure 21 .

[0068] (6) Obtaining productivity prediction maps for multiple crops: The rice plot map, corn plot map, and soybean plot map of Huihe Village and Chongzhou Base generated in step (5) are respectively imported into the ENVI software. Taking the operation of obtaining the corn yield prediction map as an example, the corn filling period yield prediction model is integrated on the basis of the corn plot map. That is, the Co-occurrence Measures tool in the ENVI software is used to obtain the red band mean and blue band mean required by the corn filling period yield prediction model. The Band Math tool is then used to bring the obtained values ​​into the corn filling period yield prediction model for calculation. After removing the outliers, the Edit ENVI Header tool is used to set the Data Ignore Value to 0 and the corn yield prediction map is exported as a tif format. The soybean yield prediction map is the same as described above, as shown in FIG. Figure 22 to Figure 27 Finally, the Band Math tool in the envi software was used to merge the corn yield forecast map and soybean yield forecast map of Huihe Village and Chongzhou Base, and finally the yield forecast map of multiple crops was obtained, such as Fig.28 , Fig.29 and Fig.30 .

[0069] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for identifying various crop plots and predicting productivity based on UAV images using DeeplabV3+, characterized by: The process includes: Obtain drone images of the study area and stitch the images; Preprocess and annotate the images and divide them into training set and validation set; The improved DeeplabV3+ deep learning network model was trained, verified and tested using the dataset to obtain the final DeeplabV3+-based drone imagery plot recognition and crop classification model. Use the trained plot recognition and crop classification models to perform plot recognition and crop classification on the pre-processed images to be detected and perform splicing processing; The spliced ​​prediction segmentation maps were further processed using envi and ArcGis software to obtain productivity prediction maps of various crops.

2. The method for identifying various crop plots and predicting productivity based on UAV images using DeeplabV3+ according to claim 1, characterized in that: The following steps are involved: (1) Obtain plot image data and use drones to collect RGB images during the crop phenological period; (2) Image preprocessing and data set construction: Preprocess the collected image data; This includes image stitching and cropping to generate a set of images with a length and width of 640 pixels, i.e., constructing the initial dataset voc2007 required for deep learning; using labelme to label different crop plots in the image, using different label names for different crop plots; Preprocess the data set required for deep learning and divide it into a training set and a validation set at a ratio of 9:

1. The training set is used as a data sample for model fitting, and the validation set is used to adjust the model's hyperparameters and to conduct a preliminary evaluation of the model's capabilities. (3) Train the convolutional neural network model; The DeeplabV3+ convolutional neural network model in deep learning is used, and the parameters of the DeeplabV3+ network model are optimized. The adam optimizer is used to improve the training efficiency. The improved network model is trained, verified and tested to obtain the optimal weight file .pth of the UAV image plot recognition and crop classification model based on DeeplabV3+, and the plot recognition and crop classification models of different crops are constructed. The model learns the characteristics of the key growth periods of different crops through training data, and realizes the automatic identification of plots of different crops. It also uses mIoU, mPA, mPrecision, and mRecall as model evaluation indicators. (4) Obtaining a map combining the original map and the crop plot segmentation map; Use the trained plot recognition and crop classification model to detect plot recognition and crop classification on a set of images with a length and width of 640 pixels, and splice the obtained segmentation prediction map to obtain a map that combines the original image with the crop plot segmentation map, which is the basic map for subsequent operations; (5) Create raster images of each crop plot; Import the image in step (4) into the envi software to obtain a raster image classification obtained by the maximum likelihood classification method; create an independent vector layer for each crop category, that is, obtain a .shp file of a single crop plot; use the ArcGis software to use the original image as the input raster and the obtained .shp files of different crops as the output range, so that the sizes of the images of each crop after cropping are the same, and export the cropped images as TIFF images, thereby obtaining plot images of different crops; (6) Obtain productivity prediction maps for multiple crops: Import the different crop plot maps generated in step (5) into the ENVI software to obtain yield prediction maps.

3. The method for identifying various crop plots and predicting productivity based on UAV images using DeeplabV3+ according to claim 2 is characterized in that: The specific method of step (5) is: The built-in Region of Interest (ROI) Tool of envi software was used to extract the ROI areas of different crops. The Maximum Likelihood Classification tool was used to obtain the raster image classification obtained by the maximum likelihood classification method, including the classification categories of different crops. The Classification to Vector tool in envi software was used to create an independent vector layer for different crop categories, that is, to obtain the .shp file of a single crop plot. The cropping tool in ArcGis software was used to take the original image as the input raster and the .shp files of different crops as the output range. One of the minimum X value and the maximum X value in the rectangle was set to 0, and one of the minimum Y value and the maximum Y value in the rectangle was set to 0, so that the sizes of the crop images after cropping were the same, so as to facilitate the subsequent image fusion operation, and the cropped images were exported as TIFF images, and the plot images of different crops were obtained.

4. The method for identifying various crop plots and predicting productivity based on UAV images using DeeplabV3+ according to claim 2, characterized in that: The specific method of step (6) is: Based on one of the crop plots, the yield prediction model for the key growth period of the crop is integrated. That is, the Co-occurrence Measures tool in the ENVI software is used to obtain the red band mean and blue band mean required by the key growth period yield prediction model. The Band Math tool is then used to bring the obtained values ​​into the key growth period yield prediction model for calculation. After removing the outliers, the Edit ENVI Header tool is used to set the Data Ignore Value to 0 and export it to a crop yield prediction map in tif format. Finally, the Band Math tool in the ENVI software was used to merge the prediction maps of other crops, and finally the yield prediction maps of multiple crops were obtained.

Images