A corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion

Abstract

The application provides a method for predicting corn yield based on a multi-modal unmanned aerial vehicle, so as to accurately predict the corn yield in a farmland, and on one hand, provides a new method for yield evaluation of a corn crop, and on the other hand, provides a new idea for yield estimation of other crops. Compared with traditional research which is mainly based on single type remote sensing data and traditional machine learning methods, the yield prediction model obtained has poor robustness, and the multi-modal fusion unmanned aerial vehicle remote sensing data can effectively improve the yield prediction efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of crop yield remote sensing prediction, and particularly relates to a corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion. BACKGROUND

[0002] Yield, as the ultimate goal of agricultural planting, is the most direct economic parameter for evaluating farmland productivity. Corn is one of the world's three major cereal crops and has important value in terms of food, feed processing and industrial production. Monitoring and accurate yield estimation of corn crops before harvest can not only improve the utilization and decision-making efficiency of corn in the current year, but also has important significance for agricultural development. The development of remote sensing technology makes it possible to monitor crops during the growth period and estimate yield. The current remote sensing monitoring methods are divided into satellite remote sensing monitoring and unmanned aerial vehicle remote sensing monitoring. Satellite remote sensing monitoring has wide range and low cost, but it is greatly affected by weather and has low resolution compared with unmanned aerial vehicle images, which makes it impossible to monitor crops at a small scale with high frequency. Unmanned aerial vehicles equipped with different types of sensors can quickly obtain crop growth information, so they are widely used in crop yield estimation, crop nutrition diagnosis, growth characteristic evaluation and other aspects. At present, various UAV sensors such as laser radar, multispectral, RGB camera have been proved to have great potential in crop monitoring. Multispectral images can calculate crop vegetation index and reflect crop growth conditions for yield estimation. Airborne laser Lidar can accurately obtain three-dimensional vegetation information of crops and improve the estimation accuracy of crop vegetation parameters at different stages. In general, this method is widely used in various crops due to its robustness and universality, however, previous yield prediction models mainly use unmanned aerial vehicle remote sensing data of a single sensor, lacking effective utilization of multi-modal crop information. Fusing multi-modal data can make up for the shortcomings of single-modal features based on one-dimensional spectral information or two-dimensional RGB image information, effectively extract multi-dimensional structural features of crops and improve yield prediction accuracy. Machine learning algorithms are very suitable for processing non-linear heteroscedastic problems and can be used for efficient data processing and data mining. SUMMARY

[0003] The application aims to provide a corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion, solving the technical problem that the existing crop yield prediction model mainly uses unmanned aerial vehicle remote sensing data of a single sensor and lacks effective utilization of multi-modal crop information.

[0004] To solve the above technical problems, the application adopts the following technical solutions:

[0005] A corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion, comprising the following steps:

[0006] S1: Using the multispectral sensor and laser radar sensor carried by the unmanned aerial vehicle, long time series multispectral images and laser point cloud images of target farmland corn from the three-leaf stage to the milk stage are obtained, and the daily average temperature T, daily maximum temperature Tmax, minimum temperature Tmin and total solar radiation (KJ·m-2) of the target farmland observed by the weather station are obtained simultaneously. -2 ·d -1 );

[0007] S2: The multispectral images and laser radar point cloud images obtained by the unmanned aerial vehicle are georeferenced by using the georeferencing tool of ArcGIS software, that is, the positional deviation of the two images is corrected.

[0008] S3: The multispectral images and laser radar point cloud images after georeferencing and radiometric calibration are down-sampled to the same spatial resolution by using the re-sampling tool in ArcGIS software.

[0009] S4: The normalized vegetation index NDVI and the soil-adjusted vegetation index SAVI of the target farmland in long time series are calculated by using the processed multispectral images, and finally the NDVI image and the SAVI image of the target farmland in long time series are obtained.

[0010] S5: Based on step S4, the daily NDVI value in the long time series NDVI image is obtained by using linear interpolation.

[0011] S6: The collected radar point cloud images are denoised and modeled by using Pix4D Mapper software to generate LAS data set, and 0.1m×0.1m digital surface model (DSM) and dense point cloud are obtained.

[0012] S7: In order to fully extract the structure information of corn, the processed point cloud data is further layered and extracted by using Agisoft software.

[0013] S8: The point cloud is classified into ground point cloud and crop point cloud by using Cloth Simulation Filter (CSF) algorithm, and then the obtained ground point cloud is fitted to obtain digital elevation model (DEM).

[0014] S9: The digital surface model and the digital elevation model are imported into ArcGIS software, and the crown height model (CHM) is obtained by processing through the raster tool.

[0015] S10: The daily NDVI value, SAVI value, point cloud data Point75-100, Point 80-100 , Point 50-100 and the daily average temperature T, daily maximum temperature Tmax, daily minimum temperature Tmin observed by the weather station in step S1 are used as input data of the machine learning model. max ​min , total solar radiation (KJ·m -2 ·d -1 ), normalized processing, and deleting abnormal data in the data set, dividing the multi-modal and multi-temporal features by time series sliding window with an overlap rate of 85%, and taking the time window as an input variable, and the new time window needs to be consistent with the original feature label;

[0016] S11: the preprocessed data is input into the three branches of the CNN-attention-LSTM model as input information, the convolution kernels in the three convolution layers adaptively extract the corresponding data features, the data set is traversed through the convolution layer, then the convolution kernel weight and the feature matrix of local sequence convolution operation are obtained, one of the convolution layers is connected with the maximum pooling layer, the pooling layer is connected with the dropout layer, the feature map after convolution retains the original sequence order, the feature map enters two consecutive LSTM layers, the feature map is converted into a hidden state through the LSTM, and then the feature map after the LSTM layer is fused to generate a new feature map;

[0017] S12: the hidden state is calculated through the self-attention mechanism to obtain a feature vector z, and the representation Ct of the hidden state is obtained through the activation function tanh, wherein W is a weight matrix, b is a bias vector, the weight a is measured by the normalization operation of the Softmax function t , and the final vector z is a weighted sum of the hidden state;

[0018] S13: the fused multi-modal features are converted through the output part, the feature samples are input into the ElasticNet network layer for training, and a yield prediction result is obtained.

[0019] S14: the actual yield is determined according to the nitrogen use efficiency (aNUE), the leaf area index LAI and the aboveground biomass AGB measured at the key growth stages of the target farmland corn from the three-leaf stage to the milk ripening stage.

[0020] Further, the multispectral image obtained by the unmanned aerial vehicle is radiometrically calibrated by the sky image obtained at the same period in step S3 to determine the actual reflectivity of the multispectral image obtained by the unmanned aerial vehicle, and the specific content is as follows:

[0021]

[0022] In the above formula, a i ×DN i +b is the radiance, E0 is the solar irradiance, different solar irradiance corresponds to each waveband, and d takes the value of 1.

[0023] Further, the Cloth Simulation Filter algorithm is used in step S8 to classify the point cloud into ground point cloud and crop point cloud, and then the obtained ground point cloud is fitted to obtain the specific steps of the digital elevation model as follows:

[0024] The basic formula of Cloth Simulation Filter algorithm is:

[0025]

[0026] Where X is the position of the "cloth" surface particle at time t, which is affected by external driving factor Fext(X, t) and internal driving factor Fint(X, t). Assuming that it is only affected by the external driving factor, set its internal driving factor to 0, and the relationship is:

[0027]

[0028] In the above formula, m is the weight of the cloth particle, Δt represents the time step, which is used to calculate the position of the iterative particle. At the same time, the internal driving factor Fint(X, t) can control the inversion problem of the particle in the blank area:

[0029]

[0030] In the above formula, d is the particle displacement, and n is the unit vector normalized to the vertical direction. Through the driving of the internal driving factor and the external driving factor, the height difference between the Lidar point cloud and the particle is calculated, and finally the ground point classification is obtained.

[0031] Further, in step S4, the specific calculation process of the normalized difference vegetation index NDVI is as follows:

[0032] (R NIR -R RED ) / (R NIR +R RED )

[0033] The specific calculation process of the soil-adjusted vegetation index SAVI is as follows:

[0034] ((R NIR -R RED ) / (R NIR +R RED +0.16))×(1+0.5)

[0035] Where, RED, GRE, BLUE, NIR represent the surface reflectivity of red channel, green channel, blue channel and near-infrared channel respectively.

[0036] Further, in step S9, the canopy height model (CHM) of the target farmland corn from the three-leaf stage to the milk stage is calculated based on the point cloud data:

[0037] CHM = DSM - DEM

[0038] The grid of the DSM and the DEM needs to be consistent during calculation.

[0039] Further, in step S11, after the feature maps passing through the LSTM layer are fused, a new hierarchical feature map is generated, and the hierarchical feature map specifically includes m hidden states xt, and the feature map representation formula is y:

[0040] y = f(x1, x2, x3,... x m )

[0041] Further, in step S12, C t , weight alpha t and the final vector z are respectively:

[0042] C t = tanh(Wx t +b)

[0043]

[0044]

[0045]

[0046] Compared with the prior art, the beneficial effects of the present application are: the present application provides a method for predicting corn yield based on multi-modal unmanned aerial vehicle, which is used for accurately predicting corn yield in farmland, on the one hand, a new method for evaluating the yield of corn crops is provided, and on the other hand, a new idea for yield estimation of other crops is provided. Compared with the traditional research which is mainly based on single type remote sensing data and traditional machine learning method, the yield prediction model obtained has poor robustness, and the use of multi-modal fusion unmanned aerial vehicle remote sensing data can effectively improve the prediction efficiency. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] The present application will be further described in detail below in combination with embodiments.

[0049] A corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion is provided in the embodiments of the present application:

[0050] A corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion includes the following steps:

[0051] S1: Use the multi-spectral sensor and laser radar sensor carried by the unmanned aerial vehicle to obtain long-time sequence multi-spectral images and laser point cloud images of corn in the target farmland from the three-leaf stage to the milk stage, and simultaneously obtain the daily average temperature T, the daily maximum temperature T max , the daily minimum temperature T min , and the total solar radiation (KJ·m -2 ·d -1 ) of the target farmland observed by the weather station;

[0052] S2: Use the geographic registration tool of the ArCGIS software to perform geographic registration on the multi-spectral images and laser radar point cloud images obtained by the unmanned aerial vehicle, that is, correct the position deviation in the two images;

[0053] S3: Use the resampling tool in the ArcGIS software to resample the multi-spectral images and laser radar point cloud images after geographic registration and radiation calibration to the same spatial resolution;

[0054] S4: Calculate the normalized vegetation index NDVI and the soil-adjusted vegetation index SAVI of the target farmland in the long-time sequence using the processed multi-spectral images, and finally obtain the NDVI image and the SAVI image of the target farmland in the long-time sequence;

[0055] S5: Based on step S4, use linear interpolation to obtain the daily NDVI value in the long-time sequence NDVI image;

[0056] S6: Use the Pix4D Mapper software to perform denoising and modeling processing on the collected radar point cloud image to generate a LAS data set, and simultaneously obtain a digital surface model (DSM) and a dense point cloud with a resolution of 0.1m×0.1m;

[0057] S7: To fully extract the corn structure information, use the Agisoft software to further extract the processed point cloud data;

[0058] S8: Use the Cloth Simulation Filter (CSF) algorithm to classify the point cloud into ground point cloud and crop point cloud, and then fit the obtained ground point cloud to obtain a digital elevation model (DEM);

[0059] S9: Import the digital surface model and the digital elevation model into the ArcGIS software, and process them through the raster tool to obtain a canopy height model (CHM);

[0060] S10: The acquired daily NDVI values, SAVI values, point cloud data Point75-100, Point80-100, Point50-100, and the daily average temperature T, daily maximum temperature T max , daily minimum temperature T min , total solar radiation (KJ·m -2 ·d -1 ) observed by the weather station in step S1 are normalized, and abnormal data in the data set are deleted. The multi-modal and multi-temporal features are divided by a time series sliding window with an overlap rate of 85%, and the obtained time window is used as an input variable. The new time window needs to be consistent with the original feature label;

[0061] S11: The preprocessed data are input into the three branches of the CNN-attention-LSTM model as input information. The convolution kernels in the three convolution layers adaptively extract the corresponding data features. The data set is traversed through the convolution layer, and then the convolution kernel weight and the feature matrix of local sequence convolution operation are obtained. One of the convolution layers is connected with the maximum pooling layer, the pooling layer is connected with the dropout layer, the feature map after convolution retains the original sequence order, the feature map enters two consecutive LSTM layers, the feature map is converted into a hidden state through the LSTM, and then the feature map after the LSTM layer is fused to generate a new feature map.

[0062] S12: The hidden state is calculated through the self-attention mechanism to obtain a feature vector z. The hidden state representation C t is obtained through the activation function tanh. In the formula, W is a weight matrix, b is a bias vector, the weight a t is measured by the normalization operation of the Softmax function, and the final vector z is a weighted sum of the hidden state.

[0063] S13: The fused multi-modal features are converted through the output part, the feature samples are input into the ElasticNet network layer for training, and the yield prediction result is obtained.

[0064] S14: The actual yield is determined according to the nitrogen use efficiency (aNUE), leaf area index LAI, and aboveground biomass AGB measured during the key growth periods of the target farmland corn from the three-leaf stage to the milk ripening stage.

[0065] The specific content of the above step S3 of performing radiometric calibration on the multi-spectral image acquired by the unmanned aerial vehicle through the sky image acquired at the same period to determine the actual reflectivity of the multi-spectral image acquired by the unmanned aerial vehicle is as follows:

[0066]

[0067] In the above formula, a i ×DN i +b is the radiant brightness, E0 is the solar irradiance, different solar irradiance corresponds to different wave bands, and d is usually 1.

[0068] The specific steps of classifying the point cloud into ground point cloud and crop point cloud by using Cloth Simulation Filter algorithm in the above step S8, and then fitting the obtained ground point cloud to obtain the digital elevation model are as follows:

[0069] The basic formula of Cloth Simulation Filter algorithm is:

[0070]

[0071] Where X is the position of the "cloth" surface particle at time t, which is affected by external driving factor Fext(X, t) and internal driving factor Fint(X, t). Assuming that it is only affected by external driving factor, set its internal driving factor to 0, and its relationship is:

[0072]

[0073] In the above formula, m is the weight of the cloth particle, Δt represents the time step, which is used to calculate the position of the iterative particle, and the internal driving factor Fint(X, t) can control the inversion problem of the particle in the blank area:

[0074]

[0075] In the above formula, is the particle displacement amount, and is the unit vector normalized to the vertical direction; by driving the internal driving factor and the external driving factor, the height difference between the Lidar point cloud and the particle is calculated, and finally the ground point classification is obtained.

[0076] The specific calculation process of normalized difference vegetation index NDVI in the above step S4 is as follows:

[0077] (R NIR -R RED ) / (R NIR +R RED )

[0078] The specific calculation process of soil-adjusted vegetation index SAVI is as follows:

[0079] ((R NIR -R RED ) / (R NIR +R RED +0.16))×(1+0.5)

[0080] Wherein, RED, GRE, BLUE, NIR represent the surface reflectance of red channel, green channel, blue channel, near-infrared channel respectively.

[0081] In the above step S9, the canopy height model (CHM) of the target farmland corn from the three-leaf stage to the milk ripening stage is calculated according to the point cloud data:

[0082] CHM = DSM-DEM

[0083] The grid of the DSM and the DEM needs to be consistent during the calculation.

[0084] In the above step S11, after the feature maps passing through the LSTM layer are fused, a new hierarchical feature map is generated, and the hierarchical feature map specifically includes m hidden states xt, and the feature map representation formula is y:

[0085] y = f(x1, x2, x3, ……xm) m )

[0086] In the above step S12, Ct, weight αt and final vector z are respectively:

[0087] C t = tanh(Wx t +b)

[0088]

[0089]

[0090]

Claims

1. A corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion, characterized in that, The method comprises the following steps: S1: Use the multispectral sensor and laser radar sensor carried by the unmanned aerial vehicle to obtain long time series multispectral images and laser point cloud images of target farmland corn from the three-leaf stage to the milk ripening stage, and obtain the daily average temperature T, daily maximum temperature T max , daily minimum temperature T min and total solar radiation KJ / m -2 d -1 of the target farmland observed by the weather station every day; S2: using the geographic registration tool of the ArCGIS software to perform geographic registration on the multispectral image and the laser radar point cloud image acquired by the unmanned aerial vehicle, i.e., correcting the position deviation in the two images; S3: using the resampling tool in the ArcGIS software to reduce the sampling of the multispectral image and the laser radar point cloud image after geographic registration and radiation calibration to the same spatial resolution; S4: calculating the normalized vegetation index NDVI and the soil-adjusted vegetation index SAVI of the target farmland long time sequence by using the processed multispectral image, and finally obtaining the NDVI image and the SAVI image of the target farmland long time sequence; S5: based on step S4, using linear interpolation to obtain the daily NDVI value in the long time sequence NDVI image; S6: using the Pix4D Mapper software to perform denoising and modeling processing on the collected radar point cloud image, generating a LAS data set, and simultaneously obtaining a 0.1m*0.1m digital surface model and a dense point cloud; S7: in order to fully extract the corn structure information, using the Agisoft software to further extract the processed point cloud data; S8: using the Cloth Simulation Filter algorithm to classify the point cloud into ground point cloud and crop point cloud, and then fitting the obtained ground point cloud to obtain a digital elevation model; S9: importing the digital surface model and the digital elevation model into the ArcGIS software, and obtaining a canopy height model through the raster tool processing; S10: The acquired daily NDVI value, SAVI value, point cloud data Point 75-100 , Point 80-100 , Point 50-100 , and the daily average temperature T, daily maximum temperature T max , daily minimum temperature T min , and total solar radiation KJ m -2 d -1 observed in step S1 are normalized and abnormal data in the data set is deleted, multi-modal and multi-temporal features are divided by time series sliding window according to an overlap rate of 85%, and the obtained time window is used as an input variable. The new time window needs to be consistent with the labeled original feature label; S11: inputting the preprocessed data as input information into three branches in the CNN-attention-LSTM model, adaptively extracting corresponding data features in the three convolutional layers, traversing the data set through the convolutional layer, and then obtaining the convolution kernel weight and the feature matrix of the local sequence convolution operation, wherein one convolutional layer is connected with a max-pooling layer, the pooling layer is connected with a dropout layer, the feature map after convolution retains the original sequence order, the feature map enters two consecutive LSTM layers, and the feature map after the LSTM layer is fused to generate a new feature map; S12: Calculate the hidden state through the self-attention mechanism to obtain a feature vector z, and obtain the representation C of the hidden state through the activation function tanh t , in which W is a weight matrix, b is a bias vector, and the normalization operation of the Softmax function measures the weight a t , and the final vector z is the weighted sum of the hidden states; S13: converting the fused multi-modal features through the output part, inputting the feature samples into an elastic regression network layer for training, and obtaining a yield prediction result; S14: determining the actual yield according to the nitrogen use efficiency, the leaf area index LAI and the aboveground biomass AGB of the target farmland corn measured from the three-leaf stage to the milk ripening stage; The specific steps of using the Cloth Simulation Filter algorithm to classify the point cloud into ground point cloud and crop point cloud, and then fitting the obtained ground point cloud to obtain a digital elevation model in step S8 are as follows: The basic formula of the Cloth Simulation Filter algorithm is: where X is the position of the surface particle of the cloth at time t, which is subjected to an external driving factor F ext (X,t) and the internal driving factor F int (X,t) is affected, assuming that it is only affected by the external driving factor, its internal driving factor is set to 0, and the relationship is: m is the weight of the cloth particle, Δt represents the time step for calculating the position of the iterative particle, while the internal driving factor F int (X,t) can control the inversion problem of the particle in the blank area: In the above formula is the particle displacement amount, is the unit vector normalized to the vertical direction; through the driving of internal and external driving factors, the height difference between the Lidar point cloud and the particle is calculated, and finally the ground point classification is obtained.

2. The corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion according to claim 1, characterized in that: In step S3, the multispectral image acquired by the unmanned aerial vehicle is radiometrically calibrated by using the sky image acquired at the same time to determine the actual reflectivity of the multispectral image acquired by the unmanned aerial vehicle, and the specific content is as follows: In the above formula, a i × DN i +b is the radiant luminance, E0 is the solar irradiance, different solar irradiance for each waveband, d is 1.

3. The corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion according to claim 1, characterized in that: The specific calculation process of the normalized difference vegetation index NDVI in step S4 is as follows: (R NIR -R RED ) / (R NIR +R RED ) The specific calculation process of the soil-adjusted vegetation index SAVI is as follows: ((R NIR -R RED ) / (R NIR +R RED +0.16))×(1+0.5) Wherein, RED, GRE, BLUE, NIR respectively represent the surface reflectivity of the red channel, the green channel, the blue channel and the near-infrared channel.

4. The corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion according to claim 1, characterized in that: In step S9, the canopy height model of the target farmland corn from the three-leaf stage to the milk ripening stage is calculated by using the point cloud data: CHM = DSM-DEM When calculating, the grid of DSM and DEM should be consistent.

5. The corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion according to claim 1, characterized in that: In step S11, after the feature maps passing through the LSTM layer are fused, a new level feature map is generated, and the level feature map specifically includes m hidden states x t , and the feature map representation formula is y: y = f(x1, x2, x3,... x m ).

6. The corn yield prediction model based on multi-modal unmanned aerial vehicle data fusion according to claim 1, wherein: In step S12, Ct, the weight α t and the final vector z are respectively: C t = tanh(Wx + b) t +b)

Images