Wheat yield estimation method based on UAV low-altitude visible light images and target detection network

Abstract

The present invention relates to a wheat yield estimation method based on low-altitude visible light images of unmanned aerial vehicles and a target detection network, comprising: planning a waypoint sampling route of a wheat plot; obtaining a high-resolution visible light image taken by a unmanned aerial vehicle at low altitude, and performing preprocessing to establish a first training set and a second training set; respectively constructing a wheat ear detection and counting model and a wheat ear grain detection and counting model based on a small target detection neural network; and calculating the estimated yield of wheat in the measured plot. The present invention does not need to enter the wheat plot for sampling, and does not need to pick wheat ears. The sampling range and sampling efficiency are greatly increased by using a fixed-point and fixed-height aerial photography sampling method of an unmanned aerial vehicle; by constructing a wheat ear detection and counting model and a wheat ear grain detection and counting model based on a target detection neural network, and calculating the wheat yield accordingly, the efficiency and accuracy of the yield estimation are improved compared with the traditional ground survey yield estimation method.

Description

Technical Field

[0001] The invention relates to the technical field of unmanned aerial vehicles and wheat yield estimation, in particular to a wheat yield estimation method based on unmanned aerial vehicle low-altitude visible light images and a target detection network. Background Art

[0002] Estimating wheat yield is an indispensable step in agricultural production. Timely and accurate estimation of wheat yield during maturity can provide data support for agricultural production management and grain yield statistics. The estimation of wheat yield is mainly based on the number of wheat ears per unit area and the number of grains on the ears. The traditional method is to pick wheat per unit area through ground survey, and then manually count and estimate. Not only is the accuracy affected by human factors, but large-scale ground surveys will also consume a lot of manpower and time costs, resulting in low yield estimation efficiency.

[0003] With the advancement of image processing and deep learning technology, image-based small target detection has become a research hotspot in recent years. However, wheat yield estimation is limited by the following aspects: how to efficiently obtain large-scale, high-resolution wheat images, especially for the object of ears and grains; traditional methods generally frame a 1 square meter area for investigation, how to conveniently and quickly calculate the sampling area for non-contact sampling; due to the complex background of wheat fields and the small size of wheat ears and grains, high requirements are placed on the small target detection network.

[0004] Therefore, a wheat yield estimation method based on UAV low-altitude visible light images and target detection network is needed. Summary of the invention

[0005] The purpose of the present invention is to provide a wheat yield estimation method based on unmanned aerial vehicle low-altitude visible light images and target detection networks, which greatly increases the sampling range and sampling efficiency, and improves the efficiency and accuracy of yield estimation.

[0006] To achieve the above object, the present invention adopts the following technical solution: a wheat yield estimation method based on UAV low-altitude visible light images and target detection network, the method comprising the following steps in order:

[0007] (1) Planning the sampling route of the wheat field: Using a drone equipped with a high-pixel, zoom visible light camera, plan the sampling points and paths of the drone waypoint flight within the wheat field;

[0008] (2) obtaining high-resolution visible light images taken by a UAV at low altitude and preprocessing them, and establishing a training set with the preprocessed high-resolution visible light images, wherein the training set is divided into a first training set and a second training set;

[0009] (3) respectively constructing a wheat ear detection and counting model and a wheat grain detection and counting model based on a small target detection neural network, inputting the first training set into the wheat ear detection and counting model for training, and inputting the second training set into the wheat grain detection and counting model for training;

[0010] (4) Calculate the estimated yield of wheat in the measured plot: Input the sampling parameters of the wheat plot to be measured and the high-resolution visible light image taken by the drone into the wheat ear detection and counting model and the wheat grain detection and counting model respectively, and calculate and output the estimated yield of wheat.

[0011] In step (1), the effective pixel of the visible light camera is better than 20 million, and the zoom ratio is greater than or equal to 4 times; the sampling points are set by an average sampling method, and 100 sampling points are set in a 10*10 manner; the parameters of the sampling points include serial number, longitude, latitude, altitude, direction angle and zoom ratio.

[0012] The step (2) specifically comprises the following steps:

[0013] (2a) Two photos are taken at each sampling point, wherein the first photo is taken normally for subsequent wheat ear detection, and the second photo is taken at a zoomed-in focal length according to a set zoom factor for subsequent ear grain detection;

[0014] (2b) marking wheat ears and grains in the effective area of ​​the captured high-resolution visible light image;

[0015] (2c) The labeled high-resolution visible light images are preprocessed by sampling rotation, flipping, scaling, and mosaicking to establish a training set, wherein the first photos of all sampling points after preprocessing constitute a first training set, and the second photos of all sampling points after preprocessing constitute a second training set.

[0016] In step (3), the construction of the wheat ear detection and counting model specifically includes the following steps:

[0017] (3a) Construct the backbone network, feature extraction network and neck module, which together form the basic detection model YOLOv5;

[0018] (3b) Combine the spatial attention module and the channel attention module to construct a convolutional attention module. Given an intermediate feature map X∈R C×H×W As input, the feature map size is H×W and the channel dimension is C; the convolutional attention module in turn derives a one-dimensional channel attention map CM∈R C×1×1 and a 2D spatial attention map SM∈R 1×H×W , the whole attention process is:

[0019]

[0020]

[0021] in is the element multiplication operation;

[0022] The one-dimensional channel attention map is generated by calculating the channel relationship of the features, using the maximum pooling operation and the average pooling operation to compress the spatial dimension of the given feature map and aggregate the spatial information of the feature map. The formula is as follows:

[0023] CM(X)=σ(MLP(avg(X))+MLP(max(X))

[0024] Among them, σ is the sigmax activation function, MLP is the multi-layer perceptron, avg is the average pooling operation, and max is the maximum pooling operation;

[0025] The two-dimensional spatial attention map is generated by the spatial relationship between features. The average pooling operation and the maximum pooling operation are applied in the channel dimension, and the generated two-dimensional map is connected. Then, the required two-dimensional spatial attention map is generated through the convolution operation. The specific formula is as follows:

[0026] SM(X′)=σ(Conv([avg(X);max(X)]))

[0027] Among them, Conv is a convolution operation with a convolution kernel size of 7x7, [avg(X); max(X)] is the connection of avg(X) and max(X);

[0028] (3c) Improve the feature extraction network in the basic detection model YOLOv5 to obtain an improved feature extraction network, add a small target detection layer, and perform small target detection after concatenating the shallow feature map with the deep feature map: In the neck module in the basic detection model YOLOv5, the feature map is further upsampled after the convolution layer output of the backbone network, so that the feature map continues to expand, and then its output is fused with the corresponding C2 layer feature map in the backbone network to generate a feature map that is conducive to small target detection;

[0029] (3d) The basic detection model YOLOv5, the convolutional attention module constructed in step (3b), and the improved feature extraction network in step (3c) jointly construct a wheat ear detection and counting model based on a small target detection neural network;

[0030] The method for constructing the wheat grain detection and counting model is consistent with the method for constructing the wheat ear detection and counting model.

[0031] The step (4) specifically comprises the following steps:

[0032] (4a) Calculate the unit area of ​​the wheat field according to the camera parameters:

[0033] (4a1) Calculate the pixel size, the calculation formula is as follows:

[0034] ps h =sl / hpn

[0035] ps v =sw / vpn

[0036] Where sl is the sensor size length of the camera, hpn is the number of horizontal pixels of the image, sw is the sensor size width of the camera, and vpn is the number of vertical pixels; ps h is the horizontal pixel size, ps v is the vertical pixel size;

[0037] (4a2) Calculate the ground sampling interval GSD, the calculation formula is as follows:

[0038] GSD h =H×ps h / f

[0039] GSD v =H×ps v / f

[0040] Among them, H is the relative height of the UAV, f is the focal length of the camera, GSD h is the sampling interval in the horizontal direction of the ground, GSD v is the sampling interval in the vertical direction of the ground;

[0041] (4a3) Calculate the unit area uA of the image, and the calculation formula is as follows:

[0042] uA=hpn×GSD h ×vpn×GSD v ;

[0043] The result of the calculation represents the actual area of ​​the ground covered by an image;

[0044] (4b) Calculation of wheat ears and grains per ear:

[0045] (4b1) inputting the first image obtained at each sampling point into the wheat ear detection and counting model, and obtaining the number of wheat ears in the image by counting the coordinate values ​​of the wheat ears;

[0046] (4b2) inputting the second image obtained from each sampling point into the wheat grain detection and counting model, and obtaining the number of grains in the image by counting the grain coordinate values;

[0047] (4c) The estimated wheat yield is calculated as follows:

[0048]

[0049] Where n is the number of sampling points, m i is the number of wheat ears at the i-th sampling point, s i is the number of spikelets at the i-th sampling point, uA1 i is the unit area of ​​the first photo of the i-th sampling point, uA2 i is the unit area of ​​the second photo of the ith sampling point, w is the thousand-grain weight, and α is the adjustment coefficient, which is set to 0.85.

[0050] The height is 3 meters, the direction angle is set to -90 degrees, and the zoom factor is set to 4 times.

[0051] It can be seen from the above technical scheme that the beneficial effects of the present invention are: first, the present invention plans the waypoint sampling route of the wheat plot by determining the position, height and other parameters of the low-flying sampling of the drone, then obtains the high-resolution visible light image taken by the drone at low altitude and preprocesses it, and then constructs the wheat ear detection and counting model and the wheat grain detection and counting model based on the target detection neural network, and finally estimates the average yield of wheat in the measured plot by automatically calculating the unit area of ​​the sampling point; second, the present invention does not need to enter the wheat plot for sampling, and does not need to pick wheat ears. The sampling range and sampling efficiency are greatly increased by the fixed-point and fixed-height aerial photography sampling of the drone; third, by constructing the wheat ear detection and counting model and the wheat grain detection and counting model based on the target detection neural network, and calculating the wheat yield accordingly, the efficiency and accuracy of the yield estimation are improved compared with the traditional ground survey yield estimation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 This is a schematic diagram of wheat ear detection and counting obtained by the present invention. DETAILED DESCRIPTION

[0054] like Figure 1 As shown, a wheat yield estimation method based on UAV low-altitude visible light images and target detection network includes the following steps in sequence:

[0055] (1) Planning the sampling route of the wheat field: Using a drone equipped with a high-pixel, zoom visible light camera, plan the sampling points and paths of the drone waypoint flight within the wheat field;

[0056] (2) obtaining high-resolution visible light images taken by a UAV at low altitude and preprocessing them, and establishing a training set with the preprocessed high-resolution visible light images, wherein the training set is divided into a first training set and a second training set;

[0057] (3) respectively constructing a wheat ear detection and counting model and a wheat grain detection and counting model based on a small target detection neural network, inputting the first training set into the wheat ear detection and counting model for training, and inputting the second training set into the wheat grain detection and counting model for training;

[0058] (4) Calculate the estimated yield of wheat in the measured plot: Input the sampling parameters of the wheat plot to be measured and the high-resolution visible light image taken by the drone into the wheat ear detection and counting model and the wheat grain detection and counting model respectively, and calculate and output the estimated yield of wheat.

[0059] In step (1), the effective pixel of the visible light camera is better than 20 million, and the zoom ratio is greater than or equal to 4 times; the sampling points are set by an average sampling method, and 100 sampling points are set in a 10*10 manner; the parameters of the sampling points include serial number, longitude, latitude, altitude, direction angle and zoom ratio.

[0060] The step (2) specifically comprises the following steps:

[0061] (2a) Two photos are taken at each sampling point, wherein the first photo is taken normally for subsequent wheat ear detection, and the second photo is taken at a zoomed-in focal length according to a set zoom factor for subsequent ear grain detection;

[0062] (2b) marking wheat ears and grains in the effective area of ​​the captured high-resolution visible light image;

[0063] (2c) The labeled high-resolution visible light images are preprocessed by sampling rotation, flipping, scaling, and mosaicking to establish a training set, wherein the first photos of all sampling points after preprocessing constitute a first training set, and the second photos of all sampling points after preprocessing constitute a second training set.

[0064] In step (3), the construction of the wheat ear detection and counting model specifically includes the following steps:

[0065] (3a) Construct the backbone network, feature extraction network and neck module, which together form the basic detection model YOLOv5;

[0066] (3b) Combine the spatial attention module and the channel attention module to construct a convolutional attention module. Given an intermediate feature map X∈R C×H×WAs input, the feature map size is H×W and the channel dimension is C; the convolutional attention module in turn derives a one-dimensional channel attention map CM∈R C×1×1 and a 2D spatial attention map SM∈R 1×H×W , the whole attention process is:

[0067]

[0068]

[0069] in is the element multiplication operation;

[0070] The one-dimensional channel attention map is generated by calculating the channel relationship of the features, using the maximum pooling operation and the average pooling operation to compress the spatial dimension of the given feature map and aggregate the spatial information of the feature map. The formula is as follows:

[0071] CM(X)=σ(MLP(avg(X))+MLP(max(X))

[0072] Among them, σ is the sigmax activation function, MLP is the multi-layer perceptron, avg is the average pooling operation, and max is the maximum pooling operation;

[0073] The two-dimensional spatial attention map is generated by the spatial relationship between features. The average pooling operation and the maximum pooling operation are applied in the channel dimension, and the generated two-dimensional map is connected. Then, the required two-dimensional spatial attention map is generated through the convolution operation. The specific formula is as follows:

[0074] SM(X′)=σ(Conv([avg(X);max(X)]))

[0075] Among them, Conv is a convolution operation with a convolution kernel size of 7x7, [avg(X); max(X)] is the connection of avg(X) and max(X);

[0076] (3c) Improve the feature extraction network in the basic detection model YOLOv5 to obtain an improved feature extraction network, add a small target detection layer, and perform small target detection after concatenating the shallow feature map with the deep feature map: In the neck module in the basic detection model YOLOv5, the feature map is further upsampled after the convolution layer output of the backbone network, so that the feature map continues to expand, and then its output is fused with the corresponding C2 layer feature map in the backbone network to generate a feature map that is conducive to small target detection;

[0077] (3d) The basic detection model YOLOv5, the convolutional attention module constructed in step (3b), and the improved feature extraction network in step (3c) jointly construct a wheat ear detection and counting model based on a small target detection neural network;

[0078] The method for constructing the wheat grain detection and counting model is consistent with the method for constructing the wheat ear detection and counting model.

[0079] The step (4) specifically comprises the following steps:

[0080] (4a) Calculate the unit area of ​​the wheat field according to the camera parameters:

[0081] (4a1) Calculate the pixel size, the calculation formula is as follows:

[0082] ps h =sl / hpn

[0083] ps v =sw / vpn

[0084] Where sl is the sensor size length of the camera, hpn is the number of horizontal pixels of the image, sw is the sensor size width of the camera, and vpn is the number of vertical pixels; ps h is the horizontal pixel size, ps v is the vertical pixel size;

[0085] (4a2) Calculate the ground sampling interval GSD, the calculation formula is as follows:

[0086] GSD h =H×ps h / f

[0087] GSD v =H×ps v / f

[0088] Among them, H is the relative height of the UAV, f is the focal length of the camera, GSD h is the sampling interval in the horizontal direction of the ground, GSD v is the sampling interval in the vertical direction of the ground;

[0089] (4a3) Calculate the unit area uA of the image, and the calculation formula is as follows:

[0090] uA=hpn×GSD h ×vpn×GSD v ;

[0091] The result of the calculation represents the actual area of ​​the ground covered by an image;

[0092] (4b) Calculation of wheat ears and grains per ear:

[0093] (4b1) inputting the first image obtained at each sampling point into the wheat ear detection and counting model, and obtaining the number of wheat ears in the image by counting the coordinate values ​​of the wheat ears;

[0094] (4b2) inputting the second image obtained from each sampling point into the wheat grain detection and counting model, and obtaining the number of grains in the image by counting the grain coordinate values;

[0095] (4c) The estimated wheat yield is calculated as follows:

[0096]

[0097] Where n is the number of sampling points, m i is the number of wheat ears at the i-th sampling point, s i is the number of spikelets at the i-th sampling point, uA1 i is the unit area of ​​the first photo of the i-th sampling point, uA2 i is the unit area of ​​the second photo of the ith sampling point; w is the thousand-grain weight, which is generally obtained based on the average value of the previous three years or the regional trial data of the variety; α is the adjustment coefficient, which is set to 0.85.

[0098] The height is 3 meters, the direction angle is set to -90 degrees, and the zoom factor is set to 4 times.

[0099] like Figure 2 As shown in the figure, the first image taken by the drone at a low altitude covers a ground area of ​​13.5 square meters. After inputting the wheat ear detection and counting model, the number of wheat ears is 1389.

[0100] To sum up, the present invention does not need to enter the wheat field for sampling, nor does it need to pick wheat ears. The sampling range and sampling efficiency are greatly increased by using the fixed-point and fixed-height aerial photography sampling method of unmanned aerial vehicles; thirdly, by constructing a wheat ear detection and counting model and a wheat grain detection and counting model based on the target detection neural network, and calculating the wheat yield accordingly, the efficiency and accuracy of the yield estimation are improved compared with the traditional ground survey yield estimation method.

Claims

1. A wheat yield estimation method based on low-altitude visible light images from unmanned aerial vehicles and a target detection network, characterized in that: The method comprises the following steps in order: (1) Planning the sampling route of the wheat field: Using a drone equipped with a high-pixel, zoom visible light camera, plan the sampling points and paths of the drone waypoint flight within the wheat field; (2) obtaining high-resolution visible light images taken by a UAV at low altitude and preprocessing them, and establishing a training set with the preprocessed high-resolution visible light images, wherein the training set is divided into a first training set and a second training set; (3) respectively constructing a wheat ear detection and counting model and a wheat grain detection and counting model based on a small target detection neural network, inputting the first training set into the wheat ear detection and counting model for training, and inputting the second training set into the wheat grain detection and counting model for training; (4) Calculating the estimated yield of wheat in the measured plot: input the sampling parameters of the drone and the high-resolution visible light image of the wheat plot to be measured into the wheat ear detection and counting model and the wheat ear grain detection and counting model, respectively, and calculate and output the estimated yield of wheat; In step (3), the construction of the wheat ear detection and counting model specifically includes the following steps: (3a) Construct the backbone network, feature extraction network and neck module, which together form the basic detection model YOLOv5; (3b) Construct a convolutional attention module by combining the spatial attention module and the channel attention module; (3c) Improve the feature extraction network in the basic detection model YOLOv5 to obtain an improved feature extraction network, add a small target detection layer, and perform small target detection after concatenating the shallow feature map with the deep feature map: In the neck module in the basic detection model YOLOv5, the feature map is further upsampled after the convolution layer output of the backbone network, so that the feature map continues to expand, and then its output is fused with the corresponding C2 layer feature map in the backbone network to generate a feature map that is conducive to small target detection; (3d) The basic detection model YOLOv5, the convolutional attention module constructed in step (3b), and the improved feature extraction network in step (3c) jointly construct a wheat ear detection and counting model based on a small target detection neural network; The method for constructing the wheat grain detection and counting model is consistent with the method for constructing the wheat ear detection and counting model.

2. The wheat yield estimation method based on UAV low-altitude visible light image and target detection network according to claim 1 is characterized in that: In step (1), the effective pixel of the visible light camera is better than 20 million, and the zoom ratio is greater than or equal to 4 times; the sampling points are set by an average sampling method, and 100 sampling points are set in a 10*10 manner; the parameters of the sampling points include serial number, longitude, latitude, altitude, direction angle and zoom ratio.

3. The wheat yield estimation method based on UAV low-altitude visible light image and target detection network according to claim 1 is characterized in that: The step (2) specifically comprises the following steps: (2a) Two photos are taken at each sampling point, wherein the first photo is taken normally for subsequent wheat ear detection, and the second photo is taken at a zoomed-in focal length according to a set zoom factor for subsequent ear grain detection; (2b) marking wheat ears and grains in the effective area of ​​the captured high-resolution visible light image; (2c) The labeled high-resolution visible light images are preprocessed by sampling rotation, flipping, scaling, and mosaicking to establish a training set, wherein the first photos of all sampling points after preprocessing constitute a first training set, and the second photos of all sampling points after preprocessing constitute a second training set.

4. The wheat yield estimation method based on UAV low-altitude visible light image and target detection network according to claim 1 is characterized in that: Step (3b) specifically means: given an intermediate feature map X∈R C×H×W As input, the feature map size is H×W and the channel dimension is C; the convolutional attention module in turn derives a one-dimensional channel attention map CM∈R C×1×1 and a 2D spatial attention map SM∈R 1×H×W , the whole attention process is: in is the element multiplication operation; The one-dimensional channel attention map is generated by calculating the channel relationship of the features, using the maximum pooling operation and the average pooling operation to compress the spatial dimension of the given feature map and aggregate the spatial information of the feature map. The formula is as follows: CM(X)=σ(MLP(avg(X))+MLP(max(X)) Among them, σ is the sigmax activation function, MLP is the multi-layer perceptron, avg is the average pooling operation, and max is the maximum pooling operation; The two-dimensional spatial attention map is generated by the spatial relationship between features. The average pooling operation and the maximum pooling operation are applied in the channel dimension, and the generated two-dimensional map is connected. Then, the required two-dimensional spatial attention map is generated through the convolution operation. The specific formula is as follows: SM(X)=σ(Chnv([avg(X);max(X)])) Among them, Conv is a convolution operation with a convolution kernel size of 7x7, and [avg(X); max(X)] is the connection between avg(X) and max(X).

5. The wheat yield estimation method based on UAV low-altitude visible light image and target detection network according to claim 1 is characterized in that: The step (4) specifically comprises the following steps: (4a) Calculate the unit area of ​​the wheat field according to the camera parameters: (4a1) Calculate the pixel size, the calculation formula is as follows: ps h =sl / hpn ps v =sw / vpn Where sl is the sensor size length of the camera, hpn is the number of horizontal pixels of the image, Sw is the sensor size width of the camera, and vpn is the number of vertical pixels; ps h is the horizontal pixel size, ps v is the vertical pixel size; (4a2) Calculate the ground sampling interval GSD, the calculation formula is as follows: GSD h =H×ps h / f GSD v =H×ps v / f Among them, H is the relative height of the UAV, f is the focal length of the camera, GSD h is the sampling interval in the horizontal direction of the ground, GSD v is the sampling interval in the vertical direction of the ground; (4a3) Calculate the unit area uA of the image, and the calculation formula is as follows: uA=hpn×GSD h ×vpn×GSD v ; The result of the calculation represents the actual area of ​​the ground covered by an image; (4b) Calculation of wheat ears and grains per ear: (4b1) inputting the first image obtained at each sampling point into the wheat ear detection and counting model, and obtaining the number of wheat ears in the image by counting the coordinate values ​​of the wheat ears; (4b2) inputting the second image obtained from each sampling point into the wheat grain detection and counting model, and obtaining the number of grains in the image by counting the grain coordinate values; (4c) The estimated wheat yield is calculated as follows: Where n is the number of sampling points, m i is the number of wheat ears at the i-th sampling point, s i is the number of spikelets at the i-th sampling point, uA1 i is the unit area of ​​the first photo of the i-th sampling point, uA2 i is the unit area of ​​the second photo of the ith sampling point, w is the thousand-grain weight, and α is the adjustment coefficient, which is set to 0.

85.

6. The wheat yield estimation method based on UAV low-altitude visible light image and target detection network according to claim 2 is characterized in that: The height is 3 meters, the direction angle is set to -90 degrees, and the zoom factor is set to 4 times.

Images