Unmanned aerial vehicle image wheat seedling identification method based on deep learning

Abstract

The invention discloses an unmanned aerial vehicle image wheat seedling identification method based on deep learning, and the method can output the spatial position, angle direction and other information of wheat seedlings, and the construction of an identification model comprises the steps: S1, data collection; s2, a data processing step; s3, a model construction step: constructing a wheat seedling recognition network and training the wheat seedling recognition network to obtain a wheat seedling recognition model; s4, performing wheat seedling identification by adopting the constructed wheat seedling identification model, and outputting and storing information such as the position and the angle of a wheat seedling prediction frame; the method is characterized by comprising the following steps: S3, newly adding a micro-scale detection layer branch, and constructing a network detection layer with four scales; classifying each wheat seedling angle based on a circular smooth label, and extracting wheat seedling angle information; and S4, removing repeated wheat seedling prediction frames based on a rotation non-maximum suppression algorithm. The scheme provided by the invention can accurately identify small-size wheat seedlings in unmanned aerial vehicle images, is suitable for complex field scenes, and provides technical reference for wheat seedling identification.

Description

Technical Field

[0001] The present invention belongs to the field of smart agriculture, and specifically is a method for identifying wheat seedlings using drone images based on deep learning. Background Art

[0002] Wheat is one of the world's major food crops. The number of wheat seedlings is an important indicator of the size of the crop population at the seedling stage. It affects the grain structure to a certain extent, and ultimately affects the yield and quality of wheat. Therefore, the identification and counting of wheat seedlings is an important task and is of great significance for the correct estimation of wheat yield.

[0003] There are three main types of existing methods for identifying wheat seedlings: the first is traditional wheat seedling identification that relies on manual surveys, but manual sampling surveys are inefficient and costly, especially when human resources are limited and the sampling area is large, sampling errors are prone to occur, affecting the final statistical results. The second machine learning method uses classifiers such as Bayes, support vector machines, and random forests to identify wheat seedlings based on color, texture, and morphological features, but the selection of features relies on expert prior knowledge, the algorithm has poor generalization, and it is difficult to adapt to changes in object size, translation, rotation, etc., and it cannot be well applied to scenes with dense distribution and severe occlusion of wheat in the field. The third is the deep learning method used in the present invention, which does not rely on manual features based on statistical laws or prior knowledge, and its automatically learned features are more in line with the nature of the data.

[0004] Due to the small size of wheat seedlings in UAV images, many occlusions and serious soil background interference, there is currently a lack of UAV-based wheat seedling identification methods. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of small size of wheat seedlings, many occlusions and serious soil background interference in drone images, and propose a method for wheat seedling recognition in drone images based on deep learning.

[0006] To solve the above problems, the technical solution of the present invention is as follows:

[0007] S1, data collection, obtaining visible light images of wheat seedlings through drones;

[0008] S2, data processing, enhancing data images, marking wheat seedlings, and obtaining training sets and test sets;

[0009] S3. Model construction, constructing a wheat seedling recognition model based on a drone platform; the detection network structure of the wheat seedling recognition model extracts shallow spatial features from multi-scale features, and fuses the shallow spatial features with deep semantic features as micro-scale detection layer branches; the wheat seedling directional features are classified by angle, and the periodicity of the angle is processed.

[0010] S4. Obtain the image to be identified, and use the wheat seedling identification model constructed in S3 to identify the wheat seedlings; combine the rotation non-maximum suppression algorithm to eliminate repeated detection frames, and output and save the identified image.

[0011] The S1 data collection step is specifically as follows: the drone flies at an altitude of 10 meters, shoots wheat seedling images vertically downward from 10:00 to 14:00, and obtains wheat seedling image data in the seedling stage.

[0012] The S2 data processing steps are specifically as follows:

[0013] S2-1, Image flip

[0014] S2-2. Image rotation

[0015] S2-3, Image brightness enhancement

[0016] S2-4, image brightness reduction

[0017] S2-5. Use the labeling tool Ro-labelimg to label all the wheat seedlings in the drone image, obtain the individual position information and angle information of the wheat seedlings, and generate the wheat seedling annotation file

[0018] S2-6. Combine the wheat seedling drone images and the corresponding annotation files, and randomly generate training and test sets in a ratio of 7:3

[0019] The steps for constructing the S3 wheat seedling recognition model are as follows:

[0020] S3-1, based on the initialization of the YOLOv5 network, extract shallower spatial features from the multi-scale features and fuse them with the deep semantic features; add a micro-scale detection layer branch, use the feature map as a new micro-scale detection layer, and build a network detection layer with four scales to generate a multi-scale feature map of wheat seedlings;

[0021] S3-2. Combining the wheat seedling multi-scale feature map and the annotated wheat seedling angle information, classify each wheat seedling angle, 1° is counted as 1 category, and a total of 360 categories; the multi-scale feature map generates a wheat seedling prediction frame with center point position, length, width and angle information;

[0022] S3-3, constructing the network loss function l based on the wheat seedling prediction box generated by the multi-scale feature map and the manually marked wheat seedling bounding box;

[0023] S3-4, setting the hyperparameters of network training, and performing training until the loss function l converges, obtaining a wheat seedling recognition model, and generating a wheat seedling prediction frame;

[0024] S3-5. Based on the rotation non-maximum suppression algorithm, the overlap between the wheat seedling prediction box and the wheat seedling boundary box is calculated, and the repeated wheat seedling detection boxes are removed to obtain the final recognition result.

[0025] In S3-1, shallower spatial features are extracted from the multi-scale features and fused with deep semantic features to generate a new feature map whose size is 1 / 4 of the input image. This feature map is used as a new micro-scale detection layer to build a detection network structure with a wider scale coverage and more detailed, which is suitable for small-sized wheat seedlings.

[0026] In S3-3, the positioning loss e is used d , classification loss e s and confidence loss e i Describe the error l of the network:

[0027] l=e d +e s +e i

[0028] In the formula, the angle feature of the wheat seedling is identified as a classification task, so the classification loss e s Added the category loss of wheat seedling angle and calculated the difference between the wheat seedling prediction box and the wheat seedling bounding box

[0029]

[0030] Among them, S 2 is the feature map size output by the detection layer, B is the number of anchor boxes, and θ is the angle category. ij Indicates whether the jth anchor in the i-th grid can detect an object (the detected value is 1, and the undetected value is 0). i (c) represents the probability that the object is a wheat ear, P i (θ) represents the probability that the object has an angle of θ.

[0031] At the same time, the complete intersection-union ratio is used to calculate the positioning loss e of the wheat seedling prediction box and the wheat seedling bounding box d :

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] Among them, ar represents the prediction box and tr represents the wheat seedling bounding box.b ,O gt ) represents the Euclidean distance between the center points of the predicted box and the bounding box, and d represents the diagonal distance of the smallest rectangle that can enclose the two boxes. gt and h gt Represents the length and width of the bounding box, w b and h b Represents the length and width of the prediction box.

[0038] In S3-4, the drone image is resampled to a resolution of 500×500 and input into the wheat seedling recognition model for training to generate a wheat seedling prediction frame; the initial learning rate is set to 0.005, and the batch size is set to 16. The stochastic gradient descent SGD method is used to optimize the learning rate during the training process, the weight decay value is set to 1e-4, and the momentum is set to 0.9.

[0039] In S3-5, the overlapping area between the wheat seedling prediction box and the corresponding wheat seedling boundary box is divided into multiple triangles with the same vertices; the area of ​​the overlapping area is obtained by counting the area of ​​each triangle and adding them up; the ratio s of the area of ​​the overlapping area to the sum of the areas of the two prediction boxes is calculated, the wheat seedling prediction box with the largest s is retained, and the remaining wheat seedling prediction boxes are discarded to obtain the final recognition result.

[0040] Preferably, in S4, the specific steps of generating the wheat seedling identification result include:

[0041] S4-1. Using the constructed wheat seedling recognition model to recognize wheat seedlings; based on the rotation non-maximum suppression algorithm, calculating the overlap between the wheat seedling prediction box and the wheat seedling boundary box, removing the repeated wheat seedling detection boxes, and obtaining the final recognition result;

[0042] S4-2. Store the category, confidence level, spatial position and other information corresponding to each wheat seedling prediction box in the form of text and pictures as the recognition result of the model.

[0043] Specifically, in S4-1, a rotating non-maximum suppression algorithm is proposed to eliminate duplicate wheat seedling prediction frames; the overlapping area between the wheat seedling prediction frame and the corresponding wheat seedling boundary frame is divided into multiple triangles with the same vertices; the area of ​​the overlapping area is obtained by counting the area of ​​each triangle and adding them up; the ratio s of the area of ​​the overlapping area to the sum of the areas of the two prediction frames is calculated, the wheat seedling prediction frame with the largest s is retained, and the remaining wheat seedling prediction frames are eliminated to obtain the final recognition result.

[0044] Beneficial effects of the present invention

[0045] Existing wheat seedling recognition schemes usually use ground equipment for shooting, which has low recognition difficulty, high cost and narrow shooting range, and is difficult to apply to large-scale farmland. At the same time, these schemes all express the detected objects in the form of horizontal detection frames, and the detection effect is not good when the wheat seedlings are densely distributed. Since the wheat seedlings in the wheat drone image have directional information, the horizontal detection frame cannot describe the precise directional information of the wheat seedlings, and cannot accurately express the morphology and directional characteristics of the wheat seedlings. The method of identifying wheat seedlings based on the horizontal detection frame does not conform to the actual situation of wheat seedlings in natural scenes. At the same time, the field environment obtained by the wheat drone image is relatively complex, and the horizontal detection frame contains too much invalid background information, which affects the recognition accuracy of the wheat seedlings. The present invention introduces the angle information of the wheat seedlings and adds a new micro-scale detection layer to identify the spatial position and angle of the wheat seedlings in the form of a directional detection frame, so as to achieve accurate recognition of the wheat seedlings in the drone image and solve the technical problem of wheat seedling recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Technical block diagram of this application

[0047] Figure 2 Schematic diagram of image processing example

[0048] Figure 3 Schematic diagram of the wheat seedling recognition model structure

[0049] Figure 4 Schematic diagram for calculating the overlapping area of ​​two wheat seedling prediction boxes DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto:

[0051] The present invention constructs a method for identifying wheat seedlings from drone images based on deep learning. The technical solution is as follows: Figure 1 The solution includes data acquisition, data processing, model construction, model training and wheat seedling detection. The specific steps are as follows:

[0052] S1. Data acquisition

[0053] S11. In the embodiment, DJI is used M210 drone equipped The ZENMUSE X4S camera was used to capture drone images of wheat seedlings at an altitude of 10 meters between 10:00 and 14:00.

[0054] S2. Data processing

[0055] S21. The present invention enhances the image data set by image flipping, image rotation, image brightness enhancement, image brightness reduction, etc. Figure 2 As shown;

[0056] S22, using the labeling tool Ro-labelimg to label all the wheat seedlings in the drone image, obtain the individual position information and angle information of the wheat seedlings, and generate a wheat seedling annotation file;

[0057] S23. Combine the wheat seedling UAV images and the corresponding annotation files, and randomly generate training sets and test sets in a ratio of 7:3.

[0058] S3. Construction and training of wheat seedling recognition model

[0059] S31, build the initial YOLOv5 network;

[0060] S32. The present invention generates a feature map that is beneficial for detecting small-sized wheat seedlings by adding a micro-scale detection layer branch, thereby obtaining a richer fine-grained feature map, such as Figure 3 As shown. A shallower spatial feature is extracted from the multi-scale feature and fused with the deep semantic feature to generate a new feature map, the size of which is 1 / 4 of the input image. This feature map is used as a new micro-scale detection layer to build a detection network structure with a wider scale coverage and more detailed detection, which is suitable for small-sized wheat seedlings;

[0061] S33, combining the wheat seedling multi-scale feature map and the annotated wheat seedling angle information, classifying each wheat seedling angle based on the circular smooth label, 1° is counted as 1 category, and a total of 360 categories; specifically, the present invention processes the distance and periodicity between wheat seedling angles based on the circular smooth label. The expression of the circular smooth label is as follows:

[0062]

[0063] Among them, θ is the angle of the wheat seedling orientation frame, g(x) is the Gaussian function, and r is the window radius of the Gaussian function. Combining the Gaussian function and setting a periodic label encoding can measure the angular distance between the wheat seedling detection frame and the wheat seedling boundary frame, thereby solving the angular periodicity problem.

[0064] S34, the present invention uses positioning loss e d , classification loss e s and confidence loss e i Describe the error l of the network:

[0065] l=e d +e s +e i

[0066] In the formula, the angle feature of wheat seedlings is identified as a classification task, so the classification loss e s Added the category loss of wheat seedling angle and calculated the difference between the wheat seedling prediction box and the wheat seedling bounding box

[0067]

[0068] Among them, S 2 is the feature map size output by the detection layer, B is the number of anchor boxes, and θ is the angle category. ij Indicates whether the jth anchor in the i-th grid can detect an object (the detected value is 1, and the undetected value is 0). i (c) represents the probability that the object is a wheat ear, P i (θ) represents the probability that the object has an angle of θ.

[0069] At the same time, the complete intersection-union ratio is used to calculate the positioning loss e of the wheat seedling prediction box and the wheat seedling bounding box d :

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] Among them, ar represents the prediction box and tr represents the wheat seedling bounding box. b ,O gt ) represents the Euclidean distance between the center points of the predicted box and the bounding box, and d represents the diagonal distance of the smallest rectangle that can enclose the two boxes. gt and h gt Represents the length and width of the bounding box, w b and h b Represents the length and width of the prediction box.

[0076] S35. The present invention resamples the drone image to a resolution of 500×500, inputs the wheat seedling recognition model for training, and generates a wheat seedling prediction frame; the initial learning rate (Learning rate) is set to 0.005, and the batch size (Batchsize) is set to 16. The stochastic gradient descent SGD method is used to optimize the learning rate during the training process, the weight decay value is set to 1e-4, and the momentum is set to 0.9.

[0077] S4. Eliminate the wheat seedling prediction frame and obtain the final recognition result

[0078] S41, after the model training is completed, the wheat seedling drone image in the test set is input into the model to obtain the wheat seedling detection frame; the present invention proposes a rotation non-maximum suppression algorithm to remove repeated wheat seedling prediction frames, and the specific steps are as follows:

[0079] The overlapping area between the wheat seedling prediction box and the corresponding wheat seedling boundary box is divided into multiple triangles with the same vertices; the area of ​​the overlapping area is obtained by counting and adding the areas of each triangle, as shown in Figure 4 As shown;

[0080] ii. Calculate the ratio s of the area of ​​the overlapping region to the sum of the areas of the two prediction frames, retain the wheat seedling prediction frame with the largest s, and discard the remaining wheat seedling prediction frames;

[0081] S42. Store the category, confidence level, spatial position and other information corresponding to each wheat seedling prediction box in the form of text and pictures as the recognition result of the model.

[0082] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for identifying wheat seedlings from drone images based on deep learning, characterized in that It includes the following steps: S1, data collection, obtaining visible light images of wheat seedlings through drones; S2, data processing, enhancing data images, marking wheat seedlings, and obtaining training sets and test sets; S3, model construction, constructing a wheat seedling recognition model based on a drone platform; the detection network structure of the wheat seedling recognition model extracts shallow spatial features from multi-scale features, and fuses the shallow spatial features with deep semantic features as a micro-scale detection layer branch; The angle classification of wheat seedlings’ directional features is performed based on circular smooth labels, and the periodicity of the angle is processed. S4, recognition result generation, using the wheat seedling recognition model constructed in S3 to recognize wheat seedlings; combining the rotation non-maximum suppression algorithm to remove duplicate detection frames; outputting and saving the position, angle and confidence information of the wheat seedling prediction frame.

2. The method according to claim 1, characterized in that In S1, the drone acquired visible light images of wheat seedlings at a resolution of 4864×3078 pixels at an altitude of 15 meters in the wheat field.

3. The method according to claim 1, characterized in that In S2, the data enhancement methods include: image flipping, image rotation, image brightness enhancement, and image brightness reduction; using the marking tool Ro-labelimg to label all wheat seedlings in the drone image, obtain the individual position information and angle information of the wheat seedlings, and generate a wheat seedling annotation file; combining the wheat seedling drone image and the corresponding annotation file, and randomly generating a training set and a test set in a quantity ratio of 7:

3.

4. The method according to claim 1, characterized in that In S3, the specific steps of model construction include: S3-1. Build the YOLOv5 initial network, extract shallower spatial features from multi-scale features, and fuse them with deep semantic features; add a micro-scale detection layer branch, use the feature map as a new micro-scale detection layer, and build a network detection layer with four scales to generate a multi-scale feature map of wheat seedlings; S3-2. Combining the wheat seedling multi-scale feature map and the annotated wheat seedling angle information, classify each wheat seedling angle based on the circular smooth label, with 1° counted as 1 category, for a total of 360 categories; the multi-scale feature map generates a wheat seedling prediction frame with center point position, length, width and angle information; S3-3, constructing the network loss function l based on the wheat seedling prediction box generated by the multi-scale feature map and the manually marked wheat seedling bounding box; S3-4. Set the hyperparameters of network training and conduct training until the loss function l converges, obtain the wheat seedling recognition model, and generate a wheat seedling prediction box.

5. The method according to claim 4, characterized in that In S3-1, the initial YOLOv5 includes large-scale, medium-scale and small-scale detection layers. The output feature map sizes are 1 / 32, 1 / 16 and 1 / 8 of the input image size, respectively, responsible for large-size targets, medium-size targets and small-size targets; shallower spatial features are extracted from multi-scale features and fused with deep semantic features to generate a new feature map with a size of 1 / 4 of the input image; this feature map is used as a new micro-scale detection layer.

6. The method according to claim 4, characterized in that In S3-2, the distance and periodicity between the angles of the wheat seedlings are processed based on the circular smooth label. The expression of the circular smooth label is as follows: Among them, θ is the angle of the wheat seedling orientation box, g(x) is the Gaussian function, and r is the window radius of the Gaussian function. Combining the Gaussian function and setting a periodic label encoding can measure the angular distance between the wheat seedling detection box and the wheat seedling boundary box, thereby solving the angular periodicity problem.

7. The method according to claim 4, characterized in that In S3-3, the positioning loss e is used d , classification loss e s and confidence loss e i Describe the error l of the network: l=e d +e s +e i In the formula, the angle feature of wheat seedlings is identified as a classification task, so the classification loss e s Add the category loss of wheat seedling angle in , and calculate the difference between the wheat seedling prediction box and the wheat seedling bounding box e s : Among them, S 2 is the feature map size output by the detection layer, B is the number of anchor boxes, and θ is the angle category; I ij Indicates whether the jth anchor in the i-th grid can detect an object: the detected value is 1, and the undetected value is 0; P i (c) represents the probability that the object is a wheat ear, P i (θ) represents the probability that the object angle is θ; P i (c) represents the probability that the object is wheat seedling, P i (θ) represents the probability that the object’s angle is θ; c∈class Indicates the total number of object categories; The complete intersection-union ratio is used to calculate the positioning loss e of the wheat seedling prediction box and the wheat seedling bounding box. d : Among them, area(·) represents the area of ​​the region, ar represents the prediction box, and tr represents the wheat seedling bounding box; l(O b ,O gt ) represents the Euclidean distance between the center points of the prediction box and the bounding box, and d represents the diagonal distance of the smallest rectangle that can enclose the two boxes; w gt and h gt Represents the length and width of the bounding box, w b and h b Represents the length and width of the predicted box; IoU is the intersection over union ratio, CIoU is the complete intersection over union ratio, υ and α represent the coefficients calculated based on the intersection over union ratio IoU and the length and width of the box; e i is the confidence loss, which is defined as: Among them, λ obj and λ noobj are the weights of the confidence errors, is the confidence of the prediction box, C i is the confidence level of the true target; It means that if there is no real target in the j-th prediction box in the i-th grid of the training image, it is 1, otherwise it is 0.

8. The method according to claim 4, characterized in that In S3-4, the drone image is resampled to a resolution of 500×500 and input into the wheat seedling recognition model for training to generate the wheat seedling prediction box; the initial learning rate is set to 0.005, and the batch size is set to 16; the stochastic gradient descent SGD method is used to optimize the learning rate during training, the weight decay value is set to 1e-4, and the momentum is set to 0.

9.

9. The method according to claim 1, characterized in that In S4, the specific steps of generating the wheat seedling recognition result include: S4-1. Using the constructed wheat seedling recognition model to recognize wheat seedlings; based on the rotation non-maximum suppression algorithm, calculating the overlap between the wheat seedling prediction box and the wheat seedling boundary box, removing the repeated wheat seedling detection boxes, and obtaining the final recognition result; S4-2. Store the category, confidence level, spatial position and other information corresponding to each wheat seedling prediction box in the form of text and pictures as the recognition result of the model.

10. The method according to claim 9, characterized in that In S4-1, a rotating non-maximum suppression algorithm is proposed to eliminate duplicate wheat seedling prediction frames; the overlapping area between the wheat seedling prediction frame and the corresponding wheat seedling boundary frame is divided into multiple triangles with the same vertices; the area of ​​the overlapping area is obtained by counting the area of ​​each triangle and adding them up; the ratio s of the area of ​​the overlapping area to the sum of the areas of the two prediction frames is calculated, the wheat seedling prediction frame with the largest s is retained, and the remaining wheat seedling prediction frames are eliminated to obtain the final recognition result.

Images