A Deep Learning-Based Method for Recognizing Wheat Seedlings from UAV Images

Abstract

This invention discloses a deep learning-based method for identifying wheat seedlings in UAV images. The method outputs information such as the spatial location and angle of the wheat seedlings. The construction of the identification model includes: S1, data acquisition; S2, data processing; S3, model construction, which involves building and training a wheat seedling identification network to obtain a wheat seedling identification model; and S4, using the constructed wheat seedling identification model to identify wheat seedlings, outputting and saving information such as the position and angle of the predicted wheat seedling bounding boxes. The key features are: S3 adds a micro-scale detection layer branch and constructs a network detection layer with four scales; classifying each wheat seedling angle based on circular smooth labels to extract wheat seedling angle information; and S4 using a rotational non-maximum suppression algorithm to remove duplicate predicted wheat seedling bounding boxes. The proposed solution can accurately identify small-sized wheat seedlings in UAV images, is suitable for complex field scenes, and provides a technical reference for wheat seedling identification.

Description

Technical Field

[0001] This invention belongs to the field of smart agriculture, specifically a method for identifying wheat seedlings from drone images based on deep learning. Background Technology

[0002] Wheat is one of the world's major food crops. The number of wheat seedlings, as an important indicator of crop population size during the seedling stage, influences the ear grain structure to a certain extent, ultimately affecting wheat yield and quality. Therefore, the identification and counting of wheat seedlings is an important task and is of great significance for accurately estimating wheat yield.

[0003] There are three main types of existing methods for identifying wheat seedlings: The first, traditional wheat seedling identification, relies on manual surveys. However, manual sampling is inefficient and costly, especially when human resources are limited and the sampling area is large, making sampling errors prone to occur and affecting the final statistical results. The second, machine learning methods, rely on color, texture, and morphological features, using classifiers such as Bayesian, Support Vector Machines, and Random Forests to identify wheat seedlings. However, feature selection depends on expert prior knowledge, resulting in poor algorithm generalization and difficulty in adapting to changes in object size, translation, and rotation, making it unsuitable for densely distributed and heavily shaded wheat fields. The third, and the deep learning method used in this invention, does not rely on manually generated features based on statistical regularities or prior knowledge. Its automatically learned features better reflect the essence of the data.

[0004] Due to issues such as the small size of wheat seedlings, significant occlusion, and severe interference from soil background in drone images, there is currently a lack of drone-based methods for identifying wheat seedlings. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of small size of wheat seedlings, many obstructions and serious interference from soil background in drone images, and to propose a method for wheat seedling recognition in drone images based on deep learning.

[0006] To solve the above problems, the technical solution of the present invention is as follows:

[0007] S1. Data acquisition: acquiring visible light images of wheat seedlings using drones;

[0008] S2. Data processing: enhance data images, label individual wheat seedlings, and obtain training and test sets;

[0009] S3. Model Construction: Construct a wheat seedling recognition model based on an unmanned aerial vehicle (UAV) platform. The detection network structure of the wheat seedling recognition model extracts shallow spatial features from multi-scale features and fuses these shallow spatial features with deep semantic features as a micro-scale detection layer branch. The wheat seedling orientation features are classified by angle and the periodicity of the angle is processed.

[0010] S4. Obtain the image to be identified, and use the wheat seedling identification model constructed in S3 to identify the wheat seedlings; combine the rotation nonmaximum suppression algorithm to remove duplicate detection boxes, and output and save the identified image.

[0011] The S1 data acquisition step is as follows: the drone flies at a height of 10 meters and takes vertical downward images of wheat seedlings from 10:00 to 14:00 to obtain seedling image data.

[0012] The S2 data processing steps are as follows:

[0013] S2-1, Image Flipping

[0014] S2-2, Image Rotation

[0015] S2-3, Image Brightness Enhancement

[0016] S2-4, Image brightness reduced

[0017] S2-5. Use the labeling tool Ro-labelimg to label all wheat seedlings within the UAV imagery, obtaining individual seedling location and angle information, and generating a wheat seedling annotation file.

[0018] S2-6. Combine the wheat seedling UAV imagery and corresponding annotation files, and randomly generate training and test sets at a ratio of 7:3.

[0019] The specific steps for constructing the S3 wheat seedling recognition model are as follows:

[0020] S3-1, based on the initial YOLOv5 network, extracts shallower spatial features from multi-scale features and fuses them with deeper semantic features; adds a micro-scale detection layer branch, uses the feature map as a new micro-scale detection layer, and constructs a network detection layer with four scales to generate multi-scale feature maps of wheat seedlings.

[0021] S3-2. Combining the multi-scale feature map of wheat seedlings and the labeled angle information of wheat seedlings, classify each wheat seedling angle, with 1° counted as 1 class, for a total of 360 classes; generate wheat seedling prediction boxes with center point position, length and width information and angle information from the multi-scale feature map;

[0022] S3-3. Construct the network's loss function l based on the wheat seedling prediction boxes generated from the multi-scale feature maps and the manually labeled wheat seedling bounding boxes;

[0023] S3-4. Set the hyperparameters for network training and train until the loss function l converges to obtain the wheat seedling recognition model and generate wheat seedling prediction boxes.

[0024] S3-5. Based on the rotational nonmaximum suppression algorithm, calculate the overlap between the wheat seedling prediction box and the wheat seedling bounding box, remove duplicate wheat seedling detection boxes, and obtain the final recognition result.

[0025] In step S3-1, shallower spatial features are extracted from the multi-scale features and fused with deeper semantic features to generate a new feature map, which is 1 / 4 the size of the input image. This feature map is used as a new micro-scale detection layer to construct a detection network structure with a wider and more detailed scale coverage, suitable for small wheat seedlings.

[0026] In S3-3, the positioning loss e is used. d Classification loss e s and confidence loss e i The error l describing the network:

[0027] l = e d +e s +e i

[0028] In the formula, the angular features of the wheat seedlings are considered as the classification task, therefore, in the classification loss e s The class loss for wheat seedling angle is added, and the difference between the predicted wheat seedling box and the bounding box is calculated.

[0029]

[0030] Among them, S 2 Let B be the size of the feature map output by the detection layer, B be the number of anchor boxes, and θ be the angle category. ij This indicates whether the j-th anchor in the i-th grid can detect an object (a value of 1 indicates a value of 0 indicates a value of 0). i (c) represents the probability that the object is an ear of wheat, P i (θ) represents the probability that the object is at an angle of θ.

[0031] Meanwhile, the localization loss e of the wheat seedling prediction box and the wheat seedling bounding box is calculated using the perfect intersection-union ratio. d :

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] Where ar represents the predicted bounding box, and tr represents the wheat seedling bounding box. l(Ob O gt ) represents the Euclidean distance between the center points of the predicted bounding box and the bounding box, and d represents the diagonal distance of the smallest rectangle that can enclose the two boxes. gt and h gt w represents the length and width of the bounding box b and h b This represents the length and width of the prediction box.

[0038] In steps S3-4, the UAV imagery is resampled to a 500×500 resolution and input into the wheat seedling recognition model for training, generating predicted wheat seedling bounding boxes. The initial learning rate is set to 0.005, and the batch size is set to 16. The stochastic gradient descent (SGD) method is used to optimize the learning rate during training, with a weight decay value set to 1e-4 and a momentum set to 0.9.

[0039] In step S3-5, the overlapping area between the wheat seedling prediction box and the corresponding wheat seedling bounding box is divided into multiple triangles with the same vertex; the area of ​​the overlapping area is obtained by counting the area of ​​each triangle and adding them together; the ratio s of the area of ​​the overlapping area to the sum of the areas of the two prediction boxes is calculated, the wheat seedling prediction box with the largest s is retained, and the remaining wheat seedling prediction boxes are removed to obtain the final recognition result.

[0040] Preferably, in S4, the specific steps for generating the wheat seedling identification result include:

[0041] S4-1. Wheat seedling recognition is performed using the constructed wheat seedling recognition model. Based on the rotational nonmaximum suppression algorithm, the overlap between the wheat seedling prediction box and the wheat seedling bounding box is calculated, and duplicate wheat seedling detection boxes are removed to obtain the final recognition result.

[0042] S4-2. Store the category, confidence level, and spatial location of each wheat seedling prediction box in the form of text and images as the model's recognition result.

[0043] Specifically, in S4-1, a rotational nonmaximum suppression algorithm is proposed to remove duplicate wheat seedling prediction boxes; the overlapping area between the wheat seedling prediction box and the corresponding wheat seedling bounding box is divided into multiple triangles with the same vertex; the area of ​​the overlapping area is obtained by counting the area of ​​each triangle and adding them together; the ratio s of the area of ​​the overlapping area to the sum of the areas of the two prediction boxes is calculated, the wheat seedling prediction box with the largest s is retained, the remaining wheat seedling prediction boxes are removed, and the final recognition result is obtained.

[0044] Beneficial effects of the present invention

[0045] Existing wheat seedling identification schemes typically use ground-based equipment for imaging, which offers relatively low identification difficulty but is costly and has a narrow imaging range, making them unsuitable for large-scale farmland applications. Furthermore, these schemes all represent detected objects using horizontal detection boxes, which are ineffective when wheat seedlings are densely distributed. Since wheat seedlings in drone images possess directional information, horizontal detection boxes cannot accurately describe their precise orientation, failing to accurately represent their morphological and directional characteristics. Methods based on horizontal detection boxes do not reflect the true state of wheat seedlings in natural scenes. Additionally, the field environment captured by drone images is complex, and horizontal detection boxes contain excessive irrelevant background information, affecting the accuracy of seedling identification. This invention addresses these challenges by introducing angular information from wheat seedlings and adding a micro-scale detection layer to identify the spatial location and angle of wheat seedlings using directional detection boxes, achieving accurate identification of wheat seedlings in drone images. Attached Figure Description

[0046] Figure 1 This is a technical block diagram of the present application.

[0047] Figure 2 Schematic diagram of an image processing example

[0048] Figure 3 Schematic diagram of the wheat seedling recognition model structure

[0049] Figure 4 A schematic diagram for calculating the area of ​​the overlapping region between two wheat seedling prediction frames. Detailed Implementation

[0050] The present invention will be further described below with reference to embodiments, but the scope of protection of the present invention is not limited thereto:

[0051] This invention constructs a deep learning-based method for identifying wheat seedlings in drone images, the technical solution of which is as follows: Figure 1 As shown. The solution includes data acquisition, data processing, model building, model training, and wheat seedling detection. The specific steps are as follows:

[0052] S1. Data Acquisition

[0053] S11. DJI is used in the embodiment. Matrice M210 UAV equipped with A ZENMUSE X4S camera was used to capture drone footage of wheat seedlings at a height of 10 meters between 10:00 and 14:00.

[0054] S2. Data Processing

[0055] S21. This invention enhances image datasets using methods such as image flipping, image rotation, image brightness enhancement, and image brightness reduction. Figure 2 As shown;

[0056] S22. Use the labeling tool Ro-labelimg to label all wheat seedlings in the UAV image, obtain the individual position and angle information of the wheat seedlings, and generate a wheat seedling annotation file;

[0057] S23. Combine the wheat seedling drone images and corresponding annotation files, and randomly generate training and test sets in a 7:3 ratio.

[0058] S3. Construction and Training of Wheat Seedling Recognition Model

[0059] S31. Construct the initial YOLOv5 network;

[0060] S32. This invention generates feature maps that are beneficial for detecting small-sized wheat seedlings by adding a new microscale detection layer branch, thus obtaining richer fine-grained data, such as... Figure 3 As shown, shallower spatial features are extracted from the multi-scale features and fused with deeper semantic features to generate a new feature map, which is 1 / 4 the size of the input image. This feature map is used as a new micro-scale detection layer to construct a detection network structure with a wider and more detailed scale coverage, suitable for small wheat seedlings;

[0061] S33. Combining the multi-scale feature map of wheat seedlings and the labeled angle information of wheat seedlings, each wheat seedling angle is classified based on a circular smooth label, with 1° counted as one category, for a total of 360 categories; specifically, this invention processes the distance and periodicity between wheat seedling angles based on circular smooth labels. The expression for the circular smooth label is as follows:

[0062]

[0063] Where θ is the angle of the wheat seedling orientation box, g(x) is the Gaussian function, and r is the window radius of the Gaussian function. By combining the Gaussian function and setting a periodic label encoding, the angular distance between the wheat seedling detection box and the wheat seedling bounding box can be measured, thereby solving the problem of angular periodicity.

[0064] S34, This invention uses positioning loss e d Classification loss e s and confidence loss e i The error l describing the network:

[0065] l = e d +e s +e i

[0066] In the formula, the angular features of the wheat seedlings are considered as the classification task, therefore, in the classification loss e s The class loss for wheat seedling angle is added, and the difference between the predicted wheat seedling box and the bounding box is calculated.

[0067]

[0068] Among them, S 2 Let B be the size of the feature map output by the detection layer, B be the number of anchor boxes, and θ be the angle category. ij This indicates whether the j-th anchor in the i-th grid can detect an object (a value of 1 indicates a value of 0 indicates a value of 0). i (c) represents the probability that the object is an ear of wheat, P i (θ) represents the probability that the object is at an angle of θ.

[0069] Meanwhile, the localization loss e of the wheat seedling prediction box and the wheat seedling bounding box is calculated using the perfect intersection-union ratio. d :

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] Where ar represents the predicted bounding box, and tr represents the wheat seedling bounding box. l(O b O gt ) represents the Euclidean distance between the center points of the predicted bounding box and the bounding box, and d represents the diagonal distance of the smallest rectangle that can enclose the two boxes. gt and h gt w represents the length and width of the bounding box b and h b This represents the length and width of the prediction box.

[0076] S35. In this invention, UAV images are resampled to a resolution of 500×500 and input into the wheat seedling recognition model for training to generate predicted wheat seedling bounding boxes. The initial learning rate is set to 0.005, and the batch size is set to 16. The stochastic gradient descent (SGD) method is used to optimize the learning rate during training, with the weight decay value set to 1e-4 and the momentum set to 0.9.

[0077] S4. Remove the predicted wheat seedling boxes and obtain the final recognition result.

[0078] S41. After the model training is complete, input the wheat seedling drone images from the test set into the model to obtain wheat seedling detection boxes; the present invention proposes a rotational nonmaximum suppression algorithm to remove duplicate wheat seedling prediction boxes, the specific steps of which are as follows:

[0079] The overlapping region between the predicted wheat seedling bounding box and the corresponding wheat seedling bounding box is divided into multiple triangles with the same vertices; the area of ​​the overlapping region is obtained by calculating the area of ​​each triangle and adding them together, such as... Figure 4 As shown;

[0080] ii. Calculate the ratio s of the area of ​​the overlapping region to the sum of the areas of the two prediction boxes, retain the wheat seedling prediction box with the largest s, and remove the remaining wheat seedling prediction boxes.

[0081] S42. Store the information such as category, confidence level and spatial location corresponding to each wheat seedling prediction box in the form of text and images as the recognition result of the model.

[0082] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for identifying wheat seedlings in drone images based on deep learning, characterized in that... It includes the following steps: S1. Data acquisition: acquiring visible light images of wheat seedlings using drones; S2. Data processing: enhance data images, label individual wheat seedlings, and obtain training and test sets; S3. Model construction: Construct a wheat seedling recognition model based on a drone platform; The detection network structure of the wheat seedling recognition model extracts shallow spatial features from multi-scale features and fuses these shallow spatial features with deep semantic features as a micro-scale detection layer branch. Based on the circular smooth label, the directional features of wheat seedlings are classified by angle, and the periodicity of the angle is handled. The specific steps in model construction include: S3-1. Construct the initial YOLOv5 network, extract shallower spatial features from the multi-scale features, and fuse them with deeper semantic features; add a micro-scale detection layer branch, use the feature map as a new micro-scale detection layer, and construct a network detection layer with four scales to generate multi-scale feature maps of wheat seedlings. S3-2. Combining the multi-scale feature map of wheat seedlings and the labeled angle information of wheat seedlings, each wheat seedling angle is classified based on a circular smooth label, with 1° counted as 1 class, for a total of 360 classes; the multi-scale feature map generates wheat seedling prediction boxes with center point position, length and width information, and angle information; the distance and periodicity between wheat seedling angles are processed based on the circular smooth label, and the expression of the circular smooth label is as follows: Where θ is the angle of the wheat seedling orientation box, g(x) is the Gaussian function, and r is the window radius of the Gaussian function; by combining the Gaussian function and setting a periodic label code, the angular distance between the wheat seedling detection box and the wheat seedling bounding box can be measured, thereby solving the problem of angular periodicity; S3-3. Construct the network's loss function l based on the wheat seedling prediction boxes generated from the multi-scale feature maps and the manually labeled wheat seedling bounding boxes; use the localization loss e. d Classification loss e s and confidence loss e i The error l describing the network: l=e d +e s +e i In the formula, the angular features of the wheat seedlings are considered as the classification task, therefore, in the classification loss e s The class loss for wheat seedling angle is added, and the difference e between the predicted wheat seedling box and the bounding box is calculated. s : Among them, S 2 Where I is the size of the feature map output by the detection layer, B is the number of anchor boxes, and θ is the angle category; ij Indicates whether the j-th anchor in the i-th grid can detect an object: a value of 1 indicates a detected object, and a value of 0 indicates an undetected object; P i (c) represents the probability that the object is an ear of wheat, P i (θ) represents the angle of the object. The probability of P i (c) represents the probability that the object is a wheat seedling, P i (θ) represents the angle of the object. The probability of c ∈ class; c ∈ class represents the total number of object categories; The localization loss e of the wheat seedling prediction box and the wheat seedling bounding box is calculated using the perfect intersection-union ratio (MIRU). d : Where area(ar∩tr) represents the area of ​​the intersection of the predicted bounding box and the wheat seedling bounding box, and area(ar∪tr) represents the area of ​​the union of the predicted bounding box and the wheat seedling bounding box, where ar represents the predicted bounding box and tr represents the wheat seedling bounding box; l(O b O gt ) represents the Euclidean distance between the center points of the predicted bounding box and the bounding box, and d represents the diagonal distance of the smallest rectangle that can enclose the two boxes; w gt and h gt w represents the length and width of the bounding box b and h b Represents the length and width of the predicted bounding box; IoU is the Cross-Union Ratio, CIoU is the Perfect Cross-Union Ratio, and υ and α represent coefficients calculated based on the Cross-Union Ratio (IoU) and the length and width of the bounding box; e i The confidence loss is defined as: Where, λ obj and λ noobj These are the weights of the confidence error, respectively. C represents the confidence level of the predicted bounding box. i Confidence level for the true target; This indicates that if the j-th predicted bounding box in the i-th grid of the training image does not contain a real target, it is 1; otherwise, it is 0. S3-4. Set the hyperparameters for network training and train until the loss function l converges to obtain the wheat seedling recognition model and generate wheat seedling prediction boxes. S4. Recognition result generation: The wheat seedling recognition model constructed in S3 is used to identify wheat seedlings; combined with the rotational nonmaximum suppression algorithm, duplicate detection boxes are removed; the position, angle and confidence information of the predicted wheat seedling boxes are output and saved.

2. The method according to claim 1, characterized in that... In S1, the drone acquires visible light images of wheat seedlings at a resolution of 4864×3078 pixels from a height of 15 meters in the wheat field.

3. The method according to claim 1, characterized in that... In S2, the data augmentation methods include: image flipping, image rotation, image brightness enhancement, and image brightness reduction; using the labeling tool Ro-labelimg to label all wheat seedlings in the UAV imagery to obtain individual wheat seedling position information and angle information, and generating wheat seedling annotation files; combining the wheat seedling UAV imagery and the corresponding annotation files to randomly generate training and testing sets at a ratio of 7:

3.

4. The method according to claim 1, characterized in that... In S3-1, the initial YOLOv5 includes large-scale, medium-scale, and small-scale detection layers, with output feature map sizes of 1 / 32, 1 / 16, and 1 / 8 of the input image size, respectively, responsible for large-sized, medium-sized, and small-sized targets. Shallower spatial features are extracted from the multi-scale features and fused with deeper semantic features to generate a new feature map with a size of 1 / 4 of the input image. This feature map is then used as a new micro-scale detection layer.

5. The method according to claim 1, characterized in that... In S3-4, the UAV imagery is resampled to a resolution of 500×500 and input into the wheat seedling recognition model for training to generate wheat seedling prediction boxes. The initial learning rate is set to 0.005, and the batch size is set to 16. The learning rate during training is optimized using the stochastic gradient descent (SGD) method, with the weight decay value set to 1e-4 and the momentum set to 0.

9.

6. The method according to claim 1, characterized in that... In S4, the specific steps for generating wheat seedling identification results include: S4-1. Wheat seedling recognition is performed using the constructed wheat seedling recognition model. Based on the rotational nonmaximum suppression algorithm, the overlap between the wheat seedling prediction box and the wheat seedling bounding box is calculated, and duplicate wheat seedling detection boxes are removed to obtain the final recognition result. S4-2. Store the category, confidence level, and spatial location of each wheat seedling prediction box in the form of text and images as the model's recognition result.

7. The method according to claim 6, characterized in that... In S4-1, a rotational nonmaximum suppression algorithm is proposed to remove duplicate wheat seedling prediction boxes. The overlapping area between the wheat seedling prediction box and the corresponding wheat seedling bounding box is divided into multiple triangles with the same vertex. The area of ​​the overlapping area is obtained by counting the area of ​​each triangle and adding them together. The ratio s of the area of ​​the overlapping area to the sum of the areas of the two prediction boxes is calculated. The wheat seedling prediction box with the largest s is retained, and the remaining wheat seedling prediction boxes are removed to obtain the final recognition result.

Images