High-precision detection method for field crop seedlings based on unmanned aerial vehicle and deep learning
By acquiring and stitching images using drones, and combining deep learning models with Intersection Area Ratio (IoA) screening, high-precision detection of field crop seedlings was achieved, solving the problems of detection accuracy and efficiency, and improving the level of intelligent agricultural production.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG AGRI UNIV
- Filing Date
- 2024-04-16
- Publication Date
- 2026-06-23
AI Technical Summary
Existing technologies struggle to achieve rapid and accurate detection of field crop seedlings, especially when dealing with overlapping targets, and they cannot handle image inputs of arbitrary sizes.
High-resolution images were acquired using drones and stitched together. After being cropped into smaller images, they were then detected using a deep learning model. The intersection area ratio (IoA) was used to filter out duplicate bounding boxes and extract seedling information.
It improves detection accuracy and efficiency, enhances the accuracy of crop seedling detection and coverage calculation, adapts to image input of any size, and reduces the labor intensity and human factor impact of traditional methods.
Smart Images
Figure CN118196669B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of smart agricultural information technology applications, artificial intelligence and deep learning, agricultural automation, and agricultural bioinformatics, specifically to a high-precision detection method for field crop seedlings based on unmanned aerial vehicles and deep learning. Background Technology
[0002] In modern agricultural production, accurate detection of crop seedlings is crucial for crop management. Traditional methods for seedling detection mainly rely on manual visual inspection and statistical sampling. These methods are labor-intensive, inefficient, and heavily influenced by human factors, making them unsuitable for large-scale agricultural production. With the development of remote sensing and drone technologies, and the ability of deep learning models to automatically learn and extract image features, remote sensing images of crop growth acquired by drones equipped with high-definition cameras, combined with deep learning models, have significantly improved the accuracy and efficiency of seedling detection, enabling rapid and accurate assessment of seedling conditions.
[0003] However, extracting accurate information about crop seedlings from high-resolution remote sensing images taken by drones still faces a series of challenges. First, drone images typically cover a wide area and are large enough to be directly used for deep learning detection. Remote sensing images are also affected by factors such as shooting angle, lighting conditions, and crop occlusion, making the detection and identification of crop seedlings more complex. Furthermore, the appearance of crop seedlings varies significantly at different growth stages, further increasing the difficulty of detection.
[0004] Therefore, researching a high-precision detection method for field crop seedlings that combines UAV remote sensing technology and deep learning algorithms can not only improve the level of intelligent agricultural production, but also promote the development of precision agriculture technology, thereby improving agricultural production efficiency and meeting the growing global food demand. Summary of the Invention
[0005] (a) Technical problems to be solved
[0006] This invention aims to solve the technical challenge of rapid and accurate detection of field crop seedlings, particularly improving detection accuracy when dealing with overlapping targets and adapting to image inputs of arbitrary sizes. This invention provides a high-precision detection method for field crop seedlings based on unmanned aerial vehicles (UAVs) and deep learning, enabling the detection and seedling condition analysis of field crop seedlings.
[0007] (II) Technical Solution
[0008] To address the aforementioned issues, this invention proposes a high-precision detection method for field crop seedlings based on unmanned aerial vehicles (UAVs) and deep learning, as detailed below.
[0009] A high-precision detection method for field crop seedlings based on UAVs and deep learning includes the following steps:
[0010] Step 1: Use an image sensor mounted on a drone platform to acquire high-resolution images of field crops in the seedling stage;
[0011] Step 2: Stitch together the high-definition images captured by the drone to obtain an orthophoto DOM with geographic coordinates.
[0012] Step 3: Crop the obtained DOM image with a specific cropping overlap to output a smaller image and record the geographic coordinates of the cropping location; the size of the smaller image meets the input requirements of the deep learning model input layer for image size and meets the DORI object detection standard;
[0013] Step 4: Use a deep learning model to detect the small-sized image, identify the area where the crop seedlings are located, and represent the outline of each crop seedling with a bounding rectangle, outputting the detection box and the confidence score of the detection box;
[0014] Step 5: Based on the cropping position recorded in Step 3, use the IoA method to filter the detection boxes output in Step 4, remove duplicate detection boxes, and map the final detection boxes onto the DOM in Step 2 to obtain the final detection box result.
[0015] Step 6: Based on the detection box results obtained in Step 5, extract seedling information from the detection box.
[0016] Preferably, in step 1, the high-definition images of field crops during the seedling stage are taken vertically from the ground by a drone, with a flight overlap of not less than 60% in the lateral direction and 70% in the heading direction; the images are in 8-bit three-channel JPG or PNG format.
[0017] Preferably, in step 3, when cropping the obtained DOM image with a specific cropping overlap, the cropping overlap satisfies the following formula:
[0018] Cropping overlap = Pixel size of crop seedling in the cropping direction / Pixel size of cropped image in the cropping direction.
[0019] Preferably, in step 4, the deep learning model includes a deep convolutional network as a feature extractor, combined with a feature pyramid network (FPN) and an anchor box mechanism to locate and identify seedlings in the field; when the deep learning model is used for detection, non-maximum suppression (NMS) is not enabled.
[0020] Preferably, the deep learning model is one of Yolov5, Yolov8, and Faster R-CNN.
[0021] Preferably, in step 5, the specific steps for removing duplicate detection boxes using the IoA method are as follows:
[0022] Step S51: Sort all detection boxes in descending order of confidence, and use the detection box with the highest confidence as the baseline detection box;
[0023] Step S52: Compare all other detection boxes with the benchmark detection box. If the IoA value of a detection box with the benchmark detection box exceeds a preset threshold, then delete the detection box. The IoA value is calculated as follows:
[0024] IoA = Area of intersecting regions / Area of the detection box itself
[0025] The intersection area here refers to the area of the intersection between this detection frame and the reference detection frame;
[0026] Step S53: Move the baseline detection box from step S52 to the output queue, and then select the detection box with the highest confidence among the remaining detection boxes as the new baseline detection box.
[0027] Step S54: Repeat steps S52-S53 until all detection boxes have been traversed;
[0028] Step S55: Use the detection boxes in the output queue as the final detection box results.
[0029] Preferably, in step 5, each detection box in the final detection box result is recorded according to geographical coordinates, and the recording method is (X, Y, W, H), with the x-coordinate of the center point X, the y-coordinate of the center point Y, the width W, and the height H.
[0030] Preferably, in step 6, the extracted seedling information includes the size, density, plant spacing, row spacing, and coverage of the seedlings;
[0031] The size of the seedlings is determined based on the width W and height H;
[0032] The seedling density is calculated using the K-means algorithm, which calculates the average distance between each seedling and its K nearest seedlings, where the default value for K is 5.
[0033] Plant spacing is calculated by clustering seedlings according to the X coordinate and then calculating the interval between seedlings in the Y direction within each group;
[0034] The row spacing is determined by clustering the seedling coordinates according to the Y coordinate, and then calculating the interval of each cluster center point in the X direction;
[0035] Coverage is determined by using the bounding rectangle of the seedling as a mask and calculating the area of the green pixels inside the mask.
[0036] (III) Beneficial Effects
[0037] Compared with the prior art, the present invention has significant positive technical effects, which are specifically manifested in the following aspects.
[0038] (1) Traditional deep learning models can only process images of a specific size due to processor performance and memory limitations, while UAV images often exceed this size limit. Existing solutions involve resampling the image, but this leads to the loss of target information and the introduction of errors. The cropping, detection, and stitching process proposed in this invention overcomes the image size limitation, enabling images of any size to be effectively detected through this process. Furthermore, an overlapping cropping method is used during the cropping process, and the stitched result is also processed by the IoA method, thereby avoiding errors caused by stitching.
[0039] (2) In existing deep learning models, Non-maximum Suppression (NMS) is a commonly used method in the post-processing stage. It evaluates the degree of overlap between targets by calculating the Intersection over Union (IoU), thereby avoiding errors caused by duplicate detection and noise. However, in agricultural scenarios, due to the high sowing density of crop seedlings and mutual occlusion caused by their growth patterns, traditional IoU-based NMS methods often fail to work effectively. This invention introduces Intersection Area Ratio (IoA) to more accurately evaluate the overlap relationship between targets. After switching to IoA, the counting accuracy increased from 90.8% to 95.3%, and the mAP of the reaction localization accuracy also increased from 86.7% to 87.1%. In addition, the computational complexity of IoA is the same as that of the conventional NMS method, so introducing IoA will hardly increase the additional computation time.
[0040] (3) Currently, image-based crop condition analysis methods lack precise measurement standards. This invention utilizes digital orthophoto images (DOM) with geographic coordinate systems acquired by UAVs to ensure that the calculated crop condition information reflects the true length parameters, thereby improving the accuracy of the analysis.
[0041] (4) Seedling cover is an important indicator for visually assessing seedling growth. However, the presence of weeds in the field makes it difficult to accurately calculate the cover. This invention uses a detection frame as a mask to exclude external weeds from the cover calculation, thereby enabling accurate measurement of the true cover during the seedling stage. Attached Figure Description
[0042] Figure 1 The method flowchart of the present invention.
[0043] Figure 2 The flowchart of the IoA method of the present invention.
[0044] Figure 3 A comparison chart of the effects of the IoA method of this invention and the conventional NMS method.
[0045] Figure 4 A schematic diagram of seedling condition parameters in an application example of this invention.
[0046] Figure 5 A schematic diagram illustrating the coverage of application examples of this invention. Detailed Implementation
[0047] To address its technical problems, this invention provides a high-precision detection method for field crop seedlings based on unmanned aerial vehicles (UAVs) and deep learning. The flowchart of this method is shown below. Figure 1 As shown.
[0048] A high-precision detection method for field crop seedlings based on UAVs and deep learning, characterized by the following steps:
[0049] Step 1: Use an image sensor mounted on a drone platform to acquire high-resolution images of field crops in the seedling stage;
[0050] Step 2: Stitch together the high-definition images captured by the drone to obtain an orthophoto DOM with geographic coordinates.
[0051] Step 3: Crop the obtained DOM image with a specific cropping overlap to output a smaller image and record the geographic coordinates of the cropping location; the size of the smaller image meets the input requirements of the deep learning model input layer for image size and meets the DORI object detection standard;
[0052] Step 4: Use a deep learning model to detect the small-sized image, identify the area where the crop seedlings are located, and represent the outline of each crop seedling with a bounding rectangle, outputting the detection box and the confidence score of the detection box;
[0053] Step 5: Based on the cropping position recorded in Step 3, use the IoA method to filter the detection boxes output in Step 4, remove duplicate detection boxes, and map the final detection boxes onto the DOM in Step 2 to obtain the final detection box result.
[0054] Step 6: Based on the detection box results obtained in Step 5, extract seedling information from the detection box.
[0055] In step 1, the crop seedlings must be clearly visible in the image when the drone collects data.
[0056] Step 3 involves processing the high-resolution images captured by the drone. These images are represented using a projected coordinate system with distances in meters. To facilitate model detection input, the DOM images are cropped into multiple smaller images of 1280×1280 size, maintaining a certain degree of overlap during the cropping process to ensure that the seedlings at the boundaries are fully displayed in at least one cropped image.
[0057] Step 4: Use a deep learning model to detect the multiple images cropped in Step 3. The detection result for each image includes five parameters: the location, size, and confidence score of each target in the image. The confidence score indicates the similarity between the detection box and the target; a higher value indicates a more accurate detection. The detection results for each image are stored in a TXT text file.
[0058] Step 5 first reads the detection results from Step 4 and uses the IoA method to deduplicate these results. This step not only removes erroneous results within each small-sized image but also eliminates duplicate results at image boundaries caused by multiple detections. The final detection results are mapped onto the DOM based on the geographic coordinates recorded in Step 3. The target location is recorded using the projected coordinate system within the geographic coordinates, and the target size information is converted to actual length units. The confidence score is no longer retained. The final result is all the detection boxes on the DOM after IoA processing.
[0059] In step 5, for details on removing duplicate detection boxes using the IoA method, please refer to [link / reference]. Figure 2 The specific steps are as follows:
[0060] Step S51: Sort all detection boxes in descending order of confidence, and use the detection box with the highest confidence as the baseline detection box;
[0061] Step S52: Compare all other detection boxes with the benchmark detection box. If the IoA value of a detection box with the benchmark detection box exceeds a preset threshold, then delete the detection box. The IoA value is calculated as follows:
[0062] IoA = Area of intersecting regions / Area of the detection box itself
[0063] The intersection area here refers to the area of the intersection between this detection frame and the reference detection frame;
[0064] Step S53: Move the baseline detection box from step S52 to the output queue, and then select the detection box with the highest confidence among the remaining detection boxes as the new baseline detection box.
[0065] Step S54: Repeat steps S52-S53 until all detection boxes have been traversed;
[0066] Step S55: Use the detection boxes in the output queue as the final detection box results.
[0067] The IoA method used in step 5 is an improvement on the NMS method used in conventional deep learning detection. The main difference between IoA and NMS is that IoA uses the area ratio (IoA) to determine whether targets are duplicates. For example... Figure 3 As shown, compared to the traditional NMS algorithm, IoA can not only produce results similar to NMS under normal circumstances, but also better handle some complex scenarios, such as target ghosting or the clipping of boundary targets.
[0068] In step 6, the extracted seedling information includes seedling size, density, plant spacing, row spacing, and coverage. The extracted seedling information is... Figure 4 and Figure 5 The visualization was presented in the middle;
[0069] The size of the seedlings is statistically determined using the width W and height H of the detection frame. Figure 4 In this context, W and H are used to represent the two characters respectively.
[0070] The seedling density is calculated using the K-means algorithm, which calculates the average distance between the center point of the detection box of each seedling and the center points of the detection boxes of its K nearest seedlings, where the default value of K is 5.
[0071] Plant spacing is calculated by clustering seedlings according to their X-coordinate and then determining the average interval between seedlings in the Y-direction within each group. Figure 4 The spacing between the two seedlings is marked in the middle;
[0072] Row spacing is determined by clustering the seedling coordinates according to the Y-coordinate, and then calculating the average interval of each cluster center point in the X-direction. Figure 4 The row spacing between the two middle columns of seedlings is marked in the middle;
[0073] Coverage is determined by using the bounding rectangle of the seedling as a mask and calculating the area of the green pixels inside the mask. Figure 5 The image shows the binarization results of the green area. The coverage is calculated by dividing the area of white pixels inside the box by the total area. White pixels outside the detection box are considered weeds and are excluded.
[0074] The specific examples described in this application are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the specific examples described, or substitute them using similar methods, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. A high-precision detection method for field crop seedlings based on UAVs and deep learning, characterized in that: Includes the following steps: Step 1: Use an image sensor mounted on a drone platform to acquire high-resolution images of field crops in the seedling stage; Step 2: Stitch together the high-definition images captured by the drone to obtain an orthophoto DOM with geographic coordinates. Step 3: Crop the obtained DOM image with a specific cropping overlap to output a smaller image and record the geographic coordinates of the cropping location; the size of the smaller image meets the input requirements of the deep learning model input layer for image size and meets the DORI object detection standard; Step 4: Use a deep learning model to detect the small-sized image, identify the area where the crop seedlings are located, and represent the outline of each crop seedling with a bounding rectangle, outputting the detection box and the confidence score of the detection box; Step 5: Based on the cropping position recorded in Step 3, use the IoA method to filter the detection boxes output in Step 4, remove duplicate detection boxes, and map the final detection boxes onto the DOM in Step 2 to obtain the final detection box results; Step 6: Based on the detection box results obtained in Step 5, extract seedling information from the detection boxes. In step 5, the specific steps for removing duplicate detection boxes using the IoA method are as follows: Step S51: Sort all detection boxes in descending order of confidence, and use the detection box with the highest confidence as the baseline detection box; Step S52: Compare all other detection boxes with the reference detection box. If the IoA value of a detection box with the reference detection box exceeds a preset threshold, then delete the detection box. The IoA value is calculated as follows: IoA = Area of intersection region / Area of detection box itself. Here, the intersection region refers to the area of the intersection region between this detection box and the reference detection box. Step S53: Move the baseline detection box from step S52 to the output queue, and then select the detection box with the highest confidence among the remaining detection boxes as the new baseline detection box. Step S54: Repeat steps S52-S53 until all detection boxes have been traversed; Step S55: Use the detection boxes in the output queue as the final detection box results.
2. The high-precision detection method for field crop seedlings based on UAVs and deep learning according to claim 1, characterized in that: In step 1, the high-resolution images of field crops during the seedling stage are taken vertically from the ground by a drone, with a flight overlap of no less than 60% in the lateral direction and 70% in the heading direction; the images are in 8-bit three-channel JPG or PNG format.
3. The high-precision detection method for field crop seedlings based on UAVs and deep learning according to claim 1, characterized in that: In step 3, when cropping the obtained DOM image with a specific cropping overlap, the cropping overlap satisfies the following formula: Cropping overlap = Pixel size of crop seedling in the cropping direction / Pixel size of cropped image in the cropping direction.
4. The high-precision detection method for field crop seedlings based on UAVs and deep learning according to claim 1, characterized in that: In step 4, the deep learning model includes a deep convolutional network as a feature extractor, combined with a feature pyramid network (FPN) and an anchor box mechanism to locate and identify seedlings in the field; when the deep learning model is used for detection, non-maximum suppression (NMS) is not enabled.
5. The high-precision detection method for field crop seedlings based on UAVs and deep learning according to claim 4, characterized in that: The deep learning model used is one of Yolov5, Yolov8, and Faster R-CNN.
6. The high-precision detection method for field crop seedlings based on UAVs and deep learning according to claim 1, characterized in that: In step 5, each detection box in the final detection box result is recorded according to geographical coordinates, and the recording method is (X, Y, W, H), with the x-coordinate of the center point X, the y-coordinate of the center point Y, the width W, and the height H.
7. The high-precision detection method for field crop seedlings based on UAVs and deep learning according to claim 6, characterized in that: In step 6, the extracted seedling information includes seedling size, density, plant spacing, row spacing, and coverage. Seedling size is determined based on width W and height H. Seedling density is calculated using the K-means algorithm to determine the average distance between each seedling and its K nearest seedlings, where K is set to a default value of 5. Plant spacing is calculated by clustering seedlings according to their X-coordinates and then calculating the interval between seedlings in the Y-direction within each group. Row spacing is determined by clustering seedlings according to their Y-coordinates and then calculating the interval between the center points of each cluster in the X-direction. Coverage is obtained by using the bounding rectangle of the seedlings as a mask and calculating the area of the green pixels inside the mask.