A method and system for measuring crop area and estimating yield of gramineous crops

Abstract

The application discloses a crop area measurement and gramineous crop yield estimation method and system, which uses a single Prompt SAM model to interpret remote sensing images obtained by a UAV to obtain the crop area of a target region, uses a mapping algorithm of remote sensing image pixel coordinates to geographic coordinates, and maps the interpretation result to geographic coordinates in real time. A YOLO series model is used for target detection of the detection target shot by the UAV at a low altitude to obtain the number of wheat ears, and then the wheat ear planting density is obtained. The results of the two models and the average quality parameter of each plant set by an artificial are used to calculate the final required yield estimation quality. The method proposes a single prompt model to achieve the task target without training in view of the lack of standard data in the agricultural field, uses a multi-model multi-task processing mode to improve the single task precision, and uses a real-time pixel coordinate to geographic coordinate mapping algorithm to accelerate the result acquisition speed.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and yield estimation of gramineous crops, specifically a method and system for general crop area measurement and yield prediction of gramineous crops. Background Technology

[0002] Rapidly and accurately obtaining information on crop planting areas is crucial for grain yield estimation and planting structure adjustment, helping relevant departments to formulate timely agricultural policies and ensure food security. Wheat, rice, and corn, the three major staple grains, are all grasses and among the world's major food crops. Timely understanding of crop planting areas is of significant social importance for monitoring crop growth and yield estimation, as well as regional food security assessments. Unmanned aerial vehicle (UAV) remote sensing technology, characterized by its high detection speed, short data update cycle, and accurate data representation, is an effective means of quickly and accurately extracting crop planting areas. With the improvement in the precision of various UAV sensors, UAV positioning accuracy has reached the centimeter level, making real-time map data processing possible without the need for pre-stitching large aerial photographs and registration before identification. This greatly improves data processing and display speed.

[0003] This invention addresses the challenges of scarce standard UAV data and complex natural conditions in agricultural remote sensing. To improve the efficiency of field identification, a method based on multi-model hybrid interpretation of field area and yield is proposed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention aims to provide a method and system for measuring crop area and predicting the yield of gramineous crops. This method employs a hybrid approach, combining a large-scale AI transformer model with a small-scale traditional YOLO convolutional neural network model to handle different tasks. First, the large model detects the crop area. Then, fixed-point sampling is used to obtain the crop cover density of the identified planting area. Combined with the average mass parameter per plant set in the algorithm, the predicted crop area and yield are calculated. This method offers improvements in processing speed and accuracy compared to traditional satellite remote sensing. Furthermore, based on UAV remote sensing images, pixel-level classification information is mapped onto an orthophoto map, enabling plot segmentation during UAV flight.

[0005] This application achieves the above effects through the following technical solution: a method for measuring crop area and predicting yield of gramineous crops, the method comprising the following steps:

[0006] S1 first uses a drone to take aerial photos at a first set altitude to obtain multiple images of crop fields and at a second set altitude to take one or more close-up images of crops. The close-up images taken by the drone are crop sampling points within the target area.

[0007] S2 uses a single manual prompt to identify the crops to be recognized and inputs them into the improved SAM large model, and then performs batch recognition.

[0008] S3 uses a trained YOLO model to identify the number of crops within a range from close-up images of crops;

[0009] S4 uses a pixel-to-geographic coordinate mapping algorithm to calculate the geographic coordinates of pixels in long-range UAV images and establishes the corresponding data structure; the pixel-to-geographic coordinate mapping algorithm is used to map pixel coordinates to geographic coordinates one-to-one.

[0010] Furthermore, S2 specifically includes:

[0011] S21, acquire several long-distance UAV aerial images of the target and extract the Prompt data from the images;

[0012] S22, input the Prompt data into the preset SAM large model to obtain a mask describing the current target area;

[0013] S23, downsample the mask to obtain the feature representation tensor of the target region in the image;

[0014] S24, average the feature representation tensor to obtain the target feature;

[0015] S25, calculate the cosine similarity between the target features in the first image and the target features in the second image to obtain the localization result of the target region in the second image;

[0016] S26, The positioning result and the second image are input into the SAM large model to obtain the second mask describing the current target area;

[0017] S27, reuse the positioning result and the second mask describing the current target region, and input them into the SAM large model to obtain the third mask describing the current target region;

[0018] S28, repeat S27 to obtain the Nth mask describing the current target region, obtain the pixel range of the target region in the image, and obtain the area of ​​the target region based on the target region and by calculating the number of pixels in the mask.

[0019] Furthermore, S3 specifically involves: acquiring several close-range UAV aerial images of the target, using YOLO series models to perform target detection on crops, and extracting the number of crop plants.

[0020] Furthermore, S3 specifically refers to:

[0021] S31. Use labelImg software to annotate the targets in a single image and generate a YOLO format annotation file, including the target's category subscript, the target's center point x coordinate, y coordinate, and the target's width and height information, and save it in a normalized format in a txt file with the same name as the image file.

[0022] S32 then divides the txt file into a training set and a test set, and performs the standard Ultralytics YOLO training process on the YOLO model;

[0023] S33 When both Precision and Recall on the test set reach 95% or higher, training ends and the corresponding model training weight results are saved; based on the weights, the number of targets in the remote sensing image can be obtained by counting the number of target detection boxes.

[0024] S34 then acquires the camera's shooting range S to obtain the crop density within the target area. The drone's flight altitude is set to H, the CMOS sensor width is L_1m (long side) and L_2m (short side), the number of pixels on the long side is N_1, the number of pixels on the short side is N_2, and the camera focal length is f. The formula for calculating the resolution of the remote sensing image is then:

[0025]

[0026] The area S = resolution × n under the corresponding pixel is calculated according to the resolution formula, and the target density ρ of the crop is finally calculated.

[0027] This application also provides a system for measuring crop area and predicting yield of gramineous crops, the system comprising:

[0028] The image acquisition module uses a drone to take aerial photos at a first set altitude to obtain multiple images of crop fields and to take one or more close-up images of crops at a second set altitude. The close-up images of crops taken by the drone are crop sampling points within the target area.

[0029] The crop identification module uses a single manual prompt to identify the crops to be identified and input them into the improved SAM large model, and then performs batch identification.

[0030] The crop quantity recognition module uses a trained YOLO model to recognize the number of crops within a range from close-up images of crops.

[0031] The geographic coordinate calculation module uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the geographic coordinates of pixels in long-range UAV images and establishes a corresponding data structure; it also uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the geographic coordinates of pixels in short-range images.

[0032] The innovation of this invention lies in:

[0033] ① The multi-model hybrid task architecture ensures the accuracy of a single task.

[0034] ② Modify the SAM model so that it can identify batches with a single prompt.

[0035] ③ It enables real-time identification of farmland areas by UAVs without the need to synthesize large aeronautical charts and then run the identification algorithm.

[0036] ④ It has better accuracy and calibrability compared to traditional NDVI. Attached Figure Description

[0037] Figure 1 Example image of mask visualization for the large SAM model;

[0038] Figure 2 This is a diagram of the transition matrix from the world coordinate system to the camera coordinate system.

[0039] Figure 3 Example diagrams of a pinhole camera model and three coordinate systems;

[0040] Figure 4 A data structure diagram for mapping pixel coordinates to geographic coordinates;

[0041] Figure 5 This is a diagram illustrating the overall solution;

[0042] Figure 6 This is a schematic diagram summarizing the structure of the Segment Anything model. Detailed Implementation

[0043] This application discloses a method and system for measuring crop area and predicting the yield of gramineous crops, which is based on two modeling methods: the SegmentAnything large model and the YOLO small model. This invention improves, expands, splits, recombines and reuses these two technologies. The implementation of the specific technical solution has many similarities with the original technology, as well as many different improvement measures, which are hereby explicitly stated.

[0044] First, a brief introduction to the Segment Anything large model and the YOLO small model:

[0045] The Segment Anything Large Model. The original paper is Kirillov A, Mintun E, Ravi N, et al. Segment anything[J]. arXiv preprint arXiv:2304.02643,2023, hereinafter referred to as the SAM large model. The model structure can be seen here. Figure 6 The input to this large model is the target intention point marker, bounding box marker, or mask information (hereinafter referred to as prompt) and the original image. The output is the mask that the model believes it is targeting and the corresponding confidence level. Its drawback is that the prompt must be manually input for each image to output a targeted target mask. In this invention, it serves as both a feature extractor and a recognition model. Since the model and its training set are open source and the weight file is large (1.5GB), this invention reuses its model weights. Because the model consists of multiple modules, it can be split into multiple modules for separate use. The main focus is on using its Image Decoder part separately and using intermediate results as additional calculation basis. Intermediate result 1: Tensor representation of image features after image encoder, with a tensor structure of [64, 64, 256].

[0046] A small YOLO model. The original project address is: https: / / github.com / ultralytics / yolov5, hereinafter referred to as yolov5. YOLO series models are commonly used for object detection. This paper applies its model structure, specifically named YOLOv5m, without reusing weights, and trains new weights for detecting wheat ears on a new training set.

[0047] Example 1

[0048] This application provides a method for measuring crop area and predicting yield of gramineous crops. The overall implementation path includes the following steps:

[0049] S1 first uses a drone to take aerial photos at a first set altitude to obtain multiple images of crop fields and at a second set altitude to take one or more close-up images of crops. The close-up images taken by the drone are crop sampling points within the target area.

[0050] S2 uses a single manual prompt to identify the crops to be recognized and inputs them into the improved SAM large model, and then performs batch recognition.

[0051] S3 uses a trained YOLO model to identify the number of crops within a range from close-up images of crops;

[0052] S4 uses a pixel-to-geographic coordinate mapping algorithm to calculate the geographic coordinates of pixels in long-range UAV images and establishes the corresponding data structure; the pixel-to-geographic coordinate mapping algorithm is used to map pixel coordinates to geographic coordinates one-to-one.

[0053] Firstly, step S2 of the above scheme provides a measurement method based on the SAM large model that can identify the area of ​​other similar crop features in batches after a single Prompt. A Prompt refers to a guidance given to the model by the user to indicate the user's intent; its data format can be pixel coordinates, bounding box data, or mask data. The point data structure is [n,2], where n is the number of points and 2 represents the x and y pixel coordinates of the points in the graph. The bounding box data structure is [n,4], where n is the number of bounding boxes and 4 represents the coordinates of the center point of the bounding box and its length and width. The mask data structure is [w,h], using 0 or 1 to indicate whether a location point is a target area. The core of the Prompt method lies in the fact that the model has learned a general object segmentation method; the user only needs to indicate the approximate location of the target, and the model can output the segmentation results.

[0054] S2 includes the following:

[0055] Users need to provide a drone aerial image (Image1) as a prompt. They can select the desired crop area, either as points or bounding boxes, and input it along with the aerial image into a pre-defined SAM model to obtain a mask (mask1) describing the target area. Figure 1As shown. Assuming the long side of the image is W and the wide side is H, the mask data structure should be [W, H]. Then, downsampling is used to interpolate the mask into a tensor of [64, 64]. Based on this tensor, the tensor representing the crop region in the image feature representation tensor is extracted, and its structure can be [k, 256]. Based on this, the feature representation tensor of the target region in the image can be extracted. Then, the average value is calculated to obtain target_feature, with a structure of [1, 256] to uniformly represent the features of the target. At this time, another image Image2 can be input, and its feature tensor is obtained through SAM, resulting in test_feature, with a structure of [64, 64, 256]. Then, the structure of test_feature is transformed into [256, 4096], and the cosine similarity is calculated with target_feature to obtain similarity, with a structure of [1, 4096]. The maximum value is extracted, and it is enlarged to the original image size W / 64 to obtain the location result of the target region in Image2. Using the location information point1 and Image2 as input, a mask mask2 is obtained through the SAM large model. Point1 and mask2 can then be reused as prompts to continue inputting into the SAM model to obtain mask3, and point1 and mask3 can be reused again to obtain mask4. Mask4 can then be used as the mask for the target region in Image2. This method can be considered a batch modification of the original SAM model. The mask can be viewed as a semi-transparent image of the same size as the original image, where the target region is opaque. This allows us to obtain the range of the target region and calculate its area by placing it over the original image.

[0056] Secondly, in step S3 of the above scheme, YOLO series models are used to perform target detection on crops and extract the number of crop plants or ears.

[0057] S3 includes the following:

[0058] First, a target detection dataset is established, using wheat ears as an example. The labelImg software is used to label wheat ears in a single image, generating a YOLO format annotation file. This file includes the target's category index, the x and y coordinates of the target's center point, and the target's width and height information, saved in a normalized format in a .txt file with the same name as the image file. The file is then divided into training and testing sets, and a standard Ultralytics YOLO training process is performed. Training ends when both the precision and recall on the testing set reach 95% or higher, and the corresponding model training weights are saved. These weights can be used to detect wheat ears; the number of targets in the remote sensing image can be obtained by counting the number of target detection boxes.

[0059] Next, the camera's shooting range S needs to be obtained to acquire the crop density within the target area. Due to the characteristics of fisheye cameras, although the field of view is widened, edge objects will be displayed as a near-side view. Furthermore, since the maximum number of bounding boxes in the YOLO model is 15120, the target area needs to be cropped. The formula for calculating the resolution of UAV remote sensing images varies depending on the camera. Assuming the UAV's flight altitude is H, the camera's CMOS sensor width is L_1m (long side) and L_2m (short side), the number of pixels on the long side is N_1, the number of pixels on the short side is N_2, and the camera's focal length is f, then the formula for calculating the resolution of the remote sensing image is:

[0060]

[0061] The area S = resolution × n under the corresponding pixel can be calculated using the resolution formula, and the target density ρ of the crop can be obtained by calculation.

[0062] Thirdly, step S4 of the above scheme is a mapping algorithm from UAV remote sensing image pixel coordinates to geographic coordinates. Its purpose is to calculate the coordinates of each pixel in the UAV remote sensing image under the Universal Transverse Mercator Grid System (hereinafter referred to as UTM). First, camera calibration is performed. Since cameras generally exhibit nonlinear radial and tangential distortion, the images captured by the UAV camera need to be corrected. This invention uses the Brown model to correct distortion. The correction process requires obtaining five distortion coefficients (k_1, k_2, p_1, p_2, k_3), i.e., the camera's intrinsic parameters. These distortion coefficients are already included in the UAV-captured JPG file and can be obtained by reading the DewarpData from the JPG file. The entire process can be implemented using methods from the OpenCV library.

[0063] After obtaining the corrected image, a pinhole camera model needs to be constructed, such as... Figure 3 As shown. Calculate the positions of the image point and real image point in the World Mapping (UTM) coordinate system. A drone equipped with GPS is required; the acquired JPG image file will include the drone's position coordinates and pose information (Latitude|Longitude|Height|Omega|Phi|Kappa), a total of six parameters. First, the latitude and longitude need to be transformed to the UTM coordinate system. This can be done using the SpatialReference method in the GDAL library to convert data between multiple standard coordinate systems. The drone's latitude and longitude coordinates are in the WSG84 Earth coordinate system, and the unit is degrees. Latitude and longitude information cannot be used to calculate collinearity equations, therefore it needs to be transformed to the UTM coordinate system. The transformation matrix is ​​shown below. Figure 2 The first matrix is ​​the rotation matrix, and the second matrix is ​​the translation matrix. Multiplying the two matrices gives the transition matrix.

[0064] Next, we need to calculate the coordinates of the camera CMOS planar pixels in the world coordinate system. The parameters we need to know beforehand include the CMOS single-pixel size, calculated as CMOS width / CMOS horizontal resolution and the camera focal length. The calculated tensor is [3, 3648, 5472], where the first dimension represents the pixel coordinates, and the second and third dimensions represent each pixel.

[0065] At this point, the collinearity equation is applied:

[0066]

[0067] Assume (x_0, y_0, z_0) is the camera origin, (x_1, y_1, z_1) is a pixel, and (x_2, y_2, z_2) is a real object point. If there is no surface elevation model, we can assume z_2 is 0, and then calculate the real coordinates x_2, y_2 of the pixel in meters. Therefore, we can construct a data structure of [4, 3648, 5427] to record the correspondence between the pixel coordinates and the world coordinates of the pixel points. Figure 4 .

[0068] At this point, the remote sensing image tile can be reconstructed by customizing the world coordinate range. Pixels in the pixel coordinates are selected in reverse order using the world coordinate system range. Subsequently, a remote sensing image tile without rotation in the world coordinate system is constructed, and RGB and coordinate system information are output, creating a Geotiff file.

[0069] In other words, compared to the original method of unifying and merging maps and then running an artificial intelligence model to identify land parcels, this method completes the classification of geographic information at the initial stage of capturing and merging orthophotos. This method can also be used for a variety of other classification tasks.

[0070] Based on the above overall plan, the actual operational workflow is as follows:

[0071] ① Determine the planting area using long-distance drone aerial photography. Existing flight planning software can be used, setting the drone's flight altitude to 80m-120m and the flight speed to be customizable, prioritizing camera performance and image clarity. Image repetition is not required at this stage; only the entire image needs to be captured. The crop area S can then be obtained using the aforementioned area measurement algorithm.

[0072] ② Determine crop planting density using close-range drone aerial photography. Existing flight planning software can be used to set the drone to a fixed point for shooting at an altitude of 3-5m. If the crop planting in the target area is relatively even, single-point measurements can be taken within the target area. If the crop planting situation in the target area is relatively complex, multiple points can be selected for shooting. The purpose of this shooting is to obtain the crop planting density. The YOLO series algorithm is used, therefore it is necessary to use grasses with relatively obvious fruits. This step can obtain the plant density ρ.

[0073] ③ The "average weight per plant" K is manually set based on historical harvest statistics.

[0074] ④ By combining the above three data points, the estimated crop yield within the drone's flight range is calculated as M = KS. ρ See the overall flowchart. Figure 5 As shown.

[0075] Example 2

[0076] This application also provides a system for measuring crop area and predicting yield of gramineous crops, the system comprising:

[0077] The image acquisition module uses a drone to take aerial photos at a first set altitude to obtain multiple images of crop fields and to take one or more close-up images of crops at a second set altitude. The close-up images of crops taken by the drone are crop sampling points within the target area.

[0078] The crop identification module uses a single manual prompt to identify the crops to be identified and input them into the improved SAM large model, and then performs batch identification.

[0079] The crop quantity recognition module uses a trained YOLO model to recognize the number of crops within a range from close-up images of crops.

[0080] The geographic coordinate calculation module uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the geographic coordinates of pixels in long-range UAV images and establishes a corresponding data structure; it also uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the geographic coordinates of pixels in short-range images.

[0081] This system employs a hybrid approach, combining a large AI-based Transformer model with a small, traditional YOLO convolutional neural network model, to handle different tasks. First, the large model detects crop area. Then, fixed-point sampling is used to obtain the crop cover density within the identified planting area. Combined with the average quality parameter per plant set in the algorithm, the system calculates estimated crop area and yield. This approach offers improvements in processing speed and accuracy compared to traditional satellite remote sensing. Furthermore, based on UAV remote sensing images, pixel-level classification information is mapped onto an orthophoto map, enabling plot segmentation during UAV flight.

[0082] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art can make various modifications and adjustments within the technical scope disclosed in the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of measuring crop area and estimating yield of a gramineous crop, characterized by, The method comprises the following steps: S1 first uses a UAV to take multiple crop field images at a first set height and one or more close-range crop images at a second set height, wherein the close-range UAV image is a crop sampling point in the target area; S2 uses a single artificial guidance prompt to identify the crop to be identified and inputs it into the improved SAM large model, and then performs batch identification; S2 specifically comprises: S21, obtaining a plurality of remote UAV aerial images of the target, and extracting prompt data in the images; S22, inputting the prompt data into the preset SAM large model to obtain a mask describing the current identification target area; S23, downsampling the mask to obtain a feature representation tensor of the target area in the image; S24, averaging the feature representation tensor to obtain a target feature; S25, calculating the cosine similarity between the target feature in the first image and the target feature in the second image to obtain the positioning result of the target area in the second image; S26, inputting the positioning result and the second image into the SAM large model to obtain a second mask describing the current identification target area; S27, multiplexing the positioning result and the second mask describing the current identification target area, and inputting them into the SAM large model to obtain a third mask describing the current identification target area; S28, repeating S27 to obtain an Nth mask describing the current identification target area, obtaining the pixel range of the target area in the image, and obtaining the area of the target area according to the target area and the number of pixels in the mask; S3 uses the trained YOLO model to identify the close-range crop image to obtain the number of crops in the range, obtains the crop coverage density of the identified planting area, and calculates the estimated results of the crop area and the crop yield according to the average quality parameter set in the algorithm; S4 uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the geographic coordinates of the pixels in the remote UAV image and establishes a corresponding data structure; the pixel coordinates and the geographic coordinates are one-to-one corresponding using the pixel coordinate to geographic coordinate mapping algorithm.

2. The method of claim 1, wherein S3 specifically comprises: obtaining a plurality of close-range UAV aerial images of the target, using a YOLO series model to detect the target of the crop, and extracting the number of crop plants.

3. The method for measuring crop area and predicting yield of gramineous crops according to claim 1, characterized in that, S3 specifically comprises: S31, using labelImg software to label the target in a single image to generate a YOLO format annotation file, including the class index of the target, the x coordinate and y coordinate of the target center point, and the width and height information of the target, and saving it in a txt file with the same name as the image file in a normalized format; S32, then divide the txt file into a training set and a test set, and perform a standard ultralytics YOLO training process on the YOLO model; S33 when the precision and recall of the test set are both above 95%, the training is ended and the corresponding model training weight result is saved; according to the weight, the number of targets in the remote sensing image can be obtained by counting the number of target detection boxes; S34 subsequently obtain the camera shooting range S to obtain the crop density in the target area, set the unmanned aerial vehicle flight height as H, the CMOS sensor width as the long side L_1m, the short side as L_2m, the sensor long side pixel number as N_1, the short side pixel number as N_2, and the camera focal length as f; then the resolution calculation formula of the remote sensing image is: ； According to the resolution formula, the area under the corresponding pixel is calculated The final crop target density is calculated .

4. A crop area measurement and gramineous crop yield estimation system characterized by comprising: The system is used to realize the estimation method of claim 1, and the system comprises: An image acquisition module, which uses an unmanned aerial vehicle to take multiple crop field images at a first set height and one or more crop close-up images at a second set height, wherein the unmanned aerial vehicle close-up image is a crop sampling point in the target area; A crop identification module, which uses a single manual guidance prompt to identify the crop to be identified and inputs it into the improved SAM large model, and then performs batch identification; A crop quantity identification module, which uses a trained YOLO model to identify the crop close-up image to obtain the number of crops in the range; A geographic coordinate calculation module, which uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the geographic coordinates of the pixel mapping in the remote distance unmanned aerial vehicle image and establishes a corresponding data structure; and uses a pixel coordinate to geographic coordinate mapping algorithm to calculate the close distance.

Images