A fruit tree statistics method based on deep learning technology
Through satellite remote sensing image and deep learning technology, combined with bilinear interpolation and non-local mean filtering, the problem of low coverage of drone remote sensing images is solved, and the rapid and accurate statistics of fruit trees in large-scale areas are achieved, and agricultural production is achieved. Provides effective guidance.
Patent Information
- Application Number
- CN202210256175.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-15
AI Technical Summary
The existing tree statistics methods mainly rely on drone remote sensing images, making it difficult to cover large areas, and the model requires specific height shooting during on-site predictions, so it cannot provide effective production guidance.
Satellite remote sensing images are used to perform fruit tree statistics, and the images are processed through bilinear interpolation and non-local mean filtering, random cropping and annotation are trained, and the number of fruit trees in large areas is counted in combination with non-maximum suppression methods.
It has achieved rapid and accurate statistics of fruit trees in large areas and provided reliable guidance for agricultural production.
Smart Images

Figure CN114596505B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a tree statistical method, and in particular to a fruit tree statistical method based on deep learning technology. Background Art
[0002] The statistics of the number of fruit trees is conducive to the scientific guidance of agricultural production for the government. In the existing tree statistics, cutting-edge deep learning technology is applied. For example, a systematic description of the statistical methods of trees in remote sensing images taken by drones using deep learning technology is made. The statistical methods are divided into two parts: model training and real-time measurement and statistics. In the model training part, image acquisition is first carried out, then the trees in the image are labeled, and finally the model is trained using deep learning technology; the real-time measurement part uses drones to obtain real-time images of regional trees, and at the same time, the image information is passed into the model to count the number of trees; in addition, deep learning technology is used to improve the original SSD neural network model. First, the environmental images taken by the drone are collected, and the images are input into the pre-trained neural network model for identifying trees for prediction.
[0003] However, in these studies, researchers used remote sensing images taken by drones to build models. The images taken by drones cover a small area, and one image cannot cover a large area at one time. When the model trained by drone remote sensing images is used for field prediction, the best prediction effect can only be obtained by taking pictures at the altitude where the drone is flying when building the model, and tree statistics in a small area cannot provide effective production guidance. Summary of the invention
[0004] The purpose of the present invention is to overcome the above-mentioned problems and provide a fruit tree statistics method based on deep learning technology. The fruit tree statistics method obtains satellite remote sensing images to count fruit trees in a large area, which can better provide assistance for production guidance.
[0005] The purpose of the present invention is achieved through the following technical solutions:
[0006] A fruit tree statistical method based on deep learning technology comprises the following steps:
[0007] Obtain satellite remote sensing images of the area to be counted;
[0008] On the satellite remote sensing image, several training sample sub-regions and several test sample sub-regions are selected respectively;
[0009] For the training sample sub-area, the bilinear interpolation method is used to sample the image, and then the non-local mean filter is used to denoise the interpolated image; the random cropping method is used to crop the area as the training sample of the deep learning model; the fruit trees in the image are annotated with rectangular boxes using the labelImg software; the training sample set and the corresponding annotation set are trained by the deep learning model using an iterative training method to obtain the fruit tree target detection model;
[0010] For the test sample sub-area, first sample and then divide to construct a test sample set;
[0011] Send the test image to the fruit tree object detection model for prediction, and obtain the bounding box coordinates of the fruit tree in each image;
[0012] After completing the prediction of all images in the test set, each bounding box coordinate relative to the sub-image is converted into a bounding box coordinate relative to the entire study area image, and the non-maximum suppression method is used to remove the repeated detection of the same fruit tree;
[0013] In the annotation file of the entire image, each row represents the rectangular coordinates of a tree, and counting the number of rows completes the regional fruit tree statistics.
[0014] In a preferred embodiment of the present invention, each sub-region is sampled and the image is upsampled to 2400×2400 pixels; thus, the fruit tree target is magnified, which can better improve the recognition effect.
[0015] A preferred embodiment of the present invention, wherein, for each enlarged sub-region, a random cropping method is used to generate 6000 640×640 pixel images from the pixel areas of 6 2400×2400 regions, each image containing 10 fruit trees, as training samples for the deep learning model.
[0016] A preferred solution of the present invention, wherein the decimal coordinates are decomposed into four adjacent integer coordinate points by bilinear interpolation using straight lines parallel to the coordinate axis, and different weights are assigned for calculation; specifically:
[0017] Find the four known pixel points closest to the corresponding coordinates, set them as Q11 = (x1, y1), Q12 = (x1, y2), Q21 = (x2, y1), Q22 = (x2, y2);
[0018] Linear interpolation in the x direction:
[0019]
[0020] Where, R1=(x,y1), R2=(x,y2);
[0021] Linear interpolation in the y direction:
[0022]
[0023] Considering the x-direction and y-direction, the result of bilinear interpolation is:
[0024]
[0025] In a preferred embodiment of the present invention, the specific operation of the non-local mean filtering is:
[0026] The pixels of the entire image are used for filtering. f(x) is a noisy image, G(x) is a noise-removed image, pixels i and j are selected in f(x), i is taken as the center point, and the neighborhood of i is selected, called the similar block comparison area. The Gaussian weighted Euclidean distance between the pixel values of each point in this neighborhood is calculated. The relationship between the input and output of the denoising process is:
[0027]
[0028] Among them, the value range of y is the entire image; ω is the similarity between image blocks x and y; is the Gaussian kernel; λ(i) is the block area in the image centered at i; h is the attenuation factor. For fruit trees planted in orchards, the color of the fruit tree crown and the background color (i.e., soil color) observed from remote sensing images are green or yellow, and the soil color in different planting areas of the orchard is different. Through the above method, continuously selecting λ(i) and λ(j) in the whole image can prevent the interference of similar background color pixels, making the calculated weights obtained at the target position more accurate.
[0029] In a preferred embodiment of the present invention, the test sample set is constructed by:
[0030] The test area of 600×600 pixels was sampled into 2400×2400 pixels, and the enlarged test image was divided into test samples of 640×640 pixels using the overlapping division method. 100 pixels were selected as the width of the test sample overlapping area; each fruit tree was completely included in at least one test sample image.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] The fruit tree statistics method of the present invention obtains satellite remote sensing images to count fruit trees in a large area. Combined with existing deep learning technology, it can achieve fast and accurate statistics and provide reliable assistance for guiding agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1This is a flowchart of the fruit tree statistical method based on deep learning technology of the present invention.
[0034] Figure 2 It is the coordinate system of the bilinear interpolation method of the present invention.
[0035] Figure 3 The satellite remote sensing image obtained by the present invention. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.
[0037] See also Figure 1 The fruit tree statistics method based on deep learning technology of this embodiment includes the following steps:
[0038] Obtain high-resolution remote sensing images from Google Earth with a resolution of 0.27 meters and a scale of 1:770. The image area should be the area to be detected, which contains a large number of fruit trees, such as a town or a county.
[0039] In the picture, select several sub-areas of 600×600 pixels (usually 6 blocks) as the image data source for training samples in the deep learning model. The sub-area should contain thousands of fruit trees, so as to provide a sufficient number of positive samples for the training set.
[0040] For each sub-region, upsample the image to 2400×2400 pixels. Fruit trees only have 20×20 pixels in satellite remote sensing images, so in order to achieve better recognition results, the fruit tree targets are enlarged. During the upsampling process, when the target image coordinate points are filled relative to the original image coordinate points, the coordinates directly converted according to the scaling ratio will have decimals, and the image coordinates are integers. Therefore, bilinear interpolation is used to upsample the image. Bilinear interpolation uses straight lines parallel to the coordinate axes to decompose the decimal coordinates into four adjacent integer coordinate points and assign different weights for calculation.
[0041] See also Figure 2 , Q11, Q12, Q21, Q22 are integer point coordinates, the corresponding coordinates of the original image are point P, the pixel of point P is the value of function f at point P = (x, y), and the values of function f at Q11 = (x1, y1), Q12 = (x1, y2), Q21 = (x2, y1), Q22 = (x2, y2) are known.
[0042] First, perform linear interpolation in the x direction:
[0043]
[0044]
[0045] Among them, R1=(x,y1), R2=(x,y2).
[0046] Then perform linear interpolation in the y direction:
[0047]
[0048] In summary, the final result of bilinear interpolation is:
[0049]
[0050] Bilinear interpolation is to find the four pixel points closest to the corresponding coordinates, calculate the weights, and determine the pixel values of the interpolated image.
[0051] The image after interpolation and upsampling has noise. For fruit trees, the presence of noise will interfere with its feature information, which will make the model learning effect poor when deep learning is used to learn the features of the fruit tree crown. Therefore, NLM is used on the interpolated image for image denoising. NLM filtering (non-local mean filtering) uses the pixels of the entire image for filtering. f(x) is a noisy image, G(x) is a noise-removed image, and pixels i and j are selected in f(x). i is taken as the center point, and the neighborhood of i is selected, called the similar block comparison area. The Gaussian weighted Euclidean distance of the pixel values between each point in this neighborhood is calculated. The relationship between the input and output of the denoising process is:
[0052] G(x)=∑ y∈I ω(x,y)f(x)
[0053]
[0054]
[0055] The value range of y is the entire image, ω is the similarity between image blocks x and y, which is a weight. Each output is weighted for the entire image, and finally the weighted sum of the entire image is calculated. is the Gaussian kernel, λ(i) is the block area in the image centered at i, and h is the attenuation factor.
[0056] For fruit trees planted in orchards, the color of the fruit tree crowns and the background color (i.e., soil color) observed from remote sensing images are green or yellow. The soil colors in different planting areas of the orchard are different. Therefore, continuously selecting λ(i) and λ(j) in the entire image can prevent the interference of similar background color pixels, making the calculated weights obtained at the target position more accurate.
[0057] For each enlarged sub-area, a random cropping method is used to generate 6,000 640×640 pixel images from the pixel areas of 6 2400×2400 areas. Each image contains about 10 fruit trees as training samples for the deep learning model. The images are cropped and used as training samples for the model because the resolution of satellite remote sensing images is large. If the downsampling factor is too large, data information is easily lost. If the downsampling factor is too small, the network model feedforward needs to store a large number of feature maps (feature matrices) in the memory, which may cause insufficient memory and prevent normal training and reasoning.
[0058] Each randomly cropped image of 640 × 640 pixels was manually labeled, and the fruit trees in the image were labeled with rectangular boxes using labelImg software.
[0059] The training sample set and the corresponding annotation set are trained using the deep learning YOLOv5 model using an iterative training method to obtain a fruit tree target detection model.
[0060] The test sample set was constructed using the same method as the training sample. The 600×600 pixel test area was upsampled to 2400×2400 pixels, and then the enlarged test image was divided into 640×640 pixel test samples using the overlapping partitioning method. 100 pixels were selected as the width of the test sample overlapping area to ensure that each fruit tree can be completely included in at least one test sample image.
[0061] The test image is sent to the fruit tree target detection model for prediction, and the bounding box coordinates of the fruit trees in each image are obtained.
[0062] After completing the prediction of all images in the test set, each bounding box coordinate relative to the sub-image is converted into a bounding box coordinate relative to the image of the entire study area, and the non-maximum suppression method is used to remove repeated detections of the same grapefruit tree.
[0063] In the annotation file txt of the entire image, each line represents the rectangular coordinates of a tree. The number of lines in txt is counted to complete the regional fruit tree statistics.
[0064] In summary, the fruit and vegetable counting method of this embodiment is specifically demonstrated:
[0065] (1) Two Google Earth high-resolution satellite remote sensing images of Shishan Town, Meizhou City were used, with a spatial resolution of 0.27 m, image sizes of 5632×5120 pixels and 6400×5888 pixels, respectively, and a scale of 1:770. Twelve training areas of 600×600 pixels were selected in the two study areas to generate training samples, and six study areas were selected in each area, such as Figure 3 .
[0066] (2) The grapefruit tree is made to have only 20×20 pixels in the 0.27-meter pixel remote sensing image. In order to obtain better recognition effect, the grapefruit tree target is enlarged by image upsampling. For the 12 training areas, the bilinear interpolation method is used to enlarge the 600×600 pixel area to 2400×2400, and the NLM filter is used to filter the noise on the interpolated image. After that, the random cropping method is used to generate 6000 640×640 pixel images from the 12 2400×2400 pixel areas as training samples for the convolutional neural network model.
[0067] (3) For the test area, the same bilinear interpolation method as the training area is used to upsample the 600×600 pixel test area to 2400×2400 pixels. Then, the overlapping partitioning method is used to divide the enlarged test area into 640×640 pixel test samples. 100 pixels are selected as the width of the test sample overlapping area to ensure that each grapefruit tree can be completely included in at least one test sample image.
[0068] (4) Use the yolov5 single-stage target detection model to train and test the dataset.
[0069] (5) After completing the prediction of all sub-images in the test set, each bounding box coordinate relative to the sub-image is converted into a bounding box coordinate relative to the image of the entire study area, and the non-maximum suppression method is used to remove the repeated detection of the same grapefruit tree.
[0070] The above is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited to the above content. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A fruit tree statistical method based on deep learning technology, It is characterized in that The following steps are involved: Obtain satellite remote sensing images of the area to be counted; On the satellite remote sensing image, several training sample sub-regions and several test sample sub-regions are selected respectively; Sample each sub-region and upsample the image to 2400×2400 pixels; For each enlarged sub-region, a random cropping method was used to generate 6000 640×640 pixel images from the pixel areas of 6 2400×2400 regions, each containing 10 fruit trees, as training samples for the deep learning model; For the training sample sub-area, the bilinear interpolation method is used to sample the image, and then the non-local mean filter is used to denoise the interpolated image; the random cropping method is used to crop the area as the training sample of the deep learning model; the fruit trees in the image are annotated with rectangular boxes using the labelImg software; the training sample set and the corresponding annotation set are trained by the deep learning model using an iterative training method to obtain the fruit tree target detection model; Bilinear interpolation uses straight lines parallel to the coordinate axes to decompose the decimal coordinates into four adjacent integer coordinate points, assigning different weights for calculation; specifically: Find the four known pixel points closest to the corresponding coordinates and set them as Q 11 =(x 1 ,y 1 ), Q 12 =(x 1 ,y 2 ),Q 21 =(x 2 ,y 1 ),Q 22 =(x 2 ,y 2 ); Linear interpolation in the x direction: wherein, R 1 =(x, y 1 ), R 2 =(x, y 2 ); Linear interpolation in the y direction: Considering the x-direction and y-direction, the result of bilinear interpolation is: For the test sample sub-area, first sample and then divide to construct a test sample set; Send the test image to the fruit tree object detection model for prediction, and obtain the bounding box coordinates of the fruit tree in each image; After completing the prediction of all images in the test set, each bounding box coordinate relative to the sub-image is converted into a bounding box coordinate relative to the image of the entire study area, and the non-maximum suppression method is used to remove the repeated detection of the same fruit tree; In the annotation file of the entire image, each row represents the rectangular coordinates of a tree, and counting the number of rows completes the regional fruit tree statistics.
2. According to the fruit tree statistical method based on deep learning technology according to claim 1, It is characterized in that The construction operation of the test sample set is: The test area of 600×600 pixels was sampled into 2400×2400 pixels, and the enlarged test image was divided into test samples of 640×640 pixels using the overlapping division method. 100 pixels were selected as the width of the test sample overlapping area; each fruit tree was completely included in at least one test sample image.
Citation Information
Patent Citations
Fruit tree identification and quantity monitoring method and system based on unmanned aerial vehicle data acquisition
CN110569786A
Unmanned aerial vehicle navel orange tree image target recognition and statistical method based on deep learning
CN114092814A
