Method for extracting density of fruit trees in forest-grain intercropping area based on remote sensing

Abstract

The invention provides a forest-grain intercropping area fruit tree density extraction method based on remote sensing, and relates to the field of fruit tree density extraction. The method comprises the following main steps: firstly, based on an instance segmentation thought, utilizing a deep learning model to replace a traditional segmentation mode to extract a land parcel from an image with ultrahigh spatial resolution as a basic calculation unit; then, on the basis of a time sequence model, features of Sentinel-2 time sequence images are extracted to distinguish common farmland plots and forest-grain intercropping plots, and plot units for subsequent fruit tree density extraction tasks are determined; then, extracting spatial information features of the fruit tree from the GF-2 image based on a semantic segmentation model; and finally, designing a template filling strategy based on a semantic segmentation result, and obtaining the planting density of the fruit trees in the forest-grain intercropping area under the constraint of a land parcel result. The extraction method is high in accuracy, high in processing speed and good in demonstration application effect, and can provide support for scientifically adjusting the planting structure of the vegetation in the forest-grain intercropping area.

Description

Technical Field

[0001] The present invention relates to the field of agricultural remote sensing monitoring, and more specifically, to a method for extracting fruit tree density in a forest-crop intercropping area based on remote sensing. Background Art

[0002] Intercropping refers to the formation of multi-level and multifunctional artificial complex groups through different combinations of various plants, making full use of the "spatial difference" and "temporal difference" formed by various plants during their growth process to maximize the limited agricultural resources. Studies have shown that at appropriate planting density, the use of intercropping planting structure can increase yields by 20% to 50%. However, improper intercropping methods may have the opposite effect. As the crowns of fruit trees grow larger year by year, fruit forests planted at high density will significantly reduce the yield of food crops under the forest. By extracting the planting density of fruit trees in the intercropping area, support can be provided for the scientific adjustment of the planting structure of vegetation in the intercropping area.

[0003] A single tree is the basic unit of a forest, and the key to extracting forest density lies in the number of trees in the forest stand. Since there is a clear difference in height and remote sensing index between the center of a tree and the border of the crown, the center of a single tree in a grove can be located through these features. Therefore, there are currently two commonly used methods for extracting tree density, namely the local maximum method based on canopy height and the local maximum method based on visible light image spectrum. The former mostly uses high-precision drone images or radar data to extract tree height and calculate forest density, but the manpower and material costs of obtaining large-area data through drones are high, which limits the feasibility of this type of method in forest density mapping in large-scale areas. Using satellite remote sensing data, a functional relationship between remote sensing index and tree position can be constructed to extract the center point of the tree and calculate its density. However, the construction process of the functional relationship of this type of method is complicated and the accuracy is difficult to improve. It is more suitable for the extraction of individual trees in areas with sparse tree planting, and is not applicable to the situation where fruit trees grow more in forest-grain intercropping areas and are dominated by jungles.

[0004] Therefore, providing a method for extracting fruit tree density that is suitable for forest-crop intercropping areas is an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing, which includes three aspects:

[0006] In the first aspect, the present invention provides a plot extraction method based on the idea of ​​instance segmentation, which uses a deep learning model to replace the traditional segmentation method to extract plots from images with ultra-high spatial resolution, as a basic calculation unit for extracting fruit tree density in forest-grain intercropping areas, including the following steps:

[0007] (1) Preprocessing GF-2 (GF-2) data to obtain ultra-high spatial resolution images includes six steps: radiometric calibration, atmospheric correction, orthorectification, image registration, image fusion and image mosaicking;

[0008] (2) A farmland zoning and stratification system was established based on the visual features of farmland images, and intercropping farmland was divided into several relatively consistent subtypes;

[0009] (3) Based on the visual features of each type of land, appropriate instance segmentation models are selected and improved to hierarchically extract each type of farmland, and the subclasses of each farmland are merged to obtain a complete farmland information extraction result map.

[0010] In the above-mentioned land parcel extraction based on the idea of ​​instance segmentation, in step (1), the specific preprocessing method for obtaining the ultra-high spatial resolution image is as follows:

[0011] (1-1) Radiation calibration, by converting the data value (DN) recorded by the sensor into apparent reflectance, thus eliminating the error of the sensor itself, it is necessary to calculate the radiation brightness value based on the absolute radiation calibration coefficient. The formula can be expressed as:

[0012] L λ =Gain·DN+Offset

[0013] Among them, L λ The converted standard radiance is obtained from the China Resources Satellite Application Center. The payload observation value (DN) and the absolute calibration coefficient offset (Offset) of the GF-2 (Gaofen-2) satellite are obtained. The sensor parameter DN value and the Offset value used in the present invention are shown in Table 1.

[0014] Table 1 Absolute radiation calibration coefficients

[0015]

[0016] (1-2) Atmospheric correction, which converts the radiation brightness or surface reflectivity into the actual reflectivity of the surface, can effectively eliminate the influence of factors such as atmosphere and illumination on the reflection of ground objects. The present invention uses the FLAASH model for atmospheric correction. The model algorithm is based on the pixel-level correction of the MODTRAN 4+ radiation transfer model, corrects the collateral effects caused by diffuse reflection, includes the classification map of cirrus and opaque clouds, and can adjust the spectrum smoothing caused by artificial suppression;

[0017] (1-3) Orthorectification, which is a process of eliminating the influence of terrain or deformation caused by camera orientation, etc., and generating a planar orthophoto. In addition to conventional geometric correction, the image distortion caused by terrain fluctuations must be corrected according to the DEM. The present invention uses the orthorectification tool RPC Orthorectification Workfolw, and the digital elevation model uses the GMTED 2010 DEM. At the same time, in order to perform subsequent image fusion, the GF-2 panchromatic image is resampled to a resolution of 1m, and the multispectral image is resampled to a resolution of 4m.

[0018] (1-4) Image registration: calibrate one image (reference image) with another image in the same area. Since the panchromatic image and multispectral image of GF-2 do not completely correspond, the multispectral image is selected as the reference image to register the panchromatic image, and the output resolution is 1m.

[0019] (1-5) Image fusion: A low-resolution multispectral image is fused with a high-resolution panchromatic image to obtain an image with both high spatial resolution and multispectral characteristics. After orthorectification, the panchromatic band of the GF-2 image has a resolution of 1 m, and the four multispectral bands have a resolution of 4 m. The present invention adopts the NNDiffuse Pan Sharpening fusion method to fuse the multispectral and panchromatic images.

[0020] (1-6) Image mosaicking: stitching two or more remote sensing images with geographic coordinates to obtain a larger and more complete remote sensing image. The present invention needs to perform image registration before mosaicking the GF-2 image. The Sentinel-2 (Mariner 2) remote sensing image is selected as the reference image, and the GF-2 image is registered as the image to be registered. Then, the seamless mosaicking tool (Seamless Mosaic) is used to mosaic the GF-2 image.

[0021] In the above plot extraction based on the idea of ​​instance segmentation, in step (2), “geographical partitioning” is also required to improve the interpretation effect of remote sensing images. The specific steps are as follows:

[0022] (2-1) With the help of road network / water system / topographic lines, the image is divided into naturally aggregated sub-regions by means of masks to ensure that the type of landforms in each sub-region is relatively uniform. This step cuts the image into several discrete geographic blocks so that the classification process can be limited to the corresponding sub-regions, thereby avoiding the propagation of classification errors. At the same time, after the image is divided into blocks, it can be used for parallel computing to improve the extraction efficiency;

[0023] (2-2) Under the constraints of each geographic block, the edges of the target objects are extracted in each area. For farmland with blurred edges, its object boundaries are obtained based on texture. On the basis of zoning based on the slope calculated based on terrain data, the differences in geometric and texture features of farmland in different areas are expressed.

[0024] In the above plot extraction based on the idea of ​​instance segmentation, in step (3), it is necessary to use the instance segmentation model to extract various types of farmland in layers. The specific steps are as follows:

[0025] (3-1) The instance segmentation algorithm used in the present invention is the Hybrid Task Cascade (HTC) model, with the skeleton being ResNeXt-101-64x4d and Deformable ConvNets v2 (DCN). 16 1024*1024 pixel plot samples drawn by visual interpretation are input into the model and trained;

[0026] (3-2) Using a sliding window, many pixel blocks are cut out from the original image from top to bottom and from left to right for instance segmentation;

[0027] (3-3) Each pixel block is divided into a target area and an ignored area. The left and upper edges of the ignored area are two pixels away from the grid boundary. When the coordinates (x, y) of the upper left corner of the detection box falls within the target area, the detection result is retained;

[0028] (3-4) When the sliding window is on the boundary of the original image, the ignored area on the edge in the same direction as the grid is merged into the target area.

[0029] In the second aspect, the extraction of intercropping plots is based on a time series model. Based on the characteristics of Sentinel-2 time series images, the present invention distinguishes between ordinary farmland plots and intercropping plots, and determines the plots for subsequent fruit tree density extraction tasks, including the following steps:

[0030] 1. Design time series features suitable for major food crops and major fruit trees, and extract the phenological change characteristics of different vegetation in long-term series Sentinel-2 images;

[0031] 2. Crop classification is performed based on the irregular time series model, and according to the classification results, the discrimination conditions of only grain crops, grain crops and fruit trees, and only fruit trees are designed to divide the farmland plots into ordinary farmland and forest-grain intercropping plots.

[0032] In the extraction of the above-mentioned intercropping plots, in step (1), the time series characteristics suitable for the main food crops and the main planted fruit trees are designed as follows:

[0033] (1-1) The present invention selects three vegetation time series features to distinguish three types of plots, namely the normalized difference vegetation index (NDVI), the normalized red edge index (NDRE1) and the red edge normalized vegetation index (NDVIre1). The calculation formula is as follows:

[0034]

[0035] In the extraction of the above-mentioned forest-crop intercropping plots, in step (2), the vegetation phenological change characteristics are extracted based on the irregular time series model. The specific steps are as follows:

[0036] (2-1) For each pixel position in the image, the present invention extracts an irregular VI time series from the original optical image and converts it into a "time-feature" matrix to characterize the crop growth pattern. In the "time-feature" matrix, the horizontal axis represents time and the vertical axis represents the value of the vegetation index;

[0037] (2-2) The present invention inputs the "time-feature" matrix obtained in the first step into the three channels of ResNet-18 to obtain a three-dimensional matrix suitable for ResNet-18, thereby extracting the characteristics of vegetation phenological changes;

[0038] The formula for stretching the VI time series into a "time-feature" matrix is ​​as follows:

[0039]

[0040] Among them, VI ori is the original value of the VI. r is the VI value after stretching, and length is the length of the time series.

[0041] (2-3) After obtaining the pixel-level classification results based on time series features, the present invention designs three discrimination conditions to distinguish between ordinary farmland and intercropping plots of forest and grain, namely 1. only grain crops exist (the distribution area of ​​grain crops is greater than 80%); 2. there are grain crops and fruit trees (the distribution area of ​​grain crops and the distribution area of ​​fruit trees are greater than 30%) and only fruit trees exist (the distribution area of ​​fruit trees is greater than 80%), and the intercropping plots of forest and grain are screened out.

[0042] The third aspect is the extraction of fruit tree density in the forest-crop intercropping area. The present invention extracts the spatial information features of fruit trees from the GF-2 image based on the semantic segmentation model, and designs a template filling strategy based on the semantic segmentation results to obtain the planting density of fruit trees under the constraints of the plot results, including the following steps:

[0043] 1. Using a semantic segmentation model based on a convolutional neural network, the fruit trees in the pre-processed GF-2 image at a resolution of 1m were extracted to obtain the semantic segmentation results of the fruit trees;

[0044] 2. Based on the template filling algorithm, different templates are formed according to the spectral characteristics of the crowns of fruit trees of different sizes, and then these generated templates are filled into the fruit tree semantic segmentation results to obtain the crown center position of each fruit tree;

[0045] 3. The planting density of fruit trees is obtained by calculating the number of center points of fruit trees within each plot element and dividing it by the element area (mu).

[0046] To extract the density of fruit trees in the intercropping area, in step (1), a semantic segmentation model is used to extract fruit trees in the ultra-high resolution image. The specific steps are as follows:

[0047] (1-1) The semantic segmentation model used in the present invention is the D-LinkNet model. D-LinkNet uses the ResNet trained on the ImageNet dataset as the network encoder and adds a dilated-convolution layer with a shortcut in the central part, so that the entire network has stronger recognition ability, a larger receptive field, and can integrate multi-scale information. 20 512*512 pixel fruit tree samples drawn by visual interpretation are input into the model and trained.

[0048] In the above-mentioned extraction of fruit tree density in the intercropping area, in step (2), the center position of the crown of each fruit tree is obtained by using the template filling algorithm. The specific steps are as follows:

[0049] (2-1) Randomly select the marking area and mark all individual fruit trees to reduce subjective bias, create a series of square empty templates of odd lengths to ensure the grid is centered, and assign each fruit tree to the template with the closest area;

[0050] (2-4) After aligning the center of the fruit tree with the center of the template, the spectral average of the connected components in each template is calculated to generate the original template. The original template series shows the typical characteristics of the crown of a single fruit tree;

[0051] (2-5) A mathematical model is constructed for the typical features of the original template, and better performance is obtained through preprocessing. Since the near-infrared response of the center of the fruit tree is stronger, the spectral characteristics of the template pixel are calculated using the following formula:

[0052] R=C

[0053]

[0054] Where R is NIR2 band (908nm), B is Red band (658.8nm), G is Green band (546.2nm), C is a constant, BG min and BG maxThey are the minimum and maximum values ​​of the B and G bands respectively, N represents the template length, and distance_to_cent represents the distance to the center of the template in pixels.

[0055] (2-6) On the target raster image, given a four-connected domain C of arbitrary shape, the pixel value in C is denoted by x i,j , where i, j are coordinates on the digital grid; given several square sub-connected domains S with different side lengths n , where n is the subscript of the connected subdomain; the pixel value in each connected subdomain is recorded as y i,j , where i and j are the coordinates of the sub-connected domain bounding box grid. The sub-connected domain S is filled into C. Each sub-connected domain can be reused any number of times. The ultimate goal is to minimize the global value G on the digital grid.

[0056] G = sum(|x i,j -y i,j |)+p+q+r

[0057] Where p is the penalty value for the sub-connected domain extending beyond C, q is the additional penalty value for the pixels not filled in C, r is the penalty value for the overlapping part of the sub-connected domain, and at most one sub-connected domain is placed at each position i, j.

[0058] (2-7) Template matching with the connected part of the fruit tree forest is achieved through convolution, and the matching results are stored in the fitness map. The fitness map of each template is calculated according to the "large first" principle. The pixels with higher values ​​in the fitness map indicate that the crown center position is more suitable. The local maxima in the suitability map are considered to be potential crown centers, and they are confirmed as crown centers after meeting specific constraints.

[0059] The extraction of the density of fruit trees in the intercropping area is as follows: In step (3), the density of fruit trees in the intercropping area is calculated. The specific steps are as follows:

[0060] (3-1) Reproject the plot results and template filling results to the same projection coordinate system, and then calculate the number of fruit trees inside each plot element in the plot vector file;

[0061] (3-2) The number of fruit trees planted per mu can represent its planting density. The conversion between square meters and public ares can be obtained by multiplying the area of ​​the plot element by 0.0015. The fruit tree planting density can be obtained by calculating the number of fruit tree center points in each plot element and dividing it by the element area (mu). The result is stored in the attribute table of the plot vector file.

[0062] Compared with the prior art, the remote sensing-based fruit tree density extraction method in the forest-crop intercropping area provided by the present invention achieves at least the following beneficial effects:

[0063] (1) The method for extracting the density of fruit trees in the forest-grain intercropping area provided by the present invention takes the plot as the smallest basic unit. The plot refers to a completely closed farmland with the same owner. It is the smallest unit in agricultural production and management. Taking the plot as the smallest basic unit lays a good foundation for the subsequent calculation of the density of fruit trees, and is more in line with the management of farmland in actual production and planting.

[0064] (2) The method for extracting the density of fruit trees in the intercropping area provided by the present invention is more suitable for the investigation of the density of fruit trees in a large-scale intercropping area. The remote sensing image based on the present invention is a pre-processed ultra-high resolution GF-2 image. Compared with the method of using radar data or ultra-high resolution images obtained by unmanned aerial vehicles, the method of improving the image resolution by data fusion in the present invention can reduce the manpower and material costs to a certain extent.

[0065] (3) The method for extracting the density of fruit trees in the intercropping area provided by the present invention uses a template filling algorithm to obtain the crown center of the fruit tree, and completes the task of high-precision stand number by generating a template. It is suitable for the planting of fruit trees in the intercropping area dominated by jungles;

[0066] (4) The present invention provides a method for extracting fruit tree density in forest-grain intercropping areas, which can provide support for scientifically adjusting the planting structure of vegetation in forest-grain intercropping areas, improve the economic benefits of forest-grain intercropping farmland, resolve the conflict between forest and grain land, and guide farmers to scientifically adjust the planting structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 A flow chart of a method for extracting the density of fruit trees in forest-crop intercropping areas based on remote sensing technology implemented by the present invention;

[0069] Figure 2 A model structure diagram of the irregular time series model used in the present invention;

[0070] Figure 3 This is a before and after comparison diagram of GF-2 data preprocessing according to the present invention;

[0071] Figure 4 The model structure diagram of the D-LinkNet semantic segmentation model used in the present invention;

[0072] Figure 5 The internal structure diagram of module B of the D-LinkNet semantic segmentation model used in the present invention;

[0073] Figure 6 This is a flow chart of the template filling algorithm used in the present invention;

[0074] Figure 7 This is a fruit tree semantic segmentation result diagram of an embodiment of the present invention;

[0075] Figure 8 This is a template filling result diagram of an embodiment of the present invention;

[0076] Fig. 9 This is the basic base map for plot attribute statistics according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Combine the following Figure 1 The process of the method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing provided by the present invention is described as follows:

[0079] 1. Preprocess the GF-2 data to obtain ultra-high spatial resolution images, including six steps: radiation calibration, atmospheric correction, orthorectification, image registration, image fusion and image mosaicking;

[0080] 2. Based on the visual features of the images of farmland plots, a farmland zoning and stratification system is established to divide the farmland into several relatively consistent subtypes. Based on the visual features presented by each type of plot, appropriate instance segmentation models are selected and improved to extract each type of farmland in a hierarchical manner, and the subtypes of each farmland are merged to obtain a complete farmland information extraction result map;

[0081] 3. Design time series features suitable for major food crops and major fruit trees, extract the phenological change characteristics of different vegetation in the long-term series Sentinel-2 images, and classify crops based on the irregular time series model. According to the classification results, design the discrimination conditions for only food crops, food crops and fruit trees, and only fruit trees, and divide the farmland into ordinary farmland and intercropped forest and grain plots;

[0082] 4. A semantic segmentation model based on a convolutional neural network was used to extract fruit trees from the preprocessed GF-2 image at a resolution of 1m to obtain the semantic segmentation results of the fruit trees. Based on the template filling algorithm, different templates were formed according to the spectral characteristics of the crowns of fruit trees of different sizes. These generated templates were then filled into the semantic segmentation results of the fruit trees to obtain the crown center position of each fruit tree.

[0083] 5. The planting density of fruit trees is obtained by calculating the number of center points of fruit trees within each plot element and dividing it by the element area (mu).

[0084] The following is an introduction to each of the above steps.

[0085] 1. Preprocessing of GF-2 images. GF-2 data is obtained from China Resources Satellite Application Center (http: / / 36.112.130.153:7777 / DSSPlatform / productSearch.html). GF-2 is the first sub-meter civilian optical satellite independently developed by my country. It has a return period of 69 days and carries two PMS (panchromatic / multispectral). The spectral range that can be obtained is shown in Table 2.

[0086] Table 2 GF-2 satellite parameters

[0087]

[0088] The preprocessing of GF-2 data includes six steps: radiation calibration, atmospheric correction, orthorectification, image registration, image fusion and image stitching:

[0089] (1) Radiometric calibration is to convert the data value (DN) recorded by the sensor into apparent reflectance to eliminate the error of the sensor itself. The radiant brightness value needs to be calculated based on the absolute radiometric calibration coefficient. The formula can be expressed as:

[0090] L λ =Gain·DN+Offset

[0091] Among them, L λ The converted standard radiance is obtained from the China Resources Satellite Application Center. The payload observation value (DN) and the absolute calibration coefficient offset (Offset) of the GF-2 satellite are obtained. The sensor parameter DN value and the Offset value used in the present invention are shown in Table 1.

[0092] Table 3 Absolute radiation calibration coefficients

[0093]

[0094]

[0095] (2) Atmospheric correction refers to converting the radiation brightness or surface reflectivity into the actual reflectivity of the ground surface, which can effectively eliminate the influence of factors such as atmosphere and illumination on the reflection of ground objects. The present invention uses the FLAASH model for atmospheric correction. The model algorithm is based on the pixel-level correction of the MODTRAN 4+ radiation transfer model to correct the collateral effects caused by diffuse reflection, including the classification map of cirrus and opaque clouds, and can adjust the spectrum smoothing caused by artificial suppression;

[0096] (3) Orthorectification, which is the process of eliminating the influence of terrain or deformation caused by camera orientation, etc., and generating a planar orthophoto. In addition to conventional geometric correction, the image distortion caused by terrain undulations must be corrected according to the DEM. The present invention uses the orthorectification tool RPC Orthorectification Workfolw, and the digital elevation model uses GMTED2010DEM. At the same time, in order to perform subsequent image fusion, the GF-2 panchromatic image is resampled to a resolution of 1m, and the multispectral image is resampled to a resolution of 4m.

[0097] (4) Image registration: calibrate one image (reference image) with another image in the same area. Since the panchromatic image and multispectral image of GF-2 do not completely correspond, the multispectral image is selected as the reference image and the panchromatic image is registered. The output resolution is 1m.

[0098] (5) Image fusion: A low-resolution multispectral image is fused with a high-resolution panchromatic image to obtain an image with both high spatial resolution and multispectral characteristics. After orthorectification, the panchromatic band of the GF-2 image has a resolution of 1 m, and the four multispectral bands have a resolution of 4 m. The present invention adopts the NNDiffuse Pan Sharpening fusion method to fuse multispectral and panchromatic images.

[0099] (6) Image mosaicking: stitching two or more remote sensing images with geographic coordinates to obtain a larger and more complete remote sensing image. Before mosaicking the GF-2 image, the present invention needs to perform image registration, select the Sentinel-2 remote sensing image as the reference image, and register the GF-2 image as the image to be registered, and then use the seamless mosaicking tool (SeamlessMosaic) to mosaic the GF-2 image.

[0100] 2. Extraction of farmland plots. Extraction of farmland plots requires "geographical zoning" to simulate the human eye's recognition process of farmland targets, gradually micro-scale the complex macroscopic spatial cognitive problems, and improve the interpretation effect of remote sensing images. First, with the assistance of road networks / water systems / topographic lines, the image needs to be divided into naturally aggregated sub-regions by masking to ensure that the types of objects in each sub-region are relatively uniform. This step cuts the image into several discrete geographic blocks, so that the classification process can be limited to the corresponding sub-regions, thereby avoiding the propagation of classification errors. At the same time, the image can be divided into blocks for parallel computing to improve the extraction efficiency. Secondly, under the constraints of each geographic block, the edge of the target object needs to be extracted in each area, and for farmland with blurred edges, its object boundary is obtained based on texture. On the basis of partitioning based on the slope calculated based on terrain data, the differences in geometric and texture features presented by farmland in different regions are expressed.

[0101] The plot extraction of farmland is obtained through the instance segmentation model, which is a combination of target detection and semantic segmentation. The target is detected in the image (target detection), and then each pixel is labeled (semantic segmentation), and a single instance can be extracted at a time. The instance segmentation algorithm used in the present invention is the Hybrid Task Cascade (HTC) model, and the skeleton is ResNeXt-101-64x4d and Deformable ConvNets v2 (DCN). The algorithm output results are post-processed by overlapping sliding window strategy, and Soft-NMS is used to improve the model performance.

[0102] The present invention uses an overlapping sliding window strategy, which means that many pixel blocks are cut out from the original image from top to bottom and from left to right for instance segmentation. Each pixel block is divided into a target area and an ignored area. The left edge and the upper edge of the ignored area are two pixels away from the grid boundary. When the coordinate (x, y) of the upper left corner of the detection box falls within the target area, the detection result is retained. When the sliding window is on the boundary of the original image, the ignored area on the edge in the same direction as the grid is merged into the target area. At the same time, the overlapping sliding window strategy should meet the following rules: 1) The sliding window should be as large as possible to ensure that each cultivated land object appears completely in at least one of the sliding windows. 2) When the coordinate (x, y) of the upper left corner of the sliding window falls within the target area, the detection result should also be retained. 3) When the sliding window is on the boundary of the original image, the edge area in the same direction as the sliding window should be merged into the target area.

[0103] 3. Extraction of forest-grain intercropping plots. The present invention uses a time series model based on the characteristics of Sentinel-2 time series images to distinguish between ordinary farmland plots and forest-grain intercropping plots, and determines the plots for subsequent fruit tree density extraction tasks. First, design time series features suitable for major food crops and major planted fruit trees, and extract the characteristics of different vegetation phenological changes in long-term series Sentinel-2 images. The present invention selects three vegetation time series features to distinguish three types of plots, namely the Normalized Difference Vegetation Index (NDVI), the Normalized Red Edge Index (NDRE1), and the Red Edge Normalized Vegetation Index (NDVIre1). The calculation formula is as follows:

[0104]

[0105] Then, the crop classification is carried out based on the irregular time series model, and the discrimination conditions of only grain crops, grain crops and fruit trees, and only fruit trees are designed according to the classification results, and the farmland plots are divided into ordinary farmland and forest-grain intercropping plots. Among them, the structure of the irregular time series model is as follows Figure 2 As shown, the two-dimensional convolutional neural network model (2DCNN) model is usually used to extract spatial and spectral features. The general input is a three-dimensional matrix containing R, G, and B channels. Since the time series feature is one-dimensional data, the main task of using 2DCNN to process time series data is to convert the one-dimensional data into a two-dimensional matrix. In the present invention, the 2DCNN model automatically learns the characteristics of crop growth patterns by stretching the VI time series into a "time-feature" matrix. The classic residual network ResNet-18 model of 2DCNN is used, and learning and training are performed after obtaining pre-training on the ImageNet dataset. The process mainly includes two steps. First, at each pixel position, an irregular VI time series is extracted from the original optical image and converted into a "time-feature" matrix to characterize the crop growth pattern. In the "time feature" matrix, the horizontal axis represents time and the vertical axis represents the value of the vegetation index; secondly, the "time-feature" matrix obtained in the first step is input into the three channels of ResNet-18 to obtain a three-dimensional matrix suitable for ResNet-18, thereby extracting the characteristics of vegetation phenological changes;

[0106] The formula for stretching the VI time series into a "time-feature" matrix is ​​as follows:

[0107]

[0108] Among them, VI ori is the original value of the VI. r is the VI value after stretching, and length is the length of the time series.

[0109] Finally, after obtaining the pixel-level classification results based on time series features, three discrimination conditions are used to distinguish ordinary farmland from intercropping plots, namely: 1. There are only grain crops (the distribution area of ​​grain crops is greater than 80%); 2. There are grain crops and fruit trees (the distribution area of ​​grain crops and the distribution area of ​​fruit trees are greater than 30%); 3. There are only fruit trees (the distribution area of ​​fruit trees is greater than 80%), and the intercropping plots are screened out.

[0110] 4. Extraction of the crown center of fruit trees in the intercropping plots. First, a semantic segmentation model based on a convolutional neural network is used to extract fruit trees from the preprocessed GF-2 image at a resolution of 1m to obtain the semantic segmentation results of the fruit trees. The semantic segmentation model uses the D-LinkNet model.

[0111] The structure of D-LinkNet network is as follows Figure 4 As shown. D-LinkNet uses Linknet with a pre-encoder as its backbone network, and has an additional dilated convolution layer in the central part. The advantage is that the dilated convolution layer is added to expand the field of view. Linknet is an effective semantic segmentation neural network, which has the advantages of skip connections, residual blocks, and encoder-decoder architecture. D-LinkNet uses ResNet trained on the ImageNet dataset as the encoder of the network, and adds a dilated-convolution layer with a shortcut in the central part, making the entire network more recognizable, with a larger receptive field, and integrating multi-scale information. Therefore, the D-LinkNet model has a stronger feature extraction capability and can achieve effective classification of heavily confused plots.

[0112] The entire network is divided into three modules: A, B, and C. Module A is the encoder part of the network, and module C is the decoder part of the network. The settings of these two parts are the same as those of the linkNet network. Module B is unique to D-LinkNet. The internal structure of this module is as follows: Figure 5 As shown in the figure, the parallel structure of the B modules can effectively fuse multi-scale features. Dilated-convolution can effectively expand the receptive field. The corresponding receptive fields from top to bottom are 31, 15, 7, 3, and 1. Finally, the results of each branch are added together to obtain the fused features.

[0113] During model training, the D-LinkNet model training parameters were set as follows: the initial learning rate was set to 0.01, and the adaptive Adam optimizer was used to automatically adjust the learning rate. The sample size was 512 pixels × 512 pixels, and a total of 100 epochs were trained (training was automatically terminated at the 72nd time). The batch size was set to 16, and the remaining parameters were default. Figure 7 This is the fruit tree semantic segmentation result diagram of the embodiment. The model segmentation result is relatively detailed and can effectively extract the fruit trees in the study area.

[0114] Secondly, based on the template filling algorithm, different templates are formed according to the spectral characteristics of the crowns of fruit trees of different sizes, and then these generated templates are filled into the fruit tree semantic segmentation results to obtain the center position of the crown of each fruit tree.

[0115] Among them, the template filling algorithm flow chart is as follows Figure 6 As shown. To calculate the density of fruit trees in a plot, it is necessary to determine the exact center position and number of the fruit trees. Due to different growth states and locations, the crowns of fruit trees are diverse. By observing the crowns of independent fruit trees, it is found that fruit trees with larger crowns often have higher infrared spectral responses in their center and lower infrared responses in the periphery. Fruit trees with smaller crowns have lower overall infrared responses. Therefore, different templates can be formed according to the spectral characteristics of the crowns of fruit trees of different sizes, and then these generated templates can be filled into the fruit tree forest, thereby converting the splitting problem of the fruit tree forest into a filling problem.

[0116] There are many fruit trees of different sizes in the study area, and the general characteristics of fruit trees can be obtained statistically, and reasonable templates can be generated. First, the annotation area is randomly selected, and all fruit trees are annotated to reduce subjective bias. Secondly, a series of square empty templates of odd lengths are created to ensure that the grid is centered and each fruit tree is assigned to the template with the closest area. After aligning the center of the fruit tree with the center of the template, the spectral average of the connected components in each template is calculated to generate the original template. The original template series shows the typical characteristics of the crown of a single fruit tree. However, due to the obvious shadows in the fruit tree images, the spectral characteristics of the fruit tree forest and the fruit tree are not exactly the same. Therefore, it is necessary to construct a mathematical model for the typical characteristics of the original template and obtain better performance through preprocessing. Since the near-infrared response is stronger in the center of the fruit tree, the spectral characteristics of the template pixel are calculated using the following formula:

[0117] R=C

[0118]

[0119] Where R is NIR2 band (908nm), B is Red band (658.8nm), G is Green band (546.2nm). C is a constant, BGmin and BG max They are the minimum and maximum values ​​of bands B and G, respectively. N represents the template length, and distance_to_cent represents the distance to the center of the template in pixels.

[0120] On the target raster image, given a four-connected domain C of arbitrary shape, the pixel value in C is recorded as x i,j , where i and j are coordinates on the digital grid. At the same time, given several square sub-connected domains S with different side lengths n , where n is the subscript of the sub-connected domain. The pixel value in each sub-connected domain is recorded as y i,j , where i and j are the coordinates of the sub-connected domain bounding box grid. Fill the sub-connected domain S into C, and each sub-connected domain can be reused any number of times. The ultimate goal is to minimize the global value G on the digital grid.

[0121] G = sum(|x i,j -y i,j |)+p+q+r

[0122] Where p is the penalty value for the sub-connected domain extending outside C, q is the additional penalty value for the pixels not filled in C, and r is the penalty value for the overlapping part of the sub-connected domain. At the same time, at most one sub-connected domain is placed at each position i, j.

[0123] Template matching with the connected part of the fruit tree forest is achieved through convolution, and the matching results are stored in the fitness map. The fitness map of each template is calculated according to the "large first" principle. The pixel with higher value in the fitness map indicates that the crown center position is more suitable. The local maxima in the suitability map are considered as potential crown centers, and they are confirmed as crown centers after meeting specific constraints.

[0124] 5. Extract the planting density of fruit trees in the intercropping plots of forest and grain. The planting density of fruit trees is obtained by calculating the number of center points of fruit trees in each plot element and dividing it by the element area (mu).

[0125] The plot, semantic segmentation and template filling results of the embodiment of the present invention are shown in FIG. Figure 8 As shown. In the present invention, the overlap of the fruit tree crowns is greater and the crowns of individual trees are smaller, so when the template is generated, the center point is more accurately determined when it is adjusted to 3*3, 5*5, 7*7 and 9*9 pixels. Since the shadows of the fruit tree crowns are inevitably present in the semantic segmentation results, a threshold of 30 is set as the boundary between the shadow and the crown according to the shadow reflectivity of the fruit tree in the figure. The reflectivity below 30 is the shadow, and the reflectivity above 30 is the potential crown pixel. It can be seen that the present invention has well extracted the center point of the fruit tree crown in the embodiment.

[0126] On this basis, the plot results and template filling results are reprojected to the same projection coordinate system, and then the number of fruit tree pixels in each feature in the plot vector file is calculated. The number of fruit trees planted per mu can represent its planting density. The conversion between square meters and public ares can be obtained by multiplying the plot element area by 0.0015. The fruit tree planting density is obtained by calculating the number of fruit tree center points in each plot element and dividing it by the element area (mu). The result is stored in the plot vector file attribute table, as shown in Table 4:

[0127] Table 4 Plot attribute results

[0128]

[0129] The present invention has been implemented on the PC platform. Experiments have shown that the solution can meet the correction requirements of the wrong merged area in the scale set, has a good sense of interaction and a fast response speed in the interactive process, realizes efficient and accurate interactive editing functions, gives full play to the advantages of the scale set model, and effectively solves the problem of wrong merged areas in the scale set model. At the same time, it combines advanced technologies such as the minimum risk segmentation model and topological map to achieve rapid positioning and correction of the wrong merged area. It can be widely used in geographic information systems, remote sensing image processing, and computer vision for image segmentation applications that require regional editing functions in scale set models, providing strong support for technological progress in these fields.

[0130] It should be noted that although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing, characterized in that It includes the following steps: Step 1: Preprocess the GF-2 data to obtain ultra-high spatial resolution images. The preprocessing process includes six steps: radiation calibration, atmospheric correction, orthorectification, image registration, image fusion and image mosaicking; Step 2: Based on the visual features of the images of farmland plots, a farmland zoning and stratification system is established to divide the farmland into several relatively consistent subtypes. Based on the visual features presented by each type of plot, appropriate instance segmentation models are selected and improved to extract each type of farmland in a hierarchical manner, and the subtypes of each farmland are merged to obtain a complete farmland information extraction result map. Step 3: Design time series features suitable for major food crops and major fruit trees, extract the phenological change characteristics of different vegetation in the long-term series Sentinel-2 images, and classify crops based on the irregular time series model. According to the classification results, design the discrimination conditions of only food crops, food crops and fruit trees, and only fruit trees, and divide the farmland plots into ordinary farmland plots and intercropping plots; Step 4: Use a semantic segmentation model based on a convolutional neural network to extract fruit trees from the preprocessed GF-2 image at a resolution of 1 m to obtain the semantic segmentation results of the fruit trees. Based on the template filling algorithm, different templates are formed according to the spectral characteristics of the crowns of fruit trees of different sizes. Then, these generated templates are filled into the semantic segmentation results of the fruit trees to obtain the crown center position of each fruit tree. Step 5, calculate the number of fruit tree center points within each plot element and divide it by the element area to obtain the planting density of fruit trees.

2. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: The method for preprocessing GF-2 data specifically includes: (1) The radiation calibration mentioned above refers to converting the data value (DN) recorded by the sensor into apparent reflectance to eliminate the error of the sensor itself. It is necessary to calculate the radiation brightness value based on the absolute radiation calibration coefficient, where L λ is the converted standard radiation brightness, Offset is the absolute calibration coefficient offset, and the formula can be expressed as: L λ =Gain·DN+Offset (2) The atmospheric correction mentioned above refers to converting the radiance or surface reflectance into the actual reflectance of the ground surface, and using the FLAASH model to perform atmospheric correction. The FLAASH model algorithm is based on the pixel-level correction of the MODTRAN 4+ radiation transfer model, correcting the cascading effects caused by diffuse reflection, including the classification map of cirrus and opaque clouds, and adjusting the spectral smoothing caused by artificial suppression; (3) The orthorectification mentioned above refers to the process of eliminating the influence of terrain or deformation caused by camera orientation to generate a planar orthophoto. In addition to conventional geometric correction, the image distortion caused by terrain undulations must be corrected based on the DEM. The orthorectification tool RPC Orthorectification Workfolw was used, and the digital elevation model used GMTED2010 DEM. At the same time, for the subsequent image fusion, the GF-2 panchromatic image was resampled to 1m resolution, and the multispectral image was resampled to 4m resolution; (4) The image registration mentioned above refers to calibrating one image (reference image) with another image in the same area. Since the panchromatic image and multispectral image of GF-2 do not completely correspond, the multispectral image is selected as the reference image to align the panchromatic image, and the output resolution is 1m; (5) The image fusion mentioned above refers to the fusion of low-resolution multispectral images with high-resolution panchromatic images to obtain an image with both high spatial resolution and multispectral characteristics. The orthorectified GF-2 image has a panchromatic band with a resolution of 1m and four multispectral bands with a resolution of 4m. The NNDiffuse Pan Sharpening fusion method is used to fuse the multispectral and panchromatic images. (6) The image mosaicking mentioned above refers to the process of stitching two or more remote sensing images with geographic coordinates to obtain a larger and more complete remote sensing image. Image registration is required before mosaicking the GF-2 image. The Sentinel-2 remote sensing image is selected as the reference image and the GF-2 image is registered as the image to be registered. Then, the seamless mosaicking tool (SeamlessMosaic) is used to mosaic the GF-2 image.

3. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: In step 2, the farmland zoning and stratification system refers to dividing the image into naturally aggregated sub-areas by means of masks with the assistance of road networks / water systems / topographic lines, ensuring that the types of objects within each sub-area are relatively uniform. Under the constraints of each geographic block, the edges of target objects are extracted in each area, and for cultivated land with blurred edges, its object boundaries are obtained based on texture. On the basis of zoning based on the slope calculated based on terrain data, the differences in geometric and texture features of cultivated land in different areas are expressed.

4. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: In step 2, the instance segmentation model is a combination of target detection and semantic segmentation. The target is detected in the image (target detection), and then each pixel is labeled (semantic segmentation). A single instance can be extracted at a time. The instance segmentation algorithm used is the Hybrid Task Cascade (HTC) model, and the skeleton is ResNeXt-101-64x4d and DeformableConvNets v2 (DCN). The algorithm output results are post-processed through the overlapping sliding window strategy, and Soft-NMS is used to improve the model performance.

5. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 4, characterized in that: The output of the algorithm is post-processed by an overlapping sliding window strategy. The overlapping sliding window strategy refers to cutting out many pixel blocks from the original image from top to bottom and from left to right for instance segmentation, dividing each pixel block into a target area and an ignored area, and the left edge and the upper edge of the ignored area are two pixels away from the grid boundary. When the coordinate (x, y) of the upper left corner of the detection box falls within the target area, the detection result is retained; when the sliding window is on the boundary of the original image, the ignored area on the edge in the same direction as the grid is merged into the target area; at the same time, the overlapping sliding window strategy should meet the following rules: 1) The sliding window should be as large as possible to ensure that each cultivated land unit appears completely in at least one of the sliding windows; 2) When the coordinate (x, y) of the upper left corner of the sliding window falls within the target area, the detection result should also be retained; 3) When the sliding window is on the boundary of the original image, the edge area in the same direction as the sliding window should be merged into the target area.

6. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: The method for extracting vegetation phenology information using a time series model is a two-dimensional convolutional neural network model (2DCNN). The model is usually used to extract spatial and spectral features. The general input is a three-dimensional matrix containing R, G, and B channels. Since the time series feature is one-dimensional data, the main task of using 2DCNN to process time series data is to convert the one-dimensional data into a two-dimensional matrix. The 2DCNN model automatically learns the characteristics of crop growth patterns by stretching the VI time series into a "time-feature" matrix. The classic residual network ResNet-18 model of 2DCNN is used, and learning and training are performed after pre-training on the ImageNet data set. The process includes two steps: First, at each pixel position, the irregular vegetation index (VI) time series is extracted from the original optical image and converted into a "time-feature" matrix to characterize the crop growth pattern. In the "time-feature" matrix, the horizontal axis represents time and the vertical axis represents the value of the vegetation index. Secondly, the "time-feature" matrix obtained in the first step is input into the three channels of ResNet-18 to obtain a three-dimensional matrix suitable for ResNet-18, so as to extract the characteristics of vegetation phenological changes; The formula for stretching the VI time series into a "time-feature" matrix is ​​as follows: Among them, VI ori is the original value of the VI. r is the VI value after stretching, and length is the length of the time series.

7. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 6, characterized in that: The vegetation index time series is a method for characterizing the characteristics of crop growth patterns. The vegetation indices are the normalized difference vegetation index (NDVI), the normalized red edge index (NDRE1) and the red edge normalized vegetation index (NDVIre1). The calculation formula is as follows:

8. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: A semantic segmentation model based on a convolutional neural network, wherein the semantic segmentation model is a D-LinkNet model, which uses Linknet with a precoder as its backbone network and has an additional dilated convolution layer (Dilatedconvolution) in the central part.

9. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: A method for obtaining the center of a fruit tree crown using a template filling algorithm, wherein the template filling algorithm refers to obtaining the general characteristics of fruit trees from a statistical perspective and generating a reasonable template: first, randomly selecting a labeled area and labeling all individual fruit trees to reduce subjective bias; second, creating a series of square empty templates of odd lengths to ensure that the grid is centered and each fruit tree is assigned to the template with the closest area; after aligning the center of the fruit tree with the center of the template, calculating the spectral average of the connected components in each template to generate an original template; the original template series exhibits the typical characteristics of the crown of an individual fruit tree; due to the obvious shadows in the fruit tree image, the spectral characteristics of the fruit tree forest and the individual fruit tree are not exactly the same, therefore, it is necessary to construct a mathematical model for the typical characteristics of the original template, and obtain better performance through preprocessing, and since the near-infrared response of the center of the fruit tree is stronger, the spectral characteristics of the template pixel are calculated using the following formula: R=C Where R is NIR2 band (908nm), B is Red band (658.8nm), G is Green band (546.2nm), C is a constant, BG min and BG max They are the minimum and maximum values ​​of the B and G bands, respectively. N represents the template length. Distance_to_cent represents the distance to the center of the template in pixels. On the target raster image, given a four-connected domain C of arbitrary shape, the pixel value in C is recorded as x i,j , where i and j are coordinates on the digital grid, and several square sub-connected domains Sn with different side lengths are given, where n is the subscript of the sub-connected domain, and the pixel value in each sub-connected domain is recorded as y i,j , fill the sub-connected domain S into C, each sub-connected domain can be reused any number of times, and calculate the global value G on the digital grid, the formula is as follows: G=sum(|x i,j -y i,j |)+p+q+r Where p is the penalty value for the sub-connected domain extending beyond C, q is the additional penalty value for the pixels not filled in C, and r is the penalty value for the overlapping part of the sub-connected domain. At the same time, at most one sub-connected domain is placed at each position i, j. The ultimate goal of the calculation is to minimize G. Template matching with the connected part of the fruit tree forest is achieved through convolution, and the matching results are stored in the fitness map. The fitness map of each template is calculated according to the "large first" principle. The pixels with higher values ​​in the fitness map indicate that the crown center position is more suitable. The local maxima in the suitability map are considered to be potential crown centers, which are confirmed as crown centers after meeting specific constraints.

10. The method for extracting fruit tree density in forest-crop intercropping areas based on remote sensing according to claim 1, characterized in that: A method for obtaining the planting density of fruit trees by calculating the number of center points of fruit trees in each plot element and dividing it by the element area (mu). The density calculation method refers to calculating the fruit tree coverage and the number of trees planted per mu, and storing the results in the plot vector file attribute table: first, the plot results and template filling results are reprojected to the same projection coordinate system, and then the number of fruit tree pixels in each element in the plot vector file is calculated; secondly, the number of fruit trees planted per mu can represent its planting density, and the conversion between square meters and public ares can be obtained by multiplying the plot element area by 0.0015, so as to obtain the fruit tree planting density by calculating the number of center points of fruit trees in each plot element and dividing it by the element area (mu), and storing the results in the plot vector file attribute table.

Images