Object-oriented and deep learning based forest gap extraction method from GF-2 image

Abstract

The application discloses a GF-2 image forest gap extraction method based on object orientation and deep learning, first, the object orientation classification method of membership function is used to extract the forest gap of the research area, then the result is verified in precision and is converted into an index map, and the forest gap sample label data set required for making a deep learning model is prepared. In the third step, the performance and accuracy of the PSPNet model and the DeepLabv3+ model for forest gap identification are compared, finally, the PSPNet model is improved by comprehensively considering, the method is that the lightweight network MobileNetV2 is used to replace the ResNet50 with a large number of parameters, the Focal Loss loss function is added to the model, the CBAM attention mechanism module is added, and the structure of the model is optimized. The results show that the method can improve the efficiency and accuracy of the model for identifying the forest gap. The method can effectively complete the forest gap identification work based on high-resolution remote sensing images, and the problem that there are few studies on the large-scale identification of forest gaps by using remote sensing technology at present.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing image recognition, and particularly relates to a GF-2 image forest gap extraction method based on object orientation and deep learning. BACKGROUND

[0002] Forest gap, also known as forest interspace, generally refers to the interspace between trees in a forest caused by diseases and insect pests, weather, wind, fire, or artificial logging, and is a common natural phenomenon in forest systems. Forest gap plays an important role in maintaining species diversity and forest material circulation in forest systems. In an ecological system, due to the presence of forest gap, the light and hydrothermal conditions in the forest will change due to the change of microenvironment, and the forest system can only be continuously updated. In actual investigation, grasping the ecological safety of the forest system and exploring the spatial distribution of forest gap in the forest system can help further study the dynamic evolution and ecological safety of the forest ecological system.

[0003] At present, many researchers still use visual interpretation method to extract forest gap based on remote sensing image, combined with field survey, and visually identify forest gap on the image according to the experience of the staff. Such method depends on the subjective consciousness and experience of the researchers, and the extraction accuracy will vary according to the artificial experience, and there is inevitably a lack of scientificity. The object-oriented classification method in current remote sensing information extraction has gradually become the mainstream method for forest gap extraction. The object-oriented classification method takes object as the minimum segmentation unit, which can well avoid the "salt and pepper phenomenon" of the traditional pixel-based classification. However, due to the fact that the object-oriented classification method depends on the experience of researchers to determine the optimal segmentation scale and construct the multi-level classification system to a certain extent, although the obtained forest gap extraction effect is good, the efficiency is not high when extracting a large number of forest gaps in a large area. With the emergence of deep learning method, it is possible to extract forest gap information in a large range with high efficiency and high accuracy. How to effectively improve the forest gap recognition efficiency and accuracy in a large area, and accurately identify forest gap by using high-resolution remote sensing image data is a key problem. Furthermore, it is necessary to ensure that the recognition model has good universality in the areas where Yunnan pine is the dominant tree species in Yunnan-Guizhou-Sichuan region. SUMMARY

[0004] In order to solve the problems in the prior art, the present application provides a GF-2 image forest gap extraction method based on object orientation and deep learning. In order to solve the above technical problems, the present application adopts the following technical solutions:

[0005] The GF-2 image forest gap extraction method based on object orientation and deep learning comprises the following steps:

[0006] (1) Data preprocessing: performing radiation calibration, geometric accuracy correction and atmospheric correction on the data;

[0007] (2) Establish object-oriented classification based on multi-level classification system;

[0008] (2.1) Determine the optimal segmentation scale parameter: use the ESP 2 plug-in of eCognition software combined with the RMAS value method to determine the optimal segmentation scale parameter of different land types in the study area, and establish a multi-level classification system;

[0009] (2.2) Select object classification features: according to the image spectrum, shape, and texture features, optimize the features for feature classification;

[0010] (2.3) Establish a fuzzy classification rule set: based on the optimized spectral, shape, and texture features, establish a fuzzy classification rule set according to the characteristics of each land class in the image, and complete the forest gap identification work;

[0011] (3) Construct forest gap sample data set: convert the forest gap results identified by the object-oriented method into an index map after manual correction, and cut it into 256*256 pixel forest gap samples with sliding window to ensure the integrity of the context semantic information;

[0012] (4) Sample data augmentation: since there is no public data set for forest gap sample data, it needs to be made by oneself, and forest gaps are less distributed in forest systems. In order to prevent model training from overfitting, the number of data sets needs to be increased;

[0013] (5) Train PSPNet and DeepLabv3+ model to identify forest gap: train the semantic segmentation model through the sample data set made by oneself, and compare the models from the aspects of forest gap extraction accuracy and model performance;

[0014] (6) Identify forest gap based on improved PSPNet model;

[0015] (6.1) Optimize MobileNetV2 for backbone network: to improve the operation speed of the model, replace the original large parameter ResNet50 network with the lightweight network MobileNetV2;

[0016] (6.2) Loss function improvement: in the model training process, to prevent the imbalance between the number of forest gap positive samples and background information negative samples, introduce the Focal Loss function to weigh the positive and negative samples by applying different weights;

[0017] (6.3) CBAM attention mechanism module: first apply channel attention mechanism to the feature map, and then apply spatial attention mechanism based on this, so that the model pays attention to the edge detail information of the forest gap;

[0018] (6.4) Model result optimization and improvement: the PSPNet model structure is improved and optimized, four upsampling are performed, the deep information and the shallow information are connected four times, the shallow information is used to the greatest extent, and the classification precision is improved;

[0019] As preferred, formula (1), (2), (3) is selected as the RMAS value calculation formula in step (2.1), and the most segmentation scale of each type of ground object is calculated in combination with the ESP 2 tool of eCognition software.

[0020]

[0021] Wherein, L represents the number of bands of the image to be segmented, and ΔC L The absolute value of the mean difference of the neighborhood of the object in L band under a single segmentation scale is represented by S L The standard deviation of the object in L band under a single segmentation scale is represented by S

[0022] As preferred, the spectral, shape and texture characteristics of the image of different ground classes are respectively counted in step (2.2), and appropriate features are selected for ground object classification.

[0023] As preferred, step (2.3) constructs a fuzzy classification rule set according to the features selected in step (2.2), completes the classification of ground object types in the study area under the multi-level classification system, and then completes the identification of forest gap.

[0024] As preferred, step (3) constructs a forest gap sample data set, evaluates the forest gap classification results obtained in step (2) in terms of accuracy, and then converts the results into an index map by combining artificial correction. Considering that the training samples should ensure the integrity of the forest gap information and the background information in the label data set should be as little as possible, finally, 256*256 pixels are selected to divide the picture by sliding window, and the forest gap sample data set for deep learning model training is obtained.

[0025] As preferred, in step (4), since the number of forest gaps distributed in the forest system is limited, and there is no public data set for forest gap sample data, the data set for model training needs to be made by oneself. Due to the limited number of samples, in order to prevent overfitting phenomenon in model training, the forest gap data set is augmented. Based on the albumentations library of OpenCV, the data set can be enhanced by operations such as flipping, translation, sharpening and brightness change.

[0026] As preferred, the step (5) trains the PSPNet model and the DeepLabv3+ model, compares the performance of the two models and the accuracy of the gap recognition, selects the average pixel accuracy (mPA), the average intersection over union (mIoU) and the speed (FPS), calculates the average pixel accuracy according to formula (4), and calculates the average intersection over union according to formula (5).

[0027]

[0028] In the formula, k represents the number of classes to be divided, i represents the actual value of the class, j represents the predicted value of the class, and p ij The larger the values of mIoU and mPA are, the better the effect of gap information extraction is, and the larger the FPS value is, indicating that the model processes more pictures per second and is more efficient.

[0029] As preferred, the step (6) improves the PSPNet model by comprehensively considering the recognition effect of the two models on the gap and the performance of the model according to the comparison of the two models in the step (5). First, ResNet50 is used as the backbone network of the model. Since the calculation amount and the parameter amount of this feature extraction network are large, the calculation speed and efficiency of the model are affected. MobileNetV2 is used as the backbone network to extract image features and improve the efficiency of the model. In the extraction of gap information based on remote sensing images, the gap belongs to positive samples, and other information belongs to negative samples. At this time, the number of gaps is much smaller than that of other information, so as to prevent background pixels from playing a leading role in the loss process. The Focal Loss function weighs the positive samples and the negative samples by applying different weights. The CBAM module is the combination of channel attention and spatial attention. Through the attention mechanism module, the model pays more attention to the edge detail information of the gap.

[0030] The present application has the following advantages over the prior art:

[0031] (1) The method has strong practicability and can be applied to large-scale gap recognition of high-resolution remote sensing images, thereby providing a technical method for post-gap dynamic change monitoring and forest system monitoring.

[0032] (2) The method uses an object-oriented method to prepare a data set for training a deep learning model, which can save a lot of manpower and time, and solve the problem of gap misclassification caused by different subjective experiences of researchers.

[0033] (3) This method takes into account the difficulty of conducting field surveys of forest gaps in plateau areas and has generalizability for identifying forest gaps in Yunnan, Guizhou and Sichuan regions where Yunnan pine is the dominant tree species. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0035] Figure 2 This invention presents the segmentation effect of forest windows based on object-oriented methods and the overall classification result diagram.

[0036] Figure 3 This is a comparison chart of the forest window extraction performance of the PSPNet model and the DeepLabv3+ model based on this invention;

[0037] Figure 4 This is a comparison chart of the forest window extraction performance between the improved PSPNet model and the unimproved model of this invention.

[0038] Figure 5 This is a diagram showing the forest gap extraction results of the improved PSPNet model in this invention for the study area. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] Example

[0041] like Figure 1 As shown, the forest window extraction method for GF-2 images based on object-oriented and deep learning includes the following steps:

[0042] (1) Data preprocessing: Radiometric calibration, geometric accuracy correction and atmospheric correction are performed on the data.

[0043] Radiometric calibration, geometric accuracy correction, and atmospheric correction were performed on the data using ENVI software. To obtain an image of the study area, different images needed to be mosaicked. ENVI's image mosaicking tool (Seamless Mosaic) was used to stitch adjacent images together to obtain the image of the study area, and the cropping tool was used to extract the required areas.

[0044] (2) Establish an object-oriented classification system based on a multi-level classification system.

[0045] (2.1) Determine the optimal segmentation scale parameters: Use the ESP 2 plugin of eCognition software in combination with the RMAS value method to determine the optimal segmentation scale parameters for different land types in the study area and establish a multi-level classification system.

[0046] Formulas (1), (2), and (3) were selected as the formulas for calculating RMAS values ​​and combined with the ESP 2 tool of the eCognition software to calculate the optimal segmentation scale for various land features.

[0047]

[0048]

[0049] Where L represents the number of band layers in the segmented image, and ΔC L S represents the absolute value of the neighborhood difference mean of an L-band object under a single segmentation scale. L It represents the standard deviation of an object in the L-band under a single segmentation scale.

[0050] (2.2) Selecting object classification features: Based on the image spectral, shape and texture features, select the best features for land cover classification.

[0051] Based on the spectral, shape, and texture characteristics of land cover in the study area, 15 characteristic parameters (as shown in Table 1) were finally selected to classify the land cover in the study area.

[0052] Table 1. Object Classification Characteristics

[0053]

[0054] (2.3) Establishing a fuzzy classification rule set: Based on the selected spectral, shape, and texture features, a fuzzy classification rule set is established according to the characteristics of various landforms in the image to complete the forest gap identification work. For example... Figure 2 The images shown are the forest window segmentation effect diagram and the overall land cover classification effect diagram, respectively.

[0055] (3) Constructing a forest window sample dataset: After manually correcting the forest window results identified by the object-oriented method, the results are converted into an index map. The sliding window is then cut into 256*256 pixel forest window samples to ensure the integrity of the contextual semantic information.

[0056] The original image size is large, and directly inputting it into the network model may result in long training times, poor performance, or even memory overflow. Therefore, it needs to be segmented pixel-level according to the standard dataset format. To ensure the integrity of the forest window and minimize the inclusion of background information, a 256*256 pixel segmentation was ultimately chosen. The sliding window method, based on a set window and step size, slides and crops from left to right and from top to bottom. Its difference from the rule-based method is that the two windows can overlap, effectively compensating for the loss of contextual information caused by cropping.

[0057] (4) Sample data augmentation: Since there is no publicly available dataset for forest window sample data, it is necessary to create it ourselves. Furthermore, forest windows are relatively rare in forest systems. To prevent overfitting during model training, it is necessary to increase the amount of dataset.

[0058] Since the number of forest gaps in a forest system is limited and their shape and form do not follow a fixed pattern, the OpenCV alginates library is used to perform operations such as flipping, translating, sharpening, and brightness changes on existing forest gap datasets. Increasing the amount of data can effectively prevent overfitting during model training.

[0059] (5) Training PSPNet and DeepLabv3+ models to identify forest windows: Training semantic segmentation models using self-made sample datasets, and comparing models in terms of forest window extraction accuracy and model performance;

[0060] To compare the extraction performance of different semantic segmentation models for forest windows, the mainstream PSPNet model and DeepLabv3+ model with good extraction performance were selected for the study. The model was trained using the forest window sample dataset prepared in steps (3) and (4) above. The average pixel precision (mPA), average intersection-over-union ratio (mIoU), and speed (FPS) were selected to evaluate the performance and extraction accuracy of the model. The average pixel precision was calculated according to formula (4), and the average intersection-over-union ratio was calculated according to formula (5).

[0061]

[0062] In the formula, k represents the number of categories to be divided, i represents the actual value of the category, j represents the predicted value of the category, and p ij This represents the number of pixels that were originally classified as class i but were predicted as class j. Higher values ​​for mIoU and mPA indicate better extraction of forest window information, while higher FPS values ​​mean the model processes more images per second, resulting in higher efficiency. Figure 3As shown in the figure, a comparison of the forest window extraction performance based on the PSPNet model and the DeepLabv3+ model is presented. mPA and mIoU are used as accuracy evaluation parameters, and FPS is used to test the computational efficiency of the models. Experimental results show that the difference in extraction accuracy between the two models is less than 1.72%, but the PSPNet model is faster and more efficient than the DeepLabv3+ model. The former achieves mPA, mIoU, and detection speed FPS of 89.10%, 81.98%, and 50.83, respectively, while the latter achieves accuracy of 90.81%, 82.57%, and detection speed of 31.65, respectively. Furthermore, visual comparison of the forest window recognition performance of the two models reveals that the PSPNet model's detected forest window boundaries more closely match the actual ground distribution of forest windows, demonstrating more accurate edge information detection. Therefore, the PSPNet model is selected for further improvement research.

[0063] (6) Identify forest windows based on the improved PSPNet model.

[0064] (6.1) Backbone network optimization MobileNetV2: To improve the model computation speed, the original ResNet50 network with a large number of parameters was replaced with the lightweight MobileNetV2 network.

[0065] ResNet50 is used as the backbone network of the PSPNet model. However, due to the large amount of computation and parameters of this feature extraction network, it will affect the model's computation speed and efficiency. MobileNetV2 is used as the backbone network to extract image features and improve the model's efficiency.

[0066] (6.2) Improved loss function: In order to prevent the imbalance in the number of positive samples in the forest window and negative samples in the background information during the model training process, the Focal Loss function is introduced to balance the positive and negative samples by applying different weights.

[0067] When extracting forest window information based on remote sensing images, forest windows are considered positive samples, while the rest are background information and are considered negative samples. In this case, the number of forest windows will be much smaller than the number of other information to prevent background pixels from playing a dominant role in the loss process. The Focal Loss function balances the positive and negative samples by applying different weights.

[0068] (6.3) CBAM attention mechanism module: First apply channel attention mechanism to feature map, then apply spatial attention mechanism to make the model notice the details of forest window edge.

[0069] The CBAM module combines channel attention and spatial attention, and through the attention mechanism module, the model pays more attention to the details of the forest window edges during extraction.

[0070] (6.4) Model Result Optimization and Improvement: The improved PSPNet model underwent four upsampling operations, connecting deep and shallow information four times to maximize the utilization of shallow information and improve classification accuracy. For example... Figure 4 As shown in the comparison chart, the improved PSPNet model and the unimproved model perform better in forest window extraction. The improved PSPNet model shows significantly better results, with improved prediction speed and accuracy compared to the other two models. It also notices the edge details of forest windows, resulting in extractions that better reflect the actual geographic conditions. The improved PSPNet model's prediction speed is 166% and 65% faster than the other two models, respectively; its mPA is 1.89% faster than the DeepLabv3+ model and 3.6% faster than the traditional PSPNet model; and its mIou is 2.53% and 3.12% faster, respectively. This demonstrates the value of the improved PSPNet model in extracting forest windows. Figure 5 The image shown is a result of the improved PSPNet model identifying forest gaps in the study area.

[0071] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A forest window extraction method for GF-2 images based on object-oriented and deep learning, characterized in that, Includes the following steps: (1) Data preprocessing: Radiometric calibration, geometric accuracy correction and atmospheric correction of the data; (2) Establish an object-oriented classification system based on a multi-level classification system; (2.1) Determine the optimal segmentation scale parameters: Use the ESP 2 plugin of eCognition software in combination with the RMAS value method to determine the optimal segmentation scale parameters for different land types in the study area and establish a multi-level classification system; (2.2) Selecting object classification features: Based on the image spectral, shape, and texture features, select the best features for land cover classification; (2.3) Establish a fuzzy classification rule set: Based on the selected spectral, shape, and texture features, establish a fuzzy classification rule set according to the characteristics of various landforms in the image to complete the forest gap identification work; (3) Construct a forest window sample dataset: After manually correcting the forest window results identified by the object-oriented method, convert them into an index map, and cut the sliding window into 256*256 pixel forest window samples to ensure the integrity of the contextual semantic information; (4) Sample data augmentation: Since there is no publicly available dataset for forest window sample data, it is necessary to create it ourselves. Furthermore, forest windows are relatively rare in forest systems. To prevent overfitting during model training, it is necessary to increase the amount of dataset. (5) Training PSPNet and DeepLabv3+ models to identify forest windows: Training semantic segmentation models using self-made sample datasets, and comparing models in terms of forest window extraction accuracy and model performance; (6) Identifying forest windows based on the improved PSPNet model; (6.1) Backbone network optimization MobileNetV2: To improve the model computation speed, the original ResNet50 network with a large number of parameters was replaced with the lightweight MobileNetV2 network; (6.2) Improved loss function: In order to prevent the imbalance in the number of positive samples in the forest window and negative samples in the background information during the model training process, the Focal Loss function is introduced to balance the positive and negative samples by applying different weights. (6.3) CBAM attention mechanism module: First, apply the channel attention mechanism to the feature map, and then apply the spatial attention mechanism to make the model notice the details of the forest window edge; (6.4) Model result optimization and improvement: The improved PSPNet model is upsampled 4 times, and the deep information and shallow information are connected 4 times to maximize the use of shallow information and improve classification accuracy.

2. The forest window extraction method for GF-2 images based on object-oriented and deep learning according to claim 1, characterized in that, The preprocessing in step (1) includes projection, cropping, radiometric calibration, geometric accuracy correction and atmospheric correction.

3. The forest window extraction method for GF-2 images based on object-oriented and deep learning according to claim 1, characterized in that, In step (2.1), formulas (1), (2), and (3) are selected as RMAS value calculation formulas and combined with the ESP 2 tool of eCognition software to calculate the optimal segmentation scale of various land features. Where L represents the number of band layers in the image being segmented, and ΔG L S represents the absolute value of the neighborhood difference mean of an L-band object under a single segmentation scale. L This represents the standard deviation of the object in the L-band under a single segmentation scale; Step (2.2) statistically analyzed the image spectral, shape, and texture features of different land cover types, and selected appropriate features for land cover classification. Step (2.3) constructs a fuzzy classification rule set based on the features selected in step (2.2) to complete the forest window identification work.

4. The forest window extraction method for GF-2 images based on object-oriented and deep learning according to claim 1, characterized in that, Step (3) constructs a forest window sample dataset, evaluates the accuracy of the forest window classification results obtained in step (2), and then combines manual correction to convert the results into an index map. Considering that the forest window information should be complete in the training samples, and the label dataset should contain as little background information as possible, finally, 256*256 pixels are selected to perform sliding window segmentation on the image to obtain the forest window sample dataset used for training the deep learning model.

5. The forest window extraction method for GF-2 images based on object-oriented and deep learning according to claim 1, characterized in that, In step (4), since the number of forest windows in the forest system is limited and there is no publicly available dataset of forest window samples, the dataset for model training needs to be created by ourselves and is limited in number. In order to prevent overfitting during model training, the forest window dataset is augmented. The OpenCV-based alumniations library can augment the dataset by performing operations such as flipping, translating, sharpening, and brightness changes.

6. The forest window extraction method for GF-2 images based on object-oriented and deep learning according to claim 1, characterized in that, In step (5), the PSPNet model and the DeepLabv3+ model were trained, and the performance of the two models and the accuracy of forest window recognition were compared. The average pixel accuracy, average intersection-union ratio and speed were selected.

7. The forest window extraction method for GF-2 images based on object-oriented and deep learning according to claim 1, characterized in that, In step (6), based on step (5), the recognition effect and model performance of the two models on forest windows were compared. The PSPNet model was improved by comprehensively considering the following: First, ResNet50 was used as the backbone network of the model. Since the computational amount and parameter amount of this feature extraction network are very large, it will affect the model's computational speed and efficiency. MobileNetV2 was used as the backbone network to extract image features and improve the model efficiency. When extracting forest window information based on remote sensing images, forest windows are positive samples, and the others are background information, which are negative samples. At this time, the number of forest windows will be much smaller than the number of other information to prevent background pixels from playing a dominant role in the loss process. The Focal Loss function weighs positive and negative samples by applying different weights. The CBAM module is a combination of channel attention and spatial attention. The attention mechanism module makes the model pay more attention to the details of the forest window edges when extracting. The PSPNet structure was optimized by performing four upsampling operations and connecting the deep information and the shallow information four times to maximize the utilization of shallow edge, shape and other information.

Images