Coastal wetland information fine extraction method based on deep learning
By introducing parallel expansion convolutional branch and efficient channel attention mechanism in DeepLab V3+ model, the coastal wetland information fine extraction model M&E-DeepLab is constructed, which solves the problems of low efficiency and insufficient accuracy of wetland information extraction, and achieves more efficient and Accurate wetland information extraction, suitable for coastal wetland environments.
Patent Information
- Application Number
- CN202410876959.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-11-12
AI Technical Summary
The existing wetland information extraction methods have problems such as low efficiency, great influence on human subjectiveness, and insufficient classification precision. Especially in coastal wetland environments, it is difficult to achieve fine wetland information extraction.
The fine extraction method of coastal wetland information based on deep learning is used to improve the model by processing the expanded convolutional branch in the hollow pyramid pooling module in parallel, combined with the efficient channel attention mechanism (ECA attention mechanism) Accuracy and robustness, a fine extraction model of coastal wetland information M&E-DeepLab was constructed.
It improves the accuracy and efficiency of wetland information extraction, reduces the phenomenon of missession, enhances the robustness of the model, can more accurately identify and segment different types of coastal wetlands, and provides scientific ecological management and protection decision support.
Smart Images

Figure CN118941939A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of wetland ecological remote sensing, and specifically relates to a method for fine extraction of coastal wetland information based on deep learning. Background Art
[0002] Wetlands, forests and oceans are known as the three major ecosystems on Earth and are known as the "Kidneys of the Earth". As one of the most productive ecosystems in the world, wetlands have multiple ecological functions such as conserving water, regulating climate, maintaining biodiversity, and "fixing carbon". They are also the largest natural source of methane (CH4). In recent years, affected by human and natural activities, my country's coastal wetlands are facing serious fragmentation and salt marsh loss. These problems not only threaten the ecological balance and biodiversity of the wetlands themselves, but also affect the ecological service functions provided by the wetlands. Accurate extraction of wetland information can not only more accurately assess the current status of wetlands and monitor ecosystem health and functional changes, but also provide scientific basis and decision-making support for wetland management, protection and restoration.
[0003] Traditional wetland information extraction methods such as field surveys and visual interpretation, unsupervised classification, and supervised classification have some limitations in application, such as low efficiency and accuracy greatly affected by human subjectivity. In recent years, with the development of deep learning and neural networks, convolutional neural networks (CNN), artificial neural networks (ANN), and fully convolutional neural networks (FCN) have been widely used in the research of wetland information extraction. Referring to previous research results, CNN was used to extract herbaceous wetland information and compared with the SVM method. The results showed that the classification accuracy of CNN was improved by about 5% compared with SVM. DeepLab V3+ refines the segmentation results by adding a decoder structure to achieve the fusion of shallow and deep features, and has achieved good segmentation results in PASCAL VOC2012, GID, CCF and other data sets. In addition, studies have shown that adding CBAM attention mechanism after deep and shallow feature fusion in the residual branch and decoder of Xception-65 can suppress the problems of too smooth boundary areas and unreasonable islands in the segmentation results of DeepLab V3+ network. However, DeepLab V3+ still has problems such as slow training speed and inability to meet the requirements of fine object segmentation.
[0004] In order to achieve the fine extraction of coastal wetland information, a fine extraction method of coastal wetland information based on deep learning is urgently needed. Summary of the invention
[0005] In view of the "salt and pepper phenomenon", low classification precision and classification efficiency in existing wetland information extraction methods, the present invention proposes a method for fine extraction of coastal wetland information based on deep learning. The coastal wetland information extraction model is based on the DeepLabV3+ model, and performs layer-by-layer feature transfer and feature fusion on the three dilated convolution branches in the hollow pyramid pooling module. At the same time, an efficient channel attention mechanism is introduced to improve the accuracy and robustness of the model without increasing the complexity of the model.
[0006] The present invention is implemented by the following technical scheme: A method for fine extraction of coastal wetland information based on deep learning, comprising the following steps:
[0007] Step A, construction of wetland classification system: referring to the land use classification system of China's land use status remote sensing monitoring database and "Wetland Classification (GB / T 24708-2009)", the data are divided into 11 categories according to the actual situation: canals / ditches, rivers, Suaeda salsa wetlands, reed wetlands, mixed wetlands, tidal flats, aquaculture ponds, ponds / reservoirs, paddy fields, dry fields and buildings (structures). Among them, mixed wetlands refer to wetland types with a high degree of interweaving of reeds, Suaeda salsa, tidal flats, canals and ditches, etc., which are difficult to distinguish due to the influence of image resolution.
[0008] Step B, remote sensing image preprocessing and deep learning sample dataset construction: obtain the original L1C-level images of the coastal wetland area from Sentinel-2, process them using Sen2Cor software (version number: 2.11) to obtain L2A-level images, combine the remote sensing images with the field survey results, and perform pixel-level annotation on the remote sensing images. Crop the L2A-level images and pixel-level annotation results.
[0009] Step C: construct a deep learning-based coastal wetland information extraction model M&E-DeepLab, and train the model using the remote sensing images and annotation results (sample data set) obtained in step B.
[0010] The coastal wetland information fine extraction model is an "encoder-decoder" structure, with the DeepLab V3+ model as the basic model. In the encoder structure, MobileNet V2 is used as the backbone feature extraction network, and the three dilated convolution branches in the hole pooling pyramid module are processed in parallel. After all branch feature maps are stacked, an efficient channel attention mechanism (ECA attention mechanism) is added to obtain deep features and pass them into the decoder structure. In the decoder structure, the shallow features and deep features obtained by the backbone feature extraction network are stacked. The encoder structure specifically includes two contents:
[0011] (1) Input the original image into the MobileNet V2 backbone feature extraction module for shallow feature extraction. The specific steps are as follows:
[0012] First, point-wise convolution is used to achieve dimensionality increase of the input feature layer. Secondly, depth wise convolution (DW) is used to extract features from the feature layer after dimensionality increase. Finally, the extracted features are convolved point-by-point again to achieve feature dimensionality reduction. MobileNet V2 uses the ReLU6 activation function with a maximum output value of 6 instead of ReLU, which avoids the numerical explosion problem caused by the ReLU activation function without limiting the output value, and can better improve the computing power and training effect of the neural network. The expression of the ReLU6 activation function is as follows:
[0013] ReLU6(x)=min(max(0,x),6) (1)
[0014] However, the convolution kernel dimension after DW processing is small, and using the ReLU6 activation function will make the neuron output 0, which is prone to gradient disappearance. MobileNet V2 uses a linear bottleneck structure to solve the gradient disappearance problem.
[0015] (2) Input shallow features into the dilated pyramid module to extract deep features. The specific steps are as follows:
[0016] In order to ensure that the number of parameters remains unchanged, expand the receptive field (RF) of the convolution kernel, make full use of global information and maintain the size of the feature map unchanged, the ASPP module (Atrous Spatial Pyramid Pooling) uses three parallel convolution branches with a convolution kernel size of 3, and the expansion rates are 6, 12, and 18 respectively. After using the dilated convolution, the actual receptive fields of the three branches are 13, 25, and 37 respectively. The actual receptive field calculation formula is as follows:
[0017] RF=k+(k-1)×(d-1) (2)
[0018] Where: RF represents the actual receptive field, k represents the convolution kernel size, and d represents the dilation rate.
[0019] In order to improve the efficiency of global information utilization, realize information sharing, and increase the feature relevance of the three dilated convolution branches, the present invention transfers the feature maps of the three parallel dilated convolution branches in the ASPP module layer by layer. The specific contents are as follows:
[0020] (a) Branch 1: The shallow features obtained by the backbone feature extraction module are input into a 1×1 convolution to perform channel compression to obtain features Figure 1 ;
[0021] (b) Branch 2: The shallow features obtained by the backbone feature extraction module are processed by a 3×3 convolution with a dilation rate of 6 to obtain the features Figure 2 ;
[0022] (c) Branch 3: Characteristics Figure 2 The input is processed by a 3×3 convolution with a dilation rate of 12 to obtain the features Figure 3 ;
[0023] (d) Branch 4: Characteristics Figure 3 The input is processed by 3×3 convolution with a dilation rate of 18 to obtain the features Figure 4 ;
[0024] (e) Branch 5: The shallow features obtained by the backbone feature extraction module are processed by global average pooling to obtain features Figure 5 ;
[0025] (f) The features obtained from the five branches Figures 1 to 5 To merge.
[0026] (g) Input the feature fusion result into the ECA attention mechanism module. The ECA attention mechanism module specifically includes the following three steps:
[0027] Perform global average pooling on the input feature map, transforming the feature map from a [H, W, C] matrix to a [1, 1, C] vector.
[0028] The convolution kernel size is calculated by the adaptive function and applied to the one-dimensional convolution to obtain the channel-by-channel weight of the feature map. The calculation formula is as follows:
[0029]
[0030] Where C represents the number of input feature map channels, b=1, and γ=2.
[0031] (h) The convolutional layer with a kernel size of 1 is then used to obtain deep features.
[0032] The decoder structure specifically includes the following steps:
[0033] (1) The shallow features obtained by the backbone feature extraction module are input into a 1×1 convolution to compress the number of channels;
[0034] (2) Upsampling the deep features obtained by the encoder structure and fusing them with the feature map compressed by the number of channels in the previous step (1);
[0035] (3) The feature fusion result is input into the convolution with a convolution kernel size of 3, and the number of channels is further adjusted;
[0036] (4) Upsample the result in (3) to obtain the final refined extraction result of coastal wetland information.
[0037] In the model, the loss function is the cross entropy loss function (CEL), the optimizer is Adam (Adaptive Gradient Algorithm), the maximum learning rate is 0.007, the gradient descent function is cos, the downsampling multiple is 8, and the training round Epoch is 40.
[0038] Step D, based on the trained coastal wetland information fine extraction model, realize the fine extraction of coastal wetland information.
[0039] The coastal wetland information fine extraction model quantitatively evaluates the model effect through four objective indicators: overall accuracy, Kappa, user accuracy, and producer accuracy.
[0040] The present invention also discloses a coastal wetland information fine extraction system based on deep learning, comprising:
[0041] The preprocessing module is used to preprocess the remote sensing images reflecting the coastal wetland area and annotate the preprocessing results based on the wetland classification system to obtain a sample data set;
[0042] The training module is used to: construct a coastal wetland information extraction model based on deep learning and train the model using a sample data set;
[0043] The fine extraction module is used to: extract the fine coastal wetland information from the input remote sensing image based on the trained fine coastal wetland information extraction model;
[0044] Wherein, the coastal wetland information fine extraction model includes:
[0045] The encoder structure is used to: input the remote sensing image into the MobileNet V2 backbone feature extraction module for shallow feature extraction; input the shallow features into the hollow pooling pyramid module to extract deep features;
[0046] The decoder structure is used to: input shallow features into 1×1 convolution for channel compression; upsample deep features and fuse them with feature maps obtained by channel compression; input the feature fusion results into convolution with a convolution kernel size of 3, further adjust the number of channels, upsample the adjusted results, and output the refined extraction results of coastal wetland information.
[0047] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for fine extraction of coastal wetland information based on deep learning is implemented.
[0048] The present invention also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for fine extraction of coastal wetland information based on deep learning.
[0049] Compared with the existing coastal wetland information extraction technology, the model proposed in this invention has the following advantages:
[0050] (1) The backbone feature extraction network replaced Xception with MobileNet V2, which avoided the problem that the advantages of the deep network structure could not be expressed due to the small internal differences among different types of wetlands. In addition, the model accuracy was improved while reducing the number of network parameters to achieve model lightweight;
[0051] (2) The three parallel dilated convolutions in the ASPP module perform layer-by-layer feature transfer processing, which can improve the efficiency of global information utilization, realize information sharing, and improve the feature correlation of the three dilated convolution branches. The addition of the ECA module improves the model accuracy without increasing the model complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0053] Figure 1 This is a flow chart of the method for fine extraction of coastal wetland information based on deep learning described in an embodiment of the present invention.
[0054] Figure 2 This is a schematic diagram of the location of the study area selected for the embodiment of the present invention.
[0055] Figure 3 The remote sensing image preprocessing result and the labeling result described in the embodiment of the present invention, wherein (a) is the remote sensing image preprocessing result, and (b) is the labeling result.
[0056] Figure 4 It is a partial cropping result of the original image and the annotation result described in the embodiment of the present invention, wherein (a) is the cropping result of the remote sensing image, and (b) is the cropping result of the annotation.
[0057] Figure 5 Schematic diagram of the network structure of the method for fine extraction of coastal wetland information based on deep learning described in an embodiment of the present invention.
[0058] Figure 6 Schematic diagram of the structure of the MobileNet V2 module described in an embodiment of the present invention, wherein (a) is an inverse residual module and (b) is a network block structure diagram.
[0059] Figure 7 Schematic diagram of the dilated convolution described in an embodiment of the present invention, wherein (a) is a dilation rate of 1, (b) is a dilation rate of 2, and (c) is a dilation rate of 4.
[0060] Figure 8 It is a schematic diagram of the structure of the improved ASPP module according to an embodiment of the present invention.
[0061] Fig. 9 Schematic diagram of the structure of the ECA attention mechanism described in an embodiment of the present invention.
[0062] Fig.10 This is a schematic diagram of the test results of the deep learning network model parameters described in an embodiment of the present invention, wherein (a) is the impact of different parameter settings on OA and Kappa, (b) is the impact of different parameter settings on Loss, and (c) is the impact of different parameter settings on Val_Loss.
[0063] Fig.11 The detailed extraction results of coastal wetland information of the comparative test described in the embodiment of the present invention, wherein (a) is the label; (b) is the extraction result of DeepLab V3+; (c) is the extraction result of the model described in the present invention; (d) is the extraction result of ERFNet; (e) is the middle-layer feature fusion extraction result; (f) is the FCN extraction result; (g) is a local enlarged view of the I frame label and the extraction results of different models in (a) to (f); (h) is a local enlarged view of the II frame label and the extraction results of different models in (a) to (f); (i) is a local enlarged view of the III frame label and the extraction results of different models in (a) to (f). DETAILED DESCRIPTION
[0064] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.
[0065] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0066] Embodiment 1
[0067] like Figure 1 As shown, this embodiment provides a method for fine extraction of coastal wetland information based on deep learning, including:
[0068] Combining remote sensing images and field survey results to construct a coastal wetland classification system:
[0069] Acquire remote sensing images of coastal wetlands, wherein the remote sensing images refer to high-resolution multispectral images provided by the Sentinel-2 satellite;
[0070] Preprocessing the coastal wetland remote sensing image to obtain a preprocessed remote sensing image;
[0071] Based on deep learning algorithms, a model for fine extraction of coastal wetland information is constructed;
[0072] Based on the coastal wetland fine extraction model, the wetland information is finely extracted from the pre-processed remote sensing image to obtain a coastal wetland fine extraction result.
[0073] In this example, the overview of the study area is as follows:
[0074] Panjin Liaohe Wetland is located along the Liaohe River and the estuary area in Panjin City, Liaoning Province. The latitude and longitude range is approximately 40°44′10″N~41°07′29″N, 121°29'57E"~122°00'40"E. It is a sedimentary plain in the lower reaches of many rivers such as Raoyang River and Liaohe River. It belongs to the estuary bay wetland. It is a wildlife and wetland type nature reserve mainly for protecting a variety of rare waterfowl such as red-crowned cranes and black-headed gulls and coastal wetland ecosystems. The location of the study area and the original image are as follows Figure 2 shown.
[0075] In this embodiment, the satellite image comes from Sentinel-2. The data is provided free of charge by the European Space Agency (ESA). The L1C-level product was acquired on October 8, 2021, with a spatial resolution of 10m. The Sentinel-2 satellite is equipped with a multi-spectral imager (MSI), with two constellations, Sentinel-2A and Sentinel-2B, covering a spectral range of 0.43-2.28μm, a total of 13 spectral bands, and a revisit period of 5 days. The spatial resolution and spectral range of each band are shown in Table 1.
[0076] Table 1
[0077]
[0078]
[0079] Note: * represents the wavelength used in this example.
[0080] In this embodiment, constructing a wetland classification system specifically includes:
[0081] With reference to the land use classification system of China's land use status remote sensing monitoring database and the "Wetland Classification (GB / T24708-2009)", the data were divided into 11 categories in combination with actual conditions: canals / ditches, rivers, winged saltwort wetlands, reed wetlands, mixed wetlands, tidal flats, aquaculture ponds, ponds / reservoirs, paddy fields, dry fields and buildings (structures). Mixed wetlands refer to wetland types with a high degree of interweaving of reeds, saltwort, tidal flats, canals and ditches, etc., which are difficult to distinguish due to the influence of image resolution.
[0082] In this embodiment, the data preprocessing process is as follows:
[0083] 1. The original L1C-level images of Sentinel-2 in the study area were processed by Sen2Cor software (version number: 2.11) to obtain L2A-level images. Figure 3 Combined with the previously constructed wetland classification system, QGIS software (version number: 3.28.14) was used to annotate the remote sensing images at the pixel level. The annotation results are shown in Figure 3 (b) as shown.
[0084] 2. Since the original image size is too large to be directly input into the neural network for training, the original image and annotation results are processed by sliding cropping, cut into 128×128 blocks, and the data set is expanded by flipping and transposing, and a total of 2988 blocks are obtained. Some cropping results are shown below: Figure 4 shown.
[0085] In this embodiment, the construction of a coastal wetland information fine extraction model based on a deep learning algorithm includes: constructing a coastal wetland information fine extraction model M&E-DeepLab based on deep learning, and training the model through the remote sensing images and annotation results obtained in step B.
[0086] The coastal wetland information fine extraction model is an "encoder-decoder" structure, with the DeepLab V3+ model as the basic model. The coastal wetland information fine extraction model structure is as follows Figure 5 As shown in the figure. In the encoder structure, MobileNet V2 is used as the backbone feature extraction network. The three dilated convolution branches in the dilated pooling pyramid module are processed in parallel, and the efficient channel attention mechanism (ECA attention mechanism) is added after all branch feature maps are stacked to obtain deep features and pass them into the decoder structure. In the decoder structure, the shallow features and deep features obtained by the backbone feature extraction network are stacked. The encoder structure specifically includes two contents:
[0087] (1) Input the original image into the MobileNet V2 backbone feature extraction module for shallow feature extraction. The specific steps are as follows:
[0088] First, point-wise convolution is used to increase the dimension of the input feature layer. Second, depth wise convolution (DW) is used to extract features from the feature layer after the dimension increase. Finally, the extracted features are convolved again point-wise to reduce the dimension of the features. The structure of MobileNet V2 is as follows: Figure 6 As shown. MobileNet V2 uses the ReLU6 activation function with a maximum output value of 6 instead of ReLU, which avoids the numerical explosion problem caused by the ReLU activation function without limiting the output value, and can better improve the computing power and training effect of the neural network. The expression of the ReLU6 activation function is as follows:
[0089] ReLU6(x)=min(max(0,x),6) (1)
[0090] However, the convolution kernel dimension after DW processing is small, and using the ReLU6 activation function will make the neuron output 0, which is prone to gradient disappearance. MobileNet V2 uses a linear bottleneck structure to solve the gradient disappearance problem.
[0091] (2) Input shallow features into the dilated pyramid module to extract deep features. The specific steps are as follows:
[0092] In order to ensure that the number of parameters remains unchanged, expand the receptive field (RF) of the convolution kernel, make full use of global information and maintain the size of the feature map unchanged, three parallel dilated convolution branches with a convolution kernel size of 3 are used in the ASPP module, and the dilation rates are 6, 12, and 18 respectively. After using the dilated convolution, the actual receptive fields of the three branches are 13, 25, and 37 respectively. The schematic diagram of the dilated convolution is shown in Figure 7 As shown, the actual receptive field calculation formula is as follows:
[0093] RF=k+(k-1)×(d-1) (2)
[0094] Where: RF represents the actual receptive field, k represents the convolution kernel size, and d represents the dilation rate.
[0095] In order to improve the efficiency of global information utilization, realize information sharing, and increase the feature relevance of the three dilated convolution branches, the present invention transfers the feature maps of the three parallel dilated convolution branches in the ASPP module layer by layer. The improved ASPP module structure is as follows: Figure 8 As shown, specifically including:
[0096] (a) Branch 1: The shallow features obtained by the backbone feature extraction module are input into a 1×1 convolution to perform channel compression to obtain features Figure 1 ;
[0097] (b) Branch 2: The shallow features obtained by the backbone feature extraction module are processed by a 3×3 convolution with a dilation rate of 6 to obtain the features Figure 2 ;
[0098] (c) Branch 3: Characteristics Figure 2 The input is processed by a 3×3 convolution with a dilation rate of 12 to obtain the features Figure 3 ;
[0099] (d) Branch 4: Characteristics Figure 3 The input is processed by 3×3 convolution with a dilation rate of 18 to obtain the features Figure 4 ;
[0100] (e) Branch 5: The shallow features obtained by the backbone feature extraction module are processed by global average pooling to obtain features Figure 5 ;
[0101] (f) The features obtained from the above five branches Figure 1 To Features Figure 5 To merge.
[0102] (g) After the fusion is completed, the feature fusion result is input into the ECA attention mechanism. Fig. 9 As shown in Figure 1, the ECA attention mechanism specifically includes:
[0103] Perform global average pooling on the input feature map, transforming the feature map from a [H, W, C] matrix to a [1, 1, C] vector.
[0104] The convolution kernel size is calculated by the adaptive function and applied to the one-dimensional convolution to obtain the channel-by-channel weight of the feature map. The calculation formula is as follows:
[0105]
[0106] Where C represents the number of input feature map channels, b=1, and γ=2.
[0107] (h) The convolutional layer with a kernel size of 1 is then used to obtain deep features.
[0108] In this embodiment, the decoder structure specifically includes two steps:
[0109] (1) The shallow features obtained by the backbone feature extraction module are input into a 1×1 convolution to compress the number of channels;
[0110] (2) Upsampling the deep features obtained by the encoder structure and fusing them with the feature map compressed by the number of channels in the previous step (1);
[0111] (3) The feature fusion result is input into the convolution with a convolution kernel size of 3, and the number of channels is further adjusted;
[0112] (4) Upsample the result in the previous step (3) to obtain the final result of the fine extraction of coastal wetland information.
[0113] This embodiment uses PyTorch as the development framework, the CUDA version is 11.7, the GPU is RTX 3080, the video memory is 16G, and the computer memory is 32G. The data is randomly divided according to a ratio of 9:1. The training data set has a total of 2689 pictures, and the verification data set has a total of 299 pictures. The accuracy and efficiency of the coastal wetland information fine extraction model proposed in the present invention are affected by factors such as the downsampling multiple, the backbone network structure, and the optimizer type. The basis for determining the three parameters is mainly based on the performance of the network on the data set (Overall Accuracy (OA), Kappa, Loss, Val_Loss). This embodiment selects MobileNet V2 and Xception as the backbone networks for experimental comparison. The downsampling multiple is selected between 8 and 16. The optimizer type is selected between SGD (Stochastic Gradient Descent) and Adam (Adaptive Gradient Algorithm). The experimental results are as follows Fig.10 shown.
[0114] See also Fig.10 In (a), M represents the use of MobileNet V2 as the backbone feature extraction network, X represents the use of Xception as the backbone feature extraction network, 8 and 16 represent different downsampling multiples, S represents the use of SGD optimizer, and A represents the use of Adam as the optimizer. Fig.10 From (a), we can see that the training effect is best when the backbone network is MobileNet V2, the downsampling factor is 8, and the optimizer is Adam. Fig.10 (b) Fig.10 (c) When epoch=40, both Loss and Val_Loss tend to be stable. The batch size is set to the maximum value within the acceptable range of the graphics card. Therefore, in the model, the backbone feature extraction network is MobileNet V2, the batch size is 8, the optimizer is Adam (Adaptive Gradient Algorithm), the maximum learning rate is 0.007, the gradient descent function is cos, the downsampling multiple is 8, the training round Epoch is 40, and the loss function is the cross entropy loss function (Cross Entropy Loss, CEL). The calculation formula is as follows:
[0115]
[0116] Where: M represents the total number of classification categories; y ic When the predicted feature type is the same as the annotated feature type, it takes 1, otherwise it takes 0; p ic represents the probability that feature i is predicted to be feature c; N represents the number of samples, i∈(1,N).
[0117] In order to verify the effectiveness of the model, this embodiment also discloses quantitative evaluation indicators for fine extraction of coastal wetland information.
[0118] This embodiment uses a confusion matrix to classify and count the validation data set, and uses four objective evaluation indicators: Overall Accuracy (OA), Producers Accuracy (PA), Users Accuracy (UA), and Kappa coefficient. The calculation formulas for the four objective evaluation indicators are as follows:
[0119]
[0120] Among them, in formulas (5) to (8), x ii represents the number of samples correctly classified into the i-th category. In formula (5) and (8) represents the total number of samples, and n is the number of categories. In formula (6) represents the number of samples of the i-th category in the true category. In formula (7) Represents the number of samples of the i-th category in the prediction results.
[0121] Embodiment 2
[0122] Comparative test
[0123] In order to verify the effectiveness of the model obtained in Example 1, it is compared with DeepLabV3+, FCN, ERFNet and the middle-level feature fusion method. The results are as follows: Fig.11 shown.
[0124] Depend on Fig.11(a)-(f) show that, overall, the various types of features extracted by DeepLab V3+ are fragmented and have serious misclassification phenomena, and the extraction effect for reservoirs is poor. There are phenomena such as interruption of flow and misclassification for small water bodies. The various types of features proposed by FCN have good integrity, but there are a large number of omissions for small water bodies such as canals / ditches and small buildings. The mid-level feature fusion method has improved the fragmentation of features in the original network classification results to a certain extent, but it is still insufficient in distinguishing small water bodies, aquaculture ponds and ponds / reservoirs. The various types of features extracted by ERFNet have obvious "salt and pepper phenomenon" and the "paddy field-aquaculture pond" confusion classification phenomenon is serious.
[0125] The model described in this invention has obvious improvement in the connectivity of small rivers compared with the other four methods, but there are still some misclassification and omission phenomena in the extraction of mixed wetlands, such as Fig.11 The canal / ditch connectivity and building integrity are significantly improved compared to the other four methods, such as Fig.11 However, in terms of the extraction of mudflats, the proposed method has the problem of misclassification of rivers and mudflats, which may be caused by the confusion of spectral information caused by suspended matter in the water at the estuary, such as Fig.11 (i) as shown.
[0126] The extraction results of different models were evaluated for accuracy, and the models were evaluated in terms of parameter quantity and training time. The results are shown in Table 2.
[0127] Table 2
[0128]
[0129] Note: UA and PA in the table represent the user accuracy and producer accuracy of each feature type respectively. The bold font represents the best result.
[0130] As can be seen from Table 2, the overall accuracy and Kappa coefficient of the model proposed in the present invention are better than those of other networks, indicating that the M&E-DeepLab model has the best classification ability for remote sensing images among the listed methods, and the extraction results are highly consistent with the actual distribution of objects. The overall accuracy of the FCN extraction results is 80.1% and the Kappa coefficient is 0.753, which is the lowest among the five methods listed, indicating that this method is less effective in extracting information from the Liaohe wetland. The training time of the middle-level feature fusion is the shortest, and it has significant advantages in the extraction of objects such as canals / ditches and rivers, but it has deficiencies in the extraction of other wetland types and non-wetland types. ERFNet has the smallest number of parameters, but it fails to effectively improve the training time. At the same time, due to parameter limitations, the overall accuracy and Kappa coefficient are not high. Therefore, the M&E-DeepLab proposed in the present invention can improve the extraction accuracy while reducing the number of parameters and shortening the training time.
[0131] Ablation test
[0132] In order to verify the effectiveness of improving the model efficiency and accuracy before and after the backbone feature extraction network is replaced, the ASPP module expansion convolution branches are changed from parallel to series, and the addition of ECANet, this embodiment conducted an ablation experiment, and the results are shown in Table 3.
[0133] Table 3
[0134]
[0135] Note: Bold fonts represent the best results.
[0136] As shown in Table 3, after replacing the backbone feature extraction network from Xception (Baseline 1) to MobileNet V2 (Baseline 2), the overall accuracy is improved by 8.579%, the Kappa coefficient is increased by 0.106, and the number of parameters is reduced to 10.7% of the original network. Compared with DeepLab V3+ (Baseline 1), on the basis of Baseline 2, branches 2, 3, and 4 in the ASPP module are changed from parallel to series, and the overall accuracy is improved by 8.198%, and the Kappa coefficient is increased by 0.101; after adding ECANet, the overall extraction accuracy is improved by 8.423%, and the Kappa coefficient is increased by 0.104; the expansion convolution branch is changed to series connection and ECANet is added at the same time, and the overall extraction accuracy is improved by 9.618%, and the Kappa coefficient is increased by 0.118.
[0137] The Xception 65 backbone feature extraction network used by DeepLab V3+ has a relatively deep number of layers, while the internal differences among the various wetland types in the experimental area are small, and the advantages of the deep network structure cannot be expressed, resulting in the overall wetland information extraction results being fragmented, and there are many misclassifications and mixed classifications between the "dry field-paddy field-pond / reservoir", "reed-salsa winged wetland", "tidal flat-pond / reservoir" and other landforms, such as Fig.11 As shown in (b). The user accuracy for mixed wetlands and canals / ditches is less than 50%. FCN is a fully convolutional neural network that does not take into account spatial information and multi-scale features. The extraction of large-scale objects is relatively complete, and the fragmentation of objects is improved. However, the extraction effect of small objects such as canals / ditches is poor. Fig.11 (f), (g), and (h) are shown in the figure. At the same time, the fully connected layer in FCN generates a large number of parameters, resulting in low model operation efficiency. The mid-level feature fusion model adds mid-level semantic features to the decoder part of the DeepLab V3+ network model, making the multi-scale connection of the feature map more complete, and the producer has the highest accuracy in the extraction of canals / ditches, rivers and other features, as shown in Table 2. However, there are still deficiencies in water connectivity and the distinction between "beach-river" and other features, such as Fig.11 As shown in (g) and (h), ERFNet uses early dimension reduction and asymmetric network structure to improve network efficiency. It has a good effect on distinguishing rivers and tidal flats, but the overall extraction results have an obvious "salt and pepper phenomenon". The extraction accuracy of canals / ditches, paddy fields and other features is low, and the "paddy field-aquaculture pond" confusion classification phenomenon is serious.
[0138] The M&E-DeepLab introduces layer-by-layer feature transfer in the ASPP module, which achieves the expansion of the receptive field and improves the information utilization rate. It also improves the model accuracy and robustness without increasing the complexity of the model through the ECA attention mechanism, effectively reducing the misclassification phenomenon in the wetland information extraction process and improving the extraction effect of various land features. Among them, the producer accuracy of seven land features, including Suaeda salsa wetland, reed wetland, mixed wetland, tidal flat, aquaculture pond, dry field, and building, is higher than that of other methods, and the user accuracy of four land features, including canal / ditch, pond / reservoir, paddy field, and building, is higher than that of other methods. The producer accuracy of canals / ditches in the extraction results of the wetland information fine extraction model of the present invention is the lowest, mainly because this type of feature belongs to a small water body, which is affected by the spatial resolution of the image and the aquatic vegetation and wet vegetation in the wetland are closely connected with the water body, resulting in poor extraction of canals / ditches; the user accuracy of mixed wetlands is the lowest, because the spectral information characteristics between different vegetation are small and difficult to distinguish, resulting in easy confusion between the three types of features such as mixed wetlands, Suaeda salsa wetlands and reed wetlands. The test results show that the overall accuracy of the network model in the Panjin Liaohe wetland information extraction results is 89.967%, and the Kappa coefficient is 0.878, indicating that it has a high consistency with the actual distribution of features.
[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A method for fine extraction of coastal wetland information based on deep learning, characterized in that: include: Preprocess the remote sensing images reflecting the coastal wetland area, and annotate the preprocessing results based on the wetland classification system to obtain a sample data set; A coastal wetland information extraction model was constructed based on deep learning, and the model was trained using a sample data set; Based on the trained coastal wetland information fine extraction model, the coastal wetland information is finely extracted from the input remote sensing image; The coastal wetland information fine extraction model comprises the following steps: The remote sensing image is input into the MobileNet V2 backbone feature extraction module to extract shallow features; the shallow features are input into the dilated pooling pyramid module to extract deep features; The shallow features are input into a 1×1 convolution to compress the number of channels; the deep features are upsampled and fused with the feature map obtained by compressing the number of channels; the feature fusion results are input into a convolution with a convolution kernel size of 3, the number of channels is further adjusted, and the adjusted results are upsampled to output the fine extraction results of coastal wetland information.
2. The method for fine extraction of coastal wetland information based on deep learning according to claim 1 is characterized in that: The remote sensing images reflecting the coastal wetland area are preprocessed, and the preprocessing results are labeled based on the wetland classification system to obtain a sample data set, including: The original L1C-level images of the Sentinel-2 satellite in the coastal wetland area were processed to obtain L2A-level remote sensing images, and the processed remote sensing images were annotated at the pixel level to obtain an annotated remote sensing image. The original images and annotated remote sensing images are processed by sliding cropping and cut into several tiles, and the sample data set is expanded by flipping and transposing.
3. The method for fine extraction of coastal wetland information based on deep learning according to claim 1 is characterized in that: The wetland classification system described is divided into the following categories: Canals / ditches, rivers, Suaeda salsa wetlands, reed wetlands, mixed wetlands, tidal flats, aquaculture ponds, ponds / reservoirs, paddy fields, dry fields and buildings (structures); among which, the mixed wetlands refer to wetland types including reeds, Suaeda salsa, tidal flats, canals and ditches, which have a high degree of interlacing of land features and are difficult to distinguish due to the influence of image resolution.
4. The method for fine extraction of coastal wetland information based on deep learning according to claim 1 is characterized in that: The step of inputting the remote sensing image into the MobileNet V2 backbone feature extraction module for shallow feature extraction includes: First, point-by-point convolution is used to increase the dimension of the input feature layer. Second, depth-wise separable convolution is used to extract features from the feature layer after dimensionality increase. Finally, point-by-point convolution is performed again on the extracted features to achieve feature dimensionality reduction.
5. The method for fine extraction of coastal wetland information based on deep learning according to claim 1 is characterized in that: The atrous pooling pyramid module uses three parallel dilated convolution branches to achieve layer-by-layer feature transfer of feature maps. The step of inputting shallow features into the dilated pyramid module to extract deep features includes: Input the shallow features into 1×1 convolution to compress the number of channels to obtain feature map 1; The shallow features are input into a 3×3 convolution with a dilation rate of 6 to obtain feature map 2; The feature map 2 is input into a 3×3 convolution with a dilation rate of 12 to obtain the feature map 3; The feature map 3 is input into a 3×3 convolution with a dilation rate of 18 to obtain the feature map 4; Perform global average pooling on shallow features to obtain feature map 5; Fuse feature maps 1 to 5; The feature fusion result is input into the ECA attention mechanism module, which performs global average pooling on the input feature map and transforms the feature map from a [H, W, C] matrix to a [1, 1, C] vector. The convolution kernel size is calculated by the adaptive function and applied to the one-dimensional convolution to obtain the channel-by-channel weight of the feature map; the calculation formula is as follows: Where C represents the number of input feature map channels, b = 1, γ = 2; Then the convolution layer with a convolution kernel size of 1 is input to obtain the deep features.
6. The method for fine extraction of coastal wetland information based on deep learning according to claim 1 is characterized in that: The method further includes: Four objective evaluation indicators, namely overall accuracy, producer accuracy, user accuracy and Kappa coefficient, were used to quantitatively evaluate the model effect of the coastal wetland information fine extraction model.
7. A coastal wetland information fine extraction system based on deep learning, characterized in that: include: The preprocessing module is used to preprocess the remote sensing images reflecting the coastal wetland area and annotate the preprocessing results based on the wetland classification system to obtain a sample data set; The training module is used to: construct a coastal wetland information extraction model based on deep learning and train the model using a sample data set; The fine extraction module is used to: extract the fine coastal wetland information from the input remote sensing image based on the trained fine coastal wetland information extraction model; Wherein, the coastal wetland information fine extraction model includes: The encoder structure is used to: input the remote sensing image into the MobileNet V2 backbone feature extraction module for shallow feature extraction; input the shallow features into the hollow pooling pyramid module to extract deep features; The decoder structure is used to: input shallow features into 1×1 convolution for channel compression; upsample deep features and fuse them with feature maps obtained by channel compression; input the feature fusion results into convolution with a convolution kernel size of 3, further adjust the number of channels, upsample the adjusted results, and output the refined extraction results of coastal wetland information.
8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for fine extraction of coastal wetland information based on deep learning as described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for fine extraction of coastal wetland information based on deep learning as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Hybrid neural network hyperspectral remote sensing image coastal wetland ground feature recognition method
CN113822230A
Coastal wetland high-definition remote sensing image ground object identification method based on double-flow coding and decoding
CN114596499A
Ground feature classification method and device based on multi-scale attention mechanism, electronic equipment and medium
CN114913436A
Cited By
Erosion gully morphological feature extraction system and method, electronic equipment and storage medium
CN121482417A
