A method for extracting impervious surfaces from remote sensing images based on deep learning

Abstract

This paper proposes a deep learning-based method for extracting impervious surfaces from remote sensing images. The method comprises the following steps: obtaining raw remote sensing data and its corresponding impervious surface map, and dividing the training data into a specific proportion; constructing an initial remote sensing image impervious surface extraction model by combining a Transformer and a CNN with an attention mechanism; training the initialization model using the training data to obtain the remote sensing image impervious surface extraction model; and obtaining a target remote sensing image, inputting the target remote sensing image into the remote sensing image impervious surface extraction model to obtain an impervious surface extraction map. The method proposed in this paper utilizes a deep neural network for impervious surface extraction, comprehensively utilizing various impervious surface features in remote sensing images to improve recognition performance.

Description

Technical Field

[0001] The invention belongs to the field of image segmentation. Background Art

[0002] An impervious surface refers to a surface where water cannot penetrate into the soil. It includes natural impervious surfaces and artificial impervious surfaces (such as asphalt and other artificially constructed surfaces that are impervious to water). Common impervious surfaces include roads, building roofs, parking lots, and tennis courts.

[0003] Urbanization has been the most significant human activity since the 20th century. Currently, approximately 50% of the world's population lives in cities. One of the most prominent characteristics of urbanization is the continuous increase in impervious surfaces. As urbanization progresses, land surrounding urban centers is continuously converted to construction. As construction land expands, impervious surfaces rapidly increase, while ecologically valuable land, such as vegetation and water sources, becomes increasingly smaller and fragmented. With the acceleration of urbanization, land use patterns are gradually shifting from being dominated by vegetation and soil to being dominated by urban impervious surfaces. Various impervious surfaces form the matrix of urban landscapes, dominating urban landscape patterns and processes. They are characteristic features of urban land cover and can be used to monitor changes in urban land cover. Furthermore, with the significant land use shifts during urbanization, large amounts of cultivated land and unused land are converted to impervious surfaces. The irrational spatial distribution of impervious surfaces has been shown to be a major factor influencing urban waterlogging and urban heat island phenomena. The application of the Impervious Surface Index to urban ecological and environmental monitoring, effects, and mechanism analysis is a current research hotspot. Its applications include dynamic monitoring of land cover, urban hydrological simulation, estimation of nonpoint source pollution loads, thermal environmental effects, regional climate change, urban expansion monitoring, and urban ecosystem assessment. Scientifically understanding the spatiotemporal variations in impervious surfaces is crucial for developing sound regional development plans, addressing urban flooding, evaluating the state of urban ecosystems, and improving the quality of urban ecological environments.

[0004] Image data obtained through remote sensing technology offers advantages such as large coverage, low cost, and repeated observations. This provides a high-quality data source for long-term impervious surface extraction, and remote sensing technology offers a more efficient, rapid, and timely means of monitoring impervious surface changes. High-precision impervious surface extraction using remote sensing imagery is crucial for monitoring urbanization levels, assessing changes in urban resources and the environment, and providing a basis for government decision-makers to plan and manage sustainable urban development.

[0005] Remote sensing-based impervious surface extraction techniques primarily include spectral mixture analysis, index methods, decision trees, regression models, object-oriented methods, support vector machines (SVMs), and artificial neural networks (ANNs). Wu and Murray proposed the VHLS model, which divides cities into a combination of four endmembers. Through a series of steps, including minimum noise component transformation (MNF) and pixel purity index (PPI), each city image is decomposed into four component images representing these four endmembers. The high-albedo component image is then summed with the low-albedo component image to obtain the impervious surface image. However, the following challenges exist: First, using only one or two fixed endmembers cannot effectively represent the various impervious surface materials in a city, resulting in a lot of bare soil being mixed with the impervious surface information. Second, because the VIS model does not account for water bodies, water information is mixed with low-albedo impervious surface information, necessitating a pre-masking to remove water bodies. Third, due to the limited spectral resolution of multispectral imagery, it is difficult to obtain the spectral characteristics of the endmembers representing pure pixels. Fourth, the computational complexity makes it difficult to extract impervious surface information over large areas. Index methods use remote sensing spectral indices to enhance and extract impervious surface information. However, most spectral indices require masking water bodies and bare land before extracting impervious surfaces. The spectral characteristics of bare land can easily be confused with buildings, and finding the optimal threshold for extracting impervious surfaces is also difficult. W. Fan et al. proposed an object-based classification method with OSM POIs. However, the availability of OSM POI data is a potential limitation.

[0006] With the development of deep learning, various convolutional neural networks have emerged in the field of semantic segmentation to achieve pixel-level image segmentation, and deep neural networks are increasingly being used in remote sensing imagery. Using deep neural networks for impervious surface extraction can comprehensively leverage the various features of impervious surfaces in remote sensing images to improve recognition. We propose a deep learning-based dual-branch architecture for extracting impervious surfaces from remote sensing images, combining a Transformer and a CNN with an attention mechanism, called TCA-NET. Specifically, the transformer branch is used to capture long-range and large-scale dependencies, while the Unified Network (UNet) branch is used to capture low-level details. In the transformer branch, a CNN-Transformer hybrid structure replaces a single transformer, avoiding the loss of detail caused by original upsampling. In the UNet branch, a coordinate attention mechanism is added to enhance the UNet's ability to capture dependencies, further improving the accuracy of small object detection. Summary of the Invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the purpose of the present invention is to propose a method for extracting impervious surfaces from remote sensing images based on deep learning, which is used to realize impervious surface extraction through deep neural networks.

[0009] To achieve the above objectives, a first embodiment of the present invention proposes a method for extracting impervious surfaces from remote sensing images based on deep learning, comprising:

[0010] Obtain the original remote sensing data and its corresponding impervious surface map, and divide the training data according to a certain ratio;

[0011] Combining the Transformer and CNN with the attention mechanism to construct an initialization remote sensing image impervious surface extraction model; training the initialization model using the training data to obtain the remote sensing image impervious surface extraction model;

[0012] A target remote sensing image is acquired, and the target remote sensing image is input into the impervious surface remote sensing image extraction model to obtain an impervious surface extraction map.

[0013] In addition, the method for extracting impervious surfaces from remote sensing images based on deep learning according to the above embodiment of the present invention may also have the following additional technical features:

[0014] Furthermore, in one embodiment of the present invention, a Transformer and a CNN are combined with an attention mechanism to construct an initialization model for extracting remote sensing images of impervious surfaces, including:

[0015] In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, and the data is sent to the transformer layer for processing;

[0016] In the transformer layer, the traditional Vit Block is repeated 12 times;

[0017] The output feature map of the transformer layer is restored to H×W×C through convolution operation.

[0018] Furthermore, in one embodiment of the present invention, a Transformer and a CNN are combined with an attention mechanism to construct an initialization model for extracting remote sensing images of impervious surfaces, including:

[0019] In the UNet branch, a coordinate attention mechanism is added, specifically adding a coordinate attention module between every two decoding blocks of UNet; the final output is H×W×C;

[0020] The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain the impervious surface extraction map.

[0021] Furthermore, in one embodiment of the present invention, the initialization model is trained using training data to obtain a remote sensing image impervious surface extraction model, including:

[0022] The output of the remote sensing image impervious surface extraction model is expressed as:

[0023] Furthermore, in one embodiment of the present invention, the initialization model is trained using training data to obtain a remote sensing image impervious surface extraction model, further comprising:

[0024] A hybrid loss function is introduced to optimize the remote sensing image impervious surface extraction model. The formula is as follows:

[0025] L total =αL Dice +βL BCE ,

[0026]

[0027] L BCE =-(true×log(pred+(1-true)×log(1-pred))),

[0028] Among them, α and β are balance factors.

[0029] To achieve the above objectives, a second embodiment of the present invention proposes a device for extracting impervious surfaces from remote sensing images based on deep learning, comprising the following modules:

[0030] The acquisition module is used to obtain the original remote sensing data and its corresponding impervious surface map, and divide the training data according to a certain ratio;

[0031] A training module is used to combine the Transformer and CNN with the attention mechanism to construct an initialization remote sensing image impervious surface extraction model; the initialization model is trained using the training data to obtain the remote sensing image impervious surface extraction model;

[0032] The extraction module is used to obtain a target remote sensing image, input the target remote sensing image into the remote sensing image impervious surface extraction model, and obtain an impervious surface extraction map.

[0033] Furthermore, in one embodiment of the present invention, the training module is also used to

[0034] In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, and the data is sent to the transformer layer for processing;

[0035] In the transformer layer, the traditional Vit Block is repeated 12 times;

[0036] Restore the output feature map of the transformer layer to H×W×C through convolution operation;

[0037] In the UNet branch, a coordinate attention mechanism is added, specifically adding a coordinate attention module between every two decoding blocks of UNet; the final output is H×W×C;

[0038] The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain the impervious surface extraction map.

[0039] Furthermore, in one embodiment of the present invention, the training module is further configured to:

[0040] A hybrid loss function is introduced to optimize the impervious surface remote sensing image extraction model.

[0041] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, which is characterized in that it includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the above-mentioned method for extracting impervious surfaces from remote sensing images based on deep learning.

[0042] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned method for extracting impervious surfaces from remote sensing images based on deep learning.

[0043] The deep learning-based impervious surface extraction method for remote sensing images proposed in an embodiment of the present invention combines Transformer and CNN with an attention mechanism, which is called TCA-NET. Specifically, the transformer branch is used to capture long-distance and large-scale dependencies, and the UNet branch is used to obtain underlying details. In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, which can avoid the loss of detail information caused by the original upsampling. In the UNet branch, a coordinate attention mechanism is added to improve UNet's ability to capture dependencies, further improving the accuracy of small object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0045] Figure 1 A flowchart of a method for extracting impervious surfaces from remote sensing images based on deep learning is provided in an embodiment of the present invention.

[0046] Figure 2 A schematic diagram of a transformer-based deep learning network model provided in an embodiment of the present invention.

[0047] Figure 3 A schematic diagram of the flow of a device for extracting impervious surfaces from remote sensing images based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0049] The following describes a method for extracting impervious surfaces from remote sensing images based on deep learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0050] Figure 1 A flowchart of a method for extracting impervious surfaces from remote sensing images based on deep learning is provided in an embodiment of the present invention.

[0051] like Figure 1 As shown in FIG, the deep learning-based remote sensing image impervious surface extraction method includes the following steps:

[0052] S101: Obtain original remote sensing data and its corresponding impervious surface map, and divide the training data according to a certain ratio;

[0053] S102: Combine Transformer and CNN with attention mechanism to construct an initialization impervious surface remote sensing image extraction model; train the initialization model with training data to obtain the impervious surface remote sensing image extraction model;

[0054] S103: Acquire a target impervious surface remote sensing image, input the target impervious surface remote sensing image into an impervious surface remote sensing image extraction model, and obtain an impervious surface extraction map.

[0055] Furthermore, in one embodiment of the present invention, the construction of an initialization impervious surface remote sensing image extraction model by combining Transformer and CNN with an attention mechanism includes:

[0056] In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, and the data is sent to the transformer layer for processing;

[0057] In the transformer layer, the traditional Vit Block is repeated 12 times;

[0058] The output feature map of the transformer layer is restored to H×W×C through convolution operation.

[0059] Furthermore, in one embodiment of the present invention, the construction of an initialization impervious surface remote sensing image extraction model by combining Transformer and CNN with an attention mechanism includes:

[0060] In the UNet branch, a coordinate attention mechanism is added, specifically adding a coordinate attention module between every two decoding blocks of UNet; the final output is H×W×C;

[0061] The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain an impervious surface extraction map.

[0062] Furthermore, in one embodiment of the present invention, the initialization model is trained using the training data to obtain an impervious surface remote sensing image extraction model, including:

[0063] The output of the impervious surface remote sensing image extraction model is expressed as:

[0064] like Figure 2As shown, this is a diagram of the transformer-based deep learning network model of the present invention.

[0065] Furthermore, in one embodiment of the present invention, the training of the initialization model using the training data to obtain the remote sensing image impervious surface extraction model further includes:

[0066] A hybrid loss function is introduced to optimize the impervious surface remote sensing image extraction model. The formula is as follows:

[0067] L total =αL Dice +βL BCE ,

[0068]

[0069] L BCE =-(true×log(pred+(1-true)×log(1-pred))),

[0070] Among them, α and β are balance factors.

[0071] The deep learning-based impervious surface extraction method for remote sensing images proposed in an embodiment of the present invention combines Transformer and CNN with an attention mechanism, which is called TCA-NET. Specifically, the transformer branch is used to capture long-distance and large-scale dependencies, and the UNet branch is used to obtain underlying details. In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, which can avoid the loss of detail information caused by the original upsampling. In the UNet branch, a coordinate attention mechanism is added to improve UNet's ability to capture dependencies, further improving the accuracy of small object detection.

[0072] In order to implement the above embodiments, the present invention also proposes a remote sensing image impervious surface extraction device based on deep learning.

[0073] Figure 3 A schematic structural diagram of a device for extracting impervious surfaces from remote sensing images based on deep learning is provided in an embodiment of the present invention.

[0074] like Figure 3 As shown, the remote sensing image impervious surface extraction device based on deep learning includes: an acquisition module 100, a training module 200, and an extraction module 300, wherein:

[0075] The acquisition module is used to obtain the original remote sensing data and its corresponding impervious surface map, and divide the training data according to a certain ratio;

[0076] A training module is used to combine the Transformer and CNN with the attention mechanism to construct an initialization remote sensing image impervious surface extraction model; the initialization model is trained using training data to obtain the remote sensing image impervious surface extraction model;

[0077] The extraction module is used to obtain the target remote sensing image, input the target remote sensing image into the impervious surface remote sensing image extraction model, and obtain the impervious surface extraction map.

[0078] Furthermore, in one embodiment of the present invention, the training module is also used to

[0079] In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, and the data is sent to the transformer layer for processing;

[0080] In the transformer layer, the traditional Vit Block is repeated 12 times;

[0081] Restore the output feature map of the transformer layer to H×W×C through convolution operation;

[0082] In the UNet branch, a coordinate attention mechanism is added, specifically adding a coordinate attention module between every two decoding blocks of UNet; the final output is H×W×C;

[0083] The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain an impervious surface extraction map.

[0084] Furthermore, in one embodiment of the present invention, the training module is further configured to:

[0085] A hybrid loss function is introduced to optimize the impervious surface remote sensing image extraction model.

[0086] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, which is characterized in that it includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the above-mentioned method for extracting impervious surfaces from remote sensing images based on deep learning.

[0087] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned method for extracting impervious surfaces from remote sensing images based on deep learning.

[0088] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0090] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limiting the present invention. A person skilled in the art may change, modify, replace, and modify the above embodiments within the scope of the present invention.

Claims

1. A method for extracting impervious surfaces from remote sensing images based on deep learning, characterized in that: The following steps are involved: Obtain the original remote sensing data and its corresponding impervious surface map, and divide the training data according to a certain ratio; Combining the Transformer and CNN with the attention mechanism to construct an initialization remote sensing image impervious surface extraction model; training the initialization model using the training data to obtain the remote sensing image impervious surface extraction model; Acquire a target remote sensing image, input the target remote sensing image into the remote sensing image impervious surface extraction model, and obtain an impervious surface extraction map; The method of combining Transformer and CNN with attention mechanism to construct an initialization model for extracting impervious surfaces from remote sensing images includes: In the transformer branch, a CNN-Transformer hybrid structure is used instead of a single transformer, and the data is sent to the transformer layer for processing; In the transformer layer, the traditional Vit Block is repeated 12 times; The output feature map of the transformer layer is restored to H×W×C through convolution operation Among them, in the UNet branch, a coordinate attention mechanism is added, specifically including adding a coordinate attention module between every two decoding blocks of UNet; the final output is H×W×C; The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain an impervious surface extraction map.

2. The method according to claim 1, characterized in that The initialization model is trained using the training data to obtain a remote sensing image impervious surface extraction model, including: The input of the remote sensing image impervious surface extraction model is expressed as:

3. The method according to claim 1, characterized in that The training of the initialization model using the training data to obtain a remote sensing image impervious surface extraction model further includes: A hybrid loss function is introduced to optimize the remote sensing image impervious surface extraction model. The formula is as follows: L total =αL Dice +βL BCE , L BCE =-(true×log(pred+(1-true)×log(1-pred))), Among them, α and β are balance factors.

4. A device for extracting impervious surfaces from remote sensing images based on deep learning, characterized in that: Includes the following modules: The acquisition module is used to obtain the original remote sensing data and its corresponding impervious surface map, and divide the training data according to a certain ratio; A training module is used to combine the Transformer and CNN with the attention mechanism to construct an initialization remote sensing image impervious surface extraction model; the initialization model is trained using the training data to obtain the remote sensing image impervious surface extraction model; An extraction module is used to obtain a target remote sensing image, input the target remote sensing image into the remote sensing image impervious surface extraction model, and obtain an impervious surface extraction map; The training module is also used to use a CNN-Transformer hybrid structure in the transformer branch instead of a single transformer, and send the data to the transformer layer for processing. In the transformer layer, the traditional Vit Block is repeated 12 times; Restore the output feature map of the transformer layer to H×W×C through convolution operation; In the UNet branch, a coordinate attention mechanism is added, specifically adding a coordinate attention module between every two decoding blocks of UNet; the final output is H×W×C; The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain an impervious surface extraction map; The training module is also used to add a coordinate attention mechanism to the UNet branch, specifically adding a coordinate attention module between every two decoding blocks of the UNet; the final output is H×W×C; The outputs of the transformer branch and the UNet branch are fused and sent to the segmentation layer to obtain an impervious surface extraction map.

5. The device according to claim 4, characterized in that The training module is further used to: A hybrid loss function is introduced to optimize the remote sensing image impervious surface extraction model. The formula is as follows: L total =αL Dice +βL BCE , L BCE =-(true×log(pred+(1-true)×log(1-pred))), Among them, α and β are balance factors.

6. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for extracting impervious surfaces from remote sensing images based on deep learning is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting impervious surfaces from remote sensing images based on deep learning is implemented.

Images