Remote sensing image building extraction method and system based on local-global features

Abstract

The invention relates to the technical field of high-resolution optical remote sensing image processing, in particular to a remote sensing image building extraction method and system based on local-global features, and the method comprises the steps: obtaining a remote sensing image containing a building, and inputting the remote sensing image into a trained building extraction network model; the model adopts a double-branch remote sensing image building network fusing local-global features, and comprises a double-branch structure encoder, a cross-feature attention fusion module and a feature refinement enhancement module. Wherein the double-branch structure encoder is further divided into a CNN branch and a Transform branch, the CNN branch is responsible for extracting local features, the Transform branch is used for extracting global features, the cross-feature attention fusion module fuses the features extracted by the CNN branch and the Transform branch, the feature refining enhancement module further enhances the feature extraction capability of the network at a decoder end, and the feature extraction capability of the network at the decoder end is enhanced. Therefore, the building in the remote sensing image can be effectively extracted.

Description

Technical Field

[0001] The invention relates to the technical field of high-resolution optical remote sensing image processing, and in particular to a remote sensing image building extraction method and system based on local-global features. Background Art

[0002] With the rapid development of satellite remote sensing and various sensor technologies, the geographical span and spatial resolution of remote sensing images have been significantly improved. Buildings are one of the land features contained in remote sensing images. Therefore, automatically, efficiently and accurately extracting building information from remote sensing images is of great significance to promoting the development of applications such as urban planning, intelligent transportation, smart cities and digital earth.

[0003] Traditional building extraction methods usually use the texture, shape, spectrum and spatial detail features of buildings to design feature extraction and pixel discrimination algorithms to extract buildings from remote sensing images. For example, threshold-based, edge-based and region-based image extraction methods. Although these traditional methods are highly mature, highly interpretable and easy to implement, they are very sensitive to external conditions such as lighting and sensor type, and can usually only extract buildings in specific scenarios, making it difficult to cope with remote sensing image building extraction tasks in different scenarios.

[0004] In recent years, deep learning has achieved good results in the semantic segmentation of high-resolution remote sensing building images due to its strong generalization and self-learning target features (such as CNN architecture and Transformer architecture). However, the size of the convolution kernel in the CNN architecture limits the size of the network's receptive field, which limits the network's ability to obtain global context information. It is difficult to cope with the characteristics of buildings in remote sensing images, such as different scales, complex textures, and large background interference. Methods that rely solely on local feature extraction still have a lot of room for improvement. The Transformer-based network is not capable of modeling fine-grained features of buildings at a local scale, which results in the network being unable to fully capture the subtle feature differences between different objects when processing remote sensing images with complex local changes.

[0005] Due to the natural complementarity of the features extracted by CNN and Transformer, combining CNN with Transformer can enable the network to extract both global and local features. However, these two sets of features have essentially different properties. Therefore, how to effectively integrate these two sets of features is the key issue in combining CNN with Transformer. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for extracting buildings from remote sensing images based on local-global features in order to address the deficiencies in the above-mentioned prior art, so as to solve the technical problem that subtle feature differences cannot be captured during building recognition in remote sensing images.

[0007] The objective of the present invention is achieved by the following technical solutions: In a first aspect, the present invention provides a method for extracting buildings from remote sensing images based on local-global features, comprising: Acquire a remote sensing image containing a building, input the remote sensing image into a trained building extraction network model, and extract the building in the remote sensing image; The building extraction network model adopts a dual-branch remote sensing image building network that integrates local and global features. The building extraction network model includes a dual-branch structure encoder, a cross-feature attention fusion module, and a feature refinement enhancement module; The dual-branch structure encoder includes a CNN branch and a Transformer branch, the CNN branch is used to extract local features, and the Transformer branch is used to extract global features; the cross-feature attention fusion module is used to fuse the features extracted by the CNN branch and the Transformer branch; the feature refinement enhancement module is used to enhance the edge details, channel dimensions and multi-scale dimensions of the building features in the input feature map.

[0008] As a further improvement of the present invention, the CNN branch includes four convolution stages, each convolution stage is used to gradually extract local texture and semantic information of the remote sensing image and establish local interactions; each convolution stage includes at least two convolution layers, and each convolution layer is connected to a batch normalization layer and a linear rectification unit.

[0009] As a further improvement of the present invention, the Trasnformer branch includes four hybrid Trasnformer blocks; the hybrid Trasnformer block includes several multi-head self-attentions, a hybrid feedforward neural network and an overlapping block merging unit; the multi-head self-attention is used to capture features in the remote sensing image to obtain a feature map; the hybrid feedforward neural network is used to enhance the features learned by the multi-head self-attention; and the overlapping block merging unit is used to downsample the feature map.

[0010] As a further improvement of the present invention, the cross-feature attention fusion module includes a channel attention fusion branch and a spatial attention fusion branch. The channel attention fusion branch is used to perform a global average pooling operation on the input feature map, and process the channel through a one-dimensional convolution layer with an adaptive size convolution kernel and a Sigmoid activation function to obtain features with redistributed weights on the channel; the spatial attention fusion branch is used to perform a preliminary fusion on the input feature map to obtain preliminary fusion features, perform global average pooling and global maximum pooling on the preliminary fusion features, and splice the channel dimensions to obtain fusion features with redistributed weights.

[0011] As a further improvement of the present invention, the building extraction network model also includes a decoder, which includes four upsampling modules, each of which includes a bilinear interpolation upsampling strategy and two convolutional layers, and each convolutional layer is connected to a batch normalization layer and a linear rectification unit.

[0012] As a further improvement of the present invention, the feature refinement and enhancement module is inserted into the decoder to enhance the edge details and channel scale of the building features in the input feature map.

[0013] As a further improvement of the present invention, the feature refinement enhancement module includes a multi-scale enhancement path unit, a channel enhancement path unit and a deformable adaptive boundary enhancement path unit; The multi-scale enhancement path unit is used to extract multi-scale feature information of buildings in remote sensing images to obtain multi-scale enhancement features. The multi-scale enhancement path unit includes four hole convolution layers with different expansion rates. The channel enhancement path unit generates channel attention weights using global average pooling, and enhances the channels of the input features according to the channel attention weights to obtain channel enhancement features; The deformable adaptive boundary enhancement path unit is used to enhance edge details of channel enhancement features and multi-scale enhancement features. The deformable adaptive boundary enhancement path unit includes a batch normalization layer, a convolution layer of a linear rectification unit, and a deformable convolution layer.

[0014] As a further improvement of the present invention, the training step of the building extraction network model includes: Batch acquiring remote sensing images containing buildings, and constructing a remote sensing building data set based on the acquired remote sensing images; the remote sensing images in the remote sensing building data set are subjected to labeling processing; Preprocess and enhance the remote sensing images in the dataset; Using the processed remote sensing building dataset, the building extraction network model will be trained end-to-end to obtain a trained building extraction network model.

[0015] As a further improvement of the present invention, the processed data set is used to perform end-to-end training on the building extraction network model, specifically including: The processed remote sensing building dataset is used to iteratively optimize the pre-built building extraction network by back propagation in combination with the boundary supervision joint loss function to obtain the trained network; the boundary supervision joint loss function is:

[0016] In the formula, L is the boundary supervision joint loss function; represents the predicted label, represents the true label, represents the cross entropy loss, which is used to measure the difference between the predicted probability of each category of the pixel and the true label; represents the dice loss, which is calculated based on the overlap coefficient between the predicted segmentation result and the true segmentation result; Represents the Laplacian convolution kernel used to extract the boundary between the predicted label and the true label; Represents the binary cross entropy loss for edge pixel supervision.

[0017] In a second aspect, the present invention provides a remote sensing image building extraction system based on local-global features, which is used to implement the above-mentioned remote sensing image building extraction method based on local-global features, comprising: A data acquisition module, used to acquire remote sensing images containing buildings; Building extraction module: Use the trained building extraction network model to extract buildings from remote sensing images; The building extraction network model adopts a dual-branch remote sensing image building network that integrates local and global features. The building extraction network model includes a dual-branch structure encoder, a cross-feature attention fusion module, and a feature refinement enhancement module; The dual-branch structure encoder includes a CNN branch and a Transformer branch, the CNN branch is used to extract local features, and the Transformer branch is used to extract global features; the cross-feature attention fusion module is used to fuse the features extracted by the CNN branch and the Transformer branch; the feature refinement enhancement module is used to enhance the edge details, channel dimensions and multi-scale dimensions of the building features in the input feature map.

[0018] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform the above-mentioned remote sensing image building extraction method based on local-global features.

[0019] In a fourth aspect, the present invention provides a computing device, comprising: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the above-mentioned remote sensing image building extraction method based on local-global features.

[0020] The beneficial effects of the present invention are as follows: the present invention provides a method for extracting buildings from remote sensing images based on local-global features, which uses an encoder with a dual-branch structure of CNN and Transformer to simultaneously capture local texture information and global feature context dependencies of buildings, effectively extracting local detail information of buildings and ensuring the accuracy of building edges and details, and better understanding the overall structure and spatial relationship of buildings. Secondly, in order to overcome the differences in features extracted by the CNN branch and the Transformer branch, the present invention designs a cross-feature attention fusion module to effectively aggregate the two sets of different features extracted by the two branches, and weights their importance, so that the model can more effectively integrate local and global features and enhance the overall understanding and recognition capabilities of buildings. The building extraction network model combines the advantages of CNN and Transformer, and can improve the accuracy of building extraction while enhancing the processing capabilities of complex scenes, reducing false detections and missed detections, and improving computational efficiency.

[0021] Furthermore, the present invention also adds a feature refinement enhancement module, which is inserted into the end of the decoder to reduce information loss during the upsampling process and refine the edges and local details of the building.

[0022] Furthermore, the CBR module in the decoder can effectively restore the spatial resolution of the feature map, extract and enhance building features, fuse multi-level feature information, and improve the robustness and generalization ability of the model through a combination of bilinear interpolation upsampling strategy, convolutional layers, batch normalization and ReLU.

[0023] In summary, the method of the present invention can not only accurately capture the local fine-grained information and global contextual information of remote sensing images, but also effectively improve the extraction accuracy of buildings in remote sensing images, and has obvious advantages over existing methods in complex task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 The network structure diagram of building extraction from remote sensing images based on dual-branch fusion of local and global features in an embodiment of the present invention is Figure 2 It is a structural diagram of the cross-feature attention fusion module CFAFM in an embodiment of the present invention.

[0026] Figure 3 It is a structural diagram of the feature refinement enhancement module FREN in an embodiment of the present invention.

[0027] Figure 4 It is a schematic diagram of the building extraction process in an embodiment of the present invention.

[0028] Figure 5 It is a building extraction effect diagram in an embodiment of the present invention.

[0029] Figure 6 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose and technical solution of the present invention clearer and easier to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] Terminology explanation: CNN: Convolutional Neural Network.

[0032] Trasnformer: Attention mechanism.

[0033] CBR: Upsampling module.

[0034] CFAFM: Cross-Feature Attention Fusion Module.

[0035] FREM: Feature Refinement Enhancement Module.

[0036] MTB:Hybrid Trasnformer Block.

[0037] EMHSA: Multi-headed self-attention.

[0038] Mix-FFN: Mixed Feedforward Neural Networks.

[0039] OPM: Overlapping Block Merging.

[0040] MLP: Multi-layer Perception Layer.

[0041] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments, wherein the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0042] Example 1 like Figure 1-Figure 5 As shown, this embodiment proposes a dual-branch remote sensing image building extraction method based on the fusion of local and global features, comprising the following steps: Obtain remote sensing images containing buildings, input the remote sensing images into the trained building extraction network model, and extract the buildings in the remote sensing images.

[0043] The building extraction network model adopts a dual-branch remote sensing image building network that integrates local and global features. Specifically, a dual-branch remote sensing image building extraction network based on the integration of local and global features is constructed. The structure of the building extraction network is as follows: Figure 2 shown.

[0044] The network structure includes a dual-branch structure encoder, a decoder, a CFAFM cross-feature attention fusion module and a FREM feature refinement enhancement module, wherein: The encoder consists of a CNN branch and a Transformer branch. The two branches extract complementary local and global features from the original remote sensing image in parallel. In order to overcome the differences in features extracted by CNN and Transformer, the features of each layer are fully interacted through the designed cross-feature attention fusion module CFAFM. In addition, the fused features are passed to the decoder through jump connections to restore the image resolution. In particular, the designed feature refinement enhancement module FREM is introduced before the last upsampling of the decoder to further enhance and refine the fused shallow features to improve the network's extraction accuracy for buildings in remote sensing images.

[0045] At the same time, in order to better supervise the boundaries of buildings in remote sensing images and reduce the impact of class imbalance problems on network training, this paper uses a joint loss function that includes the cross entropy loss function, the dice loss function, and the binary cross entropy loss boundary supervision for edge pixel supervision, which enhances the network's accurate extraction of building boundaries and reduces the impact of class imbalance problems on network training.

[0046] The dual-branch encoder part of the dual-branch remote sensing image building extraction network based on the fusion of local-global features is constructed. The main steps are: The CNN branch gradually extracts the local texture and semantic information of the remote sensing image through four convolution stages, and establishes local interactions. The convolution stage includes two convolution layers, each of which includes a convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a linear rectifier unit. The first convolution stage performs two downsamplings through two identical convolution layer blocks to extract low-level spectral texture features in the remote sensing image. Each stage from the second to the fourth convolution stage contains two convolution layers. The first convolution layer reduces the spatial resolution of the feature map by deepening the channel dimension of the network and increasing the step size of the convolution kernel to extract the deep semantic features of the remote sensing image. The second convolution layer maintains the feature dimension without changing the spatial resolution of the feature map, and further extracts the detail features. Through four consecutive convolution stages, the spatial dimension of the feature map is gradually reduced, and the channel dimension is gradually deepened, so as to obtain local interaction features at different scales.

[0047] The Transformer branch includes four hybrid Transformer blocks (MTBs), each of which consists of an efficient multi-head self-attention EMSHA, a hybrid feedforward neural network Mix-FFN, and an overlapping block merge OPM. EMHSA adopts a sequence reduction strategy, introducing a sequence scaling factor to reduce the length of the sequence to reduce the computational complexity of the self-attention mechanism and reduce unnecessary computational redundancy; Mix-FFN is used to enhance the high-level features learned by multi-head self-attention, and uses 3×3 convolution instead of position encoding to learn position information; OPM divides a larger patch into multiple smaller overlapping patches to achieve the purpose of feature map downsampling while ensuring spatial semantic coherence.

[0048] Construct the decoder part of the dual-branch remote sensing image building extraction network based on the fusion of local and global features. The specific steps include: The decoder structure mainly consists of 3 CBR modules, 4 cross-feature attention fusion modules and 1 feature refinement and enhancement module. This structure is divided into four stages. Each of the first three stages contains a CBR module and a cross-feature attention fusion module. The CBR module completes the progressive resolution recovery of the feature map. The cross-feature attention fusion module completes the full fusion of complementary features in the encoder and outputs the fused features at different dimensional levels. Each CBR module contains a bilinear interpolation upsampling strategy and two convolution layers with a convolution kernel size of 3 and a stride of 1. Each convolution layer is connected to a batch normalization layer and a linear rectifier unit. The last stage contains a cross-feature attention fusion module, a feature refinement and enhancement module and a bilinear interpolation upsampling strategy to enhance the decoder's ability to express the correlation between different dimensions of remote sensing images and reduce the chessboard effect generated during the upsampling process to output the building extraction and classification tasks in remote sensing images.

[0049] Construct a cross-feature attention fusion module CFAFM. The specific steps include: Let the feature maps of the input CFAFM be and The output feature map is .in, , , are the number of channels, height, and width of the feature map input to CFAFM, respectively.

[0050] The proposed cross-feature attention fusion module CFAFM consists of two branches: channel attention fusion branch and spatial attention fusion branch.

[0051] Specifically, the channel attention fusion branch includes two global average pooling operations, two one-dimensional convolution layers with adaptively sized convolution kernels, and two sigmoid activation functions.

[0052] The spatial attention fusion branch includes a global maximum pooling operation, a global average pooling operation, a two-dimensional convolutional layer and a Sigmoid activation function.

[0053] The specific process of CFAFM can be described as: In the channel attention fusion branch, for the local features extracted by CNN And the global features extracted by Transformer , first perform global average pooling to reshape the dimensions into , and then use one-dimensional convolution with adaptive size convolution kernel to realize cross-channel information interaction, and get the weight coefficients of the two channels through Sigmoid activation function and ,Right now: (1) (2) in, is the Sigmoid activation function, is a one-dimensional convolution with an adaptive size convolution kernel. Multiplying the two sets of weights gives a dimension of The weight matrix , then multiply the weight matrix by the two sets of features and add them to the original input features to obtain the features with redistributed weights on the channels and .Right now: (3) (4) (5) In the spatial attention fusion branch, the feature map extracted by CNN is first And the feature map extracted by Transformer Add to get the initial fusion features , and the features Perform global average pooling and global maximum pooling respectively to obtain two The spatial description of the two spaces is concatenated in the channel dimension and then used The convolution kernel of different sizes reduces its dimension and obtains the spatial weight through the Sigmoid activation function. : (6) in represents the Sigmoid function, Indicates that the convolution kernel size is used The convolutional layer, Represents the feature concatenation (Concat) operation. Then the initial fusion features and spatial weight coefficient Multiply them together to get the fusion features that redistribute the weights in the spatial dimension ,Right now: (7) Finally, the features output by the channel attention fusion branch and And the features output by the spatial attention fusion branch Add together to get the final fusion feature ,Right now: (8) Construct the feature refinement enhancement module FREM. The specific steps include: Let the feature maps of the input FREM be The output feature map is .in, , , are the number of channels, height, and width of the feature map input to CFAFM, respectively.

[0054] The proposed FREM feature refinement enhancement module includes a multi-scale enhancement path, a channel enhancement path and a deformable adaptive boundary enhancement path.

[0055] Specifically, the multi-scale enhancement path uses four dilated convolutions with different expansion rates to expand the receptive field of the convolution kernel to extract the multi-scale feature information of remote sensing image buildings, and then splices them in the channel dimension to obtain multi-scale enhanced features. ,Right now: (9) (10) in, Represents dilated convolutions with different dilation rates, Indicates the expansion rate, is the ReLU activation function, Represents a feature concatenation (Concat) operation.

[0056] The channel enhancement path uses global average pooling to generate channel attention weights , multiply it with the input feature to obtain the channel enhancement feature .Right now: (11) (12) in Indicates that the convolution kernel size is The combination of convolutional layer, BN layer and ReLU activation function, is the Sigmoid activation function, Represented as the original input features.

[0057] Finally, the features and Features The added features are used as the input of the deformable adaptive boundary enhancement path to enhance the edge details of the feature information. The deformable adaptive boundary enhancement path consists of three main parts: first, The convolution layer performs convolution operations to keep the channel dimension unchanged to extract features; secondly, the deformable convolution layer is used to learn the geometric edges and shapes of the building to enhance the expressiveness of the features; finally, The convolution layer further integrates the features and adds the input features to the output features in the channel dimension through residual connections to obtain the final refined and enhanced features. .

[0058] The constructed building extraction network model is trained, and the specific steps include: Remote sensing images containing buildings are acquired in batches, and a remote sensing building dataset is constructed based on the acquired remote sensing images; the remote sensing images in the remote sensing building dataset are labeled. The remote sensing images in the dataset are preprocessed and data enhanced.

[0059] Specifically, the constructed remote sensing building dataset is preprocessed and data enhanced, including random rotation, random horizontal and vertical flipping, Gaussian filtering, bilateral filtering, contrast enhancement, etc. of the original remote sensing data and the labeled label images.

[0060] Using the processed remote sensing building dataset, the building extraction network model will be trained end-to-end to obtain a trained building extraction network model. Specifically, the pre-processed high-resolution remote sensing building training dataset is input into the network, and the boundary supervision joint loss function is used to calculate the loss through the forward propagation algorithm; the partial derivative of the objective function with respect to the feature is solved; the gradient is obtained by the back-propagation algorithm for parameter update learning.

[0061] Boundary Supervision Joint Loss Function L The definition is as follows: (13) in represents the predicted label, represents the true label, represents the cross entropy loss, which measures the difference between the predicted probability of each category of the pixel and the true label; stands for dice loss, which is calculated based on the overlap coefficient between the predicted segmentation result and the true segmentation result. This overlap metric does not depend on the frequency of each category in the dataset, so it can better solve the imbalance problem between buildings and backgrounds; Represents the Laplacian convolution kernel used to extract the boundary between the predicted label and the true label. Represents the binary cross entropy loss for edge pixel supervision.

[0062] This embodiment selects advanced mainstream remote sensing image building extraction networks for comparative experiments, including FCN, UNet, SegNet, DeepLabV3, HRNet, Segfomer, UNetFormer, etc. Each network uses the same settings for training and testing.

[0063] See also Figure 5 , where the area encircled by the dotted line is the comparison area of ​​special concern. In general, the dual-branch remote sensing image building extraction network based on the fusion of local and global features proposed in this embodiment is superior to other comparison networks in the task of remote sensing image building extraction. Figure 5 (a) There are a large number of dense small buildings. The proposed network has better detail processing capabilities and can more accurately focus on local features such as texture and shape of small buildings; Figure 5 (b) The edge of a large building is identified. The building edges extracted by other networks all show unexpected jagged fluctuations to varying degrees. However, the network proposed in the present invention can smooth the building edges well, making them closer to the actual situation, thanks to the feature refinement enhancement module FREM and the edge supervision loss function. Figure 5 In (c), the color of the building and the background runway are very similar. Other comparison networks misclassify the background as a building to varying degrees. However, the proposed network has better global context modeling capabilities and can accurately distinguish the two even when the color texture of the building and the background are very close. Figure 5 In (d), there is a small building next to the large warehouse, and the scales of the two buildings are very different. The network proposed in this paper has better multi-scale feature extraction capabilities, which can effectively focus on the edges of buildings of different scales and enhance the correlation between regions. Therefore, buildings with large scale differences can be accurately extracted; Figure 5 (e) There are a large number of buildings with complex shapes. Comparative networks such as Segformer and DeeplabV3 all have adhesion between buildings and the extraction of corridor-shaped buildings is mostly discontinuous. The network proposed in the present invention fully integrates global context information and local context information, and performs feature enhancement and feature refinement. It can still effectively extract buildings under the conditions of background noise interference and complex foreground texture and shape.

[0064] Example 2 The present invention also provides a dual-branch remote sensing image building extraction system based on the fusion of local and global features, comprising: A data acquisition module, used to acquire remote sensing images containing buildings; Building extraction module: Use the trained building extraction network model to extract buildings from remote sensing images. In addition, the system also includes a network training module.

[0065] During the training process, the data acquisition module is used to obtain remote sensing images containing buildings, construct a remote sensing building dataset, and preprocess the dataset.

[0066] The network training module is used to train a pre-built building extraction network using the acquired remote sensing building data set to obtain a trained building extraction network, wherein the building extraction network includes a dual-branch structure encoder, a decoder, a cross-feature attention fusion module and a feature refinement enhancement module.

[0067] Example 3 In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0068] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0069] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0070] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the remote sensing image building extraction method based on local-global features in the above embodiment; the processor may load and execute the following steps: Acquire a remote sensing image containing a building, input the remote sensing image into a trained building extraction network model, and extract the building in the remote sensing image; The building extraction network model adopts a dual-branch remote sensing image building network that integrates local and global features. The building extraction network model includes a dual-branch structure encoder and a cross-feature attention fusion module. The dual-branch structure encoder includes a CNN branch and a Transformer branch. The CNN branch is used to extract local features, and the Transformer branch is used to extract global features. The cross-feature attention fusion module is used to fuse the features extracted by the CNN branch and the Transformer branch.

[0071] Example 4 See also Figure 6 , the terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the remote sensing image building extraction method based on local-global features in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the computing system composed of the remote sensing image building extraction method based on local-global features are implemented. To avoid repetition, it is not described one by one here.

[0072] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 6 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0073] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0074] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 60.

[0075] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0076] Any reference to a memory, database or other medium used in the embodiments provided in this application may include at least one of a non-volatile and a volatile memory. Non-volatile memory may include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical memory, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. Volatile memory may include a random access memory (RAM) or an external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0077] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0078] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for extracting buildings from remote sensing images based on local-global features, characterized in that: include: Acquire a remote sensing image containing a building, input the remote sensing image into a trained building extraction network model, and extract the building in the remote sensing image; The building extraction network model adopts a dual-branch remote sensing image building network that integrates local and global features. The building extraction network model includes a dual-branch structure encoder, a cross-feature attention fusion module, and a feature refinement enhancement module; The dual-branch structure encoder includes a CNN branch and a Transformer branch, the CNN branch is used to extract local features, and the Transformer branch is used to extract global features; The cross-feature attention fusion module is used to fuse the features extracted by the CNN branch and the Transformer branch; The feature refinement enhancement module is used to enhance the edge details, channel dimensions, and multi-scale dimensions of building features in the input feature map.

2. The method for extracting buildings from remote sensing images based on local-global features according to claim 1, characterized in that: The CNN branch includes four convolution stages, each of which is used to gradually extract local texture and semantic information of the remote sensing image and establish local interactions; each convolution stage includes at least two convolution layers, and each convolution layer is connected to a batch normalization layer and a linear rectification unit.

3. The method for extracting buildings from remote sensing images based on local-global features according to claim 1, characterized in that: The Trasnformer branch includes four hybrid Trasnformer blocks; the hybrid Trasnformer block includes a number of multi-head self-attentions, a hybrid feedforward neural network and an overlapping block merging unit; the multi-head self-attention is used to capture features in remote sensing images to obtain feature maps; the hybrid feedforward neural network is used to enhance the features learned by the multi-head self-attention; Overlapping block merging units are used to downsample feature maps.

4. The method for extracting buildings from remote sensing images based on local-global features according to claim 1, characterized in that: The cross-feature attention fusion module includes a channel attention fusion branch and a spatial attention fusion branch. The channel attention fusion branch is used to perform a global average pooling operation on the input feature map, and process the channel through a one-dimensional convolution layer with an adaptive size convolution kernel and a Sigmoid activation function to obtain features with redistributed weights on the channel; the spatial attention fusion branch is used to perform a preliminary fusion on the input feature map to obtain preliminary fusion features, perform global average pooling and global maximum pooling on the preliminary fusion features, and splice the channel dimensions to obtain fusion features with redistributed weights.

5. The method for extracting buildings from remote sensing images based on local-global features according to any one of claims 1 to 4, characterized in that: The building extraction network model also includes a decoder, which includes four upsampling modules, each of which includes a bilinear interpolation upsampling strategy and two convolutional layers, and each convolutional layer is connected to a batch normalization layer and a linear rectification unit.

6. The method for extracting buildings from remote sensing images based on local-global features according to claim 1, characterized in that: The feature refinement and enhancement module is inserted into the decoder to enhance the edge details, channel dimensions and multi-scale dimensions of the building features in the input feature map.

7. The method for extracting buildings from remote sensing images based on local-global features according to claim 6, characterized in that: The feature refinement enhancement module includes a multi-scale enhancement path unit, a channel enhancement path unit and a deformable adaptive boundary enhancement path unit; The multi-scale enhancement path unit is used to extract multi-scale feature information of buildings in remote sensing images to obtain multi-scale enhancement features. The multi-scale enhancement path unit includes four hole convolution layers with different expansion rates. The channel enhancement path unit generates channel attention weights using global average pooling, and enhances the channels of the input features according to the channel attention weights to obtain channel enhancement features; The deformable adaptive boundary enhancement path unit is used to enhance edge details of channel enhancement features and multi-scale enhancement features. The deformable adaptive boundary enhancement path unit includes a batch normalization layer, a convolution layer of a linear rectification unit, and a deformable convolution layer.

8. The method for extracting buildings from remote sensing images based on local-global features according to claim 1, characterized in that: The training steps of the building extraction network model include: Batch acquiring remote sensing images containing buildings, and constructing a remote sensing building data set based on the acquired remote sensing images; the remote sensing images in the remote sensing building data set are subjected to labeling processing; Preprocess and enhance the remote sensing images in the dataset; Using the processed remote sensing building dataset, the building extraction network model will be trained end-to-end to obtain a trained building extraction network model.

9. The method for extracting buildings from remote sensing images based on local-global features according to claim 8, characterized in that: The processed dataset will be used to train the building extraction network model end-to-end, including: The processed remote sensing building dataset is used to iteratively optimize the pre-built building extraction network by back propagation in combination with the boundary supervision joint loss function to obtain the trained network; the boundary supervision joint loss function is: In the formula, L is the boundary supervision joint loss function; represents the predicted label, represents the true label, represents the cross entropy loss, which is used to measure the difference between the predicted probability of each category of the pixel and the true label; represents the dice loss, which is calculated based on the overlap coefficient between the predicted segmentation result and the true segmentation result; Represents the Laplacian convolution kernel used to extract the boundary between the predicted label and the true label; Represents the binary cross entropy loss for edge pixel supervision.

10. A system for extracting buildings from remote sensing images based on local-global features, used to implement the method for extracting buildings from remote sensing images based on local-global features as claimed in any one of claims 1 to 9, characterized in that: include: A data acquisition module, used to acquire remote sensing images containing buildings; Building extraction module: Use the trained building extraction network model to extract buildings from remote sensing images; The building extraction network model adopts a dual-branch remote sensing image building network that integrates local and global features. The building extraction network model includes a dual-branch structure encoder, a cross-feature attention fusion module, and a feature refinement enhancement module; The dual-branch structure encoder includes a CNN branch and a Transformer branch, the CNN branch is used to extract local features, and the Transformer branch is used to extract global features; The cross-feature attention fusion module is used to fuse the features extracted by the CNN branch and the Transformer branch; The feature refinement enhancement module is used to enhance the edge details, channel dimensions, and multi-scale dimensions of building features in the input feature map.

Images