Multi-task learning building extraction method and system based on mutual information constraint

Abstract

The invention belongs to the technical field of urban remote sensing monitoring, and particularly relates to a multi-task learning building extraction method and system based on mutual information constraint. The invention designs a mutual information constraint multi-task learning network for accurately extracting building footprints from remote sensing images to solve the problem that the automation of building footprint extraction is hindered due to incomplete and fuzzy division caused by an existing building extraction method. The mutual information constraint multitask learning network introduces a parallel context-aware structure to capture global and detailed architectural features. In addition, a Fourier mutual information balance module is integrated to promote interaction and fusion of multi-scale context information. A multi-task learning strategy and a mutual information constraint multi-task learning network are utilized to extract building masks and contours at the same time, and a mutual information loss function is adopted to enhance information exchange between the two tasks. The method not only can restrain the shape of the building, but also can improve the ability of the model to refine the boundary of the building.

Description

Technical Field

[0001] The invention belongs to the technical field of urban remote sensing monitoring, and in particular relates to a multi-task learning building extraction method and system based on mutual information constraints. Background Art

[0002] As the core manifestation of the urbanization process, the rapid and accurate extraction of building location and shape information from high-resolution remote sensing images has become a basic requirement in urban planning, resource management, and digital city modeling. However, due to the complex diversity of building forms, dense distribution and heterogeneity of the surrounding environment, obtaining clear and complete buildings from high-resolution remote sensing images faces severe challenges.

[0003] Building extraction methods based on hand-crafted features or shallow models are difficult to fully exploit the rich contextual information and multi-scale features in images, so their performance is limited in complex scenes. In recent years, deep learning-based technologies have achieved great success in the field of computer vision. In particular, the convolutional neural network (CNN) method and the self-attention mechanism VisionTransformer (ViT) method have performed well in local detail feature extraction and global semantic feature extraction, respectively. Their powerful feature learning capabilities have also enabled them to demonstrate excellent performance in remote sensing image information extraction tasks. However, the building extraction task not only needs to capture large-scale global semantic information to determine the overall shape and location of the building, but also needs to learn fine-grained local detail features to accurately detect the building boundaries. A single convolutional neural network or visual transformer network is difficult to meet the differentiated requirements of building extraction tasks for multiple feature scales.

[0004] To this end, many studies have attempted to introduce boundary regularization or constraint networks (such as guided filtering, conditional random fields, etc.), or extract edge features through additional lightweight convolutional neural networks to better capture the contours and structures of buildings. These post-processing operations and additional optimization networks usually require complex design and calculation, resulting in increased computational cost and model complexity. In this context, multi-task learning has attracted widespread attention as a potential research direction. Multi-task learning improves the performance of the model by sharing features and solving multiple related tasks simultaneously without increasing model complexity and parameters. In the field of remote sensing building extraction, by jointly training related auxiliary tasks (such as edge detection, object detection, etc.) with the main building segmentation task, the model can learn complementary feature knowledge from different tasks and capture richer contextual semantic information, thereby improving the accuracy and robustness of building extraction.

[0005] Deep learning methods perform well in spectral and temporal feature extraction. However, due to the lack of optimized design of multi-task learning feature extraction networks and effective knowledge transfer mechanisms between tasks, the application of existing multi-task learning frameworks in remote sensing building extraction is still insufficient. The main shortcomings are as follows:

[0006] 1) In terms of the design of weight-sharing feature extraction networks, weight-sharing feature extraction networks based only on convolutional neural networks (CNN) or visual attention transformation (ViT) cannot fully meet the different requirements of different tasks for feature scales. Some studies have tried to adopt the joint structure of CNN-ViT, connecting them in series or in parallel to fully utilize their advantages. However, due to the insufficient interaction and conflict between global semantic information and local spatial information, the building extraction task has deficiencies in the accuracy of boundary details and the regularity of overall shape.

[0007] 2) The lack of knowledge transfer between tasks further limits the efficiency and accuracy of building extraction. Although different tasks share feature representations to a certain extent, they are often simply superimposed or cascaded, lacking an effective mechanism to explicitly model and maximize the mutual flow of information between different tasks, which may hinder the full transfer of knowledge between tasks. This lack of knowledge transfer not only reduces the model's opportunities to learn and improve from related tasks, but also limits the model's adaptability and generalization ability when dealing with complex scenarios. Summary of the invention

[0008] The purpose of the present invention is to overcome the problems of incomplete segmentation and irregular boundaries in building segmentation methods in traditional technologies, and to provide a multi-task learning building extraction method based on mutual information constraints.

[0009] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0010] The present invention provides a multi-task learning building extraction method based on mutual information constraint, comprising the following steps:

[0011] S1. Obtain high-resolution remote sensing images and obtain sample information;

[0012] S2. Build a multi-task learning network structure based on mutual information constraints through sample information;

[0013] S3. Extract the spatial and spectral characteristics of buildings through functions, and then extract the buildings in the remote sensing image.

[0014] Furthermore, in step S1, the acquisition of sample information includes remote sensing data preprocessing, analyzing building remote sensing image interpretation marks and constructing a sample library.

[0015] Furthermore, in step S2, the sub-steps of building the network structure include:

[0016] 1) The network adopts a context information parallel perception structure in the shared weight feature extraction stage, and obtains context features of different scales in four stages through the local context encoder (LCE) and the global context encoder (GCE);

[0017] 2) In the decoding stage, a four-stage pyramid upsampling decoder corresponding to the encoder is used to aggregate the global and local context features of each stage;

[0018] 3) Use the Fourier mutual information balancing module to enhance the multi-scale context information exchange within the model;

[0019] 4) Use the mutual information loss function to improve the information interaction between the two tasks.

[0020] Furthermore, in step 1), the local context encoder (LCE) first extracts initial image features using a 7×7 convolution with a stride of 2, and then uses a 3×3 maximum pooling layer with a stride of 2; these features are further refined by four residual blocks, which perform deeper learning of detail features; each residual block consists of consecutive 3×3 convolutions, batch normalization, and ReLU activation functions; skip connections are used within these residual blocks to maintain information flow and alleviate the gradient vanishing problem.

[0021] Furthermore, in step 1), the global context encoder (GCE) includes a PE (Patch Embedding) module, a PM (Patch Merging) module and a ViT (Vision Transformer) module; in the PE stage, two 3×3 convolutional layers divide the input image into overlapping blocks, while expanding the channel size to 96 and reducing the resolution to one-quarter of the original image; the ViT module consists of a window-based linear multi-head self-attention (WLMHSA) module, C-MLP, two batch normalization layers and two residual connections, focusing on global context extraction; in the PM stage, a 2×2 convolutional layer is used to further reduce the resolution by half while doubling the number of channels; this design enables the global context encoder to extract rich global features across multiple levels, thereby fully understanding complex visual tasks.

[0022] Furthermore, in step 2), the pyramid upsampling decoder corresponds to the four stages of the encoder, and adopts a progressive upsampling strategy to effectively fuse the multi-scale feature maps from the encoder network and generate shared semantic features; the pyramid upsampling decoder first uses 1×1 convolution to unify the feature channels into 384, and then gradually upsamples the features and adds them element-wise with the previous level features to ensure that the deep semantic information is effectively propagated to the shallow features; finally, the features of all scales are upsampled and fused to obtain a feature map rich in multi-level semantic information.

[0023] Furthermore, in step 3), the Fourier mutual information balancing module is used as an information bridge to fuse global context and local context information, and its construction steps include:

[0024] The Fourier mutual information balancing module first transforms the global and local context features F(x, y) = {F local (x,y), F global (x, y)} is converted to the frequency domain by Fourier transform Where (x, y) and (u, v) represent the row and column indices in the spatial and frequency domains, respectively:

[0025]

[0026] Local context information in the frequency domain Perform point multiplication with the adaptive high-pass filter H(u, v; θ) to filter out redundant low-frequency components while retaining high-frequency information that is critical for detail recognition, and obtain the filtered frequency domain information Then, the processed frequency domain signal is converted back to the spatial domain through inverse Fourier transform to obtain the local context feature F after high frequency enhancement. local (x, y):

[0027]

[0028] Similarly, the global context information in the frequency domain By performing a dot multiplication with the adaptive low-pass filter L(u, v; θ), the low-frequency information representing the overall semantics will be retained, while the high-frequency noise will be filtered out, and the filtered frequency domain information will be obtained. Then, the processed frequency domain signal is converted back to the spatial domain through inverse Fourier transform to obtain the local context feature F after high frequency enhancement. global (x, y):

[0029]

[0030] Among them, (H, W) are the height and width of the feature map respectively, θ is the cutoff frequency parameter, which is designed as a network parameter and learned during the deduction process; θ is a step function. When the normalized distance from the frequency component (u, v) to the center of the frequency domain is greater than the learnable cutoff frequency θ, the value of the step function is 1, and the frequency component is retained; otherwise, it is 0, and the frequency component is filtered out;

[0031] The edge attention feature map Att(x, y) is obtained by applying the Sigmoid function σ(x, y) to the local context features after high-frequency enhancement, which is used to guide the global information F global (x, y) and local information F local The fusion of (x, y) retains more local details in the edge area, while more global context information is adopted in the non-edge area; finally, through a convolution sequence f including batch normalization, ReLU activation and convolution layer conv (x, y) fuses these two context information to generate the final feature expression F FMIBM (x, y); This fused feature not only contains extensive semantic information of the global context, but also preserves the spatial and geometric details of the local context;

[0032] Att(x, y) = σ(F local (x,y)) (8)

[0033] F FMIBM (x, y) = f conv ((1-Att(x,y))·F global (x,y)+Att(x,y)·F local (x,y)) (9).

[0035] Furthermore, the mutual information constraint loss is used to explicitly model and optimize the correlation between different tasks, thereby aligning the tasks more closely:

[0036] Mutual information is an important concept in information theory, which is used to measure the correlation and mutual dependence between two random variables; specifically, for two random variables X and Y, the mutual information I(X; Y) is defined as the sum of their respective entropies minus their joint entropy:

[0037] I(X;Y)=H(X)-H(X|Y)=H(X)+H(Y)-H(X,Y) (10)

[0038] In the building extraction framework, the goal of the mutual information loss function is to maximize the mutual information between the predicted building mask and the building outline, that is, to minimize the negative of the mutual information; given the predicted building mask output P BM and the predicted building outline output PBB , first calculate their probability distribution P′ BM and P′ BB , respectively obtained by the softening operation Softmax:

[0039] P′ BM =Softmax(P BM ), P′ BB =Softmax(P BB ) (11)

[0040] Joint probability distribution P′ MB Approximation by dot product:

[0041]

[0042] Substitute these probability distributions into the definition of mutual information, calculate the mutual information, and take its inverse as the mutual information loss:

[0043]

[0044] Here, i, j are the position indices in the probability distribution graph, and for numerical stability, a small constant ∈ is added to the denominator and the logarithm parameter.

[0045] The present invention also provides a multi-task learning building extraction system based on mutual information constraints, comprising:

[0046] An acquisition module, wherein the acquisition module is used to acquire a remote sensing image to be extracted;

[0047] A preprocessing module, wherein the preprocessing module is used to preprocess the remote sensing image to be extracted to obtain a building sample;

[0048] A mutual information multi-task learning network extracts image space and spectral features through functions, performs adaptive feature selection and enhances input feature maps in channel and spatial directions, and then extracts buildings.

[0049] The beneficial effects of the present invention are:

[0050] The present invention designs a multi-task learning building extraction network based on mutual information constraints. The network uses the context information parallel perception structure to capture multi-scale context information, and aggregates global and local context information through the Fourier mutual information balance module, thereby meeting the needs of semantic information of different scales for different tasks. At the same time, the mutual information loss function enhances the information exchange between the building mask segmentation and building boundary segmentation tasks, which greatly improves the integrity and regularization of building segmentation.

[0051] Of course, any product implementing the present invention does not necessarily need to achieve all of the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0053] Figure 1 A schematic diagram of the structure of a high-resolution remote sensing building extraction network based on multi-task learning with mutual information constraints provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of the structure of a local encoder in the method provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the structure of a global encoder in the method provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the structure of a pyramid upsampling decoder in the method provided by an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of the structure of a Fourier mutual information balancing module in the method provided in an embodiment of the present invention;

[0058] Figure 6 A diagram showing the building extraction results of the method of an embodiment of the present invention implemented in a public dataset of drone buildings;

[0059] Figure 7 This is a graph of building extraction results from a satellite dataset implemented with the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Mutual Information is an important measure in information theory that describes the correlation between two random variables. It reflects how the uncertainty of the other random variable can be reduced after knowing one of the random variables. In the field of machine learning, mutual information is often used to measure the statistical correlation between input data and output features, and is applied to tasks such as feature selection or self-supervised representation learning. This feature is naturally in line with the needs of multi-task learning. Therefore, the inventors introduced it into the multi-task learning framework. By maximizing the mutual information between tasks, the mutual flow and migration of knowledge and features between different tasks are enhanced, so that each task can make full use of each other's auxiliary information to achieve a synergistic effect of "1+1>2".

[0062] Therefore, the present invention designs a multi-task learning high-resolution remote sensing building extraction framework based on mutual information constraints, which extracts buildings from high-resolution remote sensing images without using an additional edge segmentation branch network. In the proposed architecture, a parallel context-aware structure is used to extract multi-scale context, and a Fourier mutual information balance module is designed to aggregate the global and local context features of each stage. The decoding stage uses a pyramid upsampling module to process multi-scale information from the encoder and gradually restore the building features. At the same time, the inventor introduced building edge detection as an auxiliary task for building footprint extraction, and designed a mutual information loss function to enhance the information interaction between the two tasks, using building boundaries to constrain the extraction of buildings and improve the extraction of buildings.

[0063] The relevant specific embodiments of the present invention are as follows:

[0064] Embodiment 1

[0065] like Figure 1-Figure 7 As shown, the purpose of this embodiment is to provide a multi-task learning high-resolution remote sensing building extraction method based on mutual information constraints, with the core goal of "using mutual information to promote the interaction and fusion of multi-scale context information, enhance information exchange between tasks, and reduce the problems of incomplete segmentation and irregular boundaries in building extraction". A targeted deep learning algorithm is designed to improve the accuracy of building extraction and reduce spatial inconsistency in the mapping process. Under the guidance of this idea, a multi-task learning high-resolution remote sensing building extraction method based on mutual information constraints is designed.

[0066] This embodiment provides a deep learning method, which includes the following steps:

[0067] Step 1: Remote sensing image selection and sample preparation

[0068] The construction of the entire deep learning building sample database system includes target vector editing, attribute assignment, raster conversion, sample scale segmentation and enhancement, data set division, etc. The specific process is as follows:

[0069] 1) Vector Editing

[0070] According to the building crop interpretation signs formulated above, the vector boundary of the building is delineated by visual interpretation using the vector editing tools in ArcGIS software. The accuracy of vector editing will directly affect the quality of the image sample. Therefore, in order to improve the accuracy of the sample and the accuracy of the later model recognition, the error range of the delineation is strictly controlled during the delineation process to control the error to the pixel level (the boundary is accurate to a single pixel).

[0071] 2) Attribute assignment

[0072] Using the Calculate Field tool in ArcGIS software, a new attribute “gridcode” is added to the building vector data obtained during the vector editing process.

[0073] 3) Get the label

[0074] The acquisition of building labels mainly involves converting vector data after attribute assignment into raster data. In this study, the Feature to Raster tool in ArcGIS software was used to convert the edited vector data into raster data corresponding to the remote sensing image, and the obtained raster data must strictly match the corresponding image data.

[0075] 4) Sample scale segmentation and enhancement

[0076] Due to the limitation of computer hardware conditions, it is difficult to directly input images into the network. It is necessary to perform a certain degree of segmentation processing on the remote sensing images and the corresponding raster label data. This paper mainly uses the SplitRaster tool in ArcGIS software to segment the buildings and their corresponding labels. The images and label data are segmented in a size of 1024×1024, and multiple different cutting steps are taken to increase the data volume.

[0077] 5) Dataset division

[0078] The training data and test data are randomly selected in a ratio of 3:1, where the training set is used for model training and the validation set is used to supervise the convergence of the model. The validation data is based on the surveyed plot types and manual visual interpretation, combined with the complete pixel labels manually outlined in the image, to verify the accuracy of the model.

[0079] Step 2: Model network structure and module construction

[0080] 1) Network structure

[0081] The network adopts a context information parallel cognitive structure in the shared weight feature extraction stage to obtain context features of different scales in four stages through the local context encoder (LCE) and the global context encoder (GCE). The four-level pyramid upsampling structure corresponding to the encoder is used in the decoding stage to aggregate the global and local context features of each stage. The Fourier mutual information balancing module is used to enhance the multi-scale context information exchange within the model, and the mutual information loss function is introduced to improve the information interaction between the two tasks.

[0082] 2) Local Context Encoder

[0083] The local context encoder first extracts initial image features using a 7×7 convolution with a stride of 2, followed by a 3×3 max pooling layer with a stride of 2. These features are further refined by four residual blocks, which learn more detailed features. Each residual block consists of a series of 3×3 convolutions, batch normalization, and ReLU activation functions. Skip connections are used in these residual blocks to maintain information flow and alleviate the vanishing gradient problem, such as Figure 3 The LCE structure shown.

[0084] 3) Global Context Encoder

[0085] The global context encoder consists of a patch embedding (PE) module, a patch merging (PM) module, and a visual transformer (ViT) module. In the PE stage, two 3×3 convolutional layers split the input image into overlapping patches, while expanding the channel size to 96 and reducing the resolution to one-quarter of the original resolution. The ViT module consists of a window-based linear multi-head autofocus (WLMHSA) module, C-MLP, two batch normalization layers, and two residual connections, focusing on global context extraction. In the PM stage, a 2×2 convolutional layer is used to further reduce the resolution by half, while the number of channels is doubled. This design enables GCE to extract rich global features across multiple levels, thereby fully understanding complex visual tasks. Its structure is as follows Figure 4 shown.

[0086] 4) Pyramid upsampling decoder

[0087] The pyramid upsampling decoder corresponds to the four stages of the encoder, and uses a progressive upsampling strategy to effectively fuse the multi-scale feature maps from the encoder network and generate shared semantic features. This module first uses 1×1 convolution to unify the feature channels to 384, then gradually upsamples the features and adds them element-wise with the features of the previous level to ensure that deep semantic information is effectively propagated to shallow features. Finally, features of all scales are upsampled and fused to obtain feature maps rich in multi-level semantic information. Its structure is as follows Figure 4 shown.

[0088] 5) Fourier mutual information balance module

[0089] Although the global context branch is more accurate in semantics, it lacks spatial and geometric details, limiting its accuracy in complex scenes. In contrast, the local context branch provides more fine-grained information, but is accompanied by inevitable noise and redundant information. In order to achieve a dynamic balance between the two contexts and enhance the information flow between global semantics and local semantics, in this embodiment, a Fourier mutual information balance module is designed as an information bridge to fuse global context and local context information.

[0090] The Fourier mutual information balancing module first transforms the global and local context features F(x, y) = {F local (x,y), F global (x, y)} is converted to the frequency domain by Fourier transform Where (x, y) and (u, v) represent the row and column indices in the spatial and frequency domains, respectively:

[0091]

[0092] Local context information in the frequency domain Perform point multiplication with the adaptive high-pass filter H(u, v; θ) to filter out redundant low-frequency components while retaining high-frequency information that is critical for detail recognition to obtain the filtered frequency domain information Then, the processed frequency domain signal is converted back to the spatial domain through inverse Fourier transform to obtain the local context feature F after high frequency enhancement. local (x, y):

[0093]

[0094]

[0095] Similarly, the global context information in the frequency domain By performing a dot multiplication with the adaptive low-pass filter L(u, v; θ), the low-frequency information representing the overall semantics will be retained, while the high-frequency noise will be filtered out, and the filtered frequency domain information will be obtained. Then, the processed frequency domain signal is converted back to the spatial domain through inverse Fourier transform to obtain the local context feature F after high frequency enhancement. global (x, y):

[0096]

[0097] Where (H, W) are the height and width of the feature map, respectively, and θ is the cutoff frequency parameter, which is designed as a network parameter and learned during the deduction process. θ is a step function. When the normalized distance from the frequency component (u, v) to the center of the frequency domain is greater than the learnable cutoff frequency θ, the value of the step function is 1, and the frequency component is retained; otherwise, it is 0, and the frequency component is filtered out.

[0098] The edge attention feature map Att(x, y) is obtained by applying the Sigmod function σ(x, y) to the local context features after high-frequency enhancement, which is used to guide the global information F global (x, y) and local information F local The fusion of (x, y) retains more local details in the edge area, while more global context information is used in the non-edge area. Finally, through a convolution sequence f conv (x, y) (including batch normalization, ReLU activation and convolutional layers), the inventors fuse these two context information to generate the final feature expression F FMIBM (x, y). This fused feature not only contains extensive semantic information of the global context, but also preserves the spatial and geometric details of the local context.

[0099] Att(x,y)=ο(F local (x,y)) (8)

[0100] F FMIBM (x, y) = f conv ((1-Att(x,y))·F global (x,y)+Att(x,y)·F local (x,y)) (9)

[0101] 6) Mutual Information Constrained Loss

[0102] In a multi-task learning scenario, there are certain correlations and intrinsic connections between different tasks. Effectively utilizing this correlation can promote knowledge sharing and mutual gain between different tasks, which will help improve the performance and generalization ability of the overall model. Based on this consideration, this embodiment proposes a loss function based on mutual information constraints, namely, mutual information constraint loss (Mutual Information Loss), to explicitly model and optimize the correlation between different tasks, so as to align tasks more closely.

[0103] Mutual information is an important concept in information theory, which is used to measure the correlation and mutual dependence between two random variables. Specifically, for two random variables X and Y, mutual information I(X; Y) can be defined as the sum of their respective entropies minus their joint entropy:

[0104] I(X;Y)=H(X)-H(X|Y)=H(X)+H(Y)-H(X,Y) (10)

[0105] In the building extraction framework, the goal of the mutual information loss function is to maximize the mutual information between the predicted building mask and the building outline, that is, to minimize the negative of the mutual information. Given the predicted building mask output P BM and the predicted building outline output P BB The inventors first calculated their probability distribution P′ BM and P′ BB , which are obtained by the softening operation Softmax:

[0106] P′ BM =Softmax(P BM ), P′ BB =Softmax(P BB ) (11)

[0107] Joint probability distribution P′ MB It can be approximated by dot product:

[0108]

[0109] Substitute these probability distributions into the definition of mutual information, calculate the mutual information, and take its inverse as the mutual information loss:

[0110]

[0111] i, j are the position indices in the probability distribution graph, and for numerical stability, the inventors add a small constant ∈ to the denominator and the logarithm parameter.

[0112] For the content of this embodiment, the model accuracy is verified and analyzed.

[0113] 1. Verification of the accuracy of the method in this embodiment

[0114] In order to comprehensively evaluate the performance of the method proposed in this embodiment, comparative experiments were conducted with some advanced semantic segmentation methods on public drone building datasets (Massachusetts building dataset, WHU building dataset, and Inria aerial image labeling dataset). These methods include: CNN-based methods Unet, PSPnet, Deeplapv3+, Transformer-based methods STT, DC-Swin, CNN-Transformer combined methods UnetFormer, BulidFormer, BCTNet, and multi-task learning building extraction methods: CBRnet, LCSnet, BOMSCnet. This embodiment uses intersection over union (IoU), F1 score, Recall, and Precision as the main evaluation indicators to comprehensively evaluate the performance of different methods in building detection and segmentation tasks.

[0115] The results of the quantitative comparison are shown in Table 1, with the best performance for each metric marked in bold. Compared with all other methods, the method proposed in this example shows significant superiority on the UAV platform dataset. Some examples of qualitative comparison are as follows: Figure 6 As shown in the figure. The first two columns show the input image and the ground truth. The next six columns show the prediction results from Unet, Deeplapv3+, DC-Swin, BuildingFormer, and the method of this embodiment. The red circles show the differences in details between the different methods. Compared with other methods, the inventor's method can capture more complete buildings with complex structures, achieve better recognition of small buildings and buildings obscured by other objects such as trees and shadows, and suppress more interference from cement floors or plastic sheds. For example, in Figure 6 In the second row, the CNN-based method and the Transformer-based method cannot avoid the interference of the plastic shed in the image. Since the outline of the plastic shed is very similar to that of the building, it is easily mistaken for a building. In the method of this embodiment, the local features from CNN and the global information from Transformer are fused in parallel, so that the plastic shed can be distinguished from the building. However, since the feature fusion is only performed in the decoder part, many local details and global long-range dependencies cannot be fully integrated at each stage of the encoder process, resulting in inefficient expression of the extracted features. When faced with complex buildings, shadows, cluttered scene interference, etc., this may result in incomplete, missing or false recognition. It is worth noting that the Fourier mutual information balance module based on the method proposed in this embodiment can effectively provide boundary constraints for the building extraction task and improve the model's ability to recognize the boundaries of buildings with complex forms. These results demonstrate the superiority of the method of this embodiment in identifying buildings of various sizes and different appearances in complex scenes.

[0116] Table 1 Quantitative comparison with the most advanced methods on public datasets

[0117] Method F1(％) Precision(%) Recall(%) Iou(%) U-Net 84.88 86.99 82.87 73.73 PSPNet 80.32 82.89 77.91 67.11 DeepLabv3+ 82.29 84.41 80.83 72.35 MAPNet 83.39 86.84 80.2 71.51 DC-Swin 80.57 79.41 81.77 67.46 UnetFormer 84.94 86.22 83.17 74.83 BCTNet 85.74 87.57 83.99 75.04 BuildFormer 86.19 87.52 84.9 75.74 BOMSC-Net 85.13 86.64 83.68 74.71 CBRNet 85.42 86.5 84.36 74.55 This embodiment 86.51 86.73 86.29 76.23

[0118] II. Verification of the universality of the method in this embodiment

[0119] In order to verify the effectiveness of the high-resolution remote sensing building extraction method based on multi-task learning with mutual information constraints proposed in this embodiment, this embodiment has been extensively tested on satellite remote sensing images. The types of buildings tested cover a variety of complex scenes, from high-rise buildings in cities to low-rise buildings in rural areas, contiguous buildings in shanty towns, and factory buildings in industrial and mining areas. The diversity and complexity of these scenes provide a strong basis for verifying the generalization of the algorithm. Figure 7 As shown, for different types of buildings, the method proposed in this embodiment shows excellent extraction effect.

[0120] Urban high-rise buildings usually have relatively regular geometric shapes and dense arrangements, with small intervals between buildings. The shadow effects and complex structures of buildings place high demands on the extraction algorithm.

[0121] from Figure 7 It can be seen from the test results of (a) that the method of this embodiment can effectively identify and extract high-rise buildings in the city, especially in terms of the integrity of the building outline and the accuracy of the edge. Through the introduction of edge detection and mutual information loss function, the algorithm of this embodiment can accurately capture the geometric features of the building and maintain a high extraction accuracy even in a complex urban environment. Buildings in rural areas are usually sparsely distributed and irregular in shape. The roof materials and colors of such buildings are diverse, which increases the difficulty of extraction. In addition, the natural environment background in rural areas is complex, such as farmland, vegetation, etc., which can easily interfere with the extraction of buildings.

[0122] Figure 7 (b) shows the performance of the algorithm of this embodiment in extracting low-rise buildings in rural areas. The results show that the algorithm can effectively distinguish buildings from natural backgrounds and accurately extract buildings of various shapes. Buildings in shantytowns are densely packed and often have complex structures. Buildings in such areas have different shapes and are distributed irregularly.

[0123] Figure 7(c) shows the application effect of the method of this embodiment in extracting buildings in shantytowns. It can be seen that although the buildings in shantytowns are closely arranged and have complex structures, the method of this embodiment can still extract the boundaries of these buildings relatively completely. Factory buildings in industrial and mining areas usually have large roof areas and obvious geometric shapes, but there is also spectral confusion caused by a variety of building materials and colors.

[0124] Figure 7 (d) shows the building extraction results in the industrial and mining area scenario. The algorithm of this embodiment performs well in this scenario and can accurately distinguish factory buildings from other industrial facilities or open spaces. In the extraction of these large-area buildings, the edge detection task makes the entire extraction process more robust by strengthening the extraction of building contours.

[0125] The above experimental results show that the multi-task learning high-resolution remote sensing building extraction method based on mutual information constraints proposed in this embodiment shows excellent extraction performance when facing a variety of complex scenes and different types of buildings. Through the joint optimization of edge detection and mutual information loss function, the model proposed in this embodiment can not only accurately identify the boundaries of buildings, but also maintain a high extraction accuracy in complex remote sensing images. This method has shown strong adaptability in generalization tests of different types of buildings, proving its wide application potential in the field of cross-platform remote sensing building extraction.

[0126] Embodiment 2

[0127] This embodiment provides a multi-task learning high-resolution remote sensing building extraction system based on mutual information constraints, including:

[0128] An acquisition module is configured to: acquire a remote sensing image to be extracted;

[0129] A preprocessing module is configured to: preprocess the remote sensing image to be extracted to obtain a building sample;

[0130] A multi-task learning building extraction network based on mutual information constraints is configured to: extract image spatial and spectral features through functions, perform adaptive feature selection and enhance input feature maps in channel and spatial directions, and then extract buildings.

[0131] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-task learning building extraction method based on mutual information constraints, characterized in that: The steps include: S1. Obtain high-resolution remote sensing images and obtain sample information; S2. Build a multi-task learning network structure based on mutual information constraints through sample information; S3. Extract the spatial and spectral characteristics of buildings through functions, and then extract the buildings in the remote sensing image.

2. The multi-task learning building extraction method based on mutual information constraint according to claim 1 is characterized in that: In step S1, the acquisition of sample information includes remote sensing data preprocessing, analyzing building remote sensing image interpretation marks and constructing a sample library.

3. The multi-task learning building extraction method based on mutual information constraint according to claim 1 is characterized in that: In step S2, the sub-steps of building the network structure include: 1) The network adopts a context information parallel perception structure in the shared weight feature extraction stage, and obtains context features of different scales in four stages through local context encoder and global context encoder; 2) In the decoding stage, a four-stage pyramid upsampling decoder corresponding to the encoder is used to aggregate the global and local context features of each stage; 3) Use the Fourier mutual information balancing module to enhance the multi-scale context information exchange within the model; 4) Use the mutual information loss function to improve the information interaction between the two tasks.

4. The multi-task learning building extraction method based on mutual information constraint according to claim 3 is characterized in that: In step 1), the local context encoder first extracts initial image features using a 7×7 convolution with a stride of 2, and then uses a 3×3 maximum pooling layer with a stride of 2; these features are further refined by four residual blocks, which learn the detail features more deeply; each residual block consists of consecutive 3×3 convolutions, batch normalization, and ReLU activation functions; skip connections are used within these residual blocks to maintain information flow and alleviate the gradient vanishing problem.

5. The multi-task learning building extraction method based on mutual information constraint according to claim 3 is characterized in that: In step 1), the global context encoder (GCE) includes a PE module, a PM module and a ViT module; in the PE stage, two 3×3 convolutional layers divide the input image into overlapping blocks, while expanding the channel size to 96 and reducing the resolution to one quarter of the original image; The ViT module consists of a window-based linear multi-head self-attention module, C-MLP, two batch normalization layers, and two residual connections, focusing on global context extraction; In the PM stage, a 2×2 convolutional layer is used to further reduce the resolution by half while doubling the number of channels; this design enables the global context encoder to extract rich global features across multiple levels, thereby fully understanding complex visual tasks.

6. The multi-task learning building extraction method based on mutual information constraint according to claim 3 is characterized in that: In step 2), the pyramid upsampling decoder corresponds to the four stages of the encoder, and adopts a progressive upsampling strategy to effectively fuse the multi-scale feature maps from the encoder network and generate shared semantic features; the pyramid upsampling decoder first uses 1×1 convolution to unify the feature channels into 384, and then gradually upsamples the features and adds them element-wise with the previous level features to ensure that the deep semantic information is effectively propagated to the shallow features; finally, the features of all scales are upsampled and fused to obtain a feature map rich in multi-level semantic information.

7. The multi-task learning building extraction method based on mutual information constraint according to claim 3 is characterized in that: In step 3), the Fourier mutual information balance module is used as an information bridge to fuse global context and local context information, and its construction steps include: The Fourier mutual information balancing module first transforms the global and local context features F(x, y) = {F local (x,y), F global (x, y)} is converted to the frequency domain by Fourier transform Where (x, y) and (u, v) represent the row and column indices in the spatial and frequency domains, respectively: Local context information in the frequency domain Perform point multiplication with the adaptive high-pass filter H(u, v; θ) to filter out redundant low-frequency components while retaining high-frequency information that is critical for detail recognition to obtain the filtered frequency domain information Then, the processed frequency domain signal is converted back to the spatial domain through inverse Fourier transform to obtain the local context feature F after high frequency enhancement. local (x, y): Similarly, the global context information in the frequency domain By performing a dot multiplication with the adaptive low-pass filter L(u, v; θ), the low-frequency information representing the overall semantics will be retained, while the high-frequency noise will be filtered out, and the filtered frequency domain information will be obtained. Then, the processed frequency domain signal is converted back to the spatial domain through inverse Fourier transform to obtain the local context feature F after high frequency enhancement. global (x, y): Among them, (H, W) are the height and width of the feature map respectively, θ is the cutoff frequency parameter, which is designed as a network parameter and learned during the deduction process; θ is a step function. When the normalized distance from the frequency component (u, v) to the center of the frequency domain is greater than the learnable cutoff frequency θ, the value of the step function is 1, and the frequency component is retained; otherwise, it is 0, and the frequency component is filtered out; The edge attention feature map Att(x, y) is obtained by applying the Sigmoid function σ(x, y) to the local context features after high-frequency enhancement, which is used to guide the global information F global (x, y) and local information F local The fusion of (x, y) retains more local details in the edge area, while more global context information is adopted in the non-edge area; finally, through a convolution sequence f including batch normalization, ReLU activation and convolution layer conv (x, y) fuses these two context information to generate the final feature expression F FMIBM (x, y); This fused feature not only contains extensive semantic information of the global context, but also preserves the spatial and geometric details of the local context; That(x,y)=σ(F local (x,y)) (8) F FMIBM (x, y)=f conv ((1-At(x,y)) F global (x,y)+ATT(x,y) F local (x, y)) (9).

8. The multi-task learning building extraction method based on mutual information constraint according to claim 3 is characterized in that: Mutual information constraint loss is used to explicitly model and optimize the correlation between different tasks, thereby aligning the tasks more closely: Mutual information is an important concept in information theory, which is used to measure the correlation and mutual dependence between two random variables; specifically, for two random variables X and Y, the mutual information I(X; Y) is defined as the sum of their respective entropies minus their joint entropy: I(X;Y)=H(X)-H(X|Y)=H(X)+H(Y)-H(X,Y) (10) In the building extraction framework, the goal of the mutual information loss function is to maximize the mutual information between the predicted building mask and the building outline, that is, to minimize the negative of the mutual information; given the predicted building mask output P BM and the predicted building outline output P BB , first calculate their probability distribution P′ BM and P′ BB , respectively obtained by the softening operation Softmax: P′ BM =Softmax(P BM ),P′ BB =Softmax(P BB ) (11) Joint probability distribution P′ MB Approximation by dot product: Substitute these probability distributions into the definition of mutual information, calculate the mutual information, and take its inverse as the mutual information loss: where i, j are the position indices in the probability distribution graph, and a constant ∈ is added to the denominator and the parameter of the logarithm for numerical stability.

9. A multi-task learning building extraction system based on mutual information constraints, characterized in that: include: An acquisition module, wherein the acquisition module is used to acquire a remote sensing image to be extracted; A preprocessing module, wherein the preprocessing module is used to preprocess the remote sensing image to be extracted to obtain a building sample; A mutual information multi-task learning network extracts image space and spectral features through functions, performs adaptive feature selection and enhances input feature maps in channel and spatial directions, and then extracts buildings.

Images