Building image segmentation method and system based on SAM-GAN

Abstract

The application discloses a building image segmentation method and system based on a SAM-GAN, and relates to the technical field of image data processing. The method comprises the following steps: acquiring an original remote sensing image, and preprocessing the original remote sensing image to obtain a preprocessed image; generating a preliminary building segmentation result based on a preset SAM-GAN model according to the preprocessed image; the SAM-GAN model comprises a SAM generator and a double discriminator network, the double discriminator network is used for optimizing the SAM generator, and the double discriminator network is a double discriminator architecture that fuses semantic and topological constraints; and performing geometric rule adaptive optimization post-processing on the preliminary building segmentation result to obtain a final building segmentation result. The application improves the accuracy, quality and generalization ability of the building image segmentation result.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and specifically to a building image segmentation method and system based on SAM-GAN. Background Technology

[0002] With the continuous advancement of Earth observation remote sensing technology and the ongoing innovation of high-resolution remote sensing satellite technology, the acquired imagery has seen continuous improvement in both scale and quality. This has made the automated identification and segmentation of specific ground features, such as buildings and roads, using high-resolution imagery a hot topic in many scientific research and engineering fields. Buildings, as vital spaces for human life and key ground features occupying a central position on maps, possess significant research value. The automatic segmentation of buildings in images is not only valuable in critical engineering applications such as topographic mapping and door-to-door surveys, but also has profound theoretical implications: the structural height of buildings, closely related to human activities, plays a typical role in evaluating statistical methods, machine learning algorithms, and their development in image segmentation, target recognition, and regularization. Furthermore, building segmentation using high-resolution imagery provides crucial reference data and technical support for multiple fields such as disaster assessment, change monitoring, and the development of smart cities.

[0003] For decades, traditional techniques have attempted to segment buildings using textures, lines, shadows, and more complex, experience-based features. However, these methods have failed to achieve full automation of building segmentation, and no commercial software has yet been available. Recent advances in deep learning technology have spurred a new wave of research into automating building image segmentation.

[0004] While current high-resolution remote sensing image building segmentation widely employs deep learning (such as UNet, SAM, and other basic segmentation models) and generative adversarial networks (GANs), it still faces the following technical bottlenecks and shortcomings:

[0005] 1. The segmentation model output mask lacks realism and structural consistency.

[0006] End-to-end segmentation networks, exemplified by patent publication number CN114187520A, while improving the appearance quality of masks, generally suffer from issues such as mask boundary burrs, false holes, breaks, or insufficient closure. Due to the lack of explicit constraints on the segmentation contour structure, the masks output by the model... Difficult to match with real labels Structural alignment has resulted in the following error metrics remaining persistently high: The structural IoU is significantly lower than the pixel IoU. The discriminator uses a single pixel-level consistency loss, which makes it difficult to capture the topological properties of the building outline (such as the number of connected components, Euler number, etc.).

[0007] 2. The post-processing threshold for building outlines is rigid, resulting in limited generalization and automation.

[0008] Traditional DP (Douglas-Peucker algorithm) processes, such as polygon simplification, short side clipping, angle threshold trimming, and main direction alignment, largely rely on manual experience or statically set hyperparameters, such as the short side ratio threshold. Sharp angle threshold obtuse angle threshold This type of fixed threshold method experiences a significant performance drop when migrating across resolutions and scenes, and it struggles to automatically adapt to new cities and sensor conditions, lacking adaptive and intelligent optimization capabilities.

[0009] 3. GAN adversarial segmentation lacks structural innovation and is unable to significantly improve fine-grained segmentation quality.

[0010] In recent years, the ideas of GAN+ segmentation (such as pix2pix, SAM+GAN) have been widely published (see PPA-SAM, SAM-GAN for XCT, etc.), but most of them remain at the stage of single discriminator pixel adversarial, and lack targeted improvement for the topological structure (closure, no false holes, small target fracture) of remote sensing building segmentation.

[0011] In view of the above, this application is hereby submitted. Summary of the Invention

[0012] The technical problem this invention aims to solve is that existing building image segmentation methods suffer from insufficient mask realism and structural consistency, resulting in inaccurate segmentation results and poor segmentation quality. This invention aims to provide a building image segmentation method and system based on SAM-GAN, proposing an integrated innovative scheme of shared dual discriminator SAM-GAN and adaptive meta-thresholding post-processing. This invention integrates Generative Adversarial Networks (GAN), Large-Scale Segmentation Basic Model (SAM), and geometric rule adaptive optimization technology to achieve a dual improvement in segmentation realism and topological consistency, thereby enhancing the accuracy, quality, and generalization ability of building image segmentation results.

[0013] This invention is achieved through the following technical solution:

[0014] In a first aspect, the present invention provides a building image segmentation method based on SAM-GAN, the method comprising:

[0015] Acquire raw remote sensing images and preprocess them to obtain preprocessed images;

[0016] Based on the preprocessed image, preliminary building segmentation results are generated using a pre-defined SAM-GAN model. The SAM-GAN model includes a SAM generator and a dual discriminator network. The dual discriminator network is used to optimize the SAM generator and is a dual discriminator architecture that integrates semantic and topological constraints.

[0017] The preliminary building segmentation results are post-processed by adaptive optimization of geometric rules to obtain the final building segmentation results.

[0018] Furthermore, the SAM generator includes an encoder and a decoder, with the encoder used to perform preliminary feature extraction on the preprocessed image;

[0019] The decoder is used to reconstruct the initially extracted features and decode the output to produce preliminary building segmentation results. The preliminary building segmentation results include preliminary masks, low-order semantic tensors, and boundary distance feature maps.

[0020] Furthermore, the encoder is a ViT-H encoder, using the ViT-H Transformer structure disclosed by the SAM model, and the training of the first 10 epochs is frozen to ensure its zero-shot generalization performance.

[0021] The decoder consists of the original 3-layer Transformer decoder of the SAM model plus 2 layers of 5×5 depth separable convolutional layers to enhance the ability to resolve building details and textures.

[0022] Furthermore, the dual discriminator network includes a semantic discriminator and a topology discriminator, and the semantic discriminator and the topology discriminator share a single encoder;

[0023] The semantic discriminator includes a PatchGAN-style real / fake discriminator head and a multi-class semantic discriminator head. The input of the semantic discriminator is a combination of a preliminary mask, a low-order semantic tensor, and a preprocessed image grayscale image. The output of the semantic discriminator is a real / fake image and a semantic probability.

[0024] The input to the topology discriminator is a preliminary mask and a boundary distance feature map, and the output of the topology discriminator is the topology level.

[0025] Furthermore, the loss function of the dual discriminator network for:

[0026] ;

[0027] ;

[0028] ;

[0029] In the formula, The loss function for the SAM generator; For semantic discriminator The loss function; For topology discriminator The loss function; The weights are dynamic, varying with the training cycle. It increases exponentially and is used to balance the losses of the semantic discriminator and the topological discriminator.

[0030] Furthermore, the training phase of the SAM-GAN model includes:

[0031] Freeze Initialization Phase: This phase completely freezes the pre-trained ViT-H encoder and topology discriminator. Only train the decoder and semantic discriminator Relevant parameters;

[0032] Semantic adversarial training phase: In this phase, all SAM generator parameters are unfrozen, and the semantic discriminator is started. And topology discriminator It remains frozen;

[0033] Topology reinforcement training phase: This phase unfreezes the topology discriminator. This causes the loss of the topology discriminator. Explicitly differentiable constraints are imposed on the Euler characteristic difference and the connected component difference of the mask.

[0034] Furthermore, post-processing includes raster vectorization, vertex simplification, removal of short edges, removal of sharp and obtuse angles, main direction alignment, and vector rasterization;

[0035] Raster to Vector: Convert the initial mask of the preliminary building segmentation results from raster format to vector format to obtain the first intermediate result;

[0036] Vertex simplification: Vertex simplification is performed on the first intermediate result based on the DP algorithm, and the simplification threshold varies with the short side ratio threshold provided by the reinforcement learning action interface. Dynamic adjustments were made to obtain a second intermediate result;

[0037] Short edge deletion: Delete short edges from the second intermediate result to obtain the third intermediate result;

[0038] Sharp and obtuse angles are removed: Sharp angle thresholds are provided by the reinforcement learning action interface. With obtuse angle threshold Remove the sharp and obtuse angles from the third intermediate result to obtain the fourth intermediate result;

[0039] Main direction alignment: Based on the fourth intermediate result, the main direction of the building outline is determined; and the angle between each edge of the polygon and the main direction is judged and rotated and adjusted to obtain the fifth intermediate result;

[0040] Vector rasterization: The fifth intermediate result is re-rasterized into a binary mask image, which is the final building segmentation result.

[0041] Furthermore, the short side ratio threshold Sharp corner threshold With obtuse angle threshold Automatic optimization is performed through a geometric threshold policy network using reinforcement learning and meta-learning, including:

[0042] Construct a threshold based on the proportion of the shorter side Sharp corner threshold With obtuse angle threshold The resulting action space;

[0043] After the actions in the action space are executed, the environment calls the black-box post-processing to perform raster vectorization, vertex simplification, short edge removal, sharp and obtuse corner removal, main direction alignment and vector rasterization in sequence to obtain a regularized mask.

[0044] Calculate the reward based on the regularization mask. , The first item The second term represents the intersection-union ratio (IoU) of the polygon and the actual mask; Penalty for the difference in the number of vertices (simplicity penalty). The weight parameter for the second term. The number of vertices. The number of vertices of the actual labeled polygon; the third item As a topology violation penalty, The weighting parameter for the third term; the fourth term Penalty for consistency in edge directions, The weight parameter for the fourth term. Let i be the angle value of the i-th angle. The principal orientation angle of the building's outline. The number of angles; the weight parameter takes the value of ;

[0045] The PPO algorithm is used to optimize the reward R, maximizing R and obtaining the optimal short-side ratio threshold. Sharp corner threshold With obtuse angle threshold .

[0046] Secondly, the present invention provides a building image segmentation system based on SAM-GAN, the system comprising:

[0047] The acquisition and preprocessing unit is used to acquire raw remote sensing images and preprocess them to obtain preprocessed images.

[0048] The model segmentation unit is used to generate preliminary building segmentation results based on the preprocessed image and the preset SAM-GAN model. The SAM-GAN model includes a SAM generator and a dual discriminator network. The dual discriminator network is used to optimize the SAM generator. The dual discriminator network is a dual discriminator architecture that integrates semantic and topological constraints.

[0049] The post-processing unit is used to perform geometric rule adaptive optimization on the preliminary building segmentation results to obtain the final building segmentation results.

[0050] Furthermore, the dual discriminator network includes a semantic discriminator and a topology discriminator, and the semantic discriminator and the topology discriminator share a single encoder;

[0051] The semantic discriminator includes a PatchGAN-style real / fake discriminator head and a multi-class semantic discriminator head. The input of the semantic discriminator is a combination of a preliminary mask, a low-order semantic tensor, and a preprocessed image grayscale image. The output of the semantic discriminator is a real / fake image and a semantic probability.

[0052] The input to the topology discriminator is a preliminary mask and a boundary distance feature map, and the output of the topology discriminator is the topology level.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] Although existing technologies have integrated SAM models and post-processing, they mainly use a single discriminator and fixed rule thresholds, lacking explicit constraints on the realism of segmentation structures and cross-scene adaptive capabilities. This invention constructs a three-in-one closed-loop system of "generation-discrimination-adaptive post-processing" to achieve a dual improvement in segmentation realism and topological consistency, thereby enhancing the accuracy, quality, and generalization ability of building image segmentation results. Specifically, it is reflected in: (1) The SAM-GAN model (semantic discrimination + topological discrimination) with shared dual discriminators is adopted. By sharing encoder features through the discriminators, parameter saving and improved discrimination consistency are achieved. (2) A learnable and cross-domain adaptive post-processing threshold controller is innovatively introduced. Using PPO reinforcement learning and MAML meta-learning, the post-processing geometric threshold is formalized into a low-dimensional continuous control for the first time, overcoming the limitations of fixed thresholds and poor generalization in traditional post-processing. (3) It supports full automation during inference, requiring only one forward pass to generate the mask and threshold, meeting the real-time and batch mapping requirements in large-scale engineering applications. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0056] Figure 1 This is a flowchart of the building image segmentation method based on SAM-GAN of the present invention;

[0057] Figure 2 This is a flowchart of the training process for the SAM-GAN model of this invention.

[0058] Figure 3 The image shows the effect of the building image segmentation method based on SAM-GAN of this invention.

[0059] Figure 4 This is a structural block diagram of the building image segmentation system based on SAM-GAN of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0061] Existing building image segmentation methods suffer from insufficient mask realism and structural consistency, leading to inaccurate segmentation results and poor segmentation quality. This invention proposes a building image segmentation method and system based on SAM-GAN. This invention is an innovative integrated scheme combining a shared dual discriminator SAM-GAN and adaptive meta-learning post-processing to achieve the following objectives:

[0062] (1) Dual discriminator collaborative constraints improve the realism and topological consistency of mask structure segmentation.

[0063] Existing technologies use only a single discriminator to determine pixel-level consistency, making it difficult to consider the global structural closure of buildings. This invention introduces a semantic discriminator. With topology discriminator Collaborative mechanism, semantic discriminator and topology discriminator Shared underlying feature encoder The pixel-level consistency and polygon structure closure of the mask are optimized collaboratively.

[0064] The loss function employs the following dynamic weighted joint adversarial mechanism, balancing segmentation accuracy and topological constraints:

[0065]

[0066] in, The loss function for the SAM generator is the pixel-level semantic segmentation loss. Responsible for pixel-level segmentation To predict the overlap loss between the mask and the real mask, For binary cross-entropy loss; Constrain semantic consistency and topological correctness respectively. For semantic discriminator The loss function, i.e., the semantic discriminator loss, For topology discriminator The loss function, i.e., the topology discriminator loss; For dynamic weights, Achieve a dynamic balance between early pixel accuracy and later structural constraints. The maximum value of the weight. As a time-weighted factor, For training cycles.

[0067] The loss automatically increases the topological adversarial weights as training progresses, gradually enhancing the realism and integrity of the contour structure.

[0068] (2) Adaptive post-processing threshold enhances generalization and transfer capabilities.

[0069] Existing technologies use fixed empirical thresholds (such as short side ratio, angle threshold, etc.) for post-processing, which are difficult to adapt to different resolutions and city styles, resulting in limited generalization performance. This invention proposes a reinforcement learning + meta-learning (PPO+MAML) adaptive thresholding policy network, which integrates reinforcement learning and meta-learning (PPO+MAML) to optimize the traditional post-processing parameters (short side ratio threshold, etc.). Sharp angle threshold obtuse angle threshold This is the first time that the threshold has been formalized as a low-dimensional continuous control variable, and a policy network based on PPO reinforcement learning and MAML meta-learning has been designed to achieve online threshold search and cross-scenario adaptation for the black-box post-processing process. As a space for continuous actions:

[0070] ;

[0071] Here, state s includes geometric and statistical features such as mask area, number of vertices, and resolution, and strategy Implemented using a deep neural network.

[0072] By combining the reward function with constraints on polygon IoU, number of vertices, topological legality, and orientation consistency, each mask can obtain the optimal adaptive threshold.

[0073] The reward function is: The first item The second term represents the intersection-union ratio (IoU) of the polygon and the actual mask; Penalty for the difference in the number of vertices (simplicity penalty). The weight parameter for the second term. The number of vertices. The number of vertices of the actual labeled polygon; the third item As a topology violation penalty, The weighting parameter for the third term; the fourth term Penalty for consistency in edge directions, The weight parameter for the fourth term. For the first The angle value of each angle. The principal orientation angle of the building's outline. The number of angles;

[0074] The outer meta-learning loop ensures adaptation to different image resolutions / city styles in just 2-3 steps, greatly improving the system's automation and generalization capabilities. In other words, MAML enables cross-resolution migration in 2-3 steps without requiring re-mesh search, significantly improving practical deployment efficiency.

[0075] Example 1

[0076] like Figure 1 As shown, Figure 1 This is a flowchart of the building image segmentation method based on SAM-GAN of the present invention; the building image segmentation method based on SAM-GAN of the present invention includes:

[0077] Step 1: Acquire the original remote sensing image and preprocess it to obtain the preprocessed image;

[0078] In this embodiment, high-resolution raw remote sensing images are acquired by remote sensing satellites or airborne platforms. The raw remote sensing images include buildings. After preprocessing steps such as geometric correction and radiometric correction, the raw remote sensing images are converted into standard format image data, which serves as the preprocessed images.

[0079] Step 2: Based on the preprocessed image, generate preliminary building segmentation results based on the preset SAM-GAN model; the SAM-GAN model includes a SAM generator and a dual discriminator network, which is used to optimize the SAM generator.

[0080] One of the design features of this invention is that the SAM-GAN model is an improvement on the existing SAM model. While retaining the excellent zero-sample generalization ability of the SAM model, this invention introduces a specific high-frequency refinement head to enhance the ability to resolve building edge textures.

[0081] In this embodiment, the SAM generator includes an encoder and a decoder;

[0082] The encoder is used to perform preliminary feature extraction on the preprocessed image. Specifically, the preprocessed image is fed into the encoder of the SAM model (Segment Anything Model) for preliminary feature extraction. The encoder uses the publicly available ViT-H Transformer structure of SAM and freezes the training of the first 10 epochs to ensure its zero-shot generalization performance.

[0083] The decoder is used to reconstruct the initially extracted features and output the initial building segmentation results. Specifically, the features extracted by the encoder are further fed into the decoder for reconstruction. The decoder consists of the original 3-layer Transformer decoder of SAM plus 2 layers of 5×5 depth separable convolutional layers to enhance the ability to resolve building details and textures. After decoding, the initial building segmentation results are output, including the initial mask, low-order semantic tensor and boundary distance feature map.

[0084] The specific implementation scheme of the SAM generator is as follows:

[0085] First, the SAM generator uses the publicly available SAM-ViT-H / 16 model pre-trained with weights. This model consists of 32 stacked Transformer layers with a hidden dimension of 1280. To avoid gradient instability in the early training phase and to maintain the generalization properties of the pre-trained model, this proposal completely freezes the ViT-H encoder for the first 10 epochs of model training, training and fine-tuning only the decoder and the newly added refinement head to ensure the model's full adaptation to architectural texture features in remote sensing imagery. Starting from the 11th epoch, the ViT-H encoder parameters are gradually unfrozen, and adversarial joint training is performed across the entire network until the end of training (90 epochs), thereby further consolidating the model's generalization ability to remote sensing imagery features.

[0086] Secondly, based on the original three-layer Transformer decoder, this invention adds a "high-frequency thinning head" with a structure of two 5×5 depthwise-pointwise convolutions. Specifically, the channel dimension is designed as 256 → 128 → 1, and each convolution layer is followed by batch normalization (BN) and SiLU activation function. The depthwise-pointwise convolution layer significantly reduces the amount of computation and parameters compared to ordinary convolution, specifically about 25% of that of traditional convolution. After the decoder output is processed by the high-frequency thinning head, the following three branches are generated simultaneously: (1) final mask (Size: 1×1024×1024); (2) Low-order semantic feature tensor (Dimensions: 256×256×256); (3) Boundary distance map (Size: 64×256×256) The above three types of features are then sent to a dual discriminator network to achieve the unification and consistency constraints of multi-scale features.

[0087] The training process of the SAM generator in this invention uses the AdamW optimizer, with the learning rate set to [value missing]. The weight decay coefficient is Each batch processes two image tiles with a size of 1024×1024 pixels. Loss function For a linear combination of Dice and binary cross-entropy loss (BCE):

[0088] ;

[0089] in, To predict the overlap loss between the mask and the real mask, For binary cross-entropy loss;

[0090] Dynamic adversarial loss function is gradually introduced during training. :

[0091] ;

[0092] in, The current training epoch number ensures that the initial training focuses on basic pixel accuracy, while the later training emphasizes the refinement of topological structure and high-frequency textures. For semantic discriminator The loss function, i.e., the semantic discriminator loss, For topology discriminator The loss function is the topology discriminator loss.

[0093] In this embodiment, the dual discriminator network is a dual discriminator architecture that integrates semantic and topological constraints to significantly improve the accuracy and topological integrity of building image segmentation.

[0094] Dual discriminator networks include semantic discriminators. With topology discriminator Furthermore, the semantic discriminator and the topological discriminator share the same encoder. (Using a ResNet-34 architecture) to achieve efficient parameter utilization and consistent feature representation. Specifically, the input data is processed by an encoder. After initial feature extraction, the algorithm splits into two branches: one for semantic discrimination and the other for topological discrimination. Semantic Discriminator It includes a PatchGAN-style true / false discriminator and a multi-class semantic discriminator. The input to the semantic discriminator is a preliminary mask. (After 2x downsampling), low-order semantic tensor Image grayscale corresponding to the preprocessed image (Also downsampled by 2x) spliced ​​together, with a channel dimension of The output of the semantic discriminator is a true / false image and semantic probability;

[0095] Semantic discriminator The loss function is:

[0096] ;

[0097] in, For binary cross-entropy loss, Output the result to determine whether the statement is true or false. For semantic category prediction, and These are the true label and the semantic category truth value, respectively. The weight coefficients for the semantic consistency term. This represents the cross-entropy loss.

[0098] Topology discriminator This focuses on the topological integrity of the building mask. The input to the topology discriminator is the initial mask. (2x downsampling) and boundary distance feature map The channel dimension is The topology discriminator outputs a topology level. After shared ED encoding, two layers of dilated convolutions are connected to preserve spatial structure characteristics. Finally, a topology regression head (containing global average pooling and a three-layer fully connected network) outputs a predicted topology level. Used to evaluate the correctness of the topology, specific labels It is then calculated based on the Euler characteristic difference Δχ and the connected component difference ΔN.

[0099] The loss function of the topology discriminator is:

[0100] ;

[0101] in, For binary cross-entropy loss, These are the weighting coefficients of the topological regression term. For topological level regression loss, To generate a topological true / false discrimination output for the mask, This is a true topology label.

[0102] To achieve co-optimization of semantics and topology, this application introduces a dynamic weight λ(t), which varies with the training cycle. It increases exponentially, used to balance the losses of semantic and topological discrimination. That is, the loss function of a dual-discriminator network. for:

[0103] ;

[0104] ;

[0105] ;

[0106] In the formula, The loss function for the SAM generator; For semantic discriminator The loss function; For topology discriminator The loss function; The weights are dynamic, varying with the training cycle. It increases exponentially and is used to balance the losses of the semantic discriminator and the topological discriminator.

[0107] By setting dynamic weights λ(t), the training process focuses on the consistency of semantic features in the early stages and gradually strengthens the constraints on the topological structure in the later stages, so as to reduce early adversarial training oscillations and enhance convergence stability.

[0108] In this embodiment, Figure 2 This is a flowchart of the training process of the SAM-GAN model of the present invention. The adversarial scheduling training phase of the SAM-GAN model is clearly divided into three consecutive phases: the freeze initialization phase, the semantic adversarial training phase, and the topology reinforcement phase, in order to significantly improve training stability and avoid early oscillations.

[0109] (1) Freeze the initialization phase (Epoch 0–10)

[0110] This phase completely freezes the pre-trained ViT-H encoder and topology discriminator. Only train the decoder and semantic discriminator The relevant parameters; at this point, the dynamic weight λ(t) of the adversarial loss approaches zero, and the training objective degenerates into a pure segmentation loss:

[0111] ;

[0112] This is done to enable the decoder to quickly adapt to the building feature distribution without being affected by early disturbances in the GAN discrimination signal.

[0113] (2) Semantic adversarial training phase (Epoch 11–50)

[0114] This phase unfreezes all SAM generator parameters and starts the semantic discriminator. And topology discriminator The system remains frozen to ensure the stability of the semantic discrimination signal. During this stage, the adversarial loss weight λ(t) increases exponentially with each training epoch (t), defined as follows:

[0115] ;

[0116] During this period, the gradient update ratio between the discriminator and the generator is controlled at 1:1, that is, the semantic discriminator is updated once first, and then the generator is updated once.

[0117] (3) Topology reinforcement training phase (Epoch 51–90)

[0118] This phase unfreezes the topology discriminator. This causes the loss of the topology discriminator. Explicit differentiability constraints are applied to the Euler characteristic difference and connected component difference of the mask. During this stage, the adversarial loss weight λ(t) tends to its maximum value of 0.3. To further emphasize the topological closure constraint, the update ratio of the discriminator to the generator is increased to 3:1, meaning that before each generator update, the discriminator is updated three times (two of which are topological discriminator updates). One is a semantic discriminator Meanwhile, the discriminator employs spectral normalization, and the generator adds weight gradient penalties to prevent mode collapse.

[0119] This application uses a unified hyperparameter: the optimizer is AdamW, and the learning rate is... The weight decay coefficient is The batch size for a single card is set to 2 ( (pixels).

[0120] Step 3: Perform post-processing on the preliminary building segmentation results using geometric rule adaptive optimization to obtain the final building segmentation results.

[0121] In this embodiment, the post-processing is designed as a black-box post-processing pipeline for high-precision geometric regularization of building segmentation masks generated by deep learning. This pipeline also serves as the action interface for reinforcement learning (RL) threshold control strategies. Post-processing includes raster vectorization, vertex simplification, removal of short edges, removal of sharp and obtuse angles, principal direction alignment, and vector rasterization; specifically as follows:

[0122] 1) Raster-to-Vector Conversion: Converts the initial mask of the preliminary building segmentation results from raster format to vector format to obtain the first intermediate result;

[0123] The specific implementation uses OpenCV's `cv2.findContours` function, setting an 8-connectivity mode and the `CHAIN_APPROX_NONE` option to accurately extract all closed contours. Meanwhile, to avoid stray noise interfering with subsequent simplified steps, the mask can be pre-applied before contour extraction. Opening operation preprocessing.

[0124] 2) Vertex simplification (Douglas–Peucker, DP): The first intermediate result is simplified based on the DP algorithm, and the simplification threshold is dynamically adjusted according to the short side ratio threshold n1 provided by the reinforcement learning action interface to obtain the second intermediate result;

[0125] Specifically, the polygonal contours extracted in step 1) are then simplified using the dynamic programming (DP) algorithm. The simplification threshold is then set. The initial setting is 0.04 times the average of the longer side of the building outline, and the shorter side ratio threshold is provided by the reinforcement learning action interface. Dynamic adjustment, i.e. This is done to reduce the number of contour vertices while ensuring that the simplification error does not exceed 10 pixels.

[0126] 3) Short-Edge Merge: Short edges are removed from the second intermediate result to obtain the third intermediate result;

[0127] Specifically, this step iterates through the simplified polygon vertex sequence, selecting four consecutive vertices each time. The three continuous line segments formed by calculation When the middle line segment The length is less than the minimum value of two adjacent line segments multiplied by the short side ratio threshold provided by the action interface. At that time, the line segment and Extend to the intersection point q, and use the newly generated vertex sequence Replace the original four vertices to eliminate trivial boundaries.

[0128] 4) Angle Pruning: Utilizing the angle threshold provided by the reinforcement learning action interface. With obtuse angle threshold Remove the sharp and obtuse angles from the third intermediate result to obtain the fourth intermediate result;

[0129] Specifically, using the sharp corner threshold provided by the action interface With obtuse angle threshold This step iterates through all vertices of the polygon and calculates the angle θ between adjacent sides. When the angle θ is less than... or greater than If the vertex is not found, then delete the vertex and immediately reconnect adjacent vertices to avoid creating self-intersections in the polygon.

[0130] 5) Dominant-Axis Alignment: Based on the fourth intermediate result, the main direction of the building outline is determined; and the angle between each edge of the polygon and the main direction is judged and rotated and adjusted to obtain the fifth intermediate result;

[0131] Specifically, first determine the main direction of the building's outline, and select the direction of the longest side. This is the candidate main direction. If the length of all other edges is less than the length of the longest edge divided by a set value... (Default value is 4), then confirm. The main direction is then determined. Next, the angle between each edge of the polygon and the main direction is assessed and adjusted: if the angle is less than 45°, the edge is rotated until it is parallel to the main direction; if the angle is greater than or equal to 45°, the edge is rotated until it is perpendicular to the main direction. Furthermore, when two adjacent edges become parallel after rotation, an additional line segment is inserted and the intersection point is recalculated to ensure the polygon is closed.

[0132] 6) Vector-to-Raster: The fifth intermediate result is re-rasterized into a binary mask image, which is the final building segmentation result.

[0133] Specifically, using the rasterio tool's rasterize function, the normalized vector polygons are re-rasterized into a binary mask image according to the spatial affine transformation parameters and pixel resolution of the input image. Hole and outer ring information are correctly written in a specific order to ensure the rasterization result is precisely aligned with the original mask coordinate system.

[0134] After completing adversarial training, the initial mask in the above technical solution enters the geometric rule adaptive optimization stage. The initial mask first undergoes grid vectorization and global simplification using the DP algorithm, and then the adaptive threshold parameters predicted by the reinforcement learning-meta-learning policy network are applied. The steps of deleting short sides, removing sharp and obtuse angles, and aligning the main direction are controlled separately to achieve precise regularization of the building outline. Finally, the regularized polygon vector is processed by vector rasterization and masked to obtain the final building segmentation result and output it.

[0135] As a further implementation, the reinforcement learning action interface is implemented as follows:

[0136] The post-processing flow following steps 1) to 6) is encapsulated into a unified interface function postprocess. ,in , , The threshold is the output value of the continuous action interface. The reinforcement learning policy network searches for the threshold by optimizing the reward function, which is defined as:

[0137] ;

[0138] Among them, the first item The second term represents the intersection-union ratio (IoU) of the polygon and the actual mask; Penalty for the difference in the number of vertices (simplicity penalty). The weight parameter for the second term. The number of vertices. The number of vertices of the actual labeled polygon; the third item As a topology violation penalty, The weighting parameter for the third term; the fourth term Penalty for consistency in edge directions, The weight parameter for the fourth term. For the first The angle value of each angle. The principal orientation angle of the building's outline. The number of angles; weighting coefficients. , , Adjust according to application requirements.

[0139] In addition, this invention proposes for the first time a geometric threshold policy network based on reinforcement learning (PPO) and meta-learning (MAML), which formalizes the traditional experience-dependent geometric post-processing threshold problem into a low-dimensional continuous action control problem, enabling adaptive online search of thresholds and rapid cross-domain transfer capabilities.

[0140] Specifically, this invention abstracts the geometric post-processing of the entire building outline into a single-step Markov decision process (MDP). Each state s consists of the building mask and geometric and texture features extracted in real time from the image context, including mask area, perimeter, aspect ratio of the bounding rectangle, number of original vertices, roughness index, image resolution, and local gradient statistics, which are then normalized and fed into the geometric thresholding strategy network.

[0141] Construct a threshold based on the proportion of the shorter side Sharp corner threshold With obtuse angle threshold The resulting action space; the action space is defined as a three-dimensional continuous vector:

[0142] ;

[0143] Action values ​​are mapped to the actual valid range using either sigmoid or tanh, i.e.: ;

[0144] After the actions in the action space are executed, the environment calls the black-box post-processing to perform raster vectorization, vertex simplification, short edge removal, sharp and obtuse corner removal, main direction alignment and vector rasterization in sequence to obtain a regularized mask.

[0145] Calculate the reward based on the regularization mask. ;

[0146] ;

[0147] Among them, the first item The second term represents the intersection-union ratio (IoU) of the polygon and the actual mask; Penalty for the difference in the number of vertices (simplicity penalty). The weight parameter for the second term. The number of vertices. The number of vertices of the actual labeled polygon; the third item As a topology violation penalty, The weighting parameter for the third term; the fourth term Penalty for consistency in edge directions, The weight parameter for the fourth term. For the first The angle value of each angle. The principal orientation angle of the building's outline. The number of angles; the weight parameter takes the value of ;

[0148] The weight parameter values ​​are:

[0149] ;

[0150] The reward R is optimized using the Proximal Policy Optimization (PPO) algorithm, ultimately yielding the optimal short-side ratio threshold. Sharp corner threshold With obtuse angle threshold Specifically, a two-layer fully connected network (input layer width 128, hidden layer width 64) is used to model the strategy and share features of the value function. The network outputs the mean action value. Sum of logarithmic variance The CPU samples multiple building instances in parallel and calculates the dominance function (using GAE). ), and with shearing factor Update strategy.

[0151] To achieve rapid transfer of data across resolutions or different city styles, this invention introduces a Model Agnostic Meta-Learning (MAML) algorithm. Each city or resolution is treated as an independent task. The outer loop randomly selects task batches, performs a 3-step gradient update on each task using PPO, and then averages the parameters between tasks using MAML backpropagation. The resulting parent policy network, after training, requires only a very small number (2-3 steps) of PPO updates to quickly adapt to new image resolutions or city styles.

[0152] like Figure 3 As shown, Figure 3 This is a diagram illustrating the effect of the building image segmentation method based on SAM-GAN proposed in this invention; wherein... Figure 3 The left half shows the original remote sensing image, and the right half shows the building segmentation results obtained by this method (SAM-GAN). The red area represents the building mask. The results show that all major buildings are effectively extracted, with regular boundaries, clear corners, closed contours, and no topological errors such as self-intersection.

[0153] The innovative design features of this invention are:

[0154] First, the innovative dual discriminator structure of the SAM-GAN model;

[0155] This invention presents a building image segmentation method that integrates a large-scale segmentation basic model (SAM) and a generative adversarial network (GAN). The generator employs a SAM-ViT-H encoder-decoder structure modified with high frequency refinement, and the decoder consists of two concatenated layers. Depthwise separable convolutions are used to improve high-frequency boundary resolution.

[0156] In addition, the discriminator section introduces a "dual discriminator design": a semantic discriminator. With topology discriminator Shared encoder (ResNet-34) performs collaborative discrimination on the pixel-level realism of the segmentation mask and the geometric topological integrity of the contour, respectively.

[0157] Second, dynamic weights and phased adversarial training strategies

[0158] Discriminator loss and main segmentation loss are adjusted using dynamic weights. The joint loss is adaptively increased as training progresses to suppress oscillations in the early stages of adversarial training, while later stages focus on topological and high-frequency consistency. The joint loss expression is:

[0159] ;

[0160] ;

[0161] in, , This refers to the epoch number.

[0162] Third, black-box geometry post-processing pipeline and learnable threshold control.

[0163] For the mask output by deep learning models, a six-step "black box" post-processing workflow is introduced (raster vectorization → dynamic simplification → removal of short sides → removal of sharp and obtuse angles → alignment of main directions → vector back to raster), with its core parameter being the short side ratio threshold. Sharp angle threshold obtuse angle threshold Automatic optimization is achieved through downstream strategy networks.

[0164] Fourth, online search and transfer learning of geometric threshold policy networks based on reinforcement learning (PPO) and meta-learning (MAML).

[0165] For the first time, geometric rule thresholding is formalized as a low-dimensional continuous control problem. A threshold policy network is jointly trained using shearing policy gradients and model-independent meta-learning, enabling automatic online threshold search and rapid three-step transfer across image resolutions and city styles. During inference, only one forward pass of the policy network is required to determine the optimal threshold combination for each building, significantly reducing reliance on human experience and transfer costs.

[0166] Example 2

[0167] like Figure 4 As shown, Figure 4 This is a structural block diagram of the building image segmentation system based on SAM-GAN of the present invention. The difference between this embodiment and Embodiment 1 is that this embodiment provides a building image segmentation system based on SAM-GAN, which includes:

[0168] The acquisition and preprocessing unit is used to acquire raw remote sensing images and preprocess them to obtain preprocessed images.

[0169] The model segmentation unit is used to generate preliminary building segmentation results based on the preprocessed image and the preset SAM-GAN model. The SAM-GAN model includes a SAM generator and a dual discriminator network. The dual discriminator network is used to optimize the SAM generator. The dual discriminator network is a dual discriminator architecture that integrates semantic and topological constraints.

[0170] The post-processing unit is used to perform geometric rule adaptive optimization on the preliminary building segmentation results to obtain the final building segmentation results.

[0171] As a further implementation, the dual discriminator network includes a semantic discriminator and a topology discriminator, and the semantic discriminator and the topology discriminator share a single encoder;

[0172] The semantic discriminator includes a PatchGAN-style real / fake discriminator head and a multi-class semantic discriminator head. The input of the semantic discriminator is a combination of a preliminary mask, a low-order semantic tensor, and the image grayscale image corresponding to the preprocessed image. The output of the semantic discriminator is a real / fake image and a semantic probability.

[0173] The input to the topology discriminator is a preliminary mask and a boundary distance feature map, and the output of the topology discriminator is the topology level.

[0174] The execution process of each unit can be carried out according to the steps of the building image segmentation method based on SAM-GAN in Embodiment 1, and will not be described in detail in this embodiment.

[0175] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A building image segmentation method based on SAM-GAN, characterized in that, The method includes: Acquire the original remote sensing image and preprocess the original remote sensing image to obtain the preprocessed image; Based on the preprocessed image, preliminary building segmentation results are generated based on a preset SAM-GAN model; the SAM-GAN model includes a SAM generator and a dual discriminator network, the dual discriminator network is used to optimize the SAM generator, and the dual discriminator network is a dual discriminator architecture that integrates semantic and topological constraints; The preliminary building segmentation results are post-processed by geometric rule adaptive optimization to obtain the final building segmentation results; The SAM generator includes an encoder and a decoder, wherein the encoder is used to perform preliminary feature extraction on the preprocessed image; The decoder is used to reconstruct the initially extracted features and decode and output the initial building segmentation result; the initial building segmentation result includes an initial mask, a low-order semantic tensor, and a boundary distance feature map; The encoder is a ViT-H encoder, which uses the ViT-H Transformer structure disclosed by the SAM model and freezes the training of the first 10 epochs. The decoder consists of the original 3-layer Transformer decoder of the SAM model plus 2 layers of 5×5 depth separable convolutional layers. The dual discriminator network includes a semantic discriminator and a topology discriminator, and the semantic discriminator and the topology discriminator share a single encoder; The semantic discriminator includes a PatchGAN-style real / fake discriminator head and a multi-class semantic discriminator head. The input of the semantic discriminator is a combination of a preliminary mask, a low-order semantic tensor, and a preprocessed image. The output of the semantic discriminator is a real / fake image and a semantic probability. The input to the topology discriminator is a preliminary mask and a boundary distance feature map, and the output of the topology discriminator is the topology level. The post-processing includes raster vectorization, vertex simplification, removal of short edges, removal of sharp and obtuse angles, main direction alignment, and vector rasterization. Raster to Vector: The initial mask of the preliminary building segmentation result is converted from raster format to vector format to obtain the first intermediate result; Vertex simplification: Vertex simplification is performed on the first intermediate result based on the DP algorithm, and the simplification threshold varies with the short side ratio threshold provided by the reinforcement learning action interface. Dynamic adjustments were made to obtain a second intermediate result; Short edge deletion: Delete short edges from the second intermediate result to obtain the third intermediate result; Sharp and obtuse angles are removed: Sharp angle thresholds are provided by the reinforcement learning action interface. With obtuse angle threshold Sharp and obtuse angles are removed from the third intermediate result to obtain the fourth intermediate result; Main direction alignment: Based on the fourth intermediate result, the main direction of the building outline is determined; and the angle between each side of the polygon and the main direction is judged and rotated and adjusted to obtain the fifth intermediate result; Vector rasterization: The fifth intermediate result is re-rasterized into a binary mask image, which is the final building segmentation result; The short side ratio threshold Sharp corner threshold With obtuse angle threshold Automatic optimization is performed through a geometric threshold policy network using reinforcement learning and meta-learning, including: Construct a threshold based on the proportion of the shorter side Sharp corner threshold With obtuse angle threshold The resulting action space; After the actions in the action space are executed, the environment calls the black-box post-processing to perform raster vectorization, vertex simplification, short edge removal, sharp and obtuse angle removal, main direction alignment and vector rasterization in sequence to obtain a regularized mask; Calculate the reward based on the regularization mask. , The first item The second term represents the intersection-union ratio (IoU) of the polygon and the actual mask. Penalty for the difference in the number of vertices, The weight parameter for the second term. The number of vertices. The number of vertices of the actual labeled polygon; the third item As a topology violation penalty, The weighting parameter for the third term; the fourth term Penalty for consistency in edge directions, The weight parameter for the fourth term. Let i be the angle value of the i-th angle. The principal orientation angle of the building's outline. The number of angles; The PPO algorithm is used to optimize the reward R to maximize the reward R, thus obtaining the optimal short-side ratio threshold. Sharp corner threshold With obtuse angle threshold .

2. The building image segmentation method based on SAM-GAN according to claim 1, characterized in that, The loss function of the dual discriminator network for: ； ； In the formula, The loss function for the SAM generator; For semantic discriminator The loss function; For topology discriminator The loss function; The weights are dynamic, varying with the training cycle. It increases exponentially to balance the losses of the semantic discriminator and the topological discriminator; The maximum value of the weight. As a time-weighted factor, For training cycles.

3. The building image segmentation method based on SAM-GAN according to claim 1, characterized in that, The training phase of the SAM-GAN model includes: Freeze Initialization Phase: This phase completely freezes the encoder and topology discriminator, and only trains the relevant parameters of the decoder and semantic discriminator. Semantic adversarial training phase: In this phase, all SAM generator parameters are unfrozen, the semantic discriminator is started, while the topology discriminator remains frozen. Topology reinforcement training phase: In this phase, the topology discriminator is unfrozen, and the loss of the topology discriminator is explicitly differentiable to the difference in Euler characteristic of the mask and the difference in connected components.

4. A building image segmentation system based on SAM-GAN, characterized in that, The system includes: An acquisition and preprocessing unit is used to acquire raw remote sensing images and preprocess the raw remote sensing images to obtain preprocessed images. The model segmentation unit is used to generate preliminary building segmentation results based on the preprocessed image and a preset SAM-GAN model. The SAM-GAN model includes a SAM generator and a dual discriminator network. The dual discriminator network is used to optimize the SAM generator. The dual discriminator network is a dual discriminator architecture that integrates semantic and topological constraints. The post-processing unit is used to perform geometric rule adaptive optimization on the preliminary building segmentation results to obtain the final building segmentation results; The SAM generator includes an encoder and a decoder, wherein the encoder is used to perform preliminary feature extraction on the preprocessed image; The decoder is used to reconstruct the initially extracted features and decode and output the initial building segmentation result; the initial building segmentation result includes an initial mask, a low-order semantic tensor, and a boundary distance feature map; The encoder is a ViT-H encoder, which uses the ViT-H Transformer structure disclosed by the SAM model and freezes the training of the first 10 epochs. The decoder consists of the original 3-layer Transformer decoder of the SAM model plus 2 layers of 5×5 depth separable convolutional layers. The dual discriminator network includes a semantic discriminator and a topology discriminator, and the semantic discriminator and the topology discriminator share a single encoder; The semantic discriminator includes a PatchGAN-style real / fake discriminator head and a multi-class semantic discriminator head. The input of the semantic discriminator is a combination of a preliminary mask, a low-order semantic tensor, and a preprocessed image. The output of the semantic discriminator is a real / fake image and a semantic probability. The input to the topology discriminator is a preliminary mask and a boundary distance feature map, and the output of the topology discriminator is the topology level. The post-processing includes raster vectorization, vertex simplification, removal of short edges, removal of sharp and obtuse angles, main direction alignment, and vector rasterization. Raster to Vector: The initial mask of the preliminary building segmentation result is converted from raster format to vector format to obtain the first intermediate result; Vertex simplification: Vertex simplification is performed on the first intermediate result based on the DP algorithm, and the simplification threshold varies with the short side ratio threshold provided by the reinforcement learning action interface. Dynamic adjustments were made to obtain a second intermediate result; Short edge deletion: Delete short edges from the second intermediate result to obtain the third intermediate result; Sharp and obtuse angles are removed: Sharp angle thresholds are provided by the reinforcement learning action interface. With obtuse angle threshold Sharp and obtuse angles are removed from the third intermediate result to obtain the fourth intermediate result; Main direction alignment: Based on the fourth intermediate result, the main direction of the building outline is determined; and the angle between each side of the polygon and the main direction is judged and rotated and adjusted to obtain the fifth intermediate result; Vector rasterization: The fifth intermediate result is re-rasterized into a binary mask image, which is the final building segmentation result; The short side ratio threshold Sharp corner threshold With obtuse angle threshold Automatic optimization is performed through a geometric threshold policy network using reinforcement learning and meta-learning, including: Construct a threshold based on the proportion of the shorter side Sharp corner threshold With obtuse angle threshold The resulting action space; After the actions in the action space are executed, the environment calls the black-box post-processing to perform raster vectorization, vertex simplification, short edge removal, sharp and obtuse angle removal, main direction alignment and vector rasterization in sequence to obtain a regularized mask; Calculate the reward based on the regularization mask. , The first item The second term represents the intersection-union ratio (IoU) of the polygon and the actual mask; Penalty for the difference in the number of vertices, The weight parameter for the second term. The number of vertices. The number of vertices of the actual labeled polygon; the third item As a topology violation penalty, The weighting parameter for the third term; the fourth term Penalty for consistency in edge directions, The weight parameter for the fourth term. Let i be the angle value of the i-th angle. The principal orientation angle of the building's outline. The number of angles; The PPO algorithm is used to optimize the reward R to maximize the reward R, thus obtaining the optimal short-side ratio threshold. Sharp corner threshold With obtuse angle threshold .

Images