A remote sensing image building extraction model generalization method based on batch style mixing
By employing batch style blending technology in the model extraction of buildings from remote sensing images, the problem of insufficient model generalization ability is solved by utilizing geometric and color enhancement and style blending. This achieves efficient extraction of buildings from unlabeled remote sensing images and reduces the cost of manual annotation.
Patent Information
- Application Number
- CN202210835178.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-15
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2042-07-15
AI Technical Summary
Existing deep learning models lack generalization ability in building extraction from remote sensing images, resulting in unsatisfactory extraction results across sensors, time, space, and atmospheric conditions, and high costs associated with manual annotation.
A batch-style blending approach is adopted, which dynamically expands small batches of images during model training by using geometric enhancement, color enhancement, and style blending techniques to construct a broader data distribution and enhance the model's generalization ability.
It improves the accuracy of building extraction on unlabeled remote sensing imagery, reduces the cost of manual labeling, and achieves automated building extraction on any unlabeled imagery, with results approaching those of manual drawing.
Smart Images
Figure CN115311555B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a generalization method for building extraction models based on batch style mixing in remote sensing images. It only requires one training session using existing large-scale labeled remote sensing datasets to automatically extract buildings from any unlabeled remote sensing image, effectively improving the building extraction accuracy on unlabeled remote sensing images. When the quality of the unlabeled images is good, it can even achieve building extraction results close to those of manual drawing, greatly reducing labeling costs and significantly improving building extraction speed. This is an important step in the research of automated building extraction from remote sensing images and has high application value. Background Technology
[0002] Buildings are one of the most common land cover types in remote sensing imagery, especially in urban areas. The automatic extraction of buildings from remote sensing imagery is a crucial and fundamental task in the field of remote sensing image processing and analysis, and is of great significance for urban planning, topographic map production and updating, and digital city construction. With the rapid development of remote sensing technology, the resolution of remote sensing imagery is constantly improving, and the detailed and rich information provided by high-resolution imagery offers highly favorable conditions for building extraction.
[0003] In recent years, relying on the development of big data and high-performance computing equipment, deep learning technology has achieved explosive growth in fields such as computer vision and natural language processing, demonstrating its powerful performance. Applying deep learning technology to high-resolution remote sensing imagery for automated and intelligent building extraction holds enormous potential and has attracted significant attention from scholars both domestically and internationally.
[0004] Existing deep learning methods for extracting buildings from remote sensing imagery are mostly supervised semantic segmentation methods, requiring a large number of pixel-by-pixel labeled training samples for model training. Therefore, to promote the development of high-resolution remote sensing imagery building extraction research, domestic and international research institutions have invested significant resources in building extraction remote sensing datasets and have publicly released several. Within these datasets, deep learning models have achieved high building extraction accuracy. However, in practical applications, the extraction results are not ideal when these well-trained models are applied to images outside the datasets. This is because differences in acquisition conditions such as illumination radiation, photographic angle, and photographic time, as well as landscape differences between image-captured areas, create apparent differences between images inside and outside the datasets, resulting in significant differences in data distribution. General supervised deep learning models follow the independent and identically distributed (ICD) assumption, assuming that training and test data originate from the same distribution, lacking the ability to generalize to data outside the distribution. One solution is to retrain the model by manually labeling buildings on newly acquired images, but this is a very laborious and costly process and not practical.
[0005] Therefore, leveraging large-scale remote sensing databases with existing building labels to dynamically expand the sample space and enhance the generalization ability of deep learning models, thereby improving the accuracy of building extraction on unlabeled, arbitrary high-resolution remote sensing images, is an inevitable trend in the research of automated building extraction from remote sensing images. A generalization method for deep learning models used for building extraction from remote sensing images is crucial. Summary of the Invention
[0006] This invention addresses the low generalization ability of existing deep learning methods for building extraction from remote sensing imagery in practical applications. It provides a batch-style fusion-based generalization method for building extraction models from remote sensing imagery. During model training, a batch-style fusion module attached to the model performs multi-strategy dynamic image augmentation on small batches of images, promoting the model's learning of invariant features such as image color, style, and geometry. The trained building extraction model can automatically extract buildings from any unlabeled imagery, effectively improving the performance of building extraction from unlabeled remote sensing imagery. This, to some extent, solves the problem of significant performance degradation in supervised semantic segmentation deep learning methods for building extraction across sensors, time periods, spaces, and atmospheric conditions.
[0007] The technical solution adopted to achieve the purpose of this invention is: a generalization method for building extraction model based on batch style mixing of remote sensing images, comprising the following steps:
[0008] Step 1: Construct a large-scale training set based on existing images and their corresponding building annotation files, and construct a test set for unlabeled building images to be extracted.
[0009] Step 2: Construct a convolutional neural network building extraction model with an encoder-decoder structure containing positional attention and channel attention modules;
[0010] Step 3: Train the building extraction model using the training set images. During training, images from a mini-batch randomly selected from the training set are first input into the batch style blending module. Geometric enhancement, color enhancement, and deep learning-based style blending are performed with certain probabilities, progressively providing a broad data distribution for the building extraction model to learn data-invariant features. Next, the processed mini-batch images are fed into the building extraction model, the loss function is calculated, and backpropagation is used to train the model on the GPU. Training with a wider dataset enhances the generalization ability of the building extraction model.
[0011] Step 3.1, Geometric Enhancement Operation: During training, for each image to be sent into the building extraction model and its corresponding label in a mini-batch, geometric enhancement operations are performed with a certain probability, including vertical flipping, horizontal flipping, random rotation, and random scaling and cropping.
[0012] Step 3.2, Color Enhancement Operation: During the training process, for each image in the mini-batch after step 3.1, a color enhancement operation is performed with a certain probability, including brightness, color balance level, contrast, sharpness adjustment, and Gaussian random noise.
[0013] Step 3.3, Style Mixing Operation: During training, deep learning-based style mixing is performed on the mini-batches processed in Step 3.2 with a certain probability. The mini-batches output in Step 3.2 are called content mini-batches, and the images within each mini-batch are numbered sequentially. First, the order of the images in the content mini-batches is shuffled to form another mini-batchches (called style mini-batches). Images in the style mini-batches with the same number as the content mini-batches are the images to be style-mixed. The content mini-batches and style mini-batches are input into the VGG-19 structure encoder of the style transfer model to obtain the corresponding deep features f. c and f s Secondly, regarding f c Perform a restandardization operation to adjust the mean and variance of its channel dimensions, and the sum of f. s Consistent, we obtain feature f cs At this time f cs Carrying style information and f s Same or similar, but the content information remains unchanged. For f cs and f c A certain proportion of linear interpolation is performed to blend the style information carried by the two features, resulting in feature f. ccs f ccs The style transfer network's VGG-19 mirror-structure decoder outputs a style-mixed mini-batch. In the resulting style-mixed mini-batch, the content of the i-th image is the same as that of the i-th image in the content mini-batch, while the style is a mixture of the styles of the i-th image in the content mini-batch and the style mini-batch. The style transfer network encoder and decoder use model parameters pre-trained on the large-scale computer vision datasets MS-COCO and WikiArt.
[0014] Step 3.4, Building Extraction Model Training: Input the images output from the batch style mixing module into the building extraction model in small batches. For each image in the batch, the model outputs five prediction result images. Calculate the cross-entropy loss function between the prediction result images and the labels respectively, sum them in weight as the total loss function, and train the model through backpropagation.
[0015] Step 4: Using the trained building extraction model, predict the test set images to obtain pixel-level building extraction results. Stitch the results together and vectorize them to output the final building polygon vector file.
[0016] Furthermore, the specific implementation of step 1 includes the following sub-steps:
[0017] Step 1.1, Label Data Creation: Rasterize the building vector labels of the training set images, that is, mark the pixels corresponding to the building vectors as 255, and mark the remaining non-building areas as 0, to obtain the binary label data of the raster;
[0018] Step 1.2, Sample cropping: Taking into account computer performance and the size of ground features, the remote sensing images and their corresponding label data in the training set are cropped into appropriately sized sample blocks according to the land cover range without overlap; for the images to be tested, they are cropped in the same way as the images in the training set.
[0019] Furthermore, the specific implementation of step 2 includes the following sub-steps:
[0020] Step 2.1, Constructing the encoding part: Construct a VGG-16 encoder to extract features from the image through five stages of convolution and downsampling. Select parameters from a pre-trained VGG16 model on the ImageNet dataset to accelerate model convergence;
[0021] Step 2.2, Construct Attention Modules: Introduce channel attention modules and position attention modules, placing them between the encoder and decoder. Input the output of the encoding part into each of the two attention modules, and sum the output features to use as input for the decoding part.
[0022] Step 2.3, Constructing the Decoding Part: The decoder is the reverse structure of the encoder. It upsamples the feature map output by the attention module level by level and concatenates it with the same-level features in the encoder until it reaches the original image size, outputting a classification map (main classification map) of the original image size. During this process, the features at different stages are directly upsampled through four other output branches, outputting classification maps (auxiliary classification maps) of the original image size. That is, the building extraction network outputs a total of five classification maps of the original image size.
[0023] Furthermore, the specific implementation of step 4 includes the following sub-steps:
[0024] Step 4.1, Model Prediction: Load the trained model weights, input the images in the test set into the building extraction model one by one, and the main classification image output by the model is the final building prediction result image.
[0025] Step 4.2, Vectorization of prediction results: The prediction result images are stitched together to restore the original image size, and then the results are vectorized to obtain the vector files of the buildings in the test set image.
[0026] This invention has the following advantages: 1) It proposes a deep learning model generalization method for building extraction from remote sensing images based on batch style mixing. It makes full use of existing database knowledge and uses multi-strategy image dynamic augmentation technology in the plug-and-play batch style mixing module to construct a broader sample space for small batches during training. This enhances the learning and extraction of data-invariant features by the building extraction model, and the obtained model can effectively improve the building extraction effect on unlabeled remote sensing images, greatly reducing the cost of manual labeling; 2) The batch style mixing module and the building extraction model are decoupled in the generalization method. Therefore, only one pre-training on an existing dataset is required, and the building model can be used for building extraction from any unlabeled remote sensing image. It is highly efficient and has great practical value. Attached Figure Description
[0027] Figure 1 This is the overall framework diagram of the present invention.
[0028] Figure 2 This is a diagram of the architecture of the convolutional neural network building extraction model in this invention.
[0029] Figure 3 This is a flowchart of the batch style mixing module in this invention.
[0030] Figure 4 This is a flowchart of the building extraction network training process in this invention. Detailed Implementation
[0031] The specific embodiments of the present invention are described below with reference to examples and accompanying drawings.
[0032] This invention provides a generalization method for building extraction models from remote sensing images based on batch style mixing, such as... Figure 1 As shown, it includes the following steps:
[0033] Step 1: Construct a training set based on existing images and their corresponding annotation files, and construct a test set using unlabeled images; Step 2: Construct a multi-output convolutional neural network model with an encoder-decoder structure and location attention and channel attention modules; Step 3: Train the model using the training set images. In each iteration, a small batch of images is first trained through a batch style mixing module and then through a building extraction model; Step 4: Use the trained building extraction network to predict the building extraction results from the test set images, then stitch them together and vectorize them to obtain the final vector file of the building extraction results from the test set images.
[0034] First, training and test sets are constructed. For the training set images, their corresponding vector labels are rasterized to obtain the binary labels for buildings: the pixel label corresponding to the building vector is 255, and the remaining non-building areas are labeled 0. Taking into account computer performance and the size of ground features, the training set remote sensing images and their corresponding label data are cropped into appropriately sized sample blocks (e.g., 512×512 pixels) without overlap; the test set images are cropped using the same method as the training set images.
[0035] For the architecture diagram of the multi-output convolutional neural network building extraction model with encoder-decoder structure featuring positional and channel attention modules in this invention, please refer to [link to network architecture diagram]. Figure 2 The network model was built using the PyTorch deep learning framework, taking the red, green, and blue channels of an image patch (e.g., 512×512 pixels) as input. Its structure mainly consists of three parts: an encoding part, a decoding part, and an attention module. The encoding part uses a pre-trained VGG16 network on the ImageNet dataset. The network uses a downsampling layer (Stage 1–Stage 5) composed of five stages of convolution and max-pooling layers to extract features from the image. Except for the last downsampling stage, the number of feature channels is doubled after downsampling at each stage of the encoding part, ultimately resulting in a 32×32 feature map with 512 feature channels. An attention module connects the encoder and decoder. The encoder output enters the attention module, and the features output from the two attention modules are summed before entering the decoding part. Within the attention module, a channel attention module is used to extract the correlation between channel features, and a position attention module is used to extract the correlation between pixels on the feature map. The decoder structure is similar to the encoder in reverse. It performs progressive convolution and upsampling, concatenating the features at the same level in the encoder until it reaches the original image size, outputting a classification map (main classification map) of the original image size. In this process, an output branch is set before each upsampling. There are four output branches that directly upsample the features at different stages (16x, 8x, 4x, and 2x upsampling respectively) to the original image size, outputting classification maps (auxiliary classification maps). That is, the network outputs a total of five feature maps of the original image size.
[0036] For the batch style hybrid module, see the technical flowchart. Figure 3 For a mini-batch X during training, we first perform geometric enhancement and color enhancement operations with a certain probability (set to 0.5). Geometric enhancement includes vertical flipping, horizontal flipping, random rotation, and random scaling / cropping; color enhancement includes brightness, color balance level, contrast, sharpness adjustment, and Gaussian random noise. The resulting mini-batch is called the content mini-batch X. c Perform the following style blending operation with a certain probability (set to 0.5): Shuffle X along the batch dimension.c The order of the images above forms another mini-batch (called a style mini-batch) X s The images in the style sub-batch that have the same batch dimension as the content sub-batch are the corresponding images to be style-blended. Then, X... c and X s The encoder fed into the style transfer network obtains deep features f c and f s Secondly, regarding f c Perform a restandardization operation to adjust the mean and variance of its channel dimensions, and the sum of f. s Consistent, we obtain feature f cs At this time f cs Carrying style information and f s Same or similar, but with content information as f c Consistent, remaining unchanged. Next, we consider f. cs and f c A linear interpolation with a weight of 0.5 is performed to blend the style information carried by the two features, resulting in feature f. ccs Finally, f ccs The decoder outputs style mixing mini-batch X after passing through the style transfer network. ccs The obtained X ccs In the above example, the content of the i-th image is the same as that of the i-th image in the content mini-batch, while the style is a mixture of the styles of the i-th image in the content mini-batch and the style mini-batch. The encoder uses a VGG-19 architecture, and the decoder is a mirror image of the encoder. Both the encoder and decoder use parameters pre-trained on the MS-COCO and WikiArt datasets, and these parameters are frozen and not updated during style mixing.
[0037] For the training process of the building extraction network model, please refer to [link / reference]. Figure 4 During training, for each small batch of images extracted from the training set, geometric transformation, color transformation, and style mixing from the batch style mixing module are performed with a certain probability to obtain a small batch composed of dynamically expanded images. This small batch of images is then input into the building extraction model. For each image in the batch, the model outputs one main classification image and four auxiliary classification images. The cross-entropy loss function between each image and its label is calculated, and these images are weighted and summed with weights of 0.5, 0.25, 0.25, 0.25, and 0.25 to obtain the total loss function. The model is then trained via backpropagation.
[0038] After the building extraction model is trained, its optimal weights are loaded and used as the network model for prediction. The images from the test set are then input into the model, and the main classification image output by the model is the final building prediction result image.
[0039] Finally, the prediction results are vectorized. The prediction results are stitched together to restore the original image size, and then vectorized again to obtain vector files of buildings from the test set image.
[0040] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. A generalization method for building extraction models based on batch style mixing in remote sensing images, characterized in that, Includes the following steps: Step 1: Construct a large-scale training set based on existing images and their corresponding building annotation files, and construct a test set for unlabeled building images to be extracted. Step 2: Construct a convolutional neural network building extraction model with an encoder-decoder structure containing positional attention and channel attention modules; Step 3: Train the building extraction model using the training set images. During training, images randomly selected from a small batch in the training set are first input into the batch style mixing module. Geometric enhancement, color enhancement, and deep learning-based style mixing are performed with a certain probability, progressively providing a broad data distribution for the building extraction model to learn data invariant features. Second, the processed image batches are fed into the building extraction model, the loss function is calculated, and the model is trained on the GPU through backpropagation. Through training on a wider range of data, the generalization ability of the building extraction model is enhanced. The specific method of style mixing based on deep learning in the batch style mixing module is as follows: The mini-batch output after geometric enhancement and color enhancement is called the content mini-batch, and the images within the mini-batch are numbered sequentially. First, the order of the images in the content mini-batch is shuffled to form another mini-batch, called the style mini-batch. Images in the style mini-batch with the same number as the content mini-batch are the images to be style-blended. The content mini-batch and the style mini-batch are input into the VGG-19 structure encoder of the style transfer model to obtain the corresponding deep features f. c and f s Secondly, regarding f c Perform a restandardization operation to adjust the mean and variance of its channel dimensions, and the sum of f. s Consistent, we obtain feature f cs At this time f cs Carrying style information and f s Same or similar, but the content information remains unchanged; for f cs and f c A certain proportion of linear interpolation is performed to blend the style information carried by the two features, resulting in feature f. ccs ; will f ccs The VGG-19 mirror structure decoder of the style transfer network outputs a style mixing mini-batch. In the resulting style mixing mini-batch, the content of the i-th image is the same as that of the i-th image in the content mini-batch, and the style is a mixture of the styles of the i-th image in the content mini-batch and the style mini-batch. Step 4: Using the trained building extraction model, predict the test set images to obtain pixel-level building extraction results. Stitch the results together and vectorize them to output the final building polygon vector file.
2. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: Step 1 is implemented in the following sub-steps; Step 1.1, Label Data Creation: Rasterize the building vector labels of the training set images, that is, mark the pixels corresponding to the building vectors as 255, and mark the remaining non-building areas as 0, to obtain the binary label data of the raster; Step 1.2, Sample cropping: Taking into account computer performance and the size of ground features, the remote sensing images and their corresponding label data in the training set are cropped into appropriately sized sample blocks according to the land cover range without overlap; for the images to be tested, they are cropped in the same way as the images in the training set.
3. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: The geometric enhancement operations in the batch style blending module include vertical flip, horizontal flip, random rotation, and random scaling cropping.
4. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: The color enhancement operations in the batch style blending module include brightness, color balance level, contrast, sharpness adjustment, and Gaussian random noise.
5. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: The style transfer network encoder and decoder use model parameters pre-trained on the large-scale computer vision datasets MS-COCO and WikiArt.
6. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: The process of training the building extraction model in step 3 is as follows; The images output from the batch style mixing module are input into the building extraction model in small batches. For each image in the batch, the model outputs five prediction result images. The cross-entropy loss function between the prediction result images and the labels is calculated separately. They are weighted and summed as the total loss function, and the model is trained through backpropagation.
7. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: Step 2 includes the following sub-steps; Step 2.1, Constructing the encoding part: Construct an encoder in the form of VGG-16, extract features from the image through five stages of convolution and downsampling, and select VGG16 model parameters pre-trained on the ImageNet dataset to accelerate model convergence; Step 2.2, Construct Attention Modules: Introduce channel attention modules and position attention modules and place them between the encoder and decoder. Input the output of the encoding part into the two attention modules respectively, and sum the output features as the input of the decoding part; Step 2.3, Constructing the decoding part: The decoder is the opposite of the encoder in structure. It upsamples the feature map output by the attention module level by level and concatenates it with the same level features in the encoder until it reaches the original image size. It outputs a classification map of the original image size, i.e., the main classification map. In this process, the features at different stages are directly upsampled through the other four output branches to output classification maps of the original image size, i.e., auxiliary classification maps. A total of five classification maps of the original image size are output.
8. The method for generalizing a remote sensing image building extraction model based on batch style mixing as described in claim 1, characterized in that: Step 4 includes the following sub-steps; Step 4.1, Model Prediction: Load the trained building model weights, input the images in the test set into the building extraction model one by one, and the main classification image output by the model is the final building prediction result image. Step 4.2, Vectorization of prediction results: The prediction result images are stitched together to restore the original image size, and then the results are vectorized to obtain the vector files of the buildings in the test set image.