A cross-domain building extraction method based on style transfer and related equipment

Through the combination of style migration technology and invariant feature detection units, the problem of poor generalization performance of remote sensing image building extraction model in different scenarios is solved, and high-precision cross-domain building extraction is achieved, which is suitable for a variety of business needs.

CN118691979BActive Publication Date: 2024-11-22CHINESE ACAD OF SURVEYING & MAPPING
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411163379.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-22
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing remote sensing image building extraction model has poor generalization performance and migration ability on images of different scenarios, different sensors, different time periods, and different quality, and it is difficult to directly apply to actual business.

Method used

Using a cross-domain building extraction method based on style transfer, the source domain image input generative adversarial network is used to transfer the source domain image to generate target domain images, and the building extraction is performed using the supervised learning model of the invariant feature detection unit and the encoding and decoding unit. , combined with vectorization processing and pattern post-processing, the final extraction results of the building are obtained.

Benefits of technology

It realizes high-precision extraction of building extraction models under different remote sensing image styles and features, improves the adaptability and flexibility of the model, and can effectively process images with different background features, different sensor data or data during different periods, satisfying urban planning. , disaster monitoring and other business needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691979B_ABST
    Figure CN118691979B_ABST
Patent Text Reader

Abstract

The present application discloses a cross-domain building extraction method based on style transfer and related equipment, which relate to the field of image data processing. The method comprises: inputting a target domain image into a target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is an output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network; performing vectorization processing and spot post-processing operations on the preliminary building extraction result to obtain a final building extraction result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image data processing, and more specifically, to a cross-domain building extraction method based on style transfer and related equipment. Background Art

[0002] With the continuous advancement of science and technology, remote sensing images have gradually become an important means of obtaining large-scale surface information. Buildings are one of the most important geographical elements in remote sensing images. The use of artificial intelligence deep learning technology to conduct research on high-resolution remote sensing image building extraction can quickly, automatically and accurately obtain information such as the number, distribution, and area size of buildings in a specific area. It can provide effective data and technical support for various business tasks such as urban planning, disaster monitoring, environmental protection, and transportation planning, and has high practical application value.

[0003] In recent years, remote sensing image building extraction methods based on deep learning have emerged in an endless stream and have achieved very high accuracy. However, due to the complex spatiotemporal heterogeneity of remote sensing images, remote sensing images at different times and in different regions have great feature differences. Remote sensing image building extraction models are generally trained based on local areas and specific types of images. There are generally problems with poor model generalization performance and transferability, which results in the building extraction model being able to be applied only to specific data that is highly related to the training data. In actual business, the trained building extraction model is difficult to directly generalize to images of different scenes, different sensors, different time periods, and different qualities, and the extraction effect cannot meet actual requirements.

[0004] In order to solve at least some of the above problems, it is urgent to provide a building extraction method and related equipment with better cross-domain extraction capabilities. Summary of the invention

[0005] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further described in detail in the Detailed Description of the Invention section. The Summary of the Invention section of this application does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0006] In the first aspect, the present application proposes a cross-domain building extraction method based on style transfer, comprising:

[0007] Input the target domain image into the target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is an output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network;

[0008] The preliminary building extraction results are vectorized and post-processed to obtain the final building extraction results.

[0009] In a feasible implementation, the encoding and decoding unit includes an encoder and a decoder, the encoder includes a first convolutional layer, a pooling layer and multiple ResNet Block B feature extraction modules, and the decoder includes multiple upsampling modules, a second convolutional layer, a third convolutional layer and a first activation layer.

[0010] In a feasible implementation, the invariant feature detection unit includes a first feature extraction module, a second feature extraction module, a grayscale overlay layer, a fourth convolutional layer and a second activation layer. The first feature extraction module is used to extract the first feature map output by the first convolutional layer, the second feature extraction module is used to extract the second feature map output by the third convolutional layer, the grayscale overlay layer is used to superimpose the grayscales of the first feature map and the second feature map to output a superimposed feature map, and the fourth convolutional layer and the second activation layer are used to perform convolution and activation processing on the superimposed feature map to obtain invariant features.

[0011] In a feasible implementation, the style transfer model of the generative adversarial network is generated based on the CycleGAN model, and the generator of the style transfer model of the generative adversarial network is a U-type encoding and decoding structure, and the generator includes a generator encoder and a generator decoder.

[0012] In a feasible implementation, the generator encoder includes an encoder convolution layer, multiple ConvNeXt_Block deep-level feature extraction layers and multiple generator convolution downsampling layers, and each of the ConvNeXt_Block deep-level feature extraction layers is adjacent to two generator convolution downsampling layers.

[0013] In a feasible implementation, the generator decoder includes multiple deconvolution layers, multiple normalization layers, multiple activation layers, a decoder convolution layer and a decoder activation layer, and each deconvolution layer, each normalization layer and each activation layer are modularly arranged.

[0014] In a feasible implementation manner, the above-mentioned preliminary building extraction results are subjected to vectorization processing and image spot post-processing operations to obtain the final building extraction results, including:

[0015] The above-mentioned preliminary building extraction result is vectorized to obtain a vectorized result;

[0016] The vectorization results obtained above are post-processed using the Chaikin smoothing algorithm and the Douglas-Peucker thinning algorithm to obtain the final extraction results of the above buildings.

[0017] In a second aspect, the present application proposes a cross-domain building extraction device based on style transfer, comprising:

[0018] An extraction unit is used to input the target domain image into a target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is an output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network;

[0019] The acquisition unit is used to perform vectorization processing and image spot post-processing operations on the preliminary building extraction results to obtain the final building extraction results.

[0020] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the cross-domain building extraction method based on style transfer as described in any one of the above-mentioned first aspects when executing the computer program stored in the memory.

[0021] In a fourth aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the cross-domain building extraction method based on style transfer according to any one of the first aspects is implemented.

[0022] In summary, the cross-domain building extraction method based on style transfer proposed in the embodiment of the present application includes: inputting the target domain image into the target building extraction model to obtain the preliminary extraction result of the building, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and a coding and decoding unit, and the target building extraction model is a building extraction model that is pre-trained by the source domain image and the source domain label and fine-tuned by the target domain image and the target domain label, and the target domain image is the output result obtained by inputting the source domain image into the style transfer model of the generative adversarial network; the preliminary extraction result of the building is vectorized and the spot post-processing operation is performed to obtain the final extraction result of the building. The method proposed in the present application converts the source domain image into the target domain image through GAN, and the model can adapt to different remote sensing image styles and features. This method allows the building extraction model to be not only limited to images similar to the training data, but can effectively process images with different background features, different sensor data or different periods of data. The target building extraction model is first pre-trained on the source domain data, and then fine-tuned on the target domain data, so that the model can better adapt to new data environments and actual scenarios while maintaining high accuracy. The initial extraction results are at the pixel level, and then the outlines of the buildings are converted into polygons or wireframes through vectorization processing. Through post-processing steps such as removing small errors, filling gaps, and smoothing boundaries, the final building map is ensured to be more accurate and closer to the actual situation in morphology, which improves the practicality and accuracy of the extraction results. This building extraction method can support a variety of business needs, such as urban planning, disaster monitoring, environmental protection, and transportation planning, and provide high-quality data support for these fields. The method is not limited by specific sensors, specific time periods, or image quality, so that the model can be flexibly applied to image data under various conditions, improving the adaptability and flexibility of actual business. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0024] Figure 1 A schematic diagram of a cross-domain building extraction method based on style transfer provided in an embodiment of the present application;

[0025] Figure 2 A structural schematic diagram of a target building extraction model provided in an embodiment of the present application;

[0026] Figure 3A structural schematic diagram of a style transfer model of a generative adversarial network provided in an embodiment of the present application;

[0027] Figure 4 A structural schematic diagram of a generator of a style transfer model of a generative adversarial network provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of an input image of a style transfer model of a generative adversarial network provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of an output image of a style transfer model of a generative adversarial network provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of a building extraction effect image provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of a building outline before post-processing provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of a building outline after post-processing provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of a cross-domain building extraction device based on style transfer provided in an embodiment of the present application;

[0034] Fig.11 A schematic diagram of the structure of an electronic device for cross-domain building extraction based on style transfer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0036] See also Figure 1 , which is a schematic diagram of the process of a cross-domain building extraction method based on style transfer provided in an embodiment of the present application, which may specifically include:

[0037] S110, inputting the target domain image into a target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is an output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network;

[0038] For example, a style transfer model using a generative adversarial network (GAN) is used to convert source domain images into target domain images. The "source domain" refers to a set of images with high-quality annotations, while the "target domain" is a set of images with styles or features that are different from the source domain. They are obtained through style transfer technology, allowing the model to work on different types of data without training from scratch.

[0039] The target building extraction model includes an invariant feature detection unit and a coding and decoding unit. This structure allows the model to identify and locate buildings in images, where the invariant feature detection unit is responsible for identifying key features from complex backgrounds, and the coding and decoding unit processes these features and outputs preliminary extraction results of buildings.

[0040] The target building extraction model is a supervised learning model that is pre-trained using labeled training data (source domain images and labels). After pre-training, the model is fine-tuned on target domain images, using the target domain images and labels to adjust and optimize the model parameters to adapt to the new data style and characteristics.

[0041] S120, performing vectorization processing and image spot post-processing operations on the preliminary building extraction results to obtain final building extraction results.

[0042] For example, the preliminary extracted building results are usually pixel-level, that is, the model outputs whether each pixel belongs to a building. In the vectorization process, these pixel-level results are converted into vector formats, such as polygons or wireframes, which are more useful and standardized for map making and GIS applications.

[0043] Post-processing of image patches refers to optimizing and correcting the vectorized building image patches to improve the accuracy and practicality of the extracted results. This includes but is not limited to removing small errors, filling gaps, smoothing boundaries, etc., in order to make the final building image patches more accurate and close to the actual form of the actual building.

[0044] In summary, the method proposed in this application converts the source domain image into the target domain image through GAN, and the model can adapt to different remote sensing image styles and features. This method allows the building extraction model to be not limited to images similar to the training data, but can effectively process images with different background features, different sensor data or data from different periods. The target building extraction model is first pre-trained on the source domain data and then fine-tuned on the target domain data, so that the model can better adapt to new data environments and actual scenarios while maintaining high accuracy. The preliminary extraction result is pixel-level, and then the outline of the building is converted into a polygon or wireframe through vectorization. By removing small errors, filling gaps and smoothing boundaries and other post-processing steps, the final building map is ensured to be more accurate and close to the actual situation in morphology, which improves the practicality and accuracy of the extraction results. This building extraction method can support a variety of business needs, such as urban planning, disaster monitoring, environmental protection and traffic planning, and provide high-quality data support for these fields. The method is not limited by a specific sensor, a specific time period or image quality, so that the model can be flexibly applied to image data under various conditions, improving the adaptability and flexibility of actual business.

[0045] In some examples, the encoding and decoding unit includes an encoder and a decoder, the encoder includes a first convolutional layer, a pooling layer and multiple ResNet Block B feature extraction modules, and the decoder includes multiple upsampling modules, a second convolutional layer, a third convolutional layer and a first activation layer.

[0046] Exemplarily, the first convolutional layer is the initial part of the encoder, which is responsible for receiving the input target domain image and performing preliminary feature extraction. The first convolutional layer usually uses a small convolution kernel to capture the basic details and texture information of the image.

[0047] The pooling layer is located after the first convolutional layer. Its main function is to downsample, that is, to reduce the spatial dimensions (height and width) of the data, thereby reducing the amount of calculation and controlling overfitting. Through this layer, the model can extract a wider range of features while maintaining key information.

[0048] ResNet Block B is a deep network structure with residual connections that can effectively solve the gradient vanishing problem during deep network training. These modules allow signals to be passed directly to subsequent layers by adding jump connections from input to output. In the building extraction model, these ResNet Block B modules further extract complex and advanced features from the image.

[0049] The upsampling modules play a key role in the decoder. They gradually restore the dimensions of the feature map from the low resolution in the encoder to the resolution of the original image. In this process, the upsampling module increases the spatial dimensions of the feature map through interpolation and other techniques, creating conditions for the next layer to reconstruct detailed information.

[0050] The second and third convolutional layers: further refine the feature maps expanded by the upsampling module. The second and third convolutional layers improve and optimize the local details of the features by applying more convolution operations, which helps to reconstruct the edges and shapes of buildings more accurately.

[0051] The first activation layer is used to introduce nonlinearity so that the model can learn and simulate more complex functions. At the end of the decoder, the first activation layer applies a nonlinear transformation to the final output feature map to ensure that the output feature map can be correctly mapped to the pixel-level building extraction result.

[0052] In some examples, the invariant feature detection unit includes a first feature extraction module, a second feature extraction module, a grayscale overlay layer, a fourth convolutional layer and a second activation layer. The first feature extraction module is used to extract a first feature map output by the first convolutional layer. The second feature extraction module is used to extract a second feature map output by the third convolutional layer. The grayscale overlay layer is used to superimpose the grayscales of the first feature map and the second feature map to output a superimposed feature map. The fourth convolutional layer and the second activation layer are used to perform convolution and activation processing on the superimposed feature map to obtain invariant features.

[0053] Exemplarily, the first feature extraction module is used to extract the first feature map output by the first convolution layer. The first feature map is obtained after the first convolution layer processes the input target domain image, and contains the primary visual features and texture information of the image.

[0054] The second feature extraction module is used to extract the second feature map output by the third convolution layer. The second feature map contains deeper visual information, such as more complex textures and structural details, which are obtained after the first feature map is further processed by the third convolution layer.

[0055] The grayscale overlay layer is used to overlay the grayscale of the first feature map and the second feature map to output an overlay feature map. This process integrates the information of the two feature levels by overlaying the feature maps of two different levels, so that the overlay feature map includes both primary texture information and deeper structural information.

[0056] The fourth convolutional layer is used to perform convolution on the above superimposed feature map. Through this step, the superimposed feature map is further processed to extract features that are more important for subsequent tasks.

[0057] The second activation layer is used to activate the output of the fourth convolutional layer, that is, to perform nonlinear transformation on the feature map after convolution. The use of the activation layer can increase the expressive power of the model and allow the model to learn complex functions.

[0058] Through the collaborative work of these components, the invariant feature detection unit can effectively extract key features that are robust to changes in illumination, seasonality, and other environmental factors. These features provide a stable basis for the recognition and extraction of buildings, allowing the model to demonstrate high performance and accuracy under a variety of remote sensing image conditions.

[0059] In some examples, since there are areas with sudden changes in pixel grayscale values ​​on both sides of the building areas in remote sensing images of different styles, namely the inner and outer boundaries of the buildings, this model uses the inner and outer boundaries of the buildings as invariant features and performs multi-task supervised learning to improve the model's ability to recognize buildings in images of various styles.

[0060] Invariant feature labels are made using source domain images and source domain labels. The source domain images are converted into single-band grayscale images. The canny operator is used to perform edge detection on the grayscale images and binary labels to obtain image boundaries and label boundaries. The label boundaries are converted into vector diagrams, the image boundaries inside the vector range are extracted, and the image boundaries inside the vector range are gray-superimposed with the label boundaries to generate invariant feature labels.

[0061] like Figure 2The schematic diagram of the principle of a target building extraction model proposed in the embodiment of the present application is shown. The whole is divided into two parts, a codec module and an invariant feature detection unit. The codec module includes an encoder and a decoder. The encoder uses the feature extraction part of the ResNet101 network, that is, first through the first convolution layer to perform 7×7, step length 2 convolution, then through the pooling layer to perform 3×3, step length 2 pooling, and then use 3, 4, 23, 3 ResNet Block B feature extraction modules respectively. The decoder part continuously performs 4 times of 2×2 upsampling and 3×3 through multiple upsampling modules, performs convolution with a step length of 1 through the second convolution layer, and finally performs 3×3 size, step length 1 convolution through the third convolution layer. Only through the first activation can SoftMax activation be performed, and the building extraction result map is output. In the invariant feature detection unit, since the area close to the image has rich image boundary features, the first feature map output by the first convolution output layer of the encoder is subjected to 2×2 upsampling and 3×3 size, step length 1 convolution through the first feature extraction module. The area near the output end has accurate building boundary features, so the second feature map of the output of the third convolutional layer of the last module of the decoder is convolved with a size of 3×3 and a step size of 1 through the second feature extraction module. The feature maps output by the two are grayscale superimposed through the grayscale superposition layer, and finally convolved with a size of 1×1 and a step size of 1 through the fourth convolutional layer, and SoftMax activation is performed through the second activation layer to output the invariant feature map. The invariant feature map and the building extraction result map are supervised learning with the corresponding labels respectively.

[0062] In some examples, the style transfer model of the generative adversarial network is generated based on the CycleGAN model, and the generator of the style transfer model of the generative adversarial network is a U-type encoding and decoding structure, and the generator includes a generator encoder and a generator decoder.

[0063] Exemplarily, the generator encoder is the first part of the U-shaped codec structure, which is responsible for receiving the source domain image as input. Its main task is to extract the key features of the input image, which involves analyzing the image data layer by layer, from basic texture and color information to more complex visual structures. During the encoding process, the spatial dimension of the image data (i.e., the height and width of the image) will gradually decrease, while the depth or complexity of the features will gradually increase.

[0064] The generator decoder is the second part of the U-type codec structure. It receives the high-dimensional feature representation output by the encoder and begins to restore the spatial dimensions of the image, gradually constructing an image in the style of the target domain. This process usually includes multiple upsampling steps, each of which gradually increases the dimension of the feature map and finally restores it to the same size as the original image. During the decoding process, the generator decoder not only reconstructs the size of the image, but also converts the style of the source domain into the style of the target domain, which includes the adjustment of color, texture and other visual elements.

[0065] A notable feature of CycleGAN is that it can achieve unsupervised bidirectional style transfer, that is, it can not only convert from the source domain to the target domain, but also convert from the target domain back to the source domain. This ability is achieved through a cycle consistency loss, which ensures that the conversion from one domain to another can return to the original image after two conversions, thereby ensuring the consistency and accuracy of style transfer.

[0066] Through such a U-shaped encoder-decoder structure, the generator in the CycleGAN model can effectively learn and simulate the conversion between two different styles, so that the model can still achieve high-quality style transfer without explicit paired training samples. This is particularly important for application scenarios where it is difficult to obtain a large amount of annotated data, such as style conversion between images obtained by different sensors or under different conditions in remote sensing image analysis.

[0067] In some examples, the generator encoder includes an encoder convolution layer, multiple ConvNeXt_Block deep-level feature extraction layers, and multiple generator convolution downsampling layers, each of the ConvNeXt_Block deep-level feature extraction layers is adjacent to two generator convolution downsampling layers.

[0068] For example, the encoder convolution layer is located at the beginning of the generator encoder, which is responsible for receiving the input source domain image and performing preliminary feature extraction. This layer usually uses convolution kernels to capture the basic details and texture information of the image, preparing for subsequent deep feature extraction.

[0069] Each ConvNeXt_Block focuses on extracting deeper and more complex features from the input feature map, which are crucial for understanding and transferring the style of the image.

[0070] The generator convolutional downsampling layer is responsible for reducing the spatial dimension of the feature map, that is, downsampling. Downsampling is achieved by reducing the height and width of the feature map, which helps to increase the level of abstraction of the features and reduce the amount of computation. In the generator encoder, each ConvNeXt_Block deep feature extraction layer is adjacent to two generator convolutional downsampling layers, which means that after each deep feature extraction, the feature map will be downsampled to further compress and optimize the feature representation.

[0071] This configuration (each ConvNeXt_Block deep feature extraction layer is adjacent to two generator convolutional downsampling layers) ensures the efficiency and effectiveness of the generator encoder in extracting and compressing features, so that the encoder can not only extract enough useful information, but also effectively manage the dimension and complexity of the data, providing a stable and rich foundation for the style reconstruction of the decoder part.

[0072] In some examples, the generator decoder includes multiple deconvolution layers, multiple normalization layers, multiple activation layers, a decoder convolution layer, and a decoder activation layer, and each deconvolution layer, each normalization layer, and each activation layer is modularly arranged.

[0073] For example, the deconvolution layer (often called the transposed convolution layer) is used to gradually increase the spatial dimension of the feature map, that is, upsampling, which helps to restore the original size of the image from the compressed features passed from the encoder. The deconvolution layer increases the height and width of the feature map through interpolation or learned parameters, providing a basis for subsequent layers to reconstruct detailed information.

[0074] The normalization layer is responsible for standardizing the output of each layer in the generator decoder, making the network training more stable and faster. Batch normalization or other types of normalization techniques are usually used, which help the model maintain numerical stability and accelerate convergence during training by adjusting and scaling the activation output.

[0075] The activation layer is used to introduce nonlinearity so that the generator decoder is not just a stack of linear transformations, but can learn and simulate complex data mapping relationships. Activation functions include ReLU (Linear Rectification Unit) or LeakyReLU, which can process and activate the input feature map to better reconstruct the style characteristics of the target domain.

[0076] The decoder convolution layer is located at the end of the generator decoder and is mainly used to refine the feature map restored by the deconvolution layer. This layer optimizes and improves the local details of the image through further convolution processing, ensuring that the final generated target domain image is visually consistent with the desired style.

[0077] The decoder activation layer is usually located at the end of the decoder and is used for the activation processing of the final output feature map. This step is the final stage of generating the target domain image. By using activation functions such as Sigmoid or Tanh, it ensures that each pixel value of the output image is appropriately adjusted to generate visually appropriate results.

[0078] Through the combination of these modular layers, the generator decoder can effectively reconstruct high-quality images of the target domain style from the compressed feature representation. This structure ensures the efficiency of style transfer and the accuracy of the effect. In addition, the modular design allows each part to be optimized independently, improving the flexibility and maintainability of the overall network.

[0079] In some examples, a style transfer model CycleGAN-NeXt based on generative adversarial network is designed. This model uses the CycleGAN model as the basic structure, refers to the feature extraction idea of ​​the advanced convolutional neural network ConvNeXt, and improves the generator part of the model.

[0080] The overall structure of the model is as follows Figure 3 As shown in the figure, image X enters the forward generator G to generate a new image G(X) with the style of image Y. G(X) is judged by the discriminator Dy whether it belongs to image Y. G(X) then enters the reverse generator F to generate a cyclically generated image X with the original style. Through this cyclic structure, the original image can be reconstructed in theory. For image Y, it also goes through a completely symmetrical process.

[0081] The improved generator structure of this application is as follows Figure 4 As shown in the figure, the generator is a U-type codec structure. The encoder part of the generator is composed of the feature extraction part of the simplified ConvNeXt-Tiny network model. The specific structure is that after the image to be converted is input, the encoder convolution layer with a convolution kernel size of 4×4 performs preliminary feature analysis and downsampling, and then continuously inputs to the deep feature extraction layer composed of 3, 9, and 3 ConvNeXt_Blocks, respectively, interspersed with two generator convolution downsampling layers with a convolution kernel size of 1×1. The decoder part of the generator gradually restores the resolution of the image after three deconvolutions with a convolution kernel size of 3×3, normalization, and activation. Finally, the multi-channel features are integrated into a three-channel RGB image through the decoder convolution layer with a convolution kernel size of 7×7 and the decoder activation layer.

[0082] In some examples, the vectorization processing and patch post-processing operations are performed on the preliminary building extraction results to obtain the final building extraction results, including:

[0083] The above-mentioned preliminary building extraction result is vectorized to obtain a vectorized result;

[0084] The vectorization results obtained above are post-processed using the Chaikin smoothing algorithm and the Douglas-Peucker thinning algorithm to obtain the final extraction results of the above buildings.

[0085] Exemplarily, the preliminary building extraction results are vectorized to obtain vectorized results. In this step, the preliminary extracted building results, usually represented in the form of a binary image at the pixel level, are converted into a vector format. This usually involves converting the building outline formed by continuous pixels into a geometric figure, such as a polygon. The vectorized result can more accurately represent the location, shape and boundary of the building, while reducing the storage requirements of the data and facilitating subsequent graphic operations and analysis.

[0086] The vectorization results obtained above are post-processed using the Chaikin smoothing algorithm and the Douglas-Peucker thinning algorithm. This algorithm is used to smooth the edges of the vectorization results. By segmenting the edge segments and relocating the endpoints, the sharp angles are reduced, making the overall contour smoother. This is very effective in removing jagged edges that may be generated during the vectorization process.

[0087] The Douglas-Peucker thinning algorithm is used to simplify vector contours by reducing the number of vertices that describe the contour. This method evaluates the contribution of each point to the overall contour shape and removes those points that do not contribute much to the shape, thereby reducing the amount of data while maintaining the overall accuracy and recognition of the contour.

[0088] Through the application of these two algorithms, the final building extraction results are not only more accurate and beautiful visually, but also more efficient and practical in data processing and application. Such post-processing operations ensure that vectorized building data can be directly used in various professional fields such as urban planning, map making, GIS analysis, etc., improving the application value and practicality of building extraction data.

[0089] In some examples, Figure 5 A schematic diagram of an input image of a style transfer model of a generative adversarial network provided in an embodiment of the present application; Figure 6 A schematic diagram of an output image of a style transfer model of a generative adversarial network provided in an embodiment of the present application; Figure 7 A schematic diagram of a building extraction effect image provided in an embodiment of the present application; Figure 8 A schematic diagram of a building outline before post-processing provided in an embodiment of the present application; Fig. 9 A schematic diagram of a building outline after post-processing provided in an embodiment of the present application.

[0090] The method proposed in this application converts the source domain image into the target domain image through GAN, and the model can adapt to different remote sensing image styles and features. This method allows the building extraction model to be not limited to images similar to the training data, but can effectively process images with different background features, different sensor data or data from different periods. The target building extraction model is first pre-trained on the source domain data and then fine-tuned on the target domain data. This allows the model to better adapt to new data environments and actual scenarios while maintaining high accuracy.

[0091] like Fig.10 As shown, this application proposes a cross-domain building extraction device based on style migration, including:

[0092] The extraction unit 21 is used to input the target domain image into the target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is an output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network;

[0093] The acquisition unit 22 is used to perform vectorization processing and image spot post-processing operations on the preliminary building extraction results to obtain the final building extraction results.

[0094] In some examples, the encoding and decoding unit includes an encoder and a decoder, the encoder includes a first convolutional layer, a pooling layer and multiple ResNet Block B feature extraction modules, and the decoder includes multiple upsampling modules, a second convolutional layer, a third convolutional layer and a first activation layer.

[0095] In some examples, the invariant feature detection unit includes a first feature extraction module, a second feature extraction module, a grayscale overlay layer, a fourth convolutional layer and a second activation layer. The first feature extraction module is used to extract a first feature map output by the first convolutional layer. The second feature extraction module is used to extract a second feature map output by the third convolutional layer. The grayscale overlay layer is used to superimpose the grayscales of the first feature map and the second feature map to output a superimposed feature map. The fourth convolutional layer and the second activation layer are used to perform convolution and activation processing on the superimposed feature map to obtain invariant features.

[0096] In some examples, the style transfer model of the generative adversarial network is generated based on the CycleGAN model, and the generator of the style transfer model of the generative adversarial network is a U-type encoding and decoding structure, and the generator includes a generator encoder and a generator decoder.

[0097] In some examples, the generator encoder includes an encoder convolution layer, multiple ConvNeXt_Block deep-level feature extraction layers, and multiple generator convolution downsampling layers, each of the ConvNeXt_Block deep-level feature extraction layers is adjacent to two generator convolution downsampling layers.

[0098] In some examples, the generator decoder includes multiple deconvolution layers, multiple normalization layers, multiple activation layers, a decoder convolution layer, and a decoder activation layer, and each deconvolution layer, each normalization layer, and each activation layer is modularly arranged.

[0099] In some examples, the vectorization processing and patch post-processing operations are performed on the preliminary building extraction results to obtain the final building extraction results, including:

[0100] The above-mentioned preliminary building extraction result is vectorized to obtain a vectorized result;

[0101] The vectorization results obtained above are post-processed using the Chaikin smoothing algorithm and the Douglas-Peucker thinning algorithm to obtain the final extraction results of the above buildings.

[0102] like Fig.11 As shown, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, the steps of any one of the above-mentioned methods for cross-domain building extraction based on style transfer are implemented.

[0103] Since the electronic device introduced in this embodiment is a device used to implement a cross-domain building extraction device based on style transfer in the embodiment of the present application, based on the method introduced in the embodiment of the present application, the technical personnel in this field can understand the specific implementation mode of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment adopted by the technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0104] In the specific implementation process, when the computer program 311 is executed by the processor, it can achieve Figure 1 Any implementation manner in the corresponding embodiments.

[0105] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] An embodiment of the present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of cross-domain building extraction based on style transfer in the corresponding embodiment.

[0111] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0117] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cross-domain building extraction method based on style transfer, characterized in that: include: Input the target domain image into the target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is the output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network; Performing vectorization processing and image spot post-processing operations on the preliminary building extraction results to obtain final building extraction results; The encoding and decoding unit includes an encoder and a decoder, the encoder includes a first convolutional layer, a pooling layer and a plurality of ResNet Block B feature extraction modules, and the decoder includes a plurality of upsampling modules, a second convolutional layer, a third convolutional layer and a first activation layer; The invariant feature detection unit includes a first feature extraction module, a second feature extraction module, a grayscale overlay layer, a fourth convolution layer, and a second activation layer, wherein the first feature extraction module is used to extract a first feature map output by the first convolution layer, the second feature extraction module is used to extract a second feature map output by the third convolution layer, the grayscale overlay layer is used to overlay the grayscales of the first feature map and the second feature map to output a superimposed feature map, and the fourth convolution layer and the second activation layer are used to perform convolution and activation processing on the superimposed feature map to obtain invariant features; The style transfer model of the generative adversarial network is generated based on the CycleGAN model, and the generator of the style transfer model of the generative adversarial network is a U-type encoding and decoding structure, and the generator includes a generator encoder and a generator decoder; The generator encoder includes an encoder convolution layer, a plurality of ConvNeXt_Block deep-level feature extraction layers and a plurality of generator convolution downsampling layers, each of the ConvNeXt_Block deep-level feature extraction layers being adjacent to two generator convolution downsampling layers; The generator decoder includes multiple deconvolution layers, multiple normalization layers, multiple activation layers, a decoder convolution layer and a decoder activation layer, and each deconvolution layer, each normalization layer and each activation layer are modularly arranged.

2. The cross-domain building extraction method based on style transfer according to claim 1, characterized in that: The performing vectorization processing and image spot post-processing operations on the preliminary building extraction results to obtain the final building extraction results includes: performing vectorization processing on the preliminary building extraction result to obtain a vectorization result; The vectorization result is subjected to post-processing operation using Chaikin smoothing algorithm and Douglas-Peucker thinning algorithm to obtain the final extraction result of the building.

3. A cross-domain building extraction device based on style transfer, characterized in that: include: An extraction unit, used for inputting the target domain image into a target building extraction model to obtain a preliminary building extraction result, wherein the target building extraction model is a supervised learning model including an invariant feature detection unit and an encoding and decoding unit, the target building extraction model is a building extraction model that is pre-trained with source domain images and source domain labels and fine-tuned with target domain images and target domain labels, and the target domain image is an output result obtained by inputting the source domain image into a style transfer model of a generative adversarial network; An acquisition unit, used for performing vectorization processing and image spot post-processing operations on the preliminary building extraction result to obtain a final building extraction result; The encoding and decoding unit includes an encoder and a decoder, the encoder includes a first convolutional layer, a pooling layer and a plurality of ResNet Block B feature extraction modules, and the decoder includes a plurality of upsampling modules, a second convolutional layer, a third convolutional layer and a first activation layer; The invariant feature detection unit includes a first feature extraction module, a second feature extraction module, a grayscale overlay layer, a fourth convolution layer, and a second activation layer, wherein the first feature extraction module is used to extract a first feature map output by the first convolution layer, the second feature extraction module is used to extract a second feature map output by the third convolution layer, the grayscale overlay layer is used to overlay the grayscales of the first feature map and the second feature map to output a superimposed feature map, and the fourth convolution layer and the second activation layer are used to perform convolution and activation processing on the superimposed feature map to obtain invariant features; The style transfer model of the generative adversarial network is generated based on the CycleGAN model, and the generator of the style transfer model of the generative adversarial network is a U-type encoding and decoding structure, and the generator includes a generator encoder and a generator decoder; The generator encoder includes an encoder convolution layer, a plurality of ConvNeXt_Block deep-level feature extraction layers and a plurality of generator convolution downsampling layers, each of the ConvNeXt_Block deep-level feature extraction layers being adjacent to two generator convolution downsampling layers; The generator decoder includes multiple deconvolution layers, multiple normalization layers, multiple activation layers, a decoder convolution layer and a decoder activation layer, and each deconvolution layer, each normalization layer and each activation layer are modularly arranged.

4. An electronic device comprising: A memory and a processor, wherein the processor is used to implement the steps of the cross-domain building extraction method based on style transfer as described in any one of claims 1 or 2 when executing the computer program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-domain building extraction method based on style transfer as described in any one of claims 1 or 2 are implemented.