Building vector topology extraction method and device based on diffusion model

Abstract

The application relates to a building vector topology extraction method and device based on a diffusion model. The method comprises the following steps: inputting a to-be-extracted remote sensing image into a pre-constructed diffusion model to obtain a building vector topology structure of the to-be-extracted remote sensing image, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by using a preset isomorphic training strategy. Thus, the problems that the structure of a traditional diffusion model is not suitable for learning and prediction of an adjacency matrix due to information loss caused by up-sampling and down-sampling operations of a sparse matrix, the building vector extraction result is not accurate due to the randomness of a noise sampling process of the traditional diffusion model, and the traditional diffusion model is prone to overfitting in a discrimination task are solved, and automatic extraction of a building vector topology structure of a satellite or aerial remote sensing image can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, in particular to a building vector topology extraction method and device based on a diffusion model. BACKGROUND

[0002] Diffusion models have achieved success on various remote sensing generation tasks, such as image generation, denoising, super-resolution reconstruction, etc. Its excellent performance on generation tasks naturally leads to scholars' exploration of its representation ability. In recent years, some scholars have applied diffusion models to visual discrimination tasks, such as image segmentation, depth estimation, target detection, etc. However, there is no diffusion model-based method that can be applied to building vector topology extraction tasks of remote sensing images.

[0003] Building vector extraction technology of remote sensing features is usually divided into three types of methods based on segmentation, contour and node. Among them, the method based on segmentation needs to vectorize the segmentation results to obtain the vector results. It usually needs to design a fine and complex post-processing optimization link, which has large computational overhead and is not robust. The contour-based method is derived from the idea of instance segmentation, which obtains the vector topology by continuously adjusting the initial contour. Such a method depends on the quality of the initial contour, and it is difficult to apply to buildings with complex contours or holes. Finally, the node-based method simultaneously predicts key nodes and their connectivity to construct the building vector topology structure. For example, PolyWorld connects the predicted points by predicting the adjacency matrix. These methods can extract relatively regular vector topology structures, and the theory is simple and direct. However, there are three major problems in directly applying the diffusion model architecture to node-based building vector extraction:

[0004] First, the traditional diffusion model is based on the U-Net structure with up-sampling and down-sampling operations, which is not suitable for the learning and prediction of adjacency matrices. Because the adjacency matrix is a sparse matrix, the information loss brought by the up-sampling and down-sampling operations is unacceptable. In addition, due to the randomness of the noise sampling process of the traditional diffusion model, the generated result obtained each time is random and uncertain, while building vector extraction requires accurate results. Finally, the training and inference expansion of the traditional diffusion model is theoretically equivalent, but the steps are inconsistent when actually executed, which causes it to easily overfit on the discrimination task. In summary, due to the above problems, there is currently no diffusion model architecture applied to building vector topology structure extraction. SUMMARY

[0005] The application provides a building vector topology extraction method and device based on a diffusion model to solve the problems of the traditional diffusion model structure, the information loss caused by the up-sampling and down-sampling operations, the unsuitability for the learning and prediction of the adjacency matrix, the inaccuracy of the building vector extraction result caused by the randomness of the noise sampling process of the traditional diffusion model, and the overfitting of the traditional diffusion model in the discrimination task.

[0006] The first aspect embodiment of the application provides a building vector topology extraction method based on a diffusion model, including the following steps: obtaining a to-be-extracted remote sensing image; inputting the to-be-extracted remote sensing image into a pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network according to a preset isomorphic training strategy.

[0007] Optionally, the inputting of the to-be-extracted remote sensing image into the pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image includes: extracting a building multi-level edge attention map from the to-be-extracted remote sensing image by using a topological focus Swin Transformer of the multi-level diffusion Transformer network, generating a feature map by using the multi-level diffusion Transformer network with the building multi-level edge attention map as a condition, and extracting node coordinates and node features based on the feature map; generating a new visual representation based on the edge feature biased graph diffusion Transformer network with the node features as a condition to predict an adjacency matrix of the nodes; and constructing the building vector topology structure of the to-be-extracted remote sensing image according to the extracted node coordinates and the adjacency matrix.

[0008] Optionally, before the inputting of the to-be-extracted remote sensing image into the pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image, the method includes: obtaining a plurality of remote sensing images and building labeled vector data corresponding to the plurality of remote sensing images; performing data preprocessing on the plurality of remote sensing images and the building labeled vector data to obtain a training sample library; training the initial diffusion model based on the training sample library, and optimizing the initial diffusion model by using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

[0009] Optionally, the preset loss function is:

[0010]

[0011] wherein τ is a loss function sequence number, T is a total number of loss functions, including L node and L graph , and σ is a learnable parameter.

[0012] The second aspect embodiment of the application provides a building vector topology structure extraction device based on a diffusion model, comprising: an acquisition module configured to acquire a to-be-extracted remote sensing image; an extraction module configured to input the to-be-extracted remote sensing image into a pre-constructed diffusion model to obtain a building vector topology structure of the to-be-extracted remote sensing image, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by using a preset isomorphic training strategy.

[0013] Optionally, the extraction module is further configured to: extract a building multi-level edge attention map from the to-be-extracted remote sensing image by using a topological focus Swin Transformer of the multi-level diffusion Transformer network, generate a feature map by using the multi-level diffusion Transformer network with the building multi-level edge attention map as a condition, and extract node coordinates and node features based on the feature map; generate a new visual representation based on the node features by using the edge feature biased graph diffusion Transformer network to predict an adjacency matrix of the nodes; and construct the building vector topology structure of the to-be-extracted remote sensing image according to the extracted node coordinates and the adjacency matrix.

[0014] Optionally, before inputting the to-be-extracted remote sensing image into the pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image, the extraction module is further configured to: acquire a plurality of remote sensing images and building labeled vector data corresponding to the plurality of remote sensing images; perform data preprocessing on the plurality of remote sensing images and the building labeled vector data to obtain a training sample library; train the initial diffusion model based on the training sample library, and optimize the initial diffusion model by using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

[0015] Optionally, the preset loss function is:

[0016]

[0017] wherein τ is a loss function sequence number, T is a total number of loss functions, including L node and L graph , and σ is a learnable parameter.

[0018] The third aspect of the application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the building vector topology extraction method based on the diffusion model according to the above-mentioned embodiments.

[0019] The fourth aspect of the application provides a computer program product having a computer program stored thereon, which is executed by a processor to implement the building vector topology extraction method based on the diffusion model according to the above-mentioned embodiments.

[0020] In the above-mentioned embodiments, the building vector topology structure of the to-be-extracted remote sensing image is obtained by inputting the to-be-extracted remote sensing image into a pre-constructed diffusion model, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network according to a preset isomorphic training strategy. Thus, the problems of the conventional diffusion model are solved, i.e., the structure of the conventional diffusion model is not suitable for learning and prediction of the adjacency matrix due to the information loss caused by the up-sampling and down-sampling operations of the sparse matrix, and the building vector extraction result is not accurate due to the randomness of the noise sampling process, and the conventional diffusion model is prone to overfitting in the discrimination task. The diffusion model architecture can be used for building vector topology structure extraction, and automatic extraction of the building vector topology structure of satellite or aerial remote sensing images is achieved.

[0021] Additional aspects and advantages of the application will be made apparent by the following description and the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above-mentioned and / or additional aspects and advantages of the application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a building vector topology extraction method based on a diffusion model according to an embodiment of the application is shown in FIG. 1;

[0024] Figure 2 A structural diagram of a DiffVector model according to an embodiment of the application is shown in FIG. 2;

[0025] Figure 3 A structural diagram of a topology aggregation SwinTransformer (TCSwin) according to an embodiment of the application is shown in FIG. 3;

[0026] Figure 4Fig. 1 is a structural schematic diagram of a HiDiT module, an EGDiT module and an edge embedder according to an embodiment of the present application;

[0027] Figure 5 Fig. 2 is a structural schematic diagram of a node decoder and a graph decoder according to an embodiment of the present application;

[0028] Figure 6 Fig. 3 is an example diagram of a building vector topology structure extraction device based on a diffusion model according to an embodiment of the present application;

[0029] Figure 7 Fig. 4 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0030] 10 - building vector topology structure extraction device based on a diffusion model; 100 - acquisition module and 200 - extraction module. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, in which the same or similar notations are used to denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0032] A building vector topology structure extraction method and device based on a diffusion model according to an embodiment of the present application are described below with reference to the accompanying drawings. In view of the problems of the traditional diffusion model structure mentioned in the background art, i.e., the adjacency matrix is a sparse matrix, the up-sampling and down-sampling operations cause information loss, are not suitable for learning and prediction of the adjacency matrix, and the traditional diffusion model is prone to overfitting on the discrimination task due to the randomness of the noise sampling process, leading to inaccurate building vector extraction results, the present application provides a building vector topology structure extraction method based on a diffusion model. In the method, the building vector topology structure of the to-be-extracted remote sensing image is obtained by inputting the to-be-extracted remote sensing image into a pre-constructed diffusion model, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by a preset isomorphic training strategy. Thus, the problems of the traditional diffusion model structure are solved, i.e., the adjacency matrix is a sparse matrix, the up-sampling and down-sampling operations cause information loss, are not suitable for learning and prediction of the adjacency matrix, and the traditional diffusion model is prone to overfitting on the discrimination task due to the randomness of the noise sampling process, leading to inaccurate building vector extraction results, and the traditional diffusion model is prone to overfitting on the discrimination task. The diffusion model architecture can be used for building vector topology structure extraction, and automatic extraction of building vector topology structure of satellite or aerial remote sensing images is achieved.

[0033] Specifically, Figure 1 A flowchart of a building vector topology extraction method based on a diffusion model provided by an embodiment of the present application.

[0034] As Figure 1 shown, the building vector topology extraction method based on the diffusion model includes the following steps:

[0035] In step S101, a remote sensing image to be extracted is obtained.

[0036] In step S102, the remote sensing image to be extracted is input into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by a preset isomorphic training strategy.

[0037] Wherein, the initial diffusion model is a DiffVector model, the DiffVector model includes a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network, and the preset isomorphic training strategy (ITS) strictly executes the training process of the multi-level diffusion Transformer network (Hierarchical diffusion transformer, HiDiT) and the edge feature biased graph diffusion Transformer network (Edge based graph diffusion transformer, EGDiT) as the same mirror of the inference process, thereby realizing strict isomorphism of the training and inference steps.

[0038] Wherein, the structure of the multi-level diffusion Transformer network is as shown in Figure 2 The multi-level diffusion Transformer network includes a topology focused Swin Transformer network, a multi-level diffusion Transformer module, and a node decoder module.

[0039] Specifically, the structure of the topology focused Swin Transformer network is as shown in Figure 3As shown, the topologically focused SwinTransformer includes a feature encoder, a multi-level side branch outputer and a boundary feature attention module (Boundary block). The feature encoder selects Swin Transformer, and four stages in the Swin Transformer network are used to output one edge attention map (Boundary Maps) with the same size as the original picture and a feature dimension of 1. The multi-level side branch outputer mainly includes a convolution layer (Convolution Layer), a batch normalization layer (Batch Normalization, BN) and a transpose convolution layer (Transpose Concolution Layer). There are four side branch outputers, and the first three layers have the same structure: transpose convolution layer-BN layer-transpose convolution layer. The fourth layer structure is: transpose convolution layer-BN layer-transpose convolution layer-BN layer-transpose convolution layer. For the first side branch outputer, the convolution kernel size of the transpose convolution layer is 4, the step is 2, the input and output channel numbers are (128, 128) and (128, 1) respectively; for the second side branch outputer, the convolution kernel size of the transpose convolution layer is 8, 4, the step is 4, 2, the input and output channel numbers are (256, 256) and (256, 1) respectively; for the third side branch outputer, the convolution kernel size of the transpose convolution layer is 8, the step is 4, the input and output channel numbers are (512, 512) and (512, 1) respectively; for the fourth side branch outputer, the convolution kernel size of the transpose convolution layer is 8, 8, 4, the step is 4, 4, 2, the input and output channel numbers are (1024, 512), (512, 256) and (256, 1) respectively; the dimension of the BN layer is 1. The four edge attention maps output by the four side branch outputers are stacked along the feature dimension and sent to the multi-level diffusion Transformer module.

[0040] The structure of the multi-level diffusion Transformer module is as follows Figure 4(a) shown, the multi-level diffusion Transformer module includes a time encoder, a feature encoder and an attention network. The time encoder and the feature encoder each mainly include a fully connected layer (Fully connected layer), a layer normalization (Layer normalization, LN) and a rectified linear layer (ReLU). The time encoder and the feature encoder include a convolutional layer, and the size of the convolution kernel is 8 and the step is 8. The attention network mainly includes 6 Transformer encoder modules, which are composed of a multi-head self-attention layer (Multi-head self-attention layer, MHSA), a multi-head cross-attention layer (Multi-head cross-attention layer, MHCA), a feed forward layer (Feed forward layer, FF), a multi-layer perception layer and an LN layer. Among them, the multi-head self-attention layer and the multi-head cross-attention layer are subject to the formula:

[0041]

[0042] In the formula, Q, K and V represent query, index and content vectors respectively. In the embodiment of the application, Q, K and V are represented by d' after being processed by a fully connected layer with an independent input-output feature dimension (384, 1152) and being divided in the first dimension. k The feature dimension of the feature descriptor d' is 768, the softmax represents an activation function, and T represents the transpose of a matrix. The feed forward layer successively includes a fully connected layer with an input and an output channel number of (768, 768). The multi-layer perception layer is composed of an FC layer with an input and an output channel number of (384, 1536), a GELU activation function, a dropout layer, an LN layer, a fully connected layer with an input and an output channel number of (1536, 384) and a dropout layer.

[0043] The multi-scale boundary attention map and the sampling time t are respectively subjected to the feature encoder and the time encoder to obtain corresponding embeddings. The two embeddings are added to obtain a conditional embedding. The noise embedding is input into the multi-head self-attention layer for encoding. The conditional embedding and the noise encoding are integrated together and input into the cross-attention layer to output the final feature, and then the feature is input into the node decoder module.

[0044] The structure of the node decoder module is as shown in Figure 5As shown in (a), the node masker module mainly includes an MLP layer and a detection head. The MLP layer is composed of a linear layer with an input and output channel number of (384, 384), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with an input and output channel number of (384, 4096) and a Dropout layer. The detection head is composed of a convolution layer, a BN layer, an activation function and a convolution layer. The convolution kernel size of the convolution layer is 1, the step is 1, and the input and output channel numbers are (64, 64) and (64, 1) respectively. The node decoder first receives the feature map, and respectively uses the non-maximum suppression algorithm (NMS) to extract 320 target node coordinates.

[0045] The structure of the edge feature biased graph diffusion Transformer network is shown in the lower half of (b). Figure 2 As shown in the lower half of (b), the edge feature biased graph diffusion Transformer network mainly includes a position encoding module, an edge feature biased graph diffusion Transformer module (EGDiT blocks) and an adjacency matrix prediction module.

[0046] The position encoder mainly includes a linear layer with an input and output channel number of (66, 768), an LN layer, a ReLU activation function, a linear layer with an input and output channel number of (768, 768), an LN layer, a ReLU activation function, and a fully connected layer with an input and output channel number of (768, 768). The module first performs grid sampling on the feature image according to the node coordinates to obtain visual descriptors, then stacks the node coordinates and visual descriptors in the feature dimension to obtain new visual descriptors, and then sends the new visual descriptors into the position encoder, and then sends the newly generated visual descriptors into the edge feature biased graph diffusion Transformer module for further processing.

[0047] The structure of the edge feature biased graph diffusion Transformer module is shown in (b). Figure 4 As shown in (b), the edge feature biased graph diffusion Transformer module mainly includes an edge encoder, a node encoder, a time encoder, a multi-layer perception (MLP) and an attention network.

[0048] The edge encoder is shown in (b). Figure 4(c) as shown, mainly consists of an MLP layer, wherein the MLP layer consists of a linear layer with an input-output channel number of (1536, 1536), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with an input-output channel number of (1536, 12) and a Dropout layer. Specifically, first, the visual descriptor is subjected to dimension exchange in the first and second dimensions, then the visual descriptor is subjected to 320 times of replication operation, and is stacked with the transposed result in the first dimension, and then is sent into the MHSA. The node encoder mainly consists of an MLP layer, wherein the MLP layer consists of a linear layer with an input-output channel number of (768, 768), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with an input-output channel number of (768, 768) and a Dropout layer. The time encoder mainly consists of a sampling time embedding module and an MLP layer, wherein the sampling time embedding module generates sinusoidal position embedding based on the sampling time, and the MLP layer consists of a linear layer with an input-output channel number of (256, 768), a GELU, a fully connected layer with an input-output channel number of (768, 768). The multi-layer perception (MLP) mainly consists of a SiLU activation function and a fully connected layer with an input-output channel number of (768, 4068). The attention network mainly contains 12 transformer encoder modules, which consist of a multi-head self-attention layer (MHSA), a feed forward layer (FF), a multi-layer perception (MLP) and an LN layer. Among them, the multi-head self-attention layer is subject to the formula:

[0049]

[0050] In the formula, Q, K and V represent query, index and content vectors respectively. In the embodiment of the present application, Q, K and V are represented by d' after being processed by an independent input-output feature dimension (768, 2304) fully connected layer and being divided in the first dimension. kThe feature dimension of the feature descriptor d′ is 768, softmax represents the activation function, and T represents the transpose of the matrix. The feedforward layer consists of a fully connected layer with one input and one output channel of (768, 768). The Multi-Layer Perceptron (MLP) mainly consists of a linear layer with one input and one output channel of (768, 3072), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with one input and one output channel of (3072, 768), and a Dropout layer. This module obtains the node embedding from the visual descriptor through a node encoder, and adds it to the sampling time embedding obtained through a time encoder at sampling time t to obtain the conditional embedding. The conditional embedding is fed into the MLP to obtain six parameters, of which μ1, γ1, and λ1 participate in the calculation of the attention network, and μ2, γ2, and λ2 participate in the calculation of the MLP. After inputting the noise embedding and edge embedding into the attention network, a new feature descriptor is obtained.

[0051] The adjacency matrix prediction module contains two parallel adjacency matrix prediction heads and an optimization module to optimize the predicted adjacency matrix. For example... Figure 5 As shown in (b), the graph decoder's adjacency matrix prediction head contains four convolutional layers. The first three convolutional layers are followed by a batch normalization layer and a ReLU activation function. The kernel size and stride are both 1, and the number of output channels are (1536, 256), (256, 128), (128, 64), and (64, 1), respectively. First, a dimension of size 1 is added to the last dimension of the feature descriptor. Then, the descriptor is copied and transposed, and the new feature is obtained by stacking the channels with the feature descriptor. This new feature is then fed into two parallel adjacency matrix prediction heads to obtain clockwise and counterclockwise adjacency matrices. The prediction results of the first prediction head and the transpose of the prediction results of the second prediction head are added to obtain the predicted directed adjacency matrix, representing the clockwise connection relationship between nodes. Finally, an optimization module based on the Sinkhorn algorithm is used to optimize the predicted adjacency matrix. The Sinkhorn algorithm iteratively normalizes each row and column of the adjacency matrix, with 100 iterations. The final vector topology of a building can be generated from predicted nodes and a directed adjacency matrix. Specifically, non-zero values ​​on the off-diagonal of the directed adjacency matrix represent connected pairs of predicted nodes. The corresponding predicted nodes are retrieved according to their row and column indices and connected. Zero values ​​and non-zero values ​​on the diagonal of the adjacency matrix are ignored.

[0052] Optionally, in some embodiments, the building vector topology structure of the to-be-extracted remote sensing image is obtained by inputting the to-be-extracted remote sensing image into a pre-constructed diffusion model, including: extracting a building multi-level edge attention map from the to-be-extracted remote sensing image by using a topological focusing Swin Transformer of a multi-level diffusion Transformer network, and generating a feature map by using the multi-level diffusion Transformer network and taking the building multi-level edge attention map as a condition, and extracting node coordinates and node features based on the feature map; generating a new visual representation by taking the node features as a condition by using an edge feature bias-based graph diffusion Transformer network to predict an adjacency matrix of the nodes; and constructing the building vector topology structure of the to-be-extracted remote sensing image according to the extracted node coordinates and the adjacency matrix.

[0053] Optionally, in some embodiments, before the building vector topology structure of the to-be-extracted remote sensing image is obtained by inputting the to-be-extracted remote sensing image into a pre-constructed diffusion model, the method includes: obtaining a plurality of remote sensing images and a plurality of building label vector data corresponding to the remote sensing images; performing data preprocessing on the plurality of remote sensing images and the building label vector data to obtain a training sample library; training an initial diffusion model based on the training sample library, and optimizing the initial diffusion model by using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

[0054] Optionally, in some embodiments, the preset loss function is:

[0055]

[0056] wherein τ is a loss function serial number, T is a total number of loss functions, including L node and L graph , and σ is a learnable parameter.

[0057] Specifically, when the pre-constructed diffusion model is obtained, orthographic or corrected satellite or aerial remote sensing images and corresponding building label vector data are prepared, and then the satellite or aerial remote sensing images and the corresponding building label vector data are preprocessed, that is, the target building label vector data is rasterized, the image and the label raster data are resampled and cropped to obtain an image and a label raster pair with appropriate resolution and consistent size, and finally, a training sample library with appropriate slice size (such as 320x320) is prepared in combination with computer memory resources, characteristics of ground objects, and other factors, and an initial diffusion model is trained based on the training sample library. In addition, the same processing is performed on the to-be-extracted remote sensing image by using data preprocessing to establish an image prediction library for subsequent model prediction.

[0058] Further, the pre-constructed diffusion model is obtained by optimizing the initial diffusion model by using a preset loss function and an Adam optimizer.

[0059] wherein the preset loss function comprises a node detection loss function and a directed adjacency matrix prediction loss function. The node detection loss function is:

[0060]

[0061] wherein M(·) represents an operation of averaging, represents a true value node heat map, and h represents a predicted node heat map.

[0062] The directed adjacency matrix prediction loss function is:

[0063]

[0064] wherein p represents a predicted adjacency matrix, represents an adjacency matrix label.

[0065] And the final expression of the loss function of the plurality of tasks is:

[0066]

[0067] Therefore, by designing an initial diffusion model (i.e., DiffVector model), training the DiffVector model using a training sample library, using a multi-task constraint loss function, and using an Adam optimizer to optimize the DiffVector model, the trained DiffVector model is used to extract buildings from a remote sensing image to be extracted. By inputting the remote sensing image to be extracted into the pre-constructed diffusion model (i.e., the trained DiffVector model), the building vector topology structure of the remote sensing image to be extracted can be obtained.

[0068] When the building vector topology structure is obtained, a topological focusing Swin Transformer is designed to extract a building multi-level edge attention map from the input remote sensing image to be extracted. Then, a multi-level diffusion Transformer network is designed to generate a feature map conditioned on the building edge attention map to extract node coordinates and features. Then, a graph diffusion Transformer network with edge feature bias is designed to generate new visual representations conditioned on the node features to predict the adjacency matrix of the nodes. Finally, the building vector topology structure is constructed according to the extracted node coordinates and adjacency matrix.

[0069] Compared with the prior art, the advantages and beneficial effects of the embodiment of the application are that the embodiment of the application provides a building vector topology structure extraction method based on a diffusion model. On the one hand, the embodiment of the application generalizes the diffusion model architecture and paradigm to the building vector extraction task, filling the gap in the industry in the diffusion model-based remote sensing feature vector topology structure extraction technology. On the other hand, the embodiment of the application alleviates the randomness of the results of the traditional diffusion model and the training overfitting problem, providing a direction for applying the diffusion model to discriminant tasks that emphasize result certainty.

[0070] The building vector topology structure extraction method based on the diffusion model according to the embodiment of the application inputs the to-be-extracted remote sensing image into a pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by using a preset isomorphic training strategy. In this way, the structure of the traditional diffusion model is solved, the information loss caused by the up-sampling and down-sampling operations is solved, the learning and prediction of the adjacency matrix are not suitable, and the problem that the traditional diffusion model is prone to overfitting in the discriminant task is solved. The diffusion model architecture can be used for building vector topology structure extraction, and automatic extraction of the building vector topology structure of satellite or aerial remote sensing images is achieved.

[0071] Secondly, the building vector topology structure extraction device based on the diffusion model according to the embodiment of the application is described with reference to the accompanying drawings.

[0072] Figure 6 is a block schematic diagram of the building vector topology structure extraction device based on the diffusion model according to the embodiment of the application.

[0073] As shown in Figure 6 , the building vector topology structure extraction device 10 based on the diffusion model includes an acquisition module 100 and an extraction module 200.

[0074] The acquisition module 100 is configured to acquire the to-be-extracted remote sensing image, and the extraction module 200 is configured to input the to-be-extracted remote sensing image into a pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by using a preset isomorphic training strategy.

[0075] Optionally, in some embodiments, the extraction module 200 is further configured to: extract a building multi-level edge attention map from the to-be-extracted remote sensing image using a topology-focused Swin Transformer of a multi-level diffusion Transformer network, and generate a feature map using the multi-level diffusion Transformer network, with the building multi-level edge attention map as a condition, and extract node coordinates and node features based on the feature map; generate a new visual representation using a graph diffusion Transformer network based on edge feature bias, with the node features as a condition, to predict an adjacency matrix of the nodes; and construct a building vector topology structure of the to-be-extracted remote sensing image according to the extracted node coordinates and the adjacency matrix.

[0076] Optionally, in some embodiments, before the to-be-extracted remote sensing image is input into the pre-constructed diffusion model to obtain the building vector topology structure of the to-be-extracted remote sensing image, the extraction module 200 is further configured to: obtain a plurality of remote sensing images and building labeled vector data corresponding to the plurality of remote sensing images; perform data preprocessing on the plurality of remote sensing images and the building labeled vector data to obtain a training sample library; train an initial diffusion model based on the training sample library, and optimize the initial diffusion model using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

[0077] Optionally, in some embodiments, the preset loss function is:

[0078]

[0079] wherein τ is a loss function serial number, T is a total number of loss functions, including L node and L graph , and σ is a learnable parameter.

[0080] It should be noted that the foregoing explanation and description of the building vector topology structure extraction method based on the diffusion model also applies to the building vector topology structure extraction device based on the diffusion model of this embodiment, which will not be described here.

[0081] According to the building vector topology extraction device based on the diffusion model provided in the embodiment of the present application, the building vector topology structure of the to-be-extracted remote sensing image is obtained by inputting the to-be-extracted remote sensing image into the pre-constructed diffusion model, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network are trained by a preset isomorphic training strategy. Thus, the problem that the structure of the traditional diffusion model is not suitable for learning and prediction of the adjacency matrix due to the information loss caused by the up-sampling and down-sampling operations of the sparse matrix, and the problem that the traditional diffusion model is prone to overfitting in the discrimination task due to the randomness of the noise sampling process, and the problem that the building vector extraction result is not accurate are solved, and the diffusion model architecture can be used for building vector topology structure extraction, and automatic extraction of the building vector topology structure of the satellite or aerial remote sensing image is realized.

[0082] Figure 7 The electronic device provided in the embodiment of the present application is shown in the structural schematic diagram. The electronic device can include:

[0083] The memory 701, the processor 702, and the computer program stored in the memory 701 and executable on the processor 702.

[0084] The processor 702 implements the building vector topology extraction method based on the diffusion model provided in the above embodiment when executing the program.

[0085] Further, the electronic device further includes:

[0086] The communication interface 703 is used for communication between the memory 701 and the processor 702.

[0087] The memory 701 is used to store the computer program executable on the processor 702.

[0088] The memory 701 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0089] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 7 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0090] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can complete communication between each other through an internal interface.

[0091] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0092] The embodiments of the present application also provide a computer program product, which has a computer program stored thereon, and the program is executed by a processor to implement the above building vector topology extraction method based on a diffusion model.

[0093] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0094] Moreover, the terms "first", "second", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of indicated technical features. Thus, the features defined with "first", "second" can include at least one of the features explicitly or implicitly. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0095] Any process or method descriptions or descriptions of the flow diagrams described herein, or otherwise described herein, can be understood as representing at least one of the steps of a method implemented with one or more computers, as well as any other suitable process, such as a data flow or a signal flow, or other flow diagrams, and the like, and that the various steps or acts of the processes described herein can be carried out in the order specified, or in any other suitable order, and that the processes described herein can be implemented by one or more computers, or other processing devices, or any other suitable apparatus, or any combination thereof.

[0096] Logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be embodied in computer-executable instructions, or other executable instructions, that are executable by a processor, or processors, of a computer-based system, or other processing device, or apparatus, or any combination thereof, to cause the processor, or processors, to perform the steps represented in the flow diagrams, or otherwise described herein. For purposes of this description, a "computer" includes any processor-based or digitally programmed device, including a system, combination of systems, or other processing device that includes one or more processors, or other processing devices, or any combination thereof, that is capable of executing instructions. The computer-executable instructions can be loaded into and executed by a computer, or other processing device, or apparatus, or any combination thereof, to cause the computer, or other processing device, or apparatus, or any combination thereof, to perform the steps represented in the flow diagrams, or otherwise described herein. For purposes of this description, a "computer program product" can be any apparatus that can contain, store, communicate, propagate or transport the program for use by or in connection with the computer, or other processing device, or apparatus, or any combination thereof. More specific examples (a non-exhaustive list) of the computer program product include the following: an electronic connection having one or more wires (electrical, optical or other types), a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer program product can be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for example, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed, and stored in a computer memory in order to be executed. The computer program product can also be a memory having one or more wires, such as a portable memory card or a memory stick.

[0097] It should be understood that parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if realized by hardware, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0098] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer program product, which, when executed, includes one or a combination of the steps of the method embodiments.

[0099] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer program product.

[0100] The above-mentioned computer program product can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A method for extracting a building vector topology based on a diffusion model, characterized in that, The method comprises the following steps: obtaining a remote sensing image to be extracted; inputting the remote sensing image to be extracted into a pre-constructed diffusion model to obtain a building vector topological structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by using a preset isomorphic training strategy; the step of inputting the remote sensing image to be extracted into the pre-constructed diffusion model to obtain the building vector topological structure of the remote sensing image to be extracted comprises: extracting a building multi-level edge attention map from the remote sensing image to be extracted by using a topological focusing Swin Transformer of the multi-level diffusion Transformer network, generating a feature map by using the multi-level diffusion Transformer network under the condition of the building multi-level edge attention map, and extracting node coordinates and node features based on the feature map; generating a new visual representation based on the edge feature biased graph diffusion Transformer network under the condition of the node features to predict an adjacency matrix of the nodes; constructing the building vector topological structure of the remote sensing image to be extracted according to the extracted node coordinates and the adjacency matrix.

2. The method of claim 1, wherein, Before the step of inputting the remote sensing image to be extracted into the pre-constructed diffusion model to obtain the building vector topological structure of the remote sensing image to be extracted, the method comprises the following steps: obtaining a plurality of remote sensing images and building labeled vector data corresponding to the plurality of remote sensing images; performing data preprocessing on the plurality of remote sensing images and the building labeled vector data to obtain a training sample library; training the initial diffusion model based on the training sample library, and optimizing the initial diffusion model by using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

3. The method of claim 2, wherein, The preset loss function is: ； wherein, is the loss function index, is the total number of loss functions, including and , is the learnable parameter.

4. A building vector topology extraction apparatus based on a diffusion model, characterized by, comprising: an obtaining module configured to obtain a remote sensing image to be extracted; an extracting module configured to input the remote sensing image to be extracted into a pre-constructed diffusion model to obtain a building vector topological structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is obtained by optimizing an initial diffusion model, and the initial diffusion model is constructed after training a multi-level diffusion Transformer network and an edge feature biased graph diffusion Transformer network by using a preset isomorphic training strategy; the extracting module is further configured to: extract a building multi-level edge attention map from the remote sensing image to be extracted by using a topological focusing Swin Transformer of the multi-level diffusion Transformer network, generate a feature map by using the multi-level diffusion Transformer network under the condition of the building multi-level edge attention map, and extract node coordinates and node features based on the feature map; generate a new visual representation based on the edge feature biased graph diffusion Transformer network under the condition of the node features to predict an adjacency matrix of the nodes; According to the extracted node coordinates and the adjacency matrix, a building vector topological structure of the remote sensing image to be extracted is constructed.

5. The apparatus of claim 4, wherein, Before inputting the remote sensing image to be extracted into the pre-constructed diffusion model to obtain the building vector topological structure of the remote sensing image to be extracted, the extraction module is further configured to: obtain a plurality of remote sensing images and building labeled vector data corresponding to the plurality of remote sensing images; perform data preprocessing on the plurality of remote sensing images and the building labeled vector data to obtain a training sample library; train the initial diffusion model based on the training sample library, and optimize the initial diffusion model by using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

6. The apparatus of claim 5, wherein, The preset loss function is: ； wherein, is the loss function index, is the total number of loss functions, including and , is the learnable parameter.

7. An electronic device, comprising: comprise a memory and a processor; The processor runs a program corresponding to executable program code stored in the memory by reading the executable program code, to implement the building vector topological structure extraction method based on a diffusion model according to any one of claims 1-3.

8. A computer program product, the computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the building vector topological structure extraction method based on a diffusion model according to any one of claims 1-3.

Images