Building vector topological structure extraction method and device based on diffusion model

Abstract

The invention relates to a building vector topological structure extraction method and device based on a diffusion model, and the method comprises the steps: inputting a to-be-extracted remote sensing image into a pre-constructed diffusion model which is obtained through the optimization of an initial diffusion model, and obtaining a building vector topological structure of the to-be-extracted remote sensing image, the initial diffusion model is constructed after a multi-level diffusion Transform network and an edge feature offset graph diffusion Transform network are trained by a preset isomorphic training strategy. Therefore, the problems that the structure of a traditional diffusion model causes information loss and is not suitable for learning and prediction of an adjacent matrix due to the fact that the adjacent matrix is a sparse matrix and up-and-down sampling operation, and the traditional diffusion model is inaccurate in building vector extraction result due to the randomness of a noise sampling process and is not suitable for the learning and prediction of the adjacent matrix are solved. And traditional diffusion model training is easy to over-fit in a discrimination task, and automatic extraction of the building vector topological structure of the satellite or aerial remote sensing image can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for extracting building vector topological structure based on a diffusion model. Background Art

[0002] Diffusion models have been successful in various remote sensing generation tasks, such as image generation, denoising, and super-resolution reconstruction. Its outstanding performance in generation tasks has naturally attracted scholars to explore its representation capabilities. In recent years, some scholars have applied diffusion models to visual discrimination tasks, such as image segmentation, depth estimation, and target detection. However, there is no method based on diffusion models that can be applied to the task of extracting building vector topological structures from remote sensing images.

[0003] Building vector extraction technology for remote sensing objects is usually divided into three categories: segmentation-based, contour-based, and node-based methods. Among them, the segmentation-based method requires vectorization of the segmentation results to obtain vector results. It is usually necessary to design a sophisticated and complex post-processing optimization link, with high computational overhead and non-robustness. The contour-based method is derived from the idea of ​​instance segmentation. It obtains vector topology by continuously adjusting the initial contour. This type of method relies on the quality of the initial contour and is difficult to apply to buildings with complex contours or holes. Finally, the node-based method simultaneously predicts key nodes and their connectivity to construct the building vector topology structure. For example, PolyWorld connects predicted points by predicting the adjacency matrix. These methods can extract more regular vector topology structures. The theory is simple and direct, but there are three major difficulties in directly applying the architecture of the diffusion model to node-based building vector extraction:

[0004] First, the traditional diffusion model is based on the U-Net structure with up- and down-sampling operations, which is not suitable for learning and predicting the adjacency matrix. Because the adjacency matrix is ​​a sparse matrix, the information loss caused by up- and down-sampling operations is unacceptable. In addition, due to the randomness of the noise sampling process of the traditional diffusion model, the generated results are random and uncertain each time, while the building vector extraction requires a certain and accurate result. Finally, although the traditional diffusion model training and reasoning expansion are theoretically equivalent, the steps are inconsistent during actual execution, which makes it very easy to overfit in the discrimination task. In short, due to the above difficulties, there is currently no method to combine the diffusion model architecture with the building vector topology structure extraction. Summary of the invention

[0005] The present application provides a method and device for extracting building vector topological structure based on a diffusion model, so as to solve the problems that the structure of the traditional diffusion model is not suitable for learning and predicting the adjacency matrix because the adjacency matrix is ​​a sparse matrix and the up and down sampling operations cause information loss, and the traditional diffusion model leads to inaccurate building vector extraction results due to the randomness of the noise sampling process, and the traditional diffusion model training is prone to overfitting in the discrimination task.

[0006] The first aspect of the present application provides a method for extracting a building vector topology structure based on a diffusion model, comprising the following steps: obtaining a remote sensing image to be extracted; inputting the remote sensing image to be extracted into a pre-constructed diffusion model to obtain a building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias using a preset isomorphic training strategy.

[0007] Optionally, the step of inputting the remote sensing image to be extracted into a pre-built diffusion model to obtain the building vector topological structure of the remote sensing image to be extracted includes: extracting a multi-level edge attention map of the building from the remote sensing image to be extracted using the topologically focused Swin Transformer of the multi-level diffusion Transformer network, and generating a feature map based on the multi-level edge attention map of the building using the multi-level diffusion Transformer network, and extracting node coordinates and node features based on the feature map; generating a new visual representation based on the node features using the edge feature biased graph diffusion Transformer network to predict the adjacency matrix of the node; and constructing the building vector topological structure of the remote sensing image to be extracted based on the extracted node coordinates and the adjacency matrix.

[0008] Optionally, before the remote sensing image to be extracted is input into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, it includes: obtaining multiple remote sensing images and building annotation vector data corresponding to the multiple remote sensing images; performing data preprocessing on the multiple remote sensing images and the building annotation vector data to obtain a training sample library; training the initial diffusion model based on the training sample library, and optimizing the initial diffusion model using a preset loss function and Adam optimizer to obtain the pre-constructed diffusion model.

[0009] Optionally, the preset loss function is:

[0010]

[0011] Among them, τ is the loss function number, T is the total number of loss functions, including L node and L graph , σ is a learnable parameter.

[0012] The second aspect of the present application provides a device for extracting a building vector topology structure based on a diffusion model, including: an acquisition module for acquiring a remote sensing image to be extracted; an extraction module for inputting the remote sensing image to be extracted into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias using a preset isomorphic training strategy.

[0013] Optionally, the extraction module is also used to: use the topologically focused Swin Transformer of the multi-level diffusion Transformer network to extract a multi-level edge attention map of buildings from the remote sensing image to be extracted, and use the multi-level diffusion Transformer network to generate a feature map based on the multi-level edge attention map of buildings, and extract node coordinates and node features based on the feature map; use the edge feature biased graph diffusion Transformer network to generate a new visual representation based on the node features to predict the adjacency matrix of the node; and construct the building vector topology structure of the remote sensing image to be extracted based on the extracted node coordinates and the adjacency matrix.

[0014] Optionally, before inputting the remote sensing image to be extracted into a pre-constructed diffusion model to obtain the building vector topological structure of the remote sensing image to be extracted, the extraction module is also used to: obtain multiple remote sensing images and building annotation vector data corresponding to the multiple remote sensing images; perform data preprocessing on the multiple remote sensing images and the building annotation vector data to obtain a training sample library; train the initial diffusion model based on the training sample library, and use a preset loss function and Adam optimizer to optimize the initial diffusion model to obtain the pre-constructed diffusion model.

[0015] Optionally, the preset loss function is:

[0016]

[0017] Among them, τ is the loss function number, T is the total number of loss functions, including L node and L graph , σ is a learnable parameter.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for extracting building vector topology structure based on the diffusion model as described in the above embodiment.

[0019] The fourth aspect of the present application provides a computer program product on which a computer program is stored. The program is executed by a processor to implement the method for extracting building vector topology structure based on the diffusion model as described in the above embodiment.

[0020] In the above implementation, the remote sensing image to be extracted is input into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias by a preset isomorphic training strategy. Thus, the structure of the traditional diffusion model is not suitable for learning and predicting the adjacency matrix because the adjacency matrix is ​​a sparse matrix, and the up and down sampling operations cause information loss, and the traditional diffusion model is not suitable for learning and predicting the adjacency matrix, and the randomness of the noise sampling process of the traditional diffusion model leads to inaccurate building vector extraction results, and the traditional diffusion model training is prone to overfitting in the discrimination task. The diffusion model architecture can be used for extracting the building vector topology structure, and the automatic extraction of the building vector topology structure of satellite or aerial remote sensing images is realized.

[0021] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a method for extracting building vector topological structure based on a diffusion model according to an embodiment of the present application;

[0024] Figure 2 It is a structural schematic diagram of a DiffVector model according to an embodiment of the present application;

[0025] Figure 3 Schematic diagram of the structure of the topological clustering SwinTransformer (TCSwin) according to one embodiment of the present application;

[0026] Figure 4is a schematic structural diagram of a HiDiT module, an EGDiT module and an edge embedder according to an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of a node decoder and a graph decoder according to an embodiment of the present application;

[0028] Figure 6 is an example diagram of a device for extracting building vector topological structure based on a diffusion model according to an embodiment of the present application;

[0029] Figure 7 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0031] 10-a building vector topology structure extraction device based on a diffusion model; 100-an acquisition module and 200-an extraction module. DETAILED DESCRIPTION

[0032] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] The following describes the method and device for extracting the building vector topology structure based on the diffusion model of the embodiment of the present application with reference to the accompanying drawings. In view of the structure of the traditional diffusion model mentioned in the above background technology, since the adjacency matrix is ​​a sparse matrix, the up and down sampling operations cause information loss, which is not suitable for the learning and prediction of the adjacency matrix, and the traditional diffusion model has an inaccurate building vector extraction result due to the randomness of the noise sampling process, and the traditional diffusion model training is prone to overfitting in the discrimination task. The present application provides a method for extracting the building vector topology structure based on the diffusion model, in which the remote sensing image to be extracted is input into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias by a preset isomorphic training strategy. Thus, the problems that the traditional diffusion model structure is not suitable for learning and predicting the adjacency matrix due to the sparse adjacency matrix and the information loss caused by up and down sampling operations are solved, and the traditional diffusion model leads to inaccurate building vector extraction results due to the randomness of the noise sampling process, and the traditional diffusion model training is prone to overfitting in the discrimination task are solved. The diffusion model architecture can be used for the extraction of building vector topological structure, and the automatic extraction of building vector topological structure from satellite or aerial remote sensing images can be realized.

[0034] Specifically, Figure 1 A schematic flow chart of a method for extracting building vector topological structure based on a diffusion model provided in an embodiment of the present application.

[0035] like Figure 1 As shown, the building vector topology structure extraction method based on the diffusion model includes the following steps:

[0036] In step S101, a remote sensing image to be extracted is obtained.

[0037] In step S102, the remote sensing image to be extracted is input into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias using a preset isomorphic training strategy.

[0038] Among them, the initial diffusion model is the DiffVector model, which includes a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias. The preset isomorphic training strategy (ITS) strictly executes the training process of the multi-level diffusion Transformer network (Hierarchicaldiffusion transformer, HiDiT) and the edge feature biased graph diffusion Transformer network (Edge based graph diffusion transformer, EGDiT) as a mirror image of the inference process, thereby achieving strict isomorphism of the training and inference steps.

[0039] Among them, the structure of the multi-level diffusion Transformer network is as follows Figure 2 As shown, the multi-level diffusion Transformer network includes a topologically focused Swin Transformer network, a multi-level diffusion Transformer module and a node decoder module.

[0040] Specifically, the structure of the topology-focused Swin Transformer network is as follows: Figure 3As shown in the figure, the topologically focused SwinTransformer includes a feature encoder, a multi-level side branch outputter and an edge feature attention module (Boundary block). The feature encoder uses Swin Transformer. In the four stages of the Swin Transformer network, the side branch outputter is used to output an edge attention map (Boundary Maps) with a feature dimension of 1 that is the same size as the original image. The multi-level side branch outputter mainly includes a convolution layer (Convolution Layer), a batch normalization layer (Batch Normalization, BN) and a transposed convolution layer (Transpose Concolution Layer). There are four side branch outputters. The first three layers are: transposed convolution layer-BN layer-transposed convolution layer, and the fourth layer is: transposed convolution layer-BN layer-transposed convolution layer-BN layer-transposed convolution layer. For the first side branch output, the convolution kernel size of the transposed convolution layer is 4, the stride is 2, and the number of input and output channels are (128, 128) and (128, 1) respectively; for the second side branch output, the convolution kernel size of the transposed convolution layer is 8, 4, the stride is 4, 2, and the number of input and output channels are (256, 256) and (256, 1) respectively; for the third side branch output, the convolution kernel size of the transposed convolution layer is 8, the stride is 4, and the number of input and output channels are (256, 256) and (256, 1) respectively. The number of output channels is (512, 512) and (512, 1) respectively; for the fourth side branch outputter, the convolution kernel sizes of the transposed convolution layer are 8, 8, 4, and the strides are 4, 4, 2 respectively. The number of input and output channels are (1024, 512), (512, 256) and (256, 1) respectively; the dimensions of the BN layer are all 1. The four edge attention maps output by the four side branch outputters are stacked along the feature dimension and sent to the multi-level diffusion Transformer module.

[0041] The structure of the multi-level diffusion Transformer module is as follows Figure 4As shown in (a), the multi-level diffusion Transformer module includes a time encoder, a feature encoder and an attention network. Both the time encoder and the feature encoder mainly include a fully connected layer (Fully connected layer), a layer normalization (Layer normalization, LN) and a rectified linear layer (ReLU). The time encoder and the feature encoder contain convolutional layers, and the size of the convolution kernel is 8 and the step size is 8. The attention network mainly includes 6 Transformer encoder modules, which are composed of a multi-head self-attention layer (Multi-head self-attention layer, MHSA), a multi-head cross-attention layer (Multi-head cross-attention layer, MHCA), a feed forward layer (Feed forward layer, FF), a multi-layer perceptron layer and an LN layer. Among them, the multi-head self-attention layer and the multi-head cross-attention obey the formula:

[0042]

[0043] In the formula, Q, K, and V represent query, index, and content vectors, respectively. In the embodiment of the present application, Q, K, and V are represented by d′ after being processed by a fully connected layer with an independent input and output feature dimension (384, 1152) and segmented on the first dimension. k is the feature dimension of feature descriptor d′, 768, softmax represents the activation function, and T represents the transpose of the matrix. The feedforward layer contains one fully connected layer with input and output channels of (768, 768) in sequence. The multi-layer perceptron layer consists of an FC layer with input and output channels of (384, 1536), a GELU activation function, a random dropout layer, an LN layer, a fully connected layer with input and output channels of (1536, 384), and a Dropout layer.

[0044] The multi-scale boundary attention map and sampling time t are respectively embedded after passing through the feature encoder and time encoder. The two embeddings are added to obtain the conditional embedding, and the noise embedding is sent to the multi-head self-attention layer for encoding. The conditional embedding is integrated with the noise encoding and sent to the cross-attention layer to output the final features, and then the features are sent to the node decoder module.

[0045] The structure of the node decoder module is as follows Figure 5As shown in (a), the node masker module mainly includes an MLP layer and a detection head. The MLP layer consists of a linear layer with an input and output channel number of (384, 384), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with an input and output channel number of (384, 4096) and a Dropout layer. The detection head consists of a convolutional layer, a BN layer, an activation function and a convolutional layer. The convolution kernel size of the convolutional layer is 1, the step size is 1, and the input and output channel numbers are (64, 64) and (64, 1) respectively. The node decoder first receives the feature map and uses the non-maximum suppression algorithm (NMS) to extract the coordinates of 320 target nodes.

[0046] The structure of the graph diffusion Transformer network with edge feature bias is as follows Figure 2 As shown in the lower middle part, the edge feature biased graph diffusion Transformer network mainly includes a position encoding module, edge feature biased graph diffusion Transformer modules (EGDiT blocks) and an adjacency matrix prediction module.

[0047] Among them, the position encoder mainly includes a linear layer with an input and output channel number of (66, 768), an LN layer, a ReLU activation function, a linear layer with an input and output channel number of (768, 768), an LN layer, a ReLU activation function, and a fully connected layer with an input and output channel number of (768, 768). This module first performs grid sampling on the feature image according to the node coordinates to obtain a visual descriptor, and then stacks the node coordinates and the visual descriptor on the feature dimension to obtain a new visual descriptor, which is then sent to the position encoder, and then the newly generated visual descriptor is sent to the edge feature biased graph diffusion Transformer module for further processing.

[0048] The structure of the graph diffusion Transformer module with edge feature bias is as follows Figure 4 As shown in (b), the graph diffusion Transformer module with edge feature bias mainly includes edge encoder, node encoder, time encoder, multi-layer perception (MLP) and attention network.

[0049] Among them, the edge encoder is Figure 4As shown in (c), it is mainly composed of an MLP layer, in which the MLP layer consists of a linear layer with an input and output channel number of (1536, 1536), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with an input and output channel number of (1536, 12) and a Dropout layer. The specific method is: first, the first and second dimensions of the visual descriptor are swapped, then the visual descriptor is copied 320 times, and the channel stacking operation is performed on the first dimension with the transposed result, and then it is sent to the MHSA. The node encoder is mainly composed of an MLP layer, in which the MLP layer consists of a linear layer with an input and output channel number of (768, 768), a GELU activation function, a Dropout layer, an LN layer, a fully connected layer with an input and output channel number of (768, 768) and a Dropout layer. The temporal encoder is mainly composed of a sampling moment embedding module and an MLP layer, where the sampling moment embedding module generates a sinusoidal position embedding based on the sampling moment. The MLP layer consists of a linear layer with an input and output channel number of (256, 768), a GELU, and a fully connected layer with an input and output channel number of (768, 768). The multi-layer perception (MLP) is mainly composed of a SiLU activation function and a fully connected layer with an input and output channel number of (768, 4068). The attention network mainly contains 12 Transformer encoder modules, which are composed of a multi-head self-attention layer (MHSA), a feed forward layer (FF), a multi-layer perceptron (MLP) and an LN layer. Among them, the multi-head self-attention layer obeys the formula:

[0050]

[0051] In the formula, Q, K, and V represent query, index, and content vector, respectively. In the embodiment of the present application, Q, K, and V are represented by d′ after being processed by a fully connected layer with an independent input and output feature dimension (768, 2304) and segmented on the first dimension. kis the feature dimension of feature descriptor d′768, softmax represents the activation function, and T represents the transpose of the matrix. The feedforward layer contains one input and output channel (768, 768) fully connected layer in sequence. The multi-layer perceptron (MLP) mainly consists of a linear layer with input and output channels (768, 3072), a GELU activation function, a Dropout layer, a LN layer, a fully connected layer with input and output channels (3072, 768) and a Dropout layer. This module obtains node embedding of the visual descriptor through the node encoder, and adds it to the sampling time embedding obtained by the time encoder at the sampling time t to obtain the conditional embedding. The conditional embedding is sent to the multi-layer perceptron to obtain six parameters, of which μ1, γ1, λ1 participate in the calculation of the attention network, μ2, γ2, λ2 participate in the calculation of the multi-layer perceptron, and the noise embedding and edge embedding are input into the attention network to obtain a new feature descriptor.

[0052] The adjacency matrix prediction module contains two parallel adjacency matrix prediction heads and an optimization module to optimize the predicted adjacency matrix. Figure 5 As shown in (b), i.e., the graph decoder, the adjacency matrix prediction head contains 4 convolutional layers, of which the first three convolutional layers are followed by a batch normalization layer and a ReLU activation function. The size and stride of the convolution kernel are both 1, and the number of output channels are (1536, 256), (256, 128), (128, 64), and (64, 1), respectively. First, the feature descriptor is first increased by a dimension of size 1 on the last dimension, then copied and transposed, and the new features are obtained by superimposing the number of channels with the feature descriptor. The features are sent to two parallel adjacency matrix prediction heads to obtain clockwise and counterclockwise adjacency matrices. The prediction results of the first prediction head and the transposed results of the second prediction head are added to obtain the predicted directed adjacency matrix, which represents the clockwise connection relationship between nodes. Finally, the predicted adjacency matrix is ​​optimized using the optimization module based on the Sinkhorn algorithm. The Sinkhorn algorithm iteratively normalizes each row and column of the adjacency matrix, and the number of iterations is 100. The final vector topology of the building can be generated from the predicted nodes and the directed adjacency matrix. The specific method is: the non-zero value on the off-diagonal line of the directed adjacency matrix represents a connected pair of predicted nodes. The corresponding predicted nodes are taken out according to the row and column numbers and connected. The zero values ​​and non-zero values ​​on the diagonal of the adjacency matrix are ignored.

[0053] Optionally, in some embodiments, the remote sensing image to be extracted is input into a pre-built diffusion model to obtain the building vector topological structure of the remote sensing image to be extracted, including: using a topologically focused Swin Transformer of a multi-level diffusion Transformer network to extract a multi-level edge attention map of the building from the remote sensing image to be extracted, and using a multi-level diffusion Transformer network to generate a feature map based on the multi-level edge attention map of the building, and extracting node coordinates and node features based on the feature map; a graph diffusion Transformer network based on edge feature bias generates a new visual representation based on the node features to predict the adjacency matrix of the node; and constructing the building vector topological structure of the remote sensing image to be extracted according to the extracted node coordinates and adjacency matrix.

[0054] Optionally, in some embodiments, before the remote sensing image to be extracted is input into a pre-built diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, it includes: obtaining multiple remote sensing images and building annotation vector data corresponding to the multiple remote sensing images; performing data preprocessing on the multiple remote sensing images and the building annotation vector data to obtain a training sample library; training an initial diffusion model based on the training sample library, and optimizing the initial diffusion model using a preset loss function and Adam optimizer to obtain a pre-built diffusion model.

[0055] Optionally, in some embodiments, the preset loss function is:

[0056]

[0057] Among them, τ is the loss function number, T is the total number of loss functions, including L node and L graph , σ is a learnable parameter.

[0058] Specifically, when obtaining the pre-built diffusion model, it is necessary to prepare orthophotod or rectified satellite or aerial remote sensing images and corresponding building annotation vector data, and then perform data preprocessing on the satellite or aerial remote sensing images and corresponding building annotation vector data, that is, rasterize the target building annotation vector data, resample and crop the image and annotation raster data to obtain an image and annotated raster pair with appropriate resolution and consistent size, and finally, combine the computer video memory resources, the characteristics of the ground objects and other factors to make a training sample library with appropriate slice size (such as 320×320), and train the initial diffusion model based on the training sample library. In addition, the extracted remote sensing images are processed in the same way using data preprocessing to establish an image prediction library for direct prediction by subsequent models.

[0059] Furthermore, the embodiment of the present application utilizes a preset loss function and an Adam optimizer to optimize the initial diffusion model to obtain a pre-constructed diffusion model.

[0060] The preset loss functions include the node detection loss function and the directed adjacency matrix prediction loss function. The node detection loss function is:

[0061]

[0062] Where M(·) represents the operation of finding the mean value, represents the heat map of the true value node, and h represents the heat map of the predicted node.

[0063] The directed adjacency matrix prediction loss function is:

[0064]

[0065] Among them, p represents the predicted adjacency matrix, Represents the adjacency matrix labels.

[0066] The final expression of the loss function integrating multiple tasks is:

[0067]

[0068] Therefore, the embodiment of the present application designs an initial diffusion model (i.e., a DiffVector model), uses a training sample library to train the DiffVector model, uses a multi-task constrained loss function, and uses an Adam optimizer to optimize the DiffVector model. Then, the trained DiffVector model is used to extract buildings from the remote sensing image to be extracted, and the remote sensing image to be extracted is input into the pre-built diffusion model (i.e., the trained DiffVector model) to obtain the building vector topological structure of the remote sensing image to be extracted.

[0069] When obtaining the vector topological structure of the building, a topologically focused Swin Transformer is designed to extract the multi-level edge attention map of the building from the input remote sensing image to be extracted. Then a multi-level diffusion Transformer network is designed to generate a feature map based on the building edge attention map for node coordinate and feature extraction. Then, a graph diffusion Transformer network with edge feature bias is designed to generate a new visual representation based on the node features to predict the adjacency matrix of the node. Finally, the vector topological structure of the building is constructed according to the extracted node coordinates and adjacency matrix.

[0070] Compared with the prior art, the advantages and beneficial effects of the embodiments of the present application are as follows: the embodiments of the present application provide a method for extracting building vector topological structures based on a diffusion model. On the one hand, the embodiments of the present application extend the diffusion model architecture and paradigm to building vector extraction tasks, filling the gap in the industry's remote sensing ground object vector topological structure extraction technology based on diffusion models; on the other hand, the embodiments of the present application alleviate the results randomness and training overfitting problems of traditional diffusion models, and provide direction for the application of diffusion models in discrimination tasks that emphasize result certainty.

[0071] According to the method for extracting building vector topology structure based on diffusion model proposed in the embodiment of the present application, the remote sensing image to be extracted is input into the pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by the initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias by a preset isomorphic training strategy. Thus, the structure of the traditional diffusion model is not suitable for learning and predicting the adjacency matrix because the adjacency matrix is ​​a sparse matrix, and the up and down sampling operations cause information loss, and the traditional diffusion model is not suitable for learning and predicting the adjacency matrix, and the randomness of the noise sampling process of the traditional diffusion model leads to inaccurate building vector extraction results, and the traditional diffusion model training is prone to overfitting in the discrimination task. The diffusion model architecture can be used for extracting building vector topology structure, and the automatic extraction of building vector topology structure of satellite or aerial remote sensing images is realized.

[0072] Next, a building vector topology structure extraction device based on a diffusion model proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0073] Figure 6 It is a block diagram of a device for extracting building vector topological structure based on a diffusion model according to an embodiment of the present application.

[0074] like Figure 6 As shown, the building vector topology structure extraction device 10 based on the diffusion model includes: an acquisition module 100 and an extraction module 200.

[0075] Among them, the acquisition module 100 is used to obtain the remote sensing image to be extracted; the extraction module 200 is used to input the remote sensing image to be extracted into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias by a preset isomorphic training strategy.

[0076] Optionally, in some embodiments, the extraction module 200 is also used to: use a topologically focused Swin Transformer of a multi-level diffusion Transformer network to extract a multi-level edge attention map of a building from the remote sensing image to be extracted, and use a multi-level diffusion Transformer network to generate a feature map based on the multi-level edge attention map of the building, and extract node coordinates and node features based on the feature map; use a graph diffusion Transformer network based on edge feature bias to generate a new visual representation based on the node features to predict the adjacency matrix of the node; and construct a vector topological structure of the building of the remote sensing image to be extracted based on the extracted node coordinates and adjacency matrix.

[0077] Optionally, in some embodiments, before the remote sensing image to be extracted is input into a pre-built diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, the extraction module 200 is also used to: obtain multiple remote sensing images and building annotation vector data corresponding to the multiple remote sensing images; perform data preprocessing on the multiple remote sensing images and the building annotation vector data to obtain a training sample library; train an initial diffusion model based on the training sample library, and use a preset loss function and Adam optimizer to optimize the initial diffusion model to obtain a pre-built diffusion model.

[0078] Optionally, in some embodiments, the preset loss function is:

[0079]

[0080] Among them, τ is the loss function number, T is the total number of loss functions, including L node and L graph , σ is a learnable parameter.

[0081] It should be noted that the above explanation of the embodiment of the method for extracting building vector topology structure based on the diffusion model is also applicable to the device for extracting building vector topology structure based on the diffusion model of this embodiment, which will not be repeated here.

[0082] According to the device for extracting building vector topology structure based on diffusion model proposed in the embodiment of the present application, the remote sensing image to be extracted is input into the pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by the initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias by a preset isomorphic training strategy. Thus, the problem that the structure of the traditional diffusion model is not suitable for learning and predicting the adjacency matrix due to the sparse matrix of the adjacency matrix and the up and down sampling operations cause information loss, and the traditional diffusion model leads to inaccurate building vector extraction results due to the randomness of the noise sampling process, and the traditional diffusion model training is prone to overfitting in the discrimination task is solved, and the diffusion model architecture can be used for building vector topology structure extraction to realize the automatic extraction of building vector topology structure of satellite or aerial remote sensing images.

[0083] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0084] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0085] When the processor 702 executes the program, the method for extracting the building vector topology structure based on the diffusion model provided in the above embodiment is implemented.

[0086] Furthermore, the electronic device further comprises:

[0087] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0088] The memory 701 is used to store computer programs that can be executed on the processor 702 .

[0089] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0090] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0092] The processor 702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0093] The embodiment of the present application also provides a computer program product, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for extracting building vector topological structure based on the diffusion model is implemented.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer program product for use with an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer program product" can be any device that can contain, store, communicate, propagate or transmit a program for use with an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer program products (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). Furthermore, the computer program product may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example by optically scanning the paper or other medium and then editing, interpreting or, if necessary, processing in another suitable manner, and then storing it in a computer memory.

[0098] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer program product, which, when executed, includes one or a combination of the steps of the method embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer program product.

[0101] The computer program product mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for extracting building vector topological structure based on diffusion model, characterized in that: The following steps are involved: Obtain the remote sensing image to be extracted; The remote sensing image to be extracted is input into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias using a preset isomorphic training strategy.

2. The method according to claim 1, characterized in that The step of inputting the remote sensing image to be extracted into a pre-built diffusion model to obtain a building vector topological structure of the remote sensing image to be extracted includes: A topologically focused Swin Transformer of the multi-level diffusion Transformer network is used to extract a multi-level edge attention map of a building from the remote sensing image to be extracted, and a feature map is generated by the multi-level diffusion Transformer network with the multi-level edge attention map of the building as a condition, and node coordinates and node features are extracted based on the feature map; A graph diffusion Transformer network based on the edge feature bias generates a new visual representation based on the node features to predict the adjacency matrix of the node; The building vector topology structure of the remote sensing image to be extracted is constructed according to the extracted node coordinates and the adjacency matrix.

3. The method according to claim 1, characterized in that Before inputting the remote sensing image to be extracted into a pre-built diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, the method includes: Acquire multiple remote sensing images and building annotation vector data corresponding to the multiple remote sensing images; Performing data preprocessing on the multiple remote sensing images and the building annotation vector data to obtain a training sample library; The initial diffusion model is trained based on the training sample library, and the initial diffusion model is optimized using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

4. The method according to claim 3, characterized in that The preset loss function is: Among them, τ is the loss function number, T is the total number of loss functions, including L node and L graph , σ is a learnable parameter.

5. A device for extracting building vector topological structure based on diffusion model, characterized in that: include: An acquisition module is used to acquire the remote sensing image to be extracted; An extraction module is used to input the remote sensing image to be extracted into a pre-constructed diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, wherein the pre-constructed diffusion model is optimized by an initial diffusion model, and the initial diffusion model is constructed by training a multi-level diffusion Transformer network and a graph diffusion Transformer network with edge feature bias using a preset isomorphic training strategy.

6. The device according to claim 5, characterized in that The extraction module is further used for: A topologically focused Swin Transformer of the multi-level diffusion Transformer network is used to extract a multi-level edge attention map of a building from the remote sensing image to be extracted, and a feature map is generated by the multi-level diffusion Transformer network with the multi-level edge attention map of the building as a condition, and node coordinates and node features are extracted based on the feature map; A graph diffusion Transformer network based on the edge feature bias generates a new visual representation based on the node features to predict the adjacency matrix of the node; The building vector topology structure of the remote sensing image to be extracted is constructed according to the extracted node coordinates and the adjacency matrix.

7. The device according to claim 5, characterized in that Before inputting the remote sensing image to be extracted into a pre-built diffusion model to obtain the building vector topology structure of the remote sensing image to be extracted, the extraction module is further used to: Acquire multiple remote sensing images and building annotation vector data corresponding to the multiple remote sensing images; Performing data preprocessing on the multiple remote sensing images and the building annotation vector data to obtain a training sample library; The initial diffusion model is trained based on the training sample library, and the initial diffusion model is optimized using a preset loss function and an Adam optimizer to obtain the pre-constructed diffusion model.

8. The device according to claim 7, characterized in that The preset loss function is: Among them, τ is the loss function number, T is the total number of loss functions, including L node and L graph , σ is a learnable parameter.

9. An electronic device, characterized in that: Including memory and processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method for extracting building vector topology structure based on the diffusion model as described in any one of claims 1 to 4.

10. A computer program product, wherein the computer-readable storage medium stores a computer program, characterized in that: When the program is executed by a processor, the method for extracting building vector topological structure based on a diffusion model as described in any one of claims 1 to 4 is implemented.

Images