High-resolution remote sensing image building function type classification method based on multi-hop graph neural network

Abstract

The invention relates to a high-resolution remote sensing image building function type classification method based on a multi-hop graph neural network, and belongs to the field of remote sensing image classification. The method comprises the following steps: firstly, constructing a CNN-Transform double-encoder semantic segmentation model to extract building area information, then, constructing graph structure data based on the building area information, and then, constructing a multi-hop graph neural network to identify a building function type; according to the CNN-Transform double-encoder semantic segmentation model, local space detail features are kept while global context features are extracted; according to the multi-hop graph neural network, the multi-hop neighborhood features of the nodes are calculated in advance, and the multi-hop graph neural network can be used for large-scale graph structure data only by calculating once in a CPU or a distributed system. The network supports small-batch training and is easily expanded to a large data set, the complexity and calculation cost of node interaction in the training and prediction process are eliminated, the accuracy and expansibility of building function type classification are enhanced, and technical support is provided for efficiently obtaining building function types in remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image classification, and in particular relates to a method for classifying building function types in high-resolution remote sensing images based on a multi-hop graph neural network. Background Art

[0002] The information of building function types in urban areas plays an important role in application fields such as urban functional area identification and urban environmental variable inversion. Detailed information on urban building types is helpful for urban population estimation, urban land use and functional area identification, urban ecological environment assessment and other tasks. Existing building area extraction methods mostly further divide building types by analyzing the morphology, spatial relationship and other two-dimensional features of building objects extracted from images, and it is difficult to identify low-rise and high-rise buildings with similar shapes. However, the acquisition cost of high-precision LiDAR data and high-resolution stereo relative images with three-dimensional features is high, which limits their large-scale application. In recent years, deep learning technologies represented by graph neural networks have achieved excellent results in the field of computer vision. Graph neural networks are suitable for processing non-Euclidean spatial data and have superior structural feature abstraction capabilities. They can not only effectively mine the deep structural information of graph data, but also automatically integrate the rich attribute information of graph node elements, providing possibilities for the classification of building function types. Summary of the invention

[0003] The purpose of the present invention is to provide a high-resolution remote sensing image building function type classification method based on a multi-hop graph neural network. The graph neural network based on multi-hop neighborhood feature interaction overcomes the rapid expansion of adjacent node interactions in traditional GNNs methods that leads to high computational costs, and excessive adjacent node information interactions that make it difficult to distinguish between nodes of different categories, providing technical support for efficiently acquiring urban building function type information in remote sensing images.

[0004] To achieve the above object, the technical solution of the present invention is: a method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network, comprising:

[0005] Construct a CNN-Transformer dual encoder semantic segmentation model to extract building area information;

[0006] Construct graph structure data based on building area information;

[0007] Construct a multi-hop graph neural network to identify building functional types.

[0008] In one embodiment of the present invention, the method specifically comprises the following steps:

[0009] Step S1, obtaining high-resolution remote sensing images of the study area;

[0010] Step S2, constructing a CNN-Transformer dual encoder semantic segmentation model combining a CNN encoder and a Transformer encoder to extract building area information from high-resolution remote sensing images;

[0011] Step S3, preparing a binary building classification sample set, dividing the image into buildings and non-buildings, performing sliding cropping and sample set enhancement operations on the image, and training a CNN-Transformer dual encoder semantic segmentation model to extract building area information;

[0012] Step S4, constructing graph structure data of the high-resolution image based on the building area information extracted in step S3, including determining graph nodes, generating node adjacency relationships, and extracting node features;

[0013] Step S5, constructing a multi-hop graph neural network, including calculating node multi-hop neighborhood features, adding jump order encoding vectors, multi-hop neighborhood feature nonlinear interaction, and node classification;

[0014] Step S6, prepare a multi-category building function type sample set, define the building function type as high-density residential buildings, apartment buildings, comprehensive buildings, factory-type buildings, and single buildings, use a multi-hop graph neural network to identify the building function type for training and prediction, and finally identify the building function type in the study area.

[0015] In one embodiment of the present invention, in step S1, the acquired high-resolution remote sensing image of the study area is preprocessed, including radiation correction, orthorectification, atmospheric correction, image fusion, image registration and image resampling operations.

[0016] In one embodiment of the present invention, step S2 specifically includes the following steps:

[0017] Step S21, construct a CNN encoder to extract local features of the image to capture fine-grained spatial detail information of the image; specifically, use EfficientNet-B3 pre-trained on ImageNet as the backbone, which first uses a 3×3 convolutional layer Conv to extract shallow features, and then uses batch normalization Batch normal and Sigmoid weighted linear unit SiLU function; then use 4 MBConv modules to extract more levels of local spatial detail features;

[0018] Step S22, constructing a Transformer encoder to extract long-range global context features of the image; specifically, using SwinV2 pre-trained on ImageNet as the backbone, the backbone first divides the image into non-overlapping patches, and then uses a linear embedding layer to project the features of each patch into a high-dimensional space; then, extracting global context features based on the windows multi-head self-attention mechanism W-MSA and the moving windows multi-head self-attention mechanism SW-MSA;

[0019] Step S23, construct a CNN and Transformer fusion module BiFFM to fuse local spatial detail features and global context features to improve the recognition ability of buildings and non-buildings; BiFFM performs targeted processing on the features extracted by the CNN encoder and the Transformer encoder respectively; specifically, for the local spatial detail features extracted by the CNN encoder, first, construct a 3×3 deformable convolution module DFConv to extract the complex shape features of the building, and use the space and channel compression and excitation module scSE to enhance the semantic features of the building; then construct a deep separable convolution module DSConv to further refine the semantic features of the building; finally, construct P C The module performs element-by-element multiplication to extract the processed local spatial detail features, P C The calculation formula is as follows:

[0020] P C =δ(DSC(scSE(DFC(C n ))))×C n

[0021] Where δ represents the Sigmoid function, C n is the feature of the CNN encoder at level n;

[0022] For the global context features extracted by the Transformer encoder, we first construct S AM The module extracts all level features, S AM The calculation formula of the module is as follows:

[0023] S AM =DSC(AvgPool(T n ))+DSC(MaxPool(T n ))

[0024] Then construct S F The module refines the global context features, S F The calculation formula of the module is as follows:

[0025] S F=δ(DSC(ReLU(S AM )))×T n

[0026] Finally, the series S F and P C To fuse local spatial detail features and global context features;

[0027] Step S24, construct the attention gate ATG module and the attention residual ATR module as decoders to further improve the model's ability to detect buildings and generate the final building area information; ATG and ATR can enhance the representation of fine-grained spatial detail features and alleviate the boundary blur caused by shadow occlusion; specifically, the ATG module uses DSConv with different expansion rates to capture contextual information of different scales; in addition, construct F ED The module refines the features extracted by the encoder; F ED The calculation formula is as follows:

[0028] F ED =δ(BN(DSC(BR((DSC(E F )+DSC(D F )))))×E F

[0029] Where δ represents the Sigmoid function, BR represents the BN layer and the rectified linear unit ReLU, and E F represents the encoder feature, D F represents decoder features;

[0030] The ATR module is used to enhance the local spatial detail features and alleviate the blurring of building boundaries caused by shadow occlusion. The calculation formula of the ATR module is as follows:

[0031] O F =cSe(σ(BR(σ(BR(I F ))))+BN(σ(I F )))

[0032] Where σ is a 3×3 convolution, cSe refers to the spatial compression and channel excitation module for channel-level semantic feature enhancement.

[0033] In one embodiment of the present invention, step S3 specifically includes the following steps:

[0034] Step S31: Based on the high-resolution image data pre-processed in step S1, the built-in surface vector construction method of ArcGIS software is used to determine the real boundary of the building through manual visual interpretation, and field information is added for each building, where the pixel value 1 represents a building and 0 represents a non-building;

[0035] Step S32: based on step S31, convert the vector label data into raster tif format and add the map projection information of the image;

[0036] Step S33, based on the sample data set constructed in S31 and S32, the data is cropped to a size of 256×256 pixels using a sliding window method; the sample data set is divided into a training set, a validation set, and a test set according to a 6:2:2 ratio; the sample data set is simultaneously expanded using methods including horizontal and vertical flipping and affine transformation to enhance the generalization ability of the model.

[0037] In one embodiment of the present invention, step S4 specifically includes the following steps:

[0038] Step S41, segmenting the building area information to construct graph structure data; based on the building area information obtained in step S24, superimposing the original RGB image, so as to more accurately follow the real object boundary for segmentation, improve the detail expression of the object, and facilitate subsequent classification tasks; then, selecting a multi-scale segmentation algorithm, segmenting the building area information and optimizing the segmentation parameters to obtain a more ideal segmentation result;

[0039] Step S42, determine the graph nodes and generate the spatial adjacency relationship between the nodes; first, based on the image segmentation result of S41, each image object in the scene is regarded as a graph node; then, the Delaunay triangulation method is used to determine the spatial adjacency relationship of the nodes to obtain the spatial adjacency matrix; specifically, for spatially adjacent image objects, their adjacency relationship is defined as 1, and the adjacency relationship between non-adjacent objects is defined as 0. The adjacency matrix calculation formula is as follows:

[0040]

[0041] In the formula, A ij Represents the adjacency relationship between object i and object j;

[0042] Step S43, calculate node features, including node category features and node deep semantic features; first, define the node category features through manual visual interpretation; then, construct a deep feature extractor to extract the deep semantic features of the nodes; specifically, the deep feature extractor is composed of a CNN encoder and a Transformer encoder, which respectively extract local features and global features of the nodes and fuse the features; finally, concatenate the node category features and the deep semantic features as all the features of the nodes to further improve the distinguishability of the nodes.

[0043] In one embodiment of the present invention, step S5 specifically includes the following steps:

[0044] Step S51, calculating the multi-hop neighborhood features of the node; based on the original features of the node obtained in step S43, calculating the multi-hop neighborhood features of each node to extract the multi-scale neighborhood information of the node; the calculation formula of the multi-hop neighborhood features is as follows:

[0045]

[0046] In the formula, Contains multi-hop neighborhood information, is the normalized l-order adjacency matrix, X represents the node feature matrix, X p represents the pth node, represents the first-order neighborhood features of node p, Represents the Lth-order neighborhood features of node p;

[0047] Step S52: Use a linear transformation function to convert multi-hop neighborhood features into a jump embedding vector to enhance the representativeness of the node multi-hop neighborhood features; share linear transformation to reduce the number of parameters and improve computational efficiency, and ensure the comparison and fusion of features of different neighborhoods in the same space; in addition, add a jump encoding vector to the jump embedding vector to fuse and learn the jump order information; the linear transformation function and jump embedding vector calculation formula are as follows:

[0048]

[0049] Where f is the linear transformation function;

[0050]

[0051] In the formula, E order ∈R 1×L×d is a learnable multi-hop sequential encoding vector;

[0052] Step S53: nonlinear interaction of node multi-hop neighborhood features; realize nonlinear interaction of nodes through GNNs to enhance node differentiation without node interaction; specifically, first, construct a multi-hop neighborhood feature graph G hop , use standard GNNs to model interactions on multi-hop neighborhood feature maps; then, use the standard multi-head self-attention mechanism to realize jump interactions to capture the complex dependencies between multi-hop neighborhood features and generate new feature representations; the calculation formula of nonlinear interaction and attention mechanism is as follows:

[0053]

[0054] In the formula, is node v i The features after skip interaction at the kth layer, is node v i Fully connected jump feature map of

[0055]

[0056] Where W Q , W K and W V represents the learnable matrix in each interaction layer; Represents the weighted feature interaction graph of multi-hop neighborhood features, H i is node v i Features after jump interaction;

[0057] Step S54: Identify the functional type of the building by node classification; specifically, first apply the fusion function to all the jump interaction features Converted into a single representation Z∈R N×d ; Then, a linear layer with a softmax activation function is used to classify nodes; the specific calculation formula is as follows:

[0058] Z=fusion(H K ),

[0059] In the formula, fusion represents the fusion function, linear represents the linear function, and H K and Respectively represent the features of all nodes and node i after hopping interaction at the kth layer, Represents the final classification prediction result.

[0060] In one embodiment of the present invention, step S6 specifically includes: based on the results obtained in S41, a multi-category building function type sample set is prepared to train a multi-hop graph neural network model; specifically, the built-in surface vector construction method of ArcGIS software is used to define the building function type through manual visual interpretation, and field information is added, and different pixel values ​​are used to represent different building function types, including high-density residential buildings, apartment buildings, comprehensive buildings, factory-type buildings, and single buildings. The sample data set is divided into a training set, a test set, and a validation set in a ratio of 6:2:2, and the model is trained, tested, and validated respectively, and the optimal weight of the model is saved, the image of the study area is predicted, and finally the building function type information of the study area is extracted.

[0061] The present invention also provides a high-resolution remote sensing image building function type subsystem based on a multi-hop graph neural network, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0062] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) The method of the present invention solves the problem that traditional semantic segmentation methods cannot fully utilize high-level semantic features at a fine granularity by constructing a CNN-Transformer dual encoder semantic segmentation model. Through this model, it is possible to maintain local spatial detail features while extracting global context features, thereby improving the segmentation accuracy of small-sized, shadow-occluded, and other objects with similar heights to buildings. (2) After extracting building area information, the method of the present invention further constructs graph structure data, which helps capture the complex relationships and spatial layouts between buildings and provides rich context information for subsequent identification of building functional types. (3) The method of the present invention constructs a multi-hop graph neural network to identify building functional types. This method pre-calculates the multi-hop neighborhood features of nodes and only needs to be calculated once for large-scale graph structure data, effectively reducing computing and memory costs. The network supports small batch training and has strong scalability for large graph structure data sets, providing technical support for processing large-scale remote sensing image data and remote sensing extraction and classification of urban buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a method flow chart of a preferred embodiment of the present invention;

[0066] Figure 2 This is a structural diagram of a semantic segmentation network combined with a CNN-Transformer dual encoder according to a preferred embodiment of the present invention;

[0067] Figure 3 Flow chart of using multi-hop graph neural network to identify building function type classification in the preferred embodiment of the present invention

[0068] Figure 4 The result map of the building area in the study area extracted by the preferred embodiment of the present invention;

[0069] Figure 5 Resulting diagram of building function types in the study area identified by the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0071] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0072] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0073] The present invention provides a method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network, comprising:

[0074] Construct a CNN-Transformer dual encoder semantic segmentation model to extract building area information;

[0075] Construct graph structure data based on building area information;

[0076] Construct a multi-hop graph neural network to identify building functional types.

[0077] The following is the specific implementation process of the present invention.

[0078] like Figure 1 As shown, this embodiment provides a method for classifying building function types in high-resolution remote sensing images based on a multi-hop graph neural network, comprising the following steps:

[0079] Step S1: Obtain high-resolution remote sensing images of the study area and preprocess the images, including radiation correction, orthorectification, atmospheric correction, image fusion, image registration, and image resampling operations;

[0080] Step S2: Construct a semantic segmentation model combining CNN and Transformer dual encoders to extract building area information from high-resolution remote sensing images.

[0081] Step S3: Create a binary building classification sample set, divide the image into buildings and non-buildings, perform sliding cropping and sample set enhancement operations on the image, and train a CNN-Transformer dual encoder semantic segmentation model to extract building area information;

[0082] Step S4: Based on the building area information extracted in step S3, construct graph structure data of the high-resolution image, including determining graph nodes, generating node adjacency relationships, and extracting node features.

[0083] Step S5: construct a multi-hop graph neural network, including calculating node multi-hop neighborhood features, adding jump order encoding vectors, multi-hop neighborhood feature nonlinear interaction, and node classification.

[0084] Step S6: Create a multi-category sample set of building function types, define the building function types as high-density residential buildings, apartment buildings, comprehensive buildings, factory-type buildings, and single buildings, use a multi-hop graph neural network to identify the building function types for training and prediction, and finally identify the building function types in the study area.

[0085] In this example, step S1 specifically includes: selecting a high-resolution remote sensing image of the area to be studied, and preprocessing the image, including atmospheric and radiation correction, panchromatic-multispectral fusion and other operations.

[0086] In this example, step S2 specifically includes the following steps:

[0087] Step S21: Construct a CNN encoder to extract local features of the image to capture the fine-grained spatial detail information of the image. Specifically, the EfficientNet-B3 pre-trained on ImageNet is used as the backbone, which first uses a 3×3 convolutional layer (Conv) to extract shallow features, and then uses batch normalization (Batch normal) and Sigmoid weighted linear unit (SiLU) functions. Then, four MBConv modules are used to extract more levels of local spatial detail features.

[0088] Step S22: Construct a Transformer encoder to extract long-range global context features of the image. Specifically, SwinV2 pre-trained on ImageNet is used as the backbone, which first divides the image into non-overlapping patches, and then uses a linear embedding layer to project the features of each patch into a high-dimensional space. Then, global context features are extracted based on the windows multi-head self-attention mechanism (W-MSA) and the moving windows multi-head self-attention mechanism (SW-MSA).

[0089] Step S23: Construct a CNN and Transformer fusion module (BiFFM) to fuse local spatial detail features and global context features to improve the recognition ability of buildings and non-buildings. This module performs targeted processing on the features extracted by the CNN encoder and the Transformer encoder respectively. Specifically, for the local spatial detail features extracted by the CNN encoder, first, a 3×3 deformable convolution module (DFConv) is constructed to extract the complex shape features of the building, and a spatial and channel compression and excitation module (scSE) is used to enhance the semantic features of the building. Then a deep separable convolution module (DSConv) is constructed to further refine the semantic features of the building. Finally, P is constructed. C The module performs element-by-element multiplication to extract the processed local spatial detail features, P C The calculation formula is as follows:

[0090] P C =δ(DSC(scSE(DFC(C n ))))×C n

[0091] Where δ represents the Sigmoid function, C n is the feature of the CNN encoder at level n.

[0092] For the global context features extracted by the Transformer encoder, we first construct S AM The module extracts all level features, S AM The calculation formula of the module is as follows:

[0093] S AM =DSC(AvgPool(T n ))+DSC(MaxPool(T n ))

[0094] Then construct S F The module refines the global context features, S F The calculation formula is as follows:

[0095] S F =δ(DSC(ReLU(S AM )))×T n

[0096] Finally, the series S F and P C To fuse local spatial detail features and global context features.

[0097] Step S24: Construct the attention gate (ATG) module and the attention residual (ATR) module as decoders to further improve the model's ability to detect buildings and generate the final building area information. ATG and ATR can enhance the representation of fine-grained spatial detail features and alleviate the boundary blur caused by shadow occlusion. Specifically, the ATG module uses DSConv with different expansion rates to capture contextual information of different scales. In addition, construct F ED The module refines the features extracted by the encoder. ED The calculation formula is as follows:

[0098] F ED =δ(BN(DSC(BR((DSC(E F )+DSC(D F )))))×E F

[0099] Where δ represents the Sigmoid function, BR represents the BN layer and the rectified linear unit (ReLU).

[0100] The ATR module is used to enhance local spatial detail features and alleviate the blurring of building boundaries caused by shadow occlusion. The calculation formula of the ATR module is as follows:

[0101] O F =cSe(σ(BR(σ(BR(I F ))))+BN(σ(I F )))

[0102] Where σ is a 3×3 convolution, cSe refers to the spatial compression and channel excitation module for channel-level semantic feature enhancement.

[0103] In this example, step S3 specifically includes the following steps:

[0104] Step S31: Based on the high-resolution image data preprocessed in step S1, the built-in surface vector construction method of ArcGIS software is used to determine the real boundary of the building through manual visual interpretation, and field information is added for each building, where the pixel value 1 represents a building and 0 represents a non-building.

[0105] Step S32: Based on step S31, the vector label data is converted into raster tif format, and the map projection information of the image is added.

[0106] Step S33: Based on the sample data set constructed in S31 and S32, the data is cropped to a size of 256×256 pixels using a sliding window. The sample set is divided into a training set, a validation set, and a test set according to a 6:2:2 ratio. The sample data set is simultaneously expanded by horizontal and vertical flipping, affine transformation, etc. to enhance the generalization ability of the model.

[0107] In this example, step S4 specifically includes the following steps:

[0108] Step S41: Segment the building area information to construct graph structure data. Based on the building area information obtained in step S24, the original RGB image is superimposed to more accurately follow the real boundary of the object for segmentation, improve the detailed expression of the object, and facilitate subsequent classification tasks. Then, a multi-scale segmentation algorithm is selected to segment the building area information and optimize the segmentation parameters to obtain a more ideal segmentation result.

[0109] Step S42: Determine the graph nodes and generate the spatial adjacency relationship between the nodes. First, based on the image segmentation results of S41, each image object in the scene is regarded as a graph node. Then, the Delaunay triangulation method is used to determine the spatial adjacency relationship of the nodes and obtain the spatial adjacency matrix. Specifically, for spatially adjacent image objects, their adjacency relationship is defined as 1, and the adjacency relationship between non-adjacent objects is defined as 0. The adjacency matrix calculation formula is as follows:

[0110]

[0111] In the formula, A ij Represents the adjacency relationship between object i and object j.

[0112] Step S43: Calculate node features, including node category features and node deep semantic features. First, define the node category features through manual visual interpretation; then, construct a deep feature extractor to extract the deep semantic features of the nodes. Specifically, the deep feature extractor consists of a CNN encoder and a Transformer encoder, which extract local features and global features of the nodes respectively, and fuse the features. Finally, concatenate the node category features and the deep semantic features as the complete features of the node to further improve the distinguishability of the node.

[0113] In this example, step S5 specifically includes the following steps:

[0114] Step S51: Calculate the multi-hop neighborhood features of the node. Based on the original features of the node obtained in step S43, calculate the multi-hop neighborhood features of each node to extract the multi-scale neighborhood information of the node. The calculation formula of the multi-hop neighborhood features is as follows:

[0115]

[0116] In the formula, is the normalized adjacency matrix, Contains multi-hop neighborhood information.

[0117] Step S52: Use a linear transformation function to convert the multi-hop neighborhood features into a jump embedding vector to enhance the representativeness of the node's multi-hop neighborhood features. Shared linear transformations are used to reduce the number of parameters and improve computational efficiency, and to ensure that features from different neighborhoods are compared and fused in the same space. In addition, a jump encoding vector is added to the jump embedding vector to fuse and learn the jump order information. The linear transformation function and jump embedding vector calculation formula are as follows:

[0118]

[0119] Where f is the linear transformation function.

[0120]

[0121] In the formula, E order ∈R 1×L×d is a learnable multi-hop sequential encoding vector.

[0122] Step S53: Nonlinear interaction of node multi-hop neighborhood features. GNNs are used to realize nonlinear node interaction to enhance node differentiation without node interaction. Specifically, first, a multi-hop neighborhood feature graph G is constructed. hop , using standard GNNs to model interactions on multi-hop neighborhood feature graphs. Then, the standard multi-head self-attention mechanism is used to implement hop interactions to capture the complex dependencies between multi-hop neighborhood features and generate new feature representations. The calculation formula for nonlinear interactions and attention mechanisms is as follows:

[0123]

[0124] In the formula, is node v i The features after skip interaction at the kth layer, is node v i Fully connected skip feature map.

[0125]

[0126] Where W Q , W K and W V represents the learnable matrix in each interaction layer. Weighted feature interaction graph representing multi-hop neighborhood features.

[0127] Step S54: Identify the functional type of the building by node classification. Specifically, firstly, a fusion function is applied to all the jump interaction features. Converted into a single representation Z∈R N×d Then, a linear layer with a softmax activation function is used to classify nodes. The specific calculation formula is as follows:

[0128] Z=fusion(H K ),

[0129] In this example, step S6 specifically includes: based on the results obtained in S41, a multi-category building function type sample set is prepared to train a multi-hop graph neural network model. Specifically, the built-in face vector construction method of ArcGIS software is used to define the building function type through manual visual interpretation, and field information is added. Different pixel values ​​are used to represent different building function types, including high-density residential buildings, apartment buildings, comprehensive buildings, factory-type buildings, and single buildings. The sample set is divided into a training set and a validation set at a ratio of 6:2:2, and the model is trained and validated respectively, and the optimal weight of the model is saved. The image of the study area is predicted, and finally the building function type information of the study area is extracted.

[0130] This example takes the main urban area of ​​X City, X Province as the research object, uses the GF-2 high-resolution remote sensing image data in 2020, standardizes its spatial resolution to 1m after preprocessing, and uses three bands of red, green, and blue. Results After manual interpretation, this example obtains a total of 5458 pairs of building function type samples and corresponding label data with a size of 256×256 pixels, which are divided into training set, validation set, and test set in a ratio of 6:2:2 for building function type classification.

[0131] Figure 2 The structure diagram of the semantic segmentation network combined with the CNN-Transformer dual encoder constructed in this embodiment is shown. As can be seen from the figure, the model is mainly composed of three parts: CNN encoder, Transformer encoder and decoder. The CNN encoder uses EfficientNet-B3 pre-trained on ImageNet as the backbone, which first uses 3×3 convolutional layers (Conv) to extract shallow features, and then uses batch normalization (Batch normal) and Sigmoid weighted linear unit (SiLU) functions. Then, more levels of local spatial detail features are extracted through 4 MBConv modules. The Transformer encoder uses SwinV2 pre-trained on ImageNet as the backbone, which first divides the image into non-overlapping patches, and then uses a linear embedding layer to project the features of each patch into a high-dimensional space. Then, global context features are extracted based on the windows multi-head self-attention mechanism (W-MSA) and the mobile windows multi-head self-attention mechanism (SW-MSA). Finally, the attention gate (ATG) module and the attention residual (ATR) module are constructed as decoders to extract building area information.

[0132] Figure 3 The figure shows the structure of the multi-hop graph neural network constructed in this embodiment. As can be seen from the figure, the model mainly consists of calculating multi-hop neighborhood features, adding jump order vectors, multi-hop neighborhood feature nonlinear interaction, node fusion and classification.

[0133] Figure 4 and Figure 5 The figures show the partial building area extraction result and building function type recognition result of this embodiment on the GF-2 image of City X. It can be seen from the figure that the building area results extracted using the proposed method are highly consistent with the ground truth, with less misclassification and omission. The experimental results show the effectiveness of the proposed method for urban building area extraction.

[0134] The present invention also provides a high-resolution remote sensing image building function type subsystem based on a multi-hop graph neural network, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0135] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network, characterized in that: include: Construct a CNN-Transformer dual encoder semantic segmentation model to extract building area information; Construct graph structure data based on building area information; Construct a multi-hop graph neural network to identify building functional types.

2. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 1 is characterized in that: The method specifically comprises the following steps: Step S1, obtaining high-resolution remote sensing images of the study area; Step S2, constructing a CNN-Transformer dual encoder semantic segmentation model combining a CNN encoder and a Transformer encoder to extract building area information from high-resolution remote sensing images; Step S3, preparing a binary building classification sample set, dividing the image into buildings and non-buildings, performing sliding cropping and sample set enhancement operations on the image, and training a CNN-Transformer dual encoder semantic segmentation model to extract building area information; Step S4, constructing graph structure data of the high-resolution image based on the building area information extracted in step S3, including determining graph nodes, generating node adjacency relationships, and extracting node features; Step S5, constructing a multi-hop graph neural network, including calculating node multi-hop neighborhood features, adding jump order encoding vectors, multi-hop neighborhood feature nonlinear interaction, and node classification; Step S6, prepare a multi-category building function type sample set, define the building function type as high-density residential buildings, apartment buildings, comprehensive buildings, factory-type buildings, and single buildings, use a multi-hop graph neural network to identify the building function type for training and prediction, and finally identify the building function type in the study area.

3. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 2 is characterized in that: In step S1, the acquired high-resolution remote sensing images of the study area are preprocessed, including radiation correction, orthorectification, atmospheric correction, image fusion, image registration and image resampling operations.

4. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 2 is characterized in that: Step S2 specifically includes the following steps: Step S21, construct a CNN encoder to extract local features of the image to capture fine-grained spatial detail information of the image; specifically, use EfficientNet-B3 pre-trained on ImageNet as the backbone, which first uses a 3×3 convolutional layer Conv to extract shallow features, and then uses batch normalization Batch normal and Sigmoid weighted linear unit SiLU function; then use 4 MBConv modules to extract more levels of local spatial detail features; Step S22, constructing a Transformer encoder to extract long-range global context features of the image; specifically, using SwinV2 pre-trained on ImageNet as the backbone, the backbone first divides the image into non-overlapping patches, and then uses a linear embedding layer to project the features of each patch into a high-dimensional space; then, extracting global context features based on the windows multi-head self-attention mechanism W-MSA and the moving windows multi-head self-attention mechanism SW-MSA; Step S23, construct a CNN and Transformer fusion module BiFFM to fuse local spatial detail features and global context features to improve the recognition ability of buildings and non-buildings; BiFFM performs targeted processing on the features extracted by the CNN encoder and the Transformer encoder respectively; specifically, for the local spatial detail features extracted by the CNN encoder, first, construct a 3×3 deformable convolution module DFConv to extract the complex shape features of the building, and use the space and channel compression and excitation module scSE to enhance the semantic features of the building; then construct a deep separable convolution module DSConv to further refine the semantic features of the building; finally, construct P C The module performs element-by-element multiplication to extract the processed local spatial detail features, P C The calculation formula is as follows: P C =δ(DSC(scSE(DFC(C n ))))×C n Where δ represents the Sigmoid function, C n is the feature of the CNN encoder at level n; For the global context features extracted by the Transformer encoder, we first construct S AM The module extracts all level features, S AM The calculation formula of the module is as follows: S AM =DSC(AvgPool(T n ))+DSC(MaxPool(T n )) Then construct S F The module refines the global context features, S F The calculation formula of the module is as follows: S F =δ(DSC(ReLU(S AM )))×T n Finally, the series S F and P C To fuse local spatial detail features and global context features; Step S24, construct the attention gate ATG module and the attention residual ATR module as decoders to further improve the model's ability to detect buildings and generate the final building area information; ATG and ATR can enhance the representation of fine-grained spatial detail features and alleviate the boundary blur caused by shadow occlusion; specifically, the ATG module uses DSConv with different expansion rates to capture contextual information of different scales; in addition, construct F ED The module refines the features extracted by the encoder; F ED The calculation formula is as follows: F ED =δ(BN(DSC(BR((DSC(E F )+DSC(D F )))))×E F Where δ represents the Sigmoid function, BR represents the BN layer and the rectified linear unit ReLU, and E F represents the encoder feature, D F represents decoder features; The ATR module is used to enhance the local spatial detail features and alleviate the blurring of building boundaries caused by shadow occlusion. The calculation formula of the ATR module is as follows: The F =cSe(σ(BR(σ(BR(I F ))))+BN(σ(I F ))) Where σ is a 3×3 convolution, cSe refers to the spatial compression and channel excitation module for channel-level semantic feature enhancement.

5. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 2 is characterized in that: Step S3 specifically includes the following steps: Step S31: Based on the high-resolution image data pre-processed in step S1, the built-in surface vector construction method of ArcGIS software is used to determine the real boundary of the building through manual visual interpretation, and field information is added for each building, where the pixel value 1 represents a building and 0 represents a non-building; Step S32: based on step S31, convert the vector label data into raster tif format and add the map projection information of the image; Step S33, based on the sample data set constructed in S31 and S32, the data is cropped to a size of 256×256 pixels using a sliding window method; the sample data set is divided into a training set, a validation set, and a test set according to a 6:2:2 ratio; the sample data set is simultaneously expanded using methods including horizontal and vertical flipping and affine transformation to enhance the generalization ability of the model.

6. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 4 is characterized in that: Step S4 specifically includes the following steps: Step S41, segmenting the building area information to construct graph structure data; based on the building area information obtained in step S24, superimposing the original RGB image, so as to more accurately follow the real object boundary for segmentation, improve the detail expression of the object, and facilitate subsequent classification tasks; then, selecting a multi-scale segmentation algorithm, segmenting the building area information and optimizing the segmentation parameters to obtain a more ideal segmentation result; Step S42, determine the graph nodes and generate the spatial adjacency relationship between the nodes; first, based on the image segmentation result of S41, each image object in the scene is regarded as a graph node; then, the Delaunay triangulation method is used to determine the spatial adjacency relationship of the nodes to obtain the spatial adjacency matrix; specifically, for spatially adjacent image objects, their adjacency relationship is defined as 1, and the adjacency relationship between non-adjacent objects is defined as 0. The adjacency matrix calculation formula is as follows: In the formula, A ij Represents the adjacency relationship between object i and object j; Step S43, calculate node features, including node category features and node deep semantic features; first, define the node category features through manual visual interpretation; then, construct a deep feature extractor to extract the deep semantic features of the nodes; specifically, the deep feature extractor is composed of a CNN encoder and a Transformer encoder, which respectively extract local features and global features of the nodes and fuse the features; finally, concatenate the node category features and the deep semantic features as all the features of the nodes to further improve the distinguishability of the nodes.

7. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 6 is characterized in that: Step S5 specifically includes the following steps: Step S51, calculating the multi-hop neighborhood features of the node; based on the original features of the node obtained in step S43, calculating the multi-hop neighborhood features of each node to extract the multi-scale neighborhood information of the node; the calculation formula of the multi-hop neighborhood features is as follows: In the formula, Contains multi-hop neighborhood information, is the normalized l-order adjacency matrix, X represents the node feature matrix, X p represents the pth node, represents the first-order neighborhood features of node p, Represents the Lth-order neighborhood features of node p; Step S52: Use a linear transformation function to convert multi-hop neighborhood features into a jump embedding vector to enhance the representativeness of the node multi-hop neighborhood features; share linear transformation to reduce the number of parameters and improve computational efficiency, and ensure the comparison and fusion of features of different neighborhoods in the same space; in addition, add a jump encoding vector to the jump embedding vector to fuse and learn the jump order information; the linear transformation function and jump embedding vector calculation formula are as follows: Where f is the linear transformation function; In the formula, E order ∈R 1×L×d is a learnable multi-hop sequential encoding vector; Step S53: nonlinear interaction of node multi-hop neighborhood features; realize nonlinear interaction of nodes through GNNs to enhance node differentiation without node interaction; specifically, first, construct a multi-hop neighborhood feature graph G hop , use standard GNNs to model interactions on multi-hop neighborhood feature maps; then, use the standard multi-head self-attention mechanism to realize jump interactions to capture the complex dependencies between multi-hop neighborhood features and generate new feature representations; the calculation formula of nonlinear interaction and attention mechanism is as follows: In the formula, is node v i The features after skip interaction at the kth layer, is node v i Fully connected jump feature map of Where W Q , W K and W V represents the learnable matrix in each interaction layer; Represents the weighted feature interaction graph of multi-hop neighborhood features, H i is node v i Features after jump interaction; Step S54: Identify the functional type of the building by node classification; specifically, first apply the fusion function to all the jump interaction features Converted into a single representation Z∈R N×d ; Then, a linear layer with a softmax activation function is used to classify nodes; the specific calculation formula is as follows: In the formula, fusion represents the fusion function, linear represents the linear function, and H K and Respectively represent the features of all nodes and node i after hopping interaction at the kth layer, Represents the final classification prediction result.

8. The method for classifying building function types from high-resolution remote sensing images based on a multi-hop graph neural network according to claim 6 is characterized in that: Step S6 specifically includes: based on the results obtained in S41, a multi-category building function type sample set is prepared to train a multi-hop graph neural network model; specifically, the built-in face vector construction method of ArcGIS software is used to define the building function type through manual visual interpretation, and field information is added, and different pixel values ​​are used to represent different building function types, including high-density residential buildings, apartment buildings, comprehensive buildings, factory-type buildings, and single buildings. The sample data set is divided into a training set, a test set, and a validation set at a ratio of 6:2:2, and the model is trained, tested, and validated respectively, and the optimal model weight is saved to predict the image of the study area, and finally the building function type information of the study area is extracted.

9. A high-resolution remote sensing image building function type subsystem based on a multi-hop graph neural network, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images