Urban functional area identification method based on remote sensing and social perception data fusion

Through the combination of adaptive interactive fusion and hollow convolution and attention mechanism, the problems of data source heterogeneity and semantic imbalance in urban functional area identification in traditional methods are solved, and high-precision urban functional area classification and recognition effect are achieved .

CN118968235AInactive Publication Date: 2024-11-15NINGBO UNIV

Patent Information

Application Number
CN202410752145.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-12
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The traditional urban functional area identification method based on a single data source cannot fully reflect the complex functional distribution and spatial structure of the city, and the multimodal data fusion algorithm ignores the imbalance of semantic distribution between remote sensing and socially perceived data, making it difficult to make full use of complementarity. information.

Method used

Adaptive interactive fusion method is adopted to obtain multi-source heterogeneous data for preprocessing and division, and introduce hollow convolution and attention mechanisms, and weight assignment is given from the two dimensions of space and channel, combining cross entropy loss and dice category imbalance loss. Perform supervised deep network training to extract key feature information in the image.

Benefits of technology

High-precision urban functional area classification is realized, the interference of modal differences and redundant feature information is reduced, the structural information and identification effect of the fusion result is improved, and the category imbalance of urban functional area is steadily handled.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968235A_ABST
    Figure CN118968235A_ABST
Patent Text Reader

Abstract

The invention relates to a remote sensing and social perception data fused urban functional area identification method, which comprises the following steps: acquiring multi-source heterogeneous data, and constructing a data set; dividing the data set into batches for training; carrying out adaptive interactive fusion on the multi-source heterogeneous data; carrying out weight endowing on the feature map from two dimensions of space and channel; key feature information under different scales in the image is extracted; carrying out supervised deep network training, and establishing a loss model fusing an image segmentation result and a label graph; in the training process, the segmentation result is made to continuously approach the label image, and a high-precision city function area classification result graph is obtained. The method has the beneficial effects that on the basis of comprehensively utilizing each data source, the interference of redundant feature information is reduced, and the structural information and recognition effect of a fusion result are effectively improved; the contribution of key features to urban functional area identification is highlighted in a multi-scale manner, and the influence of class imbalance of urban functional areas on experimental results is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing and application, and more specifically, to an urban functional area recognition method integrating remote sensing and social perception data. Background Art

[0002] In urban planning and land use management, accurate identification of urban functional areas is crucial to improving land resource utilization efficiency and urban development quality. However, traditional methods based on a single data source have limitations and cannot fully reflect the complex functional distribution and spatial structure of the city. Therefore, the use of remote sensing and social perception multimodal data fusion technology has become an effective way to solve this problem.

[0003] Multi-source remote sensing and social perception data fusion technology can comprehensively utilize the physical information of remote sensing image data, such as spectral, texture and morphological features, combined with the elevation information of building footprints and the socio-economic information of POI (points of interest). This comprehensive use of information provided by different data sources helps to form a more comprehensive and accurate urban functional area identification result. For example, spectral information can distinguish the types of different objects, texture features can help identify the boundaries of buildings and natural cover, while elevation information can reveal the spatial distribution characteristics of objects, and POI data provides socio-economic background information of urban functional distribution. In the fusion process, appropriate algorithms and models need to be used to integrate multimodal data and solve the heterogeneity and inconsistency between different data sources. Common fusion algorithms include but are not limited to pixel-level fusion, feature-level fusion and decision-level fusion. Feature-level fusion is to combine features from different data sources or feature extraction methods to form a comprehensive feature vector before classification. Decision-level fusion is to fuse the decision results of each classifier. Most of these methods ignore modal differences, directly perform integration, and do not take into account the imbalance in semantic distribution between remote sensing and social perception data, which makes it difficult to fully utilize the complementary information between different modalities to reduce modal differences. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and propose an urban functional area identification method that integrates remote sensing and social perception data.

[0005] First, a method for identifying urban functional areas by integrating remote sensing and social perception data is provided, including:

[0006] Step 1: Acquire multi-source heterogeneous data, pre-process the multi-source heterogeneous data, and construct a data set;

[0007] Step 2: Divide the data set into batches of different sizes and perform training in batches;

[0008] Step 3: adaptively and interactively fuse the multi-source heterogeneous data;

[0009] Step 4: For the feature map obtained through adaptive interactive fusion, introduce the dilated convolution and attention mechanism to assign weights from the two dimensions of space and channel;

[0010] Step 5: Extract key feature information at different scales in the image;

[0011] Step 6: Conduct supervised deep network training and use cross entropy loss, L2 norm and dice category imbalance loss to establish a loss model for fusion image segmentation results and label images; during the training process, the segmentation results are continuously approached to the label images to obtain high-precision urban functional area classification result graphics.

[0012] Preferably, in step 1, the multi-source heterogeneous data includes remote sensing images, building footprints and points of interest; the multi-source heterogeneous data is preprocessed, including: performing radiation calibration, atmospheric correction, orthorectification, and geometric correction operations on the remote sensing images, and fusing the panchromatic and multispectral images to obtain a multispectral image with high spatial resolution; using the spatial connection tool to connect the connected plots of building footprints and extract elevation information; screening the points of interest and classifying them after screening; the building footprints and points of interest need to be rasterized into graphics.

[0013] Preferably, in step 2, the data set is divided into a training set and a test set in proportion according to the number of pixels.

[0014] Preferably, step 3 comprises:

[0015] Step 3.1: Use an encoder to extract the features of the multi-source heterogeneous data, expressed as:

[0016] Conv_EN=σ(Maxpool(Conv 3 (σ(Conv 3 (σ(Maxpool(Conv 3 (σ(Conv 3 (input))))))))))

[0017] Among them, input represents the input image features, σ(·) represents the Relu activation function, Maxpool represents the maximum pooling layer, and Conv_EN represents the output result of each encoder;

[0018] Step 3.2: The features of the multi-source heterogeneous data are fused through an adaptive weight interaction fusion module, which is expressed as:

[0019] F F ,F R ,FP =MLP(M F ,M R ,M P )

[0020] M RF =concat(F R ,F F )

[0021] M RP =concat(F R ,F P )

[0022]

[0023]

[0024] M k =concat(M i ,M j )

[0025] Among them, M F ,M R ,M P They represent the building footprint feature map, remote sensing image feature map and POI feature map respectively, and F R ,F F ,F P It represents the output result of each input feature map after passing through the multi-layer perceptron MLP. concat represents the superposition of feature maps in the channel dimension. RF 、M RP 、M i and M j represents the intermediate fusion feature, is the Hadamard product, w and w' represent the weight coefficient matrix, where w'=1-w∈[0,1], M k It is the final output fusion feature result.

[0026] Preferably, step 4 comprises:

[0027] Step 4.1: For the input features, the channel attention module is used to adaptively capture the features of different channels, assign corresponding weights to each channel space, and highlight the channel information that is meaningful to the urban functional area classification task, and obtain the output feature map M. CH , expressed as:

[0028] M CH =CAB(M k )

[0029] Among them, CAB represents the channel attention module;

[0030] Step 4.2, pass the feature map into the hole convolution layer;

[0031] Step 4.3: Pass the feature map through the spatial attention module to adaptively emphasize the features of different positions in space, expressed as:

[0032] M SP =SAB(M DI )

[0033] Among them, M DI is the output feature map of the dilated convolution, and SAB is the spatial attention module.

[0034] Preferably, in step 4.2, first, down-sampling is performed using a convolutional layer to reduce the spatial dimension of the feature map and increase the number of channels; then, three convolutional layers with different dilation rates are used, each of which contains two sublayers; the first sublayer is a depthwise separable convolutional layer, which uses a 3x3 convolution kernel with dilation rates of 1, 2, and 3, respectively, to implement a dilation convolution operation on the input features; the second sublayer is a 1x1 convolutional layer, which is used to fuse channel information; each depthwise separable volume decomposes the standard volume into depthwise convolution and pointwise convolution; expressed as:

[0035] M DC =Conv 1 (DepthConv 3 (Down_3(M CH )))

[0036] M DI =concat(M CH ,Up_3(concat(M DC ,M DC ,M DC )))

[0037] Among them, DepthConv 3 Indicates a 3x3 depth convolution with a convolution kernel, Down_3 and Up_3 represent triple down / up sampling, respectively, and three M DC Represent the deep convolution feature maps with void rates of 1, 2, and 3 respectively.

[0038] Preferably, in step 5, multiple convolutional layers, pooling layers, and fully connected layers are used to map the vectors to attention weights of the same dimension as the input features, and these weights are used to weight the multi-scale features; then, the weighted features are fused with the original input features through residual connections. Since feature maps from the same level have certain commonalities, feature block weights need to be assigned and spatially stimulated; expressed as:

[0039] M out1 =σ(Conv 1(concat(σ(Conv 1 (M in )),σ(Conv 3 (M in )),σ(Conv 5 (M in )))))

[0040]

[0041] M out3 =FC(σ(FC(GAP(concat(M in ,M out2 )))))

[0042] Among them, M in and M out3 Represent the input feature map and output feature map of the multi-scale focusing module, M out1 and M out2 represents the intermediate feature map, GAP represents the global average pooling, and FC represents the full connection; in M out3 The second fully connected layer in the calculation is followed by a sigmoid function and a Dropout regularization layer.

[0043] Preferably, in step 6, the overall loss of the network is:

[0044]

[0045] Among them, β is a constant, λ is the balance factor of the three losses;

[0046]

[0047]

[0048] exist middle, y i,k and p i,k are the true value label and predicted probability value of the k-class of the i-th sample, and C is the total number of functional area categories; In the prediction result Y pr and the true label Y re are all probability distributions after the softmax operation, and the value at each position represents the model's predicted probability for the corresponding category; In the calculation of , these probability distributions are treated as binary masks; then, the elements of the binary masks are compared one by one, and the intersection is obtained by element-by-element multiplication and the union is obtained by element-by-element addition.

[0049] In a second aspect, a system for identifying urban functional areas by integrating remote sensing and social perception data is provided, which is used to execute any of the methods described in the first aspect, including:

[0050] An acquisition module is used to acquire multi-source heterogeneous data, pre-process the multi-source heterogeneous data, and construct a data set;

[0051] A partitioning module, used to divide the data set into batches of different sizes and perform training in batches;

[0052] A fusion module, used for adaptively and interactively fusing the multi-source heterogeneous data;

[0053] The weight assignment module is used to introduce dilated convolution and attention mechanism to the feature map obtained through adaptive interactive fusion, and assign weights from two dimensions: space and channel.

[0054] Extraction module, used to extract key feature information at different scales in the image;

[0055] The training module is used for supervised deep network training. It uses cross entropy loss, L2 norm and dice category imbalance loss to establish a loss model for fusion image segmentation results and label images. During the training process, the segmentation results are continuously approached to the label images to obtain high-precision urban functional area classification result graphics.

[0056] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0057] The beneficial effects of the present invention are: the present invention simultaneously utilizes the physical space information provided by remote sensing images, the building structure information provided by building footprints, and the socio-economic information provided by POIs, and by mining the interaction between the various data sources, it can robustly and efficiently fuse the feature information between the various modalities while narrowing the modal differences, and specifically design a robust high-precision recognition model for urban functional areas. On the basis of comprehensive utilization of various data sources, the interference of redundant feature information is reduced, and the structural information and recognition effect of the fusion results are effectively improved; the contribution of key features to the recognition of urban functional areas is highlighted at multiple scales, and the impact of the imbalance of categories of urban functional areas on the experimental results is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of an urban functional area identification method that integrates remote sensing and social perception data;

[0059] Figure 2 It is a schematic diagram of the adaptive weight interaction fusion module;

[0060] Figure 3 Schematic diagram of the channel attention-dilated convolution-spatial attention joint module;

[0061] Figure 4 Schematic diagram of the multi-scale focusing module;

[0062] Figure 5 Schematic diagram of experimental results of different comparison methods in Haishu District, Ningbo;

[0063] Figure 6 Schematic diagram of experimental results of different comparison methods in Zhenhai District, Ningbo;

[0064] Figure 7 Schematic diagram of the experimental results of different comparison methods at the junction of Ningbo Jiangbei-Zhenhai-Yinzhou District. DETAILED DESCRIPTION

[0065] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0066] Embodiment 1:

[0067] Embodiment 1 of the present application provides a method for identifying urban functional areas by fusing remote sensing and social perception data. The present invention invents an adaptive weight fusion module for remote sensing data and social perception data, fully utilizing the nonlinear differences and correlations of heterogeneous data, and effectively integrating complementary information on the basis of alleviating modal differences. In addition, a multi-scale focusing module is designed to amplify the contribution of key features to the identification of urban functional areas through multiple attentions, and the loss function is improved by taking into account the extreme imbalance of urban functional area categories.

[0068] Specifically, Figure 1 As shown, the method includes:

[0069] Step 1: Acquire multi-source heterogeneous data, preprocess the multi-source heterogeneous data, and construct a data set.

[0070] In step 1, the multi-source heterogeneous data includes remote sensing images, building footprints and points of interest (POI); the multi-source heterogeneous data is preprocessed, including: using ENVI to perform radiation calibration, atmospheric correction, orthorectification, and geometric correction operations on the remote sensing images, and fusing the panchromatic and multispectral images to obtain multispectral images with high spatial resolution; building footprints and POIs are screened and cleaned using ArcMap. Among them, the building footprints use the spatial connection tool to connect the connected plots and extract the elevation information; after the POI is screened, it is classified according to the national standard "Land Use Status Classification" revised by the Ministry of Land and Resources. In addition, the building footprints and points of interest need to be rasterized into graphics.

[0071] Step 2: Divide the data set into batches of different sizes and perform training in batches.

[0072] According to the preprocessed data obtained in step 1, due to experimental hardware limitations, it is impossible to directly input all the data into the network at once. The whole image is divided into image blocks of different sizes using a multiple cropping strategy, and training is performed in batches based on the idea of ​​PATCH. In addition, when dividing the training and test sets, it is not divided proportionally according to the total number of images, but divided proportionally according to the number of pixels, which can make the sample distribution of the model more uniform, reduce the risk of overfitting caused by excessive focus on certain categories or features in model training, and improve the robustness and generalization ability of the model.

[0073] Step 3: adaptively and interactively fuse the multi-source heterogeneous data.

[0074] There is strong feature heterogeneity among the three different modal data of remote sensing images, building footprints and POI. There are similarities, differences and correlations between the data sources. It is necessary to fully explore the available feature information and discard the redundant parts. By constructing a feature interaction fusion module, adaptive gain is used to realize the projection and fusion of complementary information.

[0075] Specifically, step 3 includes:

[0076] Step 3.1: Use an encoder architecture similar to VGG16 to extract features from the three multimodal data of input remote sensing images, POIs, and building footprints using the same encoding structure. A total of four stages of Encode are used, each stage consisting of a 3*3 convolution layer and a ReLU activation function σ(·) layer. Between the convolution layer and the ReLU layer in the second and fourth stages, an additional maximum pooling layer with a pooling window size of 2*2 is used to help extract spatial features. It can be expressed as:

[0077] Conv_EN=σ(Maxpool(Conv 3 (σ(Conv3 (σ(Maxpool(Conv 3 (σ(Conv 3 (input))))))))))

[0078] Among them, input represents the input image features, σ(·) represents the Relu activation function, Maxpool represents the maximum pooling layer, and Conv_EN represents the output result of each encoder;

[0079] Step 3.2: Establish an adaptive weight interaction fusion module, such as Figure 2 As shown. F Represents the building footprint feature map, M R Represents the remote sensing image feature map, M P Represents the POI feature map, in which the remote sensing image is a bridge connecting the other two types of data. Inspired by the non-local attention module, the characteristics of different data sources are used to give these data corresponding weights, promote the positive effect of effective features on fusion, and suppress the influence of irrelevant features on the fusion results.

[0080] First, each data source passes through a shared MLP layer to promote the interaction of combined features. Then, remote sensing images are used as the main body, and building footprints and POI data are supplemented and gradually integrated. A 1*1 convolutional layer is used to learn the first two features M RF and M RP The correlation between them is then passed through a ReLU layer and a 1*1 convolution layer, and regularized using the sigmoid function to obtain the weight coefficient matrix W∈[0,1] and W'=1-W∈[0,1]. The value of W is assigned to M RF , the value of W' is assigned to M RP , and then the final output feature mapping result is obtained through weighted calculation.

[0081] It is expressed as:

[0082] F F ,F R ,F P =MLP(M F ,M R ,M P )

[0083] M RF =concat(F R ,F F )

[0084] M RP =concat(F R ,F P )

[0085]

[0086]

[0087] M k =concat(M i ,M j )

[0088] Among them, M F , M R , M P They represent the building footprint feature map, remote sensing image feature map and POI feature map respectively, and F F , F R , F P It represents the output result of each input feature map after passing through the multi-layer perceptron MLP. is the Hadamard product, M i and M j is the intermediate fusion feature, M k It is the final output fusion feature result.

[0089] Step 4: For the feature map obtained through adaptive interactive fusion, Figure 3 As shown in the figure, the hole convolution and attention mechanism are introduced to assign weights from the two dimensions of space and channel, which enhances the feature expression ability and optimizes the network performance.

[0090] It should be noted that remote sensing images cover specific areas, while social perception data are mostly vectors based on space or time, and the two are heterogeneous. Therefore, remote sensing data is used as the main body and social perception data is used as an auxiliary. After extracting their respective features using a multi-branch coding structure, these data are given corresponding weights according to the characteristics of different data sources, and different data sources are gradually fused to ensure that each fusion step fully utilizes the information of each data. The fusion results obtained by this method can greatly alleviate the modal differences, allowing the model to learn the multimodal characteristics of the data as much as possible.

[0091] Step 4 includes:

[0092] Step 4.1: For the input features, the channel attention block (CAB) is used to adaptively capture the features of different channels, assign corresponding weights to each channel space, and highlight the channel information that is meaningful to the urban functional area classification task, and obtain the output feature map M. CH , expressed as:

[0093] M CH =CAB(M k )

[0094] Among them, CAB represents the channel attention module;

[0095] Step 4.2, pass the feature map to the dilated convolution layer; first, use the convolution layer for downsampling to reduce the spatial dimension of the feature map and increase the number of channels; then, use three convolution layers with different dilation rates, each of which contains two sublayers; the first sublayer is a depthwise separable convolution layer, which uses a 3x3 convolution kernel with dilation rates of 1, 2, and 3 to implement dilated convolution operations on input features; the second sublayer is a 1x1 convolution layer, which is used to fuse channel information; each depthwise separable volume decomposes the standard volume into depthwise convolution and pointwise convolution; expressed as:

[0096] M DC =Conv 1 (DepthConv 3 (Down_3(M CH )))

[0097] M DI =concat(M CH ,Up_3(concat(M DC ,M DC ,M DC )))

[0098] Among them, DepthConv 3 Indicates a 3x3 depth convolution with a convolution kernel, Down_3 and Up_3 represent triple down / up sampling, respectively, and three M DC Represent the deep convolution feature maps with void rates of 1, 2, and 3 respectively.

[0099] Step 4.3: Pass the feature map through the Spatial Attention Block (SAB) to adaptively emphasize the features of different locations in space, expressed as:

[0100] M SP =SAB(M DI )

[0101] Among them, M DI is the output feature map of the dilated convolution, and SAB is the spatial attention module.

[0102] Step 5: Extract key feature information at different scales in the image.

[0103] For the preliminary fused feature map, in order to enhance its feature expression ability, we first introduce a dilated convolution block based on the use of channel attention and spatial attention, split the depth-separable convolution into depth-wise convolution and point-by-point convolution, so that the model retains the finer local features in the functional area and expands the receptive field. Then, based on the difference in scale space, we use a multi-scale focusing module and multiple convolution kernels of different sizes to extract key feature information at different scales, and use jump connections to improve the sensitivity to the internal details of the functional area. Due to the diversity of functional area classification, a new category imbalance loss is added on the basis of the traditional cross entropy loss. The fused classification results generated by this method can maximize the use of each data information and obtain high-precision segmentation results.

[0104] Specifically, after step 4, the feature information of spatial channel optimization is obtained. In order to further extract the key feature information at different scales in the image and enhance its perceptual representation ability. Using a variety of convolutional layers, pooling layers and fully connected layers, the vectors are mapped to attention weights of the same dimension as the input features, and these weights are used to weight the multi-scale features. Then, the weighted features are fused with the original input features through residual connections. Since feature maps from the same level have certain commonalities, feature block weights need to be assigned to perform spatial excitation. Expressed as:

[0105] M out1 =σ(Conv 1 (concat(σ(Conv 1 (M in )),σ(Conv 3 (M in )),σ(Conv 5 (M in )))))

[0106]

[0107] M out3 =FC(σ(FC(GAP(concat(M in ,M out2 )))))

[0108] Among them, M in and M out3 Represent the input feature map and output feature map of the multi-scale focusing module, M out1 and M out2 represents the intermediate feature map, GAP represents the global average pooling, and FC represents the full connection; in M out3 The second fully connected layer in the calculation is followed by a sigmoid function and a Dropout regularization layer.

[0109] Step 6: Conduct supervised deep network training and use cross entropy loss, L2 norm and dice category imbalance loss to establish a loss model for fusion image segmentation results and label images; during the training process, the segmentation results are continuously approached to the label images to obtain high-precision urban functional area classification result graphics.

[0110] In step 6, due to the significant regional differences in different urbanization stages and the extremely unbalanced distribution of functional areas, the traditional cross entropy loss function is used. On this basis, a new dice loss is added This helps the network to classify pixels more accurately and improves the performance of the model. In addition, to avoid the risk of overfitting, the network weight parameters are regularized using the L2 norm. The overall loss of the network is:

[0111]

[0112] Among them, β is a constant with a value of 0.001, and λ is a balancing factor of the three losses; by jointly utilizing these losses, the entire segmentation network can learn complementary knowledge. and The specific calculation formula is:

[0113]

[0114]

[0115] exist middle, y i,k and p i,k are the true value label and predicted probability value of the k-class of the i-th sample, and C is the total number of functional area categories; In the prediction result Y pr and the true label Y re are all probability distributions after the softmax operation, and the value at each position represents the model's predicted probability for the corresponding category; In the calculation of , these probability distributions are treated as binary masks; that is, positions with probabilities greater than a certain threshold are treated as positive, and other positions are treated as negative. Then, the elements of the binary mask are compared one by one, and the intersection is obtained by element-by-element multiplication, and the union is obtained by element-by-element addition.

[0116] Embodiment 2:

[0117] Based on Example 1, Example 2 of the present application provides a more specific method for identifying urban functional areas by integrating remote sensing and social perception data.

[0118] The remote sensing data used in the example is the existing domestically produced Gaofen-2 satellite data, including panchromatic images with a spatial resolution of 1m and multispectral images with a resolution of 4m taken in 2019. The building footprint data used is the data collected by Bigemap in 2019, and the POI data is the data of Gaode Map API in 2019. As an implementation case, the present invention uses computer software ArcMap10.8, ENVI5.3, MATLAB2021a and Pycharm, and the TensorFlow 2.5.0 framework based on Intel(R) Core(TM) i9-10920XCPU and NVIDIA RTX 3090 GPU to realize the automatic operation process. The following is combined with Figure 1 Describe in detail the steps of fusing multi-source data and applying it to classification and recognition.

[0119] Step 1: Preprocess the dataset. First, image registration is required for remote sensing images acquired under real conditions. Based on the geographic registration tool of ArcMap software, manual auxiliary point selection correction is added to finally obtain the images of each dataset after spatial registration. For building footprint data, ArcMap's spatial connection tool is used to connect intersecting building plots to form regions, select building height attributes, and output vector images after rasterization. This is because the physical attributes of building height can make up for the shortcomings of extracting features from planes and analyzing the building laws of functional areas, and the heights of different types of buildings vary greatly. For POI data, ArcMap is used to screen and clean the data, filter out invalid data such as public toilets and newsstands, and refer to the Chinese national standard "Urban Land Classification and Planning and Construction Land Standards" to divide various POI categories into 13 categories. The regional category attributes are defined by the category with the highest POI frequency, and the output image is also rasterized.

[0120] Step 2: The ratio of the training set to the test set is set to 4:1. The division according to the number of pixels makes the sample distribution of the model more uniform, so as to avoid problems such as overfitting in deep learning training and improve the robustness and generalization ability of the model.

[0121] Step 3: When using building footprint data to identify functional areas of simple structures such as scenic spots, urban green spaces, vacant land, and transportation, some data redundancy may be introduced. However, when identifying areas composed of residential, commercial, industrial and other buildings with large differences, building footprints can improve the recognition effect by enhancing the corresponding physical features. Similarly, although POI data can provide more detailed geographic entity information in scenarios with high requirements for the diversity of urban activities, in some scenarios where building structures and microscopic features need to be considered, it is necessary to combine other data sources to improve accuracy. Remote sensing images are the bridge between these two types of data. Therefore, inspired by non-local attention, by giving these data corresponding weights, different data sources are gradually fused to ensure that each fusion step fully utilizes the information of each data, alleviates modal differences, and enables the model to better learn the multimodal characteristics of the data.

[0122] Step 4: For the feature map obtained by fusing multiple data sources, a dilated convolution module is introduced based on the traditional CBAM module. This enables the model to better handle geographical structures of different scales in functional areas, expand the receptive field, and easily understand the overall structure and pattern of land use. It also retains relatively fine local features, which is very important for functional area classification because urban functional areas usually contain rich geographical details, such as building outlines, road networks, etc. The channel attention, dilated convolution, and spatial attention modules are used in serial superposition to adaptively adjust the features and use jump connections to improve sensitivity to internal details of functional areas.

[0123] Step 5: Extract key feature information at different scales from the feature map obtained in step 4 based on the difference in scale space, enhance the representation ability, optimize the network's perception and distinction performance of functional areas, and improve robustness. By integrating the features obtained from different convolution kernels, an extended feature representation is obtained, and multi-scale features are weighted by assigning certain weights. The weighted features are fused with the original input features using residual connections to form the output results.

[0124] Step 6. According to the supervised deep network training strategy, the cross entropy loss, L2 norm and dice category imbalance loss are used to establish the loss model of the fused image segmentation result and the label map, and the Adam optimizer is used for optimization. During the training process, the segmentation result is continuously close to the label image. The Epoch is set to 100 to obtain a high-precision urban functional area classification result graphic.

[0125] also, Figure 5 , Figure 6 , Figure 7The following are the results of urban functional area recognition using different comparison methods in Haishu District, Zhenhai District and the junction of Jiangbei-Zhenhai-Yinzhou District in Ningbo. RSI represents remote sensing image, FT represents building footprint, POI represents point of interest, and Label represents label. As can be seen from the figure, POI and FT are point and surface vector data, covering specific areas, and have a good improvement in the recognition effect of some areas.

[0126] Figure 5 , Figure 6 , Figure 7 The last MSDFN in the above is the method proposed by this paper, while RF, ResNet, FCN, MFN, LANet, MDL-RS-CNNs, and UisNet are the seven comparative methods in this paper. It can be seen from the recognition results of these methods that they all have certain defects. First, for any study area, the methods based on residual blocks or decision trees (ResNet and RF) have the worst results due to their simple and shallow feature extraction capabilities. The same is true for FCN, but satisfactory classification results have been achieved in slightly more complex study scenarios (the junction of Haishu District and Jiangbei-Zhenhai-Yinzhou District). Attention-based context exploration (LANet) achieves better performance by adaptively allocating weights of feature maps in spatial and channel dimensions. However, this method has certain limitations in its ability to model global relationships in multimodal data. To solve this problem, MDL-RS-CNNs proposed a cross-fusion module that is conducive to cross-modal transmission of information, showing competitive improvements. UisNet improves the model expression ability with the transformer module. Among all the comparison methods, MFN shows the best performance because it simultaneously exploits the relationship between attention context exploration and multimodal fusion, and adds auxiliary loss as supervision to improve intra-class consistency and inter-class distinguishability. The MSD3FN method proposed in this paper uses adaptive weights to achieve more stable and robust feature fusion, and uses attention, void convolution and multi-scale focusing joint modules to achieve reliable inter-semantic decoupling and intra-semantic interaction, which is better than all the comparison models. Figure 5 , Figure 6 , Figure 7 It can be seen that the recognition results obtained by the MSD3FN method have the least hollow areas, the clearest regional boundaries, the most complete details, and have better recognition effects.

[0127] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0128] Embodiment 3:

[0129] Based on Example 1, Example 3 of the present application provides an urban functional area identification system that integrates remote sensing and social perception data, including:

[0130] An acquisition module is used to acquire multi-source heterogeneous data, pre-process the multi-source heterogeneous data, and construct a data set;

[0131] A partitioning module, used to divide the data set into batches of different sizes and perform training in batches;

[0132] A fusion module, used for adaptively and interactively fusing the multi-source heterogeneous data;

[0133] The weight assignment module is used to introduce dilated convolution and attention mechanism to the feature map obtained through adaptive interactive fusion, and assign weights from two dimensions: space and channel.

[0134] Extraction module, used to extract key feature information at different scales in the image;

[0135] The training module is used for supervised deep network training. It uses cross entropy loss, L2 norm and dice category imbalance loss to establish a loss model for fusion image segmentation results and label images. During the training process, the segmentation results are continuously approached to the label images to obtain high-precision urban functional area classification result graphics.

[0136] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiment 1. Therefore, the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be repeated in this application.

Claims

1. A method for identifying urban functional areas by integrating remote sensing and social perception data, characterized in that: include: Step 1: Acquire multi-source heterogeneous data, pre-process the multi-source heterogeneous data, and construct a data set; Step 2: Divide the data set into batches of different sizes and perform training in batches; Step 3: adaptively and interactively fuse the multi-source heterogeneous data; Step 4: For the feature map obtained through adaptive interactive fusion, introduce the dilated convolution and attention mechanism to assign weights from the two dimensions of space and channel; Step 5: Extract key feature information at different scales in the image; Step 6: Conduct supervised deep network training and use cross entropy loss, L2 norm and dice category imbalance loss to establish a loss model for fusion image segmentation results and label images; during the training process, the segmentation results are continuously approached to the label images to obtain high-precision urban functional area classification result graphics.

2. The urban functional area identification method of remote sensing and social perception data fusion according to claim 1 is characterized in that: In step 1, the multi-source heterogeneous data includes remote sensing images, building footprints and points of interest; the multi-source heterogeneous data is preprocessed, including: performing radiation calibration, atmospheric correction, orthorectification, and geometric correction operations on the remote sensing images, and fusing the panchromatic and multispectral images to obtain multispectral images with high spatial resolution; using the spatial connection tool to connect the connected plots of building footprints and extract elevation information; screening the points of interest and classifying them after screening; the building footprints and points of interest need to be rasterized into graphics.

3. The urban functional area identification method of remote sensing and social perception data fusion according to claim 2 is characterized in that: In step 2, the data set is divided into a training set and a test set in proportion according to the number of pixels.

4. The urban functional area identification method of remote sensing and social perception data fusion according to claim 3 is characterized in that: Step 3 includes: Step 3.1: Use an encoder to extract the features of the multi-source heterogeneous data, expressed as: Conv_EN=σ(Maxpool〔Conv3〔σ(Conv3(σ(Maxpool(Conv3(σ(Conv3(input)))))))〕〕〕 Among them, input represents the input image features, σ(·) represents the Relu activation function, Maxpool represents the maximum pooling layer, and Conv_EN represents the output result of each encoder; Step 3.2: The features of the multi-source heterogeneous data are fused through an adaptive weight interaction fusion module, which is expressed as: F F ,F R ,F P =MLP(M F ,M R ,M P ) M RF =concat(F R ,F F ) M RP =concat(F R ,F P ) M k =concat(M i ,M j ) Among them, M F ,M R ,M P They represent the building footprint feature map, remote sensing image feature map and POI feature map respectively, and F R ,F F ,F P It represents the output result of each input feature map after passing through the multi-layer perceptron MLP. concat represents the superposition of feature maps in the channel dimension. RF 、M RP 、M i and M j represents the intermediate fusion feature, is the Hadamard product, w and w' represent the weight coefficient matrix, where w'=1-w∈[0,1], M k It is the final output fusion feature result.

5. The urban functional area identification method of remote sensing and social perception data fusion according to claim 3 is characterized in that: Step 4 includes: Step 4.1: For the input features, the channel attention module is used to adaptively capture the features of different channels, assign corresponding weights to each channel space, and highlight the channel information that is meaningful to the urban functional area classification task, and obtain the output feature map M. CH , expressed as: M CH =CAB(M k ) Among them, CAB represents the channel attention module; Step 4.2, pass the feature map into the hole convolution layer; Step 4.3: Pass the feature map through the spatial attention module to adaptively emphasize the features of different positions in space, expressed as: M SP =SAB(M DI ) Among them, M DI is the output feature map of the dilated convolution, and SAB is the spatial attention module.

6. The urban functional area identification method of remote sensing and social perception data fusion according to claim 5 is characterized in that: In step 4.2, first, the convolution layer is used for downsampling to reduce the spatial dimension of the feature map and increase the number of channels. Then, three convolution layers with different dilation rates are used, each of which contains two sublayers. The first sublayer is a depth-separable convolution layer, which uses a 3x3 convolution kernel with dilation rates of 1, 2, and 3 to implement dilation convolution operations on input features. The second sublayer is a 1x1 convolution layer, which is used to fuse channel information. Each depth-separable volume decomposes the standard volume into depth-wise convolution and point-wise convolution; it is expressed as: M DC =Conv1(DepthConv3(Down_3(M CH ))) M DI =concat(M CH ,Up_3(concat(M DC ,M DC ,M DC ))) Among them, DepthConv3 represents the depth convolution with a convolution kernel of 3x3, Down_3 and Up_3 represent three times down / up sampling respectively, and the three M DC Represent the deep convolution feature maps with void rates of 1, 2, and 3 respectively.

7. The urban functional area identification method of remote sensing and social perception data fusion according to claim 6 is characterized in that: In step 5, various convolutional layers, pooling layers, and fully connected layers are used to map the vectors to attention weights of the same dimension as the input features, and these weights are used to weight the multi-scale features; then, the weighted features are fused with the original input features through residual connections. Since feature maps from the same level have certain commonalities, feature block weights need to be assigned and spatially motivated; expressed as: M out1 =σ(Conv1(concat(σ(Conv1(M in )),σ(Conv3(M in )),σ(Conv5(M in ))))) M out3 =FC(σ(FC(GAP(concat(M in ,M out2 ))))) Among them, M in and M out3 Represent the input feature map and output feature map of the multi-scale focusing module, M out1 and M out2 represents the intermediate feature map, GAP represents the global average pooling, and FC represents the full connection; in M out3 The second fully connected layer in the calculation is followed by a sigmoid function and a Dropout regularization layer.

8. The urban functional area identification method of remote sensing and social perception data fusion according to claim 7 is characterized in that: In step 6, the overall loss of the network is: Among them, β is a constant, λ is the balance factor of the three losses; exist middle, y i,k and p i,k are the true value label and predicted probability value of the k-class of the i-th sample, and C is the total number of functional area categories; In the prediction result Y pr and the true label Y re are all probability distributions after the softmax operation, and the value at each position represents the model's predicted probability for the corresponding category; In the calculation of , these probability distributions are treated as binary masks; then, the elements of the binary masks are compared one by one, and the intersection is obtained by element-by-element multiplication and the union is obtained by element-by-element addition.

9. A system for identifying urban functional areas by integrating remote sensing and social perception data, characterized in that: Used to perform the method according to any one of claims 1 to 8, comprising: An acquisition module is used to acquire multi-source heterogeneous data, pre-process the multi-source heterogeneous data, and construct a data set; A partitioning module, used to divide the data set into batches of different sizes and perform training in batches; A fusion module, used for adaptively and interactively fusing the multi-source heterogeneous data; The weight assignment module is used to introduce dilated convolution and attention mechanism to the feature map obtained through adaptive interactive fusion, and assign weights from two dimensions: space and channel. Extraction module, used to extract key feature information at different scales in the image; The training module is used for supervised deep network training. It uses cross entropy loss, L2 norm and dice category imbalance loss to establish a loss model for fusion image segmentation results and label images. During the training process, the segmentation results are continuously approached to the label images to obtain high-precision urban functional area classification result graphics.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 8.

Citation Information

Patent Citations

  • Hyperspectral remote sensing image waveband selection method based on self-expression transfer learning

    CN115457311A

  • Urban scene understanding and function identification method and device based on geographic space data

    CN117830840A

Cited By

  • Mountain town space and traffic diversified characteristic data processing method

    CN119540768A

  • A data processing method for the spatial and traffic diversification characteristics of mountain towns

    CN119540768B

  • Ecological product development risk prediction method

    CN120297734A

  • Small-sample multi-dimensional city index prediction method and system based on semi-supervised learning, terminal and storage medium

    CN122087370A

  • A method, system, terminal, and storage medium for predicting small-sample, multi-dimensional urban indicators based on semi-supervised learning.

    CN122087370B