An urban land use mapping method based on multi-modal data collaborative perception

Abstract

The application belongs to the technical field of deep learning and remote sensing image processing, and specifically discloses a city land use mapping method based on multi-modal data collaborative perception, which comprises the following steps: performing parcel division on high-resolution remote sensing images of a research area to obtain irregular city parcels and remote sensing spectral features thereof; setting street sampling points along a road network, extracting street perception features and extracting interest point semantic features; constructing a heterogeneous graph structure based on the street sampling points, the street perception features and the interest point semantic features falling into the same parcel, and then extracting parcel-level multi-source geographic features; inputting the remote sensing spectral features and the parcel-level multi-source geographic features into a full sparse topic model for semantic alignment and fusion to generate fused parcel feature representations; and performing classification based on the fused parcel feature representations to output city land use mapping results. The application can realize high-precision and high-robustness city land use recognition under the condition that multi-source data is unevenly distributed or sparse.

Description

Technical Field

[0001] This application belongs to the field of deep learning and remote sensing image processing technology, and more specifically, relates to an urban land use mapping method based on multimodal data collaborative perception. Background Technology

[0002] Urban land use information is a crucial foundation for urban planning, resource allocation, public service layout, and refined governance. With the widespread acquisition of high-resolution remote sensing imagery, urban land use mapping using remote sensing imagery offers advantages such as wide coverage and high update efficiency. However, urban surface morphology is complex and land cover types are diverse. The same land use type may exhibit similar spectral and textural characteristics in remote sensing imagery, and different land use types may also have a high degree of confusion due to factors such as building materials, vegetation cover, and road morphology. Furthermore, urban land use inherently possesses functional semantic attributes; relying solely on the physical morphological information of remote sensing imagery often fails to accurately reflect human activities and socio-economic functions.

[0003] In recent years, multi-source geospatial big data such as street view imagery and points of interest have provided important supplements to the semantic representation of urban functions. Related methods have attempted to fuse remote sensing imagery with multi-source data to improve land use identification capabilities. However, existing fusion methods mostly employ simple stitching or rigid statistical summarization, which is difficult to effectively handle the significant differences in spatial scale, distribution, and representation space of multi-source data, and is prone to introducing information redundancy and semantic confusion. At the same time, there are complex spatial topological relationships and semantic associations among multi-source elements (such as spatial adjacency between different street view sampling points and semantic associations between similar environmental units). Without modeling of structural relationships, it is difficult to fully explore the complementary advantages of collaborative information. In addition, multi-source data suffers from imbalance and sparsity, with insufficient data coverage or sparse distribution in some areas, further leading to a decrease in model generalization.

[0004] Therefore, how to achieve high-precision and robust urban land use identification under conditions of uneven or sparse distribution of multi-source data is an urgent problem to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide an urban land use mapping method based on multimodal data collaborative perception, which can achieve high-precision and robust urban land use identification under conditions of uneven or sparse distribution of multi-source data.

[0006] To achieve the above objectives, in a first aspect, this application provides an urban land use mapping method based on multimodal data collaborative perception, comprising the following steps: S10, acquiring high-resolution remote sensing images of the study area, dividing the remote sensing images into plots based on a road network to obtain irregular urban plots and their corresponding land use labels; S20, extracting the remote sensing spectral features of the irregular urban plots; S30, setting street view sampling points along the road network, acquiring multi-directional street view images of the street view sampling points and performing semantic segmentation, extracting street view perception features; acquiring the surrounding features of the street view sampling points. Interest point data within a spatial buffer zone are used to extract semantic features of interest points; a heterogeneous graph structure is constructed based on street view sampling points falling within the same plot, the street view perception features, and the semantic features of the interest points; a graph attention network is used to process the spatial topological relationships and semantic associations of the heterogeneous graph structure to extract plot-level multi-source geographic features; S40, the remote sensing spectral features and the plot-level multi-source geographic features are input into a fully sparse topic model for semantic alignment and fusion to generate a fused plot feature representation; S50, classification is performed based on the fused plot feature representation to output urban land use mapping results.

[0007] As a further preferred option, step S10 specifically includes: Obtain the original road network data of the study area, perform topology checks and repairs on the original road network data, and obtain a topology-consistent road network; The target road network is obtained by filtering the topologically consistent road networks according to road grade. Acquire data on regions of interest in the study area, and extract category information representing land use types from the data on regions of interest as annotation samples; Using the target road network as the dividing boundary, the high-resolution remote sensing image is segmented into plots to generate irregular urban plots. The labeled samples are then spatially associated with the irregular urban plots to obtain the land use labels.

[0008] As a further preferred embodiment, before step S20, the irregular urban plots are further subjected to spatial transformation enhancement, specifically: calculating the pixel size of each plot image, cropping the small plots with pixel sizes smaller than a preset threshold based on random seed points to obtain local images; and performing Gaussian blur and random rotation processing on the local images and normal plots with pixel sizes not smaller than the preset threshold to obtain enhanced urban plots. The remote sensing spectral features extracted in step S20 are obtained by performing feature learning on the enhanced urban land parcel input convolutional neural network and then performing global average pooling on the network output features to obtain the land parcel-level spectral feature vector.

[0009] As a further preferred step, step S30 extracts street view perception features, specifically including: A set of street view sampling points is generated along the center line of the road at preset intervals, and street view images of each street view sampling point are collected in the four directions of east, south, west and north. Semantic segmentation is performed on each street view image, the pixel proportion of each semantic object category is counted, and the pixel proportions of each category in all directions are concatenated in a fixed direction order to obtain the sampling point-level street view semantic expression vector. Kernel density estimation is performed on the street scene semantic object to obtain the street scene kernel density features; the street scene perception features include the street scene semantic expression vector and the street scene kernel density features.

[0010] As a further preferred option, step S30 extracts semantic features of interest points, specifically including: Obtain raw point-of-interest data within the study area, and perform cleaning and reclassification processing; A spatial buffer is constructed with the street view sampling point as the center. The number of interest points of each interest point category falling into the buffer is counted to obtain the interest point count vector. Kernel density estimation is performed on the interest points to obtain the kernel density features of the interest points; the semantic features of the interest points include the number vector of interest points and the kernel density features of the interest points.

[0011] As a further preferred embodiment, the construction of the heterogeneous graph structure in step S30 specifically includes: For each urban plot, street view sampling points within its range are collected as nodes. The street view perception features of the corresponding nodes are spliced ​​with the semantic features of points of interest to form node features. Establish connections between any two different nodes within the same plot to form a complete graph adjacency matrix; For each edge, calculate the spatial distance and node feature difference between the two nodes it connects, and assign weights to the edges based on the spatial distance and node feature difference to obtain the edge weights; the formula for calculating the edge weights is as follows:

[0012] in, Nodes representing connections i and nodes j Edge weights; Nodes representing connections i and nodes j Spatial distance; Nodes representing connections i and nodes j Differences in node characteristics; and These are hyperparameters used to control the effects of geometric distance and semantic difference on edge weights, respectively.

[0013] As a further preferred option, step S30 extracts plot-level multi-source geographic features, specifically including: The initial feature vector of each node is input into the graph attention network, and a linear transformation is applied to the node features to map them to a unified latent space. For any pair of connected nodes within a plot, calculate the attention coefficient that simultaneously considers feature correlation and spatial topological prior edge weights, and then normalize it to obtain the attention weight. The features of neighboring nodes are weighted and summed based on attention weights, and nonlinearity, layer normalization and residual connection are introduced to update the nodes. A multi-head attention mechanism is used to learn multiple interaction modes in parallel, and the updated node representations are aggregated to obtain multi-source geographic features at the parcel level.

[0014] As a further preferred embodiment, in step S40, the remote sensing spectral features and the plot-level multi-source geographic features are input into a fully sparse topic model for semantic alignment and fusion, specifically including: The remote sensing spectral features and the plot-level multi-source geographic features are respectively mapped into feature representations in bag-of-words form; By simultaneously inputting bag-of-words feature representations of the two modalities into a fully sparse topic model with shared parameters, the topic distributions of each modality are inferred. The topic distributions corresponding to remote sensing modalities and multi-source geographic modalities are mapped to sparse semantic representations, respectively. The two types of sparse semantic representations are fused to generate a unified fused land parcel feature representation.

[0015] In a second aspect, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in any one of the above descriptions.

[0016] Thirdly, this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any one of the above statements.

[0017] Compared with existing technologies, the technical solution provided in this application has the following beneficial effects: This application proposes a multimodal data land use mapping method, which improves the stability and robustness of spectral feature extraction for small-scale and complex-morphological plots by constructing a multi-level spatial transformation enhancement module; it constructs a spatial buffer centered on street view sampling points to fuse street view perception features and point of interest semantic features, and forms a multimodal graph structure within the plot; it uses a graph attention network to simultaneously mine the spatial topological relationships and semantic associations within the plot, resulting in a more discriminative plot-level multi-source geographic representation; finally, it uses a fully sparse topic model to semantically align and fuse remote sensing features and multi-source geographic features, highlighting complementary key sparse semantics and suppressing redundant information and semantic confusion, thereby achieving more accurate, consistent, and well-generalized refined identification of urban land use even under conditions of sparse or unevenly distributed multi-source data. Attached Figure Description

[0018] Figure 1 A flowchart of the urban land use mapping method provided for this application; Figure 2 An overall framework diagram of the urban land use mapping method provided for this application; Figure 3 A flowchart of the fully sparse topic model provided for this application; Figure 4 A comparative map of urban land use mapping results in Shanghai and Wuhan, provided for this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] This application provides a method for urban land use mapping based on multimodal data collaborative perception, including the following steps.

[0021] S1. Acquire high-resolution remote sensing images of the study area and divide the land parcels based on the road network to obtain irregular urban land parcels and labels; S2. Perform multi-level spatial transformation enhancement on irregular plots, including rotation transformation, fuzzy perturbation and random pruning based on seed points; S3. Divide the enhanced plots obtained in S2 into training / validation / test subsets, and input them into a convolutional neural network for remote sensing feature extraction to obtain spectral features. ; S4. Set up street view sampling points along the road network and acquire street view images from multiple directions. Perform semantic segmentation on the street view images to extract street view category features, which together with kernel density statistical information constitute street view perception features. S5. Construct a spatial buffer centered on street view sampling points, and statistically analyze the number of interest points of different categories and kernel density within the buffer to form semantic features of interest points; S6. Construct a heterogeneous graph structure with street view sampling points as nodes within each plot. The node features include the street view perception features of S4 and the interest point semantic features of S5. S7. Utilize graph attention networks to mine spatial topological relationships and semantic associations in heterogeneous graph structures, and extract multi-source geographic features at the parcel level. ; S8. Input remote sensing spectral features and multi-source geographic features into the fully sparse topic model for semantic alignment and fusion, extract complementary key sparse semantics and reduce semantic confusion, and generate fused land parcel feature representations. S9. Input the land parcel feature representation fused from S8 into the machine learning model for classification, and output the urban land use mapping results.

[0022] This application provides a novel multimodal urban land use mapping framework. It not only effectively alleviates the problem of inconsistent scale and shape of irregular plots through multi-level spatial transformation enhancement, but also couples street view perception information and point of interest semantic information by constructing a spatial buffer, and fully explores the complementary information of physical form and functional semantics by introducing a graph attention model. Furthermore, it achieves cross-modal semantic alignment and fusion through a fully sparse topic model, and maintains high recognition accuracy and robustness even under the condition of uneven or sparse distribution of multi-source data.

[0023] Furthermore, step S1 specifically involves the following steps: S11. Download 4m high-resolution remote sensing images of the study area from Google Earth, and stitch and crop the images to the study area. S12. Obtain the road network vector data of the study area from OpenStreetMap (OSM), and complete the range clipping to obtain the original road network data. Perform topology checks and repairs on the road network, including connecting broken roads, clearing hanging lines, correcting intersection nodes, and processing duplicate line segments, to obtain a topologically consistent road network to avoid generating incorrect plots during segmentation. S13. Based on the urban land parcel division requirements, the road network is screened according to road level, retaining key roads such as main roads and secondary roads, and eliminating low-level roads or noisy roads that do not affect the land parcel boundaries, so as to obtain the target road network for land parcel division and reduce excessive fragmentation of land parcels. S14. Obtain AOI (Area of ​​Interest) data of the study area from OSM, extract the category information in AOI that can be used to characterize land use type as label fields, and perform category mapping on AOI data to form a labeled sample dataset; S15. Using the target road network obtained in S13 as the segmentation boundary, the remote sensing image in S11 is segmented into plots to generate irregular urban plots. The AOI annotation in step S14 is spatially associated with the plots to generate a corresponding land use label for each plot, and finally, urban plots and their corresponding labels are obtained.

[0024] Furthermore, step S2 specifically involves the following steps: S21. Calculate the pixel size of each plot image after division and compare it with a preset threshold to obtain narrow plots and normal plots; S22. Randomly crop all the small plots in S21 to obtain local images. The cropping ratio is about 70% of the original image. During cropping, a seed point is randomly selected as the cropping center or starting position, and the local area is extracted with the seed point as a reference. S23. Apply Gaussian blur and random rotation to the normal plots in S21 and the local plots in S22 to obtain the enhanced urban plots.

[0025] Furthermore, step S3 specifically involves the following steps: S31. Perform uniform random cropping and resampling on the enhanced land parcels, divide the land parcel dataset into training / test subsets, and input them into a convolutional neural network for feature learning to obtain the deep feature representation of each sample. S32. Perform global average pooling on the network output features to obtain the plot-level spectral feature vector, which serves as the final remote sensing spectral representation of the plot. .

[0026] Furthermore, step S4 specifically involves the following steps: S41. Based on OSM road network data, generate a set of street view sampling points along the road centerline at 100m intervals. The geographic coordinates of each sampling point are obtained. ; S42. For each street view sampling point Based on its geographical coordinates Collect street view images of this point in the four cardinal directions (north, south, east, and west), and record them as follows: Obtain the same sampling point The corresponding set of four street view images ; S43. Take each street view image acquired in S42. Input the semantic segmentation network trained on the Cityscapes dataset to perform pixel-level semantic segmentation and obtain the street scene category system. Below is a pixel-level label map of each semantic object category in each street view image; S44. For each street view image in S43 The semantic segmentation results are used to calculate the pixel percentage of each semantic object category in the image. Specifically:

[0027] in, The label image is determined to be a category The number of pixels; S45, Each street view image The pixel percentages of each category are arranged in category order to form a directional semantic vector. The four directional semantic vectors are then arranged in a fixed directional order. The data is stitched together to maintain geometric consistency of the viewpoint, resulting in a street scene semantic representation vector at the sampling point level. .

[0028] S46. Perform kernel density estimation (KDE) on street scene semantic objects to characterize their spatial clustering and obtain street scene sampling points. Street view kernel density features .

[0029] Furthermore, step S5 specifically involves the following steps: S51. Obtain the original Points of Interest (POI) data within the study area from Amap (Gaode Maps), and perform cleaning and reclassification of the POI data to map the original POIs into a set of POI categories with clear socioeconomic semantics. ; S52, Street view sampling points obtained from S4 geographic coordinates Construct a spatial buffer zone around the center For each point of interest category The number of interest points falling into the buffer is counted to obtain the sampling points. Interest point quantity vector ; S53. Further perform KDE analysis on the points of interest to obtain street view sampling points. POI kernel density characteristics .

[0030] Furthermore, step S6 specifically involves the following steps: S61, For each urban plot Collect the set of street view sampling points falling within the area of ​​this plot, denoted as the node set. Each node Corresponding to a street view sampling point ; S62. For each node, concatenate the street view perception feature vector obtained in S4 with the interest point semantic feature vector obtained in S5 to form the node feature. ; S63, with land parcel Build a graph for units For any two different nodes within the same plot and Establish connections to form an adjacency matrix of the complete graph. Specifically:

[0031] in, Represents the elements of the adjacency matrix, indicating the nodes. and Whether connected via an edge; S64. Calculate the spatial distance between two nodes for each edge. And calculate the differences in node features. Edge weights are obtained by assigning weights to edges based on spatial distance and feature similarity terms. Specifically:

[0032] in and These are hyperparameters used to control the effects of geometric distance and semantic difference on edge weights, respectively. S65. Construct corresponding maps for all plots within the study area. , forming a graph set .

[0033] Furthermore, step S7 specifically involves the following steps: S71, the training / test subset remains consistent with S31, and for each city plot in the training set... The picture , each node Initial multimodal feature vector As input to the graph attention network, It is formed by splicing together street view perception features and POI semantic features; S72, in the In layered graph attention networks, a shared linear transformation is applied to node features to map multimodal features to a unified latent space, specifically:

[0034] in The weight matrix is ​​a learnable weight matrix; S73. For any pair of connected nodes within the same plot of land Calculate the unnormalized attention coefficient This coefficient considers both feature correlation and spatial topological prior edge weights, specifically:

[0035] in, For learnable attention vectors, The strength of the a priori influence on the control space. It is the numerical stability constant; S74, for nodes The attention coefficients within the neighborhood are softmax normalized to obtain the attention weights. Specifically:

[0036] S75. Based on attention weights, the features of neighboring nodes are weighted and summed, and nonlinearity, layer normalization, and residual connections are introduced to update the node representation:

[0037] S76. Employs a multi-head attention mechanism to learn multiple interaction modes in parallel, for the first... Each attention point executes S72–S75 respectively to obtain And then splice or merge them in the hidden layer, specifically:

[0038] in, For the number of attention heads; S77, after After the layer graph attention is updated, the plots are... The final representations of all nodes within the site are aggregated to obtain the parcel-level fused features. Specifically:

[0039] S78. Perform steps S71–S77 on all plots to obtain a set of multi-source geographic features at the plot level. .

[0040] Furthermore, step S8 specifically involves the following steps: S81, Remote sensing spectral features With multi-source geographical features The features are mapped to bag-of-words representations respectively, resulting in remote sensing word representations. Representation of multi-source geographical terms Using a unified feature input format; S82, In sharing model parameters Under the conditions, and Simultaneously input the fully sparse topic model and infer its topic distribution:

[0041] S83. Map the topic distributions corresponding to remote sensing modalities and multi-source geographic modalities into sparse semantic representations, respectively, to obtain:

[0042] in, Represents the feature mapping function; S84. The two types of sparse semantic representations obtained in S83 are fused to generate a unified parcel-level fused feature representation:

[0043] Further, step S8 specifically involves: inputting the land parcel feature representation fused in S8 into the machine learning model for training, and finally outputting the urban land use mapping result.

[0044] Compared with existing technologies, the technical solution provided in this application has the following beneficial effects: This application proposes a multimodal data land use mapping method, which improves the stability and robustness of spectral feature extraction for small-scale and complex-morphological plots by constructing a multi-level spatial transformation enhancement module; it constructs a spatial buffer centered on street view sampling points to fuse street view perception features and point of interest semantic features, and forms a multimodal graph structure within the plot; it uses a graph attention network to simultaneously mine the spatial topological relationships and semantic associations within the plot, resulting in a more discriminative plot-level multi-source geographic representation; finally, it uses a fully sparse topic model to semantically align and fuse remote sensing features and multi-source geographic features, highlighting complementary key sparse semantics and suppressing redundant information and semantic confusion, thereby achieving more accurate, consistent, and well-generalized refined identification of urban land use even under conditions of sparse or unevenly distributed multi-source data.

[0045] In one embodiment, the technical solution to achieve the above objective can be: a method for urban land use mapping based on multimodal data collaborative perception is disclosed.

[0046] Please refer to Figure 1-2 , Figure 1 A flowchart of the urban land use mapping method provided in this application. Figure 2 An overall framework diagram of the urban land use mapping method provided for this application; like Figure 1 The specific operational steps of the urban land use mapping method shown are as follows: S1. Acquire high-resolution remote sensing imagery of the study area and divide the land parcels based on the road network to obtain irregular urban land parcels and labels. First, 4m high-resolution remote sensing imagery was acquired from Google Earth. This imagery was then cropped and stitched together according to the administrative boundaries of Shenzhen in 2018 and Wuhan in 2022 to obtain remote sensing imagery of Shanghai and Wuhan. Next, road network and Area of ​​Interest (AOI) data corresponding to the study area were obtained from OpenStreetMap (OSM). The road network data underwent topological correction and was filtered according to road level, retaining road elements such as arterial roads and secondary arterial roads that constrain the boundaries of urban land parcels. This yielded the final land parcel vector data. The remote sensing imagery was then spatially segmented to obtain a set of irregular urban land parcels that conform to real urban planning units. Finally, the functional category information of AOI is mapped to urban land use labels, and the AOI labels are assigned to the corresponding irregular plots through spatial overlay and correlation analysis. This results in a dataset consisting of several irregular urban plots and their corresponding land use labels, which are then randomly divided into a training plot set and a test plot set at a 1:1 ratio for subsequent feature extraction, model training, and accuracy evaluation.

[0047] S2. Perform multi-level spatial transformation enhancement on irregular plots, including rotation transformation, fuzzy perturbation, and random pruning based on seed points. The specific steps are as follows: S21. Calculate the pixel size of each plot image after division and compare it with a preset threshold to obtain narrow plots and normal plots; S22. Randomly crop all the small plots in S21 to obtain local images. The cropping ratio is about 70% of the original image. During cropping, a seed point is randomly selected as the cropping center or starting position, and the local area is extracted with the seed point as a reference. S23. Apply Gaussian blur and random rotation to the normal plots in S21 and the local plots in S22 to obtain the enhanced urban plots. S3. Divide the enhanced plots obtained in S2 into training / test subsets and input them into a convolutional neural network for remote sensing feature extraction to obtain remote sensing spectral features. The specific steps are as follows: S31. Perform uniform random cropping and resampling on the enhanced land parcels, divide the land parcel dataset into training / test subsets, and input them into a convolutional neural network for feature learning to obtain the deep feature representation of each sample. S32. Perform global average pooling on the network output features to obtain the plot-level spectral feature vector, which serves as the final remote sensing spectral representation of the plot. .

[0048] S4. Set up street view sampling points along the road network and acquire street view images from multiple directions. Perform semantic segmentation on the street view images to extract street view category features, which, together with kernel density statistical information, constitute street view perception features. The specific steps are as follows: S41. Based on OSM road network data, generate a set of street view sampling points along the road centerline at 100m intervals. The geographic coordinates of each sampling point are obtained. ; S42. For each street view sampling point Based on its geographical coordinates Collect street view images of this point in the four cardinal directions (north, south, east, and west), and record them as follows: Obtain the same sampling point The corresponding set of four street view images ; S43. Take each street view image acquired in S42. Input the semantic segmentation network trained on the Cityscapes dataset to perform pixel-level semantic segmentation and obtain the street scene category system. Below is a pixel-level label map of each semantic object category in each street view image; S44. For each street view image in S43 The semantic segmentation results are used to calculate the pixel percentage of each semantic object category in the image. Specifically:

[0049] in, The label image is determined to be a category The number of pixels; S45, Each street view image The pixel percentages of each category are arranged in category order to form a directional semantic vector. The four directional semantic vectors are then arranged in a fixed directional order. The data is stitched together to maintain geometric consistency of the viewpoint, resulting in a street scene semantic representation vector at the sampling point level. .

[0050] S46. Perform kernel density estimation (KDE) on street scene semantic objects to characterize their spatial clustering and obtain street scene sampling points. Street view kernel density features .

[0051] S5. Construct a spatial buffer centered on street view sampling points, and statistically analyze the number of interest points categories and kernel density within the buffer to form semantic features of interest points. The specific steps are as follows: S51. Obtain the original Points of Interest (POI) data within the study area from Amap (Gaode Maps), and perform cleaning and reclassification of the POI data to map the original POIs into a set of POI categories with clear socioeconomic semantics. ; S52, Street view sampling points obtained from S4 geographic coordinates Construct a spatial buffer zone around the center For each point of interest category The number of interest points falling into the buffer is counted to obtain the sampling points. Interest point quantity vector ; S53. Further perform KDE analysis on the points of interest to obtain street view sampling points. POI kernel density characteristics .

[0052] S6. Construct a heterogeneous graph structure with street view sampling points as nodes within each plot. The node features include the street view perception features from S4 and the interest point semantic features from S5. The specific steps are as follows: S61, For each urban plot Collect the set of street view sampling points falling within the area of ​​this plot, denoted as the node set. Each node Corresponding to a street view sampling point ; S62. For each node, concatenate the street view perception feature vector obtained in S4 with the interest point semantic feature vector obtained in S5 to form the node feature. ; S63, with land parcel Build a graph for units For any two different nodes within the same plot and Establish connections to form an adjacency matrix of the complete graph. Specifically:

[0053] in, Represents the elements of the adjacency matrix, indicating the nodes. and Whether connected via an edge; S64. Calculate the spatial distance between two nodes for each edge. And calculate the differences in node features. Edge weights are obtained by assigning weights to edges based on spatial distance and feature similarity terms. Specifically:

[0054] in and These are hyperparameters used to control the effects of geometric distance and semantic difference on edge weights, respectively. S65. Construct corresponding maps for all plots within the study area. , forming a graph set .

[0055] S7. Utilize graph attention networks to mine spatial topological relationships and semantic associations in heterogeneous graph structures, and extract multi-source geographic features at the parcel level. The specific steps are as follows: S71, the training / test subset remains consistent with S31, and for each city plot in the training set... The picture , each node Initial multimodal feature vector As input to the graph attention network, It is formed by splicing together street view perception features and POI semantic features; S72, in the In layered graph attention networks, a shared linear transformation is applied to node features to map multimodal features to a unified latent space, specifically:

[0056] in The weight matrix is ​​a learnable weight matrix; S73. For any pair of connected nodes within the same plot of land Calculate the unnormalized attention coefficient This coefficient considers both feature correlation and spatial topological prior edge weights, specifically:

[0057] in, For learnable attention vectors, The strength of the a priori influence on the control space. It is the numerical stability constant; S74, for nodes The attention coefficients within the neighborhood are softmax normalized to obtain the attention weights. Specifically:

[0058] S75. Based on attention weights, the features of neighboring nodes are weighted and summed, and nonlinearity, layer normalization, and residual connections are introduced to update the node representation:

[0059] S76. Employs a multi-head attention mechanism to learn multiple interaction modes in parallel, for the first... Each attention point executes S72–S75 respectively to obtain And then splice or merge them in the hidden layer, specifically:

[0060] in, For the number of attention heads; S77, after After the layer graph attention is updated, the plots are... The final representations of all nodes within the site are aggregated to obtain the parcel-level fused features. Specifically:

[0061] S78. Perform steps S71–S77 on all plots to obtain a set of multi-source geographic features at the plot level. .

[0062] S8. Input remote sensing spectral features and multi-source geographic features into a fully sparse topic model for semantic alignment and fusion, such as... Figure 3 As shown, complementary key sparse semantics are extracted and semantic confusion is reduced to generate a fused land parcel feature representation. The specific steps are as follows: S81, Remote sensing spectral features With multi-source geographical features The features are mapped to bag-of-words representations respectively, resulting in remote sensing word representations. Representation of multi-source geographical terms Using a unified feature input format; S82, In sharing model parameters Under the conditions, and Simultaneously input the fully sparse topic model and infer its topic distribution:

[0063] S83. Map the topic distributions corresponding to remote sensing modalities and multi-source geographic modalities into sparse semantic representations, respectively, to obtain:

[0064] in, Represents the feature mapping function; S84. The two types of sparse semantic representations obtained in S83 are fused to generate a unified parcel-level fused feature representation:

[0065] S9. Input the land parcel feature representation fused from S8 into the machine learning model for classification, and output the urban land use mapping results.

[0066] To verify the urban land use mapping method based on multimodal data collaborative perception proposed in this application, this example also includes the analysis and evaluation of the classification results. Figure 4 The exhibition showcased urban land use mapping for different products in Shanghai and Wuhan. Figure 4 (a) and (e) are partial high-resolution remote sensing images of Shenzhen and Wuhan, respectively. Figure 4 (b) and (f) are actual land use category labels used for reference. Figure 4 (c) and (g) are the classification results of EULUUC2.0 products. Figure 4 The middle (h) is the classification result of GUN products. Figure 4 (d) and (i) are the classification diagrams of the method proposed in this experiment. The comparison shows that the results of the method proposed in this application are closest to the actual land use category labels, effectively reducing misclassification of administrative areas, commercial areas, residential areas, etc., while also detecting a relatively complete manufacturing sector. Table 1 shows a comparison of the accuracy evaluation results of different methods. Overall, it can be seen that the method proposed in this experiment has the best classification accuracy, thus demonstrating the effectiveness of the method proposed in this application.

[0067] Table 1 compares the accuracy evaluation results of each classification method.

[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for urban land use mapping based on multimodal data collaborative perception, characterized in that, The process includes the following steps: S10, acquiring high-resolution remote sensing images of the study area, dividing the remote sensing images into plots based on the road network to obtain irregular urban plots and their corresponding land use labels; S20, extracting the remote sensing spectral features of the irregular urban plots; S30, setting street view sampling points along the road network, acquiring multi-directional street view images of the street view sampling points and performing semantic segmentation to extract street view perception features. Acquire point of interest (POI) data within the spatial buffer zone surrounding the street view sampling points and extract semantic features of POIs; construct a heterogeneous graph structure based on street view sampling points falling within the same plot, the street view perception features, and the semantic features of POIs; use a graph attention network to process the spatial topological relationships and semantic associations of the heterogeneous graph structure and extract plot-level multi-source geographic features. S40, input the remote sensing spectral features and the plot-level multi-source geographic features into the fully sparse topic model for semantic alignment and fusion to generate a fused plot feature representation; S50, classify based on the fused plot feature representation and output the urban land use mapping results.

2. The urban land use mapping method based on multimodal data collaborative perception as described in claim 1, characterized in that, Step S10 specifically includes: Obtain the original road network data of the study area, perform topology checks and repairs on the original road network data, and obtain a topology-consistent road network; The target road network is obtained by filtering the topologically consistent road networks according to road grade. Acquire data on regions of interest in the study area, and extract category information representing land use types from the data on regions of interest as annotation samples; Using the target road network as the dividing boundary, the high-resolution remote sensing image is segmented into plots to generate irregular urban plots. The labeled samples are then spatially associated with the irregular urban plots to obtain the land use labels.

3. The urban land use mapping method based on multimodal data collaborative perception as described in claim 1, characterized in that, Before step S20, the method further includes spatial transformation enhancement of the irregular urban plots, specifically: calculating the pixel size of each plot image, cropping the small plots with pixel sizes smaller than a preset threshold based on random seed points to obtain local images; and performing Gaussian blur and random rotation processing on the local images and normal plots with pixel sizes not smaller than the preset threshold to obtain enhanced urban plots. The remote sensing spectral features extracted in step S20 are obtained by performing feature learning on the enhanced urban land parcel input convolutional neural network and then performing global average pooling on the network output features to obtain the land parcel-level spectral feature vector.

4. The urban land use mapping method based on multimodal data collaborative perception as described in claim 1, characterized in that, Step S30 extracts street view perception features, specifically including: A set of street view sampling points is generated along the center line of the road at preset intervals, and street view images of each street view sampling point are collected in the four directions of east, south, west and north. Semantic segmentation is performed on each street view image, the pixel proportion of each semantic object category is counted, and the pixel proportions of each category in all directions are concatenated in a fixed direction order to obtain the sampling point-level street view semantic expression vector. Kernel density estimation is performed on the street scene semantic object to obtain the street scene kernel density features; the street scene perception features include the street scene semantic expression vector and the street scene kernel density features.

5. The urban land use mapping method based on multimodal data collaborative perception as described in claim 1, characterized in that, Step S30 extracts semantic features of interest points, specifically including: Obtain raw point-of-interest data within the study area, and perform cleaning and reclassification processing; A spatial buffer is constructed with the street view sampling point as the center. The number of interest points of each interest point category falling into the buffer is counted to obtain the interest point count vector. Kernel density estimation is performed on the interest points to obtain the kernel density features of the interest points; the semantic features of the interest points include the number vector of interest points and the kernel density features of the interest points.

6. The urban land use mapping method based on multimodal data collaborative perception as described in claim 1, characterized in that, Step S30 involves constructing a heterogeneous graph structure, specifically including: For each urban plot, street view sampling points within its range are collected as nodes. The street view perception features of the corresponding nodes are spliced ​​with the semantic features of points of interest to form node features. Establish connections between any two different nodes within the same plot to form a complete graph adjacency matrix; For each edge, calculate the spatial distance and node feature difference between the two nodes it connects, and assign weights to the edges based on the spatial distance and node feature difference to obtain the edge weights; the formula for calculating the edge weights is as follows: in, Nodes representing connections i and nodes j Edge weights; Nodes representing connections i and nodes j Spatial distance; Nodes representing connections i and nodes j Differences in node characteristics; and These are hyperparameters used to control the effects of geometric distance and semantic difference on edge weights, respectively.

7. The urban land use mapping method based on multimodal data collaborative perception as described in claim 6, characterized in that, Step S30 extracts plot-level multi-source geographic features, specifically including: The initial feature vector of each node is input into the graph attention network, and a linear transformation is applied to the node features to map them to a unified latent space. For any pair of connected nodes within a plot, calculate the attention coefficient that simultaneously considers feature correlation and spatial topological prior edge weights, and then normalize it to obtain the attention weight. The features of neighboring nodes are weighted and summed based on attention weights, and nonlinearity, layer normalization and residual connection are introduced to update the nodes. A multi-head attention mechanism is used to learn multiple interaction modes in parallel, and the updated node representations are aggregated to obtain multi-source geographic features at the parcel level.

8. The urban land use mapping method based on multimodal data collaborative perception as described in claim 6, characterized in that, In step S40, the remote sensing spectral features and the plot-level multi-source geographic features are input into the fully sparse topic model for semantic alignment and fusion, specifically including: The remote sensing spectral features and the plot-level multi-source geographic features are respectively mapped into feature representations in bag-of-words form; By simultaneously inputting bag-of-words feature representations of the two modalities into a fully sparse topic model with shared parameters, the topic distributions of each modality are inferred. The topic distributions corresponding to remote sensing modalities and multi-source geographic modalities are mapped to sparse semantic representations, respectively. The two types of sparse semantic representations are fused to generate a unified fused land parcel feature representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the steps of the method as described in any one of claims 1 to 8.

Images