Urban area representation method and system supported by multi-modal knowledge graph

Abstract

The present application relates to the field of geographic information science, and discloses a city area representation method and system supported by a multi-modal knowledge graph, which comprises: dividing a target research area into grid areas of a preset size according to a pre-constructed graph ontology, and constructing a multi-modal geographic knowledge graph; aligning remote sensing image features with street view image features, and then performing multi-scale adaptive fusion to obtain fused multi-scale features; after obtaining static embedding and dynamic embedding based on the multi-modal geographic knowledge graph, using a cross-attention mechanism to perform two-stage information fusion on entity embedding and relation embedding, embedding spatial position information, spatial relation information and multi-modal data information in the multi-modal geographic knowledge graph into a vector space to obtain a city area embedding vector. The present application effectively introduces spatial relation prior knowledge in combination with a multi-modal geographic knowledge graph, improves the quality of city area vector representation, and supports downstream city management and decision analysis tasks.

Description

Technical Field

[0001] This invention relates to the field of geographic information science and discloses a method and system for representing urban areas supported by multimodal knowledge graphs. Background Technology

[0002] Urban areas are the basic units of urban spatial structure, serving as core spatial carriers for human activities, ecological processes, resource allocation, and transportation networks, and forming complex interactive relationships with various elements of the urban system. Urban area representation can transform complex urban spatial areas into low-dimensional embedded vectors, applicable to various downstream tasks, thereby analyzing the functional attributes, spatial relationships, and evolutionary patterns of the areas.

[0003] With the advent of the geospatial big data era, the widespread acquisition and application of multimodal data such as remote sensing image data, street view image data, point of interest data, and road network data have provided a rich data foundation for urban area characterization research and significantly expanded the ability to depict the multidimensional characteristics of urban areas.

[0004] Currently, urban area representation learning methods mainly fall into two categories:

[0005] The first category is word embedding-based methods, including Word2Vec, Place2Vec, and Doc2Vec, which learn the semantic representation of regions by obtaining the semantic representation and spatial co-occurrence patterns of POIs.

[0006] The second category is based on graph neural networks, including HGI, HAFusion, and CaLLiPer, which integrate multi-source data by constructing a regional relationship network and learning regional embedding vectors.

[0007] Furthermore, multimodal knowledge graphs (MMKG) and geographic knowledge graphs (such as GeoKG, SKG, WorldKG, GIS-KG, GAKG, RSKG, and OIFKG) have made initial explorations in integrating geographic data and constructing spatial relationships. Regarding the fusion methods of RSI and SVI, existing technologies have attempted to achieve interaction between the two types of images through attention mechanisms, but most remain at an early stage of feature stitching fusion. In terms of knowledge graph representation learning models, there are translation models (such as TransE, TransH, and RotatE), semantic matching models (such as RESCAL, DistMult, and SimpIE), neural network models (such as ConvE, InteractE, and ConvHLE), and dedicated models for geographic knowledge graphs (such as SE-KGE, SSLP, and SR-KGE).

[0008] However, existing technologies still have the following technical shortcomings and deficiencies when dealing with urban area representation:

[0009] First, existing methods have significant limitations in multimodal data integration and spatial relationship representation, making it difficult to effectively capture spatial dependencies and multi-level semantic information between regions, thus restricting the comprehensive mining capabilities of multidimensional features and spatial association patterns in urban areas.

[0010] Second, existing multimodal geographic knowledge graphs are mainly limited to structured triples, making it difficult to accurately represent the complex realities of cities. Some knowledge graphs that combine multimodal data still suffer from insufficient granularity in geospatial semantic modeling, incomplete characterization of spatial dependencies between entities, and inadequate integration of multi-source data, making them unsuitable for a comprehensive representation of complex urban areas.

[0011] Third, among existing fusion methods, early fusion methods cannot effectively integrate two types of image features with large semantic information spans and obvious scale differences; deep fusion methods based on attention mechanisms lack the ability to process multi-scale semantic information in a fine-grained manner, making it difficult to capture local details and global semantics at the same time, and the fusion process is complex and lengthy, increasing computational overhead and easily leading to the loss of key information in multi-level processing, making it difficult to reduce information loss while achieving scale adaptation.

[0012] Fourth, existing representation learning methods for geographic knowledge graphs are mainly designed for structured geographic knowledge graphs and lack the ability to deeply integrate multi-source heterogeneous data. They are difficult to effectively capture the multi-dimensional spatial relationships and semantic dependencies between urban areas, which limits the ability to comprehensively and accurately represent the complex characteristics and internal structure of urban areas. Summary of the Invention

[0013] The technical problem to be solved by this invention is to overcome the shortcomings of existing urban area representation methods in multi-source heterogeneous data integration, multimodal visual semantic fusion, and geographic knowledge graph representation learning. It also addresses the problems of insufficient granularity in the construction of existing multimodal geographic knowledge graphs, lack of deep multi-scale fusion of remote sensing images and street view images, and lack of multimodal data deep fusion capability in geographic knowledge graph representation learning. The invention proposes an urban area representation method and system supported by multimodal knowledge graphs.

[0014] Therefore, the specific technical solution adopted by the present invention is as follows:

[0015] In a first aspect, this invention proposes a method for representing urban regions supported by multimodal knowledge graphs, comprising the following steps:

[0016] S1. Based on the pre-constructed geographic ontology, the target research area is divided into grid regions of a preset size. The multimodal geographic entities corresponding to the grid regions and the spatial relationships, subordinate relationships and similarity relationships between entities are determined, and a multimodal geographic knowledge graph is constructed.

[0017] S2. Based on the multimodal geographic knowledge graph, street view image data is subjected to multi-view adaptive fusion to obtain street view image features. Remote sensing image features are extracted from remote sensing image data. After aligning the remote sensing image features with the street view image features, multi-scale adaptive fusion is performed to obtain the fused multi-scale features.

[0018] S3. After obtaining static and dynamic embeddings based on the multimodal geographic knowledge graph, a two-stage information fusion of entity embedding and relation embedding is performed using a cross-attention mechanism to enhance tail entity embedding. Optimization is then performed under the guidance of a joint decision loss function, embedding spatial location information, spatial relation information, and multimodal data information from the multimodal geographic knowledge graph into a vector space to obtain urban area embedding vectors. Among these, dynamic embedding includes fused multi-scale features.

[0019] S4. Based on urban area embedding vectors, conduct land use classification, abnormal behavior prediction, housing price estimation, and urban village identification.

[0020] Furthermore, S3 includes:

[0021] S31. Based on a multimodal geographic knowledge graph, Space2Vec is used to encode geographic entity types to obtain location embeddings, and a pre-trained language model is used to obtain text embeddings for geographic entity types containing name attributes. The location embeddings and text embeddings are concatenated to obtain static embeddings.

[0022] S32. For entities in a structured multimodal geographic knowledge graph, a graph attention network is used to obtain dynamic embeddings from the multimodal geographic knowledge graph; for multimodal entities in a multimodal geographic knowledge graph, the fused multi-scale features are used as dynamic embeddings.

[0023] S33. Dimensionality reduction is performed on static and dynamic embeddings using fully connected layers. The dimensionality-reduced static and dynamic embeddings are concatenated to obtain the initial head entity embedding and the initial tail entity embedding. The pre-trained language model is then used to encode the relations in the knowledge graph to obtain the relation embedding.

[0024] S34. The initial head entity embedding and relation embedding are concatenated and deformed to obtain the first fused feature. The first fused feature is processed by convolution operation to obtain the query matrix, key matrix and value matrix. The attention weights of the query matrix and key matrix are calculated based on the cross attention mechanism. The attention weights and value matrix are weighted and aggregated and then superimposed on the original first fused feature to obtain the fused feature.

[0025] S35. The fused features are concatenated and deformed with the initial tail entity embedding to obtain the second fused features. The second fused features are processed by convolution operation to obtain the query matrix, key matrix and value matrix. The attention weights of the query matrix and key matrix are calculated based on the cross attention mechanism. The attention weights and value matrix are weighted and aggregated and then superimposed on the original second fused features to obtain the enhanced tail entity embedding.

[0026] S36. Guided by the joint decision loss function, optimization training is performed based on the enhanced tail entity embedding and the fused features. The spatial semantic information, spatial relationship information and multimodal data information of the multimodal geographic knowledge graph are embedded into the vector space to obtain the urban area embedding vector.

[0027] Furthermore, obtaining text embeddings for geographic entity types containing name attributes using a pre-trained language model includes:

[0028] The geographic entity type containing name attributes is cleaned and segmented to obtain the segmentation results;

[0029] The word segmentation results are fed into a pre-trained language model to obtain the latent vectors of each token;

[0030] Layer normalization is performed on the latent vectors of each token to obtain the text embedding.

[0031] Furthermore, S36 includes:

[0032] S361. Based on the hierarchical knowledge embedding approach, the semantic hierarchical structure of geographic entities is explicitly modeled in the embedding space, and the hierarchical structure perception loss is constructed using the edge ranking loss form.

[0033] S362. Based on the fused modal features in the enhanced tail entity embedding, a multimodal contrastive learning loss is constructed using the InfoNCE contrastive learning loss function;

[0034] S363. Optimize training under the guidance of the joint decision loss function, use hierarchical structure perception loss to accurately model the hierarchical structure relationship of geographic entities, and use multimodal contrastive learning loss to promote the visual semantic consistency alignment between different modal data. Embed spatial semantic information, spatial relationship information and multimodal data information in the multimodal geographic knowledge graph into the vector space to obtain the city area embedding vector.

[0035] Secondly, the present invention also provides a multimodal knowledge graph-supported urban area representation system, the system comprising:

[0036] The multimodal knowledge graph construction unit is used to divide the target research area into grid regions of a preset size based on the pre-constructed graph ontology, determine the multimodal geographic entities corresponding to the grid regions and the spatial relationships, subordinate relationships and similarity relationships between entities, and construct a multimodal geographic knowledge graph.

[0037] The multi-source data feature fusion unit is used to perform multi-view adaptive fusion of street view image data based on multimodal geographic knowledge graph to obtain street view image features, extract remote sensing image features from remote sensing image data, and perform multi-scale adaptive fusion after aligning the remote sensing image features with the street view image features to obtain fused multi-scale features.

[0038] The knowledge graph embedding learning unit is used to obtain static and dynamic embeddings based on multimodal geographic knowledge graphs. It then uses a cross-attention mechanism to perform two-stage information fusion of entity embeddings and relation embeddings to enhance tail entity embeddings. Under the guidance of a joint decision loss function, it optimizes the embedding of spatial location information, spatial relation information, and multimodal data information from the multimodal geographic knowledge graph into a vector space to obtain urban area embedding vectors. Among them, dynamic embedding includes fused multi-scale features.

[0039] Downstream task application units are used to carry out land use classification, abnormal behavior prediction, housing price estimation, and urban village identification based on urban area embedding vectors.

[0040] The beneficial effects of this invention are as follows:

[0041] (1) This invention designs a multi-level ontology architecture to perform refined semantic modeling of urban areas, and integrates multi-level spatial relationships such as spatial adjacency, subordinate association and functional similarity to construct a structured multimodal geographic knowledge graph, providing a comprehensive and accurate structured expression for urban areas.

[0042] (2) This invention utilizes a convolutional neural network to construct a multi-view adaptive fusion module, which performs parameter sharing and adaptive weight fusion on multiple perspectives of street view images to extract multi-view semantic features; then it constructs a multi-scale adaptive fusion module, which uses multiple convolutional kernels of different sizes to extract multi-modal features after splicing in parallel, and introduces an adaptive weight mechanism based on maximum-average joint statistics to achieve multi-scale deep fusion of remote sensing images and street view images, and extract key urban area visual semantic information.

[0043] (3) This invention combines static spatial semantic embedding and dynamic graph topological embedding to generate initial entity representations. It introduces a row-column cross attention mechanism to perform two-stage feature interaction on head entities, relations and tail entities to enhance tail entity representation. It also constructs a joint decision loss function, which includes a hierarchical perceptual loss term based on modulus distance and phase distance, and a contrastive learning loss term based on multimodal features and regional entity semantic consistency constraints. The embedding representation is jointly optimized and trained to finally obtain a low-dimensional embedding vector of urban area, supporting downstream urban management and decision analysis tasks. Attached Figure Description

[0044] The invention will now be further described with reference to the accompanying drawings.

[0045] Figure 1 This is an overall flowchart of an example of the present invention.

[0046] Figure 2 This is a multi-source geospatial data map used in this invention;

[0047] Figure 3 This is a design ontology diagram of the RegionKG construction example of the present invention;

[0048] Figure 4 This is a diagram of the Multi-View Adaptive Fusion (MVAF) module for Street View Imagery (SVI) in an example of the present invention;

[0049] Figure 5 This is a diagram of the multi-scale adaptive fusion (MSAF) module for fusing remote sensing images (RSI) and street view images (SVI) in an example of the present invention.

[0050] Figure 6 This is a diagram of the RegionKGE framework for RegionKG representation learning in an example of the present invention.

[0051] Figure 7 This is an example diagram of the cross-attention mechanism module for fusing head entity embedding and relation embedding in this invention.

[0052] Figure 8 This is a schematic diagram of the urban area representation system supported by the multimodal knowledge graph, which is an example of the present invention. Detailed Implementation

[0053] The present invention will now be described in detail with reference to the accompanying drawings, which will make the technical approach and operation steps of the present invention clearer.

[0054] This invention comprises four parts: multimodal knowledge graph construction, multi-source data feature fusion, knowledge graph embedding learning, and downstream task applications, as detailed below:

[0055] First, based on the designed RegionKG ontology, the multi-source geographic data of the target study area is processed to obtain the spatial adjacency, subordination and similarity relationships between geographic entities, and a multimodal geographic knowledge graph RegionKG is constructed.

[0056] Second, by using convolutional neural networks and adaptive fusion, remote sensing image data and street view image data in the multimodal knowledge graph are deeply fused to extract key visual semantic information of urban areas.

[0057] Third, this invention innovatively proposes a joint embedding method of static spatial semantics and dynamic graph topology, which embeds spatial location information and spatial relationship information from multimodal knowledge graphs into vector space; at the same time, it constructs a hierarchical perceptual loss function coupled with hierarchical structure and semantic similarity, and combines multimodal contrastive learning to jointly optimize the embedding representation to obtain feature vectors of urban areas.

[0058] Fourth, based on the feature vectors of urban areas, the data is applied to four downstream tasks: land use classification, abnormal behavior prediction, housing price estimation, and urban village identification, to verify the effectiveness of the regional representation.

[0059] In this invention, the knowledge graph is defined in triple form. In the formula, Represents a set of entities in a knowledge graph. Represents a set of relations. This represents the set of all fact triples in a knowledge graph.

[0060] Based on this, the present invention extends the above definition into a multimodal knowledge graph based on multimodal features. In the formula, Represents the set of modal features of entities in a knowledge graph. Corresponding to structured modal feature representation, Modal feature representation of the corresponding image, The corresponding modal feature representation after fusion. In this invention, remote sensing image data, street view image data, point of interest data, and road network data are all entities in a multimodal knowledge graph. .

[0061] Link prediction in knowledge graphs is an important validation method for knowledge graph representation learning. It involves predicting missing entities in knowledge graph triples. It can evaluate the ability of representation learning models to preserve the semantic relevance and structural topology of knowledge graphs, thereby reflecting the accuracy and effectiveness of urban area representation learning results.

[0062] In this invention, such as Figure 2As shown, multi-source geographic big data, such as remote sensing image data RSI, street view image data SVI, point of interest data POI, and road network data RN, of the target study area (such as the central urban area of ​​Changzhou) were used to learn the representation of each grid area.

[0063] RSI (Radio Sensing Index) is a direct result of Earth observation, possessing rich spatial texture features and semantic information, capable of reflecting urban structure, morphology, and functional layout. In this embodiment, the RSI of the target study area is obtained through Google Maps (https: / / mapsplatform.google.com) with a resolution of 0.5 meters. SVI (Radio Visual Identity) provides an immersive horizontal view of the city, capturing urban scenes where built environments and pedestrian interactions are most frequent. In this embodiment, panoramic SVI data within the target study area is obtained through Baidu Maps (https: / / lbsyun.baidu.com) and cropped into images corresponding to four angles. POI data is a collection of specific location data that is of significant importance or interest, including supermarkets, schools, hospitals, etc. It usually contains information such as latitude and longitude coordinates, category, and place name. In this embodiment, POI data of the target study area is obtained by calling Amapservice (https: / / lbs.amap.com), and the 23 categories corresponding to the POIs are obtained. RN is an important infrastructure supporting efficient and sustainable urban systems and is closely related to urban planning and management. In this embodiment, the required RN data is obtained through OpenStreetMap (http: / / www.openstreetmap.org / ), and the level to which each road belongs can also be obtained.

[0064] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, a method for representing urban areas supported by multimodal knowledge graphs is provided, comprising the following steps:

[0065] Step S1: Based on the pre-constructed geographic ontology (referring to the RegionKG ontology in this embodiment), the target research area is divided into grid areas of a preset size, and the multimodal geographic entities corresponding to the grid areas and the spatial relationships, subordinate relationships and similarity relationships between entities are determined to construct a multimodal geographic knowledge graph.

[0066] In this optional embodiment, S1 includes:

[0067] Step S11: Set up geographic entity types according to the hierarchical structure from low to high, and determine the relationship types of spatial relationships, subordinate relationships and similarity relationships according to the geographic entity types to obtain the RegionKG ontology;

[0068] Step S12: Acquire remote sensing image data RSI, street view image data SVI, point of interest data POI, and road network data RN within the target study area;

[0069] Step S13: Based on the RegionKG ontology, divide the target study area into grid areas of a preset size, determine the remote sensing image data RSI, street view image data SVI, point of interest data POI, and road network data RN corresponding to each grid area, establish the subordinate relationship between each grid area and the corresponding point of interest, street view image, remote sensing image, and road network, as well as the spatial relationship between each grid area and the subordinate relationship between each grid area and the administrative region.

[0070] Step S14: Obtain the proportion of various interest points in each grid area, construct the interest point proportion vector, and calculate the cosine similarity between grid areas based on the interest point proportion vector;

[0071] Step S15: Define grid region pairs with cosine similarity greater than a preset threshold as similarity relationships, and construct a multimodal geographic knowledge graph based on geographic entity type, spatial relationship, subordinate relationship and similarity relationship.

[0072] In this optional embodiment, the geographic entity types are set in a hierarchical structure from low to high, including:

[0073] The underlying geographic entity types include remote sensing image data RSI, street view image data SVI, point of interest data POI, and road network data RN;

[0074] Intermediate-layer geographic entity types include grid areas, point of interest categories, and road categories;

[0075] High-level geographic entity types include administrative regions.

[0076] Specifically, (I) RegionKG Construction: This invention designs the ontology of RegionKG: When designing the knowledge graph ontology, it is necessary to fully consider the hierarchical nesting relationship between spatial entities and the various association relationships between entities, so as to fully integrate multi-source geographic data and realize a comprehensive urban area representation.

[0077] Points of interest entities, street view image entities, remote sensing image entities, and road network entities are set as the lowest level of geographic entities; administrative region entities are set as the highest level of geographic entities; and point of interest category entities, road category entities, and grid area entities are set as intermediate level geographic entities.

[0078] In addition, the RegionKG ontology includes various relationship types, including spatial relationships, subordinate relationships, and similarity relationships, to express various semantic associations between different geographic entities.

[0079] Based on the designed RegionKG ontology, this invention instantiates geographic data:

[0080] like Figure 3 As shown, the target study area (such as the central urban area of ​​Changzhou) is divided into 6424 grid areas with a side length of 250 meters. The interest point data, street view image data, remote sensing image data, and road network data corresponding to each grid area are determined, as well as the spatial adjacency relationships between grid areas and between grid areas and administrative districts. The proportion of each type of interest point in each grid area is obtained, and the proportions of each type of interest point are used to form the interest point distribution vector of that grid area. The cosine similarity between any two grid areas is calculated based on the following formula:

[0081] ;

[0082] In the formula, and These vectors represent the proportion of Points of Interest (POIs) of various types in the two grid regions. When the calculated cosine similarity is greater than a preset threshold (e.g., 0.9), the two grid regions are defined as functionally similar regions, and a similarity relationship is established between the grid regions accordingly.

[0083] Step S2: Based on the multimodal geographic knowledge graph, street view image data is subjected to multi-view adaptive fusion to obtain street view image features. Remote sensing image features are extracted from remote sensing image data. After aligning the remote sensing image features with the street view image features, multi-scale adaptive fusion is performed to obtain the fused multi-scale features.

[0084] In this optional embodiment, S2 includes:

[0085] Step S21: Based on the multimodal geographic knowledge graph, obtain street view image data from multiple perspectives of each grid area, process the street view image data from each perspective in parallel using a shared convolutional kernel, and obtain the initial features of each perspective through batch normalization and activation function processing.

[0086] Step S22: Introduce learnable weight parameters and normalize them to obtain normalized weights. Then, based on the normalized weights, perform weighted fusion on the initial features of each viewpoint to obtain street view image features.

[0087] Step S23: Use the pre-trained ResNet50 network to extract remote sensing image features from the remote sensing image data. Perform dimension alignment processing on the remote sensing image features and street view image features respectively. Then, stitch the aligned remote sensing image features and street view image features together by channel dimension to obtain the stitched features.

[0088] Step S24: Multiple convolution kernels of different sizes are used to process the spliced ​​features in parallel and then processed by batch normalization and activation function to obtain multi-scale features; global max pooling and global average pooling are performed in parallel for each multi-scale feature, and the two pooling results are spliced ​​together. The initial weight coefficients of each scale branch are learned through one-dimensional adaptive convolution, and the adaptive weights of each scale are obtained through normalization constraints.

[0089] Step S25: Dynamically weight and fuse the multi-scale features based on adaptive weights for each scale to obtain the fused multi-scale features.

[0090] The alignment process includes: applying pooling and convolution operations to the remote sensing image features and the street view image features respectively, and unifying the spatial size and number of channels of the feature maps of the remote sensing image features and the street view image features to complete the feature dimension alignment.

[0091] Specifically, (ii) RegionKG fusion: In order to make full use of the multi-source visual data in the multimodal knowledge graph, it is necessary to perform feature fusion on remote sensing images and street view images.

[0092] (1) Adaptive fusion of street view images from multiple perspectives:

[0093] like Figure 4 As shown, considering the unique scene semantic features of different viewpoint images of street view, this invention proposes a multi-view adaptive fusion module (MVAF). A 3×3 convolutional layer with shared kernel parameters is used to process the four viewpoint images of the street view in parallel. Batch normalization and ReLU activation are performed on the initial feature map of each viewpoint to obtain the viewpoint feature map for each viewpoint. This parameter-sharing mechanism not only significantly reduces the number of model parameters but also ensures the consistency of the representation of features from different viewpoints in the same feature space.

[0094] ;

[0095] In the formula, Indicates the first Street view images from multiple perspectives; This indicates shared convolutional kernel parameters; Indicates batch normalization; Indicates the first View feature maps extracted from each viewpoint; This represents the activation function.

[0096] Considering the varying importance of different perspectives in representing urban areas, the MVAF module introduces an adaptive weighting mechanism. This mechanism utilizes learnable weight parameters. The importance weights of each viewpoint feature are dynamically adjusted based on the input data, and the learnable weight parameters are normalized using the softmax function to obtain the adaptive fusion weights for each viewpoint. Then, the view feature maps from each viewpoint are weighted and summed to obtain the multi-view fused features. :

[0097] ;

[0098] ;

[0099] In the formula, Indicates the first Learnable weight parameters corresponding to each perspective; This represents the normalized adaptive fusion weights; Indicates the activation function; This represents the multi-view fusion feature of the final output.

[0100] (2) Multi-scale adaptive fusion of remote sensing imagery and street view imagery:

[0101] like Figure 5 As shown, street view imagery and remote sensing imagery describe regional features from ground-level horizontal perspective and aerial overhead perspective, respectively, and contain complementary semantic information. To effectively integrate the advantages of these two modalities of data, this invention proposes a multi-scale adaptive fusion module (MSAF) that captures and fuses the semantic information between the two at multiple scales.

[0102] Remote sensing features of remote sensing images are extracted using a pre-trained ResNet50 network. And for remote sensing features and multi-view fusion features Dimension alignment is performed. Specifically, pooling and convolution operations are applied to the remote sensing features and the multi-view fusion features respectively to unify the spatial size and number of channels of the feature maps, thus completing feature dimension alignment. Then, the two sets of aligned features are concatenated by channel dimension to obtain the concatenated features. :

[0103] ;

[0104] In the formula, This indicates feature fusion.

[0105] Next, three convolutional kernels of different sizes (3×3, 5×5, and 7×7) were used in parallel to process and fuse the features, and batch normalization was performed on the results of each convolutional processing. Activation yields first-scale features, second-scale features, and third-scale features:

[0106] ;

[0107] In the formula, Indicates the kernel size as Convolution operations; Choose 3, 5, or 7; Indicates the kernel size as The convolutional operations extract scale features. A 3×3 convolutional kernel is used to extract local detail features, a 5×5 convolutional kernel is used to capture mid-scale block unit features, and a 7×7 convolutional kernel is used to obtain global spatial context features.

[0108] Similar to the MVAF module, the MSAF module also employs an adaptive weighting mechanism. This mechanism abandons the conventional single-path statistical mode of global average pooling and adopts a branch-independent max-average joint statistical strategy: for each scale feature (first, second, and third scale features), global max pooling and global average pooling are performed in parallel, and the pooling results are concatenated to obtain the statistical vector of that scale feature; the statistical vectors of each scale feature are then input into a one-dimensional adaptive convolutional layer to learn the initial weight coefficients corresponding to each scale, and finally, these initial weight coefficients are processed... Function normalization yields adaptive fusion weights for each scale. Finally, the features at each scale are weighted and summed to obtain the fused multi-scale features. :

[0109] ;

[0110] ;

[0111] In the formula, Represents learnable parameters; This represents the normalized weights; This represents the multi-scale features after fusion. This adaptive weighting mechanism achieves deep multi-scale fusion of remote sensing imagery and street view imagery without significantly increasing network complexity and the number of parameters.

[0112] Step S3: After obtaining static and dynamic embeddings based on the multimodal geographic knowledge graph, a two-stage information fusion of entity embedding and relation embedding is performed using a cross-attention mechanism to enhance tail entity embedding. Optimization is then performed under the guidance of a joint decision loss function, embedding spatial location information, spatial relation information, and multimodal data information from the multimodal geographic knowledge graph into a vector space to obtain the urban area embedding vector; where dynamic embedding includes fused multi-scale features.

[0113] In this optional embodiment, S3 includes:

[0114] Step S31: Based on the multimodal geographic knowledge graph, Space2Vec is used to encode the geographic entity type to obtain the location embedding, and a pre-trained language model is used to obtain the text embedding of the geographic entity type containing the name attribute. The location embedding and the text embedding are concatenated to obtain the static embedding.

[0115] Step S32: For entities in a structured multimodal geographic knowledge graph, a graph attention network is used to obtain dynamic embeddings from the multimodal geographic knowledge graph; for multimodal entities in a multimodal geographic knowledge graph, the fused multiscale features are used as dynamic embeddings.

[0116] Step S33: Use a fully connected layer to reduce the dimensionality of the static and dynamic embeddings respectively, and concatenate the dimensionality-reduced static and dynamic embeddings to obtain the initial head entity embedding and the initial tail entity embedding. Then, use a pre-trained language model to encode the relations in the knowledge graph to obtain the relation embedding.

[0117] Step S34: Concatenate and deform the initial head entity embedding and relation embedding to obtain the first fused feature. Process the first fused feature through convolution operation to obtain the query matrix, key matrix and value matrix. Calculate the attention weights of the query matrix and key matrix based on the cross attention mechanism. After weighted aggregation of the attention weights and value matrix, superimpose the original first fused feature to obtain the fused feature.

[0118] Step S35: Concatenate and deform the fused features with the initial tail entity embedding to obtain the second fused feature. Process the second fused feature through convolution operation to obtain the query matrix, key matrix and value matrix. Calculate the attention weights of the query matrix and key matrix based on the cross attention mechanism. After weighted aggregation of the attention weights and value matrix, superimpose the original second fused feature to obtain the enhanced tail entity embedding.

[0119] Step S36: Guided by the joint decision loss function, optimize the training based on the enhanced tail entity embedding and the fused features, embed the spatial semantic information, spatial relationship information and multimodal data information of the multimodal geographic knowledge graph into the vector space to obtain the urban area embedding vector.

[0120] In this optional embodiment, obtaining text embeddings for geographic entity types containing name attributes using a pre-trained language model includes:

[0121] The geographic entity type containing name attributes is cleaned and segmented to obtain the segmentation results;

[0122] The word segmentation results are fed into a pre-trained language model to obtain the latent vectors of each token;

[0123] Layer normalization is performed on the latent vectors of each token to obtain the text embedding.

[0124] In this optional embodiment, the attention weights of the query matrix and the key matrix are calculated based on the cross-attention mechanism, and the attention weights are weighted and aggregated with the value matrix, including:

[0125] Calculate the similarity between the feature vectors of each position in the query matrix and the corresponding position in the key matrix in the row and column, and normalize them to obtain the attention weights;

[0126] The attention weights are weighted and aggregated with the feature vectors in the corresponding rows and columns of the value matrix.

[0127] In this optional embodiment, S36 includes:

[0128] Step S361: Based on the hierarchical knowledge embedding approach, explicitly model the semantic hierarchical structure of geographic entities in the embedding space, and construct the hierarchical structure perception loss using the edge ranking loss form.

[0129] Step S362: Based on the fused modal features in the enhanced tail entity embedding, construct a multimodal contrastive learning loss using the InfoNCE contrastive learning loss function;

[0130] Step S363: Optimize and train under the guidance of the joint decision loss function. Use hierarchical structure perception loss to accurately model the hierarchical structure relationship of geographic entities, and use multimodal contrastive learning loss to promote the visual semantic consistency alignment between different modal data. Embed spatial semantic information, spatial relationship information and multimodal data information in the multimodal geographic knowledge graph into the vector space to obtain the city area embedding vector.

[0131] In this optional embodiment, the joint decision loss function is composed of a weighted sum of hierarchical perceptual loss and multimodal contrastive learning loss.

[0132] Specifically, (iii) RegionKG Embedding: In this invention, a knowledge graph representation learning framework, RegionKGE, is proposed for multimodal geographic knowledge graphs (RegionKG) to embed entities and relations in the multimodal knowledge graph into a vector space. This framework mainly includes three parts: static and dynamic entity embedding acquisition, tail entity enhancement based on cross-attention, and joint decision loss optimization.

[0133] (1) Acquisition of static and dynamic entity embedding:

[0134] like Figure 6 As shown, FC represents a fully connected layer, and h, r, and t represent the head entity, relation, and tail entity, respectively; the static and dynamic embeddings of entities in RegionKG are obtained respectively.

[0135] For entities containing geographic coordinate attributes (such as point of interest entities, grid area entities, etc.), the Space2Vec spatial location encoding model based on grid cells is used for location encoding, mapping the entity's two-dimensional spatial coordinates to a high-dimensional spatial location embedding, effectively capturing spatial distance, direction, and scale information. For entities containing name attributes (such as point of interest entities, road entities, administrative region entities, etc.), the pre-trained language model BERT is used to extract text semantic embeddings. Specifically, the entity text is first cleaned and segmented, and the processed text is input into the BERT model to obtain the latent vectors of each token. After layer normalization, it is used as the text semantic embedding of the entity. For entities containing both geographic coordinates and name attributes, the above location embeddings and text embeddings are concatenated to obtain a static embedding that integrates spatial semantic information.

[0136] For dynamic embeddings, different strategies are employed to acquire them separately. For structured entities (i.e., purely structured entities such as points of interest, roads, administrative regions, and grid areas), a graph attention network (GAT) is used to encode the structured topology of the multimodal knowledge graph. Specifically, the GAT uses a stacked two-layer graph attention network, with four attention heads in each layer. The multi-head attention outputs perform a concatenation operation, and feature aggregation is completed through nonlinear activation to output the dynamic embeddings of the structured entities. For multimodal entities (i.e., remote sensing image entities and street view image entities), the aforementioned multimodal fusion features are used as the dynamic embeddings of the entity.

[0137] Then, two cascaded fully connected layers (256 neurons in the first layer and 128 neurons in the second layer) are used to reduce the dimensionality of the static and dynamic embeddings respectively. The two dimensionality-reduced embeddings are then concatenated to obtain the initial head entity embedding. and initial tail entity embedding Simultaneously, the relationships in the knowledge graph are encoded based on the pre-trained BERT model to obtain the initial relationship embeddings. This ensures that relation representations and entity representations reside in the same semantic space.

[0138] (2) Tail entity embedding enhancement based on cross-attention mechanism:

[0139] like Figure 7 As shown, in order to effectively capture spatial dependencies in knowledge graphs, this invention designs a two-stage information fusion process based on a cross-attention mechanism to enhance the embedding features of tail entities.

[0140] Embed the initial head entity and initial relation embedding By splicing and deforming, the first fusion feature is obtained. In the formula, The number of channels. The query matrix is ​​extracted from the first fused feature using a 1×1 convolutional kernel. Key matrix Sum matrix ,in , , To reduce computational overhead:

[0141] ;

[0142] In the formula, This represents a convolution operation with a kernel size of 1.

[0143] Then, the cross-attention score is calculated. Unlike conventional attention mechanisms, this cross-attention mechanism only considers the interaction between each position and its positions in the same row and column. Specifically, in the query matrix... Extracting positions Corresponding feature vector and obtain location In the key matrix Construct a set of all feature vectors in the row and column corresponding to the given position. Calculate the query vector. With sets The similarity of each feature vector in the data, after... Normalization yields the attention weights:

[0144] ;

[0145] ;

[0146] In the formula, Represents a set The first in 1 eigenvector; This indicates the similarity score; This represents the normalized attention weights.

[0147] Finally, obtain the location. In the value matrix Construct a set of all feature vectors in the row and column corresponding to the given position. Using attention weights on the set The feature vectors in the first fusion are weighted and aggregated, and then fused with the original first feature vectors. Adding them together yields the enhanced first fusion feature. :

[0148] ;

[0149] The enhanced first fusion feature Reflatten and project to obtain the embedding vector. Embedded vector Embedded with the initial tail entity Repeat the cross-attention interaction process described above to obtain the enhanced tail entity embedding. .

[0150] (3) Joint decision loss function:

[0151] This invention proposes a joint decision loss function that combines hierarchical structure perception and contrastive learning. :

[0152] ;

[0153] In the formula, A loss function for the hierarchical structure; For multimodal contrastive learning loss function; Hyperparameters are used to control the relative importance of the two types of losses.

[0154] For the hierarchical structure-aware loss function, this invention maps entities and relations of the knowledge graph to polar coordinate space, explicitly modeling the semantic hierarchy of geographic entities. Triples between entities at the same level are represented by phase distance, while triples between entities at different levels are represented by modulus distance. Scoring function:

[0155] ;

[0156] In the formula, For head entity embedding, For relational embedding, For the enhanced tail entity embedding, These are learnable phase weight parameters. It is a rating function, and the first term of this rating function ( The first term represents the modulus distance between entities, used to model the hierarchical differences between entities at different levels; the second term ( () represents the phase distance between entities, used to model the semantic similarity between entities at the same level.

[0157] Based on the above scoring function, the hierarchical perception loss is calculated using the edge ranking loss form:

[0158] ;

[0159] In the formula, This represents the set of positive triples in the training set. Indicates targeting a specific head entity and relationships The constructed negative sample set, For edge parameters, This is a tail entity that was incorrectly predicted. This is the correct triplet score. These are incorrect triple scores. This loss function design encourages the model to assign higher scores to correct triples and lower scores to incorrect triples, thereby learning an embedding representation that accurately represents entity relationships in the geographic knowledge graph. Positive triples are objectively existing fact samples (head entity, relationship, tail entity) within the knowledge graph, while the negative sample set is constructed using a local random replacement strategy, obtaining different sets of positive triples based on different downstream tasks.

[0160] For the multimodal contrastive learning loss function, the enhanced multimodal fusion features will be... Embedded as a multimodal feature representation of grid region entities, using the InfoNCE contrastive learning loss function:

[0161] ;

[0162] In the formula, Embedding of Region entities; Embedding of the fused multimodal features; Indicates cosine similarity; For temperature parameters; This refers to the number of samples in the batch. This indicates the embedding of other Region entities in the batch.

[0163] Using a joint function of hierarchical perceptual loss and multimodal contrastive learning loss as the optimization objective, the initial head entity embedding, initial relation embedding, and enhanced tail entity embedding are iteratively trained to obtain the embedding vectors of each entity in the multimodal knowledge graph in the vector space. This joint decision loss framework not only accurately models the hierarchical relationships of geographic entities, but also promotes consistent alignment of multimodal visual semantics through a contrastive learning mechanism.

[0164] S4. Based on urban area embedding vectors, conduct land use classification, abnormal behavior prediction, housing price estimation, and urban village identification.

[0165] Specifically, (iv) Downstream tasks: To demonstrate the effectiveness of the learned region representation, the present invention carries out the following four downstream tasks:

[0166] (1) Land use classification. Land use refers to the way humans develop and manage land resources, including different types such as residential areas, commercial areas, industrial areas, and green spaces. The spatial layout of land use is closely related to urban planning, transportation, and resource management. Accurately identifying the land use type of a region is of great significance for urban management and planning.

[0167] (2) Prediction of abnormal behavior: The distribution of abnormal behavior is usually closely related to factors such as the region's socioeconomic status, population density, and infrastructure. Predicting the risk of abnormal behavior in a specific area helps law enforcement agencies optimize resource allocation and improve public safety.

[0168] (3) Housing price estimation: Housing prices are an important indicator reflecting the economic value and livability of a region, and are affected by various factors such as location, surrounding facilities, transportation convenience, and environmental quality. Predicting housing prices can provide a reference for urban development and investment decisions.

[0169] (4) Identification of urban villages: Urban villages are residential areas that lag behind the pace of urban development, are outside of modern urban management, and have low living standards during the rapid development of cities. Classifying urban villages can provide valuable information for sustainable urban planning.

[0170] like Figure 8 As shown, according to another embodiment of the present invention, a multimodal knowledge graph-supported urban area representation system is also provided, the system comprising:

[0171] The multimodal knowledge graph construction unit 1 is used to divide the target research area into grid regions of a preset size based on the pre-constructed RegionKG ontology, determine the multimodal geographic entities corresponding to the grid regions and the spatial relationships, subordinate relationships and similarity relationships between entities, and construct a multimodal geographic knowledge graph.

[0172] Multi-source data feature fusion unit 2 is used to perform multi-view adaptive fusion of street view image data based on multimodal geographic knowledge graph to obtain street view image features, extract remote sensing image features from remote sensing image data, and perform multi-scale adaptive fusion after aligning remote sensing image features with street view image features to obtain fused multi-scale features.

[0173] Knowledge graph embedding learning unit 3 is used to obtain static and dynamic embeddings based on multimodal geographic knowledge graphs, and then use a cross-attention mechanism to perform two-stage information fusion of entity embedding and relation embedding to enhance tail entity embedding. It is also optimized under the guidance of a joint decision loss function to embed spatial location information, spatial relation information and multimodal data information from the multimodal geographic knowledge graph into a vector space to obtain urban area embedding vectors; among which, dynamic embedding includes fused multi-scale features.

[0174] Downstream task application unit 4 is used to carry out land use classification, abnormal behavior prediction, housing price estimation and urban village identification based on urban area embedding vectors.

[0175] To verify the effectiveness of the technical solution of this invention, the following experiments were conducted: First, the experimental parameter configuration and evaluation indicators were given; then, the embedding quality was verified through a link prediction task; finally, the generalization ability of the region representation was verified through four downstream tasks; and finally, the experimental results were analyzed and discussed, as follows:

[0176] (1) Experimental Setup: In this invention, the following experimental parameters were set for the specific implementation process of RegionKG embedding representation learning: the dimensions of both static and dynamic embedding of knowledge graph entities were set to 32, and the final output entity embedding vector dimension was 64. The optimizer adopted the Adam optimization algorithm, the learning rate was set to 0.001, and the weight decay coefficient was set to... Batch size is set to 64, and training epochs are set to 800. Edge parameters (The edge distance used to control the difference between positive and negative sample scores) is set to 1.0, temperature parameter (Temperature coefficient used to control the smoothness of the distribution in the contrastive learning loss) is set to 0.1, and the contrastive learning loss weight coefficient is... Set it to 0.1.

[0177] To verify the effectiveness of this invention in learning urban area representations, link prediction and evaluation were performed on two types of triples in the knowledge graph: (grid region, spatial adjacency, grid region) and (grid region, functional similarity, grid region). , , and (mean reciprocal rank) is used as an evaluation metric to verify whether the learned region embedding vectors can effectively represent the characteristics of each grid region.

[0178] For the two classification tasks of land use classification and urban village identification, Random Forest (RF) is used as the classifier to classify the region embedding vectors. The evaluation metrics are Overall Accuracy (OA) and Kappa coefficient.

[0179] For the two regression tasks of abnormal behavior prediction and house price estimation, the Lasso regression model is used for prediction based on the regional embedding vector, and the evaluation indicators are root mean square error (RMSE) and mean absolute error (MAE).

[0180] (2) RegionKG Link Prediction: As shown in Table 1, for the link prediction task, this invention selects the following three types of knowledge graph representation learning models as the comparison baseline:

[0181] The first category is general structured knowledge graph representation learning models, including four models: RotatE, SimpIE, HAKE, and ConvHLE. Among them, RotatE, SimpIE, and HAKE are baseline models, while ConvHLE is a high-low layer feature interaction convolutional network model based on attention mechanism.

[0182] The second category is multimodal knowledge graph representation learning models, including two models: IKRL and RSME. IKRL is an image embedding knowledge graph representation learning model, while RSME is a multimodal knowledge graph representation learning model that integrates region semantic information.

[0183] The third category consists of representation learning models for geographic knowledge graphs, including three models: SSLP, SR-RotatE, and SR-HAKE. Among them, SSLP is a geographic knowledge graph embedding model that integrates spatial and semantic information, while SR-RotatE and SR-HAKE are geographic knowledge graph embedding models based on spatial relationship constraints.

[0184] Table 1. Link prediction results for different representation learning models

[0185]

[0186] The link prediction experiment results show that the RegionKGE proposed in this invention outperforms all comparative models in all evaluation metrics, verifying the effectiveness of the embedded framework of this invention and enabling comprehensive and accurate urban area representation. Specific analysis is as follows:

[0187] ① The general structured knowledge graph representation learning model is inferior to other types of models in all evaluation metrics because it does not consider image data and geospatial knowledge. This shows that structured information alone cannot support the accurate representation of urban areas.

[0188] ② While other models incorporate multimodal data or geospatial constraints and outperform general structured models, they still lag behind the RegionKGE proposed in this invention. This result demonstrates that for representation learning of multimodal geographic knowledge graphs, combining multimodal visual data and geographic semantic information is crucial.

[0189] ③ ConvHLE outperforms other general structured models in general knowledge graph research, but performs worse than other general models in representation learning for RegionKG. The reason for this is that RegionKG has a relatively small sample size, making it difficult for the complex multi-layer interactive convolutional neural network in ConvHLE to adequately fit the data. In contrast, the RegionKGF and RegionKGE proposed in this invention both employ lightweight module designs, making them suitable for representation learning tasks of small-scale multimodal geographic knowledge graphs.

[0190] ④ After removing the cross-attention mechanism, the link prediction performance decreased by about 7.1%, indicating that the relationship-aware feature enhancement module based on cross-attention can effectively improve the interaction modeling quality between head entities, relationships and tail entities.

[0191] Based on the above experimental results, this invention successfully achieved a comprehensive and accurate representation of urban areas by integrating multimodal data and geospatial knowledge and adopting a lightweight design suitable for small-scale multimodal geographic knowledge graphs, thus verifying the superiority of RegionKGE in multimodal knowledge representation in urban scenarios.

[0192] (3) Downstream tasks:

[0193] ① Land Use Classification: As shown in Table 2, this invention obtains land use data from the Third National Land Survey of Changzhou City and classifies it into six categories: Industrial Land (IND), Commercial Land (COM), Public Land (PUB), Water Body (WAT), Residential Area (RES), and Green Space (GRE). The training and test sets are divided in a 7:3 ratio. Specialized land use identification models are used for comparison; these models incorporate more socially-perceived data to better accomplish the task.

[0194] Table 2. Land use classification results based on different models

[0195]

[0196] The results show that the city region embedding vectors obtained by this invention, trained in RF, outperform other models in both OA and Kappa metrics. In terms of data sources, the method proposed in this invention achieves higher performance than the previous three models due to its ability to integrate multi-source heterogeneous data based on knowledge graphs. Compared to InceptionV3 and BERT models, although these models use more data, the model proposed in this invention can delve deeper into spatial semantic relationships and other profound information during the representation process using RegionKG, thus achieving superior results.

[0197] In summary, the urban area representation learning method proposed in this invention can effectively perceive the land use distribution in cities.

[0198] ② Abnormal Behavior Prediction: As shown in Table 3, this invention divides 1420 grid areas in Changzhou containing abnormal behavior data into training and test sets in a 7:3 ratio. Simultaneously, this invention compares DeepCrime (trained based on POIs), XGBoost (trained based on population data, POIs, and RNs), and UDAC (trained based on weather data, POI data, and taxi trip data).

[0199] Table 3. Prediction Results of Abnormal Behavior from Different Models

[0200]

[0201] The results show that the method of this invention can approach or outperform other specific models used for predicting anomalous behavior. Compared with the DeepCrime model, the knowledge graph structure of this invention can not only effectively integrate multi-source data, but also mine complex relationships, thus achieving better prediction results. Compared with XGBoost and UDAC, considering that they use data closely related to anomalous behavior such as population, weather, and travel, the experimental results of this invention are close to their results, which is acceptable.

[0202] Therefore, the experimental results demonstrate that the urban region representation results based on RegionKG are closely related to urban anomalous behavior.

[0203] ③ Housing Price Prediction: As shown in Table 4, this invention obtains housing price information for the center of Changzhou city (https: / / www.anjuke.com / ), calculates the average housing price for each grid area, and divides the 2024 areas with housing price data into a 7:3 training set and a test set, using a Lasso regression model for prediction. A specific housing price prediction model is selected for comparison.

[0204] Table 4. House price estimation results of different models

[0205]

[0206] The results show that the method proposed in this invention can approach or outperform the performance of specific housing price prediction models. This result is noteworthy, especially considering that the comparison models use specialized data features directly related to housing prices, while this invention only uses general regional representations. Housing price prediction tasks not only need to consider the characteristics of the region itself, but also the proximity relationships between regions. The urban region embedding constructed by this invention based on knowledge graphs can effectively capture both aspects of information. This result demonstrates the effectiveness of combining RegionKG to comprehensively consider regional characteristics and spatial relationships in obtaining region embeddings, and also shows that the representation results are closely related to human habitation and economic development in practical applications.

[0207] ④ Urban Village Identification: As shown in Table 5, this invention divides 1425 areas in the urban center of Changzhou that contain data on urban villages into training and test sets in a 7:3 ratio, and selects different urban village classification models for comparison to evaluate the ability of the representation results to identify urban villages.

[0208] Table 5. Classification results of urban villages using different models

[0209]

[0210] The results show that the performance of this invention is close to or better than other specific urban village classification models. Specifically, the model of this invention outperforms BMA, which uses a single data source, and HR-RSF-UV and VisionLSTM, which combine multiple data sources. This demonstrates that RegionKG can effectively integrate multiple data sources and fully mine intrinsic features to achieve better results. Furthermore, MVRAN and G-FFB, although using only RSI and SVI, have undergone complex processing of the data specifically for urban village classification tasks; therefore, the experimental results demonstrate that the models can effectively identify the areas where urban villages are located.

[0211] The results of the four downstream experiments demonstrate that the urban region embedding vectors obtained through RegionKGE in this invention, combined with a machine learning model, achieve results close to or better than those of specific models. Given that these four experiments cover important urban spatial tasks, the experimental results sufficiently prove that the representations learned by this invention can comprehensively and effectively reflect urban spatial areas, and that this method can be extended to other downstream tasks, providing support for urban management decisions.

[0212] Furthermore, this invention primarily utilizes four geographic data sources—RSI, SVI, POI, and RN—that can reflect urban spatial areas from different perspectives. These data sources are effectively integrated through RegionKG, laying the foundation for urban regional representation. Notably, the structure of RegionKG is extensible. Future development could further incorporate important information sources such as check-in data, travel data, and social media data, based on the hierarchical structure of the RegionKG ontology, to more comprehensively learn urban regional representations. Simultaneously, the incorporation of the temporal dimension into the knowledge graph construction can be explored to capture the dynamic changes in urban areas, further enhancing the spatiotemporal integrity of the representation. Of course, the inclusion of these data may also necessitate further research into new knowledge graph representation learning methods.

[0213] However, representation learning methods based on RegionKG also have some limitations. On the one hand, the region that RegionKG can represent is limited to a grid area with a side length of 250m. While this scale is sufficient for most urban analysis scenarios, it is difficult to maintain effective analysis of urban space in situations requiring more refined representations. On the other hand, the construction and maintenance costs of knowledge graphs are high. Considering that the current RegionKG based on the center of Changzhou city already contains more than 300,000 triples, expanding to a larger research scope could incur unimaginable costs.

[0214] In summary, this invention proposes a RegionKG construction module (RegionKG Construction, RegionKGC). Current urban region representation models face challenges such as the inability to effectively integrate multimodal data, fine-grained modeling of spatial semantics, and detailed characterization of entity spatial relationships. To address these issues, this invention, based on MMKG, designs a method for constructing a RegionKG that combines multimodal geographic data such as RSI, SVI, POI, and RN. This method performs refined semantic modeling of urban regions through a multi-layered ontology architecture and integrates multi-level spatial relationships to achieve a comprehensive and accurate representation of urban regions.

[0215] This invention proposes a RegionKG feature fusion module (RegionKG Fusion, RegionKGF). Current research on RSI and SVI feature integration remains at an early fusion stage, neglecting the deep interaction between their features. Therefore, this invention innovatively constructs a deep fusion method for RSI and SVI using convolutional neural networks and an adaptive fusion module, extracting key urban area visual information and enhancing the performance of representation learning models.

[0216] This invention proposes a RegionKG Embedding (RegionKGE) module. Existing representation learning methods are mainly designed for structured geographic knowledge graphs and lack the ability to deeply integrate multimodal data. To address this, this invention combines attention mechanisms, hierarchical structure awareness, and multimodal contrastive learning to propose a novel knowledge graph representation learning method. This method effectively integrates spatial semantics, spatial relationships, and multimodal geographic data from the knowledge graph to obtain urban regional feature vectors, enabling downstream experiments and supporting urban management and decision-making.

[0217] This invention proposes a region-based urban region representation learning framework to address the challenge of existing region representation methods effectively integrating multi-source heterogeneous data. Specifically, this invention proposes RegionKGC to integrate multimodal geospatial data such as RSI, SVI, POI, and RN to construct a RegionKG knowledge graph. This invention designs RegionKGF, achieving multi-scale deep fusion of RSI and SVI through an adaptive weighting mechanism. This invention innovatively proposes RegionKGE, utilizing spatial semantic information, a crisp-cross attention mechanism, and joint-decision loss to perform representation learning on RegionKG, achieving urban region embedding and enabling four downstream tasks: land use classification, abnormal behavior prediction, housing price estimation, and urban village identification. Experimental results demonstrate that the integration module RegionKGF and the integration of multimodal data effectively improve representation learning results. Compared to existing models, the RegionKGE representation learning method improves accuracy metrics MRR and HITS@1 by at least 12% and 23%, respectively.

[0218] Furthermore, the embedding vectors obtained based on RegionKGE representation learning perform at a level close to or better than existing specific models in downstream tasks. These results demonstrate that the urban region representation method proposed in this invention can integrate multimodal data and mine deep features of urban regions, making it applicable to various downstream tasks and supporting smart city decision-making.

[0219] Table 6 RegionKG Entity Embedding Table

[0220]

[0221] As shown in Table 6, for the three types of structured entities—RN, Class, and Category—this invention uses a pre-trained BERT model to extract their static embeddings. For Region entities, this invention uses the Space2Vec model to obtain static embeddings. For POI and District entities, this invention uses the BERT model to obtain semantic embeddings and Space2Vec to obtain spatial embeddings, and then concatenates them to obtain static embeddings containing spatial semantic information. For the two types of multimodal entities—RSI and SVI—considering that they have already been fused in RegionKGF, this invention uses Space2Vect to obtain the static embeddings of the fused multimodal entities based on the position information of the SVI entity. For dynamic embeddings, this invention uses GAT to obtain the embeddings of structured entities, while the dynamic embeddings of SVI and RSI multimodal entities are obtained by RegionKGF.

[0222] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for representing urban areas supported by a multi-modal knowledge graph, characterized in that, Includes the following steps: S1. Based on the pre-constructed geographic ontology, the target research area is divided into grid regions of a preset size. The multimodal geographic entities corresponding to the grid regions and the spatial relationships, subordinate relationships and similarity relationships between entities are determined, and a multimodal geographic knowledge graph is constructed. S2. Based on the multimodal geographic knowledge graph, street view image data is subjected to multi-view adaptive fusion to obtain street view image features. Remote sensing image features are extracted from remote sensing image data. After aligning the remote sensing image features with the street view image features, multi-scale adaptive fusion is performed to obtain the fused multi-scale features. S3. After obtaining static and dynamic embeddings based on the multimodal geographic knowledge graph, a two-stage information fusion of entity embedding and relation embedding is performed using a cross-attention mechanism to enhance tail entity embedding. Optimization is then performed under the guidance of a joint decision loss function, embedding spatial location information, spatial relation information, and multimodal data information from the multimodal geographic knowledge graph into a vector space to obtain urban area embedding vectors; wherein, the dynamic embedding includes fused multi-scale features. S4. Based on urban area embedding vectors, conduct land use classification, abnormal behavior prediction, housing price estimation, and urban village identification.

2. The urban region representation method supported by multimodal knowledge graph according to claim 1, characterized in that, S1 includes: S11. Set up geographic entity types according to the hierarchical structure from low to high, and determine the relationship types of spatial relationship, subordinate relationship and similarity relationship according to the geographic entity types to obtain the map ontology; S12. Acquire remote sensing image data, street view image data, point of interest data, and road network data within the target study area; S13. Based on the map ontology, the target study area is divided into grid areas of a preset size. The remote sensing image data, street view image data, point of interest data and road network data corresponding to each grid area are determined. The subordinate relationship between each grid area and the corresponding point of interest, street view image, remote sensing image and road network is established, as well as the spatial relationship between each grid area and the subordinate relationship between each grid area and the administrative region. S14. Obtain the proportion of various interest points in each grid region, construct the interest point proportion vector, and calculate the cosine similarity between grid regions based on the interest point proportion vector. S15. Define grid region pairs with cosine similarity greater than a preset threshold as similarity relationships, and construct a multimodal geographic knowledge graph based on geographic entity type, spatial relationship, subordinate relationship and similarity relationship.

3. The urban region representation method supported by multimodal knowledge graph according to claim 2, characterized in that, The hierarchical structure for setting geographic entity types, from low to high, includes: The underlying geographic entity types include remote sensing image data, street view image data, point of interest data, and road network data; Intermediate-layer geographic entity types include grid areas, point-of-interest categories, and road categories; High-level geographic entity types include administrative regions.

4. The urban region representation method supported by multimodal knowledge graph according to claim 1, characterized in that, S2 includes: S21. Based on the multimodal geographic knowledge graph, street view image data from multiple perspectives of each grid area are obtained. The street view image data from each perspective is processed in parallel using a shared convolutional kernel and then processed by batch normalization and activation function to obtain the initial features of each perspective. S22. Learnable weight parameters are introduced and normalized to obtain normalized weights. Based on the normalized weights, the initial features of each viewpoint are weighted and fused to obtain street view image features. S23. Use the pre-trained ResNet50 network to extract remote sensing image features from remote sensing image data. Perform dimension alignment processing on the remote sensing image features and street view image features respectively. Then, stitch the aligned remote sensing image features and street view image features together by channel dimension to obtain stitched features. S24. Multiple convolutional kernels of different sizes are used to process and concatenate features in parallel, and then batch normalization and activation function processing are applied to obtain multi-scale features. Global max pooling and global average pooling are performed in parallel for each multi-scale feature. After concatenating the two pooling results, the initial weight coefficients of each scale branch are learned through one-dimensional adaptive convolution, and the adaptive weights of each scale are obtained through normalization constraints. S25. Based on the adaptive weights of each scale, the multi-scale features are dynamically weighted and fused to obtain the fused multi-scale features.

5. The urban region representation method supported by multimodal knowledge graph according to claim 1, characterized in that, S3 includes: S31. Based on a multimodal geographic knowledge graph, Space2Vec is used to encode geographic entity types to obtain location embeddings, and a pre-trained language model is used to obtain text embeddings for geographic entity types containing name attributes. The location embeddings and text embeddings are concatenated to obtain static embeddings. S32. For entities in a structured multimodal geographic knowledge graph, a graph attention network is used to obtain dynamic embeddings from the multimodal geographic knowledge graph; for multimodal entities in a multimodal geographic knowledge graph, the fused multi-scale features are used as dynamic embeddings. S33. Dimensionality reduction is performed on static and dynamic embeddings using fully connected layers. The dimensionality-reduced static and dynamic embeddings are concatenated to obtain the initial head entity embedding and the initial tail entity embedding. The pre-trained language model is then used to encode the relations in the knowledge graph to obtain the relation embedding. S34. The initial head entity embedding and relation embedding are concatenated and deformed to obtain the first fused feature. The first fused feature is processed by convolution operation to obtain the query matrix, key matrix and value matrix. The attention weights of the query matrix and key matrix are calculated based on the cross attention mechanism. The attention weights and value matrix are weighted and aggregated and then superimposed on the original first fused feature to obtain the fused feature. S35. The fused features are concatenated and deformed with the initial tail entity embedding to obtain the second fused features. The second fused features are processed by convolution operation to obtain the query matrix, key matrix and value matrix. The attention weights of the query matrix and key matrix are calculated based on the cross attention mechanism. The attention weights and value matrix are weighted and aggregated and then superimposed on the original second fused features to obtain the enhanced tail entity embedding. S36. Guided by the joint decision loss function, optimization training is performed based on the enhanced tail entity embedding and the fused features. The spatial semantic information, spatial relationship information and multimodal data information of the multimodal geographic knowledge graph are embedded into the vector space to obtain the urban area embedding vector.

6. The urban region representation method supported by multimodal knowledge graph according to claim 5, characterized in that, The method of obtaining text embeddings for geographic entity types containing name attributes using a pre-trained language model includes: The geographic entity type containing name attributes is cleaned and segmented to obtain the segmentation results; The word segmentation results are fed into a pre-trained language model to obtain the latent vectors of each token; Layer normalization is performed on the latent vectors of each token to obtain the text embedding.

7. The urban region representation method supported by multimodal knowledge graph according to claim 5, characterized in that, The calculation of attention weights between the query matrix and the key matrix based on the cross-attention mechanism, and the weighted aggregation of the attention weights with the value matrix, includes: Calculate the similarity between the feature vectors of each position in the query matrix and the corresponding position in the key matrix in the row and column, and normalize them to obtain the attention weights; The attention weights are weighted and aggregated with the feature vectors in the corresponding rows and columns of the value matrix.

8. The urban region representation method supported by multimodal knowledge graph according to claim 5, characterized in that, S36 includes: S361. Based on the hierarchical knowledge embedding approach, the semantic hierarchical structure of geographic entities is explicitly modeled in the embedding space, and the hierarchical structure perception loss is constructed using the edge ranking loss form. S362. Based on the fused modal features in the enhanced tail entity embedding, a multimodal contrastive learning loss is constructed using the InfoNCE contrastive learning loss function; S363. Optimize training under the guidance of the joint decision loss function, use hierarchical structure perception loss to accurately model the hierarchical structure relationship of geographic entities, and use multimodal contrastive learning loss to promote the visual semantic consistency alignment between different modal data. Embed spatial semantic information, spatial relationship information and multimodal data information in the multimodal geographic knowledge graph into the vector space to obtain the city area embedding vector.

9. The urban region representation method supported by multimodal knowledge graph according to claim 8, characterized in that, The joint decision loss function is composed of a weighted sum of hierarchical perceptual loss and multimodal contrastive learning loss.

10. A city region representation system supported by a multimodal knowledge graph, used to implement the city region representation method supported by a multimodal knowledge graph as described in any one of claims 1-9, characterized in that, The system includes: The multimodal knowledge graph construction unit is used to divide the target research area into grid regions of a preset size based on the pre-constructed graph ontology, determine the multimodal geographic entities corresponding to the grid regions and the spatial relationships, subordinate relationships and similarity relationships between entities, and construct a multimodal geographic knowledge graph. The multi-source data feature fusion unit is used to perform multi-view adaptive fusion of street view image data based on multimodal geographic knowledge graph to obtain street view image features, extract remote sensing image features from remote sensing image data, and perform multi-scale adaptive fusion after aligning the remote sensing image features with the street view image features to obtain fused multi-scale features. The knowledge graph embedding learning unit is used to obtain static and dynamic embeddings based on multimodal geographic knowledge graphs. It then uses a cross-attention mechanism to perform two-stage information fusion of entity embeddings and relation embeddings to enhance tail entity embeddings. Under the guidance of a joint decision loss function, it optimizes the embedding of spatial location information, spatial relation information, and multimodal data information from the multimodal geographic knowledge graph into a vector space to obtain urban area embedding vectors. Among them, dynamic embedding includes fused multi-scale features. Downstream task application units are used to carry out land use classification, abnormal behavior prediction, housing price estimation, and urban village identification based on urban area embedding vectors.

Images