Urban spatial evolution prediction system fusing multi-modal data

Abstract

The invention relates to the field of city planning, and discloses a city spatial evolution prediction system fusing multi-modal data, and the system comprises a data preprocessing and multi-modal knowledge graph construction module which is used for configuring a knowledge graph ontology architecture and preprocessing multi-source city data; establishing a multi-modal knowledge graph based on the preprocessed multi-source city data and the knowledge graph ontology architecture; the model setting module is used for configuring a multi-modal representation learning model; the data integration module is used for a multi-modal representation learning model and performing deep integration on multi-modal data in a vector space to obtain vector representation with remote sensing and streetscape semantic information; and the knowledge reasoning and predicting module is used for performing knowledge reasoning on the missing part in the multi-modal knowledge graph based on the vector representation so as to predict the urban spatial evolution event. The urban spatial evolution event prediction can be realized, and the problem that the remote sensing image and the streetscape image are difficult to be deeply fused is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of urban planning, in particular to a city space evolution prediction system fusing multi-modal data. BACKGROUND

[0002] City space is a built-up regional space formed by the relative concentration of population, industrial economy and infrastructure, and is the place space of various social production and living activities and the regional carrier of urban landscape. City space evolution is the morphological remodeling, environmental change and functional transformation of city region in the process of urbanization. Urban expansion and urban renewal, as two key events of city space evolution, have a profound impact on and reconstruction of the city space pattern. Urban expansion (UE) refers to the conversion of non-constructed land such as farmland, forest and wasteland at the edge of the city into constructed land for business, education and industry. Urban renewal is the spatial reconstruction and functional optimization of the built-up area, and its fundamental purpose is to improve the urban environmental quality, improve the social and economic conditions, and improve the quality of life of residents, mainly including two forms of comprehensive reconstruction (CR) and urban rehabilitation and enhancement (URE). Among them, CR is to demolish and re-plan the original buildings in the city and fundamentally change the city space form and functional layout; URE is to maintain the basic stability of the existing city regional space pattern, and to renovate the buildings and improve the infrastructure to optimize the regional space construction. Prediction of these important city space evolution events can provide more global, strategic and forward-looking considerations for the formulation and deployment of urban planning, and help the sustainable development of cities.

[0003] Common methods for event prediction include System Dynamics (SD) and Artificial Intelligence (AI). SD integrates feedback control theory and numerical calculation models to describe and analyze the relationship between factors in complex systems and further predict the results of event changes. This method can represent complex urban systems as dynamic entities, capture nonlinear relationships through the definition of feedback loops of urban dynamics, and coordinate various factors within the urban system to predict the results of urban evolution events at the macro and micro levels. It has been widely used in disaster response, environmental monitoring, and urban development decision-making. However, SD models are highly dependent on theoretical construction for event prediction, and the assumed feedback loops and system structure require a large amount of empirical knowledge, making them knowledge-driven models. Therefore, SD can only achieve conditional prediction, and the results are subjective and easily affected by dynamic changes in the internal structure of the city system.

[0004] AI-based event prediction methods have strong high-dimensional, nonlinear, and complex data integration and processing capabilities, and can capture the complexity of urban space, making them effective data-driven methods. This data-driven approach can automatically learn urban evolution rules based on cases and make effective predictions, avoiding the limitations of subjective experience and effectively capturing the complex interactions between internal factors in the city, which makes up for the shortcomings of SD models in event prediction. Traditional machine learning models have strong classification ability and excellent generalization performance, and can learn to predict urban evolution events, such as support vector machines and random forests, which have been widely used in urban traffic management, environmental resources, and safety decision-making. At the same time, deep learning models such as Deep Neural Network (DNN), Long Short-Term Memory (LSTM), and Transformer can further integrate more complex multi-source heterogeneous data and achieve a deeper analysis of urban space, showing significant advantages in urban space prediction tasks. Although these models have shown strong predictive power, their inherent "black box" nature makes it difficult to understand the underlying principles, which is unacceptable in the field of urban planning that requires high transparency and high interpretability. In addition, regional differences between cities, uneven distribution of event labels between cities, and overfitting of AI models can all lead to insufficient generalization ability of existing methods. In summary, data-driven methods using AI for urban spatial evolution event prediction face the problems of poor interpretability and weak generalization ability.

[0005] Knowledge graph provides a new technical method for constructing objective, interpretable and transferable urban spatial evolution event prediction model, and is an effective data-knowledge joint model. Knowledge graph is essentially a semantic network, which represents the objective world through triples of "entity-relation-entity", and thus converts complex real-world information into a form that can be processed by a computer, supporting knowledge storage, query and reasoning. The knowledge graph reasoning model can reveal the deep knowledge and rules behind the data by constructing semantic associations between entities, and then be applied to prediction, classification and other downstream tasks. The event prediction method based on knowledge graph reasoning model has the advantage of strong interpretability, which represents the complex association between events and auxiliary data through triples, and reasons based on a large number of facts, and has been preliminarily applied to pollution monitoring, emergency response and traffic management. In addition, by combining the multi-source data and characteristics of the city, a general knowledge graph ontology can be designed to ensure that the reasoning model based on knowledge graph can extract a high-level knowledge framework from rich urban data, focusing on more general rules and patterns. Therefore, the knowledge graph reasoning model can effectively predict events even in the case of limited sample size, ensuring both the interpretability of the model and the good generalization ability of the model in different cities.

[0006] However, the traditional structured knowledge graph is still limited to text form, and the multi-modal information deep integration ability is weak, which greatly limits the representation ability of the real world. For example: in geographical studies, multi-modal data such as Street View Imagery (SVI), Remote Sensing Image (RSI) and social media text are often used to describe the social, economic and environmental conditions of urban space. Using only structured knowledge graph cannot capture the complex information of urban space, which is not conducive to the prediction of urban spatial evolution events. In order to overcome this problem, knowledge graphs such as UrbanKG and UrbanFloodKG attempt to combine satellite images, check-in data, POI and other multi-source data, not only retaining the original advantages of knowledge graph, but also enhancing the expression ability of complex urban space, and have been successfully applied to population flow prediction, traffic flow prediction, flood risk prediction, etc. However, the above knowledge graphs still remain in the simple storage and management of multi-modal data, and have not realized the deep fusion of multi-modal information in the vector space, resulting in insufficient multi-modal data mining, which to a large extent restricts the construction and application of knowledge graph reasoning model.

[0007] At present, there is no effective solution to the problems in the related art. SUMMARY

[0008] In view of the problems in the related art, the present application proposes a city space evolution prediction system fusing multi-modal data to overcome the above technical problems existing in the prior art.

[0009] To this end, the specific technical solutions adopted by the present application are as follows: A city space evolution prediction system fusing multi-modal data, comprising: A data preprocessing and multi-modal knowledge graph construction module is configured to configure a knowledge graph ontology architecture and preprocess multi-source city data; based on the preprocessed multi-source city data and the knowledge graph ontology architecture, a multi-modal knowledge graph is established; A model setting module is configured to configure a multi-modal representation learning model; A data integration module is configured to the multi-modal representation learning model to perform deep integration of multi-modal data in a vector space to obtain a vector representation with remote sensing and street view semantic information; A knowledge reasoning and prediction module is configured to perform knowledge reasoning on missing parts in the vector representation of the multi-modal knowledge graph to predict city space evolution events.

[0010] Further, the data preprocessing and multi-modal knowledge graph construction module comprises: A data configuration and knowledge graph ontology architecture configuration module is configured to construct a knowledge graph ontology architecture including entities, relationships and attributes; select entities related to city space evolution as the ontology of the knowledge graph, and store multi-modal data using the ontology of the knowledge graph; A data association module is configured to describe the association between entities using the ontology of the knowledge graph; The ontology architecture of the knowledge graph comprises entity categories and relationship categories, and the relationship categories include spatial relationships, subordinate relationships and cross-modal relationships.

[0011] Further, the data preprocessing and multi-modal knowledge graph construction module further comprises: A remote sensing image data processing module is configured to obtain remote sensing image data corresponding to each grid cell; label the remote sensing image data, and identify land cover entities including buildings, water bodies, sports fields, vegetation and bare land based on a target detection model; analyze the spatial relationships between the land cover entities using a multi-modal language model to extract triple knowledge and generate a remote sensing scene knowledge graph as a subgraph of the multi-modal knowledge graph; A street view image and point of interest data processing module is configured to obtain street view images of each grid cell based on the principle of complementing remote sensing images; obtain point of interest data and summarize them into several categories; A road network data processing module is configured to determine road information of each grid cell in the road network data, obtain the relationship between the grid cell and the road, and calculate the geographic spatial attributes of each grid cell.

[0012] Further, the model setting module comprises: a model selection module, configured to select a TransH model as the multi-modal representation learning model; a model running module, configured to convert the relation into a hyperplane in a vector space by the TransH model, and map the head entity and the tail entity into the hyperplane; and perform translation calculation between the entity and the relation in the hyperplane to realize embedding of the knowledge graph.

[0013] Further, the mapping calculation formula for mapping the head entity and the tail entity into the hyperplane is: ; ; The scoring function for performing translation calculation between the entity and the relation in the hyperplane is: ; In the formula, h , r , t are the head entity, relation and tail entity vectors in the triple; , are h and r vectors after mapping into the hyperplane, is r a unit normal vector of the corresponding hyperplane, is h a projection length in the direction, is the scoring function.

[0014] Further, the data integration module comprises: a feature fusion module, configured to fuse the remote sensing image feature vector and the street view image feature vector in the multi-modal knowledge graph by using a deep learning model and a full connection layer to obtain a fused multi-modal vector; a model training and vector representation form conversion module, configured to use the fused multi-modal vector as an entity vector, combine the entity and the relation of the multi-modal knowledge graph as input data, and train and optimize in the TransH model to map all entities and relations in the multi-modal knowledge graph into a feature space and convert into a vector representation form with remote sensing and street view semantic information.

[0015] Further, the feature fusion module comprises: a feature vector extraction module, configured to extract the remote sensing image feature vector and the street view image feature vector corresponding to each grid region by using a deep learning model; The feature vector splicing and fusion module is configured to splice the remote sensing image feature vector and the street view image feature vector, and reduce the dimension of the splicing result through a full connection layer to obtain a fused multi-modal vector.

[0016] Further, the knowledge reasoning and prediction module comprises: The regional event acquisition module is configured to take the existing regional entity vector, the semantic relationship vector and the different event entity vector as input data, calculate a scoring function of translation between entities and relationships in a hyperplane, and calculate a result with the lowest score as an event corresponding to the region. The prediction module is configured to perform intra-city prediction and cross-city prediction.

[0017] Further, the prediction module comprises: The target region acquisition module is configured to acquire a region in which a spatial evolution event has been observed. The intra-city potential evolution prediction module is configured to utilize the multi-modal representation learning model and predict potential evolution of other regions in the same city according to the region in which the spatial evolution event has been observed.

[0018] Further, the prediction module further comprises: The city grid region data acquisition module is configured to acquire grid region data of different cities that have experienced urban spatial evolution. The inter-city potential evolution prediction module is configured to utilize the grid region data of different cities that have experienced urban spatial evolution, combine similarities and differences between different cities, and predict potential evolution of the remaining regions of the cities.

[0019] The present application has the following advantages: (1) The present application considers that the existing urban spatial evolution event prediction technology faces challenges such as insufficient explainability and transferability, and provides an urban spatial evolution multi-modal knowledge graph (Urban Spatial Evolution Knowledge Graph, USEKG) combining remote sensing images, street view images, point of interest data, road networks and other multi-source urban geographic data, which realizes the fusion of multi-modal data in the structural space.

[0020] (2) The present application provides a new multi-modal representation learning algorithm (Urban Representation learning, ReUrbanRL) for USEKG, which realizes deep fusion between remote sensing image data and street view image data based on an attention mechanism, realizes deep fusion between multi-modal data in a vector space, and carries out prediction of urban spatial evolution events. BRIEF DESCRIPTION OF DRAWINGS

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a partial illustration of multi-source data used according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the RSI knowledge extraction process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an urban spatial evolution prediction framework according to an embodiment of the present invention; Figure 4 This is a diagram of the USEKG body according to an embodiment of the present invention; Figure 5 This is a schematic diagram of an UrbanRL knowledge graph representation learning framework according to an embodiment of the present invention; Figure 6 This is a schematic diagram of urban spatial evolution event prediction based on a USekG embodiment of the present invention; Figure 7 This is a map showing the predicted results within a city according to an embodiment of the present invention; Figure 8 This is a block diagram of an urban spatial evolution prediction system that integrates multimodal data according to an embodiment of the present invention. Detailed Implementation

[0023] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0024] According to an embodiment of the present invention, a system for predicting urban spatial evolution by fusing multimodal data is provided.

[0025] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 8 As shown, the urban spatial evolution prediction system that integrates multimodal data according to an embodiment of the present invention includes: The data preprocessing and multimodal knowledge graph construction module is used to configure the knowledge graph ontology architecture and preprocess multi-source city data; based on the preprocessed multi-source city data and the knowledge graph ontology architecture, a multimodal knowledge graph is built.

[0026] a model setting module configured to configure a multi-modal representation learning model.

[0027] a data integration module configured to perform deep integration of multi-modal data in a vector space by using the multi-modal representation learning model to obtain vector representations with remote sensing and street view semantic information.

[0028] a knowledge reasoning and prediction module configured to perform knowledge reasoning on missing parts in a multi-modal knowledge graph by using the vector representations to predict urban spatial evolution events.

[0029] In an embodiment, the data preprocessing and multi-modal knowledge graph construction module comprises: a data configuration and knowledge graph ontology architecture configuration module configured to construct a knowledge graph ontology architecture including entities, relationships and attributes; select entities related to urban spatial evolution as the ontology of the knowledge graph, and store multi-modal data by using the ontology of the knowledge graph.

[0030] a data association module configured to describe associations between entities by using the ontology of the knowledge graph.

[0031] The ontology architecture of the knowledge graph comprises entity categories and relationship categories, and the relationship categories include spatial relationships, subordinate relationships and cross-modal relationships.

[0032] In an embodiment, the data preprocessing and multi-modal knowledge graph construction module further comprises: a remote sensing image data processing module configured to obtain remote sensing image data corresponding to each grid cell; label the remote sensing image data, and identify land cover entities including buildings, water bodies, sports fields, vegetation and bare land by using a target detection model; analyze spatial relationships between the land cover entities by using a multi-modal language model to extract triple knowledge, and generate a remote sensing scene knowledge graph as a subgraph of the multi-modal knowledge graph.

[0033] a street view image and point of interest data processing module configured to obtain street view images of each grid cell based on the principle of complementarity with remote sensing images; obtain point of interest data and summarize the point of interest data into several categories.

[0034] a road network data processing module configured to determine road information of each grid cell in road network data, obtain relationships between the grid cells and the roads, and calculate geographical spatial attributes of each grid cell.

[0035] In an embodiment, the model setting module comprises: a model selection module configured to select a TransH model as the multi-modal representation learning model.

[0036] The model running module is configured to convert the relation into a hyperplane in a vector space by using a TransH model, and map the head entity and the tail entity into the hyperplane; and perform translation calculation between the entity and the relation in the hyperplane to realize embedding of the knowledge graph.

[0037] In one embodiment, the mapping calculation formula for mapping the head entity and the tail entity into the hyperplane is: ; ; The scoring function for the translation calculation between the entity and the relation in the hyperplane is: ; In the formula, h 、 r 、 t are vectors of the head entity, the relation and the tail entity in the triple respectively; 、 are vectors of the head entity and the tail entity after mapping into the hyperplane respectively, h and r are vectors of the head entity and the tail entity after mapping into the hyperplane respectively, is a unit normal vector of the hyperplane, r is a projection length in the direction of , h is the scoring function.

[0038] In one embodiment, the data integration module comprises: The feature fusion module is configured to fuse the remote sensing image feature vector and the street view image feature vector in the multi-modal knowledge graph by using a deep learning model and a full connection layer to obtain a fused multi-modal vector.

[0039] The model training and vector representation form conversion module is configured to train and optimize the fused multi-modal vector as an entity vector in the TransH model in combination with the entity and the relation of the multi-modal knowledge graph as input data, so as to map all entities and relations in the multi-modal knowledge graph into a feature space and convert them into a vector representation form with remote sensing and street view semantic information.

[0040] In one embodiment, the feature fusion module comprises: The feature vector extraction module is configured to extract the remote sensing image feature vector and the street view image feature vector corresponding to each grid region by using a deep learning model.

[0041] The feature vector splicing and fusion module is configured to splice the remote sensing image feature vector and the street view image feature vector, and reduce the dimension of the spliced result by using a full connection layer to obtain the fused multi-modal vector. ​​

[0042] In one embodiment, the knowledge reasoning and prediction module comprises: The regional event acquisition module is configured to take the existing regional entity vector, the semantic relationship vector and the different event entity vector as input data, calculate a scoring function through translation calculation between entities and relationships in a hyperplane, and obtain a result with the lowest score as a corresponding event of the region.

[0043] The prediction module is configured to perform intra-city prediction and cross-city prediction.

[0044] In one embodiment, the prediction module comprises: The target region acquisition module is configured to acquire a region in which a spatial evolution event has been observed.

[0045] The intra-city potential evolution prediction module is configured to utilize a multi-modal representation learning model and predict potential evolution of other regions in the same city according to the region in which the spatial evolution event has been observed.

[0046] In one embodiment, the prediction module further comprises: The city grid region data acquisition module is configured to acquire grid region data of regions that have experienced city spatial evolution in different cities.

[0047] The inter-city potential evolution prediction module is configured to utilize the grid region data of the regions that have experienced city spatial evolution, combine similarities and differences between different cities, and predict potential evolution of the remaining regions of the cities.

[0048] In order to facilitate understanding of the above technical solutions of the present application, the working principle of the present application in the actual process will be described in detail below.

[0049] I. Urban spatial evolution is the morphological remodeling, environmental change and functional transformation of urban areas during the urbanization process, including important events such as urban expansion and urban renewal. Existing urban spatial evolution prediction models face the problem of insufficient deep fusion of multi-modal data. Therefore, the present application realizes accurate prediction of urban spatial evolution events by reasoning through a multi-modal knowledge graph and deeply fusing multi-modal data. A multi-modal knowledge graph USEKG (knowledge graph constructed by fusing multiple data, data fusion in structural space) is constructed to cover street view images, remote sensing images, points of interest, road networks and other data, and to realize the fusion of multi-modal data in structural space. A representation learning model for multi-modal knowledge graphs is designed to embed the entities and relationships of USEKG in a vector space to realize deep fusion of multi-modal data in the vector space. Based on the representation learning results and knowledge reasoning rules, urban spatial evolution event prediction within and across cities is realized, including city repair and improvement, comprehensive reconstruction and urban expansion events, and excellent results are achieved with F1 greater than 70.8. Based on the knowledge graph vector, urban spatial evolution event prediction is carried out. The present application can realize high-precision, interpretable and transferable urban spatial evolution event prediction, and effectively solve the problem of deep fusion of remote sensing images RSI and street view images SVI, providing scientific reference for urban planners.

[0050] II. Research area and data 2.1 Research area: The present application selects the jurisdiction of City One and the jurisdiction of City Two as the research area, as shown in Figure 1 However, with the development of cities, these areas also face many challenges such as unreasonable land use, crowded old communities, and difficult industrial transformation. In the face of these difficulties, how to effectively promote urban transformation and achieve sustainable development has become a key problem to be solved. These two urban areas exhibit certain differences in positioning, policy, economy and appearance, and predicting spatial evolution events in them not only tests whether the prediction technology proposed by the present application is suitable for different urban scenarios, but also explores the feasibility of knowledge transfer. In order to achieve this goal, the two research areas are divided into grid cells with a side length of 300 meters to ensure that each grid area has sufficient data. All grid cells in the research area in 2017 are taken as the research object, and it is predicted whether each grid cell will undergo URE (micro-renovation type urban renewal), CR (demolition type urban renewal) or UE (urban expansion) in the future, or remain unchanged (UC, no urban spatial evolution event).

[0051] 2.2, Data preparation and preprocessing: With the rapid development of earth observation technology and computer information technology, geospatial big data is growing exponentially. Combining cross-modal data from different sources for urban analysis and calculation has gradually become a mainstream trend. Therefore, the following data were selected for further analysis, as shown in Figure 1 .

[0052] (1) Remote sensing image (RSI): RSI is a direct result of earth observation, with rich spatial texture features and semantic information, and is of great significance in representing geographical environment. It can be used for analysis of urban structure, form, and functional layout. RSI with a resolution of 0.5 m corresponding to the study area in 2017 was obtained, and according to the grid units after segmentation, the RSI corresponding to each grid unit was obtained.

[0053] RSI not only contains numerous object entities, but also contains relationships between objects. In order to make the knowledge graph constructed contain more geographical knowledge, a triple extraction scheme for RSI was proposed, including two steps of target detection and relationship extraction, as shown in Figure 3 . The acquired grid-scale remote sensing image was manually annotated, and on this basis, the Faster-RCNN target detection model was used to identify five types of land cover entities including Building (building), Water (water), Playground (playground), Vegetation (vegetation), and Barren (bare land). Using a multi-modal large model, the spatial relationships between entities detected in RSI were analyzed by setting the prompt words, including NorthOf (north of), SouthOf (south of), WestOf (west of), EastOf (east of), and AdjacentOf (adjacent), to extract triple knowledge and generate a remote sensing scene knowledge graph as a subgraph of the USEKG constructed later, realizing the leap from data to knowledge, as shown in Figure 2 .

[0054] (2) Street view image (SVI): SVI, as a data representing urban environment, can capture the most frequent urban scenes of building environment and pedestrian interaction, providing a close-up, three-dimensional perspective to depict the urban physical space environment, and can support urban development planning and complement RSI. The research downloaded the SVI data corresponding to each grid unit in the study area in 2017 by calling the street view API.

[0055] (3) Point of interest (POI): POI refers to a collection of data of specific locations or places with special significance or attractions, usually covering important entities such as businesses, attractions, schools, etc. within a specific geographic area, which can help understand the city's geographical environment and reflect social economy and urban vitality. POI data of the study area in 2017 is obtained, and POI has been summarized into 23 categories, which can reflect the functions of business, public service, residence and transportation in the urban area.

[0056] (4) Road network (RN): RN is the bridge between urban functional land, and the accessibility of RN can not only affect the layout of urban functional land, but also has a close relationship with the value of urban land. Therefore, RN has a profound impact on social development, economic growth, urban resilience and human activities. After obtaining the required RN data, based on geographic information systems such as ArcGIS, it is determined whether each grid unit has a road start point, end point or whether a road passes through the study unit, and the relationship between the grid unit and the road is obtained. In addition, the average population density, average building height, average annual temperature and other data of each grid area are calculated as geographic spatial attributes to improve the reliability of the prediction results of the city space evolution event.

[0057] Thirdly, the application discloses a framework for predicting city space evolution based on knowledge graph reasoning technology combined with multi-source data, as shown in Figure 4 Generally, the research framework can be divided into three steps. (1) By designing the ontology architecture of the knowledge graph and integrating the pre-processed multi-source city data, USEKG is constructed, and the fusion of multi-modal data is realized in structure. (2) By constructing a multi-modal representation learning model UrbanRL, the deep integration of multi-modal data is realized in vector space, and the embedding vector of knowledge graph entity and relationship is obtained. (3) Based on the results of representation learning, knowledge reasoning is carried out to realize the prediction of city space evolution event, and the feasibility of the prediction results of the two research areas and the cross-city space evolution prediction through knowledge transfer between the research areas is evaluated, as shown in Figure 3 3.1 Construction of multi-modal knowledge graph, realizing fusion of multi-modal data in structural space: In order to construct the city space evolution knowledge graph, it is necessary to design its ontology first. Ontology is a formal representation of knowledge, covering three elements of entity, relationship and attribute. When designing the ontology, the following principles need to be considered: (1) The ontology of the knowledge graph needs to consider as many entities related to city space evolution as possible, and can store multi-modal data.

[0058] (2) The ontology of the knowledge graph needs to cover as many relationships as possible to fully describe the association between entities.

[0059] ​Based on this, the ontology architecture of USEKG is constructed, as shown in Figure 4 It can be seen that this ontology divides 10 entity categories in urban domain and 17 relationship categories including spatial relationship, subsumption and cross-modal relationship, which can describe different categories of entities and their relationships in detail, and constitutes a comprehensive knowledge framework. In addition, rich attribute descriptions are defined for each entity to enhance the interpretability of reasoning, for example, Region has geographical spatial attributes such as average population density, average building height, and average annual temperature. The specific entity, relationship and attribute definitions are shown in Table 5-Table 6.

[0060] Table 1 Ontology information of USEKG knowledge graph USEKG contains 385281 entity nodes and 744526 triples in the study area, as shown in Table 1, which can comprehensively and richly represent the urban space and associate with the urban space evolution events, laying a foundation for subsequent prediction work.

[0061] 3.2, representation learning method, realize the fusion of multi-modal data in vector space: Representation learning of knowledge graph is to embed entities and their relationships in a continuous, low-dimensional vector space, capturing the implicit relationships between nodes and preparing for downstream applications. Existing representation learning methods cannot effectively fuse multi-modal data. Therefore, a new multi-modal representation learning method UrbanRL is proposed, as shown in Figure 5

[0062] The pre-trained ResNet50 model with removed fully connected layer is used as the feature extractor of RSI and SVI, and the mean value of all SVI feature vectors corresponding to each grid region is obtained to get the feature vectors of RSI and SVI and . The feature vectors of RSI and SVI are spliced and reduced through fully connected layer. Through this process, the two kinds of data in image modality are effectively fused, achieving information complementation while greatly reducing the computational complexity, and preparing for the following translation work.

[0063] ​TransE is a commonly used structured knowledge graph representation learning model, and its basic idea is to regard the relationship as the translation between entities, that is, the sum of the head entity and the relationship vector of the triple is close to the tail entity vector. This model has simple structure and high running efficiency, and has become the mainstream technology for studying representation learning. However, TransE is suitable for 1-1 relationships in knowledge graphs, and it is difficult to accurately represent relatively complex node relationships such as 1-N, N-1 and N-N. Considering that USEKG covers a large number of complex node relationships, selecting TransE to implement knowledge graph embedding is not an ideal choice. TransH can effectively solve this problem. TransH algorithm is an extension of TransE algorithm, which solves the shortcomings of TransE in handling many-to-one, one-to-many and many-to-many relationships. TransH maps entities and relationships to a hyperplane by introducing the concept of hyperplane and projection, so that the same entity can have different representations on the hyperplane under different relationships, thereby more accurately mining implicit relationships. TransH abstracts each relationship as a hyperplane in vector space, and maps the head entity and the tail entity to the hyperplane of the relationship (formula 1-2), and performs translation calculation between entities and relationships similar to TransE in this hyperplane (formula 3), thereby learning complex relationships and realizing the embedding of knowledge graphs.

[0064] (1) (2) (3) wherein, h , r , t correspond to the head entity, relationship and tail entity vectors in the triple, , are the vectors of h and r after mapping to the hyperplane, is the unit normal vector of r corresponding to the hyperplane, is the projection length of h in the direction, is the scoring function, and the model should try to reduce the size of the function calculation result for correct triples during training. The fused multi-modal vector is approximated as an entity vector, which is used as input data together with the entities and relationships of the structured knowledge graph in the TransH model for training and optimization. Finally, all entities and relationships in USEKG are mapped to a unified feature space and converted into vector representation forms with rich semantic information, preparing for knowledge reasoning.

[0065] 3.3, Urban space evolution event prediction: Based on the representation learning model, the missing part of the triple in the knowledge graph can be inferred and completed. In USEKG, it is explicitly specified that each Region entity in the knowledge graph will only experience one of the following four events: URE, CR, UE and UC, where URE, CR and UE are urban spatial evolution events. Therefore, predicting urban spatial evolution for each region can be converted into a link prediction task, that is, finding the unique potential Event entity that satisfies the triple (Event, OccurIn, Region). After completing the representation learning to obtain the entity and relationship vectors of the knowledge graph, the existing Region entity vector (region entity vector), OccurIn relationship vector (semantic relationship vector) and different Event entity vector (event entity vector) can be used as input data. The result with the lowest score calculated by formula (3) is the event that occurs in the region. Through the above calculation idea, two kinds of urban spatial evolution events can be predicted: urban internal prediction and cross-city prediction, as shown in Figure 6 .

[0066] (1) In terms of urban internal prediction, city one and city two are taken as the research objects. The model is based on the regions that have observed spatial evolution events to predict the potential evolution of other regions in the same city. By analyzing the existing evolution pattern, this task can provide prediction reference for the future development trajectory of each region in the city.

[0067] (2) Cross-city prediction predicts the potential evolution of the remaining regions in the research area of city two by using the grid region data of the research area of city one and the part of city two that has experienced urban spatial evolution. Not only the internal evolution law of a single city is considered, but also the similarity and difference between different cities are combined to explore the generalization ability of the model.

[0068] Four, experimental results and urban internal event prediction: 4.1, experimental setting: Embedding is performed on the constructed USEKG. In the experiment, the embedding dimension of the knowledge graph is set to 100, the learning rate is set to 0.01, and the early stopping strategy is adopted for training, that is, when the training loss no longer decreases within 10 iterations, the training is stopped. In the Intra-City Event Prediction task, the study divides the (Event, OccurIn, Region) triples of the city space evolution knowledge graph of the city one and city two research areas according to the proportion of 7:1:2 to form the training set, the validation set and the test set, and trains and evaluates the prediction results together with the remaining data in the knowledge graph. In the Cross-City Event Prediction task, all data of the city one research area in the USEKG and different proportions of the city two research area data of the spatial evolution event are selected, and the remaining city two research area grid cells are predicted for the city space evolution event and the results are evaluated. Accuracy, Precision, Recall and F1 are selected to evaluate the prediction results. Considering the imbalance of the sample quantity, the macro-averaging method is used for calculation. Through the experiment, it can be obtained that: (1) In the city space evolution event prediction task, the proposed model is more advanced than the traditional prediction model.

[0069] (2) The prediction model based on the knowledge graph shows excellent transferability.

[0070] (3) USEKG has advantages in the prediction of spatial evolution events by fusing multi-source city data.

[0071] 4.2, Intra-City Event Prediction Results: TransE, TransH and TuckER structured knowledge graph representation learning models and PoE and RSME multi-modal knowledge graph representation learning models are selected to predict the city space evolution event in the same way, and compared with the present application, as shown in Table 2.

[0072] From the experimental results, it can be seen that the prediction results of UrbanRL are better than those of the previous representation learning models in four indicators, proving that the strategy of integrating RSI and SVI data helps the effect of representation learning. At the same time, the multi-modal representation learning technology is generally significantly better than the single-modal representation learning technology, which shows that it is necessary to integrate multi-source data into the knowledge graph for the prediction of urban spatial evolution. In addition, the knowledge graph-based urban spatial evolution event prediction model has excellent prediction results in both City One and City Two, although the results differ, which may be influenced by factors such as policy positioning, historical background, and geographical location of the two cities' study areas. Therefore, this prediction model has excellent robustness and can be extended to other cities in the future to conduct suitability assessment for urban construction and support intelligent decision-making for urban planners. On this basis, the areas in City One and City Two where spatial evolution events have not yet occurred are predicted, as shown in Table 4, to assess the future construction of the city and support intelligent decision-making for urban managers. Figure 7

[0073] Table 2. Prediction result evaluation based on different representation learning models 4.3. Cross-city event prediction results: In the experiment, based on the data of City One and different proportions of data of City Two in the knowledge graph, the prediction of urban spatial evolution events in the remaining study areas of City Two where spatial evolution events have occurred is conducted, as shown in Table 3.

[0074] ​From the results, when only using the urban space evolution data of City One research area to predict City Two research area, the F1 of UE and UC is high, but the F1 of URE and CR is relatively low, which shows that due to the different urban positioning and urbanization basis of City One and City Two, the implementation of urban renewal is different. For example, the URE of City Two involves more protection of historical buildings, while City One focuses on the transformation of commercial areas. When 10% of the data of City Two research area is added, all indicators have relatively improved. When 30% of the data of City Two research area is added, the F1 of URE and CR has a significant increase compared to before, which proves that although there are some differences in the policy, implementation and requirements of urban renewal between City One research area and City Two, on the basis of the urban space evolution mode of City One, combined with sufficient urban space evolution event cases of City Two, it can help the proposed model to effectively learn and predict other areas of City Two. After adding 50% of the data of City Two research area, the prediction results of the model are still rising but relatively low. Because there are many differences between City One and City Two in many aspects, it is difficult for the prediction model based on knowledge graph to further significantly improve the performance on the basis of a large amount of data of City One. Compared with event prediction based only on the data of City Two research area, there is still a certain gap. Overall, the proposed urban space evolution prediction model can realize effective prediction across cities and can be used as an evaluation tool to provide reference for other cities to learn from the urban construction experience of City One or City Two for urbanization construction.

[0075] Table 3 Cross-city prediction results 4.4, ablation experiment results of different data sources: In the experiment, RSI, SVI, POI and RN were removed respectively, and the original results were compared to evaluate the influence of these data on urban space evolution prediction. The region also selected City One. As shown in Table 4.

[0076] Table 4 Ablation experiment results From the experimental results, all data have a certain influence on the prediction results of urban spatial evolution events, which shows that it is essential to combine multi-source geographic data for prediction. RSI contributes the most to the prediction results, followed by POI data and SVI, and RN has the least influence on the prediction results of urban spatial evolution events. In addition, RSI has a greater impact on the prediction results of UE than other events, because UE is more closely related to urban morphology, while the areas where URE and CR occur are often similar in urban appearance. SVI and POI have a greater impact on the prediction results of URE and CR than UE and UC, which is also in line with the actual situation that different urban renewal measures often need to be combined with street environment, regional function and urban vitality for specific implementation.

[0077] V. CONCLUSION Existing urban spatial evolution event prediction models often have poor interpretability and generalization ability, and cannot effectively support the evaluation and decision-making of urban construction. Therefore, based on multi-source urban geographic data and knowledge graph reasoning algorithm, the important events of urban spatial evolution are predicted. In view of the characteristics of urban spatial evolution events and multi-source big data, a multi-modal knowledge graph USEKG is constructed. The UrbanRL model is designed, and the embedding of USEKG entities and relationships is realized by training on urban data. Based on the learned representation results and knowledge reasoning rules, the prediction of urban spatial evolution events within and across cities is realized, including urban micro-update, demolition and construction type urban renewal, and urban expansion events, and excellent results (Accuracy is greater than 71.0) are achieved. The present application can intuitively express the relationship between different data through the knowledge graph, and reason based on a large number of regional spatial evolution event cases, which can meet the requirements of accuracy, interpretability and transferability of urban spatial evolution event prediction, and provide valuable reference for urban planners.

[0078] The present application considers that the existing urban spatial evolution event prediction technology faces challenges such as lack of interpretability and transferability, and designs a multi-modal knowledge graph USEKG that combines remote sensing images, street view images, point of interest data, road networks and other multi-source urban geographic data. The fusion of multi-modal data is realized in the structural space, and the research areas of two cities are taken as examples to carry out application analysis.

[0079] Table 5 Entity and attribute definition of USEKG Table 6 Relationship definition of USEKG In view of the problem that the knowledge graph in the city field cannot effectively integrate multi-modal information for reasoning, a new multi-modal representation learning technology UrbanRL for USEKG is provided, deep fusion between RSI data and SVI data is realized based on an attention mechanism, deep fusion between multi-modal data is realized in a vector space, and prediction of city space evolution events is carried out. The present application can realize high-precision, interpretable and transferable prediction of city space evolution events, and effectively solves the problem that remote sensing images and street view images are difficult to be deeply fused, thereby providing a scientific reference for city planners.

[0080] The above merely describes preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fusion multi-modal data urban space evolution prediction system, characterized in that, The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. 2.The urban spatial evolution prediction system of fusing multi-modal data according to claim 1, wherein, The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. 3.The urban spatial evolution prediction system of fusing multi-modal data according to claim 1, wherein, The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction.

4. The system for urban spatial evolution prediction fusing multi-modal data according to claim 1, wherein, The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction.

5. The system for fusing multi-modal data for urban space evolution prediction according to claim 4, wherein, The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. ； ； The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. ； In the formula, h , r , t are head entity, relation and tail entity vectors in a triple, respectively. , yes h and r The vector after mapping to the hyperplane yes r The unit normal vector corresponding to the hyperplane, yes h exist Projected length in the direction, It is a scoring function.

6. The system for urban spatial evolution prediction fusing multi-modal data according to claim 1, wherein, The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban spatial evolution prediction. The application relates to a multi-modal knowledge graph construction method and device for urban The model training and vector representation conversion module is configured to train and optimize the TransH model by taking the fused multi-modal vector as an entity vector, and combining entities and relations of the multi-modal knowledge graph as input data, so as to map all entities and relations in the multi-modal knowledge graph to a feature space and convert them into a vector representation with remote sensing and street view semantic information.

7. The system for fusing multi-modal data for urban space evolution prediction according to claim 6, wherein, The feature fusion module comprises: The feature vector extraction module is configured to extract a remote sensing image feature vector and a street view image feature vector corresponding to each grid region by using a deep learning model; The feature vector splicing and fusion module is configured to splice the remote sensing image feature vector and the street view image feature vector, and reduce the dimension of the spliced result by a full connection layer to obtain a fused multi-modal vector.

8. The system for fusing multi-modal data for urban space evolution prediction according to claim 5, wherein, The knowledge reasoning and prediction module comprises: The regional event acquisition module is configured to take the existing regional entity vector, semantic relation vector and different event entity vector as input data, calculate a scoring function by performing translation calculation between entities and relations in a hyperplane, and obtain a result with the lowest score as an event occurring in a region. The prediction module is configured to perform intra-city prediction and cross-city prediction.

9. The system for fusing multi-modal data for urban space evolution prediction according to claim 8, wherein, The prediction module comprises: The target region acquisition module is configured to acquire a region in which a spatial evolution event has been observed. The intra-city potential evolution prediction module is configured to utilize a multi-modal representation learning model, and predict potential evolution of other regions in the same city according to the region in which the spatial evolution event has been observed.

10. The system for predicting urban spatial evolution by fusing multi-modal data according to claim 8, wherein, The prediction module further comprises: The city grid region data acquisition module is configured to acquire grid region data of regions that have experienced city spatial evolution in different cities. The inter-city potential evolution prediction module is configured to utilize the grid region data of the regions that have experienced city spatial evolution, combine similarities and differences between different cities, and predict potential evolution of remaining regions of the cities.

Images