A mass land space vector data management method and system

Abstract

The application relates to a mass land space vector data management method and system, which comprises the following steps: performing identification processing on land space data to be stored to obtain space data and vector data, and establishing a first mapping and association relationship between the space data and the vector data; and storing the space data and the vector data separately, wherein the space data is stored in a space database, and the vector data is stored in an attribute database. The method has the advantages of effectively reducing the number of table objects in the database during land space data processing, reducing the difficulty of database operation and maintenance management, and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of national space data processing, and particularly relates to a massive national space vector data management method and system. BACKGROUND

[0002] In the national space data processing, the data sources are various (such as shp, gdb, GeoJSON, etc.), and the data processing amount is very large. Figure 1 As shown in the figure, the national space data can be generally divided into vector data, raster data and three-dimensional data, wherein the vector data contains spatial data and attribute data. In the prior art, the processing of the national space data mainly relies on external software such as ArcGIS or SuperMap, and the data import management is performed based on manual operation, and the storage is performed by using a relational database. Generally, the vector data is uniformly stored in the relational database by using the geometry field type provided by the relational database, and the spatial data and the attribute data in the vector data are simultaneously stored in the spatial database. When this processing mode is used, if the vector data contains a large number of geographic targets, the number of tables formed by using the relational database for storage will be extremely expanded. For example, if the imported file contains multiple layers (such as a GDB data file), multiple data tables will be generated in the spatial database, and each data table corresponds to a layer. Therefore, the number of tables in the spatial database is extremely expanded, which seriously increases the difficulty of operation and maintenance management of the database administrator. Moreover, due to the limitation of the index retrieval mechanism of the relational database, the storage and query of the vector data are very time-consuming and inefficient, which is not conducive to the query and maintenance of the system.

[0003] Moreover, in the traditional spatial database, a single storage mode cannot meet the application requirements of high concurrency and even high timeliness of big data. In the Internet application scenario, the document database is used to solve the dynamic expansion problem of the data column, which has been applied. However, the performance of the spatial data storage, query and spatial analysis is still relatively weak, which is not enough to support high-performance complex spatial analysis. Both the traditional relational database and the document database need to be based on index to perform query acceleration, and the index establishment models of the two modes depend on fixed data columns, and have poor support capability for dynamic columns, which is not suitable for data mining of the national space vector data and efficient retrieval of massive attribute information in the big data scenario. SUMMARY

[0004] The technical problem to be solved by the present application is that, in view of the technical problems existing in the prior art, the present application provides a massive national space vector data management method and system which can effectively reduce the number of table objects in the database in the processing of the national space vector data, and reduce the difficulty of database operation and maintenance management.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by the present invention is as follows: a method for managing massive land spatial vector data, wherein the land spatial vector data to be stored is identified and processed to obtain spatial data and attribute data, and a first mapping relationship is established between the spatial data and the attribute data; the spatial data and the attribute data are stored separately, the spatial data is stored in a spatial database, and the attribute data is stored in an attribute database.

[0006] Furthermore, the identification process includes: identifying the land spatial vector data to be stored to obtain spatial data and attribute data; converting the spatial data to obtain spatial data with a unified format type; parsing the attribute data into attribute list data and attribute column data, and establishing a second mapping relationship between the attribute list data and the attribute column data.

[0007] Furthermore, it also includes storing the attribute list data and the attribute column metadata in a key-value separation manner.

[0008] Furthermore, it also includes serializing the attribute list data and connecting the serialized attribute list data to the search engine.

[0009] Furthermore, it also includes: retrieving the spatial data and obtaining the corresponding attribute data based on the first mapping relationship;

[0010] or:

[0011] The attribute list data is retrieved, and the corresponding spatial data is obtained according to the first mapping relationship, and the corresponding attribute column data is obtained according to the second mapping relationship.

[0012] A massive land spatial vector data management system includes a data storage module: used to identify and process the land spatial vector data to be stored to obtain spatial data and attribute data, and establish a first mapping relationship between the spatial data and the attribute data; the spatial data and the attribute data are stored separately, with the spatial data stored in a spatial database and the attribute data stored in an attribute database.

[0013] Furthermore, the identification process in the data storage module includes: identifying the land spatial vector data to be stored to obtain spatial data and attribute data; converting the spatial data to obtain spatial data with a unified format type; parsing the attribute data into attribute list data and attribute column data, and establishing a second mapping relationship between the attribute list data and the attribute column data.

[0014] Furthermore, it also includes storing the attribute list data and the attribute column metadata in a key-value separation manner.

[0015] Furthermore, it also includes serializing the attribute list data and connecting the serialized attribute list data to the search engine.

[0016] Furthermore, it also includes a data retrieval module: used to retrieve the spatial data and obtain the corresponding attribute data according to the first mapping relationship;

[0017] or:

[0018] The attribute list data is retrieved, and the corresponding spatial data is obtained according to the first mapping relationship, and the corresponding attribute column data is obtained according to the second mapping relationship.

[0019] Compared with the prior art, the advantages of the present invention are as follows:

[0020] 1. This invention identifies and parses land spatial data into spatial data and attribute data, and stores the spatial data and attribute data separately. This can effectively reduce the number of table objects in the database, reduce the complexity of the database, and also reduce the difficulty of operation and maintenance management of the spatial database.

[0021] 2. This invention separates and stores spatial data and attribute data, while establishing a mapping relationship between the two. This allows for convenient querying and retrieval based on either spatial data or attribute data, and the corresponding attribute data or spatial data can be retrieved through the mapping relationship. This fully utilizes the spatial analysis and processing capabilities of the spatial database, significantly improving the query performance of spatial database attribute information. It also fully utilizes the search processing capabilities of the search engine after the attribute data is serialized, improving the query performance of vector data and the mining and processing capabilities of vector data.

[0022] 3. This invention solves the problem of dynamic management of attribute fields in spatial attribute tables and the need for fixed data columns in relational databases by sharing spatial data and attribute data and storing attribute data in an attribute database that uses a document-oriented database. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the composition types of territorial spatial data.

[0024] Figure 2 This is a flowchart illustrating a specific embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the structure of a vector data object in a specific embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the data retrieval process according to a specific embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0028] In this embodiment, as Figure 2 As shown, a method for managing massive land spatial vector data involves identifying and processing the land spatial vector data to be stored to obtain spatial data and attribute data, and establishing a first mapping relationship between the spatial data and attribute data. The spatial data and attribute data are stored separately; the spatial data is stored in a spatial database, and the vector data is stored in an attribute database. In this embodiment, the spatial database is a relational database, and the attribute database is a document-oriented database. Land spatial data has diverse formats and standards, such as GDB, Shapfile, GeoJSON, etc., and the file data volume is large, with a single file typically exceeding GB bytes, exhibiting characteristics of abundant and complex spatiotemporal information data.

[0029] The identification process in this embodiment includes: identifying the land space vector data to be stored to obtain spatial data and attribute data; converting the spatial data to a format to obtain spatial data with a unified format type; parsing the attribute data into attribute list data and attribute column data, and establishing a second mapping relationship between the attribute list data and the attribute column data.

[0030] In this embodiment, the method further includes storing the attribute list data and attribute list metadata separately using key-value pairs. It also includes serializing the attribute list data and then connecting the serialized attribute list data to the search engine.

[0031] In this embodiment, data retrieval and querying are also included, such as... Figure 4 As shown: spatial data is retrieved, and corresponding attribute data is obtained according to the first mapping relationship; or: attribute list data is retrieved, and corresponding spatial data is obtained according to the first mapping relationship, and corresponding attribute column data is obtained according to the second mapping relationship.

[0032] In this embodiment, specifically, it is preferred to use the DGAL tool to identify multi-source land spatial vector data and parse it according to spatial data and attribute data to generate spatial data and attribute data of a unified data format type. The spatial data specifically includes spatial information and spatial reference information, while the attribute data specifically includes attribute list information and attribute column metadata information. More specifically, it is preferred to use the bridging mode of GDAL for multi-source data parsing, such as using the OpenFileGDB driver to parse GDB data, the PGeo driver to parse MDB data, the ESRI Shapefile driver to parse Shapefile data, and the GeoJSON driver to parse GeoJSON data. It should be noted that, in addition to using the DGAL tool, other tools can also be used to identify and process land spatial data, and the specific parsing content is adapted to the specific document specifications and does not limit the scope of protection of this patent due to the content exemplified in this embodiment. For example, such as... Figure 3 As shown, according to the OGR standard specification, vector data generally includes the following classes, such as... Figure 3As shown: Geometry: The Geometry class (including OGRGeometry, etc.) encapsulates OpenGIS's vector data model, the conversion between WKB (Well-Known Binary) and WKT (Well-Known Text) formats, and the spatial reference system (projection); Spatial Reference: The OGRSpatialReference class encapsulates the definition of projection and datum; Feature: The OGRFeature class encapsulates the definition of a complete feature, which includes a geometry and a series of geometry attributes; Feature Definition: The OGRFeatureDefn class encapsulates the feature's attributes, type, name, and default spatial reference system, etc. An OGRFeatureDefn object usually corresponds to a layer; Layer: The OGRLayer class is an abstract base class that represents a layer of features within the OGRDataSource data source class. Specifically, in the extraction and mapping of vector data, the GDAL library is used to parse the data. First, the data is filtered and classified according to the file extension information of the local files to be processed. Different driver classes are loaded according to different files. The GDAL library creates a data source DataSource based on the local data path. Then, by traversing all layers, the layer object Layer is obtained by DataSource.getLayer(index). Each Layer corresponds to a SpatialReference (commonly 4326 and 4490). Each Layer will have multiple FeatureDefn and multiple Features.The vector data file can be described in the following format, DataSource = {<O1.Layer><O2.Layer>…<On.Layer>}, where: Oi.Layer = {<O1.SpatialReference, O1.FeatureDefn: [attributes, type, name of the feature], O1.Feature: [attribute data, spatial data of the feature]><O2.SpatialReference, O2.FeatureDefn: [attributes, type, name of the feature], O2.Feature: [attribute data, spatial data of the feature]>…<On.SpatialReference, On.FeatureDefn: [attributes, type, name of the feature], On.Feature: [feature]}, Oi.FeatureDefn = {<O1.name, O1.width, O1.isNullable, O1.Precesion, O1.AlternativeName, O1.default>, …<On.name, On.width, On.isNullable, On.Precesion, On.AlternativeName, On.default>}, O1.feature = {<O1.ID, O1.Geometry, O1.Attribute><O2.ID, O2.Geometry, O2.Attribute>…<On.ID, On.Geometry, On.Attribute>}. After reading the above information using the GDAL library, the spatial data and attribute data are separated, and the relationship association mapping binding is performed while separating. After completing the identification and parsing of the national territorial spatial vector data and obtaining the spatial data and attribute data, all the spatial data in the vector data is stored in the spatial database in the form of Geometry. The spatial database preferably uses a relational database, and a business table is created in the spatial database, including a Geometry field, a business ID, and a table primary key column. Further, considering the scale of the data volume, it is also possible to perform database sharding and clustering based on different strategies (such as time range, administrative division range, GeoHash, etc.), which can meet the storage of spatial data while also satisfying the efficient retrieval of spatial data. In the specific processing of this embodiment, when identifying and processing the vector data, first read information such as the name, alias, data length, data format, whether it is nullable, data precision, default value, etc. of the corresponding layer attribute columns from FeatureDefn. These metadata provide metadata dictionary support when the system displays or shares data.Simultaneously, all attribute table information in the Feature, except for spatial data, is generated into JSON. Finally, both types of information are saved to the attribute database, which is a document-oriented database. During queries, the corresponding attribute list data is first located based on the mapping relationship, and then the corresponding attribute column information is read and displayed based on the mapping relationship. It should be noted that the attribute column metadata can also be found first based on the mapping relationship, and then the attribute list data can be found and displayed. Considering the storage pressure of massive amounts of data, the attribute database can be partitioned. Furthermore, it can be stored in JSON format in a high-efficiency search engine, such as Elasticsearch (hereinafter referred to as ES). High-speed full-text search can be built using ES, and storing all attribute data in ES can provide rapid data mining capabilities. During data management, if the attribute list information in the attribute database changes, ES will be notified promptly via message notification to achieve data synchronization consistency.

[0033] In this embodiment, by identifying and parsing the land spatial vector data, spatial data and attribute data are stored separately. Furthermore, based on the characteristics of the corresponding data, attribute list data from the attribute data can be loaded into the search engine to achieve efficient query retrieval. Moreover, by establishing a mapping relationship between spatial data and attribute data, flexibly supporting both spatial and attribute-based retrieval during the search process, the corresponding attribute and spatial data can be retrieved through the mapping relationship.

[0034] A massive land spatial vector data management system includes a data storage module: used to identify and process the land spatial vector data to be stored to obtain spatial data and attribute data, and establish a first mapping relationship between the spatial data and attribute data; the spatial data and attribute data are stored separately, with the spatial data stored in a spatial database and the attribute data stored in an attribute database.

[0035] In this embodiment, the identification process in the data storage module includes: identifying the land spatial vector data to be stored to obtain spatial data and attribute data; converting the spatial data to a format to obtain spatial data with a unified format type; parsing the attribute data into attribute list data and attribute column data, and establishing a second mapping relationship between the attribute list data and the attribute column data.

[0036] In this embodiment, the attribute list data and attribute list metadata are stored separately using key-value pairs. Furthermore, the attribute list data is serialized, and the serialized attribute list data is then connected to the search engine.

[0037] In this embodiment, a data retrieval module is also included: used to retrieve spatial data and obtain corresponding attribute data according to the first mapping relationship; or: to retrieve attribute list data and obtain corresponding spatial data according to the first mapping relationship, and obtain corresponding attribute list data according to the second mapping relationship.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for managing massive land spatial vector data, characterized in that: The land spatial vector data to be stored is identified and processed to obtain spatial data and attribute data, and the attribute data is parsed into attribute list data and attribute column metadata. Establish a first mapping relationship between the spatial data and the attribute data; store the spatial data and the attribute data separately, with the spatial data stored in a relational spatial database and the attribute data stored in a document database; The attribute list data is serialized, and the serialized attribute list data is then fed into the search engine.

2. The method for managing massive land spatial vector data according to claim 1, characterized in that: The identification process includes: identifying the land spatial vector data to be stored to obtain spatial data and attribute data; converting the spatial data to obtain spatial data with a unified format type; and establishing a second mapping relationship between the attribute list data and the attribute list data.

3. The method for managing massive land spatial vector data according to claim 2, characterized in that: It also includes storing the attribute list data and the attribute column metadata in a key-value separation manner.

4. The method for managing massive land spatial vector data according to claim 2, characterized in that: Also includes: The spatial data is retrieved, and the corresponding attribute data is obtained according to the first mapping relationship; or: The attribute list data is retrieved, and the corresponding spatial data is obtained according to the first mapping relationship, and the corresponding attribute column data is obtained according to the second mapping relationship.

5. A massive land spatial vector data management system, characterized in that: It includes a data storage module: used to identify and process the land space vector data to be stored to obtain spatial data and attribute data, and to parse the attribute data into attribute list data and attribute column metadata; Establish a first mapping relationship between the spatial data and the attribute data; store the spatial data and the attribute data separately, with the spatial data stored in a relational spatial database and the attribute data stored in a document database; The attribute list data is serialized, and the serialized attribute list data is then fed into the search engine.

6. The massive land spatial vector data management system according to claim 5, characterized in that: The identification process in the data storage module includes: identifying the land spatial vector data to be stored to obtain spatial data and attribute data; converting the spatial data to obtain spatial data with a unified format type; and establishing a second mapping relationship between the attribute list data and the attribute list data.

7. The massive land spatial vector data management system according to claim 6, characterized in that: It also includes storing the attribute list data and the attribute column metadata in a key-value separation manner.

8. The massive land spatial vector data management system according to claim 6, characterized in that: It also includes a data retrieval module: used to retrieve the spatial data and obtain the corresponding attribute data according to the first mapping relationship; or: The attribute list data is retrieved, and the corresponding spatial data is obtained according to the first mapping relationship, and the corresponding attribute column data is obtained according to the second mapping relationship.

Images