Data integration method and device based on geographic space information

Abstract

The invention relates to a data integration method and device based on geographic space information. The method comprises the following steps: selecting a to-be-analyzed file, and configuring a geographic space data integration source end and a target end; obtaining metadata of the geographic space data file; performing conversion according to the obtained metadata to obtain a target end field and assembling a table building statement; creating a spatial index of the target end according to the obtained metadata; and segmenting according to the number of geographic space data files, segmenting a corresponding number of subtasks by a manage management node, and concurrently executing file analysis, SQL assembly and data writing operation. According to the method, the geographic space data file is analyzed through the program to obtain the metadata information, the analysis process of files in different formats is unified, the technical background requirement and the learning cost of operators are reduced, the data integration efficiency is improved, and the complexity and the time cost of geographic space data integration are reduced. According to the method, the memory overflow risk is reduced by adopting a block reading mode, and the stability of metadata analysis is improved.

Description

Technical Field

[0001] The present application belongs to the field of data integration technology, and in particular, relates to a data integration method, device, computer-readable storage medium, and electronic device based on geographic spatial information. Background Art

[0002] As a new type of production factor, geospatial data contains natural, economic and social information with geographic location and time, providing a spatiotemporal data foundation for the digital economy and digital society. Geospatial big data analysis refers to the storage, processing, analysis and visualization of large amounts of geospatial data to reveal patterns, trends and relationships in the data. It can not only help us better understand geospatial phenomena, but also provide solutions to various practical problems. There are many technologies involved in geospatial big data analysis, one of which is the integration of geospatial data.

[0003] The diversity of geospatial data means that we need to process data from different sources, formats, and scales. In the field of geographic information systems, common geographic data formats include but are not limited to: Shapefile, GeoJSON, GDB, KML, etc. In the actual geospatial data integration process, factors such as incompatible geospatial data formats, complex data conversion and mapping steps, limitations of open source tools and technologies, and high technical background requirements for operators have brought more complexity and higher costs to the data integration process of geospatial information data. Summary of the invention

[0004] In order to address the above problems, this application proposes a new data integration method and device based on geospatial information, aiming to solve the problems of complex and high cost of geospatial data integration process caused by factors such as incompatible geospatial information data formats, complex data conversion and mapping steps, limitations of open source tools and technologies, and high technical background requirements for operators.

[0005] In summary, this application adopts the following technical strategies: 1. Obtain the metadata information of geospatial data files by parsing the geospatial data files through the program, and use the block reading method to solve the memory overflow problem that may be caused by loading large files.

[0006] 2. Based on the metadata information of the acquired geospatial data file, the target database table field information is obtained through data conversion, and the mapping between the source geospatial geometry and attribute information and the target relational database fields is automatically established to construct the target data table creation statement. In addition, the query performance of geospatial data is optimized by automatically creating the spatial index of the target database table.

[0007] 3. Provide visual interface operation. On the basis of automatic mapping, you can manually set reasonable field types and field lengths for the fields of the target relational database to achieve more flexible and reasonable field mapping.

[0008] 4. Adopt the strategy of splitting according to the number of files, split multiple subtasks through the manage node, start the corresponding thread in each task to complete the steps of file parsing->assembling SQL->execution writing, and concurrently perform data parsing of geospatial data files and data writing operations on the target database.

[0009] In order to achieve the above objectives, this application provides the following technical solutions: A first aspect of the present application provides a data integration method based on geospatial information, the method comprising: Select the file to be parsed and configure the source and target of geospatial data integration; Get metadata for geospatial data files; The target end fields are converted based on the obtained metadata and the table creation statements are assembled; Create a spatial index on the target end based on the obtained metadata; The geospatial data files are divided according to the number, and the manage node divides the corresponding number of subtasks to concurrently execute file parsing, SQL assembly and data writing operations.

[0010] Optionally, in the method of the present application, the configuring of the source end and the target end for geospatial data integration includes: (1) Source configuration Select a file system to store geospatial data files, including FTP, SFTP, HDFS, and Minio file systems; Select a file format type of a geospatial data file, wherein the file format type includes Shapefile and GeoJSON; After determining the file format type to be parsed, the files that meet the file format type requirements in the corresponding file system are displayed in a list for the user to select; (2) Target side configuration Configure the basic information of the target data source, including the target data source type, target database name, and target table.

[0011] Optionally, in the method of the present application, the metadata of the geospatial data file includes geometric information and attribute information of the geospatial data file.

[0012] Optionally, in the method of the present application, the step of obtaining metadata of the geospatial data file includes: The GeoTools tool is used to parse the geospatial data file to obtain its metadata information, wherein the metadata information includes attribute names and corresponding data types; For large geospatial data files, the data is read in blocks when parsing the files.

[0013] Optionally, in the method of the present application, the step of reading data in blocks includes: (1) For the same geospatial data file, only one Feature data structure is obtained; (2) Use FileInputStream to create an input stream for reading geospatial data files; (3) Use the read() method of the FileInputStream object to read file data according to the preset byte block size; (4) Determine whether a complete Feature data structure has been obtained; (5) Parse the Feature data structure and extract geometric information and attribute information.

[0014] Optionally, in the method of the present application, converting the acquired metadata to obtain the target end fields and assembling the table creation statement includes: (1) Generation of target-side fields The target field names are automatically generated according to the one-to-one mapping of the attribute names in the metadata information; The data type corresponding to the attribute name in the metadata information is converted to the target end field type through the preset data type conversion rules; (2) Adjustment of the target field Pre-set the field type, field length and field precision for the target field, and use the visual interface to adjust the target field generated in the previous step; (3) Assemble the SQL statement for creating the table based on the target field.

[0015] Optionally, in the method of the present application, the geospatial data files are segmented according to the number of files, and the management node is segmented into a corresponding number of subtasks to concurrently execute file parsing, assembling SQL and data writing operations, including: After receiving the data synchronization request, the manage management node calculates the number of geospatial data files that need to be parsed; The manage management node splits the corresponding number of subtasks; The manage management node evenly distributes the geospatial data files to each subtask, and each subtask starts the corresponding thread to perform file parsing, SQL assembly, and data writing operations.

[0016] A second aspect of the present application provides a data integration device based on geographic spatial information, the device comprising: Data source configuration module: used to provide a visual interface for configuring the source and target ends of geospatial data integration; Metadata query module: used to obtain metadata of geospatial data files; Data mapping and table creation module: used to convert the target end fields according to the acquired metadata and assemble the table creation statements; Data synchronization module: used to split geospatial data files according to the number of files. The manage node divides the files into a corresponding number of subtasks and concurrently executes file parsing, SQL assembly, and data writing operations.

[0017] The device implements the steps of the aforementioned data integration method based on geographic space information when running.

[0018] A third aspect of the present application provides an electronic device, comprising: a memory and a processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the aforementioned data integration method based on geospatial information.

[0019] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the aforementioned data integration method based on geospatial information are implemented.

[0020] In summary, this application proposes a new data integration method based on geospatial information, which has the following advantages: (1) The program parses geospatial data files to obtain metadata information of geospatial data files, unifies the parsing process of files in different formats, reduces the technical background requirements and learning costs of operators for geospatial data, and improves data integration efficiency. In addition, the block reading method reduces the risk of memory overflow that may occur during the parsing of large files, optimizes memory usage, and improves the stability of metadata parsing.

[0021] (2) Based on the metadata information of the acquired geospatial data files, the target-side relational database fields are automatically inferred through data conversion, thereby simplifying the operator's task configuration operations, reducing the complexity of data conversion and mapping, improving the accuracy of field mapping, and enhancing the efficiency of creating target-side database tables.

[0022] (3) Provides a visual interface operation. When the automatic target-side field type inference is inaccurate, you can manually set a reasonable field type and field length for the field of the target-side relational database to achieve more flexible and reasonable field mapping.

[0023] (4) It can automatically establish the spatial index of the target data table based on the metadata information of the acquired geospatial data file to optimize the query performance of the geospatial data.

[0024] (5) The strategy of splitting files according to the number of files is adopted. Multiple subtasks are split through the manage node. Multi-threading is used to concurrently parse geospatial data files and write data to the target database, which can significantly improve the integration capability and efficiency of geospatial data.

[0025] (6) This application realizes the visualization and automation of geospatial data integration, reducing the complexity and time cost of geospatial data integration.

[0026] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the techniques indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 This is an overall implementation flow chart of the data integration method based on geographic spatial information according to an embodiment of the present application.

[0029] Figure 2 This is an example diagram showing the field mapping relationship in the visual interface in the embodiment of the present application.

[0030] Figure 3 This is a diagram of the task partitioning architecture of the data synchronization module in an embodiment of the present application.

[0031] Figure 4 This is a flow chart of obtaining data items by parsing files in the data synchronization module in an embodiment of the present application.

[0032] Figure 5 A schematic diagram of the composition structure of a data integration device based on geographic spatial information provided in this application.

[0033] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] As used herein, the term “including” and its variations are open inclusions, ie, “including but not limited to”; the term “based on” means “based at least in part on”; and the term “one embodiment” means “at least one embodiment”.

[0036] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0037] Explanation of terms: Geospatial data The data formats of geospatial data are generally divided into two types: vector data and raster data.

[0038] Vector data: It is data that uses X, Y, and Z coordinates to represent the location of map graphics or geographic entities. Vector data generally records coordinates to express the spatial location of geographic entities as accurately as possible. Common vector data formats include: point, line, and surface.

[0039] Common vector data formats include: SHP (Shapefile), GeoJSON (Geographic JavaScript Object Notation), GML (Geography Markup Language), KML / KMZ (Google Keyhole Markup Language), OSM (OpenStreetMap OSM XML), and DXF / DWG.

[0040] Among them, Shapefile is the most common and popular vector data format, and GeoJSON is a JSON-based geospatial data format that is also widely used.

[0041] Raster data: It consists of a matrix of cells (or pixels) organized in rows and columns (or grid), where each cell contains a value of information (such as temperature). A raster can be a digital aerial photograph, satellite imagery, digital picture, or even a scanned map.

[0042] Common raster data formats include: ERDAS Imagine (IMG), IDRISI Raster, Envi RAWRaster, GeoTIFF, BMP, and PNG.

[0043] In the embodiment of the present application, data integration is performed only on geospatial vector data.

[0044] Figure 1 The figure shows the overall implementation process of a data integration method based on geospatial information proposed in this application, which includes the following steps: Select the file to be parsed and configure the source and target of geospatial data integration; Get metadata for geospatial data files; The target side fields are converted based on the obtained metadata and the table creation statements are assembled; Create a spatial index on the target end based on the obtained metadata; The geospatial data files are divided according to the number, and the manage node divides the corresponding number of subtasks to concurrently execute file parsing, SQL assembly and data writing operations.

[0045] In order to better understand the technical solution of the present application, further explanation is given with reference to embodiments of the following scenarios.

[0046] The data integration method based on geographic spatial information in this embodiment is implemented by the following modules: 1. Data source configuration module The function of the data source configuration module is to provide a visual interface for configuring the source and target ends of geospatial data integration.

[0047] 1. Source configuration The source configuration mainly involves selecting the geospatial data files that require data integration. Geospatial data files are mainly stored in file systems such as FTP, SFTP, HDFS, and Minio. It includes the following configurations: (1) Source data source The source data source mainly selects the file system for storing geospatial data files. In this application, FTP, SFTP, HDFS, Minio and other file systems are provided for selection, and it also supports uploading local files to the above file systems.

[0048] (2) File type In this application, geospatial data file format types such as Shapefile, GeoJSON, etc. are provided for selection.

[0049] (3) Source documents After determining the file format type that needs to be parsed, the program will display the files that meet the file format type requirements in the corresponding file system in a list for users to select. One or more files can be selected. When multiple files are selected, these files need to have the same data structure, that is, the metadata structure of the geospatial data files needs to be the same.

[0050] 2. Target configuration The target-side configuration mainly involves the configuration of the basic information of the target-side data source for data integration. In this application, the geospatial data is imported into a relational database, so the basic configuration of the target-side data source includes the target-side data source type, target-side database name, and target-side table.

[0051] 2. Metadata Query Module The metadata query module mainly provides the display of the geometry and attributes of geospatial data files. In order to correctly map the geometry and attribute data in the geospatial data files to the fields of the target database table, the first step is to obtain the metadata information of the geospatial data files, which includes information such as geometry and attributes.

[0052] Since geospatial data files have multiple formats, each format has its own specific structure and parsing method. Different formats may require different parsing tools and techniques, which brings certain complexity to the metadata parsing of geospatial data files.

[0053] In the present application, a unified metadata query module is used to perform metadata parsing of geospatial data files, thereby obtaining information such as geometry information and attributes in the geospatial data files.

[0054] In order to more intuitively show the format of geospatial vector data, the GeoJSON data format is used as an example: { "type": "FeatureCollection", "features": [ { "type": "Feature", "properties": { "adcode": 110000, "name": "Beijing City", "center": [ 116.405285, 39.904989 ], "centroid": [ 116.41995, 40.18994 ], "childrenNum": 16, "level": "province", "parent": { "adcode": 100000 }, "subFeatureIndex": 0, "acroutes": [ 100000 ] }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 117.348611, 40.581141 ], [ 117.389879, 40.561593 ], ... ] ] ] } } ... ] } GeoJSON is a common file format for geographic information. As shown above, it represents geometric objects, other properties, and feature collections in the form of JSON (JavaScript Object Notation). GeoJSON supports the following geometry types: point (Point), line (LineString), surface (Polygon), multi-point (MultiPoint), multi-line (MultiLineString), multi-surface (MultiPolygon), and geometry collection (GeometryCollection).

[0055] Among them, a geographic feature can be represented by an object named "Feature", which contains the geometric information and attribute information of the geographic feature.

[0056] (1) Obtaining metadata In this application, GeoTools is used to parse GeoJSON format files and obtain metadata information.

[0057] GeoTools is an open source Java library for working with geospatial data. It provides a range of APIs for reading, writing, processing, and analyzing various geospatial data formats, such as Shapefile, GeoJSON, KML, GML, etc. GeoTools also includes some tools and components specifically for geographic information systems (GIS) and map making, such as rendering engines, symbolization, spatial indexing, projection conversion, etc.

[0058] After parsing, the metadata information of the above sample GeoJSON format file can be obtained as shown in Table 1: Table 1 Metadata information of the sample GeoJSON format file obtained by parsing In Table 1, attributes 1 to 9 are basic attribute information in properties, and attribute 10 is geometric information.

[0059] After geospatial data files of various formats are automatically parsed through a unified metadata query module, the attribute names and corresponding attribute types will be extracted. This approach not only shields the differences between different geospatial data file formats, avoids tedious and complicated parsing steps, simplifies the user's operation, but also reduces the user's learning cost and technical background requirements for geospatial data.

[0060] (2) Block reading When the geospatial data file is large, if the entire file is loaded into the memory at once during file parsing, it may cause insufficient program memory and overflow errors. In order to avoid the above problems, this application adopts a block reading method when parsing large geospatial data files to optimize memory usage and improve parsing efficiency and stability.

[0061] This application uses the FileInputStream class (an input stream for reading file byte streams) to read data on demand instead of loading the entire file into memory at once.

[0062] The specific steps are: (a) For the same geospatial data file, the data structure of each record is the same, so only the data structure of a Feature needs to be obtained; (b) Create an input stream to read geospatial data files using FileInputStream; (c) Use the read() method of the FileInputStream object to read file data in a certain byte block size; (d) Determine whether a complete Feature data structure is obtained; (e) Parse the Feature data structure and extract geometry and attribute information.

[0063] Through the above methods, memory usage can be effectively controlled to avoid program memory overflow errors caused by large files, thereby improving the stability of the parsing process.

[0064] (III) Data mapping and table creation module Importing geospatial data into a target relational database table involves mapping the geometry and attribute data in the geospatial data file to the fields of the relational database table.

[0065] This is not a simple process. Operators not only need to sort out the structure and type of geospatial data, but also need to understand the design of the relational database table on the target side and master the use of related tools and technologies. In addition, in order to ensure the success and efficiency of data writing, operators are also required to have skills such as data conversion and optimization. These contents increase the complexity and time cost of the data writing process, and have high requirements on the operator's relevant knowledge and skills.

[0066] In order to improve the efficiency of geospatial data integration, reduce the learning cost of operators, and simplify the data integration process, this application designs a data mapping and table building module to achieve automatic data conversion and mapping, thereby simplifying the table building process.

[0067] In the data mapping and table creation module, the corresponding fields of the relational database table are automatically mapped and generated mainly based on the metadata information of the geographic spatial data obtained by the metadata query module.

[0068] Here we take the import of GeoJSON format file data into the relational database PostgreSQL as an example.

[0069] 1. Generation of target-side fields As described in Table 1 above, the second and third columns are the attribute names (including geometry and attribute information) and data types in the geospatial data file obtained in the metadata query module. In order to automatically map and generate the corresponding fields of the target end relational database table, this application designs the following rules: (1) Target field name The target database table field names are automatically generated by mapping one by one the attribute names in the parsed geospatial data file.

[0070] (2) Target field type By analyzing the data types in the geospatial data files obtained by parsing, it can be seen that the attribute types in the geospatial data files are mainly summarized into the following categories: string, integer / long integer, collection, object, geometric figure, etc.

[0071] In the present application, the field type of the target-side relational database table is quickly converted through preset data type conversion rules.

[0072] Here we take the relational database PostgreSQL as an example. The preset geographic spatial data types and PostgreSQL field type conversion rules are shown in Table 2: Table 2 Conversion rules between geospatial data types and PostgreSQL field types Data Types Data Type Description Target field type Integer Integer integer Long Long integer bigint String String varchar ArrayList gather float8[] Object Object varchar POINT point geometry LINESTRING Line, Polyline geometry POLYGON noodle geometry MULTIPOINT More geometry MULTILINESTRING Multi-line geometry MULTIPOLYGON Many sides geometry POLYHEDRAL SURFACE Polyhedral Surface geometry CIRCULARSTRING Line, formed by connecting arcs geometry COMPOUNDCURVE Composite line geometry ELLIPTICALSTRING Elliptical Arc geometry BEZIER3CURVE Bezier curve geometry MULTICURVE Polyline geometry CURVEPOLYGON Composite surface geometry MULTISURFACE Many sides geometry GEOMETRYCOLLECTION Composite Objects geometry TRIANGLE triangle geometry TIN Triangulated Irregular Network geometry In Table 2, the first column is the data type of the metadata attribute in the geospatial data file, and is a Java data type, and the third column is the field type of the preset target-side relational database table field after mapping. In this embodiment, the geometric attributes of the source-side geospatial data file are all mapped to the geometry type of the PostgreSQL database. The geometry type of the PostgreSQL database can be used to store coordinate data, support the expression of geometric objects such as points, lines, polygons, etc., and provide a set of functions for processing these data.

[0073] According to the above preset data type conversion rules, the field mapping relationship in this example can be obtained as shown in Table 3: Table 3 Mapping relationship between converted geospatial data and target end fields Source attribute name Data Types Data type classification Target field name Target field type adcode Long Long integer #timg# adcode bigint name String String #timg# name varchar center ArrayList gather #timg# center float8[] Centroid ArrayList gather #timg# Centroid float8[] childrenNum Long Long integer #timg# childrenNum bigint level String String #timg# level varchar parent Object Object #timg# parent varchar subFeatureIndex Long Long integer #timg# subFeatureIndex bigint acroutes ArrayList gather #timg# acroutes float8[] geometry MultiPolygon Geometry #timg# geometry geometry 2. Adjustment of target side fields When the field type or length of the target-side relational database table obtained by the preset data type conversion rule cannot meet the actual usage requirements, in this application, a visual interface is also provided to adjust the fields of the target-side relational database table, such as Figure 2 shown.

[0074] On the basis of automatic mapping, you can manually set reasonable field types, field lengths, and precision for the fields of the target relational database to achieve more flexible and reasonable field mapping.

[0075] Through the above two steps, the SQL statement for creating a table in the target PostgreSQL database can be obtained as follows: CREATE TABLE locations ( "adcode" bigint NULL, "name" varchar(100) NULL, "center" float8[] NULL, "centroid" float8[] NULL, "childrenNum" bigint NULL, "level" varchar(100) NULL, "parent" varchar(100) NULL, "subFeatureIndex" bigint NULL, "acroutes" float8[] NULL, "geometry" geometry NULL ); Among them, locations is the name of the database table.

[0076] 3. Establishment of spatial index In PostgreSQL, a spatial index is a special index used for fast search and query on spatial data types such as points, lines, polygons, etc. By creating a spatial index on a data table, you can optimize the query efficiency of geospatial data.

[0077] In this application, based on the metadata information of the geographic spatial data obtained by the metadata query module, the spatial data type column can be automatically identified, such as the geometry attribute in this example, and the spatial index of the target database table can be created through the following SQL statement: CREATE INDEX idx_locations_geom ON locations USING GIST (geometry); Among them, idx_locations_geom is the spatial index name, locations is the database table name, and geometry is the recognized spatial data type column name.

[0078] (IV) Data synchronization module When integrating geospatial data, you often need to parse and integrate multiple large-scale geospatial data files into the target relational database table. These geographic information data files may contain millions of geometric shapes and attribute records. If the data is parsed and written to the target directly in serial, performance problems may occur, affecting the efficiency of data integration.

[0079] In the data synchronization module, in order to optimize the performance of importing geospatial data into the target database table and improve the import efficiency, in this application, a strategy of splitting according to the number of files is adopted, and multi-threading is used to concurrently perform data parsing of geospatial data files and data writing operations into the target data table.

[0080] In this solution, the core component responsible for data synchronization in the data synchronization module is manage, which is the central management node for a single synchronization task.

[0081] After receiving the request for data synchronization, the manage node will calculate the number of geospatial data files that need to be parsed. Here, it is assumed that the number of files is n. The manage node divides n subtasks into tasks and evenly distributes the geospatial data files to each task. In each task, the corresponding thread will be started to complete the steps of file parsing -> assembling SQL -> executing writing, such as Figure 3 shown.

[0082] These subtasks are executed concurrently, making full use of the resources of the operating system, which can significantly speed up the data parsing of geospatial data files and the speed of writing data into the target database, thereby improving the efficiency of data integration.

[0083] The following is a detailed description of the workflow in each task: (1) File parsing Different from the purpose of parsing geospatial data files in the metadata query module, the purpose of parsing files in the data synchronization module is to obtain data items corresponding to the geographic geometry and attributes of the source end.

[0084] GeoTools is also used here to obtain the data items of each record by parsing the file into a feature set.

[0085] like Figure 4 As shown, the specific process includes: loading GeoJSON file; creating FeatureJSON object; parsing the file into feature collection; traversing feature collection; outputting data items.

[0086] In this example, it is assumed that the parsed geospatial data of Beijing is as shown in Table 4: Table 4 Example of geospatial data of Beijing obtained by analysis (2) Assembling SQL In this embodiment, since the insertion of geographic spatial data is involved, a specific function ST_GeomFromText needs to be used to process the geographic spatial data so as to insert it into the target database data table.

[0087] The following SQL statement needs to be constructed: insert into locations (adcode, name, center, centroid, children, level, parent, subFeatureIndex, acroutes, geometry) values (110000, 'Beijing', array[116.405285, 39.904989],array[116.41995, 40.18994], 16, 'province', '{"adcode": 100000}', 0,array[100000],ST_GeomFromText('MULTIPOLYGON (((117.348611 40.581141,117.389879 40.561593,117.429915 40.576141, 117.412669 40.605226, 117.467487 40.649738, 117.46748740.649738, 117.501364 40.636569, 117.514914 40.660181,117.493973 40.675161,117.408973 40.686961, 117.342451 40.673799, 117.319662 40.657911, 117.27839440.664267, 117.208177 40.694675, 117.117018 40.70012,117.11209 40.707379,117.012308 40.693767, 116.964881 40.709647, 116.926692 40.745022, 116.92422940.773581, 116.848468 40.839264, 116.81336 40.848319,116.759773 40.889954,116.713577 40.909858, 116.722201 40.927495, 116.677853 40.970888, 116.69879541.021477, 116.688324 41.044501, 116.647672 41.059394,116.615643 41.053076,116.623034 41.021026, 116.598397 40.974503, 116.5676 40.992574, 116.51955740.98128, 116.519557 40.98128, 116.455499 40.980828, 116.44749240.953715,116.477057 40.899907, 116.398216 40.90624, 116.370499 40.94377, 116.33970240.929303, 116.334159 40.90443, 116.438253 40.81934, 116.4659740.774487,116.453651 40.765876, 116.316912 40.772221, 116.311369 40.754996, 116.27318140.762703, 116.247311 40.791707, 116.22021 40.744115, 116.20481240.740035,116.171551 40.695582, 116.162928 40.662451, 116.133979 40.666536, 116.0988740.630665, 116.005247 40.583868, 115.982457 40.578868, 115.97198640.6025,115.907929 40.617493, 115.885139 40.595229, 115.827857 40.587504, 115.81984940.55932, 115.784741 40.55841, 115.755176 40.540221, 115.73608240.503372,115.781045 40.49336, 115.771806 40.443734, 115.864197 40.359422, 115.91778440.354405, 115.95166 40.281852, 115.968907 40.264045, 115.89869 40.234354,115.870356 40.185909, 115.855574 40.188652, 115.847567 40.147036, 115.80629940.15344, 115.773654 40.176307, 115.75456 40.145663, 115.75456 40.145663,115.599959 40.119583, 115.59072 40.096239, 115.527278 40.076092, 115.48539440.040364, 115.454597 40.029825, 115.450286 39.992697, 115.428728 39.984443,115.426264 39.950502, 115.481083 39.935819, 115.522967 39.899099, 115.51557539.892212, 115.515575 39.892212, 115.526046 39.87568, 115.514344 39.837549,115.567314 39.816407, 115.552532 39.794799, 115.50572 39.784222, 115.48354739.798477, 115.483547 39.798477, 115.443511 39.785601, 115.439815 39.752022,115.486626 39.741899, 115.491554 39.670074, 115.478619 39.650723, 115.47861939.650723, 115.522351 39.640124, 115.518039 39.597252, 115.545756 39.618922,115.587024 39.589873, 115.633836 39.599557, 115.633836 39.599557, 115.66771239.615234, 115.698509 39.577881, 115.698509 39.577881, 115.699125 39.570039,115.699125 39.570039, 115.716988 39.56035, 115.716988 39.56035, 115.71883539.553891, 115.718835 39.553891, 115.720683 39.551122, 115.720683 39.551122,115.722531 39.5442, 115.721299 39.543738, 115.722531 39.5442, 115.72253139.543738, 115.721299 39.543738, 115.722531 39.543738, 115.724995 39.5442,115.724995 39.5442, 115.738545 39.540046, 115.738545 39.539585, 115.73854539.540046, 115.738545 39.539585, 115.752712 39.515581, 115.806299 39.510041,115.806299 39.510041, 115.821081 39.522968, 115.821081 39.522968, 115.82847339.541431, 115.867893 39.546507, 115.867893 39.546507, 115.91532 39.582955,115.91532 39.582955, 115.910393 39.600479, 115.910393 39.600479, 115.95720439.560812, 115.978146 39.595868, 115.995392 39.576958, 116.026189 39.587567,116.036044 39.571884, 116.09887 39.575113, 116.130283 39.567732, 116.15184139.583416, 116.198652 39.589412, 116.240536 39.564041, 116.257782 39.500344,116.307057 39.488337, 116.337854 39.455536, 116.361876 39.455074, 116.36187639.455074, 116.434557 39.442597, 116.454883 39.453226, 116.444412 39.482332,116.411767 39.482794, 116.401912 39.528046, 116.443796 39.510041, 116.43763739.526661, 116.478289 39.535431, 116.473361 39.552968, 116.50847 39.551122,116.524484 39.596329, 116.592237 39.621227, 116.592237 39.621227, 116.62057139.601863, 116.664918 39.605552, 116.723432 39.59264, 116.724048 39.59264,116.723432 39.59264, 116.724048 39.59264, 116.726512 39.595407, 116.72651239.595407, 116.709266 39.618, 116.748686 39.619844, 116.79057 39.595868,116.812128 39.615695, 116.8497 39.66777, 116.906366 39.677444, 116.9057539.688037, 116.889736 39.687576, 116.887272 39.72533, 116.916837 39.731314,116.902055 39.763523, 116.949482 39.778703, 116.918069 39.84628, 116.90759839.832494, 116.865714 39.843982, 116.812128 39.889916, 116.78441 39.891294,116.782563 39.947749, 116.757925 39.967934, 116.781331 40.034866, 116.82013540.02845, 116.831222 40.051359, 116.867562 40.041739, 116.927924 40.055024,116.945171 40.04128, 117.025243 40.030283, 117.051728 40.059605, 117.10531540.074261, 117.105315 40.074261, 117.140423 40.064185, 117.15951740.077008,117.204481 40.069681, 117.210024 40.082045, 117.224191 40.094865,117.22419140.094865, 117.254988 40.114548, 117.254988 40.114548, 117.25498840.114548, 117.274082 40.105852, 117.307343 40.136971, 117.349227 40.136513,117.36708940.172649, 117.367089 40.173106, 117.367089 40.173106, 117.36708940.172649, 117.383719 40.188195, 117.389879 40.227958, 117.351075 40.229786,117.33136540.289613, 117.295024 40.2782, 117.271618 40.325211, 117.27161840.325211, 117.243285 40.369453, 117.226039 40.368997, 117.234046 40.417312,117.26361140.442367, 117.208793 40.501552, 117.262995 40.512927, 117.24759740.539766, 117.269771 40.560684, 117.348611 40.581141, 117.34861140.581141))). '))); (3) Execute write In each task, obtain the JDBC connection to the target relational database, create an SQL execution object, and execute the SQL statement assembled in the previous step to insert data.

[0088] Figure 5 The present application shows a data integration device based on geographic space information, comprising: Data source configuration module: used to provide a visual interface for configuring the source and target ends of geospatial data integration; Metadata query module: used to obtain metadata of geospatial data files; Data mapping and table creation module: used to convert the target end fields according to the acquired metadata and assemble the table creation statements; Data synchronization module: used to split geospatial data files according to the number of files. The manage node divides the files into a corresponding number of subtasks and concurrently executes file parsing, SQL assembly, and data writing operations.

[0089] When the above device is running, the steps of the data integration method based on geographic space information disclosed in this application are implemented.

[0090] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the specified function or operation, or can be realized by a combination of special hardware and computer instructions.

[0091] like Figure 6 As shown, the embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above-mentioned data integration method based on geographic spatial information.

[0092] It is understandable that, in addition to the memory and the processor, the electronic device may also include an input device such as a keyboard, an output device such as a display, and other communication modules. The input device, the output device, and other communication modules communicate with the processor via an I / O interface (i.e., an input / output interface).

[0093] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the various steps of the data integration method based on geospatial information disclosed in the present application.

[0094] In the context of the present application, a computer-readable storage medium may be a tangible medium, more specific examples of which would include a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] In particular, according to an embodiment of the present application, the process described in the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the data integration method based on geospatial information disclosed in the present application. When the computer program is executed by a processing device, the above functions defined in the method of the embodiment of the present application are executed.

[0096] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solutions formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above public concept.

[0097] Those skilled in the art should also understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data integration method based on geographic spatial information, characterized in that: The method comprises: Select the file to be parsed and configure the source and target of geospatial data integration; Get metadata for geospatial data files; The target end fields are converted based on the obtained metadata and the table creation statements are assembled; Create a spatial index on the target end based on the obtained metadata; The geospatial data files are divided according to the number, and the manage node divides the corresponding number of subtasks to concurrently execute file parsing, SQL assembly and data writing operations.

2. The method according to claim 1, characterized in that The configuration of the geospatial data integration source end and target end includes: (1) Source configuration Select a file system to store geospatial data files, including FTP, SFTP, HDFS, and Minio file systems; Select a file format type of a geospatial data file, wherein the file format type includes Shapefile and GeoJSON; After determining the file format type to be parsed, the files that meet the file format type requirements in the corresponding file system are displayed in a list for the user to select; (2) Target side configuration Configure the basic information of the target data source, including the target data source type, target database name, and target table.

3. The method according to claim 1, characterized in that The metadata of the geospatial data file includes geometric information and attribute information of the geospatial data file.

4. The method according to claim 1, characterized in that: The step of obtaining the metadata of the geospatial data file includes: The GeoTools tool is used to parse the geospatial data file to obtain its metadata information, wherein the metadata information includes attribute names and corresponding data types; For large geospatial data files, the data is read in blocks when parsing the files.

5. The method according to claim 4, characterized in that The method of reading data in blocks includes: (1) For the same geospatial data file, only one Feature data structure is obtained; (2) Use FileInputStream to create an input stream for reading geospatial data files; (3) Use the read() method of the FileInputStream object to read file data according to the preset byte block size; (4) Determine whether a complete Feature data structure has been obtained; (5) Parse the Feature data structure and extract geometric information and attribute information.

6. The method according to claim 1, characterized in that The method of converting the obtained metadata to obtain the target end fields and assembling the table building statement includes: (1) Generation of target-side fields The target field names are automatically generated according to the one-to-one mapping of the attribute names in the metadata information; The data type corresponding to the attribute name in the metadata information is converted to the target end field type through the preset data type conversion rules; (2) Adjustment of the target field Pre-set the field type, field length and field precision for the target field, and use the visual interface to adjust the target field generated in the previous step; (3) Assemble the SQL statement for creating the table based on the target field.

7. The method according to claim 1, characterized in that The geospatial data files are divided according to the number of files, and the management node divides the corresponding number of subtasks to concurrently perform file parsing, SQL assembly and data writing operations, including: After receiving the data synchronization request, the manage management node calculates the number of geospatial data files that need to be parsed; The manage management node splits the corresponding number of subtasks; The manage management node evenly distributes the geospatial data files to each subtask, and each subtask starts the corresponding thread to perform file parsing, SQL assembly, and data writing operations.

8. A data integration device based on geographic space information, characterized in that: The device comprises: Data source configuration module: used to provide a visual interface for configuring the source and target ends of geospatial data integration; Metadata query module: used to obtain metadata of geospatial data files; Data mapping and table creation module: used to convert the target end fields according to the acquired metadata and assemble the table creation statements; Data synchronization module: used to split geospatial data files according to the number of files. The manage node divides the files into a corresponding number of subtasks and concurrently executes file parsing, SQL assembly, and data writing operations.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data integration method based on geospatial information as described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the data integration method based on geospatial information as described in any one of claims 1-7.

Images