Quick release and authentication method for vector data

Abstract

The invention discloses a method for quickly publishing and authenticating vector data, which solves the problem of service publishing and authentication of space vector data under a data lake, has the storage characteristics of a Delta-Lake framework, performs uniform format conversion on space data in different data formats and stores the space data, and realizes uniform management of the space data in different formats. On the basis of a cloud native environment, a Serverless architecture is adopted, so that a service publisher only needs to care about implementation logic of the service and does not need to care about an operation environment; an open source GeoServer (map server) is adopted as a basic engine, the high concurrency capability is expanded, and the high-frequency calling and authentication requirements of service consumers are met to the maximum extent.

Description

Technical Field

[0001] The invention relates to the technical field of electronic digital processing, and in particular to a method for rapid publishing and authentication of vector data. Background Art

[0002] The big data platform stores data in separate databases according to the vector spatial data type, and then publishes services based on the GIS engine. Disadvantages of existing technologies: They are more oriented towards publishing services stored in fixed spatial databases or fixed spatial data formats such as Shapefile. In the era of big data, data lakes are the mainstream multimodal data storage architecture. There are few technical solutions for publishing spatial data services for spatial data stored in data lakes.

[0003] The Chinese patent with the patent authorization announcement number: CN108924115B and the name "A spatial service authority control method and system" proposes a method for controlling the data range, layer, etc. of spatial service data for different roles. The disadvantages of this technology are: (1) When the same user has multiple roles, the problem of how to integrate data permissions when accessing spatial services; (2) The accuracy of spatial service data control is only at the layer and field level. The existing technical defects are: (1) The existing spatial vector data service publishing relies more on localized traditional data storage, and the spatial service publishing capability based on the data lake is weak; (2) It is impossible to realize the publishing and authentication of spatial vector data services across data sources. Summary of the invention

[0004] In order to solve the technical problems existing in the background technology, the present invention proposes a method for rapid publishing and authentication of vector data.

[0005] The technical concept of the present invention is: the Delta-Lake framework storage characteristics are used to convert and store spatial data in different data formats into a unified format, so as to achieve unified management of spatial data in different formats; based on the cloud native environment, a serverless architecture is adopted, so that service publishers only need to care about the implementation logic of the service itself without having to worry about the operating environment; open source GeoServer is used as the basic engine to expand high concurrency capabilities and maximize the high-frequency calling and authentication needs of service consumers.

[0006] A method for rapid publishing and authentication of vector data comprises the following steps:

[0007] Step A: Spatial dataset conversion: Based on the data lake, traditional spatial vector data is mapped and converted to spatial data field types and stored;

[0008] Step B: Service online configuration: Select the service data stored in step A, write the data content online, and debug the service to form the original space service;

[0009] Step C: After the original space service is completed, the service registration routing is performed: the original space service formed in step B is registered to the service gateway, and the gateway is used as a unified proxy;

[0010] Step D: User authority analysis: Integrate the authority of a single space service of the role owned by the user to calculate the scope of the user's authority to the space service;

[0011] Step E: Service data output: Request space service according to the parsed data permission range and return service data content.

[0012] Specifically, in step A, the traditional spatial vector data includes SHP, CSV, GeoJson, GDB and SDE under the ESRI standard, and the traditional spatial vector data format is converted into a String format.

[0013] As a further optimization solution of the present invention, according to different spatial vector data formats, different methods of the Apache Sedona function library are used to parse and convert them into Delta tables for unified storage in the data lake, thereby realizing spatial vector data management based on the data lake. The Delta table is a table format defined by the data lake and is a two-dimensional table structure based on column storage.

[0014] Specifically, in step B, the service script in the data writing process is written in a mixture of SQL, JAVA and Groovy. After writing is completed, based on the OGC service standard, the corresponding script is converted into the GIS spatial service standard, and the spatial field type check is performed. The OGC service standard converts the corresponding script into the GIS spatial service standard, and the spatial field type check is performed. The converted script is published to the GeoServer service engine to form the original spatial service.

[0015] As a further optimized solution of the present invention, the external exposure of the original space service in step C specifically includes:

[0016] C1: Register the service route in the HTTP container of the service instance;

[0017] C2: Register the service routes that are actually exposed to the outside world in the service gateway.

[0018] As a further optimized solution of the present invention, the service request traffic in C2 is received and processed via the gateway, and then forwarded to the service instance to complete the request according to the service load balancing strategy set by the nginx gateway.

[0019] As a further optimization scheme of the present invention, in step D, the user has at least two roles, and the service permissions of the user in each role are extracted. The spatial information and attribute information contained in each permission are union calculated to merge them into the permission scope of the user's corresponding service, and the record is encrypted and stored in the system database, and the record is associated with the user's service key.

[0020] As a further optimized solution of the present invention, it is assumed that the user has two roles A and B. The attribute information authority range of role A to spatial service 1 is attribute information range A, and the spatial information authority range is spatial information range A; the attribute information authority range of role B to spatial service 1 is attribute information range B, and the spatial information authority range is spatial information range B. The specific steps are described as follows:

[0021] Step D1: extract the A role and B role possessed by the user and the scope of authority of these roles for service 1, including the scope of attribute information and the scope of spatial information;

[0022] Step D2: perform union calculation on the attribute information ranges of role A and role B, and generate an SQL configuration statement, namely, attribute information fusion range C, which is the attribute information authority range of the user for space service 1;

[0023] Step D3: Write the user's attribute information fusion range C for space service 1 into the base_info_limit field of rel_user_service;

[0024] Step D4: perform spatial union calculation on the spatial information ranges of role A and role B to generate a spatial information fusion range C, which is the user's spatial information authority range for spatial service 1.

[0025] As a further optimized solution of the present invention, it also includes step D5: constructing a logical relationship between user information, associated service information and service authority scope information.

[0026] In some embodiments, specifically, in step E:

[0027] Step E1: When a user accesses a service, he / she enters the address of the space service proxy and the user's key;

[0028] Step E2: The service access request first reaches the service gateway, which retrieves the corresponding user information through the passed-in key, parses the service attribute information range and spatial information range of the spatial service requested by the current user, and encapsulates them into attribute information range parameter D and spatial information range parameter D according to the corresponding engine type;

[0029] Step E3: Forward the user request to the original address of the spatial service through the service routing; pass the encapsulated service attribute information range parameter D in the where parameter of the query interface of the spatial service; pass the encapsulated spatial information range parameter D in the Input Geometry parameter; finally obtain the output result and return it to the user through the service gateway address.

[0030] The method for rapid publishing and authentication of vector data proposed in the present invention has the following advantages over the prior art:

[0031] (1) It can quickly publish services for spatial vector data in the data lake and expand the rapid application of GIS spatial data in the big data environment;

[0032] (2) The present invention combines the spatial and attribute characteristics of geographic information data, achieves record-level control of spatial service data permissions, and improves the security of spatial service data during use;

[0033] (3) Provide a quick preview and authentication statistics page for published services.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention;

[0036] Figure 2 The spatial vector data format conversion mapping table in step A1 of the present invention;

[0037] Figure 3 A mapping table of the spatial vector data format conversion method in step A2;

[0038] Figure 4 It is not easy to follow the C2 process diagram;

[0039] Figure 5 This is the user rights parsing flow chart in step D;

[0040] Figure 6 This is a logical relationship diagram of the user, space service, and corresponding authority information constructed in step D5. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0042] like Figure 1 A method for rapid publishing and authentication of vector data is shown, step A: spatial data set conversion: traditional spatial vector data formats are usually SHP (ESRI Shapefile), CSV (Comma-Separated Values), GeoJson, GDB (GNU Debugger) and SDE (Spatial Database Engine) under the ESRI (Environmental Systems Research Institute) standard, etc. When checking the quality of spatial data stored in the data lake (Delta-Lake), firstly, the various spatial data sets to be stored in the data lake (Delta-Lake) are converted in format, and the spatial data spatial field types of points (spatial point features), polylines (spatial line features) and polygons (spatial surface features) in different formats are mapped and converted and stored; the details are as follows:

[0043] Step A1: According to the general format of spatial vector data, sort out the spatial field types of spatial data in different formats. All spatial fields will be uniformly stored in the data lake (Delta-Lake) as String type; the spatial vector data format conversion mapping table is as follows Figure 2 As shown, it should be noted that Figure 2 As shown, the existing spatial field type and spatial vector data format can also be stored in String type;

[0044] Step A2: According to different spatial vector data formats, different methods of Apache Sedona function library are used to parse and convert them into Delta two-dimensional tables for unified storage in the data lake (Delta-Lake), so as to realize spatial vector data management based on the data lake (Delta-Lake). The methods used for different spatial vector data are as follows: Spatial vector data format conversion method mapping table is as follows Figure 3 Shown

[0045] Note: Delta table is a table format defined by Data Lake (Delta-Lake), which is a two-dimensional table structure based on column storage.

[0046] Step B: Online service configuration: The service script is written in a mixture of SQL (Structured Query Language), JAVA (JavaScript) and Groovy. Groovy is an open source dynamic scripting language that runs on JVM. JAVA and Groovy can call each other, and Groovy can run directly on JVM (Java Virtual Machine) without compilation. Functions for accessing multiple types of data sources are pre-implemented and can be used directly in Groovy scripts to read data. After the service is written, based on the OGC (Open Geospatial Consortium) service standard, the corresponding script is converted to the GIS (Geographic Information System) spatial service standard, and the spatial field type check is performed (the spatial type field must be retained when publishing vector spatial services). After the script is converted, it will be published to the GeoServer service engine to form the original spatial service.

[0047] Step C: After the service is completed, it needs to be exposed to the outside world. At this time, two routing mapping operations are required.

[0048] Step C1: Register the service route in the HTTP (HyperText Transfer Protocol) container of the service instance. For example, if Tomcat (Tomcat is an open source Web application server) is used as the HTTP WEB container, it is necessary to dynamically bind the route to the container and set the service script that responds to the route.

[0049] Step C2: Register the service route that is actually exposed to the outside in the service gateway. In the cloud native environment, in order to achieve high availability and high concurrency of services, all request traffic will first be received and processed by the service gateway, and then forwarded to the upstream service instance to complete the request according to the service load balancing strategy set by the nginx (nginx is an open source gateway component) gateway. The specific diagram is as follows Figure 4 shown.

[0050] Step D: User authority analysis: Extract the service authority of each role that the user has, perform union calculation on the spatial information and attribute information contained in each authority, merge them into the authority scope of the user's corresponding service, and encrypt and store the record in the system database. The record is associated with the user's service key (apikey).

[0051] Core processes such as Figure 5 As shown: It is assumed that the user has two roles A and B. The attribute information authority scope of role A for spatial service 1 is attribute information scope A, and the spatial information authority scope is spatial information scope A; the attribute information authority scope of role B for spatial service 1 is attribute information scope B, and the spatial information authority scope is spatial information scope B. The specific steps are described as follows:

[0052] Step D1: extract the A role and B role possessed by the user and the scope of authority of these roles for service 1, including the scope of attribute information and the scope of spatial information.

[0053] Step D2: perform union calculation on the attribute information ranges of roles A and B to generate a new SQL configuration statement, namely, attribute information fusion range C, which is the attribute information authority range of the user for space service 1;

[0054] Step D3: Write the user's attribute information fusion range C for space service 1 into the base_info_limit field of the system rel_user_service (user and space association table);

[0055] Step D4: perform spatial union calculation on the spatial information ranges of role A and role B to generate a spatial information fusion range C, which is the user's spatial information authority range for spatial service 1;

[0056] Step D5: Construct the logical relationship between user information, associated service information, and service authority scope information, such as Figure 6 As shown, the user information table includes guaid, user ID, user name, user access key, and creation time; the user and space service association table includes guaid, role ID, service ID, service attribute information range C, service space information range C, and creation time; the service information table includes: guaid, service ID, service name, service proxy address, service original address, service type, and creation time.

[0057] The operation steps of step E are as follows:

[0058] Step E1: When a user accesses a service, he / she enters the address of the space service proxy and the user's key;

[0059] Step E2: The service access request will first go to the service gateway, where the corresponding user information is retrieved through the passed key, and then Figure 6 The mapping relationship in is used to parse the service attribute information range and spatial information range of the spatial service requested by the current user, and encapsulate them into attribute information range parameter D and spatial information range parameter D according to the corresponding engine type.

[0060] Step E3: Forward the user request to the original address of the spatial service through the service routing; pass the encapsulated service attribute information range parameter D in the where parameter of the query interface of the spatial service; pass the encapsulated spatial information range parameter D in the Input Geometry parameter; finally obtain the output result and return it to the user through the service gateway address.

[0061] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for rapid publishing and authentication of vector data, characterized in that: The steps include: Step A: Based on the data lake, traditional spatial vector data is mapped, converted and stored into spatial data field types; Step B: Select the service data stored in step A, write the data content online, and debug the service to form the original space service; Step C: register the original space service formed in step B to the service gateway, and the gateway will be the unified agent; Step D: Integrate the permissions of a single space service of the role owned by the user, and parse and calculate the scope of the user's permissions on the space service; Step E: Request space service according to the parsed data permission range and return service data content.

2. The method for rapid publication and authentication of vector data according to claim 1, characterized in that: In step A, the traditional spatial vector data includes SHP, CSV, GeoJson, GDB and SDE under the ESRI standard, and the traditional spatial vector data format is converted into String format.

3. The method for rapid publication and authentication of vector data according to claim 2, characterized in that: According to different spatial vector data formats, different methods of the Apache Sedona function library are used to parse and convert them into Delta tables for unified storage in the data lake, realizing spatial vector data management based on the data lake. The Delta table is a table format defined by the data lake and is a two-dimensional table structure based on column storage.

4. The method for rapid publishing and authentication of vector data according to claim 1, characterized in that: In the process of writing data in step B, the service script is written in a mixture of SQL, JAVA and Groovy. After writing, based on the OGC service standard, the corresponding script is converted into the GIS spatial service standard, and the spatial field type check is performed. The OGC service standard converts the corresponding script into the GIS spatial service standard, and the spatial field type check is performed. The converted script is published to the GeoServer service engine to form the original spatial service.

5. The method for rapid publishing and authentication of vector data according to claim 1, characterized in that: The specific exposure of the original space service in step C includes: C1: Register the service route in the HTTP container of the service instance; C2: Register the service routes that are actually exposed to the outside world in the service gateway.

6. The method for rapid publication and authentication of vector data according to claim 5, characterized in that: The service request traffic in C2 is received and processed by the gateway, and then forwarded to the service instance to complete the request according to the service load balancing strategy set by the nginx gateway.

7. The method for rapid publishing and authentication of vector data according to claim 1, characterized in that: In step D, the user has at least two roles. The service permissions of the user in each role are extracted, and the spatial information and attribute information contained in each permission are combined to form the permission range of the user's corresponding service. The record is encrypted and stored in the system database, and the record is associated with the user's service key.

8. The method for rapid publishing and authentication of vector data according to claim 7, characterized in that: Specifically in step D: it is assumed that the user has two roles A and B, the attribute information authority range of role A for spatial service 1 is attribute information scope A, and the spatial information authority range is spatial information scope A; the attribute information authority range of role B for spatial service 1 is attribute information scope B, and the spatial information authority range is spatial information scope B; the specific steps are described as follows: Step D1: extract the A role and B role possessed by the user and the scope of authority of these roles for service 1, including the scope of attribute information and the scope of spatial information; Step D2: perform union calculation on the attribute information ranges of role A and role B, and generate an SQL configuration statement, namely, attribute information fusion range C, which is the attribute information authority range of the user for space service 1; Step D3: Write the user's attribute information fusion range C for space service 1 into the base_info_limit field of rel_user_service; Step D4: perform spatial union calculation on the spatial information ranges of role A and role B to generate a spatial information fusion range C, which is the user's spatial information authority range for spatial service 1.

9. The method for rapid publishing and authentication of vector data according to claim 8, characterized in that: It also includes step D5: constructing a logical relationship between user information, associated service information, and service authority scope information.

10. The method for rapid publishing and authentication of vector data according to claim 1, characterized in that: Specifically in step E: Step E1: When a user accesses a service, he / she enters the address of the space service proxy and the user's key; Step E2: The service access request first reaches the service gateway, which retrieves the corresponding user information through the passed-in key, parses the service attribute information range and spatial information range of the spatial service requested by the current user, and encapsulates them into attribute information range parameter D and spatial information range parameter D according to the corresponding engine type; Step E3: forward the user request to the original address of the space service through the service routing; pass the encapsulated service attribute information range parameter D into the where parameter of the query interface of the space service; The encapsulated spatial information range parameter D is passed into the Input Geometry parameter; finally, the output result is obtained and returned to the user through the service gateway address.

Images