A method, apparatus, and storage medium for loading massive amounts of data into a web-based electronic map.

Abstract

This invention relates to a method, apparatus, and storage medium for loading massive amounts of data into a web electronic map. The method includes the following steps: on the database side, splitting massive map layer data into metadata and business data; on the server side, extracting, compressing, and transmitting the metadata and business data from the database; on the web browser client, obtaining data through an API interface; first, loading all map layer metadata; starting multiple threads to obtain business data for each layer; after each sub-thread obtains business data, parsing the business data in conjunction with the layer metadata, and feeding the parsed data back to the main thread; batch rendering the parsed data to form an electronic map, and establishing a spatial index for subsequent functional business. This invention solves the problems of excessively long network request times and slow rendering in web browser clients caused by the large amount of data loaded in a single batch when loading map data using traditional methods.

Description

Technical Field

[0001] This invention relates to the field of Web electronic map technology, and in particular to a method, apparatus and storage medium for loading massive amounts of data into a Web electronic map. Background Technology

[0002] Web-based electronic maps, also known as digital maps, are maps stored and viewed digitally using computer technology. Currently, the application of electronic maps based on HTML5 browsers is becoming increasingly widespread. The vector data loaded onto electronic maps includes spatial information elements such as points, polylines, and polygons, which include spatial coordinates and attribute information. Existing technical solutions typically use string formats such as geojson, wkt, and kml to load this vector data, which are either uncompressed or have low compression rates. When the data volume reaches 500,000 records or more, limitations imposed by server performance, network bandwidth, and client browser performance lead to excessively large data transfer volumes. This results in long page loading times, browser lag, freezing, rendering failures, inability to perform map display and roaming operations, slow spatial calculations, slow query result retrieval, and difficulty in quickly and accurately locating specific elements when the user selects them with the mouse, severely impacting the user experience. Summary of the Invention

[0003] In order to achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide a method for loading massive amounts of data into a Web electronic map, comprising the following steps:

[0004] The map layer data in the database is split into metadata and business data and stored in a spatial database; wherein, the metadata is data describing the attributes of the map layers, and the business data is the data value of each layer;

[0005] The server extracts and processes the metadata and business data of the map layer from the spatial database;

[0006] The web browser client obtains the processed metadata and business data from the server.

[0007] Load all map layer metadata;

[0008] Start multiple threads to retrieve business data for each layer;

[0009] After each sub-thread obtains the business data, it parses the business data in conjunction with the layer metadata and feeds the parsed data back to the main thread.

[0010] The batch rendering and parsing of data forms an electronic map and establishes a spatial index.

[0011] Furthermore, the metadata includes layer name, attribute name, attribute type, and attribute length;

[0012] The business data includes spatial geometric data and attribute data.

[0013] Furthermore, the processed business data obtained by the web browser client is binary data that is processed by the server-side compression algorithm according to the definition of the metadata and returned to the client.

[0014] Furthermore, the server transmits the processed metadata and business data to the client via API.

[0015] Furthermore, the compression process performed on the server side after extracting the metadata and business data of the map layer from the spatial database includes the following steps:

[0016] Categorize layer attributes;

[0017] Different compression methods are used for different types of layer attributes;

[0018] Lossless compression algorithms are used to compress business data.

[0019] Furthermore, the classification of layer attributes specifically involves classifying layer attribute types into strings, enumeration values, integers, and floating-point numbers;

[0020] The method of using different compression processing for different types of layer attributes includes the following steps:

[0021] If the business data is string data, determine whether the string values ​​of the entire layer can be classified as less than the preset value. If so, use enumerated values ​​for processing; otherwise, process according to the set delimiter.

[0022] If the business data is enumerated value data, it will be processed according to the attribute field name, enumerated value, and enumerated value index;

[0023] If the business data is integer data, determine whether the integer values ​​of the entire layer can be classified as less than a preset value. If so, use enumeration value processing; otherwise, use compression algorithm processing.

[0024] If the business data is a floating-point value, it will be converted to an integer value for processing.

[0025] Furthermore, the compression algorithm processing includes the following steps:

[0026] Find the maximum and minimum values ​​of all data, and calculate the minimum number of digits that can represent the range from the minimum to the maximum value;

[0027] Store each integer value using the minimum number of bits.

[0028] Furthermore, after each sub-thread obtains the business data, it parses the business data in conjunction with the layer metadata and feeds the parsed data back to the main thread. Specifically, after the Web browser client obtains the binary business data, it restores the original data according to the metadata definition; wherein, the parsing algorithm is the inverse method of the server-side compression algorithm.

[0029] A second objective of this invention is to provide an apparatus for loading massive amounts of data onto a web electronic map, comprising: a memory storing program code thereon; and a processor connected to the memory, wherein when the program code is executed by the processor, a method for loading massive amounts of data onto a web electronic map is implemented.

[0030] A third objective of this invention is to provide a computer-readable storage medium having program instructions stored thereon, which, when executed, implement a method for loading massive amounts of data onto a Web electronic map.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] This invention provides a method, electronic device, and storage medium for loading massive amounts of data into a web-based electronic map. Based on the unique data structure characteristics of electronic map data, a data processing algorithm is designed to solve the problems of excessively long network request times and slow rendering in web browser clients caused by the large batch size of data when loading map data using traditional methods. This invention can reduce data size by more than 70%, improve network transmission efficiency, and use multi-threading technology to load data in layers, eliminating map display lag.

[0033] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0035] Figure 1 This is a flowchart of a method for loading massive amounts of data into a Web electronic map, as described in Example 1.

[0036] Figure 2 This is a flowchart of the data compression process on the server side.

[0037] Figure 3This is a schematic diagram of the device for loading massive amounts of data onto a Web electronic map in Example 2.

[0038] Figure 4 This is a block diagram illustrating the principle of a computer-readable storage medium in Example 3. Detailed Implementation

[0039] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0040] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0041] Example 1

[0042] A method for loading massive amounts of data into a web-based electronic map is proposed, which designs a data processing algorithm based on the unique data structure characteristics of electronic map data. Specifically, such as... Figure 1 As shown, it includes the following steps:

[0043] To address the issue of slow network transmission due to the large volume of layer data, data preprocessing is performed. Specifically, at the database end, the massive map layer data is split into metadata and business data and stored in a spatial database, such as PostGIS. Metadata describes the attributes of map layers, including layer name, attribute name, attribute type, and attribute length, as shown in Table 1, which provides an example of metadata for the municipal sewage pipe network layer. Business data consists of specific data values ​​for each layer, including spatial geometric data and attribute data, and is a unique structural component of electronic map data.

[0044] Table 1. Example of metadata for municipal sewage pipe network layers.

[0045] Attribute Name Attribute type Attribute length coordinate X floating-point numbers 8 bytes coordinate Y floating-point numbers 8 bytes ID Integer 8 bytes Well bottom depth floating-point numbers 8 bytes Well type Enumeration value 20 Remark String 100

[0046] On the server side, metadata and business data are extracted and compressed from the spatial database. An API interface is developed on the server side to transmit the data externally; that is, the compressed data is transmitted to the web browser client through the developed API interface. The specific compression process includes the following steps:

[0047] Categorize layer attributes; such as Figure 2 As shown, layer attribute types are divided into string, enumeration value, integer, and floating-point number;

[0048] Different compression methods are used for different types of layer attributes; among them,

[0049] If the business data is string data, determine whether all string values ​​in the layer can be categorized into those less than a preset value. If the preset value is set to 20, then use enumeration values ​​for processing, such as classifying pipe network types into concrete pipes, reinforced concrete pipes, clay pipes, PE (polyethylene) pipes, etc.; otherwise, process them according to the set delimiter.

[0050] If the business data is enumerated value data, it will be processed according to the attribute field name, enumerated value, and enumerated value index;

[0051] If the business data is integer data, determine whether all integer values ​​in the layer can be categorized as being less than a preset value. If the preset value is set to 20, enumerate the values; otherwise, perform compression processing. The compression processing includes the following steps:

[0052] Find the maximum and minimum values ​​of all data, and calculate the minimum number of bits that can represent the range from the minimum to the maximum value.

[0053] Store each integer value using the minimum number of bits.

[0054] If the business data is floating-point value, it will be converted to integer data for processing.

[0055] In particular, the geometric objects in the layers are almost all stored in floating-point format, which can reduce the data volume by more than 70% when the data volume is large.

[0056] Then, lossless compression algorithms, such as gzip, are used to further compress the business data.

[0057] The web browser client retrieves processed metadata and business data from the server via an API interface. The processed business data retrieved by the web browser client is binary data returned to the client after being processed by the server-side compression algorithm according to the metadata definition. A detailed description of the compression process can be found in the corresponding description of server-side data compression above, and will not be repeated here.

[0058] Before the web browser client loads and renders the map, it first loads all map layer metadata, that is, it obtains data transmitted from the server through the network API interface, and the same applies to business data.

[0059] The web browser client starts multiple threads to process the business data of each layer;

[0060] After each sub-thread of the web browser client obtains the business data, it parses the data in conjunction with the layer metadata and feeds the parsed data back to the main thread of the web browser client. Specifically, after the web browser client obtains the binary business data through the network transmission API interface, it restores the original data according to the metadata definition. It should be understood that the parsing algorithm here is the inverse method of the compression algorithm on the server side mentioned above. Referring to the corresponding description of the server-side data compression process, the binary business data can be restored to the original data by following its corresponding inverse process, which will not be elaborated here. The web browser client batch renders the parsed data to form an electronic map and establishes a spatial index for subsequent functional business.

[0061] In practical applications, typically all spatial geometry data and a small amount of important attribute data are loaded, while less important attributes are queried and loaded separately in later operations. The handling of floating-point numbers for spatial geometry data is as follows:

[0062] When the spatial location x-coordinate is distributed in the range of 13190956.951 to 13201583.804 (web Mercator projection coordinate system, latitude and longitude range 18°29'46.977"-118°35'30.642"), if traditional string methods such as geojson, wkt, and kml are used, each x-coordinate number occupies 12 bytes; if binary 64-bit floating-point methods are used, it occupies 8 bytes. However, using the method provided in this invention, the difference between the minimum and maximum values ​​is first calculated to obtain a baseline integer value: (13201583.804-13190956.951)*1000 yields 10626853 as the baseline. Multiplying the difference by 1000 converts the coordinate system values ​​to integers, and storing the original coordinate values ​​with 3 decimal places satisfies the accuracy requirements. Next, calculating the minimum number of bits, log2(10626853) is 23.34, which is rounded down to 24 bits. Therefore, only 24 bits are needed to represent all integers in the range 0 to 10626853. 24 bits occupy 3 bytes (1 byte occupies 8 bits). This processing saves 62.5% of the space compared to integers and 75% compared to strings, greatly optimizing the user experience.

[0063] This invention can reduce data size by more than 70%, improve network transmission efficiency, and use multi-threading technology to load data in layers, eliminating map display lag.

[0064] Example 2

[0065] A device 200 for loading massive amounts of data onto a web-based electronic map, such as Figure 3As shown, the system includes: a memory 201 storing program code; and a processor 202 connected to the memory, which, when executed by the processor, implements a method for loading massive amounts of data onto a web electronic map. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0066] Example 3

[0067] A computer-readable storage medium, such as Figure 4 As shown, it stores program instructions, which, when executed, implement a method for loading massive amounts of data into a web electronic map. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0068] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0069] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0070] The above are merely embodiments of this specification and are not intended to limit the scope of one or more embodiments of this specification. For those skilled in the art, various modifications and variations can be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for loading mass data of a Web electronic map, characterized in that, The method comprises the following steps: The map layer data in the database is split into metadata and business data, and stored in a spatial database; wherein the metadata is data describing the attributes of the map layer, and the business data is the data value of each layer; The server extracts and processes the metadata and business data of the map layer from the spatial database; The web browser client obtains the processed metadata and business data from the server; All map layer metadata is loaded; Multi-threading is started to obtain the business data of each layer; After each sub-thread obtains the business data, the business data is parsed in combination with the layer metadata, and the parsed data is fed back to the main thread; The parsed data is batch-rendered to form an electronic map, and a spatial index is established; The compression processing of the metadata and business data of the map layer extracted from the spatial database by the server comprises the following steps: The layer attributes are classified; Different compression processing methods are adopted for different types of layer attributes; Lossless compression algorithm is adopted to compress the business data; The classification of the layer attributes specifically includes classifying the layer attribute types into strings, enumeration values, integers, and floating-point numbers; The different compression processing methods for different types of layer attributes comprise the following steps: If the business data is string data, it is judged whether the string values of the entire layer can be classified as less than a preset value; if yes, enumeration value processing is adopted, otherwise, the values are processed according to the set separator; If the business data is enumeration value data, it is processed according to the attribute field name, enumeration value, and enumeration value index; If the business data is integer data, it is judged whether the integer values of the entire layer can be classified as less than a preset value; if yes, enumeration value processing is adopted, otherwise, compression algorithm processing is performed; If the business data is floating-point value data, it is uniformly converted into integer data for processing; The compression algorithm processing comprises the following steps: The maximum value and the minimum value of all data are taken, and the minimum number of bits representing the minimum value to the maximum value range is calculated; Each integer value is stored using the minimum number of bits.

2. The method for loading mass data of Web electronic map according to claim 1, characterized in that: The metadata comprises the layer name, attribute name, attribute type, and attribute length; The business data comprises spatial geometric data and attribute data.

3. The method for loading mass data of Web electronic map according to claim 1, characterized in that: The processed business data obtained by the web browser client is binary data returned to the client after being processed by the server using the compression algorithm according to the definition of the metadata.

4. The method for loading mass data of Web electronic map according to claim 1, wherein, The server transmits the processed metadata and business data to the client through an API transmission method.

5. The method for loading mass data of Web electronic map according to claim 1, wherein: After each sub-thread obtains the business data, the business data is parsed in combination with the layer metadata, and the parsed data is fed back to the main thread; specifically, after the web browser client obtains the binary business data, the data is restored to the original data according to the metadata definition; wherein the parsing algorithm is the inverse method of the server compression algorithm.

6. An apparatus for loading mass data of a Web electronic map, characterized in that, A memory having program codes stored thereon; A processor coupled to the memory, and when the program codes are executed by the processor, the method of claim 1 is implemented. A memory having program codes stored thereon; 7. A computer-readable storage medium, characterized in that, ​

Images