Data quantitative query method and system based on spatial grids

Abstract

The invention discloses a data quantitative query method and system based on spatial grids. According to the method, data of various types and formats are obtained in various modes, and the data are read into formats which can be recognized by a system. And then, analyzing and reconstructing the data, and completing cleaning and screening operations, including data translation and analysis, spatial data processing, JSON data sorting and the like. And performing precision unification and quantitative reconstruction on the processed data, and finally storing the processed data into a database and generating a query index. The method has efficient data processing and query capabilities, and can efficiently store and quickly retrieve large-scale spatial data.

Description

Technical Field

[0001] The present invention relates to the field of big data technology and algorithm technology, and in particular to a data quantitative query method and system based on spatial grid. Background Art

[0002] With the rapid development of information technology, spatial data has been widely used in many fields such as geographic information systems (GIS), satellite remote sensing, and location services. These spatial data are usually based on spatial coordinate systems (such as longitude and latitude, plane coordinates, etc.), and have significant geometric characteristics and practical significance. However, in practical applications, the storage and query of large-scale spatial data face many problems:

[0003] Too much data: The amount of data such as high-precision, high-dimensional geographic information or satellite images is extremely large, and direct storage and management place extremely high demands on hardware resources.

[0004] Difficulty in accurate query: When querying spatial data, it is usually necessary to search based on a single coordinate point. However, due to slight deviations in the coordinate acquisition process (such as measurement errors, approximate value conversion, etc.), completely accurate queries are difficult to achieve.

[0005] Statistical issues with multi-source data: Data from different sources have different precision and resolution, which means that when performing statistical analysis, directly operating on multi-source data often leads to problems such as duplicate statistics, data bias, or inaccurate statistical results.

[0006] Low query efficiency: When faced with massive spatial data, traditional database indexing technologies (such as RT tree, R-tree, etc.) significantly reduce query efficiency when processing high-dimensional data due to the dimensionality disaster phenomenon (dimensionality expansion problem). Summary of the invention

[0007] Based on this, an embodiment of the present application provides a data quantitative query method and system based on spatial grids. The method divides the geographic space into regular grid units (such as rectangles or squares) and dynamically adjusts the grid size according to application requirements, thereby achieving efficient storage and rapid retrieval of large-scale spatial data.

[0008] In a first aspect, a data quantitative query method based on a spatial grid is provided, the method comprising:

[0009] Acquire data of various types and formats, and read the data content into a format recognizable by the system; wherein acquiring data of various types and formats at least includes acquiring spatial data through local uploading, online database reading, data interface access, and message reception;

[0010] Parse and reconstruct the data content, and clean and filter the data; the data cleaning and filtering operations at least include data translation and parsing, spatial data processing, and JSON data combing;

[0011] Unify the precision of the processed data and reconstruct the data quantization;

[0012] The data after precision unification and data quantization reconstruction are stored in the database and a query index is formed.

[0013] Optionally, the acquiring of data of multiple types and formats includes:

[0014] Local file upload supports ShapeFile vector spatial files, GeoJSON vector spatial files, WKT vector spatial text, TIF raster spatial files, JSON files, and CSV table files;

[0015] Online database reading supports Postgres spatial tables, Postgres tables, Oracle tables, MySQL tables, SQLServer tables, and ShapeFile vector spatial files, GeoJSON vector spatial files, WKT vector spatial text, TIF raster spatial files, JSON files, and CSV table files from HDFS, FTP, and NFS;

[0016] Data interface access supports HTTP interface and HTTPS interface;

[0017] The message receiving method supports messages incoming from UDP, WebSocket, and MQTT.

[0018] Optionally, the data content is parsed and reconstructed, and the data is cleaned and screened, including:

[0019] Data translation and analysis: decrypt the encrypted data in plain text according to the provided translation and analysis methods;

[0020] Spatial data processing: processing and converting spatial data, including spatial type conversion, spatial location analysis, spatial data screening, and spatial calculation;

[0021] JSON data combing: read JSON data, format the data in a visual form and display it, and provide data extraction at the specified level.

[0022] Optionally, the spatial data processing specifically includes:

[0023] Perform spatial type conversion on data, converting spatial data of different formats into a unified spatial type;

[0024] Conduct spatial position analysis to determine the spatial position relationship of data;

[0025] Perform spatial data screening and filter out spatial data that meets the requirements according to the set conditions;

[0026] Perform spatial calculations and perform geometric or geographic calculations on spatial data.

[0027] Optionally, the step of unifying the precision of the processed data and reconstructing the data quantization includes:

[0028] Data merging: according to the precision and merging logic selected by the user, data is rasterized and merged with uniform precision, and data with too fine precision is unified in precision;

[0029] Data segmentation: divide the data into grids with uniform precision according to the precision selected by the user, and unify the precision of data with too coarse precision.

[0030] Optionally, storing the data after precision unification and data quantization reconstruction into a database and forming a query index includes:

[0031] Quantitatively reconstruct the processed data, extract the specified data content through automated algorithms and reconstruct it into a quantitative structure;

[0032] Store the quantified data into the specified quantification database;

[0033] Generate pyramid indexes in the quantized database to support efficient data query.

[0034] In a second aspect, a data quantitative query system based on a spatial grid is provided, the system comprising:

[0035] An acquisition module is used to acquire data of various types and formats and read the data content into a format recognizable by the system; wherein acquiring data of various types and formats at least includes acquiring spatial data through local uploading, online database reading, data interface access, and message reception;

[0036] The processing module is used to parse and reconstruct the data content, and clean and filter the data; wherein the operations of cleaning and filtering the data at least include data translation and parsing, spatial data processing, and JSON data combing;

[0037] The reconstruction module is used to unify the precision of the processed data and reconstruct the data quantization;

[0038] The query module is used to store the data after precision unification and data quantization reconstruction into the database and form a query index.

[0039] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the data quantitative query method described in any one of the first aspects is implemented.

[0040] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data quantitative query method described in any one of the first aspects is implemented.

[0041] In a fifth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements any of the data quantitative query methods described in the first aspect.

[0042] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0043] (1) Through spatial rasterization and precision unification processing, multi-source heterogeneous data is converted into a unified format, solving the problems of inconsistent data precision and low query efficiency. At the same time, the system supports high-concurrency and multi-dimensional queries, significantly improving the speed and accuracy of data retrieval, and meeting the real-time query needs in large-scale spatial data application scenarios.

[0044] (2) It supports multiple data sources and formats, including local files, online databases, data interface access, and message reception, and can process data in multiple formats such as ShapeFile, GeoJSON, TIF, JSON, and CSV. This enables the system to flexibly adapt to data application requirements in different fields, and has wide applicability and good scalability.

[0045] (3) Through data quantization reconstruction and index generation, massive data can be efficiently stored in the database, which not only saves storage space but also improves data management efficiency. At the same time, the system can dynamically adjust the grid size, further optimize the data storage structure, reduce hardware resource requirements, and improve the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0047] Figure 1 A flowchart of a method for quantitative data query based on spatial grid provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the system usage and internal flow provided in the embodiment of the present application;

[0049] Figure 3 A block diagram of a data quantization query system based on a spatial grid provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0053] The present invention proposes a quantitative query technology and system based on spatial grids to solve the problems encountered during data storage, query and statistics due to different usage scenarios, different data sources and different data formats, such as too much data storage due to too fine data precision; the inability to find the corresponding results due to the deviation between query conditions and data due to too fine data; the inability to perform efficient combined query statistics due to multiple data formats and different precisions; and the slow spatial query speed when the data volume is large.

[0054] In order to achieve the above purpose, the present invention provides the following technical solutions, please refer to Figure 1 , which shows a flow chart of a data quantization query method based on a spatial grid provided in an embodiment of the present application, which may include the following steps:

[0055] S1, obtains data of various types and formats, and reads the data content into a format that the system can recognize.

[0056] Among them, obtaining data of various types and formats at least includes obtaining spatial data through local uploading, online database reading, data interface access, and message reception.

[0057] In this step, spatial data is obtained through a variety of methods, including local upload, online database reading, data interface access, and message reception. Local upload supports common local file formats such as ShapeFile vector spatial files, GeoJSON vector spatial files, WKT vector spatial text, TIF raster spatial files, JSON files, CSV table files, etc.; online database reading covers Postgres spatial tables, Postgres tables, Oracle tables, MySQL tables, SQLServer tables, and the above file formats stored in HDFS, FTP, and NFS; data interface access supports HTTP interface, HTTPS interface, etc., and can receive interface data from different data sources; message reception methods support UDP, WebSocket, MQTT, etc., and can receive spatial data transmitted through message queues and other methods in real time. These acquisition methods ensure that the system can receive data from different sources, in different types and formats, and read it into a format that the system can recognize, laying the foundation for subsequent processing.

[0058] S2, parses and reconstructs the data content, and cleans and filters the data.

[0059] Among them, the operations of cleaning and screening data include at least data translation and analysis, spatial data processing, and JSON data combing.

[0060] After acquiring the data, the data content is parsed and reconstructed, and cleaning and screening operations are performed at the same time. Specifically, the data translation and parsing link decrypts the encrypted data in plain text according to the provided method to ensure the readability of the data; spatial data processing covers operations such as spatial type conversion, spatial position analysis, spatial data screening, and spatial calculation. Through these processes, spatial data of different formats and precisions are converted into a unified format, and data that meets the requirements is screened out. At the same time, its spatial position relationship is analyzed to provide accurate spatial information for subsequent processing; the JSON data combing part reads data in JSON format, displays the data structure in a visual form, and provides functions such as data extraction at a specified level to facilitate the processing and analysis of complex JSON data. These operations together complete the cleaning and screening of the data, remove invalid or redundant data, retain valuable information, and improve the quality and availability of the data.

[0061] S3, unify the precision of the processed data and reconstruct the data quantization.

[0062] This step unifies the precision and reconstructs the quantization of the cleaned and screened data. The data merging operation rasterizes and merges the data with too fine precision according to the precision and merging logic set by the user to reduce the redundancy and complexity of the data; the data segmentation is for the data with too coarse precision, and rasterizes and segments it according to the uniform precision to achieve the required accuracy level. Through these two operations, the consistency of data accuracy is ensured, providing a unified data foundation for subsequent storage and query. Subsequently, the system quantitatively reconstructs the data, uses automated algorithms to extract the specified data content, and reconstructs it into a quantitative structure, further optimizing the organization of the data and preparing for efficient data storage and query.

[0063] S4, stores the data after precision unification and data quantization reconstruction into the database and forms a query index.

[0064] After the data is unified in precision and quantitatively reconstructed, the system stores it in a dedicated database. In order to support efficient query operations, the system generates a pyramid index in the database. This index structure can quickly locate and retrieve data, significantly improve query efficiency, and meet users' requirements for real-time and accuracy of data query results.

[0065] In an optional embodiment of the present application, the method can also be expanded into the following specific implementation process, such as: Figure 2 A schematic diagram of system usage and internal flow is given:

[0066] Data of various types and formats can be obtained through local upload, online database reading, data interface access, message reception, etc., and the data content can be read into a format that the system can recognize. Parse and reconstruct the data content, and complete data cleaning, screening and other operations through a combination of various processing methods. After the processed data is unified in accuracy and quantitatively reconstructed, it is stored in the system query engine database and forms a query index.

[0067] After completing the above steps, users can use the query engine to perform efficient and high-concurrency spatial queries on the stored data to ensure the real-time and accuracy of data query results.

[0068] Furthermore, the data source method and the types and formats supported for reading include:

[0069] Local file upload supports local ShapeFile vector space files, GeoJSON vector space files, WKT vector space text, TIF raster space files, JSON files, CSV table files, etc. Online database reading supports online Postgres spatial tables, Postgres tables, Oracle tables, MySQL tables, SQLServer tables, or ShapeFile vector space files, GeoJSON vector space files, WKT vector space text, TIF raster space files, JSON files, CSV table files, etc. from HDFS, FTP, NFS. Data interface access supports HTTP interface, HTTPS interface, etc. Message receiving methods support messages incoming from message sources such as UDP, WebSocket, MQTT, etc.

[0070] Furthermore, after the data is read, it can be processed by various processing methods provided by the system, including:

[0071] Data translation and analysis: decrypt the encrypted data in plain text according to the provided translation and analysis methods.

[0072] Spatial data processing, processing and converting spatial data, including spatial type conversion, spatial location analysis, spatial data screening, spatial calculation, etc.

[0073] JSON data combing: read JSON data, format the data and display it in a visual form, and provide methods such as extracting data at a specified level.

[0074] Furthermore, after the data is processed, it can be spatially rasterized and its accuracy unified, including:

[0075] Data merging: according to the precision and merging logic selected by the user, data will be rasterized and merged with uniform precision, and data with too fine precision will be unified in precision.

[0076] Data segmentation: divide the data into grids with uniform precision according to the precision of data selection, and unify the precision of data with too coarse precision.

[0077] Furthermore, the data is quantified, reconstructed and stored in the database, as follows:

[0078] The processed data is quantified and reconstructed, and the specified data content is extracted and reconstructed into a quantitative structure through an automated algorithm. The quantified data is then stored in a specified quantitative database and a pyramid index is generated.

[0079] Finally, users can use the system’s data search engine to perform high-concurrency, multi-dimensional, and high-efficiency data queries and result returns.

[0080] Please refer to Figure 3 , which shows a block diagram of a data quantization query system based on a spatial grid provided by an embodiment of the present application. Figure 3 As shown, the system may include:

[0081] An acquisition module is used to acquire data of various types and formats and read the data content into a format recognizable by the system; wherein acquiring data of various types and formats at least includes acquiring spatial data through local uploading, online database reading, data interface access, and message reception;

[0082] The processing module is used to parse and reconstruct the data content, and clean and filter the data; wherein the operations of cleaning and filtering the data at least include data translation and parsing, spatial data processing, and JSON data combing;

[0083] The reconstruction module is used to unify the precision of the processed data and reconstruct the data quantization;

[0084] The query module is used to store the data after precision unification and data quantization reconstruction into the database and form a query index.

[0085] For the specific definition of the data quantitative query system based on spatial grid, please refer to the definition of the data quantitative query method based on spatial grid above, which will not be repeated here. Each module in the above-mentioned data quantitative query system based on spatial grid can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0086] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 4 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to quantify and query data based on spatial grid data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data quantification query method based on spatial grid is implemented.

[0087] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0088] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which involves all or part of the processes in the above-mentioned embodiment method.

[0089] In one embodiment, a computer program product is also provided, including a computer program / instruction, which involves all or part of the process in the above embodiment method.

[0090] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A data quantitative query method based on spatial grid, characterized in that: The method comprises: Acquire data of various types and formats, and read the data content into a format recognizable by the system; wherein acquiring data of various types and formats at least includes acquiring spatial data through local uploading, online database reading, data interface access, and message reception; Parse and reconstruct the data content, and clean and filter the data; the data cleaning and filtering operations at least include data translation and parsing, spatial data processing, and JSON data combing; Unify the precision of the processed data and reconstruct the data quantization; The data after precision unification and data quantization reconstruction are stored in the database and a query index is formed.

2. The data quantitative query method according to claim 1, characterized in that: The acquisition of data of various types and formats includes: Local file upload supports ShapeFile vector spatial files, GeoJSON vector spatial files, WKT vector spatial text, TIF raster spatial files, JSON files, and CSV table files; Online database reading supports Postgres spatial tables, Postgres tables, Oracle tables, MySQL tables, SQLServer tables, and ShapeFile vector spatial files, GeoJSON vector spatial files, WKT vector spatial text, TIF raster spatial files, JSON files, and CSV table files from HDFS, FTP, and NFS; Data interface access supports HTTP interface and HTTPS interface; The message receiving method supports messages incoming from UDP, WebSocket, and MQTT.

3. The data quantitative query method according to claim 1, characterized in that: Parse and reconstruct the data content, and clean and filter the data, including: Data translation and analysis: decrypt the encrypted data in plain text according to the provided translation and analysis methods; Spatial data processing: processing and converting spatial data, including spatial type conversion, spatial location analysis, spatial data screening, and spatial calculation; JSON data combing: read JSON data, format the data in a visual form and display it, and provide data extraction at the specified level.

4. The data quantitative query method according to claim 3, characterized in that: Spatial data processing specifically includes: Perform spatial type conversion on data, converting spatial data of different formats into a unified spatial type; Conduct spatial position analysis to determine the spatial position relationship of data; Perform spatial data screening and filter out spatial data that meets the requirements according to the set conditions; Perform spatial calculations and perform geometric or geographic calculations on spatial data.

5. The data quantitative query method according to claim 1, characterized in that: The step of unifying the precision of the processed data and reconstructing the data quantization comprises: Data merging: according to the precision and merging logic selected by the user, data is rasterized and merged with uniform precision, and data with too fine precision is unified in precision; Data segmentation: divide the data into grids with uniform precision according to the precision selected by the user, and unify the precision of data with too coarse precision.

6. The data quantitative query method according to claim 1, characterized in that: The step of storing the data after precision unification and data quantization reconstruction in a database and forming a query index includes: Quantitatively reconstruct the processed data, extract the specified data content through automated algorithms and reconstruct it into a quantitative structure; Store the quantified data into the specified quantification database; Generate pyramid indexes in the quantized database to support efficient data query.

7. A data quantitative query system based on spatial grid, characterized in that: The system comprises: An acquisition module is used to acquire data of various types and formats and read the data content into a format recognizable by the system; wherein acquiring data of various types and formats at least includes acquiring spatial data through local uploading, online database reading, data interface access, and message reception; The processing module is used to parse and reconstruct the data content, and clean and filter the data; wherein the operations of cleaning and filtering the data at least include data translation and parsing, spatial data processing, and JSON data combing; The reconstruction module is used to unify the precision of the processed data and reconstruct the data quantization; The query module is used to store the data after precision unification and data quantization reconstruction into the database and form a query index.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images