A High-Performance and Intelligent Retrieval Method for Massive Spatial Data

Abstract

This invention relates to a high-performance and intelligent retrieval method for massive spatial data, comprising the following steps: reading data tables from a database, allocating different data tables to different nodes using a bucketing algorithm, simultaneously reading data from data tables in different nodes in a multi-threaded manner and constructing search engine document data, and synchronously writing the search engine document data in different nodes into search engine index data; during retrieval, the search engine parses the business logic, constructs search engine retrieval syntax, calls search engine query services, and queries search engine index data.

Description

Technical Field

[0001] This invention relates to a high-performance and intelligent retrieval method for massive spatial data, belonging to the field of geographic information systems. Background Technology

[0002] GIS, an interdisciplinary field that emerged after computer and spatial data analysis methods were applied to many related disciplines, has timely capitalized on the rapid development of computer technology and the increasing emphasis placed on geographic, resource, and environmental information by governments worldwide. Furthermore, the availability of powerful geospatial information acquisition tools from related technologies such as GPS, DPS, and RS has made 3D GIS a hot topic of interest for government departments, businesses, research institutions, and universities globally. Especially since the 1990s, GIS has developed into a global industry and is poised to further penetrate various sectors and even people's daily lives.

[0003] Currently, the core GIS engine of the power grid GIS platform provides vector-tiled graphic rendering capabilities and electrical topology-based analysis and calculation capabilities. The equipment query capability, which is more general, is currently implemented based on database SQL technology. Therefore, the equipment query service interface in the current power grid GIS platform public service requires the caller to explicitly specify the equipment type and related equipment attribute conditions, which is not flexible or easy to use. At the same time, the retrieval performance is insufficient for the hundreds of millions of data in the power grid and provinces and the 600 million equipment data in the headquarters. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a high-performance and intelligent retrieval method for massive spatial data.

[0005] The technical solution of the present invention is as follows:

[0006] On the one hand, this invention provides a high-performance and intelligent retrieval method for massive spatial data, comprising the following steps:

[0007] Read data tables from the database, distribute different data tables to different nodes using a bucketing algorithm, and simultaneously perform streaming partitioning and data reading on data tables in different nodes in a multi-threaded manner to build search engine document data. Simultaneously write search engine document data from different nodes into search engine index data.

[0008] When reading data from data tables on different nodes using multiple threads, there is an uneven distribution of data table sizes. By integrating a small message queue capability, the large data table is parsed and split, and the split large data table is allocated to the idle threads that have finished reading data from the small data table, ensuring that the thread pool is saturated.

[0009] When performing a search, the search engine parses the business logic, constructs the search engine retrieval syntax, calls the search engine query service, and queries the search engine index data.

[0010] In a preferred embodiment of the present invention, a data model driver is provided, which is used to remove irrelevant data tables from the database;

[0011] The data model driver has a built-in model driver table, which is used to record the data tables in the database that are allowed to be written to the search engine.

[0012] When reading data from the database, the data model driver is initialized, the model driver table information is completed, the data tables that are allowed to be written to the search engine are selected based on the model driver table, and different data tables are assigned to different nodes.

[0013] In a preferred embodiment of the present invention, when data in the database is updated, the data in the search engine is updated incrementally. The specific steps are as follows:

[0014] Based on Flink CDC, the database log information is monitored to obtain information on addition, deletion, and modification actions in the database. This information is then transmitted to the message queue in the Kafka messaging system. The search engine subscribes to the Kafka messaging system and updates its index data according to the message queue order in the system.

[0015] In a preferred embodiment of the present invention, each node is equipped with a log to record the health status of the node, and maintenance personnel can locate the node that needs maintenance by means of the log.

[0016] On the other hand, the present invention also provides a high-performance and intelligent retrieval system for massive spatial data, including a data reading module, a search engine and a retrieval module;

[0017] The data reading module is used to read data tables in the database, distribute different data tables to different nodes through a bucketing algorithm, read data from data tables in different nodes simultaneously in a multi-threaded manner and build search engine document data, and synchronously write search engine document data in different nodes into search engine index data.

[0018] The search engine has a built-in query service, which allows users to query corresponding index data.

[0019] The retrieval module is used to parse search operations, construct search engine retrieval syntax, call search engine query services, and query search engine index data.

[0020] In a preferred embodiment of the present invention, the data reading module is equipped with a data model driver, which is used to remove irrelevant data tables from the database;

[0021] The data model driver has a built-in model driver table, which is used to record the data tables in the database that are allowed to be written to the search engine.

[0022] When reading data from the database, the data model driver is initialized, the model driver table information is completed, the data tables that are allowed to be written to the search engine are selected based on the model driver table, and different data tables are assigned to different nodes.

[0023] In another aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any embodiment of the present invention.

[0024] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present invention.

[0025] The present invention has the following beneficial effects:

[0026] 1. This invention completely solves the problems of poor flexibility, ease of use, and insufficient performance in current power grid equipment retrieval. It provides users at different levels, such as provincial, municipal, and county-level units, with intelligent and high-performance retrieval capabilities that are highly matched to current business scenarios, such as line inspection, field survey, and map-based operations. It provides standardized, flexible, easy-to-use, and efficient equipment retrieval services for various applications, making business systems more user-friendly and power grid GIS platforms more practical. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention;

[0028] Figure 2 This is a schematic diagram of the multi-threading principle of the present invention;

[0029] Figure 3 This is a flowchart of the incremental update process of the present invention;

[0030] Figure 4 This is a specific application diagram of the present invention;

[0031] Figure 5 This is a system block diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0034] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0035] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0036] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0037] Example 1:

[0038] See Figure 1 A high-performance and intelligent retrieval method for massive spatial data includes the following steps:

[0039] To read data tables from the database, see [link / reference]. Figure 2 The bucketing algorithm distributes different data tables to different nodes. The multi-threaded approach simultaneously reads data from the data tables on different nodes in a streaming partition and builds search engine document data. The search engine document data on different nodes is then synchronously written into the Elasticsearch search engine index data.

[0040] When reading data from data tables on different nodes using multiple threads, there is an uneven distribution of data table sizes. By integrating a small message queue capability, the large data table is parsed and split, and the split large data table is allocated to the idle threads that have finished reading data from the small data table, ensuring that the thread pool is saturated.

[0041] Currently, the power grid data in the database is unevenly distributed, with large and small tables. The largest table has reached a capacity of 10GB. If a simple query is performed, it can easily cause the program to freeze and become unusable. First, data is read from a single table using streaming partitioning to obtain the corresponding power grid data. Then, the read data is written to the retrieval engine through multi-threaded batch requests. During this process, due to the inconsistent table sizes and the presence of large tasks, some threads are idle, failing to fully utilize machine resources and resulting in resource waste. Therefore, the application integrates a small message queue capability, which allows for deep splitting of large tasks, ensuring that the threads in the thread pool are all in a saturated state, thereby maximizing the use of machine space and performing high-performance data synchronization.

[0042] When performing a search, the Elasticsearch search engine parses the business logic, constructs the search engine's search syntax, calls the Elasticsearch query service, and queries the search engine's index data.

[0043] In a preferred embodiment of this example, a data model driver is set up, which is used to remove irrelevant data tables from the database;

[0044] The data model driver has a built-in model driver table, which is used to record the data tables in the database that are allowed to be written to the search engine.

[0045] When reading data from the database, the data model driver is initialized, the model driver table information is completed, the data tables that are allowed to be written to the search engine are selected based on the model driver table, and different data tables are assigned to different nodes.

[0046] Because power grid equipment data varies across different provinces, it directly impacts the scope of the retrieved data. Therefore, a model-driven table was designed to flexibly match the retrieval needs of different provinces, describing which tables' data can be included in the retrieval engine, thus defining the data range. During the data model-driven initialization process, relevant important information is supplemented, improving engine performance during subsequent retrieval service synchronization, reducing unnecessary queries, and alleviating database pressure. Furthermore, the data model-driven approach also provides task allocation across different nodes, matching data content from different tables to different nodes, thereby improving synchronization efficiency.

[0047] As a preferred embodiment of this example, see Figure 3Information on power grid resources and equipment changes frequently, generating incremental data. This changed equipment information needs to be synchronized to the search engine with low latency to support data retrieval. Incremental data updates are achieved by Flink CDC collecting data from PostgreSQL's WAL logs, monitoring data anomalies in the power grid resource table, parsing Insert, Update, and Delete operations, generating relevant JSON data, and streaming it to a Kafka message queue. The massive data search engine subscribes to Kafka's anomaly messages, ensuring data consistency. This approach offers advantages such as low latency, application decoupling, and no database access pressure.

[0048] By leveraging Flink CDC's operational mechanism, database log information can be asynchronously monitored to obtain information on actions such as adding, deleting, and modifying data. This does not incur additional overhead for database access, thus reducing database access pressure. The captured data is written to the message middleware via direct connection, allowing each application service to consume messages according to its own business needs. This truly achieves application decoupling and enables high-performance, high-throughput capture and monitoring of data anomalies.

[0049] In a preferred embodiment of this invention, each node is equipped with a log to record the health status of the node, and maintenance personnel can locate the node that needs maintenance by using the log.

[0050] The process of synchronizing database data to the search engine is quite long. During this process, data loss or synchronization failure may occur due to third-party factors or internal issues, making it difficult to pinpoint the location of the error. Therefore, to improve the stability of the search service, relevant health and log information is recorded at each node and presented intuitively on the page. This allows maintenance personnel to efficiently locate and quickly correct errors.

[0051] See Figure 4 Based on the above-mentioned high-performance and intelligent retrieval methods for massive spatial data, an intelligent retrieval engine for massive power grid equipment is constructed.

[0052] The massive power grid equipment intelligent retrieval engine can handle various business scenarios such as business expansion application analysis, on-site survey and design, equipment management, emergency command, route inspection, transformer area data verification, and data governance by searching by surrounding area, unit and voltage level, power station or line area, and graphic ID or resource ID. At the same time, the retrieval results can be intelligently sorted, customized sorted, multi-field matching, and multi-type matching.

[0053] By building a massive intelligent search engine for power grid equipment, we can provide users at different levels, such as provincial, municipal, and county-level units, with intelligent and high-performance search capabilities that are highly matched to their current business scenarios, such as line inspection, on-site survey, and map-based operations. This will improve the usability of business systems and enhance the practicality of the power grid GIS platform.

[0054] Example 2:

[0055] A high-performance and intelligent retrieval system for massive spatial data includes a data reading module, a search engine, and a retrieval module;

[0056] The data reading module is used to read data tables in the database, distribute different data tables to different nodes through a bucketing algorithm, read data from data tables in different nodes simultaneously in a multi-threaded manner and build search engine document data, and synchronously write search engine document data in different nodes into search engine index data.

[0057] The search engine has a built-in query service, which allows users to query corresponding index data.

[0058] The retrieval module is used to parse search operations, construct search engine retrieval syntax, call search engine query services, and query search engine index data.

[0059] In a preferred embodiment of this example, the data reading module is equipped with a data model driver, which is used to remove irrelevant data tables from the database.

[0060] The data model driver has a built-in model driver table, which is used to record the data tables in the database that are allowed to be written to the search engine.

[0061] When reading data from the database, the data model driver is initialized, the model driver table information is completed, the data tables that are allowed to be written to the search engine are selected based on the model driver table, and different data tables are assigned to different nodes.

[0062] See Figure 5 In this embodiment, the graphic browsing microservice, the element query microservice, the massive spatial retrieval microservice, and other microservices are combined to form a power grid data retrieval microservice group.

[0063] Based on the aforementioned microservice, database data is written into the search engine, and the search engine is also incrementally updated through this microservice.

[0064] This system is encapsulated into APIs using Spring Gateway and Nginx, namely the Power Grid Data Multi-condition Retrieval Service API and the Power Grid Data Multi-condition Equipment Statistics API. The retrieval function can be conveniently used by calling these APIs.

[0065] Example 3:

[0066] This embodiment proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method described in any embodiment of the present invention.

[0067] Example 4:

[0068] This embodiment proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any embodiment of the present invention.

[0069] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0070] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0071] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0072] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A high-performance and intelligent retrieval method for massive spatial data, characterized in that, Includes the following steps: Read data tables from the database, distribute different data tables to different nodes using a bucketing algorithm, and simultaneously perform streaming partitioning and data reading on data tables in different nodes in a multi-threaded manner to build search engine document data. Simultaneously write search engine document data from different nodes into search engine index data. When reading data from data tables on different nodes using multiple threads, there is an uneven distribution of data table sizes. By integrating a small message queue capability, the large data table is parsed and split, and the split large data table is allocated to the idle threads that have finished reading data from the small data table, ensuring that the thread pool is saturated. When performing a search, the search engine parses the business logic, constructs the search engine retrieval syntax, calls the search engine query service, and queries the search engine index data.

2. The high-performance and intelligent retrieval method for massive spatial data according to claim 1, characterized in that, Configure a data model driver, which is used to remove irrelevant data tables from the database; The data model driver has a built-in model driver table, which is used to record the data tables in the database that are allowed to be written to the search engine. When reading data from the database, the data model driver is initialized, the model driver table information is completed, the data tables that are allowed to be written to the search engine are selected based on the model driver table, and different data tables are assigned to different nodes.

3. The high-performance and intelligent retrieval method for massive spatial data according to claim 1, characterized in that, When data in the database is updated, the data in the search engine is updated incrementally. The specific steps are as follows: Based on Flink CDC, the database log information is monitored to obtain information on addition, deletion, and modification actions in the database. This information is then transmitted to the message queue in the Kafka messaging system. The search engine subscribes to the Kafka messaging system and updates its index data according to the message queue order in the system.

4. The high-performance and intelligent retrieval method for massive spatial data according to claim 1, characterized in that, Each node has a log that records its health status, allowing maintenance personnel to locate nodes that need maintenance.

5. A high-performance and intelligent retrieval system for massive spatial data, characterized in that, This includes a data reading module, a search engine, and a retrieval module; The data reading module is used to read data tables in the database, distribute different data tables to different nodes through a bucketing algorithm, and simultaneously perform streaming partitioning and data reading on data tables in different nodes in a multi-threaded manner to build search engine document data, and synchronously write search engine document data in different nodes into search engine index data. When reading data from data tables on different nodes using multiple threads, there is an uneven distribution of data table sizes. By integrating a small message queue capability, the large data table is parsed and split, and the split large data table is allocated to the idle threads that have finished reading data from the small data table, ensuring that the thread pool is saturated. The retrieval module is used to parse search operations, construct search engine retrieval syntax, call search engine query services, and query search engine index data.

6. The high-performance and intelligent retrieval system for massive spatial data according to claim 5, characterized in that, The data reading module is equipped with a data model driver, which is used to remove irrelevant data tables from the database; The data model driver has a built-in model driver table, which is used to record the data tables in the database that are allowed to be written to the search engine. When reading data from the database, the data model driver is initialized, the model driver table information is completed, the data tables that are allowed to be written to the search engine are selected based on the model driver table, and different data tables are assigned to different nodes.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images