Self-adaptive calculation system and method for power grid measurement

Abstract

The invention discloses a self-adaptive computing system and method for power grid measurement, the system comprises an engine core service part and a matched storage computing part, the engine core service part comprises a service layer and an engine layer, and the matched storage computing part comprises a computing layer and a storage layer. The service layer provides a unified computing engine API, an SQL development terminal, a unified data directory view and a data directory management tool, so that a user can conveniently access and manage data; the engine layer optimizes data processing performance and management through a storage calculation acceleration module, a self-adaptive push-down module, a self-adaptive path planning module and a data standardization management module; the computing layer integrates and fuses a computing engine, an interactive query engine, a stream batch computing engine and a real-time computing engine, and provides efficient data computing and analysis services; the storage layer supports cross-source heterogeneous data storage and unified metadata storage, and ensures data storage and management. The power grid measurement data searching and using efficiency can be improved, and the application development cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of digital construction of power grid measurement, and in particular to an adaptive computing system and method for power grid measurement. Background Art

[0002] With the expansion of the access scope of power grid measurement and the diversification of business scenarios, the required storage components and data distribution locations tend to be complex and diverse. In order to meet various scenarios such as instant query services, offline batch computing, and quasi-real-time computing services, data is stored in multiple copies based on the same source, and there are significant differences in data timeliness, equipment range, storage model structure, etc. on different components. In the application development process, users need to spend more manpower and time costs in data combing, data tracing, and technology selection and testing stages. This makes it difficult for users to find and use data, and the costs of data development and application transformation are high. Therefore, an adaptive computing system and method for power grid measurement is invented to solve the above problems, improve data search and use efficiency, and reduce application development costs, which has important practical significance and production value. Summary of the invention

[0003] Purpose of the invention: The present invention aims to provide an adaptive computing system and method for power grid measurement, supporting multiple scenarios such as ad hoc query, fusion computing, offline computing, real-time computing, etc., meeting different business query computing requirements in terms of timeliness and data volume, performing data source selection and execution path planning, and realizing business SQL automatic routing function.

[0004] Technical solution: In the first aspect, an adaptive computing system for power grid measurement includes an engine core service part and a supporting storage computing part, wherein the engine core service part includes a service layer and an engine layer, and the supporting storage computing part includes a computing layer and a storage layer;

[0005] The service layer provides a unified computing engine API, SQL development terminal, unified data directory view and data directory management tool to facilitate user access and management of data; wherein, the unified computing engine API serves as a unified entrance for system API-style services, enabling upper-layer business applications to call adaptive engine query and computing capabilities in a variety of ways; the SQL development terminal is a client that implements the SQL fusion query function across data sources, execution engines and clusters of the engine, and the user SQL is input into the adaptive push-down module through the built-in message pipeline, while the results returned by the lower layer are displayed on the terminal; the unified data directory view provides a tree-like structured data directory, maps and integrates the actual distribution of the system data through the metadata management platform and the data standardization management module, provides logically unified standard tables and field information, and shields the differences between different storage components and different data model structures; the unified data directory management tool implements the visual configuration of the standard data management module functions, facilitating the operation and maintenance personnel to manage and maintain the unified data directory;

[0006] The engine layer optimizes data processing performance and management through storage and computing acceleration modules, adaptive push-down modules, adaptive path planning modules and data standardization management modules; wherein the storage and computing acceleration module implements acceleration rule configuration and storage services, and provides relevant rules in the task CBO stage; the adaptive push-down module performs syntax analysis, field mapping, task distribution, dialect conversion and result integration on the SQL issued by the upper-layer business; the adaptive path planning module combines historical execution statistics, complexity optimization, execution resources and machine learning-assisted strategies to analyze the optimal path for task execution; the data standardization management module maintains and provides index mapping relationships based on metadata information, generates a unified data directory, and provides directory management functions;

[0007] The computing layer integrates the fusion computing engine, interactive query engine, stream batch computing engine and real-time computing engine to provide efficient data computing and analysis services;

[0008] The storage layer supports cross-source heterogeneous data storage and unified metadata storage, including data middle platform, data lake, relational database, time series database, metadata management engine and metadata storage, to ensure data storage and management.

[0009] According to certain embodiments of the first aspect, the storage and computing acceleration module is implemented by an acceleration rule library and an acceleration service, obtains index fields and data partition key information based on metadata information, optimizes related statements during the optimization process of the adaptive analysis module, and improves execution efficiency.

[0010] According to certain implementations of the first aspect, the adaptive push-down module is composed of various resident services, including SQL syntax parsing service, field verification service, syntax tree modification service, dialect conversion service, task disassembly and distribution service, and result splicing service;

[0011] The SQL syntax parsing service is used to understand the SQL intent, convert the user input SQL into an abstract syntax tree AST, and extract the key information of the SQL statement and pass it to the adaptive path planning module as the basis for the specific SQL execution path and plan;

[0012] The field verification service is used to match the parsing results with the content of the unified data directory, confirm the validity and legality of user input, and filter invalid requests;

[0013] The syntax tree modification service implements the function of modifying the statements in the unified view into executable real statements based on the results returned by the path planning module;

[0014] The dialect conversion service is responsible for converting the abstract syntax tree AST into the dialect of the target database, realizing the AST traversal and node modification functions, and customizing the function of replacing some function names;

[0015] The task disassembly and distribution service disassembles the AST according to the execution path rules of the logical plan and regenerates several sub-query SQL statements for direct execution to achieve task scheduling optimization;

[0016] The result stitching service integrates the sub-query results as needed and returns them to the user interface as the final query results.

[0017] According to certain implementations of the first aspect, the adaptive path planning module is implemented by a series of path planning services, and the adaptive path planning is implemented through HBO service, RBO service, CBO service, resource monitoring service, and machine learning algorithm service;

[0018] The HBO service implements the historical execution statistics function, analyzes and processes user SQL logs, the library names, table names, and key clause field names contained in SQL, summarizes the characteristics of the above historical SQL, and realizes the reuse and quick judgment of the same SQL path by searching the historical execution path planning results;

[0019] The CBO service mechanism optimizes the execution path logically while ensuring the correct execution of SQL by indexing the storage components required by the statement, estimating the amount of data in the fields used, and associating the query data dimensions.

[0020] The RBO service confirms the ambiguous table fields based on the data range and data timeliness conditions obtained from the previous analysis, combined with the rule information stored in the directory, to achieve unified view and physical view. Figure 1 One-to-one relationship;

[0021] The machine learning algorithm service automatically learns SQL features based on historical execution statistics, making up for the lack of historical rules and improving the accuracy of engine selection;

[0022] The resource monitoring service analyzes the existing resource information of the system and, based on the above adaptive strategy, selects a SQL execution path with sufficient resources within the current resource conditions.

[0023] According to certain implementations of the first aspect, the machine learning algorithm service extracts features from SQL according to strings through a natural language processing (NLP) algorithm, performs model training, and the model outputs the optimal execution path.

[0024] According to certain implementations of the first aspect, the data standardization management module is composed of a series of unified data directory related services, including metadata adaptation services, directory management services, directory generation services, directory retrieval services, and data integration services;

[0025] The metadata adaptation service connects to the underlying metadata management engine, obtains and parses metadata information from different data sources, and integrates it into a unified metadata storage;

[0026] The directory management service provides unified directory view permission control, directory attribute configuration, tag management, and directory governance functions, and provides a data management portal for operation and maintenance personnel;

[0027] The catalog generation service generates a unified data catalog structure based on the parameters set by the unified metadata storage and catalog management services and data integration services, and interacts with the front-end page through the internal protocol interface;

[0028] The catalog search service implements a unified interface for querying data catalogs in each module. During field verification, table field mapping, and table field modification, the catalog query interface is implemented through internal protocols.

[0029] Data integration service integrates duplicate data from multiple data sources. By adding configuration information through operation and maintenance personnel, it automatically realizes the logical integration of multi-source similar data and maintains the corresponding RBO rules in the path planning module.

[0030] According to certain embodiments of the first aspect, the deployment of the computing engine in the computing layer is completed by selecting computing components with corresponding characteristics on demand and completing the connection and adaptation with the storage components; the storage components in the storage layer are selected from data warehouses, data lakes, relational databases and time series databases according to actual conditions; the adaptive computing system realizes the query computing function by generating executable SQL for the computing components or directly outputting the corresponding SQL of the storage engine.

[0031] In a second aspect, an adaptive computing method for power grid measurement is provided. The method is performed according to the adaptive computing system for power grid measurement described in the first aspect, and the method comprises the following steps:

[0032] S1. The adaptive push-down module obtains the business SQL input by the user through the service layer. The business SQL is developed based on the unified data directory. After the terminal input or API parsing, the system input SQL is obtained;

[0033] S2. The SQL syntax parsing service in the adaptive push-down module obtains SQL input, extracts the SQL DDL type, table information, field information, query conditions, whether to aggregate, and whether to open a window, and passes the parsed syntax tree and its attributes to the adaptive path planning module; at the same time, the field verification service matches the source fields provided by the user with the table fields maintained in the unified data directory to check whether there are invalid input fields;

[0034] S3, the RBO service of the adaptive path planning module searches for fuzzy fields in the tables and fields submitted by the user, and confirms the unique mapping relationship between the unified view on the user side and the actual physical view of the data;

[0035] S4. According to the mapping information provided by the adaptive path planning module, the syntax tree modification service of the adaptive push-down module traverses the syntax tree, modifies the table and field accordingly, and converts the SQL input by the user under the unified view into the real SQL under the physical view;

[0036] S5. The adaptive path planning module optimizes the execution cost of the existing logical plan and performs preliminary optimization on the statement query efficiency by combining the rule base index and predicate push-down support information in the storage and computing acceleration module.

[0037] S6, Adaptive Path Planning passes the converted physical view syntax tree to the adaptive path planning module, analyzes the statements based on the HBO service, resource monitoring and machine learning, and outputs the logic execution plan value to the adaptive push-down module for subsequent task distribution;

[0038] S7. Based on the path planning, the adaptive push-down module task decomposition and distribution service is used to split the original logical execution plan as needed according to the data distribution location, and decompose it into several subtasks and subplans;

[0039] S8, using the adaptive push-down module dialect conversion service to perform statement correction and function conversion for different dialects of each data source;

[0040] S9, using the adaptive push-down module task disassembly and distribution service to distribute the executable statements to the corresponding execution components;

[0041] S10. Adopt the adaptive push-down module result splicing service to integrate and splice the lower-level execution results by integrating statements and the calculation engine, and return the results from the adaptive push-down module to the service layer to provide users with the final query results and log information.

[0042] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the adaptive computing method for power grid measurement as described in the second aspect are implemented.

[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the adaptive calculation method for power grid measurement as described in the second aspect are implemented.

[0044] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention provides an adaptive computing system for power grid measurement, which can help data analysis users to quickly query and retrieve data in a directory-like manner, reducing the cost of data tracing. Through the adaptive distribution capability of the engine, the differences between data from different data sources, different timeliness, and different storage types are shielded, and unified query and calculation of multi-source heterogeneous data is achieved from the user's perspective.

[0046] (2) Through the present invention, the original multi-source heterogeneous data is integrated, and the number of table fields is reduced by 35% compared with before integration, which significantly improves the efficiency of user data analysis. Through the present invention, the logical unification of components such as data lake, offline storage, and real-time storage is achieved, and 95% of the data access and directory display of common data sources are achieved; under the premise that the user logic is correct, the standard SQL statement is used as the unified entry dialect conversion success rate is not less than 99%.

[0047] (3) The present invention adopts a non-intrusive design with storage and computing separation, and adapts to different computing and storage components based on SQL conversion. It has high flexibility and scalability, and has good adaptability in the measurement business where the equipment is constantly updated and iterated. It adopts a modular design in terms of function, and modularly deploys services, computing, and storage components according to the different requirements of measurement scenarios, thereby reducing the overall resource consumption of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural block diagram of an adaptive computing system for power grid measurement in an embodiment of the present invention;

[0049] Figure 2 The present invention is a flowchart of the actual SQL adaptive push-down method for faceted measurement business in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings.

[0051] In terms of data query, Presto is an open source distributed SQL query engine that is widely used in the field of ad hoc data query and analysis. It connects to different data sources to achieve multi-database fusion query. The Presto core service parses, splits, distributes the input SQL to various data sources for execution, and merges and returns. Presto is based on in-memory computing and has high processing performance and query speed.

[0052] However, for measurement scenarios, the existing technology Presto supports limited computing functions. Presto mainly meets ad hoc query and analysis scenarios and adopts an in-memory computing mechanism. It consumes a lot of resources for scenarios such as offline batch computing and quasi-real-time computing. For scenarios where the timeliness and access scope of the same data are different in multiple data sources, it lacks the ability to unify the model of duplicate storage data, which makes the use and management of cross-source data more difficult, and lacks a tree-like directory classification display format. It does not have the ability to automatically route duplicate data. Data analysts need to do a lot of tracing and performance testing work in the case of multiple data sources and multiple execution engines, and the efficiency of data retrieval and use is low.

[0053] The present invention expands cross-database data analysis scenarios, supports multiple scenarios such as ad hoc query, fusion computing, offline computing, and real-time computing by adapting the stream-batch computing engine and the automatic push-down strategy, and meets different business query and computing requirements in terms of timeliness and data volume. The new integrated display capability of duplicate storage data models is added to organize and manage data in the form of data directories. Users develop based on this view, and the adaptive computing system selects data sources and plans execution paths. The automatic routing function of business SQL is realized, which reduces the workload of data tracing, technical route selection, and testing, and improves the query and computing efficiency of data analysts.

[0054] This technical solution focuses on the construction of engine core services and supporting storage and computing to create a flexible and extensible middleware system, forming a one-stop data platform capability improvement tool with unified directory view, standard SQL query and computing, and adaptive task optimization.

[0055] Reference Figure 1 The core service part of the engine includes the service layer and the engine layer, and the supporting storage and computing part includes the computing layer and the storage layer. The service layer is used to facilitate user access and management of data; the engine layer is used to optimize data processing performance and management, and supports automatic execution path planning and SQL conversion; the computing layer integrates multiple computing engines to provide efficient data computing and analysis services; the storage layer supports cross-source heterogeneous data storage and unified metadata storage to ensure data storage and management.

[0056] The core service part of the engine is the core component of the system. According to the implementation mode of the present invention, the service layer is composed of a unified computing engine API, a SQL development terminal, a unified data directory view, and a unified data directory management tool; the engine layer is composed of a storage and computing acceleration module, an adaptive push-down module, an adaptive path planning module, and a data standardization management module.

[0057] The unified computing engine API, in various forms such as JDBC, HTTP-API, SDK-API, etc., expands the application scope and adaptation capabilities of the adaptive computing engine, serves as a unified entry for system API services, and enables upper-level business applications to call the adaptive engine query and computing capabilities in various ways.

[0058] The SQL development terminal is a client that implements SQL fusion query functions across data sources, execution engines, and clusters. User SQL is input into the adaptive push-down module through the built-in message pipeline, and the results returned by the lower layer are displayed on the terminal.

[0059] The unified data directory view provides a tree-structured data directory, maps and integrates the actual distribution of system data, i.e., the physical view, through the metadata management platform and the data standardization management module, provides logically unified standard tables, fields and other information, and shields the differences between different storage components and different data model structures.

[0060] The unified data directory management tool realizes the visual configuration of the functions of the standard data management module, making it easier for operation and maintenance personnel to manage and maintain the unified data directory.

[0061] The storage and computing acceleration module implements acceleration rule configuration and storage services such as index optimization, data preheating, and operator pushdown, and provides relevant rules in the cost-based optimization (CBO) stage of the task.

[0062] The adaptive push-down module performs syntax analysis, field mapping, task distribution, dialect conversion, and result integration on the SQL sent down by the upper-level business.

[0063] The adaptive path planning module combines historical execution statistics, complexity optimization, execution resources and machine learning assistance strategies to analyze the optimal path for task execution.

[0064] The data standardization management module maintains and provides index mapping relationships based on metadata information, generates a unified data directory, and provides directory management functions.

[0065] The normal operation and task execution of the adaptive computing system requires the storage components for unified directory data storage and the computing components for execution. Figure 1 The computing layer and storage layer in the system include the following computing components, which can be selected as needed: fusion computing components such as Doris and presto, interactive query components such as Hbase and OTS, stream batch computing components such as Spark and Hive, and real-time computing components such as Flink. Storage components can select data warehouses, data lakes, relational databases, time series databases and other storage components according to actual conditions, such as Hive, Hudi, PostgreSQL, and IotDB. The adaptive computing system realizes query computing and other functions by generating executable SQL for computing components or directly outputting SQL corresponding to the storage engine.

[0066] Taking into account the optimization of data analysis management and the adaptability of measurement services, the present invention has important practical significance and application value in the field of power grid measurement data analysis.

[0067] In one embodiment, the system adaptive computing engine core service is deployed, the lower-level storage computing is selected according to the actual data storage distribution and business needs, and the lower-level computing components and storage components are connected and adapted.

[0068] Taking the measurement minimization scenario as an example, Hive, Hudi, and Hbase are deployed as storage components, Doris is used as the fusion computing component, and the Hive storage engine is used as the offline computing component.

[0069] (I) Implementation of storage and computing acceleration module

[0070] The storage and computing acceleration module is implemented by an acceleration rule library and an acceleration service. It obtains key information such as index fields and data partitions based on metadata information, optimizes related statements during the optimization process of the adaptive path planning module, and improves execution efficiency.

[0071] The acceleration rule base is a collection of optimization rules that are formulated based on metadata information, such as index fields, data partitions, etc. The acceleration service is responsible for applying the rules in the acceleration rule base to optimize data processing tasks. In the task CBO phase, the storage and computing acceleration module provides relevant rules to help the query optimizer select the most effective execution plan. For example, these acceleration rules can include:

[0072] Index optimization: by analyzing query patterns and data access characteristics, optimize index strategies to improve query efficiency;

[0073] Data preheating: predicting and preloading the data to be accessed into the cache to reduce data reading latency during querying.

[0074] Operator pushdown pushes data processing operations (such as filtering, projection, aggregation, etc.) to the data storage layer, reducing data transmission in the network and thus improving processing speed.

[0075] The storage and computing acceleration module uses metadata information (such as index fields and data partition information) to guide the configuration and application of rules. During the optimization process of the adaptive path planning module, the storage and computing acceleration module can be used to optimize SQL statements or data processing tasks to ensure that the execution path is optimal.

[0076] (II) Adaptive Push-Down Module Service Implementation

[0077] The adaptive push-down module is composed of various resident services, including SQL syntax parsing service, field verification service, syntax tree modification service, dialect conversion service, task disassembly and distribution service, and result splicing service.

[0078] The SQL syntax parsing service is used to understand the SQL intent, convert the user input SQL into an abstract syntax tree (AST), and extract key information such as the SQL DDL type, table information, field information, query conditions, aggregation, window operations, etc. The above information is passed to the adaptive path planning module as the basis for the specific SQL execution path and plan.

[0079] The field verification service calls the directory search service for standardized data management, compares and matches the parsing results of the user's SQL with the content of the unified data directory, confirms the validity and legality of the user's input, and filters invalid and illegal requests.

[0080] The syntax tree modification service implements the abstract syntax tree modification function, which is implemented through a custom modification tool class. Based on the execution path and field mapping relationship returned by the adaptive path planning module, the user SQL is developed based on the unified view provided by the data standardization management module. The service supports modifying the statement into an execution statement under the real view through the mapping relationship. It is implemented through table and field replacement. Ensure that the query statement can adapt to the actual structure and performance characteristics of the underlying data storage.

[0081] The task decomposition and distribution service regenerates several directly executable sub-query SQL statements according to the built-in rules of the logical execution plan, the execution statements under the real view, and the execution plan specified by the adaptive path planning module according to the actual data storage location and the calculation engine selection results, thus realizing fine-grained task scheduling optimization. By optimizing task scheduling, the parallel processing capability and execution efficiency of queries can be improved.

[0082] The dialect conversion service is responsible for converting the execution statements under the real view into the dialect supported by the actual execution database. Through the AST traversal and node modification functions, some function name conversion functions are customized. This service ultimately generates SQL that can be directly executed by the computing layer.

[0083] Result splicing service: If the user's SQL involves task decomposition during the push-down process, the sub-query results need to be integrated, and the appropriate engine is selected through the adaptive path planning module to perform the final merging calculation on the intermediate results, generate the final query results, and return them to the user interface or store them persistently in the corresponding data table. Ensure the integrity and correctness of the results and return the final query results according to user needs.

[0084] (III) Implementation of Adaptive Path Planning Module

[0085] The adaptive path planning module is implemented by a series of path planning services, and executes path adaptive planning through historical execution optimization (HBO) service, rule-based optimization (RBO) service, cost-based optimization (CBO) service, resource monitoring service, and machine learning algorithm service.

[0086] The HBO service implements the historical execution statistics function, analyzes and processes user SQL logs, the library names, table names, and key clause field names contained in SQL, summarizes the characteristics of the above historical SQL, and realizes the reuse and quick judgment of the same SQL path by searching the historical execution path planning results.

[0087] The CBO service mechanism uses statistical information and cost models to evaluate the cost of query operations and select the optimal execution path. For example, by analyzing the index characteristics of storage components required for statements, the estimated size of field data, and the associated query data volume, the execution path is logically optimized while ensuring that SQL is executed correctly.

[0088] The RBO service applies a set of predefined rules to optimize queries. These rules are based on database design and query characteristics. For example, based on the data range and data timeliness obtained in the previous analysis, the table fields with ambiguity are confirmed in combination with the rule information stored in the directory. This stage realizes the unified view and physical view. Figure 1 One-to-one correspondence. Combined with data distribution and the execution engine's own capabilities, the unified view and the physical view are uniquely corresponded, and the SQL function and data range required meet the requirements and are executed correctly.

[0089] The machine learning judgment service automatically learns SQL features based on historical execution statistics, makes up for the lack of historical rules, and further improves the accuracy of engine selection. SQL is extracted according to the string through natural language processing NLP algorithms such as CNN, RNN, Word2Vec, and Transformer, and the historical logs of the adaptive path planning module are used as data sets for model training. The model outputs the best execution path as a supplement to the adaptive analysis capability.

[0090] The resource monitoring mechanism analyzes the existing resource information of the system and selects a SQL execution path with sufficient resources within the resource status conditions by combining the above-mentioned adaptive strategies. For example, it monitors the usage of system resources such as CPU, memory, disk I / O, and network traffic in real time. It adjusts the query execution path according to the current resource status to avoid resource bottlenecks and improve resource utilization.

[0091] The above rules are mainly based on RBO, and other mechanisms are used for auxiliary optimization to ensure that SQL can be pushed down correctly. For different query types, one or more analysis services can be configured for comprehensive analysis, and the execution plan is returned to the adaptive push-down module to complete the subsequent process.

[0092] (IV) Implementation of data standardization management module

[0093] The data standardization management module is composed of a series of unified data directory related services, including metadata adaptation service, directory management service, directory generation service, directory retrieval service, and data integration service.

[0094] The metadata adaptation service connects to the underlying metadata management engine, obtains and parses metadata information from different data sources, and integrates them into a unified metadata storage, providing the system with standardized information about data sources, table structures, field attributes, etc.

[0095] The directory management service provides multiple functions such as unified directory view permission control, directory attribute configuration, tag management, directory governance, etc., and provides a data management portal for operation and maintenance personnel. It ensures the integrity and security of the data directory and facilitates users and system administrators to manage the data directory.

[0096] The catalog generation service generates a unified data catalog structure based on the parameters set by the unified metadata storage and catalog management services and data integration services, and interacts with the front-end page through the internal protocol interface. This simplifies the catalog creation process and ensures the standardization and consistency of the catalog structure.

[0097] The directory search service implements a unified interface for each module to query the data directory. In the process of field verification, table field mapping, table field modification, etc., the directory query interface is implemented through internal protocols. It improves data search efficiency and supports other modules (such as field verification service) to access the data directory.

[0098] Data integration service integrates duplicate data from multiple data sources. Operation and maintenance personnel can add configuration information to automatically integrate similar data from multiple sources and maintain the corresponding RBO rules in the path planning module. By identifying and integrating similar or duplicate data from different data sources, a mapping relationship between data is established. Data redundancy is eliminated, data quality is improved, and a unified data view is provided for data analysis.

[0099] The present invention establishes a unified data directory through data standardization management, logically integrates the original multi-source heterogeneous data metadata information, solves the problem of redundant storage of multiple copies of data sources, and reduces the number of table fields of the unified data directory by 35% compared with the number of original table fields, significantly improving the efficiency of user data query, analysis, and development. Through the adaptive push-down module and the adaptive path planning module, the user SQL execution path is automatically planned and adaptively pushed down to achieve the logical unification of components such as data lake, offline storage, and real-time storage, solving the problem of difficulty in designing execution paths for users and high learning costs for using multiple components; through the management service of the data standardization management module, 95% of the data access and directory display of common data sources are achieved, solving the problem that users' cross-data source data retrieval is not clear enough; through the automatic planning of execution paths based on mechanisms such as logic optimization, machine learning, and resource monitoring, under the premise that the user's business logic is correct, the unified data directory and standard SQL statements are used as the only input to solve the problems faced by users with redundant multiple data sources, more execution paths, and higher learning costs, and the success rate of full-process execution is not less than 99%.

[0100] Based on the above-mentioned adaptive computing engine middleware system, the present invention proposes an adaptive push-down method for actual SQL of vector measurement business. Figure 2 As shown, the method comprises the following steps:

[0101] S1. The adaptive push-down module obtains the business SQL input by the user through the service layer. The SQL is developed by the user based on the unified data directory. After terminal input or API parsing, the system input SQL is obtained.

[0102] S2. The SQL syntax parsing service in the adaptive push-down module obtains SQL input, generates an abstract syntax tree AST based on the SQL statement, and extracts the SQL DDL type, table information, field information, query conditions, whether to aggregate, whether to open windows and other attributes, and passes the parsed syntax tree AST and its attributes to the adaptive path planning module. At the same time, the field verification service will match the source fields provided by the user with the table fields maintained in the unified data directory to see if there are invalid input fields.

[0103] S3, the RBO service of the adaptive path planning module will search for ambiguous fields in the tables and fields submitted by the user. Some physical table fields are integrated in the standard data management module for user use. If the user SQL involves the above table fields, it is necessary to customize the RBO special rules to confirm the unique mapping relationship between the unified view on the user side and the actual physical view of the data.

[0104] S4. According to the mapping information provided by the adaptive path planning module, the syntax tree modification service of the adaptive push-down module traverses the syntax tree, modifies the table and field accordingly, and converts the SQL input by the user under the unified view into the real SQL under the physical view.

[0105] S5. The adaptive path planning module optimizes the execution cost (CBO) of the existing logical plan, and combines the information of the rule base index and predicate push-down support in the storage and computing acceleration module to preliminarily optimize the statement query efficiency.

[0106] S6. The adaptive path planning module uses HBO services, resource monitoring, and machine learning to analyze the statements on the converted physical view syntax tree as needed, and outputs the logical execution plan to the adaptive push-down module for subsequent task distribution.

[0107] S7. According to the path planning, the task decomposition and distribution service of the adaptive push-down module is used to split the original logical execution plan as needed according to the data distribution location, and decompose it into several subtasks and subplans.

[0108] S8. Use the dialect conversion service of the adaptive push-down module to perform statement correction and function conversion for different dialects of each data source.

[0109] S9. Use the task decomposition and distribution service of the adaptive push-down module to distribute the executable statements to the corresponding execution components.

[0110] S10. Adopt the result splicing service of the adaptive push-down module to integrate and splice the lower-level execution results by integrating statements and the calculation engine, and return the results from the adaptive push-down module to the service layer to provide the final query results and log information to the user.

[0111] To sum up, the advantages of the present invention are: by providing a unified data directory, optimizing data tracing, data source selection and execution component selection processes, improving data analysis efficiency, enabling measurement business data analysis, and having important practical significance and application value.

[0112] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the adaptive computing method for power grid measurement as described above are implemented.

[0113] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the adaptive calculation method for power grid measurement as described above are implemented.

[0114] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0115] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. An adaptive computing system for power grid measurement, characterized in that: It includes an engine core service part and a supporting storage computing part, wherein the engine core service part includes a service layer and an engine layer, and the supporting storage computing part includes a computing layer and a storage layer; The service layer provides a unified computing engine API, SQL development terminal, unified data directory view and data directory management tool to facilitate user access and management of data; wherein, the unified computing engine API serves as a unified entrance for system API services, enabling upper-layer business applications to call adaptive engine query and computing capabilities in a variety of ways; the SQL development terminal is a client that implements the SQL fusion query function across data sources, execution engines and clusters of the engine, and the user SQL is input into the adaptive push-down module through the built-in message pipeline, while the results returned by the lower layer are displayed on the terminal; the unified data directory view provides a tree-like structured data directory, maps and integrates the actual distribution of system data through the metadata management platform and the data standardization management module, provides logically unified standard tables and field information, and shields the differences between different storage components and different data model structures; the unified data directory management tool realizes the visual configuration of the functions of the standard data management module, which is convenient for operation and maintenance personnel to manage and maintain the unified data directory; The engine layer optimizes data processing performance and management through storage and computing acceleration modules, adaptive push-down modules, adaptive path planning modules and data standardization management modules; wherein the storage and computing acceleration module implements acceleration rule configuration and storage services, and provides relevant rules in the task-based cost optimization stage; the adaptive push-down module performs syntax analysis, field mapping, task distribution, dialect conversion and result integration on the SQL issued by the upper-layer business; the adaptive path planning module combines historical execution statistics, complexity optimization, execution resources and machine learning-assisted strategies to analyze the optimal path for task execution; the data standardization management module maintains and provides index mapping relationships based on metadata information, generates a unified data directory, and provides directory management functions; The computing layer integrates the fusion computing engine, interactive query engine, stream batch computing engine and real-time computing engine to provide efficient data computing and analysis services; The storage layer supports cross-source heterogeneous data storage and unified metadata storage, including data middle platform, data lake, relational database, time series database, metadata management engine and metadata storage, to ensure data storage and management.

2. The system according to claim 1, characterized in that The storage and computing acceleration module is implemented by an acceleration rule library and an acceleration service. It obtains index fields and key information of data partitions based on metadata information, optimizes related statements during the optimization process of the adaptive analysis module, and improves execution efficiency.

3. The system according to claim 1, characterized in that The adaptive push-down module is composed of various resident services, including SQL syntax parsing service, field verification service, syntax tree modification service, dialect conversion service, task disassembly and distribution service, and result splicing service; The SQL syntax parsing service is used to understand the SQL intent, convert the user input SQL into an abstract syntax tree AST, and extract the key information of the SQL statement and pass it to the adaptive path planning module as the basis for the specific SQL execution path and plan; The field verification service is used to match the parsing results with the content of the unified data directory, confirm the validity and legality of user input, and filter invalid requests; The syntax tree modification service implements the function of modifying the statements in the unified view into executable real statements based on the results returned by the path planning module; The dialect conversion service is responsible for converting the abstract syntax tree AST into the dialect of the target database, realizing the AST traversal and node modification functions, and customizing the function of replacing some function names; The task disassembly and distribution service disassembles the AST according to the execution path rules of the logical plan and regenerates several sub-query SQL statements for direct execution to achieve task scheduling optimization; The result stitching service integrates the sub-query results as needed and returns them to the user interface as the final query results.

4. The system according to claim 1, characterized in that The adaptive path planning module is implemented by a series of path planning services, and executes path adaptive planning through HBO service, RBO service, CBO service, resource monitoring service, and machine learning algorithm service; The HBO service implements the historical execution statistics function, analyzes and processes user SQL logs, the library names, table names, and key clause field names contained in SQL, summarizes the characteristics of the above historical SQL, and realizes the reuse and quick judgment of the same SQL path by searching the historical execution path planning results; The CBO service mechanism optimizes the execution path logically while ensuring the correct execution of SQL by indexing the storage components required by the statement, estimating the amount of data in the fields used, and associating the query data dimensions. The RBO service confirms the ambiguous table fields based on the data range and data timeliness conditions obtained in the previous analysis, combined with the rule information stored in the directory, to achieve a one-to-one correspondence between the unified view and the physical view; The machine learning algorithm service automatically learns SQL features based on historical execution statistics, making up for the lack of historical rules and improving the accuracy of engine selection. The resource monitoring service analyzes the existing resource information of the system and, based on the above adaptive strategy, selects a SQL execution path with sufficient resources within the current resource conditions.

5. The system according to claim 4, characterized in that The machine learning algorithm service extracts features from SQL as strings using the natural language processing (NLP) algorithm, performs model training, and the model outputs the best execution path.

6. The system according to claim 1, characterized in that The data standardization management module consists of a series of unified data directory related services, including metadata adaptation service, directory management service, directory generation service, directory retrieval service, and data integration service; The metadata adaptation service connects to the underlying metadata management engine, obtains and parses metadata information from different data sources, and integrates it into a unified metadata storage; The directory management service provides unified directory view permission control, directory attribute configuration, tag management, and directory governance functions, and provides a data management portal for operation and maintenance personnel; The catalog generation service generates a unified data catalog structure based on the parameters set by the unified metadata storage and catalog management services and data integration services, and interacts with the front-end page through the internal protocol interface; The catalog search service implements a unified interface for querying data catalogs in each module. During field verification, table field mapping, and table field modification, the catalog query interface is implemented through internal protocols. Data integration service integrates duplicate data from multiple data sources. By adding configuration information through operation and maintenance personnel, it automatically realizes the logical integration of multi-source similar data and maintains the corresponding RBO rules in the path planning module.

7. The system according to claim 1, characterized in that The deployment of the computing engine in the computing layer is completed by selecting computing components with corresponding characteristics on demand and completing the connection and adaptation with the storage components; the storage components in the storage layer are selected from data warehouses, data lakes, relational databases and time series databases according to actual conditions; the adaptive computing system realizes the query computing function by generating executable SQL for computing components or directly outputting the corresponding SQL of the storage engine.

8. An adaptive calculation method for power grid measurement, characterized in that: The method is implemented according to the adaptive computing system for power grid measurement according to any one of claims 1 to 7, and the method comprises the following steps: S1. The adaptive push-down module obtains the business SQL input by the user through the service layer. The business SQL is developed based on the unified data directory. After the terminal input or API parsing, the system input SQL is obtained; S2. The SQL syntax parsing service in the adaptive push-down module obtains SQL input, generates an abstract syntax tree AST and extracts SQL key attributes, and passes the parsed syntax tree and its attributes to the adaptive path planning module; at the same time, the field verification service matches the source fields provided by the user with the table fields maintained in the unified data directory to check whether there are invalid input fields; S3, the RBO service of the adaptive path planning module searches for fuzzy fields in the tables and fields submitted by the user, and confirms the unique mapping relationship between the unified view on the user side and the actual physical view of the data; S4. According to the mapping information provided by the adaptive path planning module, the syntax tree modification service of the adaptive push-down module traverses the syntax tree, modifies the table and field accordingly, and converts the SQL input by the user under the unified view into the real SQL under the physical view; S5. The adaptive path planning module optimizes the execution cost of the existing logical plan and performs preliminary optimization on the statement query efficiency by combining the rule base index and predicate push-down support information in the storage and computing acceleration module. S6. The adaptive path planning module uses HBO services, resource monitoring, and machine learning algorithms to analyze the statements in the converted physical view syntax tree, and outputs the logic execution plan to the adaptive push-down module for subsequent task distribution. S7. Based on the path planning, the task decomposition and distribution service of the adaptive push-down module is used to split the original logical execution plan as needed according to the data distribution location, and decompose it into several subtasks and subplans; S8, using the dialect conversion service of the adaptive push-down module to perform statement correction and function conversion for different dialects of each data source; S9, using the task disassembly and distribution service of the adaptive push-down module to distribute the executable statements to the corresponding execution components; S10. Adopt the result splicing service of the adaptive push-down module to integrate and splice the lower-level execution results by integrating statements and the calculation engine, and return the results from the adaptive push-down module to the service layer to provide the final query results and log information to the user.

9. A computer device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the adaptive calculation method for power grid measurement as claimed in claim 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the adaptive calculation method for power grid measurement according to claim 8 are implemented.

Images