Data query system and data query method

Abstract

The invention discloses a data query system and method, and relates to the technical field of computers. A specific embodiment of the data query system comprises a Kyuubi service layer, a federated query engine, a routing optimizer and a distributed execution engine, the Kyuubi service layer obtains a data query request which is initiated by a user and aims at a plurality of heterogeneous data sources, and a federated query engine is started according to a corresponding tenant permission, so that the federated query engine analyzes the data query request to obtain an analysis result; the route optimizer constructs a target execution plan meeting a preset cost condition according to an analysis result based on a Catalyst optimizer framework; the distributed execution engine schedules and executes the distributed task according to the target execution plan to obtain a query result; and the Kyuubi service layer also sends the query result to the tenant. According to the embodiment, high-performance and easy-to-maintain federal query service can be provided for data query, the data integration cost is reduced, and the data analysis efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data query system and a data query method. Background Technology

[0002] With the development of big data, data is becoming increasingly diverse and decentralized, typically stored in different systems. For example, data from different data sources may be stored in relational databases (MySQL, Oracle), big data platforms (Hadoop, Spark), cloud storage services (S3, HDFS), and various NoSQL databases. Traditional data querying methods usually require writing separate query statements for each data source and manually integrating the results, which is inefficient and error-prone. Therefore, a high-performance data querying method is urgently needed. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data query system and a data query method. By integrating the Kyubi service layer with a federated query engine, unified access support for multiple heterogeneous data sources is achieved. Furthermore, tenant-level resource quota control is implemented through the Kyubi service layer. Combined with a routing optimizer and a distributed execution engine, efficient execution of cross-data source queries is achieved. This provides high-performance, easy-to-maintain federated query services for data queries, reduces data integration costs, and improves data analysis efficiency.

[0004] To achieve the above objectives, according to one aspect of the present invention, a digital query system is provided, comprising: a Kyubi service layer, a federated query engine, a route optimizer, and a distributed execution engine;

[0005] The Kyuubi service layer is configured to: obtain data query requests initiated by users for multiple heterogeneous data sources, authenticate tenant permissions based on the data query requests, and start the federated query engine based on the tenant permissions corresponding to the authentication results if the authentication is successful.

[0006] The federated query engine is configured to parse data query requests and obtain the parsing results; the parsing results include: cross-source table structures and index information corresponding to multiple heterogeneous data sources;

[0007] The routing optimizer is configured as follows: based on the Catalyst optimizer framework, it constructs the target execution plan according to the parsing results, and the execution cost of the target execution plan meets the preset cost conditions;

[0008] The distributed execution engine is configured to schedule and execute distributed tasks according to the target execution plan to obtain query results;

[0009] The Kyuubi service layer is also configured to send the query results to the tenant upon receiving them from the distributed execution engine.

[0010] Optionally, the routing optimizer is further configured to: construct multiple execution plans based on the Catalyst optimizer framework and the parsing results; each execution plan includes data to be read corresponding to multiple heterogeneous data sources, and one or more nodes for data processing; determine the execution cost of each execution plan based on the execution factors corresponding to the multiple execution plans; and select the execution plan with the lowest execution cost among the multiple execution plans as the target execution plan; wherein the execution factors include any one or more of the following: network transmission cost corresponding to the data to be read, local computing cost of one or more nodes for data processing, and data size.

[0011] Optionally, the route optimizer is also configured to: divide the target execution plan into multiple stages, and determine multiple subquery plans corresponding to the multiple stages, as well as the execution order, data source and join strategy corresponding to the multiple subquery plans;

[0012] The distributed execution engine is also configured to generate subquery tasks corresponding to the subquery plan and distribute the subquery tasks to the corresponding data sources so that the data sources can execute the query to obtain the data to be read and obtain the query results.

[0013] Optionally, the Kyuubi service layer is also configured to determine the resource configuration corresponding to a tenant based on the tenant's permissions;

[0014] The distributed execution engine is also configured to dynamically allocate resources based on resource configuration, so as to utilize the allocated resources to execute distributed tasks.

[0015] Optionally, the distributed execution engine connects to multiple heterogeneous data sources through multiple connectors;

[0016] The Kyuubi service layer inherits the connector of the distributed execution engine to support data query requests for multiple heterogeneous data sources.

[0017] Optionally, the route optimizer is also configured to: subscribe to metadata change notifications; upon receiving a metadata change notification, update the data source information according to the metadata change notification, and generate an execution plan corresponding to the target query plan based on the updated data source information; the target query request is obtained through the Kyubi service layer after the data change event occurs.

[0018] Optionally, the distributed execution engine is also configured to: obtain the subquery results of each subquery task, perform cross-source aggregation operations on the subquery results, and obtain the query results;

[0019] The Kyuubi service layer is also configured to encapsulate query results into result set objects and stream these result set objects back to the tenant's corresponding client.

[0020] Optionally, the data query system further includes a monitoring device; wherein the monitoring device is configured to: detect in real time whether the Kyubi service layer, the federated query engine, the route optimizer, and the distributed execution engine are abnormal; and if any one or more of the Kyubi service layer, the federated query engine, the route optimizer, and the distributed execution engine are abnormal and the query fails, start the Kyubi service layer to execute a rollback transaction.

[0021] The Kyuubi service layer is also configured to perform a rollback transaction to respond to data query requests again.

[0022] Optionally, the route optimizer is also configured to: obtain the query data corresponding to the query request that failed, dynamically learn the query failure pattern based on the query data, and update the model for calculating the execution cost based on the learning results.

[0023] To achieve the above objectives, according to another aspect of the present invention, a data query method is provided, comprising:

[0024] The Kyuubi service layer obtains data query requests initiated by users for multiple heterogeneous data sources, and authenticates tenant permissions based on the data query requests; if authentication is successful, the federated query engine is started according to the tenant permissions corresponding to the authentication result.

[0025] The federated query engine parses data query requests and obtains the parsing results; the parsing results include: cross-source table structures and index information corresponding to multiple heterogeneous data sources;

[0026] The route optimizer is based on the Catalyst optimizer framework. It constructs a target execution plan based on the parsing results, and the execution cost of the target execution plan meets the preset cost conditions.

[0027] The distributed execution engine schedules and executes distributed tasks according to the target execution plan to obtain query results;

[0028] When the Kyuubi service layer receives the query results from the distributed execution engine, it sends the query results to the tenant.

[0029] Optionally, the routing optimizer is based on the Catalyst optimizer framework and constructs a target execution plan based on the parsing results, including: constructing multiple execution plans based on the Catalyst optimizer framework and the parsing results; each execution plan includes the data to be read corresponding to each heterogeneous data source, and one or more nodes for performing data processing;

[0030] The execution cost of each execution plan is determined based on the execution factors corresponding to the multiple execution plans. The execution factors include any one or more of the following: network transmission cost of the data to be read, local computing cost of one or more nodes performing data processing, and data size.

[0031] The execution plan with the lowest execution cost among multiple execution plans is selected as the target execution plan.

[0032] Optionally, if authentication is successful, the data query method further includes: the Kyubi service layer determining the corresponding resource configuration based on the tenant's permissions;

[0033] The distributed execution engine schedules and executes distributed tasks according to the target execution plan, including: the distributed engine dynamically allocates resources according to resource configuration in order to use the allocated resources to execute distributed tasks.

[0034] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided. The electronic device of the present invention includes: one or more processors; and a storage device communicatively connected to the one or more processors for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a data query method according to an embodiment of the present invention.

[0035] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements a data query method according to the present invention.

[0036] One embodiment of the above invention has the following advantages or beneficial effects: The Kyubi service layer obtains data query requests from tenants targeting multiple heterogeneous data sources. Upon successful tenant authentication, a federated query engine is launched. The federated query engine parses the data query requests, obtaining parsing results including cross-source table structures and index information corresponding to the heterogeneous data sources. Then, a routing optimizer, based on the Catalyst optimizer framework, constructs a target execution plan based on the parsing results. Subsequently, a distributed execution engine schedules and executes distributed tasks according to the target execution plan to obtain the query results. Finally, the Kyubi service layer sends the query results to the tenant. This achieves the integration of the Kyubi service layer and the federated query engine, thereby enabling unified access support for multiple heterogeneous data sources. Furthermore, the Kyubi service layer implements tenant-level resource quota control. Combined with the routing optimizer and the distributed execution engine, it achieves efficient execution of cross-data source queries, providing a high-performance, easily maintainable federated query service for data queries, reducing data integration costs, and improving data analysis efficiency.

[0037] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0038] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0039] Figure 1 This is a schematic diagram of the main architecture of a data query system according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram illustrating the main steps of a data query method according to an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention is shown;

[0042] Figure 4 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown. Detailed Implementation

[0043] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0044] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.

[0045] First, the relevant terms used in the embodiments of the present invention will be defined and explained.

[0046] Kyuubi Service Layer: A high-performance, distributed Thrift JDBC / ODBC service layer whose core objective is to provide enterprise-grade, multi-tenant access capabilities for big data computing engines (such as Apache Spark). It is an enterprise-standardized access gateway built on top of big data computing engines, simplifying access, management, and operation of big data platforms by providing familiar database interfaces (such as JDBC / ODBC) and multi-tenant management capabilities.

[0047] Federated queries refer to a unified SQL query processing technology across multiple independent data sources (such as MySQL, Hive, S3, etc.). It masks underlying data differences through metadata federation and execution optimization, providing a global logical view. In this embodiment of the invention, a federated query engine parses data query requests across multiple independent data sources, and a routing optimizer constructs a target execution plan to optimize and mask underlying data differences.

[0048] A distributed execution engine is a software framework whose core function is to automatically decompose a large-scale computing task into many small subtasks, distribute these subtasks to multiple computers (nodes) in a computer cluster for parallel execution, and finally aggregate the execution results of each node to generate the final output.

[0049] User equipment primarily includes devices involving communication technologies. Specifically, it includes any electronic device that a user can easily carry or operate, which can also provide remote communication capabilities with a network, such as a mobile communication terminal. Mobile communication terminals can communicate using mobile phone (wireless) networks, wireless data networks (e.g., 3G, 4G, 5G, or similar networks), Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), Wi-Max, or any other communication medium that provides access to networks such as the Internet or private networks. Specific examples include mobile terminals, tablet computers, netbooks, laptops, wearable devices (e.g., smartwatches), etc. Tenants can initiate data query requests against multiple heterogeneous data sources through user equipment, and the user equipment sends the data query request to the Kyubi service layer via the network.

[0050] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions of this invention comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0051] Big data today is characterized by diversity and decentralization, with data typically stored in different systems, such as relational databases (MySQL, Oracle), big data platforms (Hadoop, Spark), cloud storage services (S3, HDFS), and various NoSQL databases. When querying data, it's usually necessary to write separate query statements for each data source and manually integrate the results, which is inefficient and error-prone. To address this issue, some enterprises have adopted federated query solutions; however, these solutions still have limitations such as insufficient multi-tenancy and resource isolation, compatibility issues with heterogeneous data sources, and problems with query optimization and execution efficiency.

[0052] For example, in enterprise-level application scenarios, multiple users or business departments often need to use the federated query service simultaneously. However, due to the lack of a sound multi-tenant management mechanism in traditional solutions, it is impossible to effectively achieve resource isolation and quota control between users. In high-concurrency query scenarios, problems such as resource contention, performance fluctuations, and even service unavailability are likely to occur, which in turn affect the stability and service quality of the system, i.e., the problem of insufficient multi-tenancy and resource isolation.

[0053] For example, with the increasing number of data storage systems, existing federated query solutions lag behind in supporting emerging data sources. Furthermore, different data sources differ significantly in their connection protocols, authentication mechanisms, and data formats. Extending support for new data sources requires modifying the core code and recompiling and deploying it, resulting in long development cycles and difficult maintenance. Additionally, for certain specialized data sources, the lack of efficient connector implementations leads to poor query performance, highlighting the compatibility issues with heterogeneous data sources.

[0054] Furthermore, because cross-data source queries involve complex query plan generation and optimization processes, existing systems often fail to accurately assess the execution cost of each data source when handling complex join queries across multiple data sources. This leads to the generation of suboptimal query plans, resulting in higher execution costs. Moreover, the lack of effective data localization capabilities and intelligent routing mechanisms in existing systems can also cause unnecessary data transmission and computational overhead, impacting overall query performance and causing query optimization and execution efficiency issues.

[0055] In view of this, embodiments of the present invention provide a high-performance data query system and data query method, which integrates the Kyubi service layer with a federated query engine to achieve unified access support for multiple heterogeneous data sources. Furthermore, the Kyubi service layer implements tenant-level resource quota control, and combined with a routing optimizer and a distributed execution engine, it achieves efficient execution of cross-data source queries. This enables the provision of high-performance, easy-to-maintain federated query services for data queries, reduces data integration costs, and improves data analysis efficiency.

[0056] Specifically, such as Figure 1 As shown, the data query system provided in this embodiment of the invention mainly includes: a Kyubi service layer 10, a federated query engine 20, a route optimizer 30, and a distributed execution engine 40; it may also optionally include a metadata management module 50 and a monitoring device 60.

[0057] The Kyubi service layer 10 serves as the system's entry point, providing a standardized SQL interface responsible for user authentication and access control. Based on Kyubi's multi-tenant architecture, it implements user resource isolation and quota management. For example, after receiving data query requests from multiple heterogeneous data sources, the Kyubi service layer 10 authenticates tenant permissions based on the query requests. If authentication is successful, it determines the corresponding resource configuration for the tenant based on their permissions. This allows the distributed execution engine 40 to dynamically allocate resources according to the resource configuration during the execution of distributed tasks, thereby achieving multi-tenant resource isolation and quota management.

[0058] The Federated Query Engine 20 is configured to parse data query requests, obtaining parsing results including cross-source table structure and index information. For example, the Kyubi service layer 10 obtains a data query request (SQL query statement) through a standardized SQL interface. The Federated Query Engine 20 can perform lexical analysis, syntax parsing, and semantic analysis on the SQL query statement to identify cross-source table structure and index information, and generate a unified query abstract syntax tree. During this process, the Federated Query Engine 20 can parse the SQL query statement based on the metadata information of various heterogeneous data sources maintained by the metadata management module 50. Specifically, the metadata information maintained by the metadata management module 50 includes table structure, field types, and index information, providing basic data support for data query optimization.

[0059] The routing optimizer 30, based on the Catalyst optimizer framework, constructs multiple execution plans according to a cost assessment model and data source characteristics. Each execution plan includes the data to be read corresponding to each heterogeneous data source, as well as one or more nodes for data processing. Furthermore, the routing optimizer 30 determines the execution cost of each execution plan based on its corresponding execution factors, and selects the execution plan with the lowest execution cost as the target execution plan. This optimal target execution plan is then executed to achieve intelligent distribution and routing of data query requests.

[0060] The distributed execution engine 40 schedules and executes distributed tasks according to the target execution plan. For example, it generates subquery tasks based on multiple stages of the target execution plan and distributes these subquery tasks to the corresponding data sources. This ensures that the data sources execute the query and obtain the data to be read, coordinating the query execution process across different data sources to ensure efficient completion of the query tasks. The distributed execution engine 40 connects to multiple heterogeneous data sources (such as MySQL databases, S3 cloud storage services, MongoDB databases, and Hadoop data platforms) through multiple connectors (a heterogeneous data source connector cluster). This supports different types of storage systems, including relational databases, big data platforms, and cloud storage, enabling the distribution of subquery tasks to the appropriate data sources and allowing them to retrieve the data. The Kyubi service layer inherits the connectors from the distributed execution engine 40 to support data query requests from multiple heterogeneous data sources. After obtaining the query results from each subquery task, the distributed execution engine 40 can also perform cross-source aggregation of the results, ultimately returning the integrated query results to the tenant through the Kyubi service layer.

[0061] In addition, this embodiment of the invention can also provide system operation status monitoring, performance index analysis, and fault diagnosis functions through the monitoring device 60, supporting the visualized operation and maintenance management of the system. For example, during the execution of a data query, this embodiment of the invention can collect performance indicators of the Kyubi service layer 10, the federated query engine 20, the route optimizer 30, the distributed execution engine 40, and the metadata management module 50 through the monitoring device 60 to monitor the operation status of each component, thereby detecting whether any component is abnormal in real time. If any one or more components are detected to be abnormal, an alarm will be triggered and / or the backup node corresponding to the abnormal component will be automatically switched. If the monitoring device 60 detects any one or more components to be abnormal, and the query fails, the Kyubi service layer 10 can execute a transaction rollback, and the route optimizer 30 can also obtain the query data corresponding to the failed query request (e.g., obtain the corresponding query data from the logs recorded by the monitoring device 60), dynamically learn the query failure pattern based on the query data, and update the model for calculating the execution cost based on the learning results. In other words, in this embodiment of the invention, the Kyubi service layer 10 rolls back the transaction when the query fails, returns a structured error code, and supports a client retry mechanism. The route optimizer can dynamically learn failure modes and update the model for calculating execution costs to avoid inefficient paths.

[0062] The data query method provided in the embodiments of the present invention will be described in detail below. This data query method is based on... Figure 1 The data query system shown is implemented as follows. Figure 2 As shown, the data query method provided in this embodiment of the invention mainly includes the following steps:

[0063] Step S201: The Kyubi service layer obtains data query requests initiated by the tenant for multiple heterogeneous data sources, and authenticates the tenant's permissions based on the data query requests; if the authentication is successful, proceed to step S202.

[0064] Before executing the data query method, the data query system can be initialized. That is, before step S201, the data query system can be initialized to start the Kyubi service instance, load pre-configured data source connection information and system parameters, and load multi-tenant policies and resource configurations. The tenant's resource configuration corresponds to the maximum configurable resource based on the tenant's permissions. Additionally, the metadata management module 50 initializes the data gateway, establishes connections with each data source through a pre-configured connection pool, and loads dedicated drivers for each data source in the heterogeneous data source connector cluster, verifying data source reachability. Furthermore, the monitoring device 60 starts a resource usage monitoring thread to monitor the running status of each component during the data query process. Thus, initializing each component ensures the system's operating environment is ready.

[0065] Taking a JOIN query between the relational database MySQL and the big data platform Hive as an example, during the initialization process of the data query system, the system loads the corresponding connectors for MySQL and Hive respectively when it starts up, and synchronously loads tenant configurations and metadata, etc., which can be implemented through the following program:

[0066]

[0067]

[0068] The heterogeneous data source connector cluster includes multiple connectors for connecting multiple heterogeneous data sources (such as MySQL databases, S3 cloud storage services, MongoDB databases, and Hadoop data platforms). Distributed execution engines (such as Spark) establish connections with these heterogeneous data sources through this cluster. The Kyubi service layer inherits the connectors from the distributed execution engine to support data query requests from multiple heterogeneous data sources. Thus, the Kyubi service layer, acting as a gateway for the computing engine, inherits and integrates the connector ecosystem of the underlying engine, providing users with stable multi-tenant management services without requiring repeated connector configuration. For example, the Kyubi service layer receives SQL requests (data query requests) initiated by tenant clients via the JDBC / ODBC protocol and authenticates the tenant's identity and permissions based on these requests. For instance, after receiving an SQL request from a tenant, the Kyubi service layer can first verify the tenant's identity through a unified authentication and authorization layer (e.g., verifying the tenant's identity based on the username and password provided by the user). After successful tenant authentication, it then authenticates the tenant's permissions based on the tenant's group, enabling operation based on user identity and permission checks, thereby achieving secure isolation between different tenants.

[0069] After tenant identity and permission authentication are successful, the Kyubi service layer can determine the resource configuration corresponding to the tenant based on the tenant's permissions. This allows the distributed execution engine to dynamically allocate resources based on the resource configuration during the subsequent scheduling and execution of distributed tasks according to the target execution plan. For example, the Kyubi service layer can configure different Spark resources (such as executor memory and CPU core count) for engines with different sharing levels. It can also configure the resources of different engines by limiting the maximum execution time of a single SQL query and dynamically limiting the number of concurrent tasks scheduled for a single SQL query.

[0070] Step S202: The federated query engine parses the data query request and obtains the parsing results; the parsing results include: cross-source table structures and index information corresponding to multiple heterogeneous data sources.

[0071] After tenant identity and permissions are authenticated at the Kyubi service layer, the federated query engine can parse the SQL text corresponding to the SQL request, performing lexical analysis, syntax analysis, and semantic analysis to identify the data sources and query patterns involved. During this process, the SQL text can be parsed into an abstract syntax tree (AST), in which table names and column names are not yet determined. Then, based on the metadata information maintained by the metadata management module, the federated query engine obtains the cross-source table structure and index information corresponding to the SQL request. For example, by querying metadata such as management tables, functions, and partitions, the federated query engine determines the table structure and column types, and then outputs the analyzed logical plan, obtaining the corresponding parsing results (including cross-source table structure and index information). In this logical plan, the data source corresponding to this query is determined, as are the corresponding table structure and column types. When the metadata management module receives a query parsing request from the federated query engine, it can obtain the corresponding cross-source table structure and index information from locally cached data and / or data sources, and assist in generating a logical query plan. It then provides the federated query engine with relevant statistical table information, which the federated query engine uses to obtain the parsing results.

[0072] Taking the aforementioned JOIN query between the relational database MySQL and the big data platform Hive as an example, the federated query engine can parse the query request, identify the cross-source JOIN operation, and call the metadata management module to obtain the corresponding table statistics, thus obtaining the parsing result. This step can be implemented, for example, through the following procedure:

[0073]

[0074] In addition, the process by which the metadata management module analyzes and retrieves data from locally cached data and / or data sources based on query parsing requests can be implemented, for example, through the following procedure:

[0075]

[0076] Step S203: The route optimizer, based on the Catalyst optimizer framework, constructs a target execution plan according to the parsing results. The execution cost of this target execution plan meets the preset cost conditions.

[0077] In this embodiment of the invention, during the process of constructing the target execution plan, the routing optimizer can search for target historical data from historical query performance data based on the parsing results. The similarity between the cross-source table structure corresponding to the target historical data and the cross-source table structure in the parsing results is greater than a preset first threshold, and the similarity between the index information corresponding to the target historical data and the index information in the parsing results is greater than a second threshold. Then, based on the target historical data and the parsing results, an SQL query statement is constructed, and the target execution plan is determined based on the SQL query statement.

[0078] Understandably, during the processing of historical query requests, the routing optimizer also constructs historical execution plans (SQL query statements) based on the parsing results of the historical query requests. After executing the corresponding distributed tasks according to the historical execution plans, historical query performance data is obtained. During the processing of the current data query request, the routing optimizer can search for target historical data from the historical query performance data. The data source and index information corresponding to this target historical data have a high similarity to the data source and index information corresponding to the current data query request. For example, the similarity between the cross-source table structure corresponding to the target historical data and the cross-source table structure in the parsing results of the current data query request is greater than a preset first threshold, and the similarity between the index information corresponding to the target historical data and the index information in the parsing results of the current data query request is greater than a second threshold. The first and second thresholds can be set according to actual needs; they can be equal or unequal. After finding the target historical data, the routing optimizer can construct the SQL query statement corresponding to the current data query request based on the SQL query statement corresponding to the target historical data and the parsing results of the current data query request, and then determine the target execution plan based on the SQL query statement. In constructing the SQL query statement, since the target historical data is highly similar to the current data query request, part or all of the SQL query statement corresponding to the target historical data can be directly used as the SQL query statement corresponding to the current data query request, thereby improving the efficiency of generating the target execution plan.

[0079] Furthermore, in this embodiment of the invention, the routing optimizer can evaluate the execution cost of different execution plans based on the acquired metadata and cost model, and select the optimal target execution plan according to the execution cost. Specifically, the routing optimizer, based on the Catalyst optimizer framework, constructs multiple execution plans according to the parsing results. Each execution plan includes the data to be read corresponding to each heterogeneous data source, and one or more nodes for data processing. The execution cost of each execution plan is determined based on the execution factors corresponding to the multiple execution plans. The execution factors include any one or more of the following: network transmission cost corresponding to the data to be read, local computing cost of one or more nodes for data processing, and data size. The execution plan with the lowest execution cost among the multiple execution plans is selected as the target execution plan.

[0080] The routing optimizer transforms the analyzed parsing results (logical plan) according to predefined rules. For example, it matches the logical plan with specific operations (such as filtering or column pruning) based on the rules. Different combinations of operations can generate different execution plans. Each execution plan includes the data to be queried locally from the corresponding data source (the data to be read) and one or more nodes for executing the data (such as nodes for predicate pushdown or column pruning). Since the data to be queried from different data sources and the nodes for executing the data are different in different execution plans, the execution costs corresponding to different execution plans are also different. To improve query efficiency and optimize query routing, the routing optimizer selects the optimal target execution plan based on the execution cost. For example, the weight values ​​corresponding to network transmission cost, local computing cost (such as CPU resource overhead, local execution time consumption, etc.), and data scale (such as the size of the query data) can be pre-configured. The routing optimizer can determine the execution cost based on the weighted sum of the various execution factors, and then select the lowest execution cost as the target execution plan based on the execution cost of different execution plans. Understandably, after determining the target execution plan, the routing optimizer can rewrite the logical plan based on the target execution plan, that is, generate an optimized logical plan based on the execution cost. This optimized logical plan contains pushdown information at the data source level.

[0081] For example, the route optimizer can select the target execution plan through the following procedure:

[0082] Furthermore, the routing optimizer can also optimize the logical plan by incorporating historical performance data. For example, after processing historical query requests, historical query performance data corresponding to those requests can be collected and stored. When processing the current data query request, the execution cost (e.g., total execution time) of the logical plan corresponding to the data query request can be estimated based on the historical query performance data. Then, the target execution plan with the lowest execution cost is selected based on the estimation result. In this embodiment of the invention, historical performance data can be combined with execution factors such as network transmission cost, local computing cost, and data size to determine the execution cost of the execution plan. For example, historical performance data can be used as an execution factor with corresponding weight values. Then, the weighted sum value can be calculated by combining it with other execution factors such as network transmission cost, local computing cost, and data size to obtain the execution cost of the corresponding execution plan, thereby obtaining the target execution plan (optimized logical plan) with the lowest execution cost.

[0083] After generating the optimized logical plan, the routing optimizer can further transform it into one or more physical plans. The distributed execution engine then receives the optimized physical plans and breaks them down into query tasks that can be executed in parallel on a distributed cluster (such as a Spark cluster) and locally on the data source. In this embodiment, the routing optimizer divides the target execution plan into multiple stages and determines multiple subquery plans corresponding to these stages, as well as the execution order, data source, and join strategy for each subquery plan. The distributed execution engine can generate subquery tasks corresponding to the subquery plans and distribute these tasks to the appropriate data sources, enabling the data sources to execute the query, obtain the data to be read, and thus obtain the query results.

[0084] During the process of generating the physical plan based on the optimized logical plan, the routing optimizer can also generate an efficient physical plan based on the cost model or heuristic rules. The physical plan includes the data source providing the data to be queried and its connection strategy (e.g., the connection strategy may include directly pulling the data from the data source or retrieving the data from the local cache). Furthermore, since the target execution plan is broken down into multiple subquery plans corresponding to multiple stages, a subquery plan in one stage may depend on the processing results of a subquery plan in another stage; therefore, the physical plan also includes the execution order of the corresponding subquery plans. After the physical plan is generated, the distributed execution engine can generate subquery tasks for each subquery plan accordingly. For example, the Spark engine generates subquery tasks corresponding to multiple stages of the physical plan and assigns each subquery task to a task execution unit (such as an Executor) for execution.

[0085] In addition, the route optimizer can subscribe to metadata change notifications. Upon receiving a metadata change notification, it updates the data source information according to the notification and generates an execution plan corresponding to the target query request based on the updated data source information. The metadata change notification is sent by the data source based on the metadata change event, and the target query request is obtained through the Kyubi service layer after the data change event occurs.

[0086] When information in the data source changes (such as additions, updates, or deletions), i.e., a metadata change event occurs, the metadata management module can send a metadata change notification corresponding to the event to the routing optimizer that has subscribed to metadata change notifications. The routing optimizer can then update the data source information based on the data change notification. Subsequently, if a new target query plan is received, the routing optimizer can generate a corresponding execution plan based on the updated data source information. In other words, the routing optimizer can dynamically update the data source characteristics by subscribing to metadata change notifications and generate a corresponding execution plan based on the update results. This allows it to calculate the execution cost based on the latest characteristics of the data source and generate the optimal execution plan, thereby improving query efficiency.

[0087] Step S204: The distributed execution engine schedules and executes distributed tasks according to the target execution plan to obtain the query results.

[0088] During task execution, the distributed execution engine distributes subquery tasks to the corresponding data sources according to the optimized physical plan, enabling the data sources to execute the query and obtain the corresponding data to be read. For example, the distributed execution engine connects to multiple heterogeneous data sources through a heterogeneous data source connector cluster. During task execution, the subquery tasks are distributed to the data sources through the corresponding data source connectors, allowing the data sources to execute local queries. Then, the distributed execution engine obtains the subquery results through the data source connectors and performs necessary data transformation and format unification.

[0089] Furthermore, during task execution, the distributed execution engine can dynamically allocate Executor resources based on the resource configuration corresponding to tenant permissions, coordinating cross-source data reading, thereby achieving tenant-level resource quota control. Additionally, the distributed execution engine leverages the inherent advantages of the data source itself during task execution, ensuring that computational processes (filtering, projection, and aggregation, etc.) are as close to the data source as possible. For example, for data sources supporting pushdown, filtering conditions are distributed through a heterogeneous data source connector cluster, reducing data transfer volume. The distributed execution engine can also utilize distributed nodes to improve task execution efficiency; for example, for distributed data, Spark dynamic partitioning and pruning optimize read performance; for object storage, columnar pruning and compression format decoding are enabled for optimization. Thus, by using a heterogeneous data source connector cluster (such as connectors supporting protocols like JDBC, Hive, and S3) to abstract and encapsulate the local query capabilities of the data source, and combining this with the distributed execution engine's adaptive optimization strategies (such as predicate pushdown, partitioning filtering, and column pruning), data processing nodes are dynamically selected (local execution or remote computation), thereby reducing cross-source data transfer volume and ensuring compatibility with various data source characteristics. This helps solve the performance bottleneck of traditional data query scenarios that require full data retrieval and processing.

[0090] After executing a local query on the data source, the distributed execution engine can obtain the corresponding subquery results through the heterogeneous data source connector. After obtaining the subquery results for each subquery task, the distributed execution engine performs cross-source aggregation operations on the subquery results to obtain the final query results. For example, the distributed execution engine performs joins and aggregations according to SQL semantics to generate the final query results. Finally, the Kyubi service layer encapsulates the query results into a result set object and streams the result set object back to the tenant's corresponding client. During the process of obtaining query results through the heterogeneous data source connector, the distributed execution engine can automatically handle network retries and resume interrupted downloads based on the heterogeneous data source connector.

[0091] Taking the aforementioned JOIN query between the relational database MySQL and the big data platform Hive as an example, the process of generating a query plan and executing the query task can be achieved through the following procedure:

[0092]

[0093]

[0094] Step S205: Upon receiving the query results from the distributed execution engine, the Kyubi service layer sends the query results to the tenant.

[0095] The Kyuubi service layer encapsulates query results into ResultSet objects and streams these objects back to the tenant's client. For example, the Kyuubi service layer receives query results, encapsulates them into ResultSet objects according to the JDBC protocol, and streams them back to the tenant's client. After returning the query results to the tenant, the Kyuubi service layer can also record query execution logs for subsequent performance analysis. Monitoring devices can then generate visual reports based on these execution logs.

[0096] In addition, embodiments of the present invention can also provide system operation status monitoring, performance index analysis, and fault diagnosis functions through the monitoring device 60, supporting visualized operation and maintenance management of the system. For example, during the execution of data queries, embodiments of the present invention can collect performance indicators of the Kyubi service layer 10, federated query engine 20, route optimizer 30, distributed execution engine 40, and metadata management module 50 through the monitoring device 60 to monitor the operation status of each component, thereby detecting whether any component is abnormal in real time. If any one or more components are detected to be abnormal, an alarm can be initiated and / or the backup node corresponding to the abnormal component can be automatically switched. If the monitoring device 60 detects an abnormality in any one or more of the Kyubi service layer 10, federated query engine 20, route optimizer 30, and distributed execution engine 40, and the query fails, the Kyubi service layer 10 can execute a rollback transaction, and the route optimizer 30 can also obtain the query data corresponding to the failed data query request (e.g., obtain the corresponding query data from the logs recorded by the monitoring device 60), dynamically learn the query failure pattern based on the query data, and update the model for calculating execution costs based on the learning results. In other words, the Kyubi service layer 10 in this embodiment rolls back the transaction and returns a structured error code when a query fails, supporting a client retry mechanism. The route optimizer can dynamically learn failure modes and update the model for calculating execution costs to avoid inefficient paths.

[0097] Taking the aforementioned JOIN query between the relational database MySQL and the big data platform Hive as an example, the process of integrating the query results and streaming them back to the tenant client, as well as the monitoring device recording the query time, can be achieved through at least the following procedures:

[0098]

[0099] According to the data query method provided in any of the above embodiments, by integrating the Kyubi service layer with the federated query engine, unified access support for multiple heterogeneous data sources is achieved. Furthermore, tenant-level resource quota control is implemented through the Kyubi service layer. Combined with the routing optimizer and the distributed execution engine, the optimal execution plan is dynamically generated, enabling efficient execution of cross-data source queries. This provides high-performance and easily maintainable federated query services for data queries, solving the problems of resource contention and low execution efficiency in traditional query scenarios, improving resource utilization and query response speed, thereby reducing data integration costs and improving data analysis efficiency.

[0100] In addition, this embodiment of the invention also provides a data query system, which includes a Kyubi service layer, a federated query engine, a route optimizer, and a distributed execution engine; wherein, the Kyubi service layer is configured to: obtain data query requests initiated by tenants for multiple heterogeneous data sources, authenticate tenant permissions according to the data query requests, and, if the authentication is successful, start the federated query engine according to the tenant permissions corresponding to the authentication result;

[0101] The federated query engine is configured to parse data query requests and obtain the parsing results, which include the cross-source table structure and index information corresponding to multiple heterogeneous data sources.

[0102] The routing optimizer is configured as follows: based on the Catalyst optimizer framework, it constructs the target execution plan according to the parsing results, and the execution cost of the target execution plan meets the preset cost conditions;

[0103] The distributed execution engine is configured to schedule and execute distributed tasks according to the target execution plan to obtain query results;

[0104] The Kyuubi service layer is configured to send the query results to the tenant upon receiving them from the distributed execution engine.

[0105] In one embodiment of the present invention, the routing optimizer is configured as follows: based on the Catalyst optimizer framework, multiple execution plans are constructed according to the parsing results; each execution plan includes the data to be read corresponding to each heterogeneous data source, and one or more nodes for data processing; the execution cost of each execution plan is determined according to the execution factors corresponding to the multiple execution plans; the execution plan with the lowest execution cost among the multiple execution plans is selected as the target execution plan; wherein, the execution factors include any one or more of the following: the network transmission cost corresponding to the data to be read, the local computing cost of one or more nodes for data processing, and the data size.

[0106] In one embodiment of the present invention, the routing optimizer is configured to: divide the target execution plan into multiple stages, and determine multiple sub-query plans corresponding to the multiple stages, as well as the execution order, data source and connection strategy corresponding to the multiple sub-query plans;

[0107] The distributed execution engine is configured to generate subquery tasks corresponding to the subquery plan, distribute the subquery tasks to the corresponding data sources, so that the data sources can execute the query to obtain the data to be read and obtain the query results.

[0108] In one embodiment of the present invention, the Kyuubi service layer is further configured to: determine the resource configuration corresponding to the tenant based on the tenant's permissions;

[0109] The distributed execution engine is also configured to dynamically allocate resources based on resource configuration, so as to utilize the allocated resources to execute distributed tasks.

[0110] In one embodiment of the present invention, the distributed execution engine is connected to multiple heterogeneous data sources through multiple connectors; the Kyuubi service layer inherits the connectors of the distributed execution engine to support data query requests for multiple heterogeneous data sources.

[0111] In one embodiment of the present invention, the routing optimizer is further configured to: subscribe to metadata change notifications; upon receiving a metadata change notification, update the data source information according to the metadata change notification, and generate an execution plan corresponding to the target query request based on the updated data source information; the target query request is obtained through the Kyubi service layer after the data change event occurs.

[0112] In one embodiment of the present invention, the distributed execution engine is configured to: obtain the subquery results of each subquery task, perform cross-source aggregation operation on the subquery results, and obtain the query results;

[0113] The Kyuubi service layer is configured to encapsulate query results into result set objects and stream these result set objects back to the corresponding client of the tenant.

[0114] In one embodiment of the present invention, the data query system further includes: a monitoring device; wherein the monitoring device is configured to: detect in real time whether the Kyubi service layer, the federated query engine, the route optimizer and the distributed execution engine are abnormal; and if any one or more of the Kyubi service layer, the federated query engine, the route optimizer and the distributed execution engine are abnormal and the query fails, start the Kyubi service layer to perform a rollback transaction.

[0115] The Kyuubi service layer is also configured to perform a rollback transaction to respond to data query requests again.

[0116] In one embodiment of the present invention, the routing optimizer is further configured to: obtain query data corresponding to the query request that failed to query, dynamically learn the query failure pattern based on the query data, and update the model for calculating the execution cost based on the learning result.

[0117] The data query system provided in the above embodiments integrates the Kyubi service layer with the federated query engine to achieve unified access support for multiple heterogeneous data sources. Furthermore, the Kyubi service layer enables tenant-level resource quota control, and combined with the routing optimizer and distributed execution engine, it achieves efficient execution of cross-data source queries. This provides high-performance, easy-to-maintain federated query services for data queries, reduces data integration costs, and improves data analysis efficiency.

[0118] In addition, at least some embodiments of the present invention also provide an electronic device. Figure 3 A schematic diagram of an electronic device 300 according to at least one embodiment of the present invention is shown.

[0119] like Figure 3 As shown, the electronic device 300 includes one or more processors 310 and a memory 320. The memory 320 includes one or more computer program modules 321. The one or more computer program modules 321 are stored in the memory 320 and configured to be executed by the processor 310. These computer program modules 321 include instructions for executing a data query method and its additional aspects according to embodiments of the present invention. When executed by the processor 310, they can perform one or more steps of the data query method and its additional aspects according to embodiments of the present invention. The memory 320 and the processor 310 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, the bus can be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0120] For example, processor 310 may be a central processing unit (CPU), a digital signal processor (DSP), or other processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA). Processor 310 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 300 to perform desired functions.

[0121] Exemplarily, memory 320 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules 321 may be stored on the computer-readable storage medium, and processor 310 may run one or more computer program modules 321 to implement various functions of electronic device 300. The computer program modules include multiple computer-executable instructions. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0122] For example, electronic device 300 may also include input devices such as touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices such as liquid crystal displays, speakers, and vibrators; storage devices such as magnetic tapes and hard disks (HDDs or SDDs); and communication devices such as network interface cards like LAN cards and modems. The communication devices allow electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. A drive is connected to the I / O interface as needed. Removable storage media, such as disks, optical disks, magneto-optical disks, and semiconductor memories, are installed on the drive as needed so that computer programs read from them can be installed into the storage device as required.

[0123] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure). This peripheral interface can be of various types, such as a USB interface, a Lightning interface, etc. The communication device can communicate wirelessly with networks and other devices, such as the Internet, intranets and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0124] The electronic device 300 may be, for example, a system-on-a-chip (SOC) or a device including the SOC. For instance, it can be any device such as a mobile phone, tablet computer, laptop computer, e-reader, game console, television, digital photo frame, navigator, home appliance, communication base station, industrial controller, server, etc., or any combination of data processing devices and hardware. The embodiments disclosed herein do not limit this. The specific functions and technical effects of the electronic device 300 can be found in the above description of the data query method and its additional aspects according to embodiments of the present invention, and will not be repeated here.

[0125] Figure 4 A schematic diagram of a computer-readable storage medium 400 according to at least one embodiment of the present disclosure is shown.

[0126] like Figure 4 As shown, a non-transitory computer instruction 410 is stored on a computer-readable storage medium 400, which, when executed by a processor, performs one or more steps of the data query method and its additional aspects as described above.

[0127] For example, when the program code is read by a computer, the computer can execute the program code stored in the computer storage medium to perform one or more steps of, for example, the data query method and its additional aspects according to at least one embodiment of the present disclosure.

[0128] For example, the computer-readable storage medium may include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, and other computer-readable media or any combination thereof.

[0129] At least some of the embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0130] According to the technical solution of the present invention, by integrating the Kyubi service layer with the federated query engine, unified access support for multiple heterogeneous data sources is achieved. Furthermore, the Kyubi service layer enables tenant-level resource quota control. Combined with the routing optimizer and the distributed execution engine, efficient execution of cross-data source queries is achieved. This provides high-performance and easy-to-maintain federated query services for data queries, reduces data integration costs, and improves data analysis efficiency.

[0131] It should be noted that, in this document, relational terms such as "first," "second," etc., are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0132] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data query system, characterized in that, include: The Kyuubi service layer comprises a federated query engine, a route optimizer, and a distributed execution engine; among which... The Kyubi service layer is configured to: obtain data query requests initiated by tenants for multiple heterogeneous data sources, authenticate tenant permissions based on the data query requests, and, if authentication is successful, start the federated query engine based on the tenant permissions corresponding to the authentication result; The federated query engine is configured to parse the data query request and obtain the parsing result; the parsing result includes: the cross-source table structure and index information corresponding to the multiple heterogeneous data sources; The routing optimizer is configured to: construct a target execution plan based on the Catalyst optimizer framework, and the execution cost of the target execution plan satisfies a preset cost condition; The distributed execution engine is configured to schedule and execute distributed tasks according to the target execution plan to obtain query results; The Kyubi service layer is also configured to send the query results to the tenant upon receiving the query results from the distributed execution engine.

2. The data query system according to claim 1, characterized in that, The routing optimizer is also configured to: construct multiple execution plans based on the Catalyst optimizer framework and the parsing results; each execution plan includes the data to be read corresponding to each heterogeneous data source, and one or more nodes for data processing; The execution cost of each execution plan is determined based on the execution factors corresponding to the multiple execution plans; the execution plan with the lowest execution cost among the multiple execution plans is selected as the target execution plan; wherein, the execution factors include any one or more of the following: the network transmission cost corresponding to the data to be read, the local computing cost of one or more nodes performing data processing, and the data size.

3. The data query system according to claim 2, characterized in that, The routing optimizer is further configured to: divide the target execution plan into multiple stages, and determine multiple sub-query plans corresponding to the multiple stages, as well as the execution order, data source, and connection strategy corresponding to the multiple sub-query plans; The distributed execution engine is further configured to: generate subquery tasks corresponding to the subquery plan, and distribute the subquery tasks to the corresponding data sources, so that the data sources can execute the query to obtain the data to be read and obtain the query result.

4. The data query system according to claim 1, characterized in that, The Kyubi service layer is also configured to: determine the resource configuration corresponding to the tenant based on the tenant's permissions; The distributed execution engine is further configured to dynamically allocate resources according to the resource configuration, so as to use the allocated resources to execute the distributed task.

5. The data query system according to claim 1, characterized in that, The distributed execution engine is connected to multiple heterogeneous data sources through multiple connectors; The Kyubi service layer inherits the connector of the distributed execution engine to support data query requests for multiple heterogeneous data sources.

6. The data query system according to claim 1, characterized in that, The routing optimizer is also configured to: subscribe to metadata change notifications; upon receiving a metadata change notification, update the data source information according to the metadata change notification, and generate an execution plan corresponding to the target query request based on the updated data source information; The target query request is obtained through the Kyubi service layer after a data change event occurs.

7. The data query system according to claim 3, characterized in that, The distributed execution engine is further configured to: obtain the subquery results of each subquery task, perform cross-source aggregation operations on the subquery results, and obtain the query results; The Kyuubi service layer is also configured to encapsulate the query results into a result set object and stream the result set object back to the client corresponding to the tenant.

8. The data query system according to claim 1, characterized in that, Also includes: Monitoring devices; among which, The monitoring device is configured to: detect in real time whether the Kyubi service layer, the federated query engine, the route optimizer, and the distributed execution engine are abnormal; if any one or more of the Kyubi service layer, the federated query engine, the route optimizer, and the distributed execution engine are abnormal and the query fails, start the Kyubi service layer to perform a rollback transaction. The Kyuubi service layer is also configured to execute the rollback transaction to respond to the data query request again.

9. The data query system according to claim 8, characterized in that, The routing optimizer is further configured to: obtain query data corresponding to failed query requests, dynamically learn the query failure pattern based on the query data, and update the model for calculating execution cost based on the learning results.

10. A data query method, characterized in that, include: The Kyuubi service layer obtains data query requests initiated by users for multiple heterogeneous data sources, authenticates tenant permissions based on the data query requests, and starts the federated query engine based on the tenant permissions corresponding to the authentication result if the authentication is successful. The federated query engine parses the data query request and obtains the parsing result; The parsing results include: the cross-source table structure and index information corresponding to the multiple heterogeneous data sources; The routing optimizer is based on the Catalyst optimizer framework and constructs a target execution plan based on the parsing results. The execution cost of the target execution plan meets a preset cost condition. The distributed execution engine schedules and executes distributed tasks according to the target execution plan to obtain query results; Upon receiving the query results from the distributed execution engine, the Kyuubi service layer sends the query results to the tenant.

11. The data query method according to claim 10, characterized in that, The routing optimizer, based on the Catalyst optimizer framework, constructs a target execution plan based on the parsing results, including: Multiple execution plans are constructed based on the parsing results; each execution plan includes the data to be read corresponding to each heterogeneous data source, and one or more nodes for data processing. The execution cost of each execution plan is determined based on the execution factors corresponding to the multiple execution plans; the execution factors include any one or more of the following: the network transmission cost corresponding to the data to be read, the local computing cost of one or more nodes performing data processing, and the data size; The execution plan with the lowest execution cost among the multiple execution plans is selected as the target execution plan.

12. The data query method according to claim 10, characterized in that, If authentication is successful, the service layer further includes: determining the resource configuration corresponding to the tenant based on the tenant's permissions; The distributed execution engine schedules and executes distributed tasks according to the target execution plan, including: the distributed engine dynamically allocates resources according to the resource configuration to use the allocated resources to execute the distributed tasks.

13. An electronic device, characterized in that, include: One or more processors; A storage device, communicatively connected to the one or more processors, is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data query method as described in any one of claims 10-12.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data query method as described in any one of claims 10-12.

Images