Multi-data source data query method, device and system

Abstract

The present invention discloses a multi-data source data query method, device and system, which belongs to the field of federated query; Trino is used to connect all data sources to obtain metadata of each data source to shield the field differences of heterogeneous data sources, and is converted into a directed acyclic graph according to the user's selection and arrangement configuration to generate a unified query SQL statement. There is no need to use different SQL statements according to different data sources. The present application scheme uniformly arranges the metadata of heterogeneous data from multiple data sources, parses and generates a unified query SQL to realize cross-multi-data source big data query. It shields the field and SQL differences of heterogeneous data sources, and at the same time can arrange the query nodes to generate query SQL to realize convenient data analysis query. It provides excellent expansion capabilities, reduces the complexity of the system, and realizes low-threshold massive data multi-data source data arrangement analysis query.

Description

Technical Field

[0001] The present invention relates to the field of federated query, and in particular, to a method, device and system for querying data from multiple data sources. Background Art

[0002] Information is rapidly developing. In the era of big data, federated queries across large data volumes and heterogeneous data sources are becoming increasingly common. For example, data stored in Hive, EleasticSearch, and ClickHouse can be connected for correlation query analysis to yield data analysis results. Federated queries of heterogeneous data from multiple data sources with large volumes of data require cross-data connection queries. Furthermore, heterogeneous data sources generally have different query syntax and lack a unified SQL (Structured Query Language) for data querying. Therefore, in the context of big data and multiple data sources with a large amount of metadata, data analysts must always be aware of the metadata when writing query SQL. They then need to write different SQL statements based on the data sources with different metadata to obtain the final query results, making the operation complex. Summary of the Invention

[0003] In order to overcome the shortcomings of the existing technology, the present invention provides a multi-data source data query method, device and system to solve the problem that when querying metadata from multiple data sources under big data, data analysts need to clearly know the metadata at all times when writing query SQL, and then write different SQL statements according to the data sources of different metadata to obtain the final query results, which is complicated.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] In a first aspect, a method for querying data from multiple data sources is provided, comprising the following steps:

[0006] Use Trino to connect all data sources to obtain metadata for each data source, including data for all databases, tables, and columns corresponding to each data source.

[0007] receiving user selections and choreography configurations;

[0008] Converting the orchestrated data into a directed acyclic graph comprising nodes and edges;

[0009] A depth traversal algorithm is used to traverse the directed acyclic graph, and a stack and a HashMap are used to store and parse the nodes and edges of the directed acyclic graph to generate a unified query SQL;

[0010] The unified query SQL is sent to the Trino to obtain the query result.

[0011] Furthermore, Trino is used to connect all data sources to obtain metadata for each data source, including:

[0012] Use JDBC-CLIENT to connect to the Trino service;

[0013] Calling the Trino to obtain data of all databases, tables, and columns corresponding to each data source, the data including the corresponding database name, table name, or column name, and specific data;

[0014] All data are encapsulated in a unified format to obtain the metadata of each data source.

[0015] Furthermore, the calling of Trino to obtain data of all databases, tables, and columns corresponding to each data source includes:

[0016] Call Trino SHOW CATALOGS[LIKE pattern] to obtain data from the data source;

[0017] Call Trino SHOW SCHEMAS [FROM catalog] [LIKE pattern] and SHOW CREATESCHEMA schema_name to obtain database data;

[0018] Call Trino SHOW TABLES [FROM schema] [LIKE pattern] and SHOW SHOW COLUMNSFROM table [LIKE pattern] to obtain table data.

[0019] Furthermore, the receiving of user selection and arrangement configuration includes:

[0020] Receive multiple data sources, libraries, tables, and columns selected by the user; and select operations for each data source, library, table, and column, including associated query, filtering, group by aggregation, and sorting.

[0021] Furthermore, converting the arranged and configured data into a directed acyclic graph includes:

[0022] Each operation in the orchestration configuration is regarded as a node, and an edge is set between two associated operations to convert the data after the orchestration configuration into a directed acyclic graph, wherein the data is stored in the node.

[0023] Furthermore, the directed acyclic graph is traversed using a depth traversal algorithm, and the nodes and edges of the directed acyclic graph are stored and parsed using a stack and a HashMap to generate a unified query SQL, including:

[0024] Generate a stack for storing nodes of graph traversal, a HashMap for storing association relationships resolved by edge traversal, and a LIST for temporarily storing data popped from the node stack according to the directed acyclic graph;

[0025] When a JOIN NODE is encountered, the graph traversal stops, a mark is placed, and the NODE is popped from the stack and placed into the LIST. This continues until the stack is empty or a SUB SELECT subquery is encountered, at which point the popping stops.

[0026] Traverse the LIST from back to front and parse to generate a select clause corresponding to a branch;

[0027] Recursively obtain the previous node of the marked JOIN node from the HashMap until the unrelated key is found, and obtain the start node of the adjacent branch;

[0028] Start the depth traversal of the new branch until the traversal reaches the currently marked JOIN node and generates the select clause of the new branch;

[0029] Connect all branch select clauses associated with JOIN NODE with join;

[0030] Starting from the marked JOIN NODE, the subsequent nodes are deeply traversed until the entire graph is traversed and a complete unified query SQL is generated.

[0031] Furthermore, generating a stack for storing nodes of the graph traversal and a HashMap for storing association relationships resolved by edge traversal according to the directed acyclic graph includes:

[0032] Push the nodes into the stack in the traversal order, and store the nodeId of the FROM and END nodes of the edge resolution as the key-value of the HashMap.

[0033] Furthermore, the multiple data sources include: Hive, HBase, EleasticSearch, Kudu, Kafka, MySQL, and Redis.

[0034] In a second aspect, a multi-data source data query device is provided, comprising:

[0035] A metadata acquisition module is used to connect all data sources using Trino to obtain metadata for each data source, including data from all databases, tables, and columns corresponding to each data source.

[0036] The orchestration configuration module is used to receive user selections and orchestrate configurations;

[0037] A data conversion module, configured to convert the orchestrated data into a directed acyclic graph comprising nodes and edges;

[0038] An SQL generation module is used to traverse the directed acyclic graph using a depth traversal algorithm, and to store and parse the nodes and edges of the directed acyclic graph using a stack and a HashMap to generate a unified query SQL;

[0039] The result query module is used to send the unified query SQL to the Trino to obtain the query result.

[0040] Thirdly, we provide a multi-data source data query system, including:

[0041] processor;

[0042] a memory for storing instructions executable by the processor;

[0043] The processor is configured to execute the method described in any one of the technical solutions of the first aspect.

[0044] Beneficial effects:

[0045] The technical solution of this application provides a multi-data source data query method, device and system, which uses Trino to connect all data sources to obtain the metadata of each data source, so as to shield the field differences of heterogeneous data sources, and convert it into a directed acyclic graph according to the user's selection and arrangement configuration to generate a unified query SQL statement. There is no need to use different SQL statements according to different data sources. This application solution uniformly arranges the metadata of heterogeneous data from multiple data sources, parses and generates a unified query SQL to realize cross-multi-data source big data query. It shields the field and SQL differences of heterogeneous data sources, and can also arrange the query nodes to generate query SQL to realize convenient data analysis query. It provides excellent expansion capabilities, reduces the complexity of the system, and realizes low-threshold massive data multi-data source data arrangement analysis query. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a flow chart of a multi-data source data query method provided by an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of an arrangement configuration provided by an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a directed acyclic graph structure provided by an embodiment of the present invention;

[0050] Figure 4 This is a diagram of a graph traversal of a directed acyclic graph provided by an embodiment of the present invention;

[0051] Figure 5 The present invention provides a schematic diagram of the structure of a multi-data source data query device. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of the present invention are described in detail below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some of the embodiments of this application, rather than all of them. Based on the embodiments in this application, all other implementation methods obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] First embodiment, refer to Figure 1 , an embodiment of the present invention provides a multi-data source data query method, comprising the following steps:

[0054] S11: Use Trino to connect all data sources to obtain metadata for each data source. The metadata includes data for all databases, tables, and columns corresponding to each data source.

[0055] S12: receiving user selection and arrangement configuration;

[0056] S13: Convert the orchestrated data into a directed acyclic graph, which includes nodes and edges.

[0057] S14: Use the depth traversal algorithm to traverse the directed acyclic graph, and use stacks and HashMaps to store and parse the nodes and edges of the directed acyclic graph to generate unified query SQL;

[0058] S15: Send the unified query SQL to Trino to obtain the query result.

[0059] The multi-data source data query method provided by the embodiment of the present invention adopts Trino to connect all data sources to obtain the metadata of each data source, so as to shield the field differences of heterogeneous data sources, and convert it into a directed acyclic graph according to the user's selection and arrangement configuration to generate a unified query SQL statement. There is no need to use different SQL statements according to different data sources. The present application scheme uniformly arranges the metadata of heterogeneous data from multiple data sources, parses and generates a unified query SQL to realize cross-multi-data source big data query. It shields the field and SQL differences of heterogeneous data sources, and at the same time can arrange the query nodes to generate query SQL to realize convenient data analysis query. It provides excellent expansion capabilities, reduces the complexity of the system, and realizes low-threshold massive data multi-data source data arrangement analysis query.

[0060] In a second embodiment, the present invention provides a specific method for querying data from multiple data sources, comprising the following steps:

[0061] Use Trino to connect to all data sources to obtain metadata for each data source. This metadata includes data for all databases, tables, and columns corresponding to each data source. Specifically, use a JDBC-CLIENT to connect to the Trino service. Call Trino to obtain data for all databases, tables, and columns corresponding to each data source, including the corresponding database, table, or column name, as well as the specific data. All data is then encapsulated in a unified format to obtain metadata for each data source. Calling Trino to obtain data for all databases, tables, and columns corresponding to each data source involves: calling Trino SHOW CATALOGS [LIKE pattern] to obtain data source data; calling Trino SHOW SCHEMAS [FROM catalog] [LIKE pattern] and SHOW CREATE SCHEMA schema_name to obtain database data; and calling Trino SHOW TABLES [FROM schema] [LIKE pattern] and SHOW SHOW COLUMNS FROM table [LIKE pattern] to obtain table data. This step unifies the different fields or data in heterogeneous data sources into the same metadata format, facilitating subsequent orchestration, configuration, and querying.

[0062] Receive user selections and configurations; Figure 2As shown, the system receives multiple data sources, libraries, tables, and columns selected by the user, and selects operations for each data source, library, table, and column. Operations include join queries, filtering, groupby aggregation, and sorting. Specific operations are selected based on the user's actual needs.

[0063] The data after the orchestration configuration is converted into a directed acyclic graph (DAG). Specifically, each operation in the orchestration configuration is regarded as a node, and an edge is set between two related operations to convert the data after the orchestration configuration into a directed acyclic graph. The data is stored in the nodes. Figure 3 As shown, a directed acyclic graph includes nodes and edges; the data structure of the graph is defined as: nodes (NODE) ​​and edges (EDGE);

[0064] A deep traversal algorithm is used to traverse the directed acyclic graph, and a stack and HashMap are used to store and parse the nodes and edges of the directed acyclic graph to generate a unified query SQL; specifically, Figure 4 As shown:

[0065] a: DAG graph traversal order is as follows Figure 4 The sequence number (downward arrow + digital number) is used. When a JOIN NODE is encountered, the current depth traversal is stopped, a mark is made, and the depth traversal of the next branch associated with the JOIN NODE is started.

[0066] EDGE also participates in traversal as a virtual node in the graph. There are three types of storage data structures: stack - stores the NODE nodes of the graph traversal, HashMap - stores the associations resolved by the EDGE node traversal, and list - temporarily stores the data popped from the NODE stack.

[0067] B: Graph traversal, ordinary NODE nodes are directly pushed into the stack in the traversal order, while the EDGE node parses the node ID of the FROM and END nodes and stores them as the key-value of the HashMap.

[0068] c: When encountering JOIN NODE (the traversal order is Figure 4 1, 2, 3 in the graph), stop traversing the graph and mark it. Start popping the NODE from the stack and putting it into the LIST until the stack is empty or a SUB SELECT subquery is encountered, then stop popping the NODE.

[0069] d: Traverse the LIST from back to front, parse and generate the corresponding select clause, such as: SELECT NODE is converted to SELECT columes FROM catalog.schema.table (columes and catalog.schema.table are attributes of SELECT NODE), and when a FILTER NODE is encountered, it is spliced ​​with WHERE columes = condition (columes operator condition is an attribute of FILTER NODE);

[0070] e: Recursively get the previous node of the current marked JOIN node from the EDGE node in the HashMap until there is no associated key, and the start node of the adjacent branch can be obtained. And start the depth traversal of the new branch until the traversal reaches the current marked JOIN node (the traversal order is Figure 4 4, 5, 6 in the sequence), repeat (c), (d), and (e) to generate the select clause of the new branch.

[0071] f: Process the JOIN NODE and connect the branch select clauses associated with the JOIN NODE with join (obtain the attributes of the JOIN NODE node as the connection direction and connection conditions).

[0072] Starting from the marked JOIN NODE, traverse the subsequent nodes (corresponding to Figure 4 If a new JOIN NODE is encountered (numbers 7 and 8), repeat steps (d), (e), and (f) until the entire graph is traversed and the complete query SQL is generated.

[0073] Send the unified query SQL to Trino to get the query results. Execute and query in Trino through JDBC-CLIENT, and return the results of the federated query.

[0074] Trino is a high-performance distributed computing engine for big data analysis based on SQL. It is fully based on interactive queries for in-memory analysis, abstracting SQL into operators to form a pipeline. Currently, it supports reading and querying data sources such as Hive, HBase, EleasticSearch, Kudu, Kafka, MySQL, and Redis.

[0075] The specific multi-data source data query method provided by the embodiment of the present invention uses Trino to connect all data sources to obtain the metadata of each data source to shield the field differences of heterogeneous data sources, and convert it into a directed acyclic graph according to the user's selection and arrangement configuration to generate a unified query SQL statement. There is no need to use different SQL statements according to different data sources. The application scheme uniformly arranges the metadata of heterogeneous data from multiple data sources, parses and generates a unified query SQL to realize cross-multi-data source big data query. It shields the field and SQL differences of heterogeneous data sources, and at the same time can arrange the query nodes to generate query SQL to realize convenient data analysis query. It provides excellent expansion capabilities, reduces the complexity of the system, and realizes low-threshold massive data multi-data source data arrangement analysis query.

[0076] In a third embodiment, the present invention provides a multi-data source data query device, such as Figure 5 Shown, including:

[0077] The metadata acquisition module 51 is used to connect to all data sources using Trino to obtain metadata for each data source. The metadata includes data for all databases, tables, and columns corresponding to each data source. Specifically, the metadata acquisition module 51 uses JDBC-CLIENT to connect to the Trino service. Trino is called to obtain data for all databases, tables, and columns corresponding to each data source. The data includes the corresponding database name, table name, or column name, as well as specific data. All data is then packaged in a unified format to obtain metadata for each data source. Calling Trino to obtain data for all databases, tables, and columns corresponding to each data source includes: calling Trino SHOW CATALOGS [LIKE pattern] to obtain data source data; calling Trino SHOW SCHEMAS [FROM catalog] [LIKE pattern] and SHOW CREATESCHEMA schema_name to obtain database data; and calling Trino SHOW TABLES [FROM schema] [LIKE pattern] and SHOW SHOW COLUMNS FROM table [LIKE pattern] to obtain table data.

[0078] As an optional implementation method of an embodiment of the present invention, multiple data sources include: Hive, HBase, EleasticSearch, Kudu, Kafka, MySQL, and Redis.

[0079] The orchestration configuration module 52 is used to receive user selections and orchestration configurations. Specifically, the orchestration configuration module 52 receives multiple data sources, libraries, tables, and columns selected by the user, as well as operation selections for each data source, library, table, and column. The operation selections include associated queries, filtering, group by aggregation, and sorting.

[0080] The data conversion module 53 is used to convert the orchestrated data into a directed acyclic graph, which includes nodes and edges. Specifically, the data conversion module 53 regards each operation in the orchestration configuration as a node and sets an edge between two related operations to convert the orchestrated data into a directed acyclic graph, and the data is stored in the nodes.

[0081] The SQL generation module 54 is used to traverse the directed acyclic graph using a deep traversal algorithm, and to store and parse the nodes and edges of the directed acyclic graph using a stack and a HashMap to generate a unified query SQL. Specifically, the SQL generation module 54 generates a stack for storing the nodes of the graph traversal, a HashMap for storing the association relationships parsed by the edge traversal, and a LIST for temporarily storing the data popped from the node according to the directed acyclic graph. When a JOIN NODE is encountered, the graph traversal is stopped and marked, and the NODE is popped from the stack and placed into the LIST until the stack is empty or a SUB SELECT subquery is encountered, in which case the popping is stopped. The LIST is traversed from back to front, and the select clause corresponding to a branch is generated by parsing. The previous node of the marked JOIN node is recursively obtained from the HashMap until the unrelated key is found, and the start node of the adjacent branch is obtained. The depth traversal of the new branch is started until the currently marked JOIN node is traversed, and the select clause of the new branch is generated. All branch select clauses associated with a NODE are connected using join; starting from the marked JOIN NODE, subsequent nodes are deeply traversed until the entire graph is traversed, generating a complete unified query SQL.

[0082] Among them, the nodes are pushed into the stack in the traversal order, and the nodeId of the FROM and END nodes of the edge parsing are stored as the key-value of the HashMap.

[0083] The result query module 55 is used to send the unified query SQL to Trino to obtain the query result.

[0084] The multi-data source query device provided by the present invention includes a metadata acquisition module that uses Trino to connect all data sources to obtain metadata for each data source; an orchestration configuration module that receives user selections and orchestration configurations; a data conversion module that converts the orchestrated data into a directed acyclic graph (DAG), which includes nodes and edges; an SQL generation module that uses a deep traversal algorithm to traverse the DAG, and simultaneously uses a stack and a HashMap to store and parse the nodes and edges of the DAG to generate a unified query SQL; and a result query module that sends the unified query SQL to Trino to obtain the query results. The multi-data source query device provided by the present invention uses Trino to connect all data sources to obtain metadata for each data source, shielding field differences between heterogeneous data sources. Based on the user's selections and orchestration configurations, the device converts the metadata into a DAG to generate a unified query SQL statement, eliminating the need to use different SQL statements for different data sources. This solution uniformly orchestrates metadata for heterogeneous data from multiple data sources, parses, and generates a unified query SQL statement to enable cross-data-source big data queries. This shields field and SQL differences between heterogeneous data sources, while orchestrating query nodes to generate query SQL for convenient data analysis and querying. It provides excellent expansion capabilities, reduces system complexity, and enables low-threshold data orchestration, analysis, and query of massive data from multiple data sources.

[0085] In a fourth embodiment, the present invention provides a multi-data source data query system, comprising:

[0086] processor;

[0087] a memory for storing processor-executable instructions;

[0088] The processor is configured to be used for the multi-data source data query method provided by the first embodiment or the second embodiment.

[0089] The multi-data source data query system provided by the embodiment of the present invention stores the executable instructions of the processor in the memory. When the executable instructions are executed, the processor can use Trino to connect to all data sources to obtain the metadata of each data source to shield the field differences of heterogeneous data sources. According to the user's selection and arrangement configuration, it is converted into a directed acyclic graph to generate a unified query SQL statement. There is no need to use different SQL statements according to different data sources. The present application scheme uniformly arranges the metadata of heterogeneous data from multiple data sources, parses and generates a unified query SQL to realize cross-multi-data source big data query. It shields the field and SQL differences of heterogeneous data sources, and at the same time can arrange the query nodes to generate query SQL to realize convenient data analysis query. It provides excellent expansion capabilities, reduces the complexity of the system, and realizes low-threshold massive data multi-data source data arrangement analysis query.

[0090] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0091] It should be noted that, in the description of this application, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0092] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0093] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0094] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0095] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0096] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0097] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0098] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-data source data query method, characterized in that: The following steps are involved: Use Trino to connect all data sources to obtain metadata for each data source, including data for all databases, tables, and columns corresponding to each data source. receiving user selections and choreography configurations; Converting the orchestrated data into a directed acyclic graph comprising nodes and edges; The converting the orchestrated data into a directed acyclic graph includes: taking each operation in the orchestration configuration as a node, setting an edge between two associated operations, so as to convert the orchestrated data into a directed acyclic graph, wherein the data is stored in the nodes; The directed acyclic graph is traversed using a depth traversal algorithm, and the nodes and edges of the directed acyclic graph are stored and parsed using a stack and a HashMap to generate a unified query SQL; the directed acyclic graph is traversed using a depth traversal algorithm, and the nodes and edges of the directed acyclic graph are stored and parsed using a stack and a HashMap to generate a unified query SQL, including: generating a stack for storing nodes for graph traversal, a HashMap for storing association relationships parsed by edge traversal, and a LIST for temporarily storing data popped from the node according to the directed acyclic graph; when a JOIN NODE is encountered, the graph traversal is stopped, and a mark is added, the NODE is popped from the stack and placed into the LIST until the stack is empty or a SUB SELECT subquery is encountered, then the popping is stopped; the LIST is traversed from back to front, and the parsed select clause corresponding to a branch is generated; and the marked JOIN is recursively obtained from the HashMap. The node's previous node is searched until the unrelated key is reached, and the starting node of the adjacent branch is obtained. A deep traversal of the new branch is started until the currently marked JOIN node is reached, generating the select clause for the new branch. All branch select clauses associated with the JOIN NODE are connected using join. Starting from the marked JOIN NODE, the subsequent nodes are deeply traversed until the entire graph is traversed, generating a complete unified query SQL. The unified query SQL is sent to the Trino to obtain the query result.

2. The method according to claim 1, wherein: Trino is used to connect all data sources to obtain metadata for each data source, including: Use JDBC-CLIENT to connect to the Trino service; Calling the Trino to obtain data of all databases, tables, and columns corresponding to each data source, the data including the corresponding database name, table name, or column name, and specific data; All data are encapsulated in a unified format to obtain the metadata of each data source.

3. The method according to claim 2, wherein: The calling of Trino to obtain data of all databases, tables, and columns corresponding to each data source includes: Call Trino SHOW CATALOGS [ LIKE pattern ] to obtain data from the data source; Call Trino SHOW SCHEMAS [ FROM catalog ] [ LIKE pattern ] and SHOW CREATESCHEMA schema_name to obtain database data; Call Trino SHOW TABLES [ FROM schema ] [ LIKE pattern ] and SHOW SHOWCOLUMNS FROM table [ LIKE pattern ] to obtain table data.

4. The method according to claim 1, wherein: The receiving user's selection and arrangement configuration includes: Receive multiple data sources, libraries, tables, and columns selected by the user; and select operations for each data source, library, table, and column, including associated query, filtering, group by aggregation, and sorting.

5. The method according to claim 1, wherein: The step of generating a stack for storing nodes of the graph traversal and a HashMap for storing association relationships resolved by the edge traversal according to the directed acyclic graph includes: Push the nodes into the stack in the traversal order, and store the nodeId of the FROM and END nodes of the edge resolution as the key-value of the HashMap.

6. The method according to claim 1, wherein: The multiple data sources include: Hive, HBase, EleasticSearch, Kudu, Kafka, MySQL, and Redis.

7. A multi-data source data query device, characterized in that: The multi-data source data query method according to any one of claims 1 to 6, wherein the multi-data source data query device comprises: A metadata acquisition module is used to connect all data sources using Trino to obtain metadata for each data source, including data from all databases, tables, and columns corresponding to each data source. The orchestration configuration module is used to receive user selections and orchestrate configurations; A data conversion module, configured to convert the orchestrated data into a directed acyclic graph comprising nodes and edges; An SQL generation module is used to traverse the directed acyclic graph using a depth traversal algorithm, and to store and parse the nodes and edges of the directed acyclic graph using a stack and a HashMap to generate a unified query SQL; The result query module is used to send the unified query SQL to the Trino to obtain the query result.

8. A multi-data source data query system, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the method according to any one of claims 1 to 6.

Images