Multi-source query method, electronic equipment and storage medium

Abstract

The invention discloses a multi-source query method, electronic equipment and a storage medium. The method comprises the steps that a multi-source query engine obtains a query statement input by a target; analyzing whether the query statement contains dimension table elements or not; in response to dimension table elements contained in the query statement, creating a target dimension table corresponding to the dimension table elements; and performing a multi-source query task based on the target dimension table. According to the scheme, the cross-source query efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data distributed computing technology, and in particular to a multi-source query method, electronic equipment, and storage medium. Background Art

[0002] In the prior art, in order to solve the problem that different data sources cannot access each other when performing cross-source queries, Flink is usually used as a general distributed computing engine to perform cross-source queries on various data sources. However, this method requires the target to first develop a CREATE statement to create a Flink dimension table before directly querying the extended system in the Flink SQL engine. When developing the CREATE statement, the target needs to pay attention to the mapping relationship between the extended system and the Flink data type, and needs to ensure the accuracy of the mapping relationship at all times. This process is extremely time-consuming and complicated. Summary of the invention

[0003] The present application at least provides a multi-source query method, electronic device, and storage medium, which can improve the efficiency of cross-source queries.

[0004] In a first aspect, the present application provides a multi-source query method, which includes: a multi-source query engine obtains a query statement of a target input; parses the query statement to see whether it contains a dimension table element; in response to the query statement containing the dimension table element, creates a target dimension table corresponding to the dimension table element; and performs a multi-source query task based on the target dimension table.

[0005] The creation of a target dimension table corresponding to the dimension table element includes: obtaining table structure information corresponding to the dimension table element from a unified metadata service; generating a table creation statement based on the table structure information; and generating a target dimension table based on the table creation statement.

[0006] Wherein, based on the table structure information, a table creation statement is generated, including: converting the original field type in the table structure information into a data type supported by the multi-source query engine; assembling the converted table structure information to obtain a table creation statement; and / or, after generating a target dimension table based on the table creation statement, including: storing the table structure information in a directory of the multi-source query engine.

[0007] Before creating the target dimension table corresponding to the dimension table element, the method further includes: based on the dimension table element, determining whether the multi-source query engine has cached the target database where the target dimension table is located; in response to the multi-source query engine not caching the target database, creating the target database; in response to the target database being created or the multi-source query engine having cached the target database, determining whether the multi-source query engine has cached the table structure information corresponding to the dimension table element; in response to the multi-source query engine not caching the table structure information corresponding to the dimension table element, executing creation of the target dimension table corresponding to the dimension table element.

[0008] The creating of the target database includes: acquiring database information from a unified metadata service based on an identifier of the target database; and creating the target database based on the database information.

[0009] Among them, in response to the query statement containing a dimension table element, before judging whether the multi-source query engine has cached the target database where the target dimension table is located based on the dimension table element, it also includes: judging whether the multi-source query engine has cached the target data directory where the target dimension table is located based on the dimension table element; in response to the multi-source query engine not caching the target data directory, creating the target data directory; in response to the target data directory being created or the multi-source query engine having cached the target data directory, executing based on the dimension table element to judge whether the multi-source query engine has cached the target database where the target dimension table is located.

[0010] The step of creating a target data directory includes: obtaining data directory information from a unified metadata service based on an identifier of the target data directory; and creating the target data directory based on the data directory information.

[0011] Among them, parsing whether the query statement contains dimension table elements includes: parsing the query statement to obtain a syntax tree; judging whether each node in the syntax tree is a dimension table node; in response to the node being a dimension table node, determining that the query statement contains dimension table elements, and the dimension table node is used as a dimension table element; and / or, performing a multi-source query task based on a target dimension table, including: generating at least one connector description information based on the type of the target dimension table; creating a task execution graph corresponding to the query statement based on the connector description information; and / or, the multi-source query engine is a Flink SQL engine, and the query statement is an SQL query statement.

[0012] A second aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the multi-source query method in the first aspect.

[0013] A third aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the multi-source query method in the first aspect is implemented.

[0014] In the above scheme, after obtaining the query statement of the target input, the multi-source query engine parses the query statement to determine whether the query statement contains dimension table elements. If the query statement contains dimension table elements, a target dimension table corresponding to the dimension table element is automatically created, and a multi-source query task is performed based on the target dimension table. Compared with the existing technology, there is no need for the target itself to create dimension table data, thereby simplifying the multi-source query task steps to improve the efficiency of the query task.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0017] Figure 1 It is a schematic diagram of the framework of an embodiment of the automatic dimension table generation system of the present application; Figure 2 It is a flowchart of an embodiment of the multi-source query method of the present application; Figure 3 It is a flowchart of another embodiment of the multi-source query method of the present application; Figure 4 It is a flowchart of an embodiment of a multi-source query device of the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0018] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0019] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0020] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0021] In existing cross-source query tasks, the target will face a variety of data structures, including structured data (relational database data), semi-structured data (CSV (Comma-Separated Values, character-separated values), XML (Extensible Markup Language, Extensible Markup Language), JSON (JavaScript Object Notation, JavaScript Object Notation), etc.), unstructured data (email, documents, PDF (Portable Document Format, portable file format)) and binary data (images, audio, video). At the same time, these data are stored in different data sources, such as traditional relational databases (MySQL, Oracle, PostgreSQL, etc.), NoSQL databases (HBase, Kudu, Redis, MongoDB, Neo4j, etc.), and file systems (local file systems, HDFS (Hadoop Distributed File System, distributed file systems), etc.). The existence of multi-source heterogeneous data causes the following difficulties for unified data analysis: 1. High learning cost. To perform cross-source query tasks on multiple data sources, the target needs to be familiar with the query primitives of different data sources. If the storage source does not support query primitives, such as the file system, the target also needs to implement the query through programming, which is difficult to use and leads to high learning cost. 2. Different data sources cannot access each other. When performing query analysis, the target needs to frequently switch between different platforms, which is extremely cumbersome. 3. Association analysis is difficult and usually requires programming to complete in cross-source queries.

[0022] To address the above issues, the commonly used solution is to use Flink as a general distributed computing engine to perform cross-source query tasks on various data sources. The overall idea of ​​this solution is: Step 1: Create the dimension table of the input data source from the target: CREATE TABLE source_table(id INT, name STRING) WHITH ('connector'='hive',...); Step 2: Create the dimension table of the output data source from the target: CREATE TABLE sink_table(id INT, name STRING) WHITH ('connector'='jdbc', 'username'='user', 'password'='123456', ...); Step 3: The target writes business SQL (Structured Query Language): INSERT INTO sink_table SELECT * FROM source_table.

[0023] In this process, when developing CREATE statements, you need to pay attention to the mapping relationship between the extension system and Flink data types and ensure the accuracy of the mapping relationship. This process is extremely time-consuming and complicated.

[0024] The present application provides a dimension table automatic generation system, which solves the problem that the process is cumbersome and error-prone when creating Flink dimension tables. At the same time, the system obtains metadata information from a unified metadata service to avoid the security problem caused by exposing the password on the login page when the target explicitly creates the dimension table.

[0025] Specifically, see Figure 1 , Figure 1 It is a schematic diagram of the framework of an embodiment of the dimension table automatic generation system 100 of the present application. The dimension table automatic generation system 100 includes a Catalog (catalog) automatic creation unit 110 and a ClsGenericInMemoryCatalog unit 120. After receiving the data catalog information transmitted by the parser of the multi-source query engine 200, the Catalog automatic creation unit 110 interacts with the unified metadata service 210 in real time, and automatically creates a Catalog of type ClsGenericInMemoryCatalog according to the interaction result. The ClsGenericInMemoryCatalog unit 120 is a self-developed Catalog (of type ClsGenericInMemoryCatalog, inheriting GenericInMemoryCatalog), which is responsible for obtaining database information and table structure information from the unified metadata service, wherein GenericInMemoryCatalog is a Catalog implemented based on memory, and all metadata are only available within the life cycle of the session (i.e., within the life cycle of a Flink task once). The dimension table automatic generation system 100 is located in a multi-source query engine 200 . The multi-source query engine 200 further includes a unified metadata service 210 and a connector 220 .

[0026] The ClsGenericInMemoryCatalog unit 120 also includes a database automatic creation unit 121 and a dimension table automatic creation unit 122. The database automatic creation unit 121 is a subunit of the ClsGenericInMemoryCatalog unit 120, and is used to process the automatic creation of the target database. Its working principle is that after receiving the identifier of the target database transmitted by the multi-source query engine 200, the ClsGenericInMemoryCatalog unit 120 interacts with the unified metadata service 210 in real time, and automatically creates the target database according to the interaction result. The dimension table automatic creation unit 122 is a subunit of the ClsGenericInMemoryCatalog unit 120, and is used to process the automatic creation of the target dimension table. Its working principle is that after receiving the identifier of the target database transmitted by the multi-source query engine 200, the dimension table automatic creation unit 122 interacts with the unified metadata service 210 in real time, and automatically creates the target dimension table according to the interaction result.

[0027] The dimension table automatic creation unit 122 includes a table structure acquisition unit 1221, a field type conversion unit 1222, and a dimension table creation statement assembly unit 1223. The table structure acquisition unit 1221 is a subunit of the dimension table automatic creation unit 122, and is used to acquire table structure information from the unified metadata service 210. The field type conversion unit 1222 is a subunit of the dimension table automatic creation unit 122, and is used to convert the original field type in the table structure information acquired by the table structure acquisition unit 1221 into a data type supported by the multi-source query engine 200. The dimension table creation statement assembly unit 1223 is a subunit of the dimension table automatic creation unit 122, and is used to assemble the table structure information converted by the field type conversion unit 1222 into a table creation statement supported by the multi-source query engine 200.

[0028] After the multi-source query engine 200 is integrated with the dimension table automatic generation system 100, a target dimension table can be established according to the query statement input by the target, and connector description information can be generated according to the type of the established target dimension table, and a task execution graph can be generated using the connector description information. The task execution graph is connected to the corresponding connector 220 through the connector description information, and the external resource library is connected through the connector 220 to obtain external resources. The multi-source query engine 200 combines the target dimension table and the external resources to execute the corresponding task according to the task execution graph.

[0029] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of the multi-source query method of the present application. Specifically, it may include the following steps: Step S210: the multi-source query engine obtains the query statement input by the target.

[0030] In some embodiments, after receiving the query statement of the target input, the multi-source query engine 200 may send the query statement to the dimension table automatic generation system 100, and the dimension table automatic generation system 100 processes the query statement. Prior to this, the dimension table automatic generation system 100 is not enabled by default, and is enabled only when the cls.url parameter (the parameter is used to specify the location of the unified metadata service 210) is set in Table Config, that is, when the dimension table automatic generation system 100 is initialized. This configurable method can be more compatible with open source systems. The code example for enabling the dimension table automatic generation system 100 is as follows: StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings); tableEnv.getConfig().getConfiguration().setString("cls.url", "http: / / localhost:7890").

[0031] It is understandable that the code for starting the dimension table automatic generation system 100 can be modified according to actual usage, and no specific limitation is made here.

[0032] In some embodiments, before the multi-source query engine 200 is enabled, a transmission protocol may be set between the multi-source query engine 200 and the dimension table automatic generation system 100. For example, protocols such as TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) may be set, which are not specifically limited here.

[0033] In some embodiments, the multi-source query engine 200 may be a Flink SQL engine, and the query statement may be an SQL query statement. In addition, the multi-source query engine 200 may also be an APACHE HIVE engine, and the corresponding query statement may be an SQL-like query statement. Therefore, no specific limitation is made here regarding the multi-source query engine 200 and the query statement.

[0034] Step S220: parse the query statement to determine whether it contains dimension table elements.

[0035] In some embodiments, each field in the query statement is detected to detect whether the field in the query statement is a table field. For example, when the query statement is an SQL query statement, the SQL query statement input by the target is: INSERT INTO sink_table SELECT * FROM source_table, and the multi-source query engine 200 detects the SQL query statement and obtains that "sink_table" and "source_table" in the SQL query statement are table fields. Therefore, the detection result obtained is that the query statement contains a dimension table element.

[0036] In other embodiments, the parser of the multi-source query engine 200 may be used to parse the query statement to obtain a syntax tree, and the multi-source query engine 200 may be used to determine whether each node in the syntax tree is a dimension table node. For example, the syntax tree may be traversed to analyze each node in the syntax tree to determine whether the node is a dimension table node. In response to the node being a dimension table node, it is determined that the query statement contains a dimension table element, and the dimension table node is used as a dimension table element. If the entire syntax tree is traversed and it is found that the syntax tree does not contain a dimension table node, the entire process ends.

[0037] Step S230: in response to the query statement containing a dimension table element, creating a target dimension table corresponding to the dimension table element.

[0038] In some embodiments, after determining that the query statement contains dimension table elements, the multi-source query engine 200 can use DDL (Data Definition Language) statements to define the target dimension table, wherein the DDL statement is used to describe the structure, data type, partition, attribute, etc. of the dimension table. After that, the connector of the target dimension table is configured. Specifically, the built-in JDBC (Java Database Connectivity) connector of the multi-source query engine 200 can be used to connect to the MySQL database. This connector allows the multi-source query engine 200 to interact with MySQL through the JDBC interface, thereby realizing data reading and writing. After defining the target dimension table and configuring the connector, the target dimension table can be used to perform JOIN operations in the tasks executed by the multi-source query engine 200 (JOIN is an operation in Flink used to associate stream data with static data).

[0039] In other embodiments, when the multi-source query engine 200 is a Flink SQL engine, in the Flink data directory system, Catalog, Database (database) and Table constitute the basic architecture of Flink data management. Therefore, when creating a target dimension table in Flink, it is necessary to create a data management architecture of Catalog, Database (database) and Table in the Flink data directory system. For details, please refer to steps S231 to S236.

[0040] Step S231: Based on the dimension table elements, determine whether the multi-source query engine has cached the target data directory where the target dimension table is located.

[0041] In some embodiments, the multi-source query engine 200 may be detected, and a first determination result may be obtained as to whether a target data directory corresponding to a target table corresponding to a dimension table element is stored in a storage space of the multi-source query engine 200 .

[0042] Step S232: In response to the multi-source query engine not caching the target data directory, creating the target data directory.

[0043] In some embodiments, when the first judgment result obtained is that the multi-source query engine 200 has not cached the target data directory, the data directory information can be obtained from the unified metadata service 210 based on the identifier of the target data directory. Then, the target data directory is created based on the data directory information. Specifically, the identifier of the target data directory can be the name of the target data directory. After the Catalog automatic creation unit 110 in the dimension table automatic generation system 100 receives the name of the target data directory, it interacts with the unified metadata service 210, obtains the data directory information using the unified metadata service 210, and then creates the target data directory based on the obtained data directory information. It can be understood that the identifier of the target data directory can also be a unique identifier, etc., which is not specifically limited here.

[0044] In some other embodiments, when the first judgment result is that the target data directory has been created or the multi-source query engine 200 has cached the target data directory, step S233 is executed.

[0045] Step S233: Based on the dimension table elements, determine whether the multi-source query engine has cached the target database where the target dimension table is located.

[0046] In some embodiments, after the target data directory is created or the multi-source query engine 200 has cached the target data directory, it may be further determined whether the multi-source query engine 200 has cached the target database where the target dimension table is located to obtain a second determination result.

[0047] Step S234: In response to the multi-source query engine not caching the target database, creating a target database.

[0048] In some embodiments, when the second judgment result is that the multi-source query engine 200 does not cache the target database, a new target database needs to be created. Based on the identifier of the target database, the database information can be obtained from the unified metadata service 210, and the target database is created based on the database information. Specifically, the identifier of the target database can be the name of the target database, and the target database is created using the database automatic creation unit 121 in the dimension table automatic generation system 100. First, after receiving the name of the target database, the database automatic creation unit 121 interacts with the unified metadata service 210 to obtain the database information, and then creates the target database based on the database information.

[0049] Step S235: In response to the target database being created or the multi-source query engine having cached the target database, it is determined whether the multi-source query engine has cached the table structure information corresponding to the dimension table element.

[0050] In some embodiments, if the second judgment result is that the target database has been created or the multi-source query engine 200 has cached the target database, it is possible to further judge whether the multi-source query engine 200 has cached the table structure information corresponding to the dimension table element to obtain a third judgment result.

[0051] Step S236: in response to the multi-source query engine not caching the table structure information corresponding to the dimension table element, executing creation of a target dimension table corresponding to the dimension table element.

[0052] In some embodiments, if the third judgment result obtained indicates that the multi-source query engine does not cache the table structure information corresponding to the dimension table element, a target dimension table corresponding to the dimension table element is created. Specifically, refer to steps S2361 to S2363.

[0053] Step S2361: Acquire table structure information corresponding to dimension table elements from the unified metadata service.

[0054] In some embodiments, the table structure information can be obtained by using the table structure acquisition unit 1221 in the dimension table automatic creation unit 122. Specifically, the table structure acquisition unit 1221 interacts with the unified metadata service 210 according to the identifier of the target database parsed by the multi-source query engine 200, and determines the table structure information corresponding to the dimension table element through the identifier of the target database.

[0055] Step S2362: Generate a table creation statement based on the table structure information.

[0056] In some embodiments, a creation module in the multi-source query engine 200 may be used to generate a table creation statement according to the table structure information.

[0057] In other embodiments, the original field type in the table structure information can be converted into a data type supported by the multi-source query engine; the converted table structure information is assembled to obtain a table creation statement. Specifically, the field type conversion unit 1222 in the dimension table automatic creation unit 122 can be used to convert the original field type in the table structure information, thereby obtaining a data type supported by the multi-source query engine. Then, the dimension table creation statement assembly unit 1223 in the dimension table automatic creation unit 122 is used to assemble the table structure information converted by the field type conversion unit 1222 into a table creation statement supported by the multi-source query engine 200.

[0058] Step S2363: Generate a target dimension table based on the table creation statement.

[0059] In some embodiments, the dimension table automatic generation system 100 combines the above target data catalog, target database and table creation statement to generate the data management architecture of Catalog, Database and Table, and the corresponding target dimension table will also be generated. To facilitate subsequent operations on the target dimension table, the table structure information can be stored in the directory of the multi-source query engine 200.

[0060] Step S240: performing a multi-source query task based on the target dimension table.

[0061] In some embodiments, after the target dimension table is created, at least one connector description information may be generated based on the type of the target dimension table, and then a task execution graph corresponding to the query statement may be created based on the connector description information. A corresponding connector 220 is generated through the connector description information in the task execution graph, and an external resource library is connected through the connector 220 to obtain external resources, and the corresponding task is executed according to the task execution graph in combination with the target dimension table and the external resources.

[0062] In a specific implementation scenario, the multi-source query engine 200 is a Flink SQL engine, and when the query statement is an SQL query statement, after the target enters INSERT INTO sink_table SELECT * FROMsource_table in the multi-source query engine 200, the multi-source query engine 200 starts to automatically create the target dimension table. Figure 3 The steps of automatically creating the target dimension table performed by the multi-source query engine 200 are as follows: 1. The Flink SQL engine parses the SQL query statement and generates a SQL syntax tree; 2. Traverse each node in the SQL syntax tree, analyze each node in the syntax tree, and determine whether the node is a dimension table node in the SQL syntax tree. If the node is a dimension table node, determine that the query statement contains a dimension table element, and the dimension table node is used as a dimension table element, and execute the following 3, 4, and 5 in sequence; if not, continue to traverse each node in the syntax tree; 3. Determine whether the Flink SQL engine has cached the target data directory where the target dimension table is located. If so, proceed to the following 4. If the Flink SQL engine has not cached the target data directory, create the target data directory. Specifically, after the Catalog automatic creation unit 110 in the dimension table automatic generation system 100 receives the name of the target data directory, it interacts with the unified metadata service 210, uses the unified metadata service 210 to obtain data directory information, and then automatically creates a target data directory of type ClsGenericInMemoryCatalog according to the obtained data directory information; 4. Determine whether the Flink SQL engine has cached the target database where the target dimension table is located. If so, proceed to the following 5. If not, proceed to the database automatic creation unit 121. After receiving the name of the target database, the database automatic creation unit 121 interacts with the unified metadata service 210 to obtain database information, and then creates the target database according to the database information; 5. Determine whether the Flink SQL engine has cached the table structure information corresponding to the dimension table elements. If so, return to the above 2 to continue traversing the SQL syntax tree; if not, enter the dimension table automatic creation unit 122. In the dimension table automatic creation unit 122, the target dimension table is automatically created through the following three steps: a) The table structure acquisition unit 1221 acquires the table structure information from the unified metadata service 210. If the table structure information is acquired, it is handed over to the field type conversion unit 1222. Otherwise, the dimension table automatic creation unit 122 exits; b) The field type conversion unit 1222 converts the original field type corresponding to the table structure information into a data type supported by the Flink SQL engine; c) The dimension table creation statement assembling unit 1223 automatically assembles SQL table creation statements according to the data types supported by the Flink SQL engine obtained in the first two steps, and stores the table structure information in the directory of the multi-source query engine 200 to complete the creation of the target dimension table of the dimension table node, and then returns to step 2; 6. After traversing all nodes in the SQL syntax tree, the Flink SQL engine generates connector description information based on the type of the established target dimension table; 7. The Flink SQL engine uses the connector description information to generate a task execution graph until the end.

[0063] Compared with the existing technology, this application only needs to submit a single statement to complete business development, which greatly improves development efficiency. And compared with the open source solution of the existing technology, the target password is exposed when the CREATE statement is directly entered on the login page. This application solves the problem of connector password exposure by canceling the explicit development dimension table creation statement.

[0064] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0065] See also Figure 4 , Figure 4 It is a flowchart of an embodiment of a multi-source query device 400 of the present application. The multi-source query device 400 includes an acquisition module 410, a parsing module 420, a creation module 430 and a task execution module 440. The acquisition module 410 executes a multi-source query engine to acquire a query statement of a target input. The parsing module 420 performs parsing to determine whether the query statement contains a dimension table element. The creation module 430 executes a response to the query statement containing a dimension table element to create a target dimension table corresponding to the dimension table element. The task execution module 440 performs a multi-source query task based on the target dimension table.

[0066] In some embodiments, the creation module 430 creates a target dimension table corresponding to the dimension table element, including: obtaining table structure information corresponding to the dimension table element from a unified metadata service; generating a table creation statement based on the table structure information; and generating a target dimension table based on the table creation statement.

[0067] In some embodiments, the creation module 430 generates a table creation statement based on the table structure information, including: converting the original field type in the table structure information into a data type supported by the multi-source query engine; assembling the converted table structure information to obtain a table creation statement; and / or, after generating a target dimension table based on the table creation statement, including: storing the table structure information in a directory of the multi-source query engine.

[0068] In some embodiments, before the creation module 430 creates the target dimension table corresponding to the dimension table element, it also includes: based on the dimension table element, determining whether the multi-source query engine has cached the target database where the target dimension table is located; in response to the multi-source query engine not caching the target database, creating the target database; in response to the target database being created or the multi-source query engine having cached the target database, determining whether the multi-source query engine has cached the table structure information corresponding to the dimension table element; in response to the multi-source query engine not caching the table structure information corresponding to the dimension table element, executing the creation of the target dimension table corresponding to the dimension table element.

[0069] In some embodiments, the creation module 430 executes creation of a target database, including: acquiring database information from a unified metadata service based on an identifier of the target database; and creating the target database based on the database information.

[0070] In some embodiments, the creation module 430 executes in response to a query statement containing a dimension table element, and before determining whether the multi-source query engine has cached the target database where the target dimension table is located based on the dimension table element, it also includes: determining whether the multi-source query engine has cached the target data directory where the target dimension table is located based on the dimension table element; creating the target data directory in response to the multi-source query engine not caching the target data directory; and determining whether the multi-source query engine has cached the target database where the target dimension table is located based on the dimension table element in response to the target data directory having been created or the multi-source query engine having cached the target data directory.

[0071] In some embodiments, the creation module 430 executes creation of a target data directory, including: acquiring data directory information from a unified metadata service based on an identifier of the target data directory; and creating the target data directory based on the data directory information.

[0072] In some embodiments, the parsing module 420 performs parsing of whether the query statement contains dimension table elements, including: parsing the query statement to obtain a syntax tree; determining whether each node in the syntax tree is a dimension table node; in response to the node being a dimension table node, determining that the query statement contains a dimension table element, and the dimension table node serves as a dimension table element.

[0073] In some embodiments, the task execution module 440 executes a multi-source query task based on a target dimension table, including: generating at least one connector description information based on the type of the target dimension table; and creating a task execution graph corresponding to the query statement based on the connector description information.

[0074] See also Figure 5 , Figure 51 is a schematic diagram of a framework of an embodiment of an electronic device 50 of the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above multi-source query method embodiments. In a specific implementation scenario, the electronic device 50 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0075] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps in any of the above-mentioned multi-source query method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0076] See also Figure 6 , Figure 6 60 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor, and the program instructions 601 are used to implement the steps in any of the above-mentioned multi-source query method embodiments.

[0077] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0078] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0079] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0080] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A multi-source query method, characterized in that: include: The multi-source query engine obtains the query statement input by the target; Analyze whether the query statement contains dimension table elements; In response to the query statement containing the dimension table element, creating a target dimension table corresponding to the dimension table element; A multi-source query task is performed based on the target dimension table.

2. The method according to claim 1, characterized in that: The step of creating a target dimension table corresponding to the dimension table element includes: Acquire table structure information corresponding to the dimension table element from a unified metadata service; Generate a table creation statement based on the table structure information; Based on the table creation statement, the target dimension table is generated.

3. The method according to claim 2, characterized in that The generating a table creation statement based on the table structure information includes: Converting the original field type in the table structure information into a data type supported by the multi-source query engine; Assembling the converted table structure information to obtain the table creation statement; And / or, after generating the target dimension table based on the table creation statement, comprising: The table structure information is stored in a directory of the multi-source query engine.

4. The method according to claim 1, characterized in that: Before creating the target dimension table corresponding to the dimension table element, the method further includes: Based on the dimension table element, determining whether the multi-source query engine has cached the target database where the target dimension table is located; In response to the multi-source query engine not caching the target database, creating the target database; In response to the target database being created or the multi-source query engine having cached the target database, determining whether the multi-source query engine has cached the table structure information corresponding to the dimension table element; In response to the multi-source query engine not caching the table structure information corresponding to the dimension table element, the creating of the target dimension table corresponding to the dimension table element is performed.

5. The method according to claim 4, characterized in that The creating of the target database comprises: Based on the identifier of the target database, obtaining database information from a unified metadata service; The target database is created based on the database information.

6. The method according to claim 4, characterized in that In response to the query statement containing the dimension table element, before judging whether the multi-source query engine has cached the target database where the target dimension table is located based on the dimension table element, the method further includes: Based on the dimension table element, determining whether the multi-source query engine has cached the target data directory where the target dimension table is located; In response to the multi-source query engine not caching the target data directory, creating the target data directory; In response to the target data directory being created or the multi-source query engine having cached the target data directory, the step of determining whether the multi-source query engine has cached the target database where the target dimension table is located based on the dimension table element is performed.

7. The method according to claim 6, characterized in that The step of creating the target data directory comprises: Based on the identifier of the target data directory, obtaining data directory information from a unified metadata service; The target data directory is created based on the data directory information.

8. The method according to claim 1, characterized in that The analyzing whether the query statement contains dimension table elements includes: Parsing the query statement to obtain a syntax tree; Determine whether each node in the syntax tree is a dimension table node; In response to the node being a dimension table node, determining that the query statement includes a dimension table element, and the dimension table node serves as the dimension table element; And / or, performing a multi-source query task based on the target dimension table includes: generating at least one connector description information based on the type of the target dimension table; Based on the connector description information, create a task execution graph corresponding to the query statement; And / or, the multi-source query engine is a Flink SQL engine, and the query statement is an SQL query statement.

9. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the multi-source query method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the multi-source query method according to any one of claims 1 to 8 is implemented.

Images