Data query method and device, server and storage medium

Abstract

This application provides a data query method, apparatus, server, and storage medium, relating to the field of data query technology. The data query method includes: obtaining a target query statement; if the target query statement is a joint query statement, obtaining metadata of the data table to be queried based on the target query statement; decomposing the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried; and querying a first cluster and a second cluster respectively based on the first type of query statement and the second type of query statement to obtain query results, wherein the first cluster and the second cluster are data clusters of different types. By decomposing the target query statement into a first type of query statement and a second type of query statement for querying in the first and second clusters respectively, users only need to use the target query statement once to perform a dual-source joint query in the first and second clusters, improving the user experience.

Description

Technical Field

[0001] This application relates to the field of data query technology, and more specifically, to a data query method, apparatus, server, and storage medium. Background Technology

[0002] With the explosive growth of data volume, data analytics engines have become a core tool for enterprise decision-making. The use of data analytics engines for data querying is also becoming increasingly common.

[0003] In related technologies, queries are performed within a single database based on query statements, and different databases have different advantages. Some databases can achieve fast queries but perform poorly when handling complex query operations, while others can handle complex query operations but have lower query efficiency.

[0004] In related technologies, when faced with complex query statements, users need to query different databases separately, which reduces the user experience. Summary of the Invention

[0005] The purpose of this application is to provide a data query method, apparatus, server, and storage medium to address the shortcomings of the prior art and solve the aforementioned technical problems.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a data query method, including: Retrieve the target query statement; If the target query statement is a join query statement, then the metadata of the data table to be queried is obtained according to the target query statement; Based on the metadata of the data table to be queried, the target query statement is decomposed into a first type of query statement and a second type of query statement; Based on the first type of query statement and the second type of query statement, queries are performed on the first cluster and the second cluster respectively to obtain query results, wherein the first cluster and the second cluster are data clusters of different types.

[0007] Optionally, before obtaining the target query statement, the method further includes: Synchronize the metadata of all data tables in the first cluster and the second cluster to the preset metadata database; The step of obtaining the metadata of the data table to be queried based on the target query statement includes: Based on the target query statement, the metadata of the data table to be queried is obtained from the preset metadata database.

[0008] Optionally, the step of querying the first cluster and the second cluster respectively according to the first type of query statement and the second type of query statement to obtain query results includes: The first type of query statement is decomposed into multiple first sub-query statements, and the second type of query statement is decomposed into multiple second sub-query statements; Based on the multiple first sub-query statements, the first cluster is queried to obtain the first query result; Based on the multiple second sub-query statements, the second cluster is queried to obtain the second query result; The first query result and the second query result are correlated to obtain the query result.

[0009] Optionally, the step of querying the first cluster based on the plurality of first sub-query statements to obtain the first query result includes: Determine whether any of the multiple first subqueries involve online analytical queries; If no online analytical query is involved, the first cluster is queried according to the multiple first sub-query statements to obtain the first query result; If the online analysis query is involved, the first cluster is queried according to multiple first sub-query statements to obtain matching first data information; The first query result is obtained based on the first data information and the plurality of first sub-query statements.

[0010] Optionally, obtaining the first query result based on the first data information and the plurality of first sub-query statements includes: Using the engine of the second cluster, the first data information is queried according to the multiple first sub-query statements to obtain the first query result.

[0011] Optionally, the step of querying the second cluster based on the plurality of second sub-query statements to obtain the second query result includes: Determine whether any of the multiple second subqueries involve online analytical queries; If an online analytical query is involved, the second cluster is queried according to the multiple second sub-query statements to obtain the second query result; If the online analysis query is not involved, the second cluster is queried according to multiple second sub-query statements to obtain matching second data information; The second query result is obtained based on the second data information and the plurality of second sub-query statements.

[0012] Optionally, obtaining the second query result based on the second data information and the plurality of second sub-query statements includes: Using the engine of the first cluster, the second data information is queried according to the multiple second sub-query statements to obtain the second query result.

[0013] Secondly, embodiments of this application also provide a data query device, including: The acquisition module is used to acquire the target query statement; if the target query statement is a query statement of a join query, then the metadata of the data table to be queried is acquired according to the target query statement. The decomposition module is used to decompose the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried; The query module is used to query the first cluster and the second cluster respectively according to the first type of query statement and the second type of query statement, and obtain query results, wherein the first cluster and the second cluster are data clusters of different types.

[0014] Optionally, before obtaining the target query statement, the apparatus further includes: The synchronization module is used to synchronize the metadata of all data tables in the first cluster and the second cluster to the preset metadata database; The acquisition module is specifically used to acquire the metadata of the data table to be queried from the preset metadata database according to the target query statement.

[0015] Optionally, the query module is specifically configured to decompose the first type of query statement into multiple first sub-query statements and the second type of query statement into multiple second sub-query statements; query the first cluster according to the multiple first sub-query statements to obtain a first query result; query the second cluster according to the multiple second sub-query statements to obtain a second query result; and perform a correlation operation on the first query result and the second query result to obtain the query result.

[0016] Optionally, the query module is specifically used to determine whether the plurality of first sub-query statements involve an online analytical query (OLA); if no OLA is involved, the first cluster is queried according to the plurality of first sub-query statements to obtain the first query result; if the OLA is involved, the first cluster is queried according to the plurality of first sub-query statements to obtain matching first data information; and the first query result is obtained according to the first data information and the plurality of first sub-query statements.

[0017] Optionally, the query module is specifically used to use the engine of the second cluster to query the first data information according to the plurality of first sub-query statements to obtain the first query result.

[0018] Optionally, the query module is specifically used to determine whether the plurality of second sub-query statements involve an online analytical query (OLA) query; if an OLA query is involved, the second cluster is queried according to the plurality of second sub-query statements to obtain the second query result; if the OLA query is not involved, the second cluster is queried according to the plurality of second sub-query statements to obtain matching second data information; and the second query result is obtained according to the second data information and the plurality of second sub-query statements.

[0019] Optionally, the query module is specifically used to use the engine of the first cluster to query the second data information according to the plurality of second sub-query statements to obtain the second query result.

[0020] Thirdly, embodiments of this application also provide a server, including: a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the data query method described in any of the first aspects above.

[0021] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when read and executed, implements the data query method described in any of the first aspects above.

[0022] The beneficial effects of this application are as follows: This application provides a data query method, apparatus, server, and storage medium. The data query method may include: obtaining a target query statement; if the target query statement is a joint query statement, obtaining the metadata of the data table to be queried based on the target query statement; decomposing the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried; and querying a first cluster and a second cluster respectively based on the first type of query statement and the second type of query statement to obtain query results, wherein the first cluster and the second cluster are data clusters of different types. By decomposing the target query statement into a first type of query statement and a second type of query statement for querying in the first cluster and the second cluster respectively, users only need to use the target query statement once to perform a dual-source joint query in the first cluster and the second cluster, thus improving the user experience. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a data query method provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating a data query method provided in this application embodiment. Figure 2 ; Figure 3 This is a schematic diagram of the structure of a data query system provided in an embodiment of this application; Figure 4 A flowchart illustrating a data query method provided in this application embodiment. Figure 3 ; Figure 5 A flowchart illustrating a data query method provided in this application embodiment. Figure 4 ; Figure 6 A flowchart illustrating a data query method provided in this application embodiment. Figure 5 ; Figure 7 A complete architecture diagram corresponding to the flow of a data query method provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0026] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0027] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0028] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0030] This application provides a data query method applicable to any server in a data query system. The data query system comprises multiple servers; if one server fails, the system can switch to one of the remaining servers, thus improving the reliability of the data query. The data query system may further include a first cluster and a second cluster, with multiple servers communicating with the first cluster and the second cluster respectively.

[0031] Figure 1 A flowchart illustrating a data query method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method may include: S101. Obtain the target query statement.

[0032] The target query statement can be an SQL (Structured Query Language) query statement.

[0033] In some implementations, users can input their target query statement through a client interface. The server then receives the query statement, parses its syntax to determine if it conforms to standard SQL syntax rules, and returns a corresponding message to the user if a syntax error is detected; otherwise, it proceeds to the next processing step.

[0034] S102. If the target query statement is a join query statement, then obtain the metadata of the data table to be queried based on the target query statement.

[0035] In this embodiment of the application, the target query statement carries annotation information. It is determined whether the annotation information in the target query statement is a joint query annotation. If the annotation information in the target query statement is a joint query annotation, then the target query statement is determined to be a joint query statement.

[0036] It should be noted that the target query statement can include any of the following annotations: / db:hive / 、 / db:ck / 、 / db:all / ;in, / db:hive / This is an annotation for querying the first cluster; / db:ck / To query the annotations of the second cluster, / db:all The ' / ' symbol is an annotation for a join query. The join query refers to a two-source join query performed on the first and second clusters.

[0037] In some implementations, a preset evaluator is used to parse the target query statement to obtain the identifier of the data table to be queried, and the metadata of the data table to be queried is obtained based on the identifier of the data table to be queried in the parsing result.

[0038] S103. Based on the metadata of the data table to be queried, decompose the target query statement into a first type of query statement and a second type of query statement.

[0039] Among them, the first type of query statement and the second type of query statement are used to query different types of data clusters respectively.

[0040] In some implementations, the metadata of the data table to be queried and the target query statement are analyzed to obtain analysis results; based on the analysis results, the complex target query statement is decomposed into a first type of query statement and a second type of query statement.

[0041] It's worth noting that the analysis results include, but are not limited to, information such as the number of joins, the number of table rows, and partitions. Based on this information, the execution plan complexity of the target query can be determined. The number of joins refers to the number of join operations executed in the target query. Join operations are a core component of relational database management systems, used to combine data rows from two or more database tables based on join conditions.

[0042] S104. Based on the first type of query statement and the second type of query statement, perform queries on the first cluster and the second cluster respectively to obtain the query results.

[0043] The first cluster and the second cluster are different types of data clusters.

[0044] In some implementations, a query is performed on the first cluster according to a first type of query statement, and a query is performed on the second cluster according to a second type of query statement to obtain the query results.

[0045] It should be noted that the first cluster is good at batch processing of massive amounts of data and can stably execute complex query operations, but its query efficiency is low. The second cluster is good at fast queries, but its performance is poor when processing complex operations.

[0046] In this embodiment, the target query statement is decomposed into a first type of query statement and a second type of query statement. The advantages of the first cluster and the second cluster are utilized to perform queries in the first cluster and the second cluster respectively. For the target query statement, a dual-source joint query can be achieved in the first cluster and the second cluster.

[0047] In summary, this application provides a data query method, which may include: obtaining a target query statement; if the target query statement is a joint query statement, obtaining the metadata of the data table to be queried based on the target query statement; decomposing the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried; and querying a first cluster and a second cluster respectively based on the first type of query statement and the second type of query statement to obtain query results, wherein the first cluster and the second cluster are data clusters of different types. By decomposing the target query statement into a first type of query statement and a second type of query statement for querying in the first cluster and the second cluster respectively, users only need to use the target query statement once to perform a dual-source joint query in the first cluster and the second cluster, thus improving the user experience.

[0048] Moreover, for the target query statement, the advantages of both the first and second clusters can be combined to obtain query results quickly and accurately.

[0049] Optionally, the first cluster can be a Hive cluster. Hive is a data warehouse built on top of the Hadoop ecosystem, adept at batch processing massive amounts of data. By transforming SQL into distributed computing tasks such as MapReduce (a distributed parallel computing framework) or Tez / Spark (a directed acyclic graph execution framework), it can stably execute extremely complex operations such as multi-table joins and nested queries. However, its drawbacks are equally apparent: low query efficiency, with query latency typically on the order of minutes or even hours, making it unsuitable for interactive, ad-hoc queries.

[0050] Alternatively, the second cluster could be the CK cluster. CK clusters, representing MPP (Massively Parallel Processing) columnar databases, are renowned for their exceptional query performance, particularly excelling at aggregation and filtering analysis of single or small tables. Through key technologies such as vectorized execution, columnar storage, and data pre-aggregation, they achieve "second-level" or even "sub-second-level" response times in OLAP (Online Analytical Processing) scenarios. However, CK clusters perform poorly when handling complex multi-table join operations, especially in scenarios with data skew or large table joins, limiting their application in complex business scenarios.

[0051] Optionally, Figure 2 A flowchart illustrating a data query method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, before the process of obtaining the target query statement in S101 above, the method may further include: S201. Synchronize the metadata of all data tables in the first and second clusters to the preset metadata database.

[0052] The default metadata database consists of metadata databases from multiple servers.

[0053] Figure 3 This is a schematic diagram of the structure of a data query system provided in an embodiment of this application; as shown below. Figure 3 As shown, the first and second clusters include data and metadata of data tables. Using a preset tool, the metadata of all data tables in the first and second clusters is synchronized to a preset metadata database on multiple servers. The preset tool can be an ETL (Extract, Transform, Load) tool.

[0054] In addition, multiple servers can include: server A, server B, and server C.

[0055] The process of obtaining the metadata of the data table to be queried based on the target query statement in S102 above may include: S202. Based on the target query statement, retrieve the metadata of the data table to be queried from the preset metadata database.

[0056] In some implementations, a preset evaluator is used to parse the target query statement to obtain the identifier of the data table to be queried, and the metadata of the data table to be queried is obtained from a preset metadata database based on the identifier of the data table to be queried in the parsing result.

[0057] It should be noted that the parsing results may include, but are not limited to: the number of Join tables, Join type, whether a subquery exists, and the complexity of aggregate functions.

[0058] In this embodiment of the application, the preset metadata database includes the following information: storage source (first cluster or second cluster), database name, specific layered database, table name, number of table data rows, partition field, number of partitions, and operation time.

[0059] For example, the first cluster is a Hive cluster. The user_orders table (user order association table) in Hive's ODS (Operational Data Store) database is recorded as follows: Hive, ods, user_orders, 125000000, dt, 28, 2025-10-10 14:00:05, where Hive is the Hive cluster, ods is the database name, user_orders is the table name, 125000000 is the number of rows in the table, dt is the partition field, 28 is the number of partitions, and 2025-10-10 14:00:05 is the operation time.

[0060] Optionally, Figure 4 A flowchart illustrating a data query method provided in this application embodiment. Figure 3 ;like Figure 4 As shown, the process in S104 above, which queries the first cluster and the second cluster respectively based on the first type of query statement and the second type of query statement to obtain the query results, may include: S301. Decompose the first type of query statement into multiple first subquery statements, and decompose the second type of query statement into multiple second subquery statements.

[0061] In this embodiment of the application, in order to reduce the load of a single query, improve parallel processing capabilities, and adapt to the technical characteristics of different data clusters, the server further performs logical decomposition on the decomposed first type of query statement and second type of query statement.

[0062] In some implementations, the first type of query statement is decomposed into multiple first subquery statements one by one, and the second type of query statement is decomposed into multiple second subquery statements one by one.

[0063] It should be noted that the first type of query statement can be decomposed first, and then the second type of query statement can be decomposed; or the second type of query statement can be decomposed first, and then the first type of query statement can be decomposed; or the first type of query statement and the second type of query statement can be decomposed simultaneously. This application embodiment does not impose specific limitations on this.

[0064] S302. Based on multiple first subquery statements, query the first cluster to obtain the first query result.

[0065] In some implementations, based on the analysis results, multiple first subqueries are used to query the first cluster to obtain first query results. The analysis results are used to indicate whether the multiple first query results involve OLAP (Online Analytical Processing) queries.

[0066] It should be noted that OLAP queries are analytical queries for multidimensional data, allowing users to quickly explore and aggregate massive amounts of data from different angles, granularities, and levels.

[0067] S303. Based on multiple second subquery statements, query the second cluster to obtain the second query result.

[0068] In some implementations, based on the analysis results, multiple second sub-queries are used to query the second cluster to obtain second query results. The analysis results indicate whether the multiple second query results involve OLAP (Online Analytical Processing) queries.

[0069] It is worth noting that the process of S202 can be executed first, followed by the process of S203; or the process of S203 can be executed first, followed by the process of S202; or the processes of S202 and S203 can be executed simultaneously. This application embodiment does not impose specific restrictions on this.

[0070] S304. Perform a join operation on the first query result and the second query result to obtain the query result.

[0071] In this embodiment of the application, after obtaining the first query result and the second query result, the first query result and the second query result are cached on the server or on the server's disk, and the cached first query result and the second query result are associated to obtain the query result.

[0072] In addition, the server can send the query results to the client, and the client can receive and display the query results.

[0073] Optionally, Figure 5 A flowchart illustrating a data query method provided in this application embodiment. Figure 4 ;like Figure 5 As shown, the process in S202 above, which queries the first cluster based on multiple first sub-query statements to obtain the first query result, may include: S401. Determine whether multiple first subquery statements involve online analytical queries.

[0074] Online analytical query (OLAP) is also known as a query that performs online analysis.

[0075] In some implementations, based on the analysis results, it is determined whether multiple first subqueries involve online analytical queries (OLA) to obtain a determination result. This determination result indicates whether the first subqueries involve OLA or not.

[0076] It should be noted that Online Analytical Query is a query technique specifically designed for multidimensional data analysis, allowing decision-makers to explore large amounts of historical data in an interactive, multidimensional way.

[0077] S402. If no online analytical query is involved, query the first cluster based on multiple first subquery statements to obtain the first query result.

[0078] In some implementations, if no online analytical query is involved and only multiple association operations exist, the first cluster is queried directly based on multiple first subquery statements to obtain the first query result. Here, the first cluster is a Hive cluster.

[0079] In addition, the first query result is cached on any one of the multiple servers. Since the cached data of the first query result is large, the intermediate data of the first query result is written to the disk of the server.

[0080] S403. If online analytical query is involved, query the first cluster based on multiple first subquery statements to obtain the matching first data information.

[0081] In some implementations, if online analytical queries are involved, the Hive cluster is queried based on multiple first subqueries to obtain matching first data information, and the matching first data information is cached on the server.

[0082] S404. Based on the first data information and multiple first subquery statements, obtain the first query result.

[0083] The first data information is detailed data. The first data information is queried based on multiple first subquery statements to obtain the first query result.

[0084] It should be noted that the first query result is cached on the server. If the cached data of the first query result is large, the intermediate data corresponding to the first query result is written to the server's disk.

[0085] Optionally, based on the first data information and multiple first subquery statements, the first query result is obtained, including: Using the engine of the second cluster, the first data information is queried based on multiple first subquery statements to obtain the first query result.

[0086] In some implementations, the CK engine is used to query the first data information based on multiple first subquery statements to obtain the first query result.

[0087] In summary, this application proactively migrates the computation task of the first subquery to the engine of the second cluster when it detects that online analytical processing (OLAP) is involved, achieving an innovative architecture that allows data retrieval and optimal computation selection. This not only significantly improves the response speed of complex analytical queries but also optimizes resource allocation between heterogeneous clusters, effectively solving the performance bottleneck problem caused by engine limitations in traditional solutions. This solution greatly improves the overall query efficiency and user experience of the system while ensuring data consistency.

[0088] Optionally, Figure 6 A flowchart illustrating a data query method provided in this application embodiment. Figure 5 ;like Figure 6 As shown, the process in S303 above, which queries the second cluster based on multiple second sub-query statements to obtain the second query result, may include: S501. Determine whether multiple second subquery statements involve online analytical queries.

[0089] Based on the analysis results, it is determined whether multiple second subqueries involve online analytical queries (OLMs). The determination result indicates whether the second subqueries involve OLMs or not.

[0090] S502. If online analytical queries are involved, the second cluster is queried based on multiple second subquery statements to obtain the second query results.

[0091] In some implementations, if online analytical queries are involved and there are a few table joins, the second cluster is queried directly based on multiple second subqueries to obtain the second query results. The second cluster is the CK cluster.

[0092] In addition, the second query result is cached on any of the multiple servers. Since the cached data of the second query result is large, the intermediate data of the second query result is written to the disk of the server.

[0093] S503. If online analytical queries are not involved, the second cluster is queried based on multiple second subquery statements to obtain matching second data information.

[0094] Among them, "not involving online analytical queries" refers to the occurrence of multiple Join relationships.

[0095] In some implementations, if online analytical queries are not involved, the CK cluster is queried based on multiple second subquery statements to obtain matching second data information, and the matching second data information is cached on the server.

[0096] S504. Based on the second data information and multiple second subquery statements, obtain the second query result.

[0097] The second data information is detailed data. The second data information is queried based on multiple second sub-query statements to obtain the second query result.

[0098] It should be noted that the second query result is cached on the server. If the cached data of the second query result is large, the intermediate data corresponding to the second query result will be written to the server's disk.

[0099] Optionally, the process of obtaining the second query result based on the second data information and multiple second subquery statements in S504 above may include: Using the engine of the first cluster, the second data information is queried based on multiple second subquery statements to obtain the second query result.

[0100] In some implementations, the Hive engine is used to query the second data information based on multiple second subqueries to obtain the second query results.

[0101] In this embodiment, a correlation operation is performed on the first query result and the second query result. Based on the number of correlated items in the first query result and the second query result, a target engine is determined from the engines of the first cluster and the second cluster. The target engine is then used to perform the correlation operation on the first query result and the second query result to achieve result set merging.

[0102] If there are two related items, the target engine is the engine of the second cluster, and the engine of the second cluster is used to perform the join operation on the first query result and the second query result. If there are multiple related items, the target engine is the engine of the first cluster, and the engine of the first cluster is used to perform the join operation on the first query result and the second query result.

[0103] In summary, this application can distinguish between online analytical queries and non-online analytical queries; fully leverages the extreme performance advantages of the second cluster (such as CK) in OLAP scenarios; in non-OLAP scenarios, it first queries the CK cluster and then uses the Hive engine for the query; and it provides high-quality, low-latency intermediate results support for cross-cluster joint queries. This not only improves query efficiency and user experience but also constructs an intelligent joint query for different data clusters, fully utilizing the respective query advantages of the second and first clusters.

[0104] Figure 7 A complete architecture diagram corresponding to the flow of a data query method provided in this application embodiment; as shown Figure 7 As shown, it includes: obtaining the client request, determining the input category of the target query statement in the client request, and if the input category is / db:hive If the input category is / , then the Hive cluster is queried directly based on the target query statement; if the input category is / db:ck If / , then the CK cluster is queried directly based on the target query statement.

[0105] If the input category is / db:all If / , then the data query method provided in this application embodiment is executed. Specifically, according to the target query statement, the metadata of the data table to be queried is obtained from the preset metadata database; then the server decomposes the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried; then, a complexity estimator is used to query the Hive cluster and the CK cluster respectively based on the first type of query statement and the second type of query statement.

[0106] This includes multiple join queries and OLAP queries for Hive and CK clusters. The query results are cached on the server.

[0107] In summary, the embodiments of this application can solve the technical problems in the prior art where CK clusters cannot efficiently handle complex joins, and Hive clusters have excessively high query latency, making it difficult to use the two in a unified manner. This application can meet the needs of business scenarios requiring data analysis and processing of multiple table joins and OLAP.

[0108] The following describes the data query apparatus, server, and storage medium used to execute the data query method provided in this application. For the specific implementation process and technical effects, please refer to the relevant content of the above-mentioned data query method, which will not be repeated below.

[0109] Figure 8 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes: The acquisition module 101 is used to acquire the target query statement; if the target query statement is a query statement of a joint query, then the metadata of the data table to be queried is acquired according to the target query statement. The decomposition module 102 is used to decompose the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried. The query module 103 is used to query the first cluster and the second cluster respectively according to the first type of query statement and the second type of query statement to obtain query results, wherein the first cluster and the second cluster are data clusters of different types.

[0110] Optionally, before obtaining the target query statement, the apparatus further includes: The synchronization module is used to synchronize the metadata of all data tables in the first cluster and the second cluster to the preset metadata database; The acquisition module is specifically used to acquire the metadata of the data table to be queried from the preset metadata database according to the target query statement.

[0111] Optionally, the query module 103 is specifically configured to decompose the first type of query statement into multiple first sub-query statements and decompose the second type of query statement into multiple second sub-query statements; query the first cluster according to the multiple first sub-query statements to obtain a first query result; query the second cluster according to the multiple second sub-query statements to obtain a second query result; and perform a correlation operation on the first query result and the second query result to obtain the query result.

[0112] Optionally, the query module 103 is specifically used to determine whether the plurality of first sub-query statements involve online analytical query (OLA); if no OLA is involved, the first cluster is queried according to the plurality of first sub-query statements to obtain the first query result; if OLA is involved, the first cluster is queried according to the plurality of first sub-query statements to obtain matching first data information; and the first query result is obtained according to the first data information and the plurality of first sub-query statements.

[0113] Optionally, the query module 103 is specifically used to use the engine of the second cluster to query the first data information according to the plurality of first sub-query statements to obtain the first query result.

[0114] Optionally, the query module 103 is specifically used to determine whether the plurality of second sub-query statements involve an online analytical query; if an online analytical query is involved, the second cluster is queried according to the plurality of second sub-query statements to obtain the second query result; if the online analytical query is not involved, the second cluster is queried according to the plurality of second sub-query statements to obtain matching second data information; and the second query result is obtained according to the second data information and the plurality of second sub-query statements.

[0115] Optionally, the query module 103 is specifically used to use the engine of the first cluster to query the second data information according to the plurality of second sub-query statements to obtain the second query result.

[0116] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0117] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0118] Figure 9 This application provides a schematic diagram of the structure of a server, as shown in the embodiment of the present application. Figure 9 As shown, the server includes: processor 201 and memory 202.

[0119] The memory 202 is used to store programs, and the processor 201 calls the programs stored in the memory 202 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described in detail here.

[0120] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0124] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data query method, characterized in that, include: Retrieve the target query statement; If the target query statement is a join query statement, then the metadata of the data table to be queried is obtained according to the target query statement; Based on the metadata of the data table to be queried, the target query statement is decomposed into a first type of query statement and a second type of query statement; Based on the first type of query statement and the second type of query statement, queries are performed on the first cluster and the second cluster respectively to obtain query results, wherein the first cluster and the second cluster are data clusters of different types.

2. The method according to claim 1, characterized in that, Before obtaining the target query statement, the method further includes: Synchronize the metadata of all data tables in the first cluster and the second cluster to the preset metadata database; The step of obtaining the metadata of the data table to be queried based on the target query statement includes: Based on the target query statement, the metadata of the data table to be queried is obtained from the preset metadata database.

3. The method according to claim 1, characterized in that, The step of querying the first cluster and the second cluster according to the first type of query statement and the second type of query statement respectively, and obtaining query results, includes: The first type of query statement is decomposed into multiple first sub-query statements, and the second type of query statement is decomposed into multiple second sub-query statements; Based on the multiple first sub-query statements, the first cluster is queried to obtain the first query result; Based on the multiple second sub-query statements, the second cluster is queried to obtain the second query result; The first query result and the second query result are correlated to obtain the query result.

4. The method according to claim 3, characterized in that, The step of querying the first cluster based on the plurality of first sub-query statements to obtain the first query result includes: Determine whether any of the multiple first subqueries involve online analytical queries; If no online analytical query is involved, the first cluster is queried according to the multiple first sub-query statements to obtain the first query result; If the online analysis query is involved, the first cluster is queried according to multiple first sub-query statements to obtain matching first data information; The first query result is obtained based on the first data information and the plurality of first sub-query statements.

5. The method according to claim 4, characterized in that, The step of obtaining the first query result based on the first data information and the plurality of first sub-query statements includes: Using the engine of the second cluster, the first data information is queried according to the multiple first sub-query statements to obtain the first query result.

6. The method according to claim 3, characterized in that, The step of querying the second cluster based on the plurality of second sub-query statements to obtain the second query result includes: Determine whether any of the multiple second subqueries involve online analytical queries; If an online analytical query is involved, the second cluster is queried according to the multiple second sub-query statements to obtain the second query result; If the online analysis query is not involved, the second cluster is queried according to multiple second sub-query statements to obtain matching second data information; The second query result is obtained based on the second data information and the plurality of second sub-query statements.

7. The method according to claim 6, characterized in that, The step of obtaining the second query result based on the second data information and the plurality of second sub-query statements includes: Using the engine of the first cluster, the second data information is queried according to the multiple second sub-query statements to obtain the second query result.

8. A data query device, characterized in that, include: The retrieval module is used to retrieve the target query statement; If the target query statement is a join query statement, then the metadata of the data table to be queried is obtained according to the target query statement; The decomposition module is used to decompose the target query statement into a first type of query statement and a second type of query statement based on the metadata of the data table to be queried; The query module is used to query the first cluster and the second cluster respectively according to the first type of query statement and the second type of query statement, and obtain query results, wherein the first cluster and the second cluster are data clusters of different types.

9. A server, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the data query method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when read and executed, implements the data query method according to any one of claims 1-7.

Images