Query request processing method and device, storage medium and computer equipment

Abstract

The invention provides a query request processing method and device, a storage medium and computer equipment. The method comprises the following steps: obtaining a query request, and analyzing the query request to obtain a query plan; generating a plurality of query fragments corresponding to each query step, wherein when the plurality of query fragments corresponding to the same query step are executed, a query function which is the same as that of the corresponding query step is realized; when each query step is executed, a plurality of query fragments corresponding to each query step are scheduled to different execution units at the same time to be executed in parallel; determining the execution result of each query step according to the execution result of the target query fragment corresponding to each query step, wherein the target query fragment is the query fragment which is executed first in the plurality of query fragments which are executed in parallel; and determining a target query result corresponding to the query request according to the execution result corresponding to each query step and the execution sequence. The method can improve the processing efficiency of the query request.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to a query request processing method, device, storage medium and computer equipment. Background Art

[0002] Structured Query Language (SQL) is the mainstream interactive language used by databases or data warehouses. When an object uses SQL query language to initiate a query request to a database or data warehouse, the database or data warehouse usually first parses the query request to generate an execution plan, and then queries the data stored in the database or data warehouse according to the execution plan to obtain the query results.

[0003] However, the query request processing method adopted in the related art has low efficiency in query request processing. Summary of the invention

[0004] The embodiments of the present disclosure provide a query request processing method, apparatus, storage medium and computer equipment, which can improve the query request processing efficiency.

[0005] According to one aspect of the present disclosure, a query request processing method is provided, comprising:

[0006] Obtaining a query request, and parsing the query request to obtain a query plan, wherein the query plan includes a plurality of query steps and an execution order between the plurality of query steps;

[0007] Generate a plurality of query fragments corresponding to each of the query steps, wherein the plurality of query fragments corresponding to the same query step realize the same query function as the corresponding query step when executed;

[0008] When executing each of the query steps, the multiple query fragments corresponding to each of the query steps are simultaneously scheduled to different execution units for parallel execution;

[0009] Determine the execution result of each query step according to the execution result of the target query slice corresponding to each query step, wherein the target query slice is the query slice that is executed first among the multiple query slices executed in parallel;

[0010] The target query result corresponding to the query request is determined according to the execution result corresponding to each query step and the execution order.

[0011] According to one aspect of the present disclosure, there is provided a query request processing device, comprising:

[0012] An acquisition unit, configured to acquire a query request, and parse the query request to obtain a query plan, wherein the query plan includes a plurality of query steps and an execution order between the plurality of query steps;

[0013] A generating unit, configured to generate a plurality of query fragments corresponding to each of the query steps, wherein the plurality of query fragments corresponding to the same query step realize the same query function as the corresponding query step when executed;

[0014] A scheduling unit, configured to schedule the multiple query fragments corresponding to each query step to different execution units for parallel execution when executing each query step;

[0015] A first determining unit, configured to determine an execution result of each query step according to an execution result of a target query shard corresponding to each query step, wherein the target query shard is a query shard that is executed first among a plurality of query shards executed in parallel;

[0016] The second determining unit is used to determine the target query result corresponding to the query request according to the execution result corresponding to each query step and the execution order.

[0017] Optionally, in some embodiments, the first determining unit includes:

[0018] A detection subunit, used for detecting the execution progress of each query slice during the parallel execution of multiple query slices corresponding to each query step;

[0019] The determination subunit is used to determine the execution result of the corresponding query step according to the execution result of the target query slice among the multiple query slices when it is detected that the execution of the target query slice is completed.

[0020] Optionally, in some embodiments, the query request processing device provided by the present disclosure further includes:

[0021] A termination subunit, used to terminate the execution process of other query slices except the target query slice among the multiple query slices;

[0022] The release subunit is used to release the computing resources of the different execution units.

[0023] Optionally, in some embodiments, the detection subunit is further used for:

[0024] During the parallel execution of the plurality of query fragments corresponding to each of the query steps, obtaining the execution status information sent by the different execution units;

[0025] The determining subunit is further used for:

[0026] When it is detected that the execution status information sent by the target execution unit indicates that the execution of the corresponding query slice is completed, the execution result of the corresponding query step is determined according to the execution result of the target execution unit on the corresponding target query slice.

[0027] Optionally, in some embodiments, the acquiring unit includes:

[0028] A first acquisition subunit is used to acquire a query request, wherein the query request includes a structured query statement;

[0029] A parsing subunit, used to parse the structured query statement to obtain a corresponding abstract syntax tree;

[0030] The first generating subunit is used to generate a query plan according to the abstract syntax tree.

[0031] Optionally, in some embodiments, the first generating subunit includes:

[0032] A first generating module, configured to generate a plurality of candidate query plans according to the abstract syntax tree;

[0033] A calculation module, used to calculate the query resource consumption corresponding to each candidate query plan;

[0034] The first determination module is used to determine a query plan from the multiple candidate query plans according to the query resource consumption.

[0035] Optionally, in some embodiments, the first determining module includes:

[0036] A sorting submodule, used to sort the multiple candidate query plans in order from low to high according to the query resource consumption;

[0037] The determination submodule is used to determine that the candidate query plan ranked first is the generated query plan.

[0038] Optionally, in some embodiments, the generating unit comprises:

[0039] A second acquisition subunit is used to acquire the query function corresponding to each query step;

[0040] The third acquisition subunit is used to obtain statistical information and metadata of business data in the target database;

[0041] The second generating subunit is used to generate a plurality of query fragments realizing each of the query functions according to the statistical information and the metadata.

[0042] Optionally, in some embodiments, the second generating subunit includes:

[0043] A second determination module, configured to determine a combination of multiple query operators used to implement each query function according to the statistical information and the metadata;

[0044] The second generating module is used to generate multiple query fragments according to the combination of the multiple query operators.

[0045] Optionally, in some embodiments, the second generating subunit includes:

[0046] A detection module, used for performing multi-query shard detection on each query function according to the statistical information and the metadata;

[0047] The third generating module is used to generate multiple query shards that implement each query function according to the statistical information and the metadata when the detection result of the multiple query shard detection indicates that the query function can be implemented by multiple query shards.

[0048] Optionally, in some embodiments, the scheduling unit includes:

[0049] A fourth acquisition subunit, used to acquire execution capability information of each execution unit;

[0050] The scheduling subunit is used to schedule the multiple query fragments corresponding to each query step to different execution units for parallel execution according to the execution capability information when executing each query step.

[0051] According to one aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the query request processing method as described above when executing the computer program.

[0052] According to one aspect of the present disclosure, a storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the query request processing method as described above is implemented.

[0053] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the query request processing method as described above.

[0054] In an embodiment of the present disclosure, a query request is obtained and parsed to obtain a query plan, wherein the query plan includes multiple query steps and an execution order between the multiple query steps; multiple query slices corresponding to each query step are generated, and the multiple query slices corresponding to the same query step realize the same query function as the corresponding query step when executed; when each query step is executed, the multiple query slices corresponding to each query step are simultaneously scheduled to different execution units for parallel execution; the execution result of each query step is determined according to the execution result of the target query slice corresponding to each query step, and the target query slice is the query slice that is executed first among the multiple query slices executed in parallel; the target query result corresponding to the query request is determined according to the execution result corresponding to each query step and the execution order.

[0055] Thus, the query request processing method provided by the present disclosure generates multiple query fragments corresponding to each query step, and then when executing each query step, the multiple query fragments corresponding to the query step to be executed are simultaneously dispatched to multiple different execution units for parallel execution, and the execution result of the query step is determined according to the execution result of the query fragment that is executed the fastest. In this way, it can be ensured that each query step can be executed using the fastest execution method, thereby greatly improving the processing efficiency of the query request.

[0056] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0058] Figure 1 is a system architecture diagram of the query request processing method according to the embodiment of the present disclosure;

[0059] Figure 2 is a flowchart of a query request processing method provided by the present disclosure;

[0060] Figure 3 It is a schematic diagram of the processing process of a query request in the related art;

[0061] Figure 4 is a schematic diagram of the processing process of a query request in the present disclosure;

[0062] Figure 5is a schematic diagram of a grouping plan generated in the query request processing method provided by the present disclosure;

[0063] Figure 6 A schematic diagram of the process of executing the grouping plan in the present disclosure;

[0064] Figure 7 Another flowchart of the query request processing method provided by the present disclosure;

[0065] Figure 8 A schematic diagram of the process of determining the query shard group execution result for the group execution operator;

[0066] Fig. 9 A timing flow chart of the execution of the group query plan provided by the present disclosure;

[0067] Fig.10 A schematic diagram of the structure of a query request processing device provided in an embodiment of the present disclosure;

[0068] Fig.11 is a terminal structure diagram for implementing various methods according to an embodiment of the present disclosure;

[0069] Fig.12 It is a server structure diagram for implementing various methods according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0071] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0072] Database: It can be regarded as an electronic filing cabinet - a place to store electronic files. Objects can add, query, update, delete, etc. to the data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application.

[0073] Data Warehouse: A strategic collection of all types of data to support decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes. It provides guidance for business process improvement, cost, quality, and control for enterprises that need business intelligence. Unlike databases, which usually serve business, data warehouses usually serve analysis. Traditional databases serve business processes with small amounts of data, while data warehouses serve business analysis with large amounts of data.

[0074] Structured Query Language (SQL): The mainstream interactive language used in databases or data warehouses. It is a special-purpose programming language and a database query and programming language used to access data and query, update, and manage relational database systems.

[0075] Query Plan: The internal data structure converted after the query is parsed and processed by the database / data warehouse. It defines the steps for executing the query statement and the input / output relationship of each step.

[0076] Query Operator: The implementation of basic relational operations defined in relational algebra in the database / data warehouse kernel, such as Scan, Aggregate, Join, etc.

[0077] Query fragment: A section in a query plan, consisting of one or more operators, that implements a specific data processing.

[0078] When a data warehouse or database product processes an SQL query, it generally generates a query plan through a query parsing and optimization module, and then the execution engine schedules and executes the query operator according to the query plan. Therefore, the quality of the query plan is crucial to the end-to-end performance of query processing. The generation of a query plan not only depends on the query request, but also on some external information, such as some set rules and statistical information of business data stored in the database or data warehouse. However, this external information will change dynamically in real time with operations such as writing, updating, and deleting data in the database or data warehouse. Therefore, when executing a query according to the generated query plan, there may be a problem that the external information has changed, resulting in the generated query plan not being the optimal query plan, thereby affecting the processing efficiency of the query request. In this regard, the present disclosure provides a query request processing method in order to improve the processing efficiency of query requests.

[0079] System architecture and scenario description of the application of the embodiments of the present disclosure

[0080] Figure 1 1 is a system architecture diagram used by the query request processing method according to an embodiment of the present disclosure, and includes a terminal 140, the Internet 130, a gateway 120, a server 110, and the like.

[0081] The terminal 140 includes desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, dedicated terminals, intelligent voice interaction devices, smart home appliances, aircraft, etc. In addition, it can be a single device or a collection of multiple devices. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0082] The server 110 refers to a computer system that can provide certain services to the terminal 140. Compared with the ordinary terminal 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0083] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0084] The query request processing method of the embodiment of the present disclosure can be specifically implemented in the server 110 in the system architecture. Specifically, when the query request processing method is implemented in the server 110, the server 110 can obtain the query request, parse the query request, and obtain a query plan, which includes multiple query steps and the execution order between the multiple query steps; generate multiple query slices corresponding to each query step, and when the multiple query slices corresponding to the same query step are executed, they realize the same query function as the corresponding query step; when executing each query step, the multiple query slices corresponding to each query step are simultaneously scheduled to different execution units for parallel execution; determine the execution result of each query step according to the execution result of the target query slice corresponding to each query step, and the target query slice is the query slice that is executed first among the multiple query slices executed in parallel; determine the target query result corresponding to the query request according to the execution result and execution order corresponding to each query step.

[0085] The embodiments of the present disclosure can be applied in a variety of scenarios, such as online analytical processing (OLAP) query scenarios of data warehouses, online transaction processing (OLTP) query scenarios of data warehouses, hybrid transaction / analytical processing (HTAP) query scenarios, and artificial intelligence prediction query (AI predicate query) scenarios.

[0086] General description of the disclosed embodiments

[0087] According to an embodiment of the present disclosure, a query request processing method is provided. The method can be used in query request processing scenarios such as OLAP query, OLTP query, HTAP query, and AI prediction query of the aforementioned data warehouse, and can also be applied to query request processing scenarios of other databases.

[0088] like Figure 2 FIG. 1 is a flow chart of a query request processing method provided by the present disclosure. The method can be applied to a query request processing device, which can be integrated in a computer device, and the computer device can specifically be a server corresponding to a database or a data warehouse. The query request processing method may include:

[0089] Step 210: Obtain a query request, and parse the query request to obtain a query plan.

[0090] Among them, as described above, the query request processing method provided by the present disclosure can be applied to process query requests in a database, and can also be used in a data warehouse to process query requests for OLAP queries, OLTP queries, and HTAP queries, or it can also be used to process query requests in artificial intelligence prediction queries. Among them, in different scenarios, the query request processing method provided by the present disclosure has roughly the same processing flow for query requests. Therefore, in the embodiments of the present disclosure, the query request processing method provided by the present disclosure can be specifically described in detail by taking the processing of query requests in a database as an example.

[0091] When an object uses a terminal (database client) to access the content stored in the database, a query request can be sent to the database. The query request can contain some query condition information, including but not limited to the data table information to be queried, the screening condition information, the data table processing method information, etc. When the database server obtains the query request sent by the object through the client, the query request can be parsed and a query plan can be generated according to the parsing result. The generated query plan includes multiple query steps and the execution order between the multiple query steps.

[0092] In the related art, the parsing operation of the query request is generally implemented by the parser in the database server, and the SQL language in the query request is parsed and processed into an abstract syntax tree (AST). Then the optimizer generates the optimal query plan based on the parsed abstract syntax tree combined with the statistical information, metadata and rule information of the business data in the database, and finally the executor executes the query according to the execution plan to obtain the query result. The optimizer generates the optimal query plan based on the parsed abstract syntax tree combined with the statistical information, metadata and rule information of the business data in the database, and the process can be algebraic optimization and physical optimization. Among them, algebraic optimization changes the order and combination of operations in the query statement, and does not involve the underlying access path. There are many access plans for a query statement, and the efficiency of these different execution plans is different. It is not enough to perform algebraic optimization alone. Physical optimization is to select efficient and reasonable operation algorithms or access paths to obtain optimized query plans. Among them, physical optimization methods include rule-based heuristic optimization and cost estimation-based optimization. Whether it is an algebraic optimization method or a physical optimization method or a combination of the two, the optimal query plan is selected from multiple query plans that can achieve the goal of the query request based on the abstract syntax tree obtained by parsing combined with the reservation rules and statistical data of the business data.

[0093] When the optimizer in the related technology generates a query plan, it selects the best query plan from multiple query plans that can achieve the query target corresponding to the query request through a preset optimization scheme combined with the abstract syntax tree obtained by parsing the query request and the statistical information of the business data in the database. This process is actually a process of eliminating other query plans that are determined to be non-optimal according to the optimization principle. However, since there is a delay between the optimizer generating the query plan and the executor executing the query plan, and the statistical information of the business data in the database will change in real time with operations such as data writing, updating, and deleting, when the executor executes the optimal query plan determined by the optimizer, the so-called optimal query plan may no longer be the optimal query plan. In some cases, the optimal query plan determined by the optimizer is already a less efficient query plan when the executor executes it, which may have a negative impact on the query efficiency.

[0094] like Figure 3 As shown, it is a schematic diagram of the query request processing process in the related art. As shown in the figure, after receiving the query request, the parser 310 can parse the query request and output an abstract syntax tree; then the parsed abstract syntax tree is sent to the optimizer 320, so that the optimizer 320 generates a query plan according to the abstract syntax tree, the metadata of the business data and the statistical information of the business data; wherein the statistical information of the business data can be specifically collected from the business data by the statistical information collector 340; finally, the executor 330 executes the query according to the query plan generated by the optimizer 320 in combination with the business data, and the executor 330 outputs the execution result.

[0095] In some technologies, targeted optimization is performed through the Adaptive Query Execution (AQE) solution. In this solution, the query is re-optimized according to the current statistical information during the execution phase of the query plan to generate a new execution plan, such as dynamically merging and adjusting partitions (shuffle partitions), dynamically switching join strategies, etc. The essence of this solution is delayed optimization, and the plan is dynamically adjusted during the execution phase. However, since each query step is executed on different components during the execution phase, it will lead to the loss of global information, which makes it impossible to perform global optimization adjustments, and only local optimization effects can be achieved. From a global perspective, this result may not be the best solution, so the improvement effect of this solution is limited.

[0096] In the embodiment of the present disclosure, when parsing the query request and generating the query plan, it is not necessary to use the optimization scheme for optimization, and it is only necessary to generate a query plan that can achieve the query target of the query request. The query plan includes multiple query steps and the execution order between the multiple query steps. Each query step corresponds to a query function, and there can actually be multiple query steps to achieve the query function.

[0097] In some embodiments, obtaining a query request and parsing the query request to obtain a query plan includes:

[0098] Obtaining a query request, where the query request includes a structured query statement;

[0099] Parse the structured query statement to obtain the corresponding abstract syntax tree;

[0100] Generate a query plan based on the abstract syntax tree.

[0101] Among them, in the embodiment of the present disclosure, after obtaining a query request containing a structured query statement, the structured query statement can be parsed to obtain the corresponding abstract syntax tree. Among them, the parsing process is the process of converting the SQL query statement submitted by the object into an internal data structure that can be understood and executed by the database system. During the parsing process, the database will perform lexical, grammatical and semantic analysis on the query statement to ensure the accuracy and legality of the query. Query parsing can be specifically implemented by a parser. The purpose of query parsing is to convert the query statement into a query plan or optimizer input that can be executed by the database so that the database can select an effective execution plan and generate corresponding execution code. The parsing process can also detect errors and abnormal operations in the query statement and provide corresponding error information to the query object. Through parsing, the database can understand the query behavior in advance and provide accurate results for the object. Among them, the lexical analysis process divides the query statement into different lexical units; then, the grammatical analysis process generates the corresponding abstract syntax tree according to the SQL grammar rules.

[0102] Then, the corresponding query plan can be generated according to the parsed abstract syntax tree. Specifically, the process of generating a query plan corresponding to a query request according to the abstract syntax tree can also be applied to some preset rule data and statistical information of business data stored in the database. Among them, the statistical information can specifically include the number of rows in the table, the number of rows after deduplication, null values, maximum and minimum values, etc. In this way, a query plan can be generated that can achieve the query target of the query request in the database.

[0103] In some embodiments, generating a query plan according to an abstract syntax tree includes:

[0104] Generate multiple candidate query plans according to the abstract syntax tree;

[0105] Calculate the query resource consumption corresponding to each candidate query plan;

[0106] A query plan is determined from among multiple candidate query plans according to query resource consumption.

[0107] As described above, in the aforementioned public embodiment, according to the process of generating a query plan based on the abstract syntax tree, any query plan that can achieve the query target corresponding to the query request can be generated, that is, the query plan does not need to be optimized. In the disclosed embodiment, multiple candidate query plans can be generated according to the abstract syntax tree first, and then a query plan can be determined from multiple candidate query plans by using a cost estimation optimization method. Specifically, after generating multiple candidate query plans, the query resource consumption corresponding to each candidate query plan can be calculated respectively. Then, the generated query plan is determined from multiple candidate query plans according to the query resource consumption corresponding to each candidate query plan. Among them, calculating the query resource consumption corresponding to each candidate query plan is to calculate the execution cost of the execution plan corresponding to each candidate query plan. The execution cost is closely related to the state of the database, and specifically closely related to the statistical data of the business data stored in the aforementioned database. That is, the execution cost of the execution plan corresponding to each candidate query plan can be calculated by using a corresponding cost calculation method combined with statistical data.

[0108] In some embodiments, determining a query plan from a plurality of candidate query plans according to the query resource consumption includes:

[0109] Sort multiple candidate query plans according to the query resource consumption from low to high;

[0110] The candidate query plan ranked first is determined to be the generated query plan.

[0111] In the embodiment of the present disclosure, a candidate query plan with the minimum query resource consumption can be used as the generated query plan to reduce the query resource consumption caused by the query request while ensuring the query request processing efficiency is improved. For each query step, a variety of different operator combinations can be used to implement the function required to be implemented in the query step. Each operator combination that can implement the function required to be implemented in the query step can be called a query shard.

[0112] Specifically, after calculating the query resource consumption corresponding to each candidate query plan, the query resource consumption corresponding to each query plan can be sorted. Specifically, the sorting can be performed from high to low, and then the query plan ranked last is determined as the generated query plan; or, the sorting can be performed from low to high, and then the query plan ranked first is determined as the generated query plan.

[0113] Step 220: Generate multiple query fragments corresponding to each of the query steps.

[0114] In the related art, after the planner generates a query plan, the generated query plan can be delivered to the executor, and then the executor schedules the corresponding execution operator based on the received query plan, accesses the data table of the business data in the database and processes it, and then returns the query result. In the embodiment of the present disclosure, the generated query plan is only a preliminary query plan, which reveals the general process required to achieve the query target of the query request but not the necessary process. In the embodiment of the present disclosure, after generating a query plan including multiple query steps and the execution order between the multiple query steps, multiple query shards corresponding to each query step can be further generated. Among them, the multiple query shards corresponding to each query step here can specifically be all query shards that can achieve the same query function as the query step. The multiple query shards corresponding to each query step can be called the shard group corresponding to the query step.

[0115] In some embodiments, generating multiple query fragments corresponding to each query step includes:

[0116] Get the query function corresponding to each query step;

[0117] Obtain statistical information and metadata of business data in the target database;

[0118] Multiple query shards are generated to implement each query function based on statistical information and metadata.

[0119] In the disclosed embodiment, in the query plan generated, each query step corresponds to a query function. Wherein, the query function indicates the intermediate result obtained by the query step processing. For example, the function of a query step is to find out the number of boys aged over 10 years old among the third-grade students, that is, the query step corresponds to an intermediate result, and the intermediate result is the number of boys aged over 10 years old among the third-grade students. Wherein, there can be multiple method paths to realize the query function or obtain the intermediate result, for example, the third-grade students can be first screened out, and then the students aged over 10 years old can be further screened out from the screened third-grade students, and then the boys can be further screened out from the third-grade students aged over 10 years old; or, the boys in the school can be first screened out, and then the boys aged over 10 years old can be screened out from the boys, and finally the boys aged over 10 years old can be screened out from the third grade. The above two methods are just examples, and other methods can also be used to realize the query function or obtain the intermediate result.

[0120] Therefore, in the embodiment of the present disclosure, after parsing the query request to obtain a query plan that can realize the query target corresponding to the query request, the query function of each query step contained in the query plan can be obtained, and then multiple query slices that can realize the query function can be determined based on the obtained query function. Specifically, in the embodiment of the present disclosure, multiple query slices that can realize the query function can be determined based on the query function. Specifically, a traversal method can be used, that is, traversing all methods that can obtain intermediate results corresponding to the query function from the input data processing, and each achievable method corresponds to a query slice, so that all query slices corresponding to each query function can be traversed.

[0121] Specifically, before determining the query slice corresponding to each query function, the statistical information and metadata of the business data in the target database can be obtained first, and then multiple query slices for each query function can be generated based on the statistical information and metadata. Among them, the statistical information can specifically include: data table information and index information, etc.; the data table information can include the total number of tuples, tuple length, occupied fast-ripening and occupied overflow blocks of each data table, etc. In addition, it can also include column information of each data table, including the number of different values ​​of each column, the maximum and minimum values ​​of the column, whether an index has been established on the column, and what kind of index has been established. The index information can include the number of index levels, the number of different index values, the selection cardinality of the index, and the number of leaf nodes of the index. Metadata, also known as intermediary data or relay data, is data that describes data, mainly describing the attribute information of the data table stored in the database, and is used to support functions such as indicating storage location, historical data, resource search, and file records. Metadata is a kind of electronic catalog. In order to achieve the purpose of compiling a catalog, the content or characteristics of the data can be described and collected, thereby achieving the purpose of assisting data query.

[0122] In some embodiments, multiple query fragments for implementing each query function are generated based on statistical information and metadata, including:

[0123] Determine the combination of multiple query operators used to implement each query function based on statistical information and metadata;

[0124] Generate multiple query shards based on the combination of multiple query operators.

[0125] In an embodiment of the present application, after obtaining the statistical information and metadata of the business data stored in the database or data warehouse, the specific process of generating multiple query slices to implement each query function based on the obtained statistical information and metadata can be to traverse the combination of different query operators that can implement the query function in combination with the aforementioned statistical information and metadata, and then generate multiple query slices based on the combination of multiple query operators. Among them, the query operator here is specifically an execution operator that can be executed by the database or data warehouse system, such as a scan operator, an aggregate operator, etc. The process of generating multiple query slices based on the combination of multiple query operators can specifically be to arrange and combine the different operators in each query operator combination according to different execution orders, and then detect whether the query function implemented by each combination in combination with the statistical information and metadata is the same as the query function of the corresponding query step. After eliminating the combination that cannot implement the same query function as the corresponding query step, multiple query slices corresponding to each query step can be obtained.

[0126] Alternatively, in some embodiments, some intermediate results may be pre-processed and generated based on statistical data and metadata in a database and saved in a cache for later use. In this case, the intermediate results may be obtained without calling multiple query operators. In this case, the operation of directly calling the intermediate result may also be a query fragment corresponding to the intermediate data.

[0127] In some embodiments, multiple query fragments for implementing each query function are generated based on statistical information and metadata, including:

[0128] Perform multi-query sharding detection on each query function based on statistics and metadata;

[0129] When the detection result of the multi-query shard detection indicates that the query function can be implemented by multiple query shards, multiple query shards that implement each query function are generated according to the statistical information and the metadata.

[0130] In some embodiments, some intermediate results may only need to be processed simply to obtain, without the need for processing by multiple operators. In this case, the query function may only correspond to one query fragment. In this case, it is not necessary to generate multiple query fragments corresponding to the query step. For example, a certain intermediate result is that the male students in the student information table need to be screened out. This step only needs to be screened in one step to obtain it, without the need for multiple operator combinations to achieve it. Therefore, in the disclosed embodiment, before generating multiple query fragments corresponding to each query function, each query function can be tested for multiple query fragments according to statistical information and metadata. The purpose of the multiple query fragment test can be to detect whether each query function has multiple query fragments that can achieve the function. If the test result of the multiple query fragment test indicates that the query function can be achieved by a combination of multiple query fragments, multiple query fragments that achieve the corresponding query function are generated according to the statistical information and metadata. On the contrary, if the test result of the multiple query fragment test indicates that the query function does not have multiple query fragments that can achieve the function, it is not necessary to generate multiple query fragments corresponding to the query function, and further improve the processing efficiency of the query request.

[0131] Step 230: When executing each of the query steps, the multiple query fragments corresponding to each of the query steps are simultaneously scheduled to different execution units for parallel execution.

[0132] As described above, in the related art, a query shard is screened for each query step, and then a query shard combination screened by different query steps is used to generate the optimal target query plan, and then the executor executes the query according to the target query plan to obtain the query result. Since the generation of the target query plan depends largely on the statistical information of the business data in the database, and the real-time change of the statistical information causes the generated target query plan to be not the optimal query plan when it is executed, and the optimal query plan may be eliminated when the optimizer optimizes. To solve this problem, the query request processing method provided by the embodiment of the present disclosure does not eliminate the query plan shards when the query plan is generated, but traverses the query shards corresponding to each query step and retains the multiple query shards corresponding to each query step, and determines the query shard with the highest real-time efficiency by executing the multiple query shards corresponding to each query step in parallel by multi-threading when the executor executes. That is, compared with the related art in which the main process of query optimization is placed in the optimizer, the embodiment of the present disclosure places the main process of query optimization in the executor, determines the optimal (most efficient) execution plan during execution, so as to execute the query in real time in the most efficient way, thereby greatly improving the processing efficiency of the query request.

[0133] Specifically, when executing each query step, multiple query slices corresponding to each query step generated in step 220 can be simultaneously scheduled to different execution units for parallel execution. Among them, the execution unit here can be a thread or an independent execution device. That is, compared with related schemes, this solution may consume more computing resources, thereby achieving higher query request processing efficiency. With the continuous development of computer technology, computing power is also constantly improving. In some scenarios, computing power is no longer a limitation on query efficiency, but the design of query algorithms is a limitation on query efficiency. In this scenario, the query request processing method provided by the present disclosure can greatly improve the query request processing efficiency at the expense of certain computing power resources. Among them, in the embodiments of the present disclosure, different execution units can be equivalent, that is, different execution units have the same execution capabilities, and the execution efficiency of different query slices is only related to the query slices themselves, and has nothing to do with the execution capabilities of the execution units.

[0134] In some embodiments, when executing each query step, multiple query shards corresponding to each query step are simultaneously scheduled to different execution units for parallel execution, including:

[0135] Obtaining execution capability information of each execution unit;

[0136] When each query step is executed, multiple query fragments corresponding to each query step are simultaneously scheduled to different execution units for parallel execution according to the execution capability information.

[0137] Among them, in some embodiments, the execution capabilities of different execution units may also differ. For example, some execution units are good at processing a single operator of a large amount of data, and some execution units are good at switching processing of multiple operators of a small amount of data, etc., that is, different execution units have different execution capability information. In the embodiment of the present disclosure, before scheduling query slices to different execution units for synchronous and parallel execution, the execution capability information of each execution unit can be obtained first. Then, when executing each query step, the multiple query slices corresponding to each query step can be simultaneously scheduled to different execution units for parallel execution according to the execution capability information. Specifically, the execution unit corresponding to each query slice can be determined according to the properties of the specific operators contained in each query slice and the execution capability information corresponding to each execution unit, so as to further improve the execution efficiency of each query slice.

[0138] Step 240: Determine the execution result of each query step according to the execution result of the target query slice corresponding to each query step.

[0139] In the disclosed embodiment, when executing each query step, multiple query slices corresponding to the query step are simultaneously scheduled to different execution units for simultaneous and parallel execution, and the multiple query slices executed simultaneously and in parallel have the same query function, that is, the execution results (intermediate results of the output) of different query slices are the same. Therefore, the execution result of any query slice among the multiple query slices executed simultaneously and in parallel can be used as the execution result of the query step. In this way, in order to improve the processing efficiency of the query request, the execution result of the target query slice that obtains the execution result first can be determined as the execution result of the corresponding query step. In this way, for each query step with multiple query slices in the query plan, the corresponding execution result can be obtained with the minimum execution time, so that the processing efficiency of the query request can be improved to the greatest extent.

[0140] In some embodiments, determining the execution result of each query step according to the execution result of the target query shard corresponding to each query step includes:

[0141] During the parallel execution of multiple query shards corresponding to each query step, detecting the execution progress of each query shard;

[0142] When it is detected that the execution of a target query shard among multiple query shards is completed, the execution result of the corresponding query step is determined according to the execution result of the target query shard.

[0143] Among them, in the embodiment of the present disclosure, during the execution of each query step with multiple query shards, when the multiple shards corresponding to the query step are simultaneously scheduled to multiple execution units for synchronous and parallel execution, the execution progress of each query shard can be detected in real time. When the execution progress of multiple synchronously and parallelly executed query shards is detected to have been completed for a certain target query shard, the execution result of the corresponding query step can be determined according to the execution result of the target query shard.

[0144] In the disclosed embodiment, it is not necessary to determine the optimal query slice among multiple query slices and then execute the optimal query slice; instead, multiple query slices corresponding to each query step are directly dispatched to different execution units for synchronous parallel execution, and which query slice is the query slice with the highest execution efficiency is determined during the execution process. This avoids the problem that the query slice with the highest execution efficiency determined by pre-optimization is inaccurate, which not only fails to improve the processing efficiency of query requests, but may even reduce the processing efficiency of query requests to a certain extent.

[0145] In some embodiments, after detecting that the target query shard among the multiple query shards has been executed, determining the execution result of the corresponding query step according to the execution result of the target query shard, the method further includes:

[0146] Terminate the execution process of other query shards except the target query shard among the multiple query shards;

[0147] Release computing resources of different execution units.

[0148] Among them, in the embodiment of the present disclosure, when it is detected that one of the target query fragments of a query step that is executed in parallel in multiple execution units is completed, the execution process of other query fragments that are executed in parallel can be terminated at the same time, and the computing resources in the multiple execution units can be released. In this way, it can be avoided that multiple execution units continue to execute query fragments and cause a waste of execution resources. These execution units can be put into the execution process of multiple query fragments of the next query step.

[0149] After the execution result of the corresponding query step is determined according to the execution result of the target query shard, the execution result can be further input into the next query step. Here, the next query step can be determined according to the aforementioned execution plan, which includes the execution order between multiple query steps.

[0150] In this way, the disclosed embodiment can further save the consumption of computing resources corresponding to each query step on the basis of ensuring the query request processing efficiency, thereby further improving the processing efficiency of the entire query request processing process.

[0151] In some embodiments, during the parallel execution of multiple query fragments corresponding to each query step, detecting the execution progress of each query fragment includes:

[0152] During the parallel execution of multiple query shards corresponding to each query step, obtaining execution status information sent by different execution units;

[0153] When it is detected that the execution of the target query shard among the multiple query shards is completed, the execution result of the corresponding query step is determined according to the execution result of the target query shard, including:

[0154] When it is detected that the execution status information sent by the target execution unit indicates that the execution of the corresponding query slice is completed, the execution result of the corresponding query step is determined according to the execution result of the target execution unit on the corresponding target query slice.

[0155] In an embodiment of the present disclosure, multiple query fragments corresponding to each query step can be synchronously executed by multiple execution units in the executor, and multiple execution units in the executor can send execution status information to the processor during or after execution, and the processor can determine the execution progress of each query fragment based on the execution status information sent by the received execution unit. In this way, when the processor detects that the execution status information sent by the target execution unit indicates that the corresponding query fragment has been executed, the execution operation of other execution units can be terminated and their corresponding computing resources can be released. Then, the execution result output by the target execution unit is determined as the execution result of the current execution step.

[0156] In some embodiments, the detection of the execution progress of each query shard in multiple execution units that are executed synchronously and in parallel and the determination of the execution result of the query step can be specifically implemented by a group execution operator (group polling).

[0157] like Figure 4 As shown, it is a schematic diagram of the processing process of the query request in the present disclosure. As shown in the figure, after the parser 310 parses the acquired query request to obtain the corresponding abstract syntax tree, the grouping plan generator 410 generates a grouping plan according to the corresponding abstract syntax tree, the metadata of the business data in the database, and the statistical information. Among them, the grouping plan includes multiple query steps and the execution order between the query steps, each query step includes a corresponding query sharding group, and each query sharding group includes multiple query shards with the same query function as the query step. When the grouping plan executor 420 executes the query task according to the grouping plan and the business data, it executes each query step one by one according to the execution order between the query steps in the grouping plan. When executing a certain query step, it first determines whether the query step has multiple query shards with the same query function as the query step. When the query step has multiple query shards, the multiple query shards corresponding to the query step are simultaneously scheduled to different execution units for synchronous parallel execution. When executing multiple query shards synchronously and in parallel, the group plan executor 420 can perform real-time detection on the execution process of different query shards in multiple execution units. When it is detected that the execution of the target query shard in a certain target execution unit is completed, the execution process of other execution units is terminated, and the execution result output by the target execution unit is input as the execution result of the query step into the next query step for execution until the final query result corresponding to the query request is obtained.

[0158] like Figure 5As shown, it is a schematic diagram of the grouping plan generated in the query request processing method provided by the present disclosure. As shown in the figure, the grouping plan includes query step one 510, query step two 520 and query step three 530, and each query step includes multiple query shards 511. When executing these plans, the grouping plan executor 420 executes the query steps one by one according to the execution order between the query steps. When executing each query step, the grouping plan executor 420 simultaneously schedules the multiple query shards corresponding to the query step to multiple execution units for parallel execution. When any of the target query shards is executed, the execution of other query shards is terminated, and the execution result of the target query shard is output to the next query step as the execution result of the query step.

[0159] like Figure 6 As shown, it is a schematic diagram of the process of executing the grouping plan in the present disclosure. As shown in the figure, when a query step in the grouping plan is executed, the grouping execution operator 610 simultaneously dispatches multiple query slices corresponding to the query step to different execution units for parallel execution. As shown in the figure, the query slice may include one execution step or multiple execution steps, and the multiple execution steps in different query slices may also have different execution order relationships. During the execution process of each query slice, the execution progress will be fed back to the grouping execution operator 610 in real time. When the grouping execution operator 610 detects that any target query slice in the multiple query slices executed in parallel is completed, the execution of the query slices in other execution units is terminated, and the computing resources of each execution unit are released, and the execution result of the target query slice is output as the execution result of the query step.

[0160] Step 250: determining a target query result corresponding to the query request according to the execution result corresponding to each query step and the execution order.

[0161] Among them, for each query step in the query plan, if it has multiple query shards, the query request processing method provided by the present disclosure is used to perform multi-threaded synchronous execution on the multiple query shards corresponding to each query step, and the execution result of the query shard that obtains the execution result the fastest is used as the execution result of the query step. The execution result obtained by executing each query step can be circulated according to the execution order between the query steps in the query plan until the last query step is executed and the target query result corresponding to the query request is obtained.

[0162] In summary, the query request processing method provided by the present invention is adopted, by obtaining a query request and parsing the query request, a query plan is obtained, and the query plan includes multiple query steps and the execution order between the multiple query steps; multiple query slices corresponding to each query step are generated, and the multiple query slices corresponding to the same query step realize the same query function as the corresponding query step when they are executed; when each query step is executed, the multiple query slices corresponding to each query step are simultaneously scheduled to different execution units for parallel execution; the execution result of each query step is determined according to the execution result of the target query slice corresponding to each query step, and the target query slice is the query slice that is executed first among the multiple query slices executed in parallel; the target query result corresponding to the query request is determined according to the execution result corresponding to each query step and the execution order.

[0163] Thus, the query request processing method provided by the present disclosure generates multiple query fragments corresponding to each query step, and then when executing each query step, the multiple query fragments corresponding to the query step to be executed are simultaneously dispatched to multiple different execution units for parallel execution, and the execution result of the query step is determined according to the execution result of the query fragment that is executed the fastest. In this way, it can be ensured that each query step can be executed using the fastest execution method, thereby greatly improving the processing efficiency of the query request.

[0164] The embodiments of the present disclosure are described in detail in combination with specific application scenarios.

[0165] like Figure 7 FIG. 1 is another flow chart of the query request processing method provided by the present disclosure. The method will be described in detail by taking the query request processing method applied in a data warehouse as an example. The method specifically includes the following steps:

[0166] Step 701: The data warehouse server receives a query request sent by a terminal.

[0167] Among them, the data warehouse is a database that provides various types of data support for the enterprise-level decision-making process. The data warehouse is presented to the object in the form of an application, providing various types of analysis and decision support services for the enterprise-level objects. The object can access the data warehouse through the client of the data warehouse application loaded in the terminal and send a query request to the server of the data warehouse. After receiving the query request sent by the object through the client loaded in the terminal, the data warehouse server can perform various query operations such as access, call, and analysis on the corresponding data in the data warehouse according to the query request, and then return the query results to the terminal for display.

[0168] In the related art, after receiving a query request sent by a client loaded in a terminal, the data warehouse server can parse the query request and generate an optimal query plan, and then perform corresponding query operations on the business data stored in the data warehouse based on the query plan, and return the query results. However, the statistical data relied on when generating the query plan in the related art will change over time, resulting in the generated query plan being no longer the optimal query plan when it is executed due to the time difference between the plan generation time and the plan execution time, which will lead to low query request processing efficiency.

[0169] In order to solve the problem that there is a time difference between the query plan generation time and the plan execution time, the statistical information changes and the executed plan is not the optimal plan, resulting in low query request execution efficiency, this solution provides a solution for generating a grouping plan and determining the optimal execution plan in the plan execution stage to improve the query request processing efficiency. The solution is described in detail below.

[0170] In the disclosed embodiment, similarly, the data warehouse server corresponding to the data warehouse application receives the query request sent by the object through the terminal, and then starts to execute the corresponding query operation based on the query request. Specifically, the data warehouse server can receive the query request sent by the terminal through the data transceiver unit, and then send the received query request to the parser for processing.

[0171] Step 702: The parser parses the received query request to obtain an abstract syntax tree.

[0172] After receiving the query request sent by the data transceiver unit, the parser in the data warehouse server can parse the query request. Similar to the related art, the parsing of the query request can specifically include lexical analysis, syntax analysis and semantic analysis of the SQL statement in the query request to obtain the abstract syntax tree corresponding to the query request. This process has been described in detail in the above example and will not be repeated here.

[0173] Step 703: The statistical information collector collects statistical information of the business data in the data warehouse.

[0174] Among them, the data warehouse server also includes a statistical information collector, which can collect statistical information of business data in the data warehouse in real time or intermittently. The statistical information of business data specifically includes data table information and index information, etc.; the data table information can include the total number of tuples, tuple length, number of occupied fast-ripening blocks and number of occupied overflow blocks of each data table, etc. In addition, it can also include column information of each data table, including the number of different values ​​of each column, the maximum and minimum values ​​of the column, whether an index has been established on the column, and what kind of index has been established. Index information can include the number of index levels, the number of different index values, the selection cardinality of the index, and the number of leaf nodes of the index. The specific collection process of statistical information is also described in detail in the aforementioned example, which is consistent with the collection process of statistical information of business data in the data warehouse in the relevant technology, and will not be repeated here.

[0175] Step 704: The grouping plan generator generates a grouping query plan according to the abstract syntax tree, statistical information, and metadata of the business data.

[0176] After the parser parses the query request to obtain an abstract syntax tree, and the statistical information collector collects the statistical information of the business data in the data warehouse to obtain statistical information, the parsed abstract syntax tree and the collected statistical information are sent to the grouping plan generator for plan generation. The grouping plan generator can further obtain the metadata of the business data in the data warehouse, and then generate a grouping query plan based on the abstract syntax tree, metadata and statistical information. The grouping query plan contains multiple query sharding groups, each query sharding group contains multiple plan fragments with the same function, and these plan fragments can be called query shards. The multiple query shards contained in the same query sharding group have a unique output to the outside, that is, the multiple query shards contained in the same query sharding group implement the same query function.

[0177] That is, in the disclosed embodiment, when generating a query plan, the plan generator does not screen the query plan to determine the optimal query plan, but traverses all query plans that can achieve the goal corresponding to the query request. Each query plan is composed of multiple query shards, and query shards with the same functions in the query shards are divided into the same query shard group. This method can avoid mistakenly screening out the optimal query plan (the most efficient query plan) in the plan generation stage, but instead generates all query plans, and then executes the query with the most efficient execution plan in the execution stage, thereby greatly improving the processing efficiency of query requests.

[0178] Step 705: When executing any query shard group in the group query plan, the group plan executor uses the group execution scheduler to schedule multiple query shards in the query shard group to different threads.

[0179] After the group plan generator generates a group query plan based on the abstract syntax tree, statistical information, and metadata of the business data, the group query plan can be output to the group plan executor to execute the query. Specifically, when executing the group query plan, specifically when executing any query shard group in the group query plan, the group plan executor can first initialize the operating environment and state of the query shard group, and then allocate corresponding equipment or process / thread resources to each query shard in the query shard group for operation. In this embodiment, a corresponding thread can be specifically allocated to each query shard for operation.

[0180] Step 706: The group execution scheduler controls different threads to execute multiple query fragments simultaneously.

[0181] Among them, after the group execution scheduler allocates a corresponding thread to each query shard in the query shard group, it can control different threads to run multiple query shards in parallel at the same time.

[0182] Different from the related art, after generating a group query plan, the query request processing method provided by the present invention does not run a single query plan, but rather performs multi-threaded synchronous running on multiple query shards in the query shard group in the group query plan.

[0183] Step 707: During the simultaneous execution of multiple query fragments, the group execution scheduler detects the execution progress of the query fragment in each thread in real time.

[0184] When multiple query shards in a query shard group are synchronously executed in multiple different threads, the group execution scheduler can detect the execution progress of the query shards in each thread in real time.

[0185] Step 708: When it is detected that the execution of the query slice in any target thread is completed, the group execution scheduler terminates the execution of the query slice in all threads and releases the computing resources of the threads.

[0186] When the group execution scheduler detects that one of the multiple query shards in the query shard group has completed execution, it can determine that the query shard group has completed execution. At this time, the execution operation of the threads executing each query shard in the query shard group can be terminated, and the corresponding software and hardware resources can be released so that multiple threads can be put into the execution of the next query shard group. This can also save the computing power resources consumed by execution and further improve the execution efficiency of query requests.

[0187] Step 709: The group execution operator uses the execution result of the query shard in the target thread as the execution result of the query shard group.

[0188] At the same time, the group execution operator can use the execution results of the query shards in the target thread as the execution results of the query shard group.

[0189] like Figure 8 As shown, it is a schematic diagram of the process of the group execution operator determining the execution result of the query shard group. Among them, as shown in the figure, the group execution operator can be implemented based on the queue and semaphore mechanism. Specifically, an execution result queue 830 can be defined, and the content stored in the queue is (shard id, execution result data), and the group execution operator 610 waits for the data in the execution result queue 830. The first thread 810 executes query shard 1, and the second thread 820 executes query shard 2. No matter which thread is completed first, the corresponding shard id and execution result data can be sent to the corresponding position in the execution result queue 830 for storage. When the group execution operator 610 detects that data is received in the execution result queue 830, it can be determined that a thread has completed the execution of the corresponding query shard. At this time, the group execution operator can terminate the execution process in other threads and release the software and hardware resources of each thread. At the same time, the group execution operator 610 can obtain the corresponding execution result data from the execution result queue 830 for subsequent processing.

[0190] Step 710 , the group plan executor inputs the execution result of the query shard group to the next query shard group for further query, until the target query result corresponding to the query request is output.

[0191] Among them, for any query shard group, the group plan executor can use the group execution operator to schedule the multiple query shards contained therein to different threads for synchronous execution, and obtain the unique execution result output by the query shard group until the last query shard group is executed and the target query result corresponding to the query request is output.

[0192] like Fig. 9As shown, it is a timing flow chart of the execution of the group query plan provided by the present disclosure. As shown in the figure, first, the group plan generator 410 generates a group query plan according to the statistical information, the AST corresponding to the query request, and the metadata of the business data. Then, the group plan generator 410 sends the generated group query plan to the group execution scheduler 910. The group execution scheduler 910 schedules the query shard 1 in a query shard group in the group query plan to the first thread 810 for execution, and at the same time schedules the query shard 2 in the query shard group to the second thread 820 for execution. The first thread 810 and the second thread 820 simultaneously execute the execution operations on the query shard 1 and the query shard 2. If the first thread 810 is executed first, and the execution result of the query shard 1 is output to the group execution operator 610. At the same time, the group execution scheduler 910 sends a termination execution instruction to the second thread 820 to terminate the execution of the query shard 2. After receiving the termination execution instruction sent by the group execution scheduler 910, the second thread 820 terminates the execution operation of the query shard 2 being executed therein. After receiving the execution result of query shard 1 sent by the first thread, the group execution operator outputs the execution of query shard 1 as the execution result of the query shard group.

[0193] Among them, when the query request processing solution provided by the present disclosure is applied in the OLAP query scenario, in the OLAP application scenario, some pre-calculated mechanisms, such as AGG, can be used to improve the query performance. However, as mentioned above, with the addition, deletion and modification of data, the pre-calculated AGG will also be updated at any time, and its results may be in an invalid state. When using the traditional solution, the pre-calculated result cannot support the query when it is in an invalid state, and the AGG result can only be calculated at runtime, which will consume a lot of resources and time; using the solution provided by the present disclosure, it will detect whether the current AGG status is available at runtime. If available, the pre-calculated AGG can be directly used as the query result as a query shard selection, so that resource consumption will be minimized and performance will be greatly improved.

[0194] When the query request processing solution provided by the present disclosure is applied in an OLTP scenario or a HTAP scenario, since in the OLTP / HTAP application scenario, the query performance can be improved through the index, but similar to the previous scenario, as the data is updated, the index also needs to be updated in real time. At the moment of query execution, the index may also be in an unavailable state. For TP scenarios, the correctness of the results must be guaranteed. If the solution in the related art is used, when the index is unavailable, it can only fall back to the full table scan solution; using the solution provided by the present disclosure, the index scan will be attempted to be executed at the same time as the query shard at runtime. If the current index is in an available state, the query shard will be executed quickly and the result will be returned, avoiding the full table scan, and the query performance will be greatly improved.

[0195] When the query request processing solution provided in the present disclosure is applied in the artificial intelligence prediction query scenario, since some data warehouses support AI models, AI models can be used for prediction through queries. As the underlying data changes, the AI ​​model will also change accordingly. This feature is called concept drift in the AI ​​field. In the solutions of related technologies, when encountering such a situation, either an inaccurate model is used or the query cannot be made; using the solution provided in the present disclosure, the prediction results of the best model can be obtained dynamically and in real time at runtime.

[0196] The query request processing method provided by the present disclosure, after generating all query shards that can realize the query target of the query request according to the parsing results of the query request and the statistical data of the business data, divides the query shards with the same function into a query shard group to obtain a group query plan. When executing the group query plan, the query shards in the same query shard group are scheduled to different processes for simultaneous execution, and then the output of the query shard group is determined by the query shard that obtains the execution result the fastest, and the execution efficiency of the query shard group is taken as the fastest execution result. In this way, the fastest query execution can be achieved for any query shard group, thereby greatly improving the processing efficiency of query requests. The specific instructions are as follows:

[0197] Assume that there are n query shards with the same function in a query shard group, denoted as PF i (i∈[1,n]), where the actual execution time of the i-th shard is t i , where max(t i ), min(t i ) represent the maximum and minimum actual execution time of n query shards, respectively. In the related scheme, one shard is selected for execution in the planning phase, so the actual execution time t of the shard satisfies min(t i )≤t≤max(t i ). However, when the solution provided by the present disclosure is adopted, the execution result of the fastest executed shard will be returned in actual execution, so the actual execution time t of the query shard group satisfies t=min(t i ). Assume that a query plan has m groupable shards, and its execution time is the execution time T of the m query shards grouped together. PF and the execution time T of other query shards c From the above analysis, we can see that the total time consumed by using the relevant scheme to execute the query request is:

[0198]

[0199] The total query time of the solution adopted by this disclosure is:

[0200] From the above comparison, it can be seen that the solution provided by the present disclosure can reduce the total time consumption of query requests, that is, the processing efficiency of query requests is greatly improved.

[0201] Description of the apparatus and device of the present disclosure

[0202] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0203] It should be noted that in each specific embodiment of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with the relevant laws, regulations and standards of the relevant region. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present application will be obtained.

[0204] Fig.10 The structure diagram of the query request processing device 1000 provided in the embodiment of the present disclosure. The query request processing device 1000 includes:

[0205] The acquisition unit 1010 is used to acquire a query request, and parse the query request to obtain a query plan, where the query plan includes multiple query steps and an execution order between the multiple query steps;

[0206] A generating unit 1020, configured to generate a plurality of query fragments corresponding to each query step, wherein the plurality of query fragments corresponding to the same query step realize the same query function as the corresponding query step when being executed;

[0207] A scheduling unit 1030 is used to schedule multiple query fragments corresponding to each query step to different execution units for parallel execution when executing each query step;

[0208] A first determining unit 1040 is used to determine the execution result of each query step according to the execution result of the target query shard corresponding to each query step, where the target query shard is the query shard that is executed first among the multiple query shards executed in parallel;

[0209] The second determining unit 1050 is used to determine the target query result corresponding to the query request according to the execution result and execution order corresponding to each query step.

[0210] Optionally, in some embodiments, the first determining unit includes:

[0211] A detection subunit, used to detect the execution progress of each query shard during the parallel execution of multiple query shards corresponding to each query step;

[0212] The determination subunit is used to determine the execution result of the corresponding query step according to the execution result of the target query shard among the multiple query shards when it is detected that the execution of the target query shard is completed.

[0213] Optionally, in some embodiments, the query request processing device provided by the present disclosure further includes:

[0214] A termination subunit, used to terminate the execution process of other query shards except the target query shard among the multiple query shards;

[0215] The release subunit is used to release the computing resources of different execution units.

[0216] Optionally, in some embodiments, the detection subunit is further used for:

[0217] During the parallel execution of multiple query shards corresponding to each query step, obtaining execution status information sent by different execution units;

[0218] Identify subunits. Also used for:

[0219] When it is detected that the execution status information sent by the target execution unit indicates that the execution of the corresponding query slice is completed, the execution result of the corresponding query step is determined according to the execution result of the target execution unit on the corresponding target query slice.

[0220] Optionally, in some embodiments, the acquiring unit includes:

[0221] A first acquisition subunit is used to acquire a query request, where the query request includes a structured query statement;

[0222] The parsing subunit is used to parse the structured query statement and obtain the corresponding abstract syntax tree;

[0223] The first generating subunit is used to generate a query plan according to the abstract syntax tree.

[0224] Optionally, in some embodiments, the first generating subunit includes:

[0225] A first generation module, used to generate multiple candidate query plans according to the abstract syntax tree;

[0226] A calculation module, used to calculate the query resource consumption corresponding to each candidate query plan;

[0227] The first determination module is used to determine a query plan from multiple candidate query plans according to the query resource consumption.

[0228] Optionally, in some embodiments, the first determining module includes:

[0229] A sorting submodule, used to sort multiple candidate query plans in order from low to high according to the query resource consumption;

[0230] The determination submodule is used to determine that the candidate query plan ranked first is the generated query plan.

[0231] Optionally, in some embodiments, the generating unit comprises:

[0232] A second acquisition subunit is used to acquire the query function corresponding to each query step;

[0233] The third acquisition subunit is used to obtain statistical information and metadata of business data in the target database;

[0234] The second generation subunit is used to generate multiple query fragments that implement each query function according to the statistical information and metadata.

[0235] Optionally, in some embodiments, the second generating subunit includes:

[0236] A second determination module is used to determine a combination of multiple query operators used to implement each query function according to the statistical information and metadata;

[0237] The second generation module is used to generate multiple query fragments according to a combination of multiple query operators.

[0238] Optionally, in some embodiments, the second generating subunit includes:

[0239] A detection module is used to perform multi-query shard detection on each query function based on statistical information and metadata;

[0240] The third generating module is used to generate multiple query shards that implement each query function according to the statistical information and metadata when the detection result of the multiple query shard detection indicates that the query function can be implemented by multiple query shards.

[0241] Optionally, in some embodiments, the scheduling unit includes:

[0242] A fourth acquisition subunit, used to acquire execution capability information of each execution unit;

[0243] The scheduling subunit is used to schedule multiple query fragments corresponding to each query step to different execution units for parallel execution according to the execution capability information when executing each query step.

[0244] The query request processing device provided by the embodiment of the present disclosure obtains the query request through the acquisition unit 1010, and parses the query request to obtain a query plan, wherein the query plan includes multiple query steps and the execution order between the multiple query steps; the generation unit 1020 generates multiple query slices corresponding to each query step, and the multiple query slices corresponding to the same query step realize the same query function as the corresponding query step when executed; when executing each query step, the scheduling unit 1030 schedules the multiple query slices corresponding to each query step to different execution units for parallel execution at the same time; the first determination unit 1040 determines the execution result of each query step according to the execution result of the target query slice corresponding to each query step, and the target query slice is the query slice that is executed first among the multiple query slices executed in parallel; the second determination unit 1050 determines the target query result corresponding to the query request according to the execution result corresponding to each query step and the execution order.

[0245] Thus, the query request processing device provided by the present disclosure generates multiple query fragments corresponding to each query step, and then when executing each query step, simultaneously dispatches the multiple query fragments corresponding to the executed query step to multiple different execution units for parallel execution, and determines the execution result of the query step according to the execution result of the query fragment that is executed the fastest. In this way, it can be ensured that each query step can be executed using the fastest execution method, thereby greatly improving the processing efficiency of the query request.

[0246] Reference Fig.11 , Fig.11The structural block diagram of the terminal 140 for implementing the query request processing method of the embodiment of the present disclosure is as follows: the terminal 140 includes: a radio frequency (RF) circuit 1110, a memory 1111, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WiFi) module 1170, a processor 1180, and a power supply 1190. Those skilled in the art can understand that Fig.11 The structure of the terminal 140 shown does not constitute a limitation on a mobile phone or a computer, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0247] The RF circuit 1110 may be used for receiving and sending signals during information transmission or communication. In particular, after receiving downlink information from the base station, the information is sent to the processor 1180 for processing. In addition, the uplink data is sent to the base station.

[0248] The memory 1111 may be used to store software programs and modules. The processor 1180 executes various functional applications and query request processing of the terminal by running the software programs and modules stored in the memory 1111 .

[0249] The input unit 1130 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1130 may include a touch panel 1131 and other input devices 1132 .

[0250] The display unit 1140 may be used to display input information or provided information and various menus of the terminal. The display unit 1140 may include a display panel 1141 .

[0251] The audio circuit 1160 , the speaker 1161 , and the microphone 1162 may provide an audio interface.

[0252] In this embodiment, the processor 1180 included in the terminal 140 can execute the query request processing method of the previous embodiment.

[0253] The terminal 140 of the embodiment of the present disclosure includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The embodiment of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0254] Fig.12A block diagram of the structure of a portion of a server 110 for implementing the query request processing method of an embodiment of the present disclosure. The server 110 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1222 (for example, one or more processors) and storage devices 1232, and one or more storage media 1230 (for example, one or more mass storage devices) storing application programs 1242 or data 1244. Among them, the storage device 1232 and the storage medium 1230 may be temporary storage or permanent storage. The program stored in the storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server 110. Furthermore, the central processing unit 1222 may be configured to communicate with the storage medium 1230 to execute a series of instruction operations in the storage medium 1230 on the server 110.

[0255] The server 110 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258, and / or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0256] The central processor 1222 in the server 110 may be used to execute the query request processing method of the embodiment of the present disclosure.

[0257] The embodiments of the present disclosure further provide a storage medium, which is used to store program codes, and the program codes are used to execute the query request processing methods of the aforementioned embodiments.

[0258] The embodiment of the present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the query request processing method.

[0259] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0260] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0261] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0262] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0263] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0264] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0265] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0266] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0267] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A query request processing method, characterized in that: The method comprises: Obtaining a query request, and parsing the query request to obtain a query plan, wherein the query plan includes a plurality of query steps and an execution order between the plurality of query steps; Generate a plurality of query fragments corresponding to each of the query steps, wherein the plurality of query fragments corresponding to the same query step realize the same query function as the corresponding query step when executed; When executing each of the query steps, the multiple query fragments corresponding to each of the query steps are simultaneously scheduled to different execution units for parallel execution; Determine the execution result of each query step according to the execution result of the target query slice corresponding to each query step, wherein the target query slice is the query slice that is executed first among the multiple query slices executed in parallel; The target query result corresponding to the query request is determined according to the execution result corresponding to each query step and the execution order.

2. The method according to claim 1, characterized in that Determining the execution result of each query step according to the execution result of the target query slice corresponding to each query step includes: During the parallel execution of the multiple query slices corresponding to each of the query steps, detecting the execution progress of each query slice; When it is detected that the execution of a target query slice among the multiple query slices is completed, the execution result of the corresponding query step is determined according to the execution result of the target query slice.

3. The method according to claim 2, characterized in that When it is detected that the target query slice among the multiple query slices has been executed, after determining the execution result of the corresponding query step according to the execution result of the target query slice, the method further includes: terminating the execution process of other query shards among the multiple query shards except the target query shard; The computing resources of the different execution units are released.

4. The method according to claim 2, characterized in that: During the parallel execution of the multiple query fragments corresponding to each query step, detecting the execution progress of each query fragment comprises: During the parallel execution of the plurality of query fragments corresponding to each of the query steps, obtaining the execution status information sent by the different execution units; When it is detected that the execution of the target query slice among the multiple query slices is completed, determining the execution result of the corresponding query step according to the execution result of the target query slice, includes: When it is detected that the execution status information sent by the target execution unit indicates that the execution of the corresponding query slice is completed, the execution result of the corresponding query step is determined according to the execution result of the target execution unit on the corresponding target query slice.

5. The method according to claim 1, characterized in that The obtaining of the query request and parsing the query request to obtain a query plan includes: Obtaining a query request, wherein the query request includes a structured query statement; Parsing the structured query statement to obtain a corresponding abstract syntax tree; A query plan is generated according to the abstract syntax tree.

6. The method according to claim 5, characterized in that Generating a query plan according to the abstract syntax tree includes: Generate multiple candidate query plans according to the abstract syntax tree; Calculate the query resource consumption corresponding to each candidate query plan; A query plan is determined from among the multiple candidate query plans according to the query resource consumption.

7. The method according to claim 6, characterized in that Determining a query plan from among the multiple candidate query plans according to the query resource consumption includes: Sorting the plurality of candidate query plans according to the query resource consumption in ascending order; The candidate query plan ranked first is determined to be the generated query plan.

8. The method according to claim 1, characterized in that The generating of a plurality of query fragments corresponding to each of the query steps includes: Obtaining the query function corresponding to each of the query steps; Obtain statistical information and metadata of business data in the target database; A plurality of query fragments implementing each of the query functions are generated according to the statistical information and the metadata.

9. The method according to claim 8, characterized in that The generating a plurality of query fragments implementing each of the query functions according to the statistical information and the metadata includes: Determine, according to the statistical information and the metadata, a combination of multiple query operators used to implement each of the query functions; A plurality of query fragments are generated according to a combination of the plurality of query operators.

10. The method according to claim 8, characterized in that The generating a plurality of query fragments implementing each of the query functions according to the statistical information and the metadata includes: Performing multi-query shard detection on each query function according to the statistical information and the metadata; When the detection result of the multiple query shard detection indicates that the query function can be implemented by multiple query shards, multiple query shards that implement each of the query functions are generated according to the statistical information and the metadata.

11. The method according to any one of claims 1 to 10, characterized in that When executing each of the query steps, the multiple query fragments corresponding to each of the query steps are simultaneously scheduled to different execution units for parallel execution, including: Obtaining execution capability information of each execution unit; When executing each of the query steps, the multiple query fragments corresponding to each of the query steps are simultaneously scheduled to different execution units for parallel execution according to the execution capability information.

12. A query request processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a query request, and parse the query request to obtain a query plan, wherein the query plan includes a plurality of query steps and an execution order between the plurality of query steps; A generating unit, configured to generate a plurality of query fragments corresponding to each of the query steps, wherein the plurality of query fragments corresponding to the same query step realize the same query function as the corresponding query step when executed; A scheduling unit, configured to schedule the multiple query fragments corresponding to each query step to different execution units for parallel execution when executing each query step; A first determining unit, configured to determine an execution result of each query step according to an execution result of a target query shard corresponding to each query step, wherein the target query shard is a query shard that is executed first among a plurality of query shards executed in parallel; The second determining unit is used to determine the target query result corresponding to the query request according to the execution result corresponding to each query step and the execution order.

13. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the query request processing method according to any one of claims 1 to 11 is implemented.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the query request processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the query request processing method according to any one of claims 1 to 11.

Images