Engine selection method and device and computer storage medium

Abstract

The invention discloses an engine selection method and device and a computer storage medium, and the method comprises the steps: generating a plurality of to-be-selected engine strategies through responding to at least one piece of query request information; for each to-be-selected engine, determining a quantitative value based on a quantitative standard of the to-be-selected engine and at least one piece of index data of the query request information corresponding to the to-be-selected engine, and determining a comprehensive performance score of the to-be-selected engine based on the quantitative value; for each to-be-selected engine strategy, determining a fitness value for representing query performance based on the comprehensive performance score of each to-be-selected engine in the to-be-selected engine strategy; and determining a target engine strategy based on the fitness value. That is to say, the target engine strategy is determined by calculating the fitness values of the multiple generated to-be-selected engine strategies, the multiple query tasks are processed by the multiple engine strategies at the same time based on the target engine strategy, overload of a single query engine is avoided, and the stability and processing efficiency of the big data platform are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an engine selection method, device, and computer storage medium. Background Art

[0002] With the explosive growth of data volume and the diversification of business needs, different query engines have their own advantages in processing different types of tasks.

[0003] However, in current applications, existing big data systems typically use a single query engine to handle all SQL queries. This results in a failure to fully leverage the strengths of different query engines, making it difficult to optimize query performance and utilize resources effectively. Furthermore, when the data volume is large or the query logic is complex, query engine overload can lead to long execution times and high resource consumption, compromising the stability and overall efficiency of the big data platform. Summary of the Invention

[0004] The present invention provides an engine selection method, device and computer storage medium to solve the problem in the prior art that a single query engine processes all query tasks, resulting in query engine overload and affecting the stability and overall efficiency of a big data platform.

[0005] In a first aspect, an embodiment of the present application provides an engine selection method, the method comprising:

[0006] In response to at least one query request information, generating a plurality of strategies for engines to be selected, wherein the strategies for engines to be selected include a first correspondence between at least one engine to be selected and the query request information;

[0007] For each engine to be selected, based on the quantitative criteria of the engine to be selected and at least one indicator data of the query request information corresponding to the engine to be selected, a quantitative value of each indicator data is determined, and a comprehensive performance score of the engine to be selected is determined based on the quantitative value, wherein the quantitative value is used to represent the complexity of the query request information, and the comprehensive performance score is used to represent the query performance of the engine to be selected;

[0008] For each engine strategy to be selected, determining a fitness value for characterizing the query performance of the engine strategy to be selected based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected;

[0009] A target engine strategy is determined from the engine strategies to be selected based on the fitness value.

[0010] In some optional implementations, determining the comprehensive performance score of the engine to be selected based on the quantified value includes:

[0011] Determining a first weight coefficient corresponding to a quantized value of each indicator data based on a preset second correspondence between the to-be-selected engine and the first set of weight coefficients;

[0012] The comprehensive performance score is determined based on the quantized values ​​of the indicator data and the first weight coefficients corresponding to the respective indicators.

[0013] In some optional implementations, determining a fitness value for characterizing the query performance of the engine strategy to be selected based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected includes:

[0014] Determining, based on a preset third correspondence between the strategy for the engine to be selected and the second set of weight coefficients, a second weight coefficient corresponding to the comprehensive performance score of each engine to be selected in the strategy for the engine to be selected;

[0015] The fitness value is determined based on the comprehensive performance score of each of the engines to be selected and the second weight coefficient corresponding to each.

[0016] In some optional implementations, after determining, for each engine strategy to be selected, a fitness value for characterizing the query performance of the engine strategy to be selected based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected, the method further includes:

[0017] Determining a first sub-target engine strategy to be selected from the engine strategies to be selected based on the fitness value;

[0018] Performing genetic transformation on the first sub-goal to-be-selected engine strategy to obtain the second sub-goal to-be-selected engine strategy;

[0019] Based on the second sub-target engine strategy to be selected, the multiple engine strategies to be selected are updated, and the step of determining the quantitative value of each indicator data is returned until the change between the current fitness value and the last fitness value obtained is less than a preset threshold, and the most recently obtained second sub-target engine strategy to be selected is used as the target engine strategy.

[0020] In some optional implementations, determining a first sub-target engine strategy to be selected from the engine strategies to be selected based on the fitness value includes:

[0021] The fitness values ​​of each engine strategy to be selected are summed up to obtain a first sum value;

[0022] For each engine strategy to be selected, calculating a quotient of the fitness value of the engine strategy to be selected divided by the first sum value;

[0023] Based on the quotient value corresponding to each engine strategy to be selected, the first sub-goal engine strategy to be selected is determined.

[0024] In some optional implementations, determining the first sub-goal engine strategy to be selected based on the quotient value corresponding to each engine strategy to be selected includes:

[0025] The engine strategy to be selected corresponding to the maximum quotient value is used as the first sub-target engine strategy to be selected;

[0026] Or, determining the engine strategy to be selected for the first sub-goal based on a first absolute value of a difference between the maximum quotient and the random number, and a second absolute value of a difference between the second sum and the random number;

[0027] The second sum value is used to represent the sum of the maximum quotient value and other quotient values ​​except the maximum quotient value.

[0028] In some optional implementations, performing a genetic transformation on the first sub-goal to-be-selected engine strategy to obtain the second sub-goal to-be-selected engine strategy includes:

[0029] Performing a cross transformation based on the first sub-target engine strategy to be selected and the third sub-target engine strategy to be selected determined from the engine strategies to be selected to obtain a child engine strategy;

[0030] The child engine strategy is mutated and transformed to obtain the second sub-target engine strategy to be selected.

[0031] In some optional implementations, updating the multiple to-be-selected engine strategies based on the second sub-goal to-be-selected engine strategy includes:

[0032] The second sub-target to-be-selected engine replaces the to-be-selected engine strategy corresponding to the minimum quotient value to obtain multiple updated to-be-selected engine strategies;

[0033] Or, add the second sub-target to-be-selected engine to the

[0034] In a second aspect, an embodiment of the present application provides an engine selection device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a method as described in any one of the first aspects.

[0035] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method as described in any one of the first aspects.

[0036] The beneficial effects of the present invention are as follows:

[0037] The present invention provides an engine selection method, device, and computer storage medium. The method generates multiple engine selection strategies in response to at least one query request information, wherein the engine selection strategy includes a first correspondence between at least one engine to be selected and the query request information; for each engine to be selected, based on the quantitative criteria of the engine to be selected and at least one indicator data of the query request information corresponding to the engine to be selected, determines a quantitative value of each indicator data, and determines a comprehensive performance score of the engine to be selected based on the quantitative value, wherein the quantitative value is used to characterize the complexity of the query request information, and the comprehensive performance score is used to characterize the query performance of the engine to be selected; for each engine selection strategy, based on the comprehensive performance score of each engine to be selected in the engine selection strategy, determines a fitness value used to characterize the query performance of the engine selection strategy; and determines a target engine strategy from the engine selection strategy based on the fitness value. In other words, the present invention determines a target engine strategy by calculating the fitness values ​​of the multiple engine selection strategies generated, and implements multiple engine strategies to process multiple query tasks simultaneously based on the target engine strategy, thereby avoiding overloading of a single query engine and improving the stability and processing efficiency of the big data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of an application scenario of an engine selection method provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of an engine selection method provided in an embodiment of the present application;

[0041] Figure 3 A flowchart of another engine selection method provided in an embodiment of the present application;

[0042] Figure 4 A flowchart of another engine selection method provided in an embodiment of the present application;

[0043] Figure 5 A flowchart of another engine selection method provided in an embodiment of the present application;

[0044] Figure 6A flowchart of another engine selection method provided in an embodiment of the present application;

[0045] Figure 7 A structural diagram of an engine selection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0047] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. The character " / " generally indicates that the objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where each of a, b, c can be an element itself, or a set containing one or more elements.

[0048] Throughout this application, the terms "exemplary," "in some embodiments," and "in other embodiments" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0049] It should be pointed out that the words "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0050] The following is an explanation of some technical terms involved in this application.

[0051] 1) SQL (Structured Query Language) query: is a special-purpose programming language used to access and manage data in relational databases. SQL queries allow users to perform a series of operations such as creating databases, defining table structures, inserting data, querying data, updating data, deleting data, and controlling access rights to databases.

[0052] 2) Hive query engine: It is a Hadoop-based data warehouse tool used to extract, transform, and load large-scale data sets stored in the Hadoop distributed file system.

[0053] 3) Flink query engine: It is an open-source, high-speed, real-time data stream processing and batch data processing system for distributed data.

[0054] 4) StarRocks Query Engine: It is a high-performance analytical database that combines MPP (Massively Parallel Processing) and OLAP (Online Analytical Processing) technologies to quickly process massive amounts of data and support high-performance multidimensional analysis and complex data query operations.

[0055] With the explosive growth of data volumes and the diversification of business needs, different query engines have their own advantages when handling different types of tasks. For example, the Hive query engine is suitable for large-scale batch data processing, the Flink query engine is suitable for real-time data stream processing, and the StarRocks query engine is suitable for real-time data analysis and interactive queries.

[0056] However, in current practical applications, existing big data systems typically choose a single query engine to handle all SQL query tasks. This results in a failure to fully leverage the strengths of different query engines during query processing, making it difficult to optimize query performance and rationally utilize resources. For example, the Hive query engine significantly increases execution time when processing large-scale, complex queries, Flink experiences increased processing latency when facing massive, complex streaming queries, and StarRocks also faces performance bottlenecks when processing extremely large, complex queries. When the data volume is excessive or the query logic is overly complex, the query engine overloads, resulting in long execution times and high resource consumption, which in turn affects the stability and overall efficiency of the big data platform.

[0057] To solve this technical problem, an embodiment of the present application provides an engine selection method, device and computer storage medium. The method generates multiple engine selection strategies in response to at least one query request information. The engine selection strategies include a first correspondence between at least one engine to be selected and the query request information. First, the quantitative values ​​of each indicator data of the query request information corresponding to the engine to be selected are determined. Then, the comprehensive performance score of the engine to be selected is determined based on the quantitative values. After that, the fitness of each engine selection strategy is determined based on the comprehensive performance score. The target engine strategy is determined from the engine selection strategies based on the fitness value, thereby enabling multiple engine strategies to process multiple query tasks simultaneously, avoiding overload of a single query engine, and thereby improving the stability and processing efficiency of the big data platform.

[0058] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0059] Figure 1 The schematic diagram exemplarily illustrates an application scenario of an engine selection method provided by an embodiment of the present application. The application scenario diagram includes a terminal device 110 and a server 120.

[0060] It should be noted that the engine selection method in the embodiments of the present application can be executed by an electronic device, which can be a server 120 or a terminal device 110. That is, the method can be executed by the server 120 or the terminal device 110 alone, or by the server 120 and the terminal device 110 together. For example, when the method is executed by the server 120 and the terminal device 110 together, the subject inputs a query request into the terminal device 110, the terminal device 110 sends the query engine to the server 120, and the server 120 generates multiple engine strategies to be selected. For each engine to be selected, based on the quantitative criteria of the engine to be selected and at least one indicator data of the query request information corresponding to the engine to be selected, the server 120 determines the quantitative value of each indicator data, and determines the comprehensive performance score of the engine to be selected based on the quantitative value. Then, for each engine strategy to be selected, the server 120 calculates the comprehensive performance score of each engine to be selected based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected. Finally, the server 120 determines a target engine strategy from the engine strategies to be selected based on the fitness value, and the server 120 feeds the target engine strategy back to the terminal device 110, so that the terminal device 110 executes the query request based on the target engine strategy.

[0061] In an optional implementation, the terminal device 110 and the server 120 may communicate via a communication network.

[0062] In an optional implementation, the communication network is a wired network or a wireless network.

[0063] It should be noted that the application scenarios of the engine selection method described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Persons skilled in the art will appreciate that, as new business scenarios emerge, the technical solutions provided by the embodiments of this application will also be applicable to similar technical problems. Furthermore, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of this application.

[0064] Based on the above Figure 1 The application scenarios and related technical introductions shown are as follows: Figure 2 A flow chart showing an engine selection method according to an embodiment of the present application is shown as follows: Figure 2 As shown, the following steps are included:

[0065] S201: Generate multiple strategies for engines to be selected in response to at least one query request, wherein the strategies for engines to be selected include a first correspondence between at least one engine to be selected and the query request;

[0066] For example, if the engines to be selected include the Hive query engine, the Flink query engine, and the StarRocks engine, and if five query request information is received, each generated engine selection strategy includes five engines to be selected. For example, using engine selection strategy 1 in the following table as an example, the Hive query engine is used to process query requests 1 and 4, the Flink query engine is used to process query requests 2 and 5, and the StarRocks query engine is used to process query request 3.

[0067] Table 1

[0068]

[0069] It should be noted that before generating multiple engine strategies to be selected, the first weight coefficient set, the second weight coefficient set and other related parameters in this application are also initialized.

[0070] S202. For each engine to be selected, based on the quantitative criteria of the engine to be selected and at least one indicator data of the query request information corresponding to the engine to be selected, determine a quantitative value of each indicator data, and determine a comprehensive performance score of the engine to be selected based on the quantitative value, wherein the quantitative value is used to represent the complexity of the query request information, and the comprehensive performance score is used to represent the query performance of the engine to be selected;

[0071] In one possible implementation, the indicator data of the query request information corresponding to the engine to be selected may include query SQL complexity, query cycle quantity, query timeliness, query response time range, and query order of magnitude. Based on the quantitative standards of the engine to be selected and the above indicator data, the quantitative value of each indicator data is determined.

[0072] For example, the quantification C of the query SQL complexity C can be determined by the number of subqueries, the number of join operations, and the number of aggregate functions. The query SQL complexity quantification C∈[0,100]. The higher the complexity of the query request information, the larger the value of the query SQL complexity quantification C, indicating that the query calculation requirement is higher. The formula is as follows:

[0073] C = w1×(number of subqueries)+w2×(number of join operations)+w3×(number of aggregate functions).

[0074] Among them, w1, w2, and w3 are pre-set weight coefficients corresponding to the number of subqueries, the number of join operations, and the number of aggregate functions, respectively. The number of subqueries, the number of join operations, and the number of aggregate functions are determined based on the query request information. The number of subqueries represents the number of subqueries in the query request information; the number of join operations represents the number of join operations used in the query request information, including inner joins and outer joins; and the number of aggregate functions represents the number of aggregate functions such as SUM, AVG, COUNT, and MAX in the query request information.

[0075] Exemplarily, the query cycle quantity is quantized P∈(1, 2, 3), where when the query request information is a real-time query, the query cycle quantity is quantized P=1, when the query request information is a near real-time query, the query cycle quantity is quantized P=2, and when the query request information is a batch query, the query cycle quantity is quantized P=3.

[0076] Exemplarily, the quantification of query timeliness is T∈(0.8, 0.5, 0.2). Based on the quantification standard of the engine to be selected, when the query request information is determined to be highly timely, T=0.8; when the query request information is determined to be of medium timeliness, T=0.5; when the query request information is determined to be of low timeliness, T=0.2.

[0077] For example, the query response time range is quantified as R∈(0-1 seconds, 1-5 seconds, 5-10 seconds). The query response time range quantification R is determined based on the query request information requirements based on the quantification criteria of the engine to be selected. 0-1 seconds indicates a fast response to the query request and is suitable for scenarios requiring timely feedback. 1-5 seconds is suitable for general business queries. 5-10 seconds is suitable for query requests with complex calculations and large data volumes.

[0078] Exemplarily, the query magnitude quantization S quantifies the magnitude of the query request information based on the quantization standard of the engine to be selected. The query magnitude quantization S∈(0.3, 0.5, 0.7), where S=0.3 when the query request information is determined to be small-scale data, S=0.5 when the query request information is determined to be medium-scale data, and S=0.7 when the query request information is determined to be large-scale data. For example, taking the Hive query engine and the Flink query engine as examples, if the data volume of query request information 1 is equivalent to small-scale data for the Hive query engine, the query magnitude quantization S for query request information 1 of the Hive query engine is 0.3. If the data volume of query request information 1 is equivalent to large-scale data for the Flink query engine, the query magnitude quantization S for query request information 1 of the Flink query engine is 0.7.

[0079] In one possible implementation, Figure 3 FIG. 1 is a flow chart of another engine selection method provided in an embodiment of the present application, and the steps are as follows:

[0080] S301: Determine a first weight coefficient corresponding to a quantized value of each indicator data based on a preset second correspondence between the engine to be selected and the first weight coefficient set;

[0081] The first weight coefficient set includes w C ,w P ,w T ,w R ,w S , where w C To query the first weight coefficient corresponding to the SQL complexity quantification C, w P The first weight coefficient corresponding to the query period quantity P, w T The first weight coefficient corresponding to the query timeliness quantification T, w R The first weight coefficient corresponding to the query response time range quantification R, w S The first weight coefficient corresponding to the query magnitude quantization S.

[0082] After determining the first weight coefficient corresponding to the quantized value of each indicator data, the process further includes normalizing each quantized value. The formula is as follows:

[0083]

[0084] Among them, X represents any one of the indicator data that needs to be normalized among the query SQL complexity quantification C, query cycle quantification P, query timeliness quantification T, query response time range quantification R, and query magnitude quantification S, and N XRepresents the normalized value, min(X) represents the minimum value of the indicator data X obtained in the processor for all query request information, and max(X) represents the maximum value of the indicator data X obtained in the processor for all query request information.

[0085] Taking the query SQL complexity metric C as an example, if the query SQL complexity metric C that needs to be normalized is 25, the minimum value min(C) of the query SQL complexity metric C obtained from all query request information is 10, and the maximum value max(C) of the query SQL complexity metric C obtained from all query request information is 40, then the normalized query SQL complexity metric N is calculated. C is 0.5.

[0086] S302: Determine a comprehensive performance score based on the quantified values ​​of the indicator data and the first weight coefficients corresponding to the respective indicators.

[0087] For example, each quantized value is normalized to obtain the normalized query SQL complexity quantification N C , query cycle quantity N P , Query timeliness quantification N T , query response time range quantification N R and query magnitude quantization N S The first set of weight coefficients includes w C ,w P ,w T ,w R ,w S .

[0088] Determine the comprehensive performance score S total The formula is as follows:

[0089] S total =w C ×C+w P ×P+w T ×T+w R ×R+w S ×S.

[0090] S203: For each engine strategy to be selected, based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected, determine a fitness value for characterizing the query performance of the engine strategy to be selected;

[0091] Exemplarily, the fitness value is used to characterize the query performance of the engine strategy to be selected. The larger the fitness value, the better the comprehensive performance of the engine strategy to be selected corresponding to the fitness value. The smaller the fitness value, the worse the comprehensive performance of the engine strategy to be selected corresponding to the fitness value.

[0092] In one possible implementation, Figure 4 FIG. 1 is a flow chart of another engine selection method provided in an embodiment of the present application, and the steps are as follows:

[0093] S401: Determine a second weight coefficient corresponding to the comprehensive performance score of each to-be-selected engine of the to-be-selected engine strategy based on a third correspondence between the preset to-be-selected engine strategy and the second weight coefficient set;

[0094] Taking the received query request information as an example of 5 query request information, each generated engine strategy to be selected includes 5 engines to be selected, and the third correspondence between the second weight coefficient in the second weight coefficient set and the 5 engines to be selected is pre-set. Among them, the second weight coefficient in the second weight coefficient set can be a1, a2, a3, a4, a5, and the comprehensive performance score of each engine to be selected in the engine strategy to be selected can be S total_1 、S total_2 、S total_3 、S total_4 、S total_5 .

[0095] S402: Determine a fitness value based on the comprehensive performance score of each to-be-selected engine and its corresponding second weight coefficient.

[0096] Exemplarily, the formula for determining the fitness value may be:

[0097] Fitness value = S total_1 ×a1+S total_2 ×a2+S total_3 ×a3+S total_4 ×a4+S total_5 ×a5.

[0098] S204: Determine a target engine strategy from the engine strategies to be selected based on the fitness value.

[0099] In one possible implementation, since the fitness value is used to characterize the query performance of the engine strategy to be selected, the larger the fitness value, the better the comprehensive performance of the engine strategy to be selected corresponding to the fitness value. Therefore, the strategy to be selected corresponding to the maximum fitness value can be used as the target engine strategy.

[0100] Further, Figure 5, which is a flow chart of another engine selection method provided by an embodiment of the present application. This embodiment of the present application can also generate a new engine selection strategy by iteratively updating the engine selection strategy until a better overall performance engine selection strategy is obtained. That is, for each engine selection strategy, after determining a fitness value representing the query performance of the engine selection strategy based on the overall performance score of each engine selection strategy, the method further includes:

[0101] S501, determining a first sub-target engine strategy to be selected from the engine strategies to be selected based on the fitness value;

[0102] In a possible implementation, an engine strategy to be selected is selected from the engine strategies to be selected as the first sub-target engine strategy to be selected, a new engine strategy to be selected is generated by performing genetic transformation on the first sub-target engine strategy to be selected, and then an iterative update is performed based on the newly generated engine strategy to be selected. Specifically, Figure 6 As shown, the steps for determining the engine strategy to be selected for the first sub-goal are as follows:

[0103] S601: summing up the fitness values ​​of each to-be-selected engine strategy to obtain a first sum value;

[0104] For example, taking the five engine strategies to be selected generated in Table 1 as an example, if the fitness value of engine strategy 1 to be selected is 0.8, the fitness value of engine strategy 2 to be selected is 0.6, the fitness value of engine strategy 3 to be selected is 0.9, the fitness value of engine strategy 4 to be selected is 0.7, and the fitness value of engine strategy 5 to be selected is 0.5, then the first sum value obtained after the addition calculation is 3.5.

[0105] S602: For each engine strategy to be selected, calculate the quotient of the fitness value of the engine strategy to be selected divided by the first sum value;

[0106] For example, for the engine strategy 1 to be selected, the quotient value 1 = 0.8 / 3.5 = 0.2286, for the engine strategy 2 to be selected, the quotient value 2 = 0.6 / 3.5 = 0.1714, for the engine strategy 3 to be selected, the quotient value 3 = 0.9 / 3.5 = 0.2571, for the engine strategy 4 to be selected, the quotient value 4 = 0.7 / 3.5 = 0.2, and for the engine strategy 1 to be selected, the quotient value 1 = 0.5 / 3.5 = 0.1429.

[0107] S603: Determine the first sub-target engine strategy to be selected based on the quotient value corresponding to each engine strategy to be selected.

[0108] Furthermore, in a possible implementation, the to-be-selected engine strategy corresponding to the maximum quotient value is used as the first sub-target to-be-selected engine strategy;

[0109] For example, based on the quotient values ​​1, 2, 3, 4, and 5 calculated above, the larger the value, the better the performance of the engine strategy to be selected. Therefore, engine strategy 3, corresponding to the maximum quotient value 3, can be selected as the first sub-target engine strategy to be selected.

[0110] Furthermore, in another possible embodiment, the engine strategy to be selected for the first sub-target is determined based on the first absolute value of the difference between the maximum quotient value and the random number, and the second absolute value of the difference between the second sum value and the random number value; wherein the second sum value is used to represent the sum of the maximum quotient value and other quotient values ​​except the maximum quotient value.

[0111] In a specific embodiment, the first absolute value of the difference between the maximum quotient value and the random value is calculated; the maximum value of the quotient values ​​of each engine strategy to be selected, excluding the maximum quotient value, is taken as the target quotient value, the sum of the maximum quotient value and the target quotient value is taken as the second sum, and the second absolute value of the difference between the second sum and the random value is calculated; the second sum is taken as the maximum quotient value, and the process returns to the step of determining the target quotient value until the quotient value of each engine strategy to be selected is calculated; the minimum absolute value among the first absolute value and at least one second absolute value is determined, and the engine strategy to be selected corresponding to the minimum absolute value is taken as the first sub-target engine strategy to be selected.

[0112] For example, if the random value is 0.65, and 5 engine strategies to be selected are generated, the quotient value 1 of the engine strategy to be selected is 0.2286, the quotient value 2 of the engine strategy to be selected is 0.1714, the quotient value 3 of the engine strategy to be selected is 0.2571, the quotient value 4 of the engine strategy to be selected is 0.2, and the quotient value 1 of the engine strategy to be selected is 0.1429.

[0113] That is, the maximum quotient is 3, the calculated first absolute value is 0.3929, and the maximum value outside the maximum quotient is used as the target quotient, that is, the quotient is 1 as the target quotient, the calculated second sum 1 is 0.4857, and the second absolute value 1 is 0.1643;

[0114] Using the second sum 1 as the maximum quotient, returning to the step of determining the target quotient, using the quotient 4 as the target quotient, the second sum 2 is calculated to be 0.6857, and the second absolute value 2 is 0.0357;

[0115] Taking the second sum 2 as the maximum quotient, returning to the step of determining the target quotient, taking quotient 2 as the target quotient, the second sum 3 is calculated to be 0.8571, and the second absolute value 3 is 0.2071;

[0116] The second sum 3 is taken as the maximum quotient, and the process returns to the step of determining the target quotient. The quotient 5 is taken as the target quotient, and the second sum 4 is calculated to be 1, and the second absolute value 4 is 0.45.

[0117] The minimum absolute value between the first absolute value and at least one second absolute value is determined to be the second absolute value 2, and the to-be-selected engine strategy 4 corresponding to the quotient 4 corresponding to the second absolute value 2 is used as the first sub-target to-be-selected engine strategy.

[0118] S502: Perform genetic transformation on the first sub-goal to-be-selected engine strategy to obtain the second sub-goal to-be-selected engine strategy;

[0119] In one possible implementation, a crossover transformation is performed based on the first sub-target engine strategy to be selected and the third sub-target engine strategy to be selected determined from the engine strategies to be selected to obtain a child engine strategy. A crossover transformation is a genetic algorithm that reorganizes parental genetic information to produce new children. In this application, the child engine strategy is obtained by performing a crossover transformation using the first sub-target engine strategy to be selected and the third sub-target engine strategy to be selected as parents.

[0120] For example, using the strategy for the first sub-target engine to be selected, engine 4: Hive, Flink, Flink, StarRocks, Hive, as the first sub-target engine to be selected, engine 2: Flink, StarRocks, Hive, Flink, StarRocks, as the third sub-target engine to be selected. Crossover transformation is performed by selecting crossover potentials. For example, using crossover point 2 as an example, engine 4 is split into Hive, Flink and Flink, StarRocks, Hive, and engine 2 is split into Flink, StarRocks and Hive, Flink, StarRocks. The resulting offspring engine strategy is Hive, Flink, Hive, Flink, StarRocks.

[0121] The offspring engine strategy is mutated to obtain the second sub-target engine strategy to be selected. The mutation transformation in the genetic algorithm is to randomly transform the engine to be selected in the strategy to be selected, thereby avoiding falling into the local optimal solution during the iterative update process, improving the global search capability, and achieving a more reasonable application of the engine strategy.

[0122] For example, taking the offspring engine strategy of Hive, Flink, Hive, Flink, and StarRocks as an example, a genetic transformation is performed based on the to-be-selected engines Hive, Flink, and StarRocks by randomly selecting an engine to be selected from the offspring engine strategy and replacing it with another to-be-selected engine. This can result in the second sub-target to-be-selected engine strategy of StarRocks, Flink, Hive, Flink, and StarRocks, or the second sub-target to-be-selected engine strategy of StarRocks, Hive, Hive, Flink, and StarRocks. The above embodiments are merely for a clearer understanding of the technical solutions of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0123] S503. Update the multiple engine strategies to be selected based on the second sub-target engine strategy to be selected, and return to the step of determining the quantitative value of each indicator data until the change between the current fitness value and the last fitness value is less than a preset threshold, and use the most recently obtained second sub-target engine strategy to be selected as the target engine strategy.

[0124] In one possible implementation, the second sub-target engine to be selected is replaced with the engine to be selected strategy corresponding to the minimum quotient value to obtain multiple updated engine to be selected strategies; since the engine to be selected strategy corresponding to the maximum quotient value represents the most preferred strategy among the engine to be selected strategies, by replacing the engine to be selected strategy corresponding to the minimum quotient value, the engine to be selected strategy with better comprehensive performance is obtained in the iterative update process.

[0125] For example, taking Table 1 above as an example, the engine strategy to be selected in Table 1 is used as the engine strategy to be selected before the update. According to the quotient value calculated above, engine 2 to be selected is determined to be the engine to be selected corresponding to the minimum quotient value. Therefore, as shown in Table 2, the second sub-target engine strategy to be selected StarRocks, Flink, Hive, Flink, StarRocks obtained above can be replaced with engine 2 to obtain multiple updated engine strategies to be selected.

[0126] Table 2

[0127]

[0128] In one possible implementation, the second sub-target engine to be selected is added to the engine to be selected strategy, resulting in multiple updated engine strategies to be selected. By adding the second sub-target engine to the engine to be selected strategy, the variety of engine strategies to be selected is increased, thereby achieving a more optimal solution during the iterative update process. As shown in Table 3, the second sub-target engine to be selected strategy StarRocks, Flink, Hive, Flink, and StarRocks obtained above can be added to the engine to be selected strategy to obtain multiple updated engine strategies to be selected.

[0129] Table 3

[0130]

[0131]

[0132] After obtaining the updated engine strategy to be selected, return to the step of determining the quantitative value of each indicator data until the change between the current fitness value and the last fitness value obtained is less than the preset threshold, and use the second sub-target engine strategy to be selected obtained most recently as the target engine strategy, and output the target engine strategy.

[0133] It should be noted that after the query request task is completed, the query performance information is also collected, including query response time, resource usage, etc., and fed back into the performance evaluation model to evaluate the target engine strategy selected this time.

[0134] Based on the same inventive concept, an engine selection device is also provided in the embodiment of the present application, such as Figure 7 As shown, a structural diagram of an engine selection device provided in an embodiment of the present application includes at least one processor 701; a memory 702 communicatively connected to the at least one processor 701; and a bus 703; wherein 702 stores instructions that can be executed by the at least one processor 701, and the instructions are executed by the at least one processor 701 to enable the at least one processor 701 to execute any one of the above-mentioned engine selection methods.

[0135] Memory 702 is used to store computer programs executed by processor 701. Memory 702 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0136] Memory 702 can be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 702 can be a combination of the above memories.

[0137] The processor 701 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 701 is configured to implement the engine selection method in the above embodiment when calling the computer program stored in the memory 702 .

[0138] Based on the same inventive concept, an embodiment of the present application further provides a computer storage medium, which stores a computer program, and the computer program is used to enable a computer to execute any one of the above-mentioned engine selection methods.

[0139] The present invention provides an engine selection method, device, and computer storage medium. The method generates multiple engine selection strategies in response to at least one query request information, wherein the engine selection strategy includes a first correspondence between at least one engine to be selected and the query request information; for each engine to be selected, based on the quantitative criteria of the engine to be selected and at least one indicator data of the query request information corresponding to the engine to be selected, determines a quantitative value of each indicator data, and determines a comprehensive performance score of the engine to be selected based on the quantitative value, wherein the quantitative value is used to characterize the complexity of the query request information, and the comprehensive performance score is used to characterize the query performance of the engine to be selected; for each engine selection strategy, based on the comprehensive performance score of each engine to be selected in the engine selection strategy, determines a fitness value used to characterize the query performance of the engine selection strategy; and determines a target engine strategy from the engine selection strategy based on the fitness value. In other words, the present invention determines a target engine strategy by calculating the fitness values ​​of the multiple engine selection strategies generated, and implements multiple engine strategies to process multiple query tasks simultaneously based on the target engine strategy, thereby avoiding overloading of a single query engine and improving the stability and processing efficiency of the big data platform.

[0140] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0141] The present application is described above with reference to block diagrams and / or flow charts illustrating methods, apparatus (systems) and / or computer program products according to embodiments of the present application. It should be understood that a block of a block diagram and / or flow chart, as well as a combination of blocks of a block diagram and / or flow chart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer and / or other programmable data processing device to produce a machine such that instructions executed by the computer processor and / or other programmable data processing device create a method for implementing the functions / actions specified in the block diagram and / or flow chart block.

[0142] Accordingly, the present application may also be implemented using hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, the present application may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in conjunction with an instruction execution system. In the context of the present application, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, transmit, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0143] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An engine selection method, characterized in that: The method comprises: In response to at least one query request information, generating a plurality of strategies for engines to be selected, wherein the strategies for engines to be selected include a first correspondence between at least one engine to be selected and the query request information; For each engine to be selected, based on the quantitative criteria of the engine to be selected and at least one indicator data of the query request information corresponding to the engine to be selected, a quantitative value of each indicator data is determined, and a comprehensive performance score of the engine to be selected is determined based on the quantitative value, wherein the quantitative value is used to represent the complexity of the query request information, and the comprehensive performance score is used to represent the query performance of the engine to be selected; For each engine strategy to be selected, determining a fitness value for characterizing the query performance of the engine strategy to be selected based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected; A target engine strategy is determined from the engine strategies to be selected based on the fitness value.

2. The method according to claim 1, wherein Determining the comprehensive performance score of the engine to be selected based on the quantified value includes: Determining a first weight coefficient corresponding to a quantized value of each indicator data based on a preset second correspondence between the to-be-selected engine and the first set of weight coefficients; The comprehensive performance score is determined based on the quantized values ​​of the indicator data and the first weight coefficients corresponding to the respective indicators.

3. The method according to claim 1, wherein The determining, based on the comprehensive performance score of each engine to be selected in the engine to be selected strategy, a fitness value for characterizing the query performance of the engine to be selected strategy includes: Determining, based on a preset third correspondence between the strategy for the engine to be selected and the second set of weight coefficients, a second weight coefficient corresponding to the comprehensive performance score of each engine to be selected in the strategy for the engine to be selected; The fitness value is determined based on the comprehensive performance score of each of the engines to be selected and the second weight coefficient corresponding to each.

4. The method according to claim 1, wherein After determining, for each engine strategy to be selected, a fitness value for characterizing the query performance of the engine strategy to be selected based on the comprehensive performance score of each engine to be selected in the engine strategy to be selected, the method further includes: Determining a first sub-target engine strategy to be selected from the engine strategies to be selected based on the fitness value; Performing genetic transformation on the first sub-goal to-be-selected engine strategy to obtain the second sub-goal to-be-selected engine strategy; Based on the second sub-target engine strategy to be selected, the multiple engine strategies to be selected are updated, and the step of determining the quantitative value of each indicator data is returned until the change between the current fitness value and the last fitness value is less than a preset threshold, and the most recently obtained second sub-target engine strategy to be selected is used as the target engine strategy.

5. The method according to claim 4, wherein The determining of a first sub-target engine strategy to be selected from the engine strategies to be selected based on the fitness value includes: The fitness values ​​of each engine strategy to be selected are summed up to obtain a first sum value; For each engine strategy to be selected, calculating a quotient of the fitness value of the engine strategy to be selected divided by the first sum value; Based on the quotient value corresponding to each engine strategy to be selected, the first sub-goal engine strategy to be selected is determined.

6. The method according to claim 5, wherein The determining of the first sub-goal engine strategy to be selected based on the quotient value corresponding to each engine strategy to be selected includes: The engine strategy to be selected corresponding to the maximum quotient value is used as the first sub-target engine strategy to be selected; Or, determining the engine strategy to be selected for the first sub-goal based on a first absolute value of a difference between the maximum quotient and the random number, and a second absolute value of a difference between the second sum and the random number; The second sum value is used to represent the sum of the maximum quotient value and other quotient values ​​except the maximum quotient value.

7. The method according to claim 4, wherein The performing of genetic transformation on the first sub-goal to-be-selected engine strategy to obtain the second sub-goal to-be-selected engine strategy includes: Performing a cross transformation based on the first sub-target engine strategy to be selected and the third sub-target engine strategy to be selected determined from the engine strategies to be selected to obtain a child engine strategy; The child engine strategy is mutated and transformed to obtain the second sub-target engine strategy to be selected.

8. The method according to claim 4, wherein The updating of the plurality of to-be-selected engine strategies based on the second sub-goal to-be-selected engine strategy includes: The second sub-target to-be-selected engine replaces the to-be-selected engine strategy corresponding to the minimum quotient value to obtain multiple updated to-be-selected engine strategies; Alternatively, the second sub-target engine to be selected is added to the strategy of the engine to be selected to obtain multiple updated strategies of the engine to be selected.

9. An engine selection device, characterized in that: The invention comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that The computer storage medium stores a computer program for causing a computer to execute the method according to any one of claims 1 to 8.

Images