Query task allocation method and device, equipment and storage medium

Abstract

The application discloses a query task allocation method and device, equipment and storage medium, and belongs to the technical field of big data. The method comprises the following steps: receiving a plurality of query tasks of a user; determining a loss value corresponding to each query task when the query task is not allocated to an optimal query engine according to characteristic data of each query task in the plurality of query tasks, resources of each query engine in a plurality of query engines, and historical query behavior data of the user; and allocating the plurality of query tasks in a size order of the loss value corresponding to each query task, wherein a query task with a large loss value is preferentially allocated. In the application, a query task with a large loss value is preferentially allocated, and a query task with exclusivity and urgency is preferentially considered, so that the allocation of the query engine is more reasonable, and the overall utilization efficiency of the query engine resources is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a query task allocation method, apparatus, device and storage medium. Background Technology

[0002] Ad-hoc query analysis is one of the core issues in the field of big data. Existing technologies include a resource-based query engine selection method, which chooses the optimal query engine based on the available resources of each query engine and the resource requirements of the query task. In this method, users cannot choose the query engine themselves; the system performs an optimal selection process for each submitted query task, choosing the most suitable engine. The selection factor is mainly the matching degree between the available resources of each query engine and the query task. This method is suitable for scenarios with low task intensity and relatively idle overall resources. However, big data query tasks are not evenly distributed; they are often clustered by time windows, such as user behavior analysis after a sports event or analysis of daily and weekly business reports. In the case of large-scale queries, the optimal query engine determined by the above query engine selection method for each query task may not be the globally optimal one, resulting in unreasonable overall utilization of query engine resources and reduced overall efficiency.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a query task allocation method, apparatus, device, and storage medium, which aims to solve the problem of low overall utilization efficiency of query engine resources when query tasks are aggregated in large quantities in the prior art.

[0005] To achieve the above objectives, this application provides a query task allocation method, which includes the following steps:

[0006] Receive multiple query tasks from users;

[0007] Based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, determine the loss value corresponding to each query task when it is not assigned to the best query engine;

[0008] The multiple query tasks are allocated according to the order of the loss values ​​corresponding to each query task, with priority given to query tasks with larger loss values.

[0009] Optionally, the step of determining the loss value corresponding to each query task when it is not assigned to the best query engine, based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, includes:

[0010] Based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, determine the allocation score corresponding to each query task when it is assigned to each query engine;

[0011] Based on the allocation score corresponding to each query task when it is assigned to each query engine, determine the loss value corresponding to each query task when it is not assigned to the best query engine.

[0012] Optionally, the step of determining the loss value of each query task when it is not assigned to the best query engine, based on the allocation score corresponding to each query task when it is assigned to each query engine, includes:

[0013] Obtain the highest and second-highest scores from the allocation scores corresponding to each query task when it is assigned to each query engine.

[0014] The difference between the highest score and the second highest score is calculated as the loss value for each query task when it is not assigned to the best query engine.

[0015] Optionally, the step of determining the allocation score corresponding to each query task when it is assigned to each query engine based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data includes:

[0016] A target classifier is trained and generated based on the user's historical query behavior data.

[0017] Based on the target classifier, the feature data of each query task in the plurality of query tasks, and the resources of each query engine in the plurality of query engines, the receiving probability corresponding to each query task when it is received by each query engine is determined.

[0018] Based on the receiving probability of each query task when it is received by each query engine, the allocation score of each query task when it is assigned to each query engine is determined.

[0019] Optionally, the step of allocating the multiple query tasks according to the order of the loss values ​​corresponding to each query task includes:

[0020] The steps of determining the query engine to which each query task is assigned, based on the preset query engine allocation rules, are executed in order of the magnitude of the loss value corresponding to each query task, until the multiple query tasks are assigned.

[0021] Optionally, the step of determining the query engine to which the query task is assigned according to a preset query engine allocation rule includes:

[0022] Based on the allocation score corresponding to the query task when it is assigned to each query engine and the allocation relationship between the query task and each query engine, the query task is assigned to a query engine that meets the preset conditions.

[0023] Determine whether the query task and the query engine assigned to the query task meet the constraints.

[0024] If the constraints are met, the query task allocation is complete;

[0025] If the constraints are not met, the allocation relationship between the query task and the query engine to which the query task is assigned is modified to prohibit allocation, and the process returns to the step of allocating the query task to a query engine that meets the preset conditions based on the allocation score corresponding to the query task when it is assigned to each query engine and the allocation relationship between the query task and each query engine.

[0026] The preset conditions include that the assigned query engine has the highest allocation score, and the allocation relationship between the assigned query engine and the query task is allocation; the constraint conditions include that the running time period of the query task coincides with the idle time period of the assigned query engine.

[0027] Optionally, the step of receiving multiple query tasks from a user includes:

[0028] It can receive multiple query tasks from users within a time window.

[0029] Furthermore, to achieve the above objectives, this application also provides a query task allocation device, the device comprising:

[0030] The query task receiving module is used to receive multiple query tasks from users;

[0031] The loss value determination module is used to determine the loss value of each query task when it is not assigned to the best query engine, based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data.

[0032] The query task allocation module is used to allocate the multiple query tasks according to the order of the loss value corresponding to each query task, wherein the query task with the larger loss value is allocated first.

[0033] In addition, to achieve the above objectives, this application also provides a query task allocation device, the device comprising: a memory, a processor, and a query task allocation program stored in the memory and executable on the processor, the query task allocation program being configured to implement the steps of the query task allocation method as described above.

[0034] In addition, to achieve the above objectives, this application also provides a storage medium storing a query task allocation program, which, when executed by a processor, implements the steps of the query task allocation method as described above.

[0035] This application discloses a query task allocation method, apparatus, device, and storage medium. Compared with existing technologies where query engine resources are underutilized when a large number of query tasks are aggregated, this application receives multiple query tasks from a user; determines the loss value of each query task when it is not assigned to the optimal query engine based on the feature data of each query task, the resources of each query engine, and the user's historical query behavior data; and allocates the multiple query tasks according to the order of their loss values, with priority given to query tasks with larger loss values. In other words, this application preferentially allocates query tasks with larger loss values ​​and prioritizes exclusive and urgent query tasks, making the allocation of query engines more reasonable and improving the overall utilization efficiency of query engine resources. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the structure of the query task allocation device of the hardware operating environment involved in the embodiments of this application;

[0039] Figure 2 This is a flowchart illustrating the first embodiment of the query task allocation method of this application;

[0040] Figure 3 This is a scenario application diagram of the resource-based query engine selection method in the background technology of this application;

[0041] Figure 4 A scenario application diagram for the query task allocation method in this application;

[0042] Figure 5 This is a schematic diagram of the functional modules of the first embodiment of the query task allocation device of this application.

[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure for allocating query tasks in the hardware operating environment involved in the embodiments of this application.

[0046] like Figure 1 As shown, the query task allocation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the query task allocation device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0048] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a query task allocation program. The operating system is a program that manages and controls the hardware and software resources of the query task allocation device, and supports the operation of the query task allocation program and other software or programs.

[0049] exist Figure 1 In the query task allocation device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the query task allocation device calls the query task allocation program stored in the memory 1005 through the processor 1001 and executes the query task allocation method provided in the embodiments of this application.

[0050] This application provides a method for querying task allocation, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of a query task allocation method according to this application.

[0051] In this embodiment, the query task allocation method includes:

[0052] Step S10: Receive multiple query tasks from the user;

[0053] Step S20: Based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, determine the loss value corresponding to each query task when it is not assigned to the best query engine.

[0054] Step S30: Allocate the multiple query tasks according to the order of the loss values ​​corresponding to each query task, wherein the query task with the larger loss value is allocated first.

[0055] Compared to existing technologies where query engine resource utilization is low when large numbers of query tasks are aggregated, this embodiment receives multiple query tasks from a user; determines the loss value of each query task when it is not assigned to the optimal query engine based on the feature data of each query task, the resources of each query engine, and the user's historical query behavior data; and allocates the multiple query tasks according to the order of their loss values, prioritizing the allocation of query tasks with larger loss values. In other words, this embodiment preferentially allocates query tasks with larger loss values, giving priority to exclusive and urgent query tasks, resulting in a more rational allocation of query engines and improved overall query engine resource utilization efficiency.

[0056] The specific steps are as follows:

[0057] Step S10: Receive multiple query tasks from the user.

[0058] It's important to note that in practical applications, big data query tasks are not evenly distributed but often clustered according to time windows, which are time periods. This means that for the same set of multiple query engines, there may be a situation where a large number of query tasks occur within a first time window, while a small number, or even none, occur within an adjacent second time window. In the second time window, query tasks are less concentrated, and the overall resources of the query engines are relatively idle. In this case, grouping the query tasks from the first and second time windows together and rearranging the order in which they are assigned to engines, prioritizing exclusive and urgent tasks, would cause the query tasks from the second time window to crowd out the query engine resources of the query tasks from the first time window, reducing the timeliness of the first time window's queries. Therefore, to ensure the timeliness of query tasks within each time window, all query tasks in the set to be assigned are within a single time window. In other words, in this embodiment, receiving multiple query tasks from a user means receiving multiple query tasks from the user within a single time window.

[0059] Step S20: Based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, determine the loss value corresponding to each query task when it is not assigned to the best query engine.

[0060] Specifically, step S20 includes:

[0061] Step S21: Based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, determine the allocation score corresponding to each query task when it is assigned to each query engine.

[0062] Specifically, step S21 includes:

[0063] Step S211: Train and generate a target classifier based on the user's historical query behavior data.

[0064] In this embodiment, the user's historical query behavior data includes the feature data of historical query tasks, the resources of each query engine corresponding to the historical query behavior, and the correct reception probability of the historical query task being received by each query engine corresponding to the historical query behavior.

[0065] The characteristic data of historical query tasks includes the metadata of the historical query tasks and the relevance of the historical query tasks to each query engine corresponding to the historical query behavior. The metadata of historical query tasks includes the number of partitions in the queried table, whether it is dynamically partitioned, the number of small files, and the file format, etc.; the relevance of historical query tasks to each query engine corresponding to the historical query behavior includes whether the tables involved in the historical query task have been recently queried by the query engine, and whether the historical query task meets the requirements of the query engine, etc. (For example, for performance reasons, dynamically partitioned query tasks are not recommended to be submitted to the Spark query engine for execution).

[0066] The resources for each query engine corresponding to historical query behavior include: the idle resources of the node corresponding to the query engine, such as the number of remaining NodeManagers, memory capacity and Vcores of the Yam cluster corresponding to the Hive query engine, and the number of ImpalaServers of the Impala cluster.

[0067] The step of training and generating a target classifier based on the user's historical query behavior data specifically includes:

[0068] The feature data of the historical query tasks and the resources of each query engine corresponding to the historical query behavior are divided into training set and validation set;

[0069] The logistic regression classifier is trained using the data in the training set to obtain the trained logistic regression classifier.

[0070] The data in the validation set is input into the trained logistic regression classifier, and the model acceptance probability of each query engine corresponding to the historical query task is obtained when the historical query behavior is received is calculated.

[0071] Based on the correct reception probability, calculate the loss error of the model reception probability;

[0072] If the loss error is greater than the preset error threshold, the regression coefficients of the logistic regression classifier are corrected, and the process returns to the step of training the logistic regression classifier using the data in the training set to obtain the trained logistic regression classifier, until the loss error is less than or equal to the preset error threshold.

[0073] If the loss error is less than or equal to the preset error threshold, then the trained logistic regression classifier is the target classifier.

[0074] Step S212: Based on the target classifier, the feature data of each query task in the plurality of query tasks, and the resources of each query engine in the plurality of query engines, determine the receiving probability corresponding to each query task when it is received by each query engine.

[0075] It should be noted that the characteristic data for each query task includes the query task's metadata and its relevance to the query engine. The query task's metadata includes the number of partitions in the queried table, whether it is dynamically partitioned, the number of small files, and the file format. The relevance of the query task to the query engine includes whether the tables involved in the query task have been recently queried by the query engine, and whether the query task meets the query engine's requirements (for example, for performance reasons, dynamically partitioned query tasks are not recommended to be submitted to the Spark query engine for execution).

[0076] It should be noted that the resources of each query engine include: the idle resources of the node corresponding to the query engine, such as the number of remaining NodeManagers, memory capacity and Vcores of the Yam cluster corresponding to the Hive query engine, and the number of ImpalaServers of the Impala cluster.

[0077] In this embodiment, the feature data of each query task and the resources of each query engine are input into the target classifier, so that the receiving probability of each query task being received by each query engine can be calculated.

[0078] Step S213: Determine the allocation score for each query task when it is assigned to each query engine based on the receiving probability of each query task being received by each query engine.

[0079] In this embodiment, the allocation score can be represented by the following calculation formula:

[0080]

[0081] Among them, score ij Let p represent the assigned score, i represent the query task number (i is a positive integer), and j represent the query engine number. If there are m query engines, then the query engine numbers are 1, 2, 3, ..., m. ij a represents the probability of query task i being received by query engine j. ij This indicates that a query engine can only receive one query task within a time window. ij =0 indicates that query engine j cannot receive query task i within a time window, a ij =1 indicates that query engine j can receive query task i within a time window.

[0082] In this embodiment, the allocation score corresponding to query engine 1 and query task i is determined. Then, in the aforementioned formula for calculating the allocation score, a i1 =0, a i2 =1, a i3 =1, ...,aim =1;

[0083] Once the assigned score for query engine 2 and query task i is determined, then in the aforementioned formula for calculating the assigned score, a i1 =1, a i2 =0, a i3 =1, ...,a im =1;

[0084] Once the assigned score for query engine 3 and query task i is determined, then in the aforementioned formula for calculating the assigned score, a i1 =1, a i2 =1, a i3 =0, ..., a im =1;

[0085] …

[0086] Once the assigned score for query engine m and query task i is determined, then in the aforementioned formula for calculating the assigned score, a i1 =1, a i2 =1, a i3 =1, ...,a im =0.

[0087] Step S22: Determine the loss value of each query task when it is not assigned to the best query engine, based on the allocation score corresponding to each query task when it is assigned to each query engine.

[0088] Specifically, step S22 includes:

[0089] Step S221: Obtain the highest and second-highest scores among the allocation scores corresponding to each query task when it is assigned to each query engine;

[0090] Step S222: Calculate the difference between the highest score and the second highest score, as the loss value for each query task when it is not assigned to the best query engine.

[0091] In this embodiment, the loss value can be represented by the following calculation formula:

[0092] lost i =firstMax(score) i1 ..., score im )-secondMax(score i1 ..., score im )

[0093] Among them, lost iThis represents the loss value for each query task when it is not assigned to the best query engine. The loss value refers to the loss caused by the query task not being prioritized and not assigned to its best matching query engine, resulting in a lower overall score for the set of tasks to be assigned. `firstMax(score)` i1 ..., score im i represents the highest score among the allocation scores corresponding to each query engine when the query task is assigned to each query engine, and secondMax(score) i1 ..., score im The score represents the second-highest score among the allocation scores for each query engine when the query task is assigned to it. i1 The score represents the allocation score when query task i is assigned to query engine 1. im The assignment score represents the score at which query task i is assigned to query engine m.

[0094] It should be noted that, in this embodiment, the loss value is used to describe the exclusivity of the query engine corresponding to the highest score in the query task and the urgency of the query task allocation.

[0095] In this embodiment, after determining the assigned query engine for each query task, the allocation score of each query task is added to the allocation score of the assigned query engine to obtain the overall allocation score of the received multiple query tasks.

[0096] In other words, if the query task with the largest loss value is not prioritized for allocation, and the query task is not assigned to its best matching query engine (the query engine corresponding to the highest score among the allocation scores of each query task when it is assigned to each query engine), once the best matching query engine is preempted by other query tasks, it will cause a huge loss in the allocation of query tasks, which will affect the overall allocation score of the multiple received query tasks, and thus affect the overall resource utilization efficiency of the query engine.

[0097] For example, if query task i is assigned to the Spark query engine with a score of 3, to the Hive query engine with a score of 5, and to the Impala query engine with a score of 11, then the highest score is 11, the second highest score is 5, and the loss value for query task i when it is not assigned to the best query engine is 11-5=6.

[0098] Step S30: Allocate the multiple query tasks according to the order of the loss values ​​corresponding to each query task, wherein the query task with the larger loss value is allocated first.

[0099] Specifically, step S30 includes:

[0100] The steps of determining the query engine to which each query task is assigned, based on the preset query engine allocation rules, are executed in order of the magnitude of the loss value corresponding to each query task, until the multiple query tasks are assigned.

[0101] The step of determining the query engine assigned to the query task according to preset query engine allocation rules includes:

[0102] Step A1: Based on the allocation score corresponding to the query task when it is assigned to each query engine and the allocation relationship between the query task and each query engine, assign the query task to a query engine that meets the preset conditions.

[0103] Step A2: Determine whether the query task and the query engine assigned to the query task meet the constraints.

[0104] Step A3: If the constraints are met, the query task allocation is complete.

[0105] Step A4: If the constraints are not met, the allocation relationship between the query task and the query engine to which the query task is assigned is modified to prohibit allocation, and the process returns to the step of allocating the query task to a query engine that meets the preset conditions based on the allocation score corresponding to the query task when it is assigned to each query engine and the allocation relationship between the query task and each query engine.

[0106] The preset conditions include that the assigned query engine has the highest allocation score, and the allocation relationship between the assigned query engine and the query task is allocation; the constraint conditions include that the running time period of the query task coincides with the idle time period of the assigned query engine.

[0107] In other words, in this embodiment, the query task with the highest loss value is assigned first according to the order of the loss values ​​corresponding to each query task. The assignment process is as follows: Query engines with the same assignment relationship as the first-ranked query task are selected from all query engines. Based on the assignment score of each query engine when the first-ranked query task is assigned, the query engine with the highest assignment score is selected from the selected query engines and assigned to the first-ranked query task. It is then determined whether the assigned query engine meets the constraints. If the assigned query engine is not assigned any query tasks within the same time window, then the assigned query engine meets the constraints, and the assignment of the first-ranked query task ends. If the assigned query engine is assigned any query tasks within the same time window, then the assigned query engine does not meet the constraints. In this case, the assignment relationship between the first-ranked query task and the assigned query engine is modified to prohibit assignment, and the process returns to the step of selecting query engines with the same assignment relationship as the first-ranked query task from all query engines, and a new query engine is selected. Following the allocation process of assigning the first-ranked query task, query engines are then assigned to query tasks sorted by their loss values ​​in turn.

[0108] It should be noted that in this embodiment, within the same time window, a query engine can only be assigned to one query task. If, within the same time window, the query engine assigned to the query task that meets the preset conditions is occupied by other query tasks with higher priority, i.e., the running time of the query task does not overlap with the idle time of the assigned query engine, then the allocation relationship between the query task and its assigned query engine that meets the preset conditions needs to be corrected to disallow allocation, and another query engine should be selected for allocation instead of still assigning it to the query engine that meets the preset conditions. This would be detrimental to the overall utilization of query engine resources.

[0109] Specific examples are provided:

[0110] If both Hive and Spark query engines are available, and query task 1 and query task 2 are submitted sequentially within the same time window with a very short submission interval, the resource-based query engine selection method in the background technology and the query task allocation method of this application are used to process query task 1 and query task 2 respectively.

[0111] Reference Figure 3 , Figure 3 This is a scenario application diagram of the resource-based query engine selection method in the background technology of this application.

[0112] The resource-based query engine selection method in the background technology is used to process query task 1 and query task 2. That is, query task 1 and query task 2 are assigned according to the submission order of the query tasks. Query task 1 is preferentially assigned the Hive query engine with the highest score. Therefore, query task 2 can only choose the Spark query engine. That is, the total allocation score of the set to be assigned is 3+4=7 points.

[0113] Reference Figure 4 , Figure 4 This is a scenario application diagram for the query task allocation method in this application.

[0114] The query task allocation method of this application is used to process query task 1 and query task 2. First, the loss values ​​of query task 1 and query task 2 are calculated. For query task 1, the loss value is 3-2=1; for query task 2, the loss value is 7-4=3. Therefore, the tasks are rearranged, and query task 2 is preferentially allocated. Query task 2 selects the Hive query engine with the highest allocation score, and query task 1 selects the Spark query engine. That is, the overall allocation score of the set to be allocated is 7+2=9 points.

[0115] In the case of batch submission of query tasks, the query task allocation method of this application, due to the resource-based query engine selection method in the background technology, makes the allocation of query engines more reasonable and improves the overall utilization efficiency of query engine resources.

[0116] This application also provides a query task allocation device, referring to... Figure 5 , Figure 5 This is a schematic diagram of the functional modules of a first embodiment of a query task allocation device according to this application.

[0117] A query task allocation device, comprising:

[0118] The query task receiving module 10 is used to receive multiple query tasks from users.

[0119] The loss value determination module 20 is used to determine the loss value corresponding to each query task when it is not assigned to the best query engine, based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data.

[0120] The query task allocation module 30 is used to allocate the multiple query tasks according to the order of the loss values ​​corresponding to each query task, wherein the query task with the larger loss value is allocated first.

[0121] Optionally, the loss value determination module includes:

[0122] The allocation score determination unit is used to determine the allocation score corresponding to each query task when it is assigned to each query engine, based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data.

[0123] The loss value determination unit is used to determine the loss value of each query task when it is not assigned to the best query engine, based on the allocation score corresponding to each query task when it is assigned to each query engine.

[0124] Optionally, the loss value determination unit includes:

[0125] The filtering subunit is used to obtain the highest and second-highest scores among the allocation scores corresponding to each query task when it is assigned to each query engine.

[0126] The loss value determination subunit is used to calculate the difference between the highest score and the second highest score, which serves as the loss value for each query task when it is not assigned to the best query engine.

[0127] Optionally, the score allocation determination unit includes:

[0128] The classifier training subunit is used to train and generate a target classifier based on the user's historical query behavior data;

[0129] The receiving probability determination subunit is used to determine the receiving probability of each query task when it is received by each query engine, based on the target classifier, the feature data of each query task in the plurality of query tasks, and the resources of each query engine in the plurality of query engines.

[0130] The allocation score determination subunit is used to determine the allocation score corresponding to each query task when it is allocated to each query engine, based on the receiving probability corresponding to each query task when it is received by each query engine.

[0131] Optionally, the query task allocation module is used to implement:

[0132] The steps of determining the query engine to which each query task is assigned, based on the preset query engine allocation rules, are executed in order of the magnitude of the loss value corresponding to each query task, until the multiple query tasks are assigned.

[0133] Optionally, the query task allocation module includes:

[0134] The initial allocation subunit is used to allocate the query task to a query engine that meets preset conditions based on the allocation score corresponding to the query task when it is allocated to each query engine and the allocation relationship between the query task and each query engine. The preset conditions include that the allocated query engine has the highest allocation score and the allocation relationship between the allocated query engine and the query task is allocation.

[0135] The first judgment subunit is used to determine whether the query task and the query engine assigned to the query task meet the constraint conditions.

[0136] The allocation determination subunit is used to complete the allocation of the query task if the constraint conditions are met, and if the constraint conditions are not met, to modify the allocation relationship between the query task and the query engine to which the query task is allocated to to prohibit allocation, and return to the step of allocating the query task to a query engine that meets preset conditions based on the allocation score corresponding to the query task when it was allocated to each query engine and the allocation relationship between the query task and each query engine. The constraint conditions include the running time period of the query task coinciding with the idle time period of the allocated query engine.

[0137] Optionally, the query task receiving module is used to implement:

[0138] It can receive multiple query tasks from users within a time window.

[0139] The specific implementation of the query task allocation device in this application is basically the same as the embodiments of the query task allocation method described above, and will not be repeated here.

[0140] This application also proposes a storage medium storing a query task allocation program, which, when executed by a processor, implements the steps of the query task allocation method described above.

[0141] The specific implementation of the storage medium in this application is basically the same as the embodiments of the query task allocation method described above, and will not be repeated here.

[0142] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0143] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0145] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A query task allocation method, characterized in that, The query task allocation method includes the following steps: Receive multiple query tasks from users; Based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data, determine the allocation score corresponding to each query task when it is assigned to each query engine; Based on the allocation score corresponding to each query task when it is assigned to each query engine, determine the loss value corresponding to each query task when it is not assigned to the best query engine. Obtain the highest and second-highest scores from the allocation scores corresponding to each query task when it is assigned to each query engine. Calculate the difference between the highest score and the second highest score, and use it as the loss value for each query task when it is not assigned to the best query engine; The loss value is the magnitude of the loss caused by the allocation score of the entire set to be allocated when the query task is not assigned to the best query engine. The multiple query tasks are allocated according to the order of the loss values ​​corresponding to each query task, with priority given to query tasks with larger loss values.

2. The task allocation method as described in claim 1, characterized in that, The step of determining the allocation score corresponding to each query task when it is assigned to each query engine based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data includes: A target classifier is trained and generated based on the user's historical query behavior data. Based on the target classifier, the feature data of each query task in the plurality of query tasks, and the resources of each query engine in the plurality of query engines, the receiving probability corresponding to each query task when it is received by each query engine is determined. Based on the receiving probability of each query task when it is received by each query engine, the allocation score of each query task when it is assigned to each query engine is determined.

3. The task allocation method as described in claim 1, characterized in that, The step of allocating the multiple query tasks according to the order of the loss values ​​corresponding to each query task includes: The steps of determining the query engine to which each query task is assigned, based on the preset query engine allocation rules, are executed in order of the magnitude of the loss value corresponding to each query task, until the multiple query tasks are assigned.

4. The task allocation method as described in claim 3, characterized in that, The step of determining the query engine to which the query task is assigned according to a preset query engine allocation rule includes: Based on the allocation score corresponding to the query task when it is assigned to each query engine and the allocation relationship between the query task and each query engine, the query task is assigned to a query engine that meets the preset conditions. Determine whether the query task and the query engine assigned to the query task meet the constraints. If the constraints are met, the query task allocation is complete; If the constraints are not met, the allocation relationship between the query task and the query engine to which the query task is assigned is modified to prohibit allocation, and the process returns to the step of allocating the query task to a query engine that meets the preset conditions based on the allocation score corresponding to the query task when it is assigned to each query engine and the allocation relationship between the query task and each query engine. The preset conditions include that the assigned query engine has the highest allocation score, and the allocation relationship between the assigned query engine and the query task is allocation; the constraint conditions include that the running time period of the query task coincides with the idle time period of the assigned query engine.

5. The task allocation method as described in claim 1, characterized in that, The step of receiving multiple query tasks from a user includes: Receive multiple query tasks from users within a time window; All query tasks in the set to be assigned are within a single time window.

6. A query task allocation device, characterized in that, The device includes: The query task receiving module is used to receive multiple query tasks from users; The loss value determination module is used to determine the allocation score corresponding to each query task when it is assigned to each query engine based on the feature data of each query task in the plurality of query tasks, the resources of each query engine in the plurality of query engines, and the user's historical query behavior data. Based on the allocation score corresponding to each query task when it is assigned to each query engine, the module determines the loss value corresponding to each query task when it is not assigned to the best query engine. The filtering subunit is used to obtain the highest and second-highest scores among the allocation scores corresponding to each query task when it is assigned to each query engine. The loss value determination subunit calculates the difference between the highest score and the second highest score as the loss value for each query task when it is not assigned to the best query engine; wherein, the loss value is the magnitude of the loss caused by the allocation score of the entire set to be allocated when the query task is not assigned to the best query engine. The query task allocation module is used to allocate the multiple query tasks according to the order of the loss value corresponding to each query task, wherein the query task with the larger loss value is allocated first.

7. A query task allocation device, characterized in that, The device includes: a memory, a processor, and a query task allocation program stored in the memory and executable on the processor, the query task allocation program being configured to implement the steps of the query task allocation method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a query task allocation program, which, when executed by a processor, implements the steps of the query task allocation method as described in any one of claims 1 to 5.

Images