Load balancing method and device, program product, equipment and storage medium

By building multi-backpack problems and optimizing task allocation, the problems of task processing efficiency and node overload in distributed databases are solved, and task processing efficiency is maximized and node load balancing is achieved.

CN118916166APending Publication Date: 2024-11-08BEIJING OCEANBASE TECHNOLOGY CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411046478.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the distribution of subtask load allocation after the data query task split, distributed database cannot guarantee task processing efficiency, resulting in overloading of database instances.

Method used

By obtaining the task overhead of each task and the remaining processing capacity of each node, we build multi-backpack problems, optimize task allocation to maximize task processing efficiency, ensure that each task is assigned to a unique node and the node is not loaded, and priority allocation and compensation are adopted The allocation mechanism handles tasks that cannot be assigned immediately.

Benefits of technology

It realizes that while ensuring task processing efficiency and node overloading, the task processing efficiency of the distributed database system is maximized and the data query task is completed at the fastest speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916166A_ABST
    Figure CN118916166A_ABST
Patent Text Reader

Abstract

The invention provides a load balancing method and device, a program product, equipment and a storage medium. The method comprises the steps of obtaining task overhead of each task in a plurality of tasks to be distributed and residual processing capacity of each node in a plurality of nodes of a distributed database system; each task can be allocated to a unique node and the residual processing capacity of each node is not less than 0 as constraint conditions, task processing efficiency maximization of the distributed database system is used as an optimization target, and the task processing efficiency of the distributed database system is optimized based on the task overhead of each task and the residual processing capacity of each node. Determining an execution node of each task in the plurality of nodes; and sending each task to an execution node, so that at least one node in the plurality of nodes executes the received to-be-allocated task. According to the method, the task processing efficiency of the distributed database system can be maximized under the conditions that each task can have an execution node, each task is not repeatedly distributed with nodes, and each node is not overloaded.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a load balancing method and apparatus, program product, device, and storage medium. Background Art

[0002] With the rapid development of the Internet and informatization, the generation of data is exploding, so the requirements for databases and their management are getting higher and higher. Distributed databases have multiple database instances, which can execute data query tasks in parallel, making the data query efficiency of distributed databases higher. Therefore, when a distributed database processes a data query task, it needs to be split into several subtasks and distributed as load to different database instances. In the process of load distribution, it is necessary to take into account both the task processing efficiency and the processing capacity of the database instance.

[0003] However, in the related art, when the distributed database distributes the load of the subtasks after splitting the data query task, it often cannot guarantee the task processing efficiency or causes the database instance to be overloaded. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a load balancing method and apparatus, a program product, a device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a load balancing method is proposed, the method comprising:

[0007] Obtaining the task overhead of each of the multiple tasks to be assigned and the remaining processing capacity of each of the multiple nodes of the distributed database system;

[0008] With constraints that each task can be assigned to a unique node and that the remaining processing capacity of each node is not less than 0, and with maximizing the task processing efficiency of the distributed database system as an optimization goal, determining an execution node for each task from the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node;

[0009] Each task is sent to an execution node of the task, so that at least one node among the multiple nodes executes the received task to be assigned.

[0010] In a possible embodiment of the present specification, maximizing the task processing efficiency of the distributed database system is characterized by minimizing the remaining processing capacity of the distributed database system.

[0011] In a possible embodiment of the present specification, the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, the optimization goal of maximizing the task processing efficiency of the distributed database system, and the determination of the execution node of each task from the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node include:

[0012] The MKP is constructed by taking each task as an item in the multi-knapsack problem MKP, taking the task overhead of the task as the weight of the corresponding item in the MKP, taking each node as a backpack in the MKP, taking the processing capacity of the node as the capacity of the corresponding backpack in the MKP, and taking the reciprocal of the remaining processing capacity of the distributed database system as the value of all backpacks in the MKP;

[0013] The MKP is optimized based on the task overhead of each task and the remaining processing capacity of each node, and an execution node of each task is determined among the multiple nodes according to the optimization result.

[0014] In a possible embodiment of the present specification, the obtaining of the task overhead of each task in the plurality of tasks to be assigned includes:

[0015] The overhead of the task is determined according to at least one of the following information of the task: task type, task content, and data targeted by the task.

[0016] In a possible embodiment of the present specification, obtaining the remaining processing capacity of each of the multiple nodes of the distributed database system includes:

[0017] The remaining processing capacity of the node is determined according to the CPU status, memory status, and storage status of the node.

[0018] In a possible embodiment of this specification, the method further includes:

[0019] If each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0 as constraint conditions, and the task processing efficiency of the distributed database system is maximized as the optimization goal, and based on the task overhead of each task and the remaining processing capacity of each node, it is impossible to determine an execution node for each task among the multiple nodes, at least one task among the multiple tasks is marked as a priority assignment;

[0020] With the constraints that each of the at least one task marked as priority allocation can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, based on the task overhead of each task in the at least one task marked as priority allocation and the remaining processing capacity of each node, the execution node of each task in the at least one task marked as priority allocation is determined among the multiple nodes.

[0021] In a possible embodiment of this specification, the method further includes:

[0022] After a preset period of time has passed after determining the execution node of each task in at least one task marked as priority allocation, with the constraints that each remaining task can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, the execution node of each remaining task is determined among the multiple nodes based on the task overhead of each remaining task and the remaining processing capacity of each node.

[0023] In a possible embodiment of this specification, the method further includes:

[0024] A data query task is received, and the data query task is split into a plurality of sub-query tasks, wherein the plurality of sub-query tasks serve as a plurality of tasks to be assigned.

[0025] In a possible embodiment of this specification, the method further includes:

[0026] The execution results of multiple sub-query tasks belonging to the same query task are merged into the execution result of the query task.

[0027] According to a second aspect of one or more embodiments of this specification, a load balancing device is provided, the device comprising:

[0028] An acquisition module, used to acquire the task overhead of each task in the plurality of tasks to be assigned, and the remaining processing capacity of each node in the plurality of nodes of the distributed database system;

[0029] An allocation module, configured to determine an execution node for each task from among the plurality of nodes based on the task overhead of each task and the remaining processing capacity of each node, with each task being assignable to a unique node and the remaining processing capacity of each node being not less than 0 as constraints, and maximizing the task processing efficiency of the distributed database system as an optimization goal;

[0030] The sending module is used to send each task to the execution node of the task, so that at least one node among the multiple nodes executes the received task to be assigned.

[0031] In a possible embodiment of the present specification, maximizing the task processing efficiency of the distributed database system is characterized by minimizing the remaining processing capacity of the distributed database system.

[0032] In a possible embodiment of this specification, the allocation module is used to:

[0033] The MKP is constructed by taking each task as an item in the multi-knapsack problem MKP, taking the task overhead of the task as the weight of the corresponding item in the MKP, taking each node as a backpack in the MKP, taking the processing capacity of the node as the capacity of the corresponding backpack in the MKP, and taking the reciprocal of the remaining processing capacity of the distributed database system as the value of all backpacks in the MKP;

[0034] The MKP is optimized based on the task overhead of each task and the remaining processing capacity of each node, and an execution node of each task is determined among the multiple nodes according to the optimization result.

[0035] In a possible embodiment of the present specification, the obtaining of the task overhead of each task in the plurality of tasks to be assigned includes:

[0036] The overhead of the task is determined according to at least one of the following information of the task: task type, task content, and data targeted by the task.

[0037] In a possible embodiment of the present specification, obtaining the remaining processing capacity of each of the multiple nodes of the distributed database system includes:

[0038] The remaining processing capacity of the node is determined according to the CPU status, memory status, and storage status of the node.

[0039] In a possible embodiment of this specification, the device further includes:

[0040] a priority marking module, for marking at least one of the multiple tasks as a priority assignment if, based on the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, the optimization goal is to maximize the task processing efficiency of the distributed database system, and if it is impossible to determine an execution node for each task among the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node;

[0041] A priority allocation module is used to determine the execution node of each task in at least one task marked as priority allocation among the multiple nodes based on the task overhead of each task in at least one task marked as priority allocation and the remaining processing capacity of each node, with the constraints that each task in at least one task marked as priority allocation can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal.

[0042] In a possible embodiment of this specification, the device further includes:

[0043] A compensation allocation module is used to determine the execution node of each remaining task in at least one task marked as priority allocation after a preset period of time has passed, with each remaining task being able to be allocated to a unique node and the remaining processing capacity of each node being not less than 0 as constraints, and with maximizing the task processing efficiency of the distributed database system as an optimization goal, based on the task overhead of each remaining task and the remaining processing capacity of each node, to determine the execution node of each remaining task among the multiple nodes.

[0044] In a possible embodiment of this specification, the method further includes:

[0045] A data query task is received, and the data query task is split into a plurality of sub-query tasks, wherein the plurality of sub-query tasks serve as a plurality of tasks to be assigned.

[0046] In a possible embodiment of this specification, the method further includes:

[0047] The execution results of multiple sub-query tasks belonging to the same query task are merged into the execution result of the query task.

[0048] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0049] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor implements the method as described in the first aspect by running the executable instructions.

[0053] According to a fifth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0054] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0055] The load balancing method provided in the embodiment of this specification obtains the task overhead of each task in the multiple tasks to be assigned and the remaining processing capacity of each node in the multiple nodes of the distributed database system; and takes each task as a constraint condition that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, takes the task processing efficiency of the distributed database system to be maximized as the optimization goal, and determines the execution node of each task in the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node; finally, each task is sent to the execution node of the task, so that at least one node in the multiple nodes executes the received task to be assigned. Each task can be assigned to a unique node, which can ensure that each task can have an execution node and that each task is not repeatedly assigned a node; the remaining processing capacity of each node is not less than 0, which can ensure that each node is not overloaded; therefore, the method can maximize the task processing efficiency of the distributed database system under the condition that each task can have an execution node, each task is not repeatedly assigned a node, and each node is not overloaded, that is, it can complete the data query task at the fastest speed, thereby achieving the two goals of task processing efficiency and node overload at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the structure of a distributed database provided by an exemplary embodiment.

[0057] Figure 2 It is a schematic diagram of a SQL execution distribution mechanism provided by an exemplary embodiment.

[0058] Figure 3 It is a flowchart of a load balancing method provided by an exemplary embodiment.

[0059] Figure 4 It is a flowchart of a load balancing method provided by another exemplary embodiment.

[0060] Figure 5 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0061] Figure 6 It is a block diagram of a load balancing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0063] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0064] Based on the technical problems raised in the background technology, at least one embodiment of this specification provides a load balancing method, which can be applied to a distributed database system, such as the attached Figure 1 The distributed database system based on the LSM-Tree storage engine shown in the figure can split the tasks such as data query tasks when they are received and distribute them among multiple nodes of the distributed database system, so that the data query tasks can be executed with maximum efficiency and each node can be guaranteed not to be overloaded, that is, the task overhead of all tasks executed by each node does not exceed its processing capacity.

[0065] Please refer to the attached Figure 2 , which graphically illustrates the content of the method, that is, the method provides a SQL execution distribution mechanism, which is used to distribute SQL (Structured Query Language) statements with different execution costs to nodes with different processing capabilities for execution.

[0066] Please refer to the attached Figure 3 , which exemplarily shows the process of the load balancing method, including steps S301 to S303.

[0067] In step S301 , the task overhead of each task among the multiple tasks to be assigned and the remaining processing capacity of each node among the multiple nodes of the distributed database system are obtained.

[0068] The multiple tasks to be assigned may be subtasks of a data query task received by the distributed database system. For example, the method may receive a data query task and split the data query task into multiple subquery tasks, wherein the multiple subquery tasks are used as multiple tasks to be assigned. The data query task received by the distributed database system may be represented by an SQL statement, and when an SQL execution plan is formulated for the above SQL statement, it may be split into multiple minimum units that can be executed independently, and these minimum units are subquery tasks.

[0069] The task overhead can be the processing power occupied by the node when executing the task, such as the occupied CPU, memory, storage, etc. Exemplarily, the step of obtaining the task overhead of each task in the multiple tasks to be assigned may include: determining the overhead of the task according to at least one of the following information of the task: task type, task content, and data targeted by the task. In other words, dimensions such as task type, task content, and data targeted by the task can all affect the task overhead, and the task overhead influencing factors of different options of each dimension can be determined by controlling variables for each dimension that affects the task overhead, and then in this step, the task overhead of the task is determined by the options of each dimension of each task.

[0070] The node of a distributed database system is a database instance of the distributed database system. The remaining processing capacity of a node is the task processing capacity that the node can provide; or it can be understood that the maximum processing capacity of a node when it is not processing any task is the remaining processing capacity at that time, and when the node is processing at least one task, the difference between the maximum processing capacity and the total task overhead of the processed task is the remaining processing capacity at that time. Exemplarily, the step of obtaining the remaining processing capacity of each of the multiple nodes of the distributed database system includes: determining the remaining processing capacity of the node according to the CPU state, memory state, and storage state of the node. In other words, dimensions such as CPU state, memory state, and storage state can all affect the processing capacity of the node. The processing capacity influencing factors of different options of each dimension can be determined by controlling variables for each dimension that affects the processing capacity, and then the remaining processing capacity of the node is determined by the options of each dimension of each node in this step.

[0071] In step S302, with the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, the execution node of each task is determined among the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node.

[0072] Among them, each task can be assigned to a unique node, which can ensure that each task can have an execution node and that each task is not repeatedly assigned to a node; the remaining processing capacity of each node is not less than 0, which can ensure that each node is not overloaded.

[0073] The maximization of the task processing efficiency of the distributed database system can ensure the highest processing effect of the data query tasks obtained by splitting the tasks to be assigned. For example, the maximization of the task processing efficiency of the distributed database system can be characterized by minimizing the remaining processing capacity of the distributed database system.

[0074] This step completes the allocation of multiple tasks to be assigned on multiple nodes of the distributed database system by maximizing the task processing efficiency of the distributed database system under constraints. During the allocation process, a task is assigned to a node, and the node is the execution node of the task. Optionally, this step can construct the above optimization process into a multiple knapsack problem MKP (Multiple Knapsack Problem) and optimize the MKP to complete the task allocation. The MKP refers to multiple backpacks, each with its own capacity limit. The optimization goal of the problem is to assign different items to these backpacks in order to maximize the total value of the items in all backpacks, while ensuring that the capacity limit of any backpack is not violated, and each item can only be assigned to one backpack, and each backpack cannot be overloaded.

[0075] The following example explains the process of building and optimizing MKP in detail.

[0076] First, each task is regarded as an item in the multi-knapsack problem MKP, the task overhead of the task is regarded as the weight of the corresponding item in the MKP, each node is regarded as a backpack in the MKP, the processing capacity of the node is regarded as the capacity of the corresponding backpack in the MKP, and the inverse of the remaining processing capacity of the distributed database system is regarded as the value of all backpacks in the MKP, so as to construct the MKP.

[0077] For example, the optimization objective of MKP is represented by the following formula:

[0078]

[0079] The constraints of MKP are represented by the following formula:

[0080]

[0081]

[0082] In the above formula, s iis the remaining processing capacity of the ith node, w j is the task cost of the jth task, W i is the total processing capacity of the ith node,

[0083] Next, the MKP is optimized based on the task overhead of each task and the remaining processing capacity of each node, and an execution node of each task is determined among the multiple nodes according to the optimization result.

[0084] By constructing and optimizing MKP in the above manner, the optimization of the optimization target in this step can be completed more conveniently, and the corresponding constraints can be met during the optimization process, so that the task allocation method provided by the method can be completed smoothly and the allocation efficiency and allocation effect can be improved.

[0085] In step S303, each task is sent to an execution node of the task, so that at least one node among the multiple nodes executes the received task to be assigned.

[0086] It should be understood that if the multiple sub-query tasks obtained by splitting the received data query task in step S201 are used as multiple tasks to be assigned, the method can also merge the execution results of multiple sub-query tasks belonging to the same query task into the execution result of the query task.

[0087] The load balancing method provided in this specification can maximize the task processing efficiency of the distributed database system while ensuring that each task has an execution node, each task is not repeatedly assigned to a node, and each node is not overloaded. That is, it can complete data query tasks at the fastest speed, thereby achieving the two goals of maximizing task processing efficiency and not overloading nodes at the same time.

[0088] In some embodiments of this specification, the method may also include: Figure 4 The following steps are shown:

[0089] Step S401: If the constraints are that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, and the optimization goal is to maximize the task processing efficiency of the distributed database system, based on the task overhead of each task and the remaining processing capacity of each node, if it is impossible to determine an execution node for each task among the multiple nodes, then at least one task among the multiple tasks is marked as priority allocation.

[0090] For example, a preset proportion (such as half or one third) of the tasks among multiple tasks are randomly marked as priority allocation.

[0091] For another example, the remaining processing capacity of the distributed database system is calculated, and the total task overhead of the tasks marked as priority allocation is made smaller than the remaining processing capacity of the distributed database system.

[0092] Step S402: With the constraints that each of the at least one task marked as priority allocation can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, based on the task overhead of each task in the at least one task marked as priority allocation and the remaining processing capacity of each node, determine the execution node of each task in the at least one task marked as priority allocation among the multiple nodes.

[0093] For example, according to the above Figure 3 This step is performed in the manner described in step S302:

[0094] First, each task is regarded as an item in the multi-knapsack problem MKP, the task overhead of the task is regarded as the weight of the corresponding item in the MKP, each node is regarded as a backpack in the MKP, the processing capacity of the node is regarded as the capacity of the corresponding backpack in the MKP, and the reciprocal of the remaining processing capacity of the distributed database system is regarded as the value of all backpacks in the MKP, so as to construct the MKP;

[0095] Next, the MKP is optimized based on the task overhead of each task in at least one task marked as priority allocation and the remaining processing capacity of each node, and the execution node of each task in at least one task marked as priority allocation is determined among the multiple nodes according to the optimization result.

[0096] Step S403: After a preset period of time has passed after determining the execution node of each task in at least one task marked as priority allocation, with the constraints that each remaining task can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, the execution node of each remaining task is determined among the multiple nodes based on the task overhead of each remaining task and the remaining processing capacity of each node.

[0097] For example, according to the above Figure 3 This step is performed in the manner described in step S302:

[0098] First, each task is regarded as an item in the multi-knapsack problem MKP, the task overhead of the task is regarded as the weight of the corresponding item in the MKP, each node is regarded as a backpack in the MKP, the processing capacity of the node is regarded as the capacity of the corresponding backpack in the MKP, and the reciprocal of the remaining processing capacity of the distributed database system is regarded as the value of all backpacks in the MKP, so as to construct the MKP;

[0099] Next, the MKP is optimized based on the task overhead of each remaining task and the remaining processing capacity of each node, and an execution node for each remaining task is determined from among the plurality of nodes according to the optimization result.

[0100] In this embodiment, if according to the attached Figure 3 The method shown cannot complete task allocation under the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, that is, the remaining processing capacity of the distributed database system cannot bear the task overhead of multiple tasks to be assigned. In this case, some tasks can be assigned first, and the remaining tasks can be assigned after a certain period of time, so as to asynchronously assign these tasks and maximize the processing efficiency of tasks. Figure 3 This step is performed in the manner described in step S302.

[0101] In the related art, when the remaining processing capacity of the distributed database system cannot bear the task overhead of multiple tasks to be assigned, the allocation is often tried again after a certain period of time until the remaining processing capacity of the distributed database system can bear the task overhead of multiple tasks to be assigned, and then the task allocation is completed to execute each task. However, this embodiment asynchronously allocates and executes multiple tasks in the above manner, thereby avoiding the waste of computing power of the distributed database system (in the related art, waiting before re-allocation will cause waste of existing computing power of the distributed database), and ensuring the task processing efficiency to the greatest extent.

[0102] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0103] Please refer to Figure 6 , the load balancing device can be used for Figure 5 The load balancing device may include:

[0104] An acquisition module 601 is used to acquire the task overhead of each task in the multiple tasks to be assigned and the remaining processing capacity of each node in the multiple nodes of the distributed database system;

[0105] An allocation module 602 is used to determine an execution node for each task from the plurality of nodes based on the task overhead of each task and the remaining processing capacity of each node, with each task being assignable to a unique node and the remaining processing capacity of each node being not less than 0 as constraints, and maximizing the task processing efficiency of the distributed database system as an optimization goal;

[0106] The sending module 603 is used to send each task to the execution node of the task, so that at least one node among the multiple nodes executes the received task to be assigned.

[0107] In a possible embodiment of the present specification, maximizing the task processing efficiency of the distributed database system is characterized by minimizing the remaining processing capacity of the distributed database system.

[0108] In a possible embodiment of this specification, the allocation module is used to:

[0109] The MKP is constructed by taking each task as an item in the multi-knapsack problem MKP, taking the task overhead of the task as the weight of the corresponding item in the MKP, taking each node as a backpack in the MKP, taking the processing capacity of the node as the capacity of the corresponding backpack in the MKP, and taking the reciprocal of the remaining processing capacity of the distributed database system as the value of all backpacks in the MKP;

[0110] The MKP is optimized based on the task overhead of each task and the remaining processing capacity of each node, and an execution node of each task is determined among the multiple nodes according to the optimization result.

[0111] In a possible embodiment of the present specification, the obtaining of the task overhead of each task in the plurality of tasks to be assigned includes:

[0112] The overhead of the task is determined according to at least one of the following information of the task: task type, task content, and data targeted by the task.

[0113] In a possible embodiment of the present specification, obtaining the remaining processing capacity of each of the multiple nodes of the distributed database system includes:

[0114] The remaining processing capacity of the node is determined according to the CPU status, memory status, and storage status of the node.

[0115] In a possible embodiment of this specification, the device further includes:

[0116] a priority marking module, for marking at least one of the multiple tasks as a priority assignment if, based on the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, the optimization goal is to maximize the task processing efficiency of the distributed database system, and if it is impossible to determine an execution node for each task among the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node;

[0117] A priority allocation module is used to determine the execution node of each task in at least one task marked as priority allocation among the multiple nodes based on the task overhead of each task in at least one task marked as priority allocation and the remaining processing capacity of each node, with the constraints that each task in at least one task marked as priority allocation can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal.

[0118] In a possible embodiment of this specification, the device further includes:

[0119] A compensation allocation module is used to determine the execution node of each remaining task in at least one task marked as priority allocation after a preset period of time has passed, with each remaining task being able to be allocated to a unique node and the remaining processing capacity of each node being not less than 0 as constraints, and with maximizing the task processing efficiency of the distributed database system as an optimization goal, based on the task overhead of each remaining task and the remaining processing capacity of each node, to determine the execution node of each remaining task among the multiple nodes.

[0120] In a possible embodiment of this specification, the method further includes:

[0121] A data query task is received, and the data query task is split into a plurality of sub-query tasks, wherein the plurality of sub-query tasks serve as a plurality of tasks to be assigned.

[0122] In a possible embodiment of this specification, the method further includes:

[0123] The execution results of multiple sub-query tasks belonging to the same query task are merged into the execution result of the query task.

[0124] One or more embodiments of the present specification also provide a computer program product, including a computer program / instruction, which implements the steps of the method provided in any of the above embodiments when executed by a processor.

[0125] One or more embodiments of the present specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method provided in any of the above embodiments.

[0126] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0127] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0128] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0129] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0131] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0133] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0134] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0135] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A load balancing method, the method comprising: Obtaining the task overhead of each of the multiple tasks to be assigned and the remaining processing capacity of each of the multiple nodes of the distributed database system; With constraints that each task can be assigned to a unique node and that the remaining processing capacity of each node is not less than 0, and with maximizing the task processing efficiency of the distributed database system as an optimization goal, determining an execution node for each task from the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node; Each task is sent to an execution node of the task, so that at least one node among the multiple nodes executes the received task to be assigned.

2. According to the load balancing method of claim 1, maximizing the task processing efficiency of the distributed database system is characterized by minimizing the remaining processing capacity of the distributed database system.

3. The load balancing method according to claim 2, wherein the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, the optimization goal is to maximize the task processing efficiency of the distributed database system, and based on the task overhead of each task and the remaining processing capacity of each node, determine the execution node of each task from the multiple nodes, including: The MKP is constructed by taking each task as an item in the multi-knapsack problem MKP, taking the task overhead of the task as the weight of the corresponding item in the MKP, taking each node as a backpack in the MKP, taking the processing capacity of the node as the capacity of the corresponding backpack in the MKP, and taking the reciprocal of the remaining processing capacity of the distributed database system as the value of all backpacks in the MKP; The MKP is optimized based on the task overhead of each task and the remaining processing capacity of each node, and an execution node of each task is determined among the multiple nodes according to the optimization result.

4. The load balancing method according to claim 1, wherein the step of obtaining the task overhead of each task in the plurality of tasks to be assigned comprises: The overhead of the task is determined according to at least one of the following information of the task: task type, task content, and data targeted by the task.

5. The load balancing method according to claim 1, wherein obtaining the remaining processing capacity of each of the multiple nodes of the distributed database system comprises: The remaining processing capacity of the node is determined according to the CPU status, memory status, and storage status of the node.

6. The load balancing method according to claim 1, further comprising: If each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0 as constraint conditions, and the task processing efficiency of the distributed database system is maximized as the optimization goal, and based on the task overhead of each task and the remaining processing capacity of each node, it is impossible to determine an execution node for each task among the multiple nodes, at least one task among the multiple tasks is marked as a priority assignment; With the constraints that each of the at least one task marked as priority allocation can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, based on the task overhead of each task in the at least one task marked as priority allocation and the remaining processing capacity of each node, the execution node of each task in the at least one task marked as priority allocation is determined among the multiple nodes.

7. The load balancing method according to claim 6, further comprising: After a preset period of time has passed after determining the execution node of each task in at least one task marked as priority allocation, with the constraints that each remaining task can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal, the execution node of each remaining task is determined among the multiple nodes based on the task overhead of each remaining task and the remaining processing capacity of each node.

8. The load balancing method according to claim 1, further comprising: A data query task is received, and the data query task is split into a plurality of sub-query tasks, wherein the plurality of sub-query tasks serve as a plurality of tasks to be assigned.

9. The load balancing method according to claim 8, further comprising: The execution results of multiple sub-query tasks belonging to the same query task are merged into the execution result of the query task.

10. A load balancing device, comprising: An acquisition module, used to acquire the task overhead of each task in the plurality of tasks to be assigned, and the remaining processing capacity of each node in the plurality of nodes of the distributed database system; An allocation module, configured to determine an execution node for each task from among the plurality of nodes based on the task overhead of each task and the remaining processing capacity of each node, with each task being assignable to a unique node and the remaining processing capacity of each node being not less than 0 as constraints, and maximizing the task processing efficiency of the distributed database system as an optimization goal; The sending module is used to send each task to the execution node of the task, so that at least one node among the multiple nodes executes the received task to be assigned. 11 . The load balancing device according to claim 10 , wherein maximizing the task processing efficiency of the distributed database system is characterized by minimizing the remaining processing capacity of the distributed database system.

12. The load balancing device according to claim 11, wherein the allocation module is used for: The MKP is constructed by taking each task as an item in the multi-knapsack problem MKP, taking the task overhead of the task as the weight of the corresponding item in the MKP, taking each node as a backpack in the MKP, taking the processing capacity of the node as the capacity of the corresponding backpack in the MKP, and taking the reciprocal of the remaining processing capacity of the distributed database system as the value of all backpacks in the MKP; The MKP is optimized based on the task overhead of each task and the remaining processing capacity of each node, and an execution node of each task is determined among the multiple nodes according to the optimization result.

13. The load balancing device according to claim 10, further comprising: a priority marking module, for marking at least one of the multiple tasks as a priority assignment if, based on the constraints that each task can be assigned to a unique node and the remaining processing capacity of each node is not less than 0, the optimization goal is to maximize the task processing efficiency of the distributed database system, and if it is impossible to determine an execution node for each task among the multiple nodes based on the task overhead of each task and the remaining processing capacity of each node; A priority allocation module is used to determine the execution node of each task in at least one task marked as priority allocation among the multiple nodes based on the task overhead of each task in at least one task marked as priority allocation and the remaining processing capacity of each node, with the constraints that each task in at least one task marked as priority allocation can be allocated to a unique node and the remaining processing capacity of each node is not less than 0, and with the maximization of the task processing efficiency of the distributed database system as the optimization goal.

14. The load balancing device according to claim 13, further comprising: A compensation allocation module is used to determine the execution node of each remaining task in at least one task marked as priority allocation after a preset period of time has passed, with each remaining task being able to be allocated to a unique node and the remaining processing capacity of each node being not less than 0 as constraints, and with maximizing the task processing efficiency of the distributed database system as an optimization goal, based on the task overhead of each remaining task and the remaining processing capacity of each node, to determine the execution node of each remaining task among the multiple nodes.

15. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

16. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

17. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Dynamic fragmentation method for three-dimensional matrix task, electronic equipment and medium

    CN122431846A