A load balancing method for distributed jobs, electronic equipment and storage medium

Abstract

The application discloses a load balancing method for distributed jobs, an electronic device and a storage medium. The load balancing method for distributed jobs is applied to a metadata service node and comprises the following steps: determining a trigger and a default execution node of the trigger meeting a preset condition, and acquiring resource usage of all execution nodes; determining a target execution node from all the execution nodes according to the resource usage, and establishing dynamic distribution information of the trigger and the target execution node in a preset global trigger execution chain table; and sending a trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes a job task corresponding to the trigger. According to the embodiment of the application, the dynamic distribution of the execution node of the trigger is realized, the problem of long job execution time caused by the unbalanced load of the execution node is prevented, and the execution efficiency of the job task is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a load balancing method, electronic device, and storage medium for distributed operations. Background Technology

[0002] A Distributed Processing Cluster (DPC) consists of three parts: a plan generation node (SQLProcessor, SP), a data storage node (Backend Processor, BP), and a metadata server node (MetadataProcessor, MP). The SP receives client requests and generates execution plans; the BP stores user data, executes the SP's scheduling instructions, and returns the execution results to the SP; the MP stores metadata and provides metadata services to the SP and BP.

[0003] In the DPC system, when creating a job, an execution node can be specified (the execution node can be any of the SP nodes). The job will then be executed on the specified execution node. If no execution node is specified, the SP node with the smallest available node number will be selected for execution. If an execution node is specified, the job will always execute on the same node once created. With a large number of jobs, some node servers may experience high resource usage, leading to load imbalance and longer job execution times. If no execution node is specified, more jobs will execute on the SP nodes with smaller node numbers, also resulting in load imbalance and longer job execution times. Therefore, load balancing of execution nodes has become a pressing issue. Summary of the Invention

[0004] This invention provides a load balancing method, electronic device, and storage medium for distributed jobs to solve the problems of uneven load on execution nodes and long job execution time.

[0005] According to one aspect of the present invention, a load balancing method for distributed jobs is provided, wherein the method is applied to a metadata service node and includes:

[0006] Determine if the trigger and its default execution node meet preset conditions, and obtain the resource usage of all execution nodes;

[0007] Based on the resource usage, determine the target execution node among all the execution nodes, and establish dynamic distribution information between the trigger and the target execution node in the preset global trigger execution chain list;

[0008] The trigger identifier of the trigger is sent to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger.

[0009] According to another aspect of the present invention, a load balancing method for distributed jobs is provided, wherein the method applied to the target execution node includes:

[0010] Receive the trigger identifier sent by the metadata service node, and add the trigger identifier to the preset local trigger execution list;

[0011] The job tasks corresponding to the triggers are executed according to the order of the trigger identifiers in the preset local trigger execution list.

[0012] According to another aspect of the present invention, a load balancing device for distributed operations is provided, wherein the device is applied to a metadata service node and includes:

[0013] The status acquisition module is used to determine whether the trigger and the default execution node of the trigger meet the preset conditions, and to acquire the resource usage status of all execution nodes.

[0014] The information generation module is used to determine the target execution node among all the execution nodes based on the resource usage, and to establish dynamic distribution information between the trigger and the target execution node in a preset global trigger execution chain list;

[0015] The job distribution module is used to send the trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger.

[0016] According to another aspect of the present invention, a load balancing device for distributed operations is provided, wherein the device applied to a target execution node includes:

[0017] The identifier receiving module is used to receive trigger identifiers sent by the metadata service node and add the trigger identifiers to a preset local trigger execution list;

[0018] The job execution module is used to execute the job tasks corresponding to the triggers according to the order of the trigger identifiers in the preset local trigger execution chain.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the load balancing method for distributed jobs as described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the load balancing method for distributed jobs as described in any embodiment of the present invention.

[0024] The technical solution of this invention determines that the trigger and the default execution node of the trigger meet preset conditions, obtains the resource usage of all execution nodes, determines the target execution node among all execution nodes based on the resource usage, and establishes dynamic distribution information between the trigger and the target execution node in a preset global trigger execution chain. The trigger identifier of the trigger is sent to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger. This realizes the dynamic distribution of the execution node of the trigger, prevents the problem of long job execution time caused by uneven load of execution nodes, improves the execution efficiency of job tasks, and enhances the user experience.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a load balancing method for distributed jobs provided according to Embodiment 1 of the present invention;

[0028] Figure 2 This is a flowchart of another load balancing method for distributed jobs provided according to Embodiment 2 of the present invention;

[0029] Figure 3 This is a flowchart of another load balancing method for distributed jobs provided according to Embodiment 3 of the present invention;

[0030] Figure 4This is a flowchart of another load balancing method for distributed jobs provided according to Embodiment 4 of the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of a distributed job load balancing device provided in Embodiment 5 of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of a distributed job load balancing device according to Embodiment Six of the present invention;

[0033] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the load balancing method for distributed jobs according to embodiments of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "default," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] It should be noted that in a distributed computing cluster (DPC), only one MP node can be configured, and all metadata information is stored on that MP; multiple BP nodes can be configured, with different user data stored on each BP node; multiple SP nodes can be configured, and connecting to any SP node can obtain complete database services.

[0037] In database management, for tasks with fixed processes, such as regularly backing up the database or regularly generating data statistics reports, these can be configured as jobs and executed on a schedule using triggers.

[0038] Example 1

[0039] Figure 1 This is a flowchart of a distributed job load balancing method according to Embodiment 1 of the present invention, applied to a metadata service node. This embodiment is applicable to the case of load balancing of job tasks of triggers. The method can be executed by a distributed job load balancing device, which can be implemented in hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0040] S110. Determine if the trigger and the default execution node of the trigger meet the preset conditions, and obtain the resource usage of all execution nodes.

[0041] A trigger is a special stored procedure associated with table events in a database. Its execution is not initiated by a program or manually, but rather triggered by an event. For example, when an operation (insert, delete, or update) is performed on a table in the database, the trigger can be activated. A trigger is a database object associated with a table; when a specified event occurs on the table containing the trigger and the defined conditions are met, the set of statements defined in the trigger will be executed. Each trigger has a default execution node.

[0042] The default execution node refers to the default execution node of the trigger. In actual operation, the default execution node can be the plan generation node. The execution node refers to the node that executes the job task of the trigger, and can be any plan generation node. There can be an association between the trigger and the default execution node. For example, the trigger identifier of the trigger can be associated and stored with the node identifier of the default execution node, and the corresponding default execution node can be queried by the trigger identifier.

[0043] Resource usage can be understood as the usage and resource consumption of execution nodes. For example, resource usage may include, but is not limited to, the utilization rate of the Central Processing Unit (CPU) and memory consumption.

[0044] In one embodiment, determining that the trigger and its default execution node meet preset conditions includes: the default execution node of the trigger is a load balancing allowed node; and the trigger has not been assigned.

[0045] In this context, a load balancing allow node can be understood as a node that allows the job task associated with a trigger to be distributed to other execution nodes for execution. Load balancing allow nodes may have a special identifier, such as "AT_ALL_SP$". In one embodiment, the creation of this special identifier may include setting the execution node to a special identifier during the creation of the specified execution node for the job task, indicating that the job task can be dynamically distributed to any execution node for execution. When a job task specifies a special identifier, the execution node of the trigger is automatically set to this special identifier when the trigger is created for the job task; alternatively, the special identifier can be directly set for the specified execution node.

[0046] In this embodiment of the invention, the default execution node of the trigger can be determined, as well as the node type of the default execution node. Simultaneously, the allocation status of the trigger can be determined. When the default execution node of the trigger is a load-balanced allowed node and the trigger is not allocated, it can be determined that the trigger and its default execution node meet preset conditions. At this point, the resource usage of all execution nodes can be extracted.

[0047] In actual operation, after a trigger is assigned to an execution node, the trigger and its corresponding execution node can be stored in a preset global trigger execution list to indicate that the trigger has been assigned. The preset global trigger execution list can be retrieved to check if a trigger identifier exists. If a trigger identifier exists in the preset global trigger execution list, it is determined that the trigger does not meet the preset conditions; if no trigger identifier exists in the preset global trigger execution list, it is determined that the trigger meets the preset conditions. When both the trigger and its corresponding default execution node meet the preset conditions, the resource usage of all execution nodes is obtained. In one embodiment, the CPU utilization and memory consumption of all execution nodes can be determined as the resource usage.

[0048] S120. Determine the target execution node among all execution nodes based on resource usage, and establish dynamic distribution information between triggers and target execution nodes in the preset global trigger execution chain.

[0049] The pre-defined global trigger execution list can be understood as a pre-created linked list storing the dynamic distribution information of all triggers and target execution nodes in the distributed computing cluster. Dynamic distribution information can be understood as information indicating that a trigger has been published. In actual operation, dynamic distribution information can be the association between target execution nodes and triggers. For example, the association between the node identifier of the target execution node and the trigger identifier of the trigger can be established as dynamic distribution information.

[0050] In this embodiment, the resource usage of all execution nodes can be compared to determine the execution node with the lowest resource usage as the target node. In actual operation, the execution node with the lowest CPU utilization and memory consumption among all execution nodes can be selected as the target execution node. Alternatively, a load balancing algorithm, such as Weighted Random Scheduling or Processing Capacity Scheduling, can be used to select the optimal execution node as the target execution node. The node identifier of the target execution node is then determined, and the association between the trigger identifier and the node identifier is established as dynamic distribution information, stored in a preset global trigger execution list. In one embodiment, the trigger identifier may include a trigger identity document (ID); the node identifier may include a node number.

[0051] S130. Send the trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger.

[0052] In this context, a job task can be understood as a task that is executed periodically via triggers. For example, a job task may include, but is not limited to, periodically backing up the database and periodically generating data statistics reports. For tasks in the database whose processes remain unchanged, they can be configured as a job task.

[0053] In this embodiment, dynamic distribution information associated with the trigger can be determined, node identifiers can be extracted from the dynamic distribution information, and trigger identifiers can be sent to the target execution node according to the node identifiers, so that the target execution node executes the job task corresponding to the trigger. In actual operation, the corresponding dynamic distribution information can be determined according to the trigger identifier of the trigger, the node identifier of the target execution node can be queried, and trigger identifiers can be sent to the target execution node according to the node identifiers.

[0054] In this embodiment of the invention, by determining that the trigger and the default execution node of the trigger meet preset conditions, the resource usage of all execution nodes is obtained. Then, based on the resource usage, the target execution node among all execution nodes is determined, and dynamic distribution information between the trigger and the target execution node is established in a preset global trigger execution chain. The trigger identifier of the trigger is sent to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger. This realizes the dynamic distribution of the execution node of the trigger, prevents the problem of long job execution time caused by uneven load of execution nodes, improves the execution efficiency of job tasks, and enhances the user experience.

[0055] In one embodiment, after sending the trigger identifier of the trigger to the target execution node according to the dynamic distribution information, the method further includes:

[0056] Receive execution completion information from the target execution node;

[0057] Remove the dynamic dispatch information from the preset global trigger execution chain based on the execution completion information.

[0058] The execution completion information can be understood as the completion information returned by the target execution node after completing the job task, in order to notify the metadata service node that the job task has been completed.

[0059] In this embodiment of the invention, upon receiving execution completion information from the target execution node, the trigger identifier contained in the execution completion information can be determined. The dynamic distribution information associated with the trigger identifier is then queried from the preset global trigger execution list, and the corresponding dynamic distribution information is removed from the preset global trigger execution list.

[0060] In one embodiment, the load balancing method for distributed jobs further includes:

[0061] If it is determined that the default execution node of the trigger does not meet the load balancing requirement, the trigger identifier of the trigger will be sent to the default execution node so that the default execution node can execute the job task corresponding to the trigger.

[0062] In an embodiment of the invention, when the default execution node of the trigger is not a load balancing allowed node, it can be assumed that the job task corresponding to the trigger is not allowed to be distributed to other execution nodes. The trigger identifier of the trigger can be sent to the default execution node so that the default execution node can execute the job task corresponding to the trigger.

[0063] In one embodiment, when the metadata service node fails, if the metadata service node is a single node, after the metadata service node restarts, a preset global trigger execution list is recreated, preset local trigger execution lists on all execution nodes are collected, and all trigger identifiers and corresponding execution node identifiers are added to the preset global trigger execution list.

[0064] When a backup node is configured on the metadata service node, after a master-slave switch occurs, the new master node will recreate a preset global trigger execution list, collect the preset local trigger execution lists on all execution nodes, and add all trigger identifiers and corresponding execution node identifiers to the preset global trigger execution list.

[0065] Example 2

[0066] Figure 2This is a flowchart of another load balancing method for distributed jobs according to Embodiment 2 of the present invention. This embodiment is a further optimization and extension based on the above embodiments, and can be combined with various optional technical solutions in the above embodiments, such as... Figure 2 As shown, the method includes:

[0067] S210. Determine that the trigger and the default execution node of the trigger meet the preset conditions, and obtain the resource usage of all execution nodes.

[0068] S220. Compare the resource usage across all execution nodes; the monitoring items for resource usage should include at least: CPU utilization and memory consumption.

[0069] In this embodiment of the invention, the memory of the execution nodes can be read to determine the resource usage of each execution node. In actual operation, all monitoring items related to the resource usage of each execution node can be determined, such as CPU utilization and memory consumption, and the same monitoring items across all execution nodes can be compared. For example, the CPU utilization and memory consumption of all execution nodes can be compared separately, and the values ​​of CPU utilization and memory consumption for each execution node can be compared to determine the target execution node.

[0070] S230. Select the execution node with the smallest value in the resource usage monitoring item among all execution nodes as the target execution node.

[0071] In the embodiments of the invention, after comparing the resource usage of all execution nodes, the execution node with the smallest value in the resource usage monitoring item can be selected as the target execution node. In one embodiment, the execution node with the smallest CPU utilization and memory consumption among all execution nodes can be selected as the target execution node; alternatively, a weighted random load balancing scheduling algorithm, a processing capacity load balancing scheduling algorithm, or the like can be used to select the optimal execution node as the target execution node.

[0072] S240, Extract the node identifier of the target execution node and the trigger identifier of the trigger.

[0073] Here, the node identifier can be understood as identification information that distinguishes execution nodes; for example, the node identifier may include the node number. The trigger identifier can be understood as identification information that indicates the trigger; for example, the trigger identifier may include the trigger ID.

[0074] In the embodiments of the invention, the corresponding node identifier and trigger identifier can be extracted from the memory of the target execution node and the trigger, respectively.

[0075] S250. Establish the association between node identifiers and trigger identifiers as dynamic distribution information, and store the dynamic distribution information in the preset global trigger execution list.

[0076] In the embodiments of the invention, the association between node identifier and trigger identifier can be determined, and the association can be stored as dynamic distribution information in a preset global trigger execution chain.

[0077] S260, Read the dynamic dispatch information of the preset global trigger execution list.

[0078] S270. Determine the trigger identifier and node identifier stored in the dynamic distribution information.

[0079] In the embodiments of the invention, dynamic distribution information can be extracted to determine the trigger identifier and node identifier stored in the dynamic distribution information.

[0080] S280. Send the trigger identifier to the target execution node according to the node identifier, so that the target execution node executes the job task corresponding to the trigger.

[0081] In this embodiment of the invention, by determining that the trigger and its default execution node meet preset conditions, the resource usage of all execution nodes is obtained. Comparing the resource usage of all execution nodes, the execution node with the smallest monitored value for resource usage is selected as the target execution node. The node identifier of the target execution node and the trigger identifier of the trigger are extracted, and a relationship between the node identifier and the trigger identifier is established as dynamic distribution information. This dynamic distribution information is stored in a preset global trigger execution list, allowing for a clearer view of the trigger distribution. By reading the dynamic distribution information from the preset global trigger execution list, and then determining the trigger identifier and node identifier stored in the dynamic distribution information, the trigger identifier is sent to the target execution node according to the node identifier. This enables the execution of the job task corresponding to the trigger through the target execution node, while simultaneously achieving dynamic distribution of the trigger's execution node.

[0082] Example 3

[0083] Figure 3 This is a flowchart of another load balancing method for distributed jobs according to Embodiment 3 of the present invention, applied to the target execution node. This embodiment can be applied to the case of load balancing of job tasks of triggers, such as... Figure 3 As shown, the method includes:

[0084] S310. Receive the trigger identifier sent by the metadata service node and add the trigger identifier to the preset local trigger execution list.

[0085] The preset local trigger execution list can be understood as a pre-created linked list of triggers stored in the target execution node to execute job tasks. The corresponding job tasks can be executed according to the order of the trigger identifiers in the preset local trigger execution list.

[0086] In an embodiment of the invention, after obtaining the trigger identifier sent by the metadata service node, the trigger identifier can be added to a preset local trigger execution list to execute the job task corresponding to the trigger.

[0087] S320. Execute the job tasks corresponding to the triggers according to the order of the trigger identifiers in the preset local trigger execution chain.

[0088] In the embodiments of the invention, the order of trigger identifiers in the preset local trigger execution chain can be determined, and the job tasks corresponding to the triggers can be executed sequentially according to the order of the trigger identifiers.

[0089] In one embodiment, the target execution node obtains the first trigger ID in the preset local trigger execution chain, queries the specific trigger information based on the trigger ID, and the trigger information may include the job identifier corresponding to the trigger, and executes the job task.

[0090] In this embodiment of the invention, by receiving trigger identifiers sent by metadata service nodes and adding the trigger identifiers to a preset local trigger execution list, the corresponding job tasks are executed according to the order of the trigger identifiers in the preset local trigger execution list, thereby achieving orderly execution of job tasks and improving the execution efficiency of job tasks.

[0091] In one embodiment, the load balancing method for distributed jobs further includes:

[0092] After the job task corresponding to the trigger is completed, the execution completion information is sent to the metadata service node, and the trigger identifier is removed from the preset local trigger execution chain.

[0093] The execution completion information includes at least the trigger identifier.

[0094] In this embodiment, after the job task corresponding to the trigger is completed, execution completion information with the trigger identifier can be generated and sent to the metadata service node to notify the metadata service node that the job task corresponding to the trigger has been completed. Simultaneously, the trigger identifier is removed from the preset local trigger execution chain to prevent duplicate execution of the job task.

[0095] In one embodiment, when the metadata service node fails, if the metadata service node is a single node, after the metadata service node restarts, a preset global trigger execution list is recreated, preset local trigger execution lists on all execution nodes are collected, and all trigger identifiers and corresponding execution node identifiers are added to the preset global trigger execution list.

[0096] When a metadata service node is configured with a backup node, after a master-slave switch, the new master node will recreate a preset global trigger execution list, collect preset local trigger execution lists from all execution nodes, and add all trigger identifiers and their corresponding execution node identifiers to the preset global trigger execution list. In one embodiment, because the metadata service node stores important metadata information, the DPC system supports configuring multiple backup nodes for the metadata service node to ensure system reliability. When the master node of the metadata service node fails, a node is selected from the backup nodes to serve as the new master node and continue providing services.

[0097] In one embodiment, when the metadata service node detects a failure in the execution node, the metadata service node will remove all dynamic distribution information corresponding to the failed execution node from the preset global trigger execution chain.

[0098] If an execution node fails and restarts quickly (the preset local trigger execution list on the execution node will be empty after restarting), the execution node will send a message to the metadata service node to notify the metadata service node that the current execution node has restarted. The metadata service node will then remove all dynamic distribution information corresponding to that execution node from the preset global trigger execution list (if the information does not exist in the preset global trigger execution list, it will not be removed).

[0099] Example 4

[0100] Figure 4 This is a flowchart of another load balancing method for distributed jobs according to Embodiment 4 of the present invention. This embodiment, based on the above embodiments, uses MP nodes as metadata service nodes and SP nodes as execution nodes. Taking the trigger ID as the trigger identifier, the SP node number as the execution node identifier, and the target SP node as the target execution node as an example, this further illustrates a load balancing method for distributed jobs. Figure 4 As shown, the method includes:

[0101] S410, MP node checks if there is a special flag in the default execution node of the trigger; if so, execute S420; otherwise, execute S430.

[0102] In this embodiment, when the default execution node has a special marker, it can be considered as a load balancing allowed node.

[0103] In one embodiment, when creating a job task and specifying an execution node, the execution node can be set to a special flag, such as "AT_ALL_SP$", indicating that the job task can be dynamically distributed to any SP node for execution. If a job task specifies a special flag, the execution node of the trigger will be automatically set to that special flag when a trigger is created for the job task.

[0104] S420. When the MP node checks that there is no trigger ID in the preset global trigger execution chain, the MP node monitors and collects the resource usage of each SP node and determines the target SP node according to the resource usage.

[0105] Resource usage includes, but is not limited to, CPU utilization and memory consumption. Load balancing algorithms, such as Weighted Random Scheduling or Processing Capacity Scheduling, can be used to select the optimal SP node as the target SP node.

[0106] In this embodiment, if the trigger ID exists in the preset global trigger execution chain, it means that the trigger has been dispatched and the corresponding job task has not yet been completed, so it will not be dispatched again.

[0107] S430. Send the trigger ID of the trigger to the default execution node so that the default execution node executes the job task corresponding to the trigger.

[0108] S440 and MP nodes send the trigger ID to the target SP node and add a dynamic distribution information entry to the preset global trigger execution chain, recording the trigger ID and the SP node number of the target SP node.

[0109] When the S450 and SP nodes receive the trigger ID sent by the MP node, they add the trigger ID to the preset local trigger execution list and send the execution completion information back to the MP node after the job task is executed.

[0110] In one embodiment, the SP node will sequentially execute the job tasks corresponding to the triggers in the preset local trigger execution chain. After each job task is completed, it will send a message to the MP node to notify the MP node that the job task has been completed.

[0111] In one embodiment, the SP node obtains the first trigger ID from a preset local trigger execution list, retrieves the specific trigger information based on the trigger ID, and the trigger information may include the job identifier corresponding to the trigger. The SP node then executes the job task. After the job task is completed, the SP node sends execution completion information to the MP node, notifying the MP node that the job execution is complete. This execution completion information includes the trigger ID. After sending the execution completion information, the SP node removes the current trigger ID from the preset local trigger execution list and then obtains the next trigger ID to continue executing the job.

[0112] When S460 and MP nodes receive a message from SP node, they confirm that the SP node's job task has been completed and remove the dynamic distribution information corresponding to the corresponding trigger ID from the preset global trigger execution list.

[0113] In one embodiment, when an MP node fails, if the MP node is a single node, after the MP node restarts, the global trigger execution list is recreated, the preset local trigger execution lists on all SP nodes are collected, and all trigger IDs and corresponding SP node numbers are added to the preset global trigger execution list.

[0114] When an MP node is configured with a backup node, after a master-slave switch, the new master node will recreate a preset global trigger execution list, collect preset local trigger execution lists from all SP nodes, and add all trigger IDs and corresponding SP node numbers to the preset global trigger execution list. Because MP nodes store important metadata, the DPC system supports configuring multiple backup nodes for MP nodes to ensure system reliability. When the MP master node fails, a node is selected from the MP backup nodes to continue providing service as the new master node.

[0115] In one embodiment, when an MP node detects a failure in an SP node, the MP node removes all dynamic distribution information corresponding to the failure SP node from the preset global trigger execution list.

[0116] If an SP node fails and restarts quickly (the preset local trigger execution list on the SP node is empty after restarting), the SP node will send a message to the MP node to notify the MP node that the current SP node has performed a restart operation. The MP node will then remove all dynamic distribution information corresponding to that SP node from the preset global trigger execution list (if the information does not exist in the preset global trigger execution list, it will not be removed).

[0117] Example 5

[0118] Figure 5This is a schematic diagram of a distributed job load balancing device according to Embodiment 5 of the present invention, applied to a metadata service node. Figure 5 As shown, the load balancing device for the distributed job includes: a situation acquisition module 51, an information generation module 52, and a job distribution module 53.

[0119] Among them, the situation acquisition module 51 is used to determine whether the trigger and the default execution node of the trigger meet the preset conditions, and to acquire the resource usage of all execution nodes.

[0120] The information generation module 52 is used to determine the target execution node among all execution nodes based on resource usage, and to establish dynamic distribution information between triggers and target execution nodes in the preset global trigger execution chain list;

[0121] The job distribution module 53 is used to send the trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger.

[0122] In this embodiment of the invention, the status acquisition module determines that the trigger and the default execution node of the trigger meet preset conditions, acquires the resource usage of all execution nodes, and the information generation module determines the target execution node among all execution nodes based on the resource usage. It then establishes dynamic distribution information between the trigger and the target execution node in a preset global trigger execution chain. The job distribution module sends the trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes the job task corresponding to the trigger. This achieves dynamic distribution of the trigger's execution node, preventing the problem of long job execution time caused by uneven load on execution nodes, improving the execution efficiency of job tasks, and enhancing the user experience.

[0123] In one embodiment, the situation acquisition module 51 includes:

[0124] The node condition determination unit is used to determine the default execution node of the trigger as a load balancing allowed node;

[0125] Trigger condition determination unit, used when the trigger is not assigned.

[0126] In one embodiment, the information generation module 52 includes:

[0127] The data comparison unit is used to compare the resource usage across all execution nodes; the monitoring items for resource usage include at least: CPU utilization and memory consumption.

[0128] The node determination unit is used to select the execution node with the smallest value of the resource usage monitoring item among all execution nodes as the target execution node.

[0129] The identifier extraction unit is used to extract the node identifier of the target execution node and the trigger identifier of the trigger.

[0130] The information establishment unit is used to establish the association between node identifiers and trigger identifiers as dynamic distribution information, and stores the dynamic distribution information in the preset global trigger execution linked list.

[0131] In one embodiment, the job distribution module 53 includes:

[0132] The information reading unit is used to read the dynamic distribution information of the preset global trigger execution list; the identifier determination unit is used to determine the trigger identifier and node identifier stored in the dynamic distribution information.

[0133] The job distribution unit is used to send the trigger identifier to the target execution node according to the node identifier, so that the target execution node can execute the job task corresponding to the trigger.

[0134] In one embodiment, the load balancing device for distributed jobs further includes:

[0135] The information receiving module is used to receive execution completion information fed back by the target execution node;

[0136] The dynamic information removal module is used to remove dynamically distributed information from the preset global trigger execution chain according to the execution completion information.

[0137] In one embodiment, the load balancing device for distributed jobs further includes:

[0138] The task distribution module is used to send the trigger identifier to the default execution node when the default execution node of the trigger does not meet the load balancing requirement, so that the default execution node can execute the job task corresponding to the trigger.

[0139] The distributed job load balancing device provided in the embodiments of the present invention can execute the distributed job load balancing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] Example 6

[0141] Figure 6 This is a schematic diagram of a load balancing device for distributed operations according to Embodiment Six of the present invention, applied to a target execution node. Figure 6 As shown, the load balancing device for the distributed job includes: an identifier receiving module 61 and a job execution module 62.

[0142] Among them, the identifier receiving module 61 is used to receive the trigger identifier sent by the metadata service node and add the trigger identifier to the preset local trigger execution list;

[0143] The job execution module 62 is used to execute the job tasks corresponding to the triggers according to the order of the trigger identifiers in the preset local trigger execution chain.

[0144] In this embodiment of the invention, the identifier receiving module receives the trigger identifier sent by the metadata service node and adds the trigger identifier to a preset local trigger execution list. The job execution module executes the job tasks corresponding to the triggers according to the order of the trigger identifiers in the preset local trigger execution list, thereby realizing the orderly execution of job tasks and improving the execution efficiency of job tasks.

[0145] In one embodiment, the load balancing device for distributed jobs further includes:

[0146] The data removal module is used to send execution completion information to the metadata service node after the job task corresponding to the trigger is completed, and to remove the trigger identifier from the preset local trigger execution chain; wherein, the execution completion information includes at least the trigger identifier.

[0147] The distributed job load balancing device provided in the embodiments of the present invention can execute the distributed job load balancing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0148] Example 7

[0149] Figure 7 This is a schematic diagram of an electronic device implementing the load balancing method for distributed jobs according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0150] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as load balancing methods for distributed jobs.

[0153] In some embodiments, the distributed job load balancing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the distributed job load balancing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the distributed job load balancing method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0160] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A load balancing method for distributed jobs, characterized in that, Applied to a metadata service node, comprising: Determine that a trigger and a default execution node of the trigger meet preset conditions, and acquire resource usage of all execution nodes; Determine a target execution node from all execution nodes according to the resource usage, and establish dynamic distribution information of the trigger and the target execution node in a preset global trigger execution linked list; Send a trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes a job task corresponding to the trigger; The method further comprises: Compare the resource usage of all execution nodes; the monitoring items of the resource usage at least include a central processing unit usage rate and memory consumption; Take an execution node with the minimum value of the monitoring items of the resource usage as a target execution node; Extract a node identifier of the target execution node and a trigger identifier of the trigger; Establish an association relationship between the node identifier and the trigger identifier as dynamic distribution information, and store the dynamic distribution information in the preset global trigger execution linked list.

2. The method of claim 1, wherein, The method further comprises: The default execution node of the trigger is a load balancing allowed node; The trigger is not assigned.

3. The method of claim 1, wherein, The method further comprises: Read the dynamic distribution information of the preset global trigger execution linked list; determine a trigger identifier and a node identifier stored in the dynamic distribution information; Send the trigger identifier to a target execution node according to the node identifier, so that the target execution node executes a job task corresponding to the trigger.

4. The method of claim 1, wherein, After the trigger identifier of the trigger is sent to the target execution node according to the dynamic distribution information, the method further comprises: Receive execution completion information fed back by the target execution node; Remove the dynamic distribution information from the preset global trigger execution linked list according to the execution completion information.

5. The method of claim 1, wherein, The load balancing method of the distributed job further comprises: When it is determined that the default execution node of the trigger does not meet the load balancing condition, send a trigger identifier of the trigger to the default execution node, so that the default execution node executes a job task corresponding to the trigger.

6. A load balancing method for distributed jobs, characterized in that, Applied to a target execution node, comprising: Receive a trigger identifier sent by a metadata service node, and add the trigger identifier to a preset local trigger execution linked list; Execute a job task corresponding to the trigger according to an arrangement order of the trigger identifier in the preset local trigger execution linked list; The trigger identifier is determined by the metadata service node, and the trigger and the default execution node of the trigger meet preset conditions. Resource usage of all execution nodes is obtained, and the resource usage of all execution nodes is compared. The monitoring items of the resource usage include at least a central processing unit usage rate and memory consumption. An execution node with the minimum value of the monitoring items of the resource usage of all the execution nodes is taken as a target execution node. A node identifier of the target execution node and a trigger identifier of the trigger are extracted, an association between the node identifier and the trigger identifier is established as dynamic distribution information, the dynamic distribution information is stored in a preset global trigger execution linked list, and the trigger identifier of the trigger is sent to the target execution node according to the dynamic distribution information.

7. The method of claim 6, wherein, The load balancing method of the distributed job further includes: After the job task corresponding to the trigger is executed, execution completion information is fed back to the metadata service node, and the trigger identifier is removed from the preset local trigger execution linked list; The execution completion information at least includes a trigger identifier.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the load balancing method of the distributed job in any one of claims 1-5 or claims 6-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the load balancing method of the distributed job in any one of claims 1-5 or claims 6-7 when executed.

Images