A method for dynamically and adaptively adjusting task sharding based on ElasticJob

Abstract

The present invention discloses a method for dynamically and adaptively adjusting task sharding based on ElasticJob. The custom sharding strategy in the present invention includes custom IPs, which allows the node instances to be sorted preferentially according to the instance IPs configured as specified; at the same time, the custom sharding strategy of the present invention also includes dynamic adaptation, which allows the node instances to be sorted according to the current task duration; through the custom sharding method of "fixed" plus "dynamic", the custom adjustment of the instance order is realized, which meets the requirement of evenly adjusting the tasks with serious time consumption to other instance machines for execution, further balancing the application instance resources, avoiding multiple tasks being pressed on the same machine, avoiding the risk of resource tension on a certain machine, avoiding the hidden danger of insufficient memory, improving the original random scheduling used in the market, making it closer to the actual usage scenario and more flexible.

Description

Technical Field

[0001] The present invention relates to the field of slicing of scheduled tasks, and in particular to a method for dynamically and adaptively adjusting task slicing based on elasticjob. Background Art

[0002] The task scheduling framework elasticJob supports parallel scheduling, distributed scheduling coordination, and elastic expansion. It is a distributed scheduling solution. ElasticJob achieves distributed scheduling through sharding. It is widely used in many business scenarios of the current Internet.

[0003] In the actual scenario, the monthly payable report scheduled task has only one shard, and the monthly income report scheduled task also has only one shard. It is possible that these two tasks are only run on one machine according to the default sharding rule of elasticjob, while other application machines are running idle. The ideal state is that if the two time-consuming tasks can be evenly distributed on the application machines, it will be more reasonable. The reason is that when there are multiple tasks, and each of them has only one shard with a long execution time, multiple tasks are run on a fixed machine, which increases the burden on this machine and may eventually cause the machine to run out of memory. However, several instance machines have been running idle, and the machine resource allocation is seriously unbalanced, and the purpose of resource balancing has not been achieved;

[0004] Based on the existing sharding mechanism of elasticjob, the default sharding mechanism is a sharding strategy based on the average distribution algorithm, which has a disadvantage: once the number of shards is less than the number of job servers, the job will always be assigned to the server with the first IP address, resulting in idle servers with the last IP address. The other two sharding modes redistribute server loads based on the hash value of the task name, but this has clear restrictions on the definition of the task name, which must be restricted by the task name hash value, and is not flexible enough to meet actual business scenarios. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a method for dynamically and adaptively adjusting task sharding based on ElasticJob. Based on ElasticJob, the order of the incoming default node instances is adjusted, and the order of the node instances is customized. The custom sharding strategy in the present invention includes custom IPs, which can prioritize the order of node instances according to the instance IPs configured as specified. At the same time, the custom sharding strategy of the present invention also includes dynamic adaptation, which can sort the order of node instances according to the current task duration. Through the "fixed" plus "dynamic" custom sharding method, the custom adjustment of the instance order is realized, which satisfies the balanced adjustment of time-consuming tasks to other instance machines, further balances the application instance resources, avoids multiple tasks being pressed on the same machine, avoids causing resource tension on a certain machine, and avoids the hidden danger of insufficient memory. By statistically analyzing the task duration in real time, the sharding strategy of the job is dynamically adjusted, and at the same time, it has the function of executing according to the specified application IP, improving the original random scheduling used in the market, making it more in line with the actual usage scenario and more flexible.

[0006] The present invention provides the following technical solutions:

[0007] The present invention provides a method for dynamically and adaptively adjusting task sharding based on ElasticJob, including the following steps:

[0008] I. Key database fields in the ER diagram of the task execution information table

[0009] 1). jobName: The task name, which refers to the name of each task and serves as the unique identifier in this table;

[0010] 2). jobCurrentIp: The current execution IP of the task, which refers to the application machine IP where the current task is executed;

[0011] 3). jobRuleType: The task rule type: dynamic adaptation; specified IP; default; for dynamic adaptation, the order of node instances can be sorted according to the current task duration; for custom IP, the order of node instances can be prioritized according to the instance IPs configured as specified;

[0012] 4). jobRuleValue: The task rule value: when the rule type is specified IP, the rule value here is the corresponding IP for task execution; when the rule type is dynamic adaptation, the rule value here corresponds to the application machine;

[0013] II. Initialization sequence diagram of the task execution information table

[0014] 1). When the task starts, query whether there is a record in the task information table according to the task name;

[0015] 2). Check if there is a record in the task information table. When there is no record, add a record to the task information table (add the current executing IP); when there is a record, update the record in the task information table (update the current executing IP).

[0016] 3). After the normal business execution of the task, update the record in the task information table (update the task duration).

[0017] III. Sequence Diagram of Custom Sharding Strategy

[0018] 1). Query the task information table according to the task name.

[0019] 2). When the sharding strategy is dynamic adaptation, first sort in descending order according to the task duration, then place the instances with slower duration in the priority execution queue in the order of node JobInstance instances, and finally place the instances with slower duration in the subsequent execution queue in the order of node JobInstance instances.

[0020] 3). When the sharding strategy is specified IP, first obtain the specified IP value in the task information table, then place the instances with the specified IP in the priority execution queue in the order of node JobInstance instances, and finally set the remaining JobInstance nodes.

[0021] 4). When the sharding strategy is default, perform sharding according to the existing rules of ElasticJob.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. It realizes the custom adjustment of the instance order, avoids multiple tasks being pressed on the same machine, and avoids resource tension on a certain machine.

[0024] 2. It meets the requirement of evenly adjusting the tasks with serious duration to other instance machines for execution.

[0025] 3. It balances the resources of application instances and avoids the hidden danger of memory shortage.

[0026] 4. By statistically analyzing the task duration in real time, the sharding strategy of the job is dynamically adjusted. At the same time, it has the function of specifying the application IP for execution, improving the original random scheduling in the market, making it more in line with the actual usage scenario and more flexible.

[0027] 5. Since the coupling of this invention with the product is relatively low, it can be applied to fields such as acquiring products, payment products, and financial products that require task debugging in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, but do not constitute a limitation to the present invention. In the accompanying drawings:

[0029] Figure 1 is the ER diagram of the task execution information table of the present invention;

[0030] Figure 2 is the initialization timing diagram of the task execution information table of the present invention

[0031] Figure 3 is the timing diagram of the custom sharding strategy of the present invention. Detailed implementation manners

[0032] The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention. Among them, the same reference numerals in the accompanying drawings all refer to the same components.

[0033] Embodiment 1

[0034] The key of the present invention is the incoming default node instance List <jobinstance>Adjust the order to customize the order of node instances.

[0035] For example Figures 1-3 , the present invention provides a method for dynamically and adaptively adjusting task sharding based on elasticjob, including the following steps:

[0036] I. Key database fields in the ER diagram of the task execution information table (as Figure 1 shown)

[0037] 1). jobName: The task name, which refers to the name of each task and serves as the unique identifier in this table;

[0038] 2). jobCurrentIp: The current execution IP of the task, which refers to the IP of the application machine where the current task is executed;

[0039] 3). jobRuleType: The task rule type: dynamic adaptation; specified IP; default; For dynamic adaptation, the order of node instances can be sorted according to the current task duration; for custom IP, the order of node instances can be sorted preferentially according to the specified instance IP;

[0040] 4). jobRuleValue: The task rule value: when the rule type is specified IP, the rule value here is the corresponding IP for task execution; when the rule type is dynamic adaptation, the rule value here corresponds to the application machine;

[0041] II. Initialization sequence diagram of the task execution information table (as Figure 2 shown)

[0042] 1). When the task starts, query the task information table according to the task name to check if there is a record;

[0043] 2). Determine whether there is a record in the task information table. When there is no record, add a record to the task information table (add the current execution IP); when there is a record, update the record in the task information table (update the current execution IP);

[0044] 3). After the normal business execution of the task, update the record in the task information table (update the task duration);

[0045] III. Sequence diagram of the custom sharding strategy (as Figure 3 shown)

[0046] 1). Query the task information table according to the task name;

[0047] 2). When the sharding strategy is dynamically adapted, first sort by task duration from largest to smallest, then sequentially place the JobInstance instances of the nodes with slower duration in the priority execution queue, and finally sequentially place the JobInstance instances of the nodes with slower duration in the subsequent execution queue;

[0048] 3). When the sharding strategy is specified IP, first obtain the specified IP value in the task information table, then sequentially place the JobInstance instances with the specified IP in the priority execution queue, and finally set the remaining JobInstance nodes;

[0049] 4). When the sharding strategy is default, sharding is performed according to the existing rules of ElasticJob.

[0050] Furthermore, it is as follows:

[0051] 1). A new task execution information table with the task name as the unique identifier is added, which contains the task duration time to distinguish which tasks have a long execution time. The rule types of the tasks are dynamically adapted; specified IP; default. For dynamic adaptation, the node instances can be sorted sequentially according to the current task duration; for custom IP, the node instances can be sorted preferentially according to the specified configured instance IP;

[0052] 2). After each task execution is completed, the duration in the task execution information table is updated, so that it is possible to know in real time and dynamically which tasks have a long execution time;

[0053] Based on ElasticJob, implement the JobShardingStrategy interface and override the sharding method. When performing sharding, query the task execution information table, retrieve the sharding strategy type inside, and perform different sharding strategies according to different values of the sharding strategy type;

[0054] 3). When the sharding strategy is dynamically adapted, first sort by task duration from largest to smallest, then sequentially place the JobInstance instances of the nodes with slower duration in the priority execution queue, and finally sequentially place the JobInstance instances of the nodes with slower duration in the subsequent execution queue. Assume that there are three machines for the current node instances, and the default order at system load is A, B, C. In the task execution information table, the task monthSumJob is completed on machine A and takes 600 seconds; the task weekSumJob is completed on machine C and takes 680 seconds. The sharding strategy of the newly configured task daySumJob is dynamically adapted. After the sharding strategy is adjusted, the node instance List <jobinstance>The order will be adjusted from the default A, B, C when the system is loaded to B, A, C. Eventually, the task daySumJob with only one shard will be executed in instance B where resources are idle;

[0055] 4). When the sharding strategy is to specify an IP, first obtain the specified IP value in the task information table, then the node JobInstance instance order will place the instance with the specified IP in the priority execution queue, and finally set the remaining JobInstance nodes. Assume that there are three machines for the current node instances, and the default order is A, B, C. The sharding strategy for the newly configured task hourSumJob is to specify an IP. After the sharding strategy is adjusted, the priority of the instance C corresponding to the specified IP will be adjusted, that is, the node instance List <jobinstance>The order will be adjusted from the default A, B, C when the system is loaded to C, A, B. Eventually, the hourSumJob task with only one shard will be executed in instance C where resources are idle.

[0056] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.< / jobinstance> < / jobinstance> < / jobinstance>

Claims

1. A method for dynamically and adaptively adjusting task sharding based on ElasticJob, characterized in that, it includes the following steps: I. Set the key database fields of the ER diagram of the task execution information table 1). jobName (task name), which refers to the name of each task and serves as the unique identifier in this table; 2). jobCurrentIp (task current execution IP), which refers to the application machine IP where the current task is executed; 3). jobRuleType (task rule type): dynamic adaptation; specified IP; default; for dynamic adaptation, the node instance order can be sorted according to the current task duration; for custom IP, the node instance order can be sorted preferentially according to the specified instance IP; 4). jobRuleValue (task rule value): when the rule type is specified IP, the rule value is the IP corresponding to the task execution; when the rule type is dynamic adaptation, the rule value takes the application machine; II. Sequence diagram for initializing the task execution information table 1). When the task starts, query whether there is a record in the task information table according to the task name; 2). Judge whether there is a record in the task information table. When there is no record, add a record to the task information table and add the IP of the current execution machine; when there is a record, update the record in the task information table and update the IP of the current execution machine; 3). After the business corresponding to the task is executed, update the record in the task information table and update the task duration; III. Sequence diagram for custom sharding strategy 1). Query the task information table according to the task name; 2). When the sharding strategy is dynamic adaptation, first sort from large to small according to the task duration, then place the instances with long duration in the node JobInstance instance order in the priority execution queue, and finally place the instances with short duration in the node JobInstance instance order in the subsequent execution queue; 3). When the sharding strategy is specified IP, first obtain the specified IP value in the task information table, then place the instances with the specified IP in the node JobInstance instance order in the priority execution queue, and finally set the execution queue for the remaining JobInstance nodes; 4). When the sharding strategy is default, sharding is performed according to the default rules of ElasticJob.

Images