Distributed cluster scheduling method and system based on decentration

Abstract

The invention discloses a decentration-based distributed cluster scheduling method and system, and relates to the field of distributed technologies. The method comprises the following steps: acquiring a resource use condition and a message queue corresponding to a total node in a current distributed system through a main node, and performing fragmentation processing on the message queue based on a preset fragmentation strategy, the resource use condition and a preset adjustment strategy to generate a target task scheduling strategy; determining a corresponding target scheduling task based on a target task scheduling strategy through a target node, executing the target scheduling task based on a preset time window mechanism to generate a heartbeat report and state information, and sending the heartbeat report and the state information to a main node; and summarizing the heartbeat report and the state information corresponding to the target scheduling task through the main node to generate a global scheduling summarizing report. Through the technical scheme of the invention, the pressure of the central scheduler can be obviously reduced, and the real-time performance, the flexibility and the resource utilization rate of the distributed system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed technology, and particularly relates to a distributed cluster scheduling method and system based on decentralization. BACKGROUND

[0002] With the rapid development of information technology and the increasing demand for computing, high-frequency second-level timing tasks are gradually applied to distributed systems to realize application scenarios that require fast response.

[0003] In the existing distributed system, high-frequency second-level timing tasks are usually managed by a centralized scheduler. Specifically, high-frequency second-level timing tasks are triggered by timing, and then distributed to each execution node for processing by the central scheduler.

[0004] However, high-frequency second-level timing tasks usually bring great pressure to the central scheduler, affecting its stability and response speed. Moreover, frequent task scheduling and state synchronization may cause network communication to become a performance bottleneck, reducing the real-time performance, flexibility and resource utilization of the system. Therefore, how to significantly reduce the pressure on the central scheduler and improve the real-time performance, flexibility and resource utilization of the distributed system is a problem to be solved at present. SUMMARY

[0005] The present application provides a distributed cluster scheduling method and system based on decentralization, which can solve the scheduling problem of high-frequency second-level timing tasks in the existing distributed system.

[0006] According to an aspect of the present application, a distributed cluster scheduling method based on decentralization is provided, comprising:

[0007] The master node obtains the resource usage and message queue of all nodes in the current distributed system, and performs sharding processing on the message queue based on a preset sharding strategy, resource usage and preset adjustment strategy, to generate a target task scheduling strategy corresponding to the current distributed system; wherein the target task scheduling strategy includes a target scheduling task corresponding to a target node in the current distributed system.

[0008] The target node determines the corresponding target scheduling task based on the target task scheduling strategy, executes the target scheduling task based on a preset time window mechanism, generates a heartbeat report and state information corresponding to the target scheduling task, and sends the heartbeat report and state information corresponding to the target scheduling task to the master node.

[0009] The master node processes the heartbeat report and state information corresponding to the target scheduling task to generate a global scheduling summary report corresponding to the current distributed system.

[0010] According to another aspect of the present application, there is provided a decentralized distributed cluster scheduling system, comprising:

[0011] a master node configured to obtain resource usage and message queues of all nodes in a current distributed system, and perform sharding processing on the message queues based on a preset sharding strategy, resource usage and a preset adjustment strategy, to generate a target task scheduling strategy corresponding to the current distributed system; wherein the target task scheduling strategy comprises target scheduling tasks corresponding to target nodes in the current distributed system;

[0012] a target node configured to determine target scheduling tasks corresponding to the target task scheduling strategy, execute the target scheduling tasks based on a preset time window mechanism, generate heartbeat reports and state information corresponding to the target scheduling tasks, and send the heartbeat reports and state information corresponding to the target scheduling tasks to the master node;

[0013] the master node configured to aggregate the heartbeat reports and state information corresponding to the target scheduling tasks, to generate a global scheduling aggregation report corresponding to the current distributed system.

[0014] The technical solution of the embodiment of the present application obtains resource usage and message queues of all nodes in a current distributed system through a master node, and performs sharding processing on the message queues based on a preset sharding strategy, resource usage and a preset adjustment strategy, to generate a target task scheduling strategy corresponding to the current distributed system. Then, a target node determines target scheduling tasks corresponding to the target task scheduling strategy, executes the target scheduling tasks based on a preset time window mechanism, generates heartbeat reports and state information corresponding to the target scheduling tasks, and sends the heartbeat reports and state information corresponding to the target scheduling tasks to the master node. Finally, the master node aggregates the heartbeat reports and state information corresponding to the target scheduling tasks, to generate a global scheduling aggregation report corresponding to the current distributed system. Due to the introduction of real-time dynamic sharding technology and localized task scheduling mechanism, the scheduling problem of high-frequency second-level timing tasks in the existing distributed system is solved, which can significantly reduce the pressure on the central scheduler and improve the real-time performance, flexibility and resource utilization of the distributed system.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to make the technical solution in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0017] Figure 1 is a flow chart of a decentralized distributed cluster scheduling method according to the first embodiment of the present application;

[0018] Figure 2 is a flow chart of a decentralized distributed cluster scheduling method according to the second embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a decentralized distributed cluster scheduling flow according to the second embodiment of the present application;

[0020] Figure 4 is a structural schematic diagram of a decentralized distributed cluster scheduling system according to the third embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the technical solution in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0022] It should be noted that the terms "target", "basis" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] It is worth noting that in the technical solution of the present application, the collected information is information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal; if the user chooses to refuse, enter the expert decision-making process.

[0024] Embodiment one

[0025] Figure 1 A flowchart of a decentralized distributed cluster scheduling method provided for the first embodiment of the present application, the present embodiment can be applicable to the scheduling of high-frequency second-level timing tasks in a distributed system, and the method can be executed by a decentralized distributed cluster scheduling system, which can be realized in the form of hardware and / or software. As shown in Figure 1 The method comprises:

[0026] S110, obtaining the resource usage and message queue of all nodes in the current distributed system through the master node, and performing sharding processing on the message queue based on a preset sharding strategy, resource usage and a preset adjustment strategy to generate a target task scheduling strategy corresponding to the current distributed system; wherein the target task scheduling strategy comprises a target scheduling task corresponding to a target node in the current distributed system.

[0027] The master node can refer to a core working node responsible for decision-making, coordination and management in a distributed system. For example, a working executor. In an embodiment of the application, the master node can be used to allocate task shards. In an optional implementation, before the master node obtains the resource usage and message queue of all nodes in the current distributed system, the method can further include: determining the master node of the current distributed system based on a preset node creation rule through a temporary node. The current distributed system can refer to a distributed system selected for high-frequency second-level timing task scheduling at the current time. In general, the current distributed system can be determined according to actual application requirements. The temporary node can refer to a technical means for implementing master node election. The preset node creation rule can refer to a rule predefined for limiting the master node election process. For example, the preset node creation rule can be to create a temporary node in a predefined path for all nodes in the current distributed system, and determine the node with a successful creation status as the master node. Specifically, before the master node obtains the resource usage and message queue of all nodes in the current distributed system, the master node can be generated through the state election of the temporary node created by all nodes in the current distributed system. Thus, an effective basis can be provided for subsequent operations.

[0028] The resource usage can refer to the real-time state of various hardware and software resources consumed by each node in the current distributed system during runtime. For example, the resource usage can include computing resource usage, memory resource usage, storage resource usage, network resource usage, and application layer resource usage, etc. The message can refer to information used to structure and display tasks that need to be scheduled. In other words, the message is a structured data block containing all information required for task execution. In general, one message corresponds to one task, and the message can include the unique identifier of the task corresponding to the message, task execution information, task parameters, and metadata, etc. The message queue can refer to a collection of various messages corresponding to the same current distributed system within a specified time period.

[0029] The preset sharding strategy can refer to a rule preset for limiting the sharding and partitioning process of the message queue. For example, the preset sharding strategy can include specific implementation steps and operation objects of the sharding and partitioning process. The preset adjustment strategy can refer to a rule preset for checking and adjusting the preliminary generated scheduling strategy. In an optional implementation, the preset adjustment strategy can include a preset failover strategy, a preset priority scheduling strategy, and a preset resource dynamic allocation strategy. The preset failover strategy can refer to a strategy preset for adjusting the node failure condition. For example, the preset failover strategy can be to reassign tasks in the failed node to other nodes. The preset priority scheduling strategy can refer to a strategy preset for scheduling tasks according to priority. For example, the preset priority scheduling strategy can be to schedule high-priority tasks to an independent queue to allocate more thread resources. The preset priority scheduling strategy can also be to promote the priority of a low-priority task if the low-priority task is waiting for more than 30 seconds, to prevent starvation, etc. The preset resource dynamic allocation strategy can refer to a strategy preset for expanding according to the current running state of the distributed system. For example, the preset resource dynamic allocation strategy can include a horizontal expansion strategy or a vertical expansion strategy. The task scheduling strategy can refer to the allocation result generated after the message tasks in the message queue are scheduled and allocated. Generally, the task scheduling strategy can include the physical mapping relationship between the message tasks and the corresponding allocation nodes. The target task scheduling strategy can refer to the final generated task scheduling strategy corresponding to the current distributed system.

[0030] In S120, the target node determines a corresponding target scheduling task based on the target task scheduling strategy, executes the target scheduling task based on a preset time window mechanism, generates a heartbeat report and state information corresponding to the target scheduling task, and sends the heartbeat report and state information corresponding to the target scheduling task to the master node.

[0031] Among them, the target node may refer to the node selected to perform the scheduling task in the current distributed system. In an optional embodiment, the target node includes: the master node and the slave nodes in the full node except the master node. The slave node may refer to the auxiliary node that executes the master node instructions in the current distributed system. Equivalently, in an embodiment of the present invention, the master node is responsible for generating the target task scheduling policy corresponding to the current distributed system, and is required to perform the corresponding tasks in accordance with the target task scheduling policy, while the slave node only needs to perform the corresponding tasks in accordance with the target task scheduling policy. It is worth noting that in the same distributed system, the master node and the remaining slave nodes together constitute the full node. During the working process of the master node, the remaining slave nodes monitor the temporary node corresponding to the master node. If the master node fails, the temporary node will be deleted, and the remaining slave nodes can re-trigger the election process to determine a new master node to ensure the operating efficiency of the system.

[0032] In an optional embodiment, the target node may include: a task processing unit and a heartbeat reporting module. The task processing unit may refer to a unit used to actually process the task. For example, a unit that performs actual data processing. Typically, a slave node may include multiple task processing units. The heartbeat reporting module may refer to a module used to periodically send survival status to the master node. Typically, each slave node includes a heartbeat reporting module.

[0033] Among them, the target scheduling task may refer to the task that matches the target node in the target task scheduling strategy. That is, the task assigned to the target node in the target task scheduling strategy. The preset time window mechanism may refer to a pre-set time value for limiting the task completion time. Exemplarily, the preset time window mechanism may be 10 seconds. Thus, by introducing a time window, tasks can be allowed to be completed within a set time range, reducing dependence on precise scheduling and improving system fault tolerance. The heartbeat report may refer to information used to indicate the survival status of a node. Exemplarily, the heartbeat report may include lightweight data such as a unique node identifier, a timestamp, and a basic survival signal. Status information may refer to information used to indicate the detailed operating status of a node. Exemplarily, status information may include data such as memory, task list, queue depth, and error log.

[0034] S130. The master node aggregates the heartbeat report and status information corresponding to the target scheduling task to generate a global scheduling summary report corresponding to the current distributed system.

[0035] The aggregation processing can refer to operations of aggregating heartbeat reports and state information corresponding to the same target scheduling task, and aggregating heartbeat reports and state information corresponding to different target scheduling tasks. The global scheduling aggregation report can refer to a scheduling report generated after the heartbeat reports and state information corresponding to all nodes in the current distributed system are subjected to the aggregation processing.

[0036] The technical scheme of the embodiment of the application obtains the resource usage and the message queue corresponding to all nodes in the current distributed system through the master node, performs the sharding processing on the message queue based on the preset sharding strategy, the resource usage and the preset adjustment strategy, generates the target task scheduling strategy corresponding to the current distributed system, determines the target scheduling task corresponding to the target node based on the target task scheduling strategy, executes the target scheduling task based on the preset time window mechanism, generates the heartbeat report and the state information corresponding to the target scheduling task, and sends the heartbeat report and the state information corresponding to the target scheduling task to the master node. Finally, the master node aggregates the heartbeat report and the state information corresponding to the target scheduling task, and generates the global scheduling aggregation report corresponding to the current distributed system. Due to the introduction of the real-time dynamic sharding technology and the localized task scheduling mechanism, the scheduling problem of the high-frequency second-level timing task in the existing distributed system is solved, the pressure on the central scheduler is significantly reduced, and the real-time performance, flexibility and resource utilization of the distributed system are improved.

[0037] Embodiment Two

[0038] Figure 2 A flowchart of a distributed cluster scheduling method based on decentralization is provided for the second embodiment of the application. The embodiment is refined based on the above-mentioned embodiment. In the embodiment, the operation of performing the sharding processing on the message queue based on the preset sharding strategy, the resource usage and the preset adjustment strategy, and generating the target task scheduling strategy corresponding to the current distributed system is refined. Specifically, the operation can include: performing the sharding processing on the message queue based on the preset sharding strategy and the resource usage through the task management module, and generating the basic task scheduling strategy corresponding to the current distributed system; and dynamically adjusting the basic task scheduling strategy based on the preset adjustment strategy through the task scheduling module, and generating the target task scheduling strategy corresponding to the current distributed system. As shown in FIG. 8, the method includes: Figure 2

[0039] S210, obtaining the resource usage and the message queue corresponding to all nodes in the current distributed system through the master node.

[0040] ​In an optional embodiment, the master node may include: a task management module, a task scheduling module, and a status report generation module. The task management module may refer to a module in the master node for managing tasks. For example, the task management module may implement operations such as task creation, allocation, and monitoring. The task scheduling module may refer to a module in the master node for task distribution. The status report generation module may refer to a module in the master node for generating summary reports.

[0041] In an optional embodiment, obtaining a message queue through the master node may include: receiving a message queue corresponding to the current distributed system through a central scheduler, and sending the message queue to the master node. The central scheduler may refer to a core coordination component in a distributed system. Specifically, when the central scheduler receives an external request, it may create a message queue based on the external request and send the message queue to the master node so that the master node can assign tasks to the message queue. Thus, by sharing the task assignment work of the central scheduler with the master node, the pressure on the central scheduler can be reduced and the stability and response speed of the central scheduler can be improved.

[0042] S220. Slice the message queue based on a preset slicing strategy and resource usage through a task management module to generate a basic task scheduling strategy corresponding to the current distributed system.

[0043] The basic task scheduling strategy may refer to a preliminarily generated task scheduling strategy corresponding to the current distributed system.

[0044] In an optional embodiment, the task management module performs sharding processing on the message queue based on a preset sharding strategy and resource usage to generate a basic task scheduling strategy corresponding to the current distributed system, including:

[0045] Step a1: shard the message queue based on a preset hash sharding strategy through a task management module to obtain a logical shard set corresponding to the message queue; wherein the logical shard set includes a target logical shard.

[0046] The preset hash sharding strategy can refer to a preset strategy for logically sharding the message queue. For example, the preset hash sharding strategy can be to distribute tasks corresponding to task unique identifiers with the same hash value to the same shard according to hash values of the task unique identifiers in the message queue. The logical shard can refer to a virtual shard divided in a logical level. For example, in an embodiment of the present application, the logical shard can be 128. Generally, one logical shard contains all tasks with the same hash value. The logical shard set can refer to a set composed of logical shards corresponding to the same message queue. The target logical shard can refer to a logical shard selected for strategy generation in the logical shard set. For example, the target logical shard can be any one of the logical shards in the logical shard set.

[0047] In step a2, the task management module dynamically maps the logical shard set based on the preset weighted round robin algorithm and the resource usage, determines the physical queues corresponding to the logical shard set, and takes the mapping relationship between the target logical shard and the corresponding physical nodes in the physical queues as the basic task scheduling strategy.

[0048] The preset weighted round robin algorithm can refer to a preset rule for limiting the dynamic round robin distribution process. For example, the preset weighted round robin algorithm can be to dynamically map the target logical shard in the logical shard set according to the weight. The physical node can refer to an actually existing worker node in the current distributed system. For example, the physical node can be a master node, or a slave node other than the master node. The physical queue can refer to a set composed of various physical nodes corresponding to the same current distributed system.

[0049] Specifically, after obtaining the message queue corresponding to the current distributed system through the master node, the message queue can be logically divided using hash values of unique identifiers of various messages in the message queue to generate a logical shard set corresponding to the message queue. Then, the target logical shard in the logical shard set is dynamically mapped to the physical queue based on the resource usage through the preset weighted round robin algorithm. In this way, by splitting a large number of messages in the message queue into multiple small blocks and distributing them to multiple queues, dynamic real-time sharding can be achieved, which can enable the system to quickly adapt to changes in business requirements and achieve load balancing.

[0050] In an optional embodiment, the task management module dynamically maps the logical shard set based on a preset weighted polling algorithm and resource usage to determine the physical queue corresponding to the logical shard set, including: adjusting the basic weight corresponding to the full node based on the resource usage corresponding to the full node through the task management module to determine the target weight corresponding to the full node; dynamically mapping the logical shard set based on a preset weighted polling algorithm and the target weight through the task management module to determine the physical queue corresponding to the logical shard set.

[0051] Among them, the basic weight may refer to the weight ratio relationship between physical nodes in a pre-set physical queue. Exemplarily, the basic weight may be the weight ratio relationship obtained after initialization. Generally, the weight may reflect the processing capability of the physical node, and the higher the weight, the stronger the processing capability. For example, taking a physical queue containing three physical nodes A, B and C, with a corresponding weight of 3:2:1 as an example, the dynamic mapping process may be: the 1st to 3rd logical shards are allocated to A, the 4th to 5th logical shards are allocated to B, the 6th logical shard is allocated to C, the 7th to 9th logical shards are allocated to A, and the cycle continues. The target weight may refer to the final weight obtained after adjusting the basic weights corresponding to all nodes based on resource usage.

[0052] Specifically, after generating the logical shard set corresponding to the message queue, the basic weight corresponding to the full node can be adjusted based on the resource usage corresponding to the full node to determine the target weight corresponding to the full node. Afterwards, the logical shard set is dynamically mapped based on the preset weighted polling algorithm and the target weight to determine the physical queue corresponding to the logical shard set. For example, taking the physical queue containing three physical nodes A, B and C, the corresponding basic weight is 3:2:1, and the resource usage is that the load of physical node B suddenly increases as an example, the weight of physical node B can be reduced from 2 to 1, and the weight of physical node C can be increased from 1 to 2, and the generated target weight can be 3:1:2. Then, according to the dynamic mapping process: the 1st to 3rd logical shards are allocated to A, the 4th logical shard is allocated to B, the 5th to 6th logical shards are allocated to C, the 7th to 9th logical shards are allocated to A, and the cycle continues. Each logical shard in the logical shard set is mapped to the physical queue. Therefore, by adjusting the basic weights based on resource usage, the dynamic mapping process can quickly adapt to changes in business needs and provide an effective basis for subsequent operations.

[0053] S230: Dynamically adjust the basic task scheduling strategy based on a preset adjustment strategy through a task scheduling module to generate a target task scheduling strategy corresponding to the current distributed system.

[0054] Specifically, after generating the basic task scheduling strategy corresponding to the current distributed system, the basic task scheduling strategy can be dynamically adjusted in sequence according to the preset failover strategy, the preset priority scheduling strategy and the preset resource dynamic allocation strategy in the preset adjustment strategy. For example, in the preset failover strategy, the state of each physical node in the physical queue can be obtained based on the heartbeat mechanism of the database, and the tasks of the abnormal node can be redistributed to the healthy node. In the preset priority scheduling strategy, messages of different priorities can be adjusted to the corresponding physical nodes according to the pre-divided task priority. In the preset resource dynamic allocation strategy, resource expansion can be performed according to the preset threshold, system hardware resources and message queue state. For example, the system hardware resource is the CPU utilization, the CPU utilization threshold > 80% triggers expansion, the memory occupancy rate and the disk IO waiting time; the message queue state is: the number of backlog tasks and the backlog task threshold > 1000 triggers dynamic sharding expansion. When the system hardware resources and the message queue state reach the preset threshold, the node resource configuration can be temporarily improved for key task shards, such as increasing the CPU core number from 4 cores to 8 cores, to realize vertical expansion. Horizontal expansion can also be realized by adding new executor nodes to the physical queue.

[0055] S240, asynchronously pulling, by the task processing unit, a target scheduling task corresponding to the target node based on the current node load condition of the target node and the target task scheduling strategy.

[0056] The node load condition can refer to information indicating the pressure and capacity saturation of the current processing task of the node. The current node load condition can refer to the real-time load condition of the target node at the current time. For example, the current node load condition can include queue length, response delay, and task backlog amount. Asynchronous pulling can refer to a non-blocking task acquisition mechanism. Generally, after the target node initiates a pulling request, it does not need to wait synchronously for data to return, but processes the data when it is ready through callback, event notification or polling.

[0057] Specifically, after generating the target task scheduling strategy corresponding to the current distributed system, the target node can asynchronously pull the target scheduling task corresponding to the target node in the target task scheduling strategy from the master node according to the corresponding current node load condition. For example, if the number of target scheduling tasks corresponding to the target node in the target task scheduling strategy is 40, and the current node load condition can execute 25 tasks, then 20 target scheduling tasks can be asynchronously pulled each time. In this way, the efficiency of task scheduling can be improved by asynchronously pulling the scheduling tasks.

[0058] S250, the target node generates heartbeat report and state information corresponding to the target scheduling task based on the preset time window mechanism, and sends the heartbeat report and state information corresponding to the target scheduling task to the master node.

[0059] Specifically, after the task processing unit asynchronously pulls the target scheduling task corresponding to the target node based on the current node load condition of the target node and the target task scheduling strategy, the target scheduling task can be executed based on the preset time window mechanism. Further, the heartbeat report module in the target node generates a heartbeat report corresponding to the target scheduling task, and sends the heartbeat report to the master node. In addition, the state information, such as time consumption, success or failure, is written into the local log in a batch submission manner, and the local log is sent to the master node through the channel. Thus, by autonomously completing task pulling, execution and state feedback on the executor node, local task scheduling can be realized, and the dependence on the central scheduler is reduced.

[0060] S260, the master node processes the heartbeat report and state information corresponding to the target scheduling task to generate a global scheduling summary report corresponding to the current distributed system.

[0061] The technical scheme of the embodiment of the application acquires the resource usage and message queue corresponding to all nodes in the current distributed system through the master node. Then, the task management module performs sharding processing on the message queue based on the preset sharding strategy and resource usage to generate a basic task scheduling strategy corresponding to the current distributed system. The task scheduling module dynamically adjusts the basic task scheduling strategy based on the preset adjustment strategy to generate a target task scheduling strategy corresponding to the current distributed system. Further, the task processing unit asynchronously pulls the target scheduling task corresponding to the target node based on the current node load condition of the target node and the target task scheduling strategy. The target node executes the target scheduling task based on the preset time window mechanism to generate a heartbeat report and state information corresponding to the target scheduling task, and sends the heartbeat report and state information corresponding to the target scheduling task to the master node. Finally, the master node processes the heartbeat report and state information corresponding to the target scheduling task to generate a global scheduling summary report corresponding to the current distributed system. Due to the introduction of real-time dynamic sharding technology and local task scheduling mechanism, the scheduling problem of high-frequency second-level timing tasks in the existing distributed system is solved, the pressure on the central scheduler is significantly reduced, and the real-time performance, flexibility and resource utilization of the distributed system are improved.

[0062] Figure 3As shown in the schematic diagram of the distributed cluster scheduling process based on decentralization provided by the embodiment of the application. Specifically, first, the central scheduler receives the message queue corresponding to the current distributed system, and sends the message queue to the master node. Then, the task management module in the master node performs sharding processing on the message queue based on the preset sharding strategy and resource usage, generates the basic task scheduling strategy corresponding to the current distributed system, and dynamically adjusts the basic task scheduling strategy based on the preset adjustment strategy through the task scheduling module in the master node, to generate the target task scheduling strategy corresponding to the current distributed system. Further, the task processing unit in the target node asynchronously pulls the target scheduling task corresponding to the target node based on the current node load condition of the target node and the target task scheduling strategy, and executes the target scheduling task based on the preset time window mechanism. Wherein, the target node includes: the master node and the slave node in the full node except the master node. Further, the heartbeat report module in the target node generates the heartbeat report corresponding to the target scheduling task, and sends the heartbeat report to the master node, and sends the state information to the master node through the channel. Finally, the state report generation module in the master node processes the heartbeat report and the state information corresponding to the target scheduling task, generates the global scheduling summary report corresponding to the current distributed system, and sends it to the central scheduling module.

[0063] Embodiment three

[0064] Figure 4 The structure schematic diagram of the distributed cluster scheduling system based on decentralization provided by the third embodiment of the application. As shown in the figure, the system includes a master node 310 and a target node 320. Figure 4

[0065] The master node 310 is configured to obtain the resource usage and message queue of the full node in the current distributed system, and perform sharding processing on the message queue based on the preset sharding strategy, resource usage and preset adjustment strategy, to generate the target task scheduling strategy corresponding to the current distributed system. The target task scheduling strategy includes the target scheduling task corresponding to the target node 320 in the current distributed system.

[0066] The target node 320 is configured to determine the corresponding target scheduling task based on the target task scheduling strategy, execute the target scheduling task based on the preset time window mechanism, generate the heartbeat report and state information corresponding to the target scheduling task, and send the heartbeat report and state information corresponding to the target scheduling task to the master node 310.

[0067] The master node 310 is configured to process the heartbeat report and state information corresponding to the target scheduling task, and generate the global scheduling summary report corresponding to the current distributed system. ​

[0068] According to the technical solution of the embodiment of the present invention, the master node 310 is used to obtain the resource usage and message queues corresponding to all nodes in the current distributed system, and to perform sharding processing on the message queue based on the preset sharding strategy, resource usage and preset adjustment strategy, and to generate a target task scheduling strategy corresponding to the current distributed system. Afterwards, the target node 320 is used to determine the corresponding target scheduling task based on the target task scheduling strategy, execute the target scheduling task based on the preset time window mechanism, generate a heartbeat report and status information corresponding to the target scheduling task, and send the heartbeat report and status information corresponding to the target scheduling task to the master node 310. Finally, the master node 310 is used to summarize the heartbeat report and status information corresponding to the target scheduling task, and generate a global scheduling summary report corresponding to the current distributed system. Due to the introduction of real-time dynamic sharding technology and localized task scheduling mechanism, the scheduling problem of high-frequency second-level scheduled tasks in the existing distributed system is solved, which can significantly reduce the pressure on the central scheduler and improve the real-time performance, flexibility and resource utilization of the distributed system.

[0069] Optionally, the decentralized distributed cluster scheduling system may also include: a temporary node, which is used to determine the master node 310 corresponding to the current distributed system among the full nodes of the current distributed system based on preset node creation rules before obtaining the resource usage and message queue corresponding to the full nodes in the current distributed system through the master node.

[0070] Optionally, the master node 310 may specifically include: a task management module and a task scheduling module;

[0071] The task management module is used to perform sharding on the message queue based on a preset sharding strategy and resource usage, and generate a basic task scheduling strategy corresponding to the current distributed system;

[0072] The task scheduling module is used to dynamically adjust the basic task scheduling strategy based on a preset adjustment strategy to generate a target task scheduling strategy corresponding to the current distributed system.

[0073] Optionally, the task management module can be specifically used to: shard the message queue based on a preset hash sharding strategy to obtain a logical shard set corresponding to the message queue; wherein the logical shard set includes a target logical shard; dynamically map the logical shard set based on a preset weighted polling algorithm and resource usage, determine the physical queue corresponding to the logical shard set, and use the mapping relationship between the target logical shard and the corresponding physical node in the physical queue as the basic task scheduling strategy.

[0074] Optionally, the task management module can be specifically configured to: adjust a basic weight corresponding to the full-amount node based on a resource usage of the full-amount node, and determine a target weight corresponding to the full-amount node; and dynamically map the logical shard set based on a preset weighted round robin algorithm and the target weight, and determine a physical queue corresponding to the logical shard set.

[0075] Optionally, the preset adjustment strategy can include a preset failover strategy, a preset priority scheduling strategy and a preset resource dynamic allocation strategy.

[0076] Optionally, the target node 320 can specifically include a task processing unit.

[0077] The task processing unit is configured to asynchronously pull a target scheduling task corresponding to the target node 320 based on a current node load condition of the target node 320 and the target task scheduling strategy.

[0078] Optionally, the decentralized distributed cluster scheduling system can further include a central scheduler configured to receive a message queue corresponding to a current distributed system, and send the message queue to the master node 310.

[0079] Optionally, the target node can include a master node and a slave node other than the master node in the full-amount node.

[0080] The decentralized distributed cluster scheduling system provided by the embodiments of the present application can execute the decentralized distributed cluster scheduling method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0081] It should be understood that various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0082] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A distributed cluster scheduling method based on decentralization, characterized in that: include: Obtain resource usage and message queues corresponding to all nodes in the current distributed system through the master node, and perform sharding on the message queue based on a preset sharding strategy, resource usage, and a preset adjustment strategy to generate a target task scheduling strategy corresponding to the current distributed system; wherein the target task scheduling strategy includes a target scheduling task corresponding to a target node in the current distributed system; The target node determines the corresponding target scheduling task based on the target task scheduling strategy, executes the target scheduling task based on a preset time window mechanism, generates a heartbeat report and status information corresponding to the target scheduling task, and sends the heartbeat report and status information corresponding to the target scheduling task to the master node; The master node aggregates and processes the heartbeat reports and status information corresponding to the target scheduling tasks to generate a global scheduling summary report corresponding to the current distributed system.

2. The method according to claim 1, characterized in that Before obtaining the resource usage and message queues corresponding to all nodes in the current distributed system through the master node, the method further includes: The master node corresponding to the current distributed system is determined in all nodes of the current distributed system through a temporary node based on a preset node creation rule.

3. The method according to claim 1, characterized in that The master node includes: a task management module and a task scheduling module; The master node performs sharding processing on the message queue based on a preset sharding strategy, resource usage, and a preset adjustment strategy, and generates a target task scheduling strategy corresponding to the current distributed system, including: The task management module performs sharding processing on the message queue based on a preset sharding strategy and resource usage, and generates a basic task scheduling strategy corresponding to the current distributed system; The basic task scheduling strategy is dynamically adjusted based on a preset adjustment strategy by a task scheduling module to generate a target task scheduling strategy corresponding to the current distributed system.

4. The method according to claim 3, characterized in that The task management module performs sharding processing on the message queue based on a preset sharding strategy and resource usage to generate a basic task scheduling strategy corresponding to the current distributed system, including: The task management module performs sharding processing on the message queue based on a preset hash sharding strategy to obtain a logical shard set corresponding to the message queue; wherein the logical shard set includes a target logical shard; The task management module dynamically maps the logical shard set based on a preset weighted polling algorithm and resource usage, determines the physical queue corresponding to the logical shard set, and uses the mapping relationship between the target logical shard and the corresponding physical node in the physical queue as the basic task scheduling strategy.

5. The method according to claim 4, characterized in that The dynamically mapping the logical shard set based on a preset weighted round-robin algorithm and resource usage by the task management module to determine the physical queue corresponding to the logical shard set includes: The task management module adjusts the basic weights of all nodes based on the resource usage of all nodes to determine the target weights of all nodes. The task management module dynamically maps the logical shard set based on a preset weighted polling algorithm and a target weight, and determines the physical queue corresponding to the logical shard set.

6. The method according to claim 1, characterized in that The preset adjustment strategy includes: a preset failover strategy, a preset priority scheduling strategy and a preset resource dynamic allocation strategy.

7. The method according to claim 1, characterized in that The target node includes: a task processing unit; The determining, by the target node, the corresponding target scheduling task based on the target task scheduling policy includes: The target scheduling task corresponding to the target node is asynchronously pulled by the task processing unit based on the current node load situation corresponding to the target node and the target task scheduling strategy.

8. The method according to claim 1, characterized in that Get the message queue through the master node, including: The central scheduler receives the message queue corresponding to the current distributed system and sends the message queue to the master node.

9. The method according to claim 1, characterized in that The target nodes include: the master node and the slave nodes in the full set of nodes except the master node.

10. A distributed cluster scheduling system based on decentralization, characterized in that: include: The master node is used to obtain the resource usage and message queues corresponding to all nodes in the current distributed system, and to perform sharding on the message queues based on a preset sharding strategy, resource usage, and preset adjustment strategy, to generate a target task scheduling strategy corresponding to the current distributed system; wherein the target task scheduling strategy includes a target scheduling task corresponding to a target node in the current distributed system; The target node is configured to determine a corresponding target scheduling task based on the target task scheduling strategy, execute the target scheduling task based on a preset time window mechanism, generate a heartbeat report and status information corresponding to the target scheduling task, and send the heartbeat report and status information corresponding to the target scheduling task to the master node; The master node is used to aggregate and process the heartbeat reports and status information corresponding to the target scheduling tasks, and generate a global scheduling summary report corresponding to the current distributed system.

Images