Presto task scheduling method and device

Abstract

The invention discloses a Presto task scheduling method and a Presto task scheduling device. According to the method, after a task request is received, tasks are allocated to matched clusters according to user information and task information in the request, then task types are further analyzed, execution modes of the tasks are determined, whether the tasks belong to task types needing to exclusively share resources or not is judged, and if the tasks belong to resource exclusive tasks, the tasks are allocated to the clusters. Tasks are allocated to reserved single tasks and special resource nodes, and when the tasks exclusively share working node resources, no other types of tasks are executed on the nodes, so that the tasks exclusively share the resources on specified cluster nodes, physical isolation of the working nodes in a cluster is realized, the performance problem caused by resource contention is avoided, and the service life of the cluster is prolonged. Therefore, the method is suitable for a complex query environment shared by multiple users, efficient execution of the important tasks can be ensured even when a large number of tasks are processed, and the overall reliability and stability of the system are improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data permission management, and more particularly to a task scheduling method and apparatus for Presto. Background Art

[0002] Presto is a multi-functional open-source distributed SQL query engine mainly used for quickly and parallelly processing large-scale data sets. It can query and analyze various data sources, efficiently execute queries from multiple data sources and return results, and is widely used.

[0003] The task assignment logic of Presto is that the coordinator receives and parses SQL queries, generates query plans, and decomposes them into multiple tasks (Tasks). These tasks are further divided into drivers and then assigned to multiple worker nodes (Workers). Among them, when Presto selects the strategy of worker nodes, the system dynamically monitors and adjusts the load of worker nodes to ensure that tasks are evenly distributed on available nodes. However, when there are a large number of tasks, there may be resource competition among multiple tasks. On the one hand, it will affect the execution efficiency and response time of tasks; on the other hand, if a task runs abnormally, it will affect the execution of other tasks and further affect the stability of system performance. In addition, the possible mutual influence among multiple task executions brings great difficulties to system maintenance including fault troubleshooting, performance tuning, and resource management, and the system maintenance complexity and cost are high. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a task scheduling method and apparatus for Presto, which can enhance the stability of the system and effectively reduce the risk of performance fluctuations.

[0005] In a first aspect, an embodiment of the present application provides a task scheduling method for Presto, including:

[0006] After receiving a task request, extracting user information and task information from the request;

[0007] According to the resource status of the current cluster and the task information, matching a cluster that meets the resource requirements of the task from available clusters as the target cluster;

[0008] Identifying whether the task is a resource-exclusive task;

[0009] If so, selecting an exclusive worker node from the target cluster as the task execution node; wherein, the exclusive worker node is a single-task and dedicated resource node;

[0010] Invoking the task execution node to respond to the task request.

[0011] In one embodiment, before selecting an exclusive working node from the target cluster, it further includes:

[0012] If the task is a resource-exclusive task, determine whether the current time belongs to the exclusive time window set for the task;

[0013] If it belongs, execute the step of selecting an exclusive working node from the target cluster.

[0014] In one embodiment, it further includes:

[0015] If the current time does not belong to the exclusive time window, or the task is not a resource-exclusive task, determine the task execution node according to the node load situation in the target cluster according to the load balancing principle.

[0016] In one embodiment, identifying whether the task is a resource-exclusive task includes:

[0017] Judge whether the priority level of the task is a high priority level;

[0018] If so, judge whether the permission level of the submitter reaches the resource-exclusive setting level according to the user information;

[0019] If it reaches, determine that the task is a resource-exclusive task.

[0020] In one embodiment, identifying whether the task is a resource-exclusive task includes:

[0021] Extract the task information in the historical task registration corresponding to the user information as the historical task information;

[0022] Judge whether there is historical execution information in the historical task information that matches the current task;

[0023] If there is, determine whether the resource requirement of the task reaches the high requirement level according to the historical execution information;

[0024] If it reaches, determine that the task is a resource-exclusive task.

[0025] In one embodiment, it further includes:

[0026] Continuously monitor the running state of the cluster;

[0027] If the running state shows that the cluster load reaches the high load standard, suspend the scheduling of new tasks.

[0028] In one embodiment, after continuously monitoring the running state of the cluster, it further includes:

[0029] If the execution status of the running status display task is abnormal, reassign the nodes of the task;

[0030] If the node status of the running status display is abnormal, mark the abnormal node and reassign all tasks of the abnormal node.

[0031] In one embodiment, before matching a cluster that meets the resource requirements of the task from available clusters according to the resource status of the current cluster and the task information, it further includes:

[0032] Determine whether there is a specified cluster in the task information;

[0033] If it exists, use the specified cluster as the target cluster.

[0034] In a second aspect, an embodiment of the present application provides a task scheduling device for Presto, including:

[0035] An information extraction unit, configured to extract user information and task information from the request after receiving a task request;

[0036] A cluster matching unit, configured to match a cluster that meets the resource requirements of the task from available clusters according to the resource status of the current cluster and the task information, as the target cluster;

[0037] A task type identification unit, configured to identify whether the task is a resource exclusive task; if so, trigger the exclusive node determination unit;

[0038] The exclusive node determination unit is configured to select an exclusive working node from the target cluster as the task execution node; wherein, the exclusive working node is a single-task, dedicated resource node;

[0039] A request response unit, configured to call the task execution node to respond to the task request.

[0040] In one embodiment, it further includes: a registration information management unit, configured to store and manage user information, task information, cluster information, exclusive working node information, and working node information.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. Description of the Drawings

[0042] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes, and advantages of the present application will become more apparent:

[0043] Figure 1The flowchart shows the task scheduling method of Presto provided by the embodiments of the present application;

[0044] Figure 2 The flowchart shows the node task allocation process under the exclusive time window setting provided by the embodiments of the present application;

[0045] Figure 3 The exemplary structural block diagram shows the task scheduling device of Presto provided by the embodiments of the present application. Detailed implementation manners

[0046] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention, rather than limiting the invention. Additionally, it should be noted that for the sake of description, only the parts related to the invention are shown in the drawings.

[0047] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments. Although the embodiments of the present application provide the method operation instruction steps as shown in the following embodiments or drawings, in the method, more or fewer operation instruction steps may be included based on routine or non-creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. When the method is actually processed or the device executes, it can be executed in the order shown in the embodiments or drawings or executed in parallel.

[0048] Presto is a distributed SQL query engine designed to efficiently execute SQL queries in a big data environment. However, its design focus is on high-performance data processing and query, rather than physical isolation of tasks. This design framework will expose the following disadvantages in actual use:

[0049] 1. Resource competition: Multiple tasks share the same physical resources (such as CPU, memory, and disk), which may lead to resource competition, thereby affecting the execution efficiency and response time of tasks.

[0050] 2. Performance fluctuation: In an environment of resource competition, the high load or abnormal situation of one task may have a negative impact on the performance of other tasks, resulting in performance fluctuation and instability.

[0051] 3. Maintenance complexity: Running multiple tasks in the same physical environment may increase the complexity of system maintenance, including challenges in fault troubleshooting, performance tuning, and resource management, etc.; if an independent cluster is built, the daily maintenance cost will be greatly increased.

[0052] 4. Reduced fault tolerance: In an environment without physical isolation, faults or resource contention issues may more easily affect the stability and fault tolerance of the entire system, increasing the risk and scope of system failures.

[0053] 5. Resource waste: Due to the lack of physical isolation, the system may not be able to effectively allocate and utilize resources, resulting in resource waste and low efficiency problems, thus affecting cost control and operational efficiency.

[0054] In response to this, this application proposes a task scheduling method for Presto, which performs secondary development on Presto and introduces the feature of physical isolation. This improvement enables the system to independently allocate tasks that need to be executed separately to dedicated physical resources, avoiding competition with other task resources, thereby significantly improving the execution efficiency and response speed of these tasks. Physical isolation also enhances the stability of the system, effectively reducing the risk of performance fluctuations caused by resource contention. At the same time, physical isolation also helps to limit the scope of influence that abnormal nodes may cause, improving the overall reliability and fault tolerance of the system.

[0055] It should be noted that the acquisition or use of data in the embodiments of this application requires user consent. Relevant data can only be obtained after user authorization, and the acquisition or use of data complies with the provisions of relevant laws and regulations.

[0056] Please refer to Figure 1 , Figure 1 which shows a schematic flowchart of the task scheduling method for Presto provided in an embodiment of this application. As Figure 1 shown, the method includes:

[0057] S101. After receiving a task request, extract user information and task information from the request;

[0058] After receiving a task request, extract user information and task information from the task request. User information mainly includes username, user group, priority, department, etc., mainly including personal information and permission information, which can help the system make intelligent decisions during task allocation. Task information mainly includes the name, priority, task type, required resources, etc. of the task. Task information can be used for reasonable isolation and scheduling of tasks during resource allocation.

[0059] S102. According to the resource status of the current cluster and the task information, match a cluster that meets the resource requirements of the task from the available clusters as the target cluster;

[0060] Continuously monitor the resource status of all available clusters, including the current load, idle resources, network status, etc. of each cluster. According to the available resources of each node or resource pool in the current cluster, as well as the resources required by the task (such as CPU, memory, storage, network bandwidth, etc.), select a cluster from all available clusters that can meet the resource requirements of the task. Among them, meeting the resource requirements of the task means that the computing, storage, network and other resources required during task execution match the resources provided by the cluster, ensuring that the task can run normally and efficiently on the selected cluster without failure or performance degradation due to insufficient resources or restrictions.

[0061] S103. Identify whether the task is a resource-exclusive task; if so, execute S104;

[0062] Further analyze the task type to determine whether the task belongs to the task type that requires exclusive resources, that is, a resource-exclusive task, so as to determine the execution mode of the task.

[0063] In this method, the concept of a resource-exclusive task is proposed, which refers to a task that requires the allocation of exclusive resources to avoid performance bottlenecks caused by resource contention of other tasks during task execution and ensure the smooth execution of the task. The identification criteria for resource-exclusive tasks can be customized according to the requirements in the actual application scenario, such as high priority or high resource requirements, etc., which are not limited here.

[0064] It should be noted that if the task type is not a resource-exclusive task, its processing method is not limited in this embodiment and can refer to the introduction of the following embodiments.

[0065] S104. Select an exclusive working node from the target cluster as the task execution node;

[0066] If the current task is identified as a resource-exclusive task, then select an exclusive working node from the determined target cluster as the task execution node. Among them, the exclusive working node is a pre-configured single-task, dedicated resource node. Single-task means that this node runs only one task at any time point to avoid resource contention caused by running multiple tasks simultaneously; dedicated resource means that the hardware or software resources of this node are only used for the current node and are not shared with other nodes. Among them, the dedicated resources can be dedicated only after the exclusive working node is started (that is, when there is a task occupying) to avoid resource waste.

[0067] The configured quantity of exclusive working nodes in each cluster is not limited. The available exclusive working nodes can be randomly selected from the cluster, or other selection rules for exclusive working nodes can be set, which can be set accordingly according to the actual setting requirements.

[0068] The dedicated working node only serves this task, ensuring complete exclusivity of resources and interference-free operation. This can maximize the computing power and resource utilization rate of the node, avoid other (non-important) tasks competing for resources with this task, and ensure that important tasks can be completed with optimal performance.

[0069] Through the mechanism of exclusive resources, high-level tasks can make full use of all computing resources of the node, such as CPU, memory, storage, and bandwidth, etc., thus significantly reducing the execution time of tasks. Whether it is a complex computing task or a resource-intensive query, it can obtain the optimal execution efficiency in an exclusive environment.

[0070] S105. Invoke the task execution node to respond to the task request.

[0071] The dedicated working node is a type of node specifically configured separately from the working nodes in this method and belongs to a kind of working node. After using a certain dedicated working node as the task execution node, the specific invocation of the dedicated working node can refer to the invocation mode of the working node. It should be noted that after a dedicated working node is occupied by a certain task, it cannot be called by other tasks anymore to ensure the exclusivity of a single task.

[0072] It should be noted that although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired result. For example, S103 can be executed simultaneously with S102, that is, the identification of resource-exclusive tasks can be carried out simultaneously with the determination of the target cluster.

[0073] Based on the above introduction, for the method provided in this embodiment, after receiving a task request, the task is allocated to a matching cluster according to the user information and task information in the request, and then the task type is further analyzed to determine the execution mode of the task, and it is judged whether the task belongs to the task type that requires exclusive resources. If it belongs to a resource-exclusive task, the task is allocated to the reserved single-task and dedicated resource nodes. During the period when the task exclusively occupies the resources of the working node, no other types of tasks are executed on these nodes, enabling the task to exclusively occupy resources on the specified cluster node, thereby achieving physical isolation of the working nodes within the cluster and avoiding performance problems caused by resource contention, thus ensuring the efficient execution of important tasks. Therefore, this method is applicable to complex query environments shared by multiple users and can ensure the efficient execution of important tasks even when processing a large number of tasks, thereby improving the overall reliability and stability of the system.

[0074] In one embodiment, for the processing of resource-exclusive tasks, a dedicated time window can be further configured to support task isolation by time period.

[0075] Specifically, when the determination task is a resource exclusive task, before selecting an exclusive working node from the target cluster in S104, it is possible to further determine whether the current time belongs to the exclusive time window set for the task; if so, execute the step of selecting an exclusive working node from the target cluster.

[0076] Figure 2 The figure shows a schematic diagram of the node task allocation process under a setting of exclusive time window. First, it is determined whether the cluster is in the exclusive time window. If it is not in the exclusive time window, the task can be allocated to any executable node in the entire cluster. If it is in the exclusive time window, the task execution queue is obtained, and an exclusive working node that can be allocated is queried from it. It is determined whether there is an executable exclusive working node in the current cluster. If there is, the task is directly allocated to this node. If not, it can be further determined whether there is an exclusive working node that can be allocated in the public queue. If there is one in the public queue, the task is directly allocated to this node. If there is none in the public queue, to ensure the rapid response of the task, it can be allocated to any executable node in the entire cluster. Only the above steps are taken as an example here, and the implementation of other methods can refer to the above introduction and will not be elaborated here.

[0077] In this embodiment, a resource exclusive working time, that is, an exclusive time window, is set for the resource exclusive task. The exclusive time window refers to a specific time range during which the task exclusively occupies the specified resources. Within this time window, only this task can run on the exclusive working node, and other tasks cannot preempt or share the same resources. If the task is determined to be executed within its exclusive time window, the system will allocate the task to run on the exclusive working node in the cluster. These nodes only serve this task during this time period, ensuring the complete exclusivity of resources and interference-free operation. While in the non-exclusive time window, or on non-exclusive working nodes, they do not have exclusive privileges, that is, they can be executed in the same way as ordinary tasks.

[0078] Under the above configuration, users can independently use the cluster resources within the specified time period, and outside this time period, all resources of the cluster are open to all users. This method not only ensures the smooth execution of tasks and the timely output of results, but also significantly improves the stability and overall performance of the system.

[0079] The method provided in this embodiment, by setting a time window for task isolation, enables important tasks to be executed flexibly while minimizing the waste of cluster resources and significantly ensuring the overall resource utilization rate. Especially in a complex query environment, this method performs particularly outstandingly. It can maintain the efficient operation of the system under high task load without affecting the system performance.

[0080] If it does not currently belong to the exclusive time window, or the current task is not a resource-exclusive task, then at this time, the task will share the resources of the cluster with other tasks. The processing method for tasks in these two cases is not limited in this method and can refer to the implementation methods in related technologies. To improve resource utilization, in one embodiment, according to the principle of load balancing, the task execution node can be determined based on the node load situation in the target cluster. That is, the system will schedule the task to a suitable node according to the current load situation of the cluster, so as to ensure that even in the state of resource sharing, the task can obtain sufficient resource support. This process is still based on the principles of load balancing and maximizing resource utilization to ensure the smooth execution of tasks under limited resources.

[0081] In the above embodiments, the identification criteria for resource-exclusive tasks are not limited. For better understanding, two implementation methods are introduced in this embodiment.

[0082] Method 1: Identify the task type according to the priority of the task and the privilege level of the submitter to ensure that high-priority tasks obtain the execution right first.

[0083] S31. Determine whether the priority level of the task is high priority; if so, execute S32;

[0084] S32. According to the user information, determine whether the privilege level of the submitter reaches the resource-exclusive setting level; if it reaches, determine that the task is a resource-exclusive task.

[0085] This method regards high-priority tasks with high importance as resource-exclusive tasks, and high-priority tasks obtain exclusive resources first, which can ensure that key services or computationally intensive tasks will not be interfered by other low-priority tasks. Moreover, only when the high-priority task is satisfied and the submitter has sufficient privileges, the task will be determined as a resource-exclusive task, preventing ordinary users from abusing exclusive resources and affecting the fairness of the system.

[0086] Method 2: Since the task has high resource requirements for itself, to avoid affecting the execution of other tasks, it is executed separately as a resource-exclusive task.

[0087] S33. Extract the task information in the historical task registration corresponding to the user information as the historical task information;

[0088] S34. Determine whether there is historical execution information in the historical task information that matches the current task; if so, execute S35;

[0089] S35. Determine whether the resource requirements of the task reach the high-demand level according to the historical execution information; if it reaches, determine that the task is a resource-exclusive task.

[0090] For example, if the past execution records of a certain task indicate that it usually consumes more than 90% of the CPU resources, the system can automatically identify it as a resource-exclusive task when submitting a similar task in the future.

[0091] If a user submits multiple similar tasks, the system can discover through historical execution data that these tasks always consume a large amount of CPU / memory, and thus automatically allocate exclusive resources to this task without the need for manual adjustment by the user. This reduces human intervention while increasing the success rate of the task, reduces the failures caused by resource contention during the execution of intensive resource-consuming tasks, improves the stability of the system, and enhances the intelligence level of scheduling.

[0092] In one embodiment, to facilitate the intelligent scheduling and resource allocation of tasks within the system, the running states of clusters can be continuously monitored above multiple clusters. Among them, the running states include the current load, idle resources, network state, node state, and task execution state, etc., of the available clusters. Through global monitoring, the latest resource status of all clusters in the system is obtained, providing accurate data support for decision-making.

[0093] In global resource monitoring, how to perform feedback optimization of system operation based on the monitoring data is not limited herein. In this embodiment, two dynamic scheduling strategies are introduced to dynamically adjust the resource allocation strategy according to the actual states of each cluster.

[0094] Specifically, if the running state shows that the cluster load reaches the high-load standard, the scheduling of new tasks is suspended.

[0095] For example, when the load of a certain cluster is too high, the decision-making center will schedule the newly submitted tasks to a cluster with a lower load to avoid resource competition and system performance degradation. This dynamic scheduling mechanism ensures the load balance of the system and avoids task delay or failure caused by the overload of a certain cluster. It should be noted that after reaching the high-load standard, if the cluster load decreases and is lower than the high-load standard, the scheduling of new tasks can be restarted to reduce the task processing burden on other clusters.

[0096] Another dynamic adjustment mechanism under global monitoring. Specifically, after continuously monitoring the running states of the clusters, if the running state shows that the execution state of the task is abnormal, the task is reallocated to nodes; if the running state shows that the node state is abnormal, the abnormal node is marked, and all tasks of the abnormal node are reallocated.

[0097] After the task is scheduled to a node, the system will continuously monitor the execution status of the task, track its resource usage, task progress, and potential anomalies or bottlenecks. If an abnormal situation such as insufficient resources or overloading is detected in a certain node during the execution process, the system can dynamically adjust the execution node of the task, reallocate the task to a more suitable node to ensure the stability and efficiency of task execution. This dynamic adjustment mechanism can optimize resource allocation in a timely manner during runtime, avoiding system overload or task failure. In addition to node status monitoring, if an exception occurs during the execution of the task, such as task execution interruption, task failure, etc., to ensure the smooth execution of the task, node reallocation of the abnormal task can be quickly initiated to achieve rapid retry of the task and avoid task execution failure caused by node reasons. It should be noted that for node reallocation after task failure, it can be further set to pause reallocation and output corresponding prompt information after the number of reallocations reaches a certain threshold, so as to promptly discover and handle the corresponding problems and avoid waste of resources.

[0098] In one embodiment, the scheduling policy can also be flexibly configured according to user preferences. Specifically, before executing S102 to match a cluster that meets the resource requirements of the task from available clusters according to the resource status and task information of the current cluster, it can first be determined whether a specified cluster exists in the task information; if it exists, the specified cluster is used as the target cluster. If it does not exist, S102 can continue to be executed.

[0099] For example, some tasks need to run on a specific cluster. To meet different usage requirements, the target cluster can be specified. Then, during the process of allocating a cluster to the task, these specified rules can be combined to automatically complete the task allocation, ensuring the intelligence and efficiency of the scheduling process. This not only enables the optimal configuration of resources in the system but also ensures that the differentiated requirements of different application scenarios can be met in a complex and changeable operating environment. Therefore, this method can effectively address the scheduling challenges in a large-scale multi-task environment.

[0100] In addition to the above flexible scheduling, before task execution, it can also be flexibly configured according to different task requirements. For example, for some long-running tasks, the system can automatically schedule them to nodes with relatively idle resources according to their execution progress and resource consumption to avoid resource waste; for sudden emergency tasks, the system can also temporarily interrupt low-priority tasks to free up resources to ensure the rapid execution of emergency tasks.

[0101] Through the above task execution scheduling mechanism, the system can effectively allocate resources reasonably in different scenarios, not only ensuring the exclusive use of resources by high-priority tasks during the exclusive time period, but also ensuring the smooth completion of tasks through intelligent scheduling during the non-exclusive time period. In addition, continuous task monitoring and dynamic adjustment further improve the execution efficiency of tasks and the overall stability of the system. It realizes the efficient execution of tasks and the optimal utilization of cluster resources, and is particularly suitable for complex query environments that require high performance and high stability.

[0102] Further reference is made to Figure 3 , which shows an exemplary structural block diagram of a task scheduling device of Presto according to an embodiment of the present application. This device is built on the Coordinator (coordinating node) in the Presto cluster, and mainly includes:

[0103] An information extraction unit 101, configured to extract user information and task information from a request after receiving a task request;

[0104] A cluster matching unit 102, configured to match, according to the resource status of the current cluster and the task information, a cluster that meets the resource requirements of the task from available clusters as the target cluster;

[0105] A task type recognition unit 103, configured to recognize whether a task is a resource exclusive task; if so, trigger an exclusive node determination unit;

[0106] An exclusive node determination unit 104, configured to select an exclusive working node from the target cluster as the task execution node; wherein, the exclusive working node is a single-task, dedicated resource node;

[0107] A request response unit 105, configured to call the task execution node to respond to the task request.

[0108] In one embodiment, a registration information management unit may be further provided in the task scheduling device of Presto, configured to store and manage relevant information for task scheduling, including but not limited to user information, task information, cluster information, exclusive working node information, and working node information.

[0109] User information: Each user needs to register their personal information and permissions in the system, and this information will help the system make intelligent decisions during task allocation. User information may include username, user group, priority, department affiliation, etc.

[0110] Task information: Each submitted task needs to be registered in the system, including the name, priority, task type, required resources, etc. of the task. This information will help the system reasonably isolate and schedule tasks during resource allocation.

[0111] Cluster Information: The system maintains a cluster resource database that records detailed information about each available cluster, such as the cluster's scale, performance metrics, and the number of available resources, to ensure that the system can accurately select the most suitable cluster for tasks.

[0112] Dedicated Worker Nodes: For high-priority tasks or tasks that require exclusive resources, the system can allocate dedicated worker nodes to these tasks. These nodes are specifically registered in the system, and their status, load conditions, historical task execution situations, etc. are recorded to ensure that they can be quickly and accurately allocated to critical tasks when needed.

[0113] Worker Node Details: The system manages the detailed information of each worker node, including the node's hardware configuration (such as CPU, memory, storage), network situation, current load, fault history, etc. This data can ensure that tasks are allocated to the most suitable nodes, avoiding resource waste or improper allocation.

[0114] Through the comprehensive management of the above information, the system can not only ensure the accuracy of task allocation and execution but also continuously monitor and optimize the resource usage of tasks to ensure the efficient operation of the system.

[0115] To facilitate understanding, in this embodiment, the interactions between the coordination node and the worker nodes in several business scenarios are introduced.

[0116] 1. Register Nodes to the Discovery Server

[0117] Every 8 seconds, the worker node sends a

[0118] PUT / v1 / announcement / {node_id} request for registration;

[0119] The server returns 202 Accepted, indicating that the node registration is successful.

[0120] 2. Update the Node List Cache

[0121] Every 10 seconds, the coordination node requests the node list from the discovery server: sends a GET

[0122] / v1 / service / presto / general request;

[0123] The server returns services (the node list), that is, the current available Presto node information.

[0124] 3. Conduct Health Checks

[0125] Every 0.5 seconds, the coordination node performs a health check on all nodes in the node list, including traversing the services (node list) to send a HEAD / request to each worker node; if 404 Not Found is returned, it means the node is unavailable.

[0126] 4. Update the node list of the coordinator

[0127] Every 5 seconds, the coordination node checks the status of each worker node: send a GET

[0128] / v1 / info / state, and the returned values are ACTIVE (active) or SHUTTING_DOWN (shutting down);

[0129] After the server returns the list of all nodes, filter the healthy nodes and update the available node list of the coordinator for query scheduling.

[0130] It should be understood that the various units described in the above device correspond to the respective steps in the method described in the reference Figure 1 description. Thus, the operations and features described above for the method also apply to the device and the units contained therein, and will not be elaborated here. The device can be pre-implemented in the browser or other secure applications of the server, or can be loaded into the browser or its secure application of the server by means such as downloading. The corresponding units in the device can cooperate with the units in the server to implement the solution of the embodiments of the present application.

[0131] For several units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more of the above-described units can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0132] It should be noted that for the details not disclosed in the Presto task scheduling device of the embodiments of the present application, please refer to the details disclosed in the above embodiments of the present application, and will not be elaborated here.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operation instructions of devices and methods according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the aforementioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two connected blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that executes the specified functions or operation instructions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0134] The units or modules involved in the embodiments described in the present application can be implemented in software or in hardware. The described units or modules can also be provided in a processor. Among them, the names of these units or modules do not, in some cases, constitute a limitation on the units or modules themselves.

[0135] The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features having similar functions disclosed in the present application.

Claims

1. A task scheduling method for Presto, characterized in that, Including: After receiving a task request, extract user information and task information from the request; According to the resource status of the current cluster and the task information, match a cluster that meets the resource requirements of the task from available clusters as the target cluster; Identify whether the task is a resource exclusive task; If so, select an exclusive working node from the target cluster as the task execution node; wherein, the exclusive working node is a single-task and dedicated resource node; Call the task execution node to respond to the task request.

2. The method according to claim 1, wherein Before selecting an exclusive working node from the target cluster, it further includes: If the task is a resource exclusive task, determine whether the current time belongs to the exclusive time window set for the task; If it belongs, execute the step of selecting an exclusive working node from the target cluster.

3. The method according to claim 2, characterized in that, It also includes: If the current time does not belong to the exclusive time window, or the task is not a resource exclusive task, determine the task execution node according to the node load situation in the target cluster according to the load balancing principle.

4. The method according to claim 1, wherein The identifying whether the task is a resource exclusive task includes: Judge whether the priority level of the task is a high priority level; If so, judge whether the permission level of the submitter reaches the resource exclusive setting level according to the user information; If it reaches, determine that the task is a resource exclusive task.

5. The method according to claim 1, characterized in that, The identifying whether the task is a resource exclusive task includes: Extract the task information in the historical task registration corresponding to the user information as the historical task information; Judge whether there is historical execution information in the historical task information that matches the current task; If there is, determine whether the resource requirements of the task reach the high demand level according to the historical execution information; If it reaches, determine that the task is a resource exclusive task.

6. The method according to claim 1, characterized in that, It also includes: Continuously monitor the running status of the cluster; If the running status shows that the cluster load reaches the high load standard, suspend new task scheduling.

7. The method according to claim 6, wherein After the continuously monitoring the running status of the cluster, it further includes: If the running status shows that the execution status of the task is abnormal, reassign the nodes for the task; If the running status shows that the node status is abnormal, mark the abnormal node and reassign all tasks of the abnormal node.

8. The method according to claim 1, wherein Before matching a cluster that meets the resource requirements of the task from available clusters according to the resource status of the current cluster and the task information, it further includes: Judge whether there is a specified cluster in the task information; If there is, use the specified cluster as the target cluster.

9. A task scheduling device for Presto, characterized in that, Including: An information extraction unit, configured to extract user information and task information from the request after receiving a task request; A cluster matching unit, configured to match a cluster that meets the resource requirements of the task from available clusters according to the resource status of the current cluster and the task information as the target cluster; A task type identification unit, configured to identify whether the task is a resource exclusive task; If so, trigger the exclusive node determination unit; The exclusive node determination unit is configured to select an exclusive working node from the target cluster as the task execution node; wherein, the exclusive working node is a single-task and dedicated resource node; A request response unit, configured to call the task execution node to respond to the task request.

10. The device according to claim 9, characterized in that, It also includes: A registration information management unit is used to store and manage user information, task information, cluster information, dedicated worker node information, and worker node information.

Images