Task scheduling method, device and equipment for multi-tenant isolation on cloud
Through the combination of distributed message queues and exclusive message queues, the fairness and efficiency of multi-tenant task scheduling in cloud computing is solved, effective restrictions on tenant tasks and reasonable allocation of resources are achieved, and the stability of cloud computing services is improved .
Patent Information
- Application Number
- CN202411155368.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-21
AI Technical Summary
In cloud computing, there are fairness and efficiency problems in multi-tenant workflow task scheduling, especially in burst requests or performance bottlenecks, which affects the normal requests of other users.
Load balancing of the service cluster through distributed message queues, scheduling tasks to the target service nodes, and setting up exclusive message queues and concurrency for each tenant to control the execution of tasks.
The fairness and efficiency of multi-tenant task scheduling on the cloud is achieved, the number of tasks per tenant and the number of tasks of the same type is limited, resource overload is avoided, and the stability of cloud computing services is improved.
Smart Images

Figure CN119030982A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cloud computing technology, and in particular to a method, device and equipment for scheduling tasks for multi-tenant isolation on a cloud. Background Art
[0002] As a service provider, cloud computing vendors use management and control software to process user requests to create, delete, or schedule resources. When a user, as a tenant, requests a service provider to create, delete, or schedule resources, the service provider will automatically generate multiple tasks according to user needs through the management and control software, such as configuring cloud computing devices, calling internal services, and data persistence operations. The workflow scheduling system responds to the call of the management and control software and is responsible for scheduling and executing these tasks reasonably to ensure that resources can eventually be delivered to users on demand. The workflow scheduling system will arrange the tasks and parameters that need to be executed in a single user request into a workflow, and then execute the tasks in this workflow serially or in parallel according to the arrangement order.
[0003] The workflow scheduling system needs to provide task scheduling for multiple users at the same time. If some users have sudden requests or some tasks cause performance bottlenecks in the backend service, the normal requests of other users will inevitably be affected. Therefore, how to achieve fair and efficient scheduling of workflow tasks for multiple tenants on the cloud is a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a method, apparatus and device for task scheduling with multi-tenant isolation on the cloud, so as to improve the efficiency and fairness of workflow task scheduling for multi-tenants on the cloud.
[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:
[0006] In a first aspect, an embodiment of the present application provides a method for scheduling tasks for multi-tenant isolation on a cloud, the method comprising:
[0007] Receive a first workflow of a first tenant on the cloud, perform load balancing on multiple service nodes in a service cluster through a distributed message queue, and schedule tasks of the first workflow to a target service node;
[0008] Storing the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node;
[0009] The target service node controls the execution of tasks of the first workflow based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant.
[0010] In a second aspect, an embodiment of the present application provides a task scheduling device for multi-tenant isolation on a cloud, the device comprising:
[0011] A scheduling unit, configured to receive a first workflow of a first tenant on the cloud, load balance a plurality of service nodes in a service cluster through a distributed message queue, and schedule tasks of the first workflow to a target service node;
[0012] A storage unit, configured to store the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node;
[0013] A control unit is used to control the task execution of the first workflow through the target service node based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant.
[0014] In a third aspect, an embodiment of the present application provides a task scheduling device for multi-tenant isolation on the cloud, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the task scheduling method for multi-tenant isolation on the cloud as described above is implemented.
[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the task scheduling method for multi-tenant isolation on the cloud as described above.
[0016] It can be seen that the embodiments of the present application have the following beneficial effects:
[0017] In an embodiment of the present application, after receiving the first workflow of the first tenant on the cloud, the tasks of the first workflow are scheduled to the target service node through a distributed message queue to achieve load balancing of multiple service nodes in the service cluster. A dedicated message queue corresponding to the tenant is set in each service node. The tasks of the first workflow are added to the dedicated message queue of the first tenant in the target service node. Tasks are read from the dedicated message queue according to the dedicated concurrency configured for the first tenant to control the execution of the tasks of the first workflow. Therefore, the number of tasks for each tenant and the number of tasks of the same type can be limited through the tenant's dedicated message queue and dedicated concurrency, so as to achieve isolation of multiple tenants on the cloud and ensure the fairness and efficiency of workflow task scheduling of multiple tenants on the cloud. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A schematic diagram of an exemplary application scenario provided for an embodiment of the present application;
[0019] Figure 2 A flowchart of a task scheduling method for multi-tenant isolation on the cloud provided in an embodiment of the present application;
[0020] Figure 3 This is a schematic diagram of the working process of the scheduler in the embodiment of the present application;
[0021] Figure 4 A schematic diagram of a process for dynamically adjusting a concurrency threshold in an embodiment of the present application;
[0022] Figure 5 A schematic diagram of the process of obtaining workflow information in an embodiment of the present application;
[0023] Figure 6 This is a schematic diagram of the working process of the counter in the embodiment of the present application;
[0024] Figure 7 A schematic diagram of the implementation process of a task scheduling method for multi-tenant isolation on the cloud in an embodiment of the present application;
[0025] Figure 8 A schematic diagram of a task scheduling device for multi-tenant isolation on a cloud provided in an embodiment of the present application;
[0026] Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0028] In order to facilitate understanding and explanation of the technical solution provided by the embodiments of the present application, the background technology of the embodiments of the present application will be described below.
[0029] For cost optimization or product design needs, cloud computing vendors often use a workflow scheduling system to provide scheduling services for different users (i.e. tenants) in the same region. The workflow scheduling system arranges the tasks and parameters that need to be executed in a single user request into a workflow, and then executes the tasks in the workflow in serial or parallel order according to the arrangement sequence.
[0030] If some users have sudden requests, processing these users' requests will affect the normal requests of other users. Alternatively, some tasks cause performance bottlenecks in the backend service, such as server failures, insufficient backend service processing performance, etc., and processing these tasks will also affect the normal requests of other users. How to achieve isolation of multiple tenants on the cloud and ensure that the workflow tasks of multiple tenants on the cloud are fairly and efficiently scheduled is the main technical problem to be solved by the embodiments of this application.
[0031] In order to facilitate understanding of the task scheduling method for multi-tenant isolation on the cloud provided by the embodiment of the present application, Figure 1 See the example scenario shown in the figure. Figure 1 As shown, this figure is a schematic diagram of an exemplary application scenario provided in an embodiment of the present application.
[0032] The embodiment of the present application is applied to a workflow scheduling system, and the workflow scheduling system can be composed of four modules: a memory, a scheduler, an executor, and a counter. At the same time, the workflow scheduling system is a multi-node system, which is composed of a service cluster of multiple service nodes, and the service nodes can be computing devices such as servers. In the workflow scheduling system, the service nodes may include software modules such as a memory, a scheduler, an executor, and a counter. In addition, the workflow scheduling system also includes an independent scheduling node and a centralized cache node. The scheduling node includes a scheduler, and the centralized cache node includes a memory.
[0033] Among them, the scheduler is the core module of the task scheduling method for multi-tenant isolation on the cloud provided by the embodiment of the present application, which is used to dynamically schedule tasks of multiple tenants according to the system resource situation. The memory uses multi-level caching technology to store the metadata required for workflow execution and can persist the data. The executor can execute tasks according to task names and parameters. The counter is used to record the current execution status of the workflow, and the optimistic locking technology is used to ensure the concurrency safety of tasks.
[0034] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.
[0035] To facilitate understanding of the embodiments of the present application, a method for scheduling tasks for multi-tenant isolation on a cloud provided by the embodiments of the present application is described below in conjunction with the accompanying drawings.
[0036] See also Figure 2 As shown, this figure is a flowchart of a task scheduling method for multi-tenant isolation on the cloud provided by an embodiment of the present application, such as Figure 2 As shown, the method may include S201-S204:
[0037] S201: Receive a first workflow of a first tenant on the cloud, perform load balancing on multiple service nodes in a service cluster through a distributed message queue, and schedule tasks of the first workflow to a target service node.
[0038] When a tenant requests a service provider to create, delete or schedule resources, the service provider will generate multiple tasks according to the tenant's needs through the management and control software. The management and control software calls the workflow scheduling system, and the workflow scheduling system implements the scheduling of tasks in the workflow. Usually, a workflow includes at least one task, and each task includes the tenant information requesting the task, so the task and the tenant have a corresponding relationship. After the workflow scheduling system obtains the first workflow of the first tenant, it will assign the tasks in the first workflow to the target service node. The first tenant is any tenant on the cloud, and the target service node is any service node in the workflow scheduling system.
[0039] In actual applications, the scheduler in the scheduling service node can add the tasks in the first workflow to the distributed message queue, and distribute each task to the service node through the distributed message queue. By distributing tasks to service nodes through the distributed message queue, the load balancing of multiple service nodes in the service cluster can be guaranteed as much as possible.
[0040] S202: Storing the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node.
[0041] In each service node, a dedicated message queue is established for each tenant. After the task of the first workflow is assigned to the target service node, it is added to the dedicated message queue established for the first tenant in the target service node. For example, the target service node has a dedicated message queue for tenant 1, a dedicated message queue for tenant 2, etc., and the task of the first workflow corresponds to tenant 1, then the task of the first workflow is added to the dedicated message queue of tenant 1. Establishing a dedicated message queue for each tenant ensures that the tasks of different tenants are isolated during scheduling.
[0042] In actual applications, the scheduler may store the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node.
[0043] S203: Controlling the task execution of the first workflow through the target service node based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant.
[0044] Each tenant has an exclusive concurrency to limit the tasks requested by each tenant. The exclusive message queue of the first tenant includes at least one task of the first workflow. Tasks are read from the exclusive message queue of the first tenant in the order in which they are added. The total number of tasks read must be less than or equal to the exclusive concurrency of the first tenant. For example, the exclusive concurrency of the current tenant's exclusive message queue is 5. If the number of tasks in the exclusive message queue is more than 5, 5 tasks can be read; if the number of tasks in the exclusive message queue is not more than 5, all tasks in the exclusive message queue can be read.
[0045] In practical applications, the scheduler of the target service node can read the task of the first workflow from the dedicated message queue of the first tenant based on the dedicated concurrency configured for the first tenant, and then trigger the target service node to execute the task of the first workflow.
[0046] In actual applications, the scheduler passes the task name, task parameters and other information of the task to be executed to the executor, which actually executes the task of the first workflow and can return the execution result to the scheduler so that the scheduler can continue to schedule tasks.
[0047] In a possible implementation, S203, by means of the target service node, based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant, a specific implementation of controlling the execution of the task of the first workflow may include:
[0048] Reading tasks from the dedicated message queue of the first tenant in sequence through the target service node based on the dedicated concurrency configured for the first tenant and the task concurrency corresponding to each task type, so that the total number of tasks read is less than or equal to the dedicated concurrency configured for the first tenant and satisfies the task concurrency corresponding to each task type;
[0049] The read task of the first workflow is executed through the target service node.
[0050] In an embodiment of the present application, in addition to setting the exclusive concurrency of the tenant, the task concurrency corresponding to each task type can also be set to limit the flow of tasks of the same task type requested by a tenant at the same time. Thus, when reading tasks from the exclusive message queue of the tenant, in addition to satisfying the exclusive concurrency set for the tenant, each task read also needs to satisfy the task concurrency corresponding to the task type of the task. For example, the exclusive concurrency set for the first tenant is 5, the task concurrency corresponding to task type a is 2, the task concurrency corresponding to task type b is 2, and the task concurrency corresponding to task type c is 2. Tasks are read in sequence from the exclusive message queue of the first tenant. The task type of task 1 read is task type a, the task type of task 2 is task type a, the task type of task 3 is task type b, the task type of task 4 is task type c, and the task type of task 5 is task type a. At this time, task 5 does not satisfy the task concurrency corresponding to task type a, and the task cannot be read temporarily. The task type of task 6 that continues to be read is task type b. Then the task of the first workflow read is executed through the target service node.
[0051] In this way, by setting the tenant's exclusive concurrency and the task concurrency corresponding to each task type, concurrency limiting can be implemented for tasks requested by each tenant, ensuring fairness in task scheduling among multiple tenants.
[0052] In addition, in actual applications, other types of concurrency can also be set, such as setting the concurrency of the interface corresponding to the task request service, etc.
[0053] In the embodiment of the present application, the tenant's exclusive concurrency and the task concurrency corresponding to the task type can be automatically and dynamically adjusted according to the task execution status. For related instructions, please refer to the subsequent embodiments and will not be repeated here.
[0054] See also Figure 3 As shown, a schematic diagram of the work process of the scheduler is shown. After obtaining the workflow, the scheduler in the scheduling service node adds the tasks in the workflow to the task message queue, and distributes the tasks in the task message queue to different service nodes, for example, the tasks are distributed to service node 1, service node 2, and service node 3. The scheduler in each service node adds the tasks to the exclusive message queue of the tenant corresponding to each task. For example, the tasks assigned to service node 1 are all from tenant 1, and these tasks are added to the exclusive message queue of tenant 1 in service node 1; the tasks assigned to service node 2 are all from tenant 1, and these tasks are added to the exclusive message queue of tenant 1 in service node 2; the tasks assigned to service node 3 are all from tenant 2, and these tasks are added to the exclusive message queue of tenant 2 in service node 3. The scheduler of each service node reads the tasks from the exclusive message queue according to the exclusive concurrency of the tenant and the task concurrency corresponding to the task type, that is, after reading one or more tasks, the task concurrency check is performed. If the read task meets the concurrency, the task can be executed by the executor in the service node. If the read task does not meet the concurrency, the task is read again from the exclusive message queue.
[0055] In this way, in an embodiment of the present application, after receiving the first workflow of the first tenant on the cloud, the tasks of the first workflow are scheduled to the target service node through a distributed message queue to achieve load balancing of multiple service nodes in the service cluster. An exclusive message queue corresponding to the tenant is set in each service node. The tasks of the first workflow are added to the exclusive message queue of the first tenant in the target service node. Tasks are read from the exclusive message queue according to the exclusive concurrency configured for the first tenant to control the execution of the tasks of the first workflow. Therefore, the number of tasks for each tenant and the number of tasks of the same type can be limited through the tenant's exclusive message queue and exclusive concurrency, so as to achieve isolation of multiple tenants on the cloud and ensure the fairness and efficiency of workflow task scheduling of multiple tenants on the cloud.
[0056] In the embodiment of the present application, it is also possible to analyze whether the service cluster has workflow overload and analyze the abnormal source to dynamically adjust the concurrency.
[0057] In a possible implementation, the method for scheduling tasks for multi-tenant isolation on the cloud provided by the embodiment of the present application may also include:
[0058] A1: Determine whether the service cluster has workflow overload based on the number of remaining tasks in the exclusive message queues of each tenant in the service cluster and the tenant's remaining exclusive concurrency.
[0059] A2: If there is a workflow overload, determine the abnormal task type based on the execution time of multiple tasks of the same task type.
[0060] A3: Adjust the task concurrency corresponding to the abnormal task type.
[0061] In an embodiment of the present application, it is possible to perform a timed execution to determine whether the entire service cluster has a workflow overload. Specifically, by recording the number of remaining tasks in the exclusive message queue of each tenant in the service cluster and the remaining exclusive concurrency of the tenant, it is determined whether the service cluster has a workflow overload. The number of remaining tasks is the number of accumulations in the exclusive message queue. If the number of remaining tasks in multiple exclusive message queues is in a state of exceeding the number threshold within a time period exceeding the time threshold, it means that the exclusive message queue is in a task accumulation state for a long time, and it can be determined that the service cluster has a workflow overload. The remaining exclusive concurrency refers to the difference between the exclusive concurrency set for the tenant and the number of tasks read from the exclusive message queue of the tenant. For example, if the exclusive concurrency set for the tenant is 5, and the number of tasks currently read from the exclusive message queue of the tenant is 3, then the remaining exclusive concurrency refers to 2. The larger the remaining exclusive concurrency of each exclusive message queue, the more idle the exclusive message queue is. On the contrary, if the remaining exclusive concurrency of multiple exclusive message queues is 0 within a time period exceeding the time threshold, it means that the exclusive message queue is in a task accumulation state for a long time, and it can be determined that the service cluster has a workflow overload.
[0062] If the service cluster has workflow overload, you can collect the execution time of each task and calculate the average execution time of tasks of the same task type. If the average execution time is greater than the standard execution time of tasks of this task type, it means that the service providing the task may be abnormal, and the task type can be determined as an abnormal task type. For example, the standard execution time of tasks of task type a is 10 seconds, and the average execution time of tasks of task type a is 20 seconds, which means that the service providing the execution of tasks of task type a is abnormal, and task type a is an abnormal task type.
[0063] Therefore, logs can be output and the task concurrency corresponding to the abnormal task type can be lowered. Since the abnormal task type may cause performance bottlenecks in its backend service, lowering the task concurrency corresponding to the abnormal task type and reducing the number of tasks executed by the abnormal task type will help restore the performance of the backend service of the abnormal task type.
[0064] In a possible implementation, the method for scheduling tasks for multi-tenant isolation on the cloud provided by the embodiment of the present application may also include:
[0065] B1: Determine whether the service cluster has workflow overload based on the number of remaining tasks in the exclusive message queue of each tenant in the service cluster and the remaining exclusive concurrency of the tenant.
[0066] B2: If there is a workflow overload, determine the abnormal tenant based on the number of tasks corresponding to the tenant.
[0067] B3: Adjust the exclusive concurrency of abnormal tenants.
[0068] In the embodiment of the present application, it is possible to periodically determine whether the entire service cluster has a workflow overload. If the service cluster has a workflow overload, the number of tasks corresponding to each tenant can also be counted. If the number of tasks corresponding to a tenant exceeds a threshold, it means that the tenant has submitted too many task requests, and the tenant can be determined as an abnormal tenant.
[0069] Therefore, logs can be output and the exclusive concurrency corresponding to abnormal tenants can be lowered. An abnormal tenant may request a large number of tasks, causing workflow overload. Lowering the exclusive concurrency of the tenant ensures that the execution time of workflow tasks of other tenants does not increase significantly, significantly improving the stability of cloud computing services.
[0070] In addition, if the abnormal tenant determined last time is no longer an abnormal tenant during the process of re-determining whether the service cluster has workflow overload, the exclusive concurrency of the tenant can be increased. Similarly, if the abnormal task type determined last time is no longer an abnormal task type, the task concurrency corresponding to the task type can also be increased. This achieves dynamic adjustment of concurrency.
[0071] See also Figure 4As shown, a schematic diagram of the process of dynamically adjusting the concurrency is shown. When a performance bottleneck occurs in the service backend or a request burst occurs in some tenants, the scheduler can also dynamically adjust the concurrency according to the task execution status. In the process of determining whether the service cluster has workflow overload, it can be determined whether the service cluster has workflow overload according to the number of remaining tasks in the exclusive message queue of each tenant and the remaining exclusive concurrency of the tenant. In the process of analyzing the abnormal source, the abnormal task type can be determined according to the execution time of multiple tasks of the same task type, and the abnormal tenant can be determined according to the number of tasks corresponding to the tenant. After determining the workflow overload, the state of the workflow overload, the abnormal task type, and the abnormal tenant can be recorded in the log. In the process of adaptive adjustment of concurrency, the exclusive concurrency corresponding to the abnormal tenant and the task concurrency corresponding to the abnormal task type can be adjusted. Thus, when a request burst or a service performance bottleneck causes workflow overload, the concurrency can be dynamically adjusted according to the execution status of the task.
[0072] During the working process of the scheduler and the executor, it is also necessary to obtain the workflow information of the first workflow from the memory. In a possible implementation, the task scheduling method for multi-tenant isolation on the cloud provided by the embodiment of the present application may also include:
[0073] The workflow information of the first workflow is obtained from the multi-level cache based on the priority order of the multi-level cache.
[0074] In an embodiment of the present application, the memory uses a multi-level cache technology, so when the workflow information is needed, the workflow information of the first workflow can be obtained from the multi-level cache. Among them, the multi-level cache includes a local cache, a centralized cache, and a persistent storage sorted in descending order of priority. That is, in the process of obtaining workflow information from the multi-level cache, the workflow information is obtained from the local cache first, the workflow information is obtained from the centralized cache secondly, and the workflow information is obtained from the persistent storage last. Each service node includes a local cache and a persistent cache, and the centralized cache service node includes a centralized cache. The local cache is used to store data that will not be modified during the life cycle of a single workflow, such as task parameters, workflow orchestration information, header information, and static task context information; while the centralized cache stores the above data, it also stores workflow execution status information, dynamic task context data, and is responsible for data synchronization between multiple service nodes; the persistent storage is responsible for storing the data in the centralized cache to the local disk to prevent the loss of cached data.
[0075] In a possible implementation, the specific implementation of obtaining the workflow information of the first workflow from the priority order multi-level cache based on the multi-level cache may include:
[0076] C1: Acquire the workflow information of the first workflow from the local cache of the target service node.
[0077] C2: If the workflow information of the first workflow does not exist in the local cache of the target service node, the workflow information of the first workflow is obtained from the centralized cache, and the workflow information of the first workflow is saved in the local cache of the target service node.
[0078] C3: If the workflow information of the first workflow does not exist in the centralized cache, the workflow information of the first workflow is obtained from the persistent storage of the target service node, and the workflow information of the first workflow is saved in the local cache of the target service node.
[0079] See also Figure 5 As shown, a schematic diagram of the process of obtaining workflow information is shown. After requesting to obtain the workflow information of the first workflow, query the local cache of the target service node to determine whether the workflow information exists in the local cache. If the workflow information exists in the local cache, the workflow information is obtained. If the workflow information does not exist in the local cache, query the centralized cache to determine whether the workflow information exists in the centralized cache. If the workflow information exists in the local cache, obtain the workflow information and save the workflow information to the local cache. If the workflow information does not exist in the centralized cache, query the persistent storage, obtain the workflow information, and save the workflow information to the local cache.
[0080] Compared with the usual use of databases to save workflow information, the use of multi-level caches for storage and synchronization of workflow information makes reading workflow information more efficient.
[0081] In an embodiment of the present application, after the current task (target task) of the first workflow is successfully executed, the execution status information of the executed tasks of the first workflow in the centralized cache needs to be modified. Since the tasks of a single workflow may be executed concurrently by different service nodes, different service nodes may modify the execution status information in the centralized cache at the same time. Therefore, the counter uses an optimistic lock to ensure the concurrent safety when modifying the execution status information. In a possible implementation method, the task scheduling method for multi-tenant isolation on the cloud provided in an embodiment of the present application may also include:
[0082] D1: After the current task of the first workflow is successfully executed, the execution status information of the executed tasks of the first workflow is read from the centralized cache.
[0083] D2: Write the current task execution success information of the first workflow into the centralized cache.
[0084] D3: Read the execution status information of the executed tasks of the first workflow from the centralized cache again.
[0085] D4: If the execution status information of the executed task of the first workflow read twice is consistent, the current task execution success information of the first workflow is added to the execution status information of the executed task of the first workflow.
[0086] D5: If the execution status information of the executed tasks of the first workflow read twice is inconsistent, repeat D1 to read the execution status information of the executed tasks of the first workflow from the centralized cache and subsequent steps.
[0087] After the current task of the first workflow is successfully executed, for example, if the task is the target task, the counter needs to write the status of the successful execution of the target task into the centralized cache. The centralized cache stores the execution status information of the executed tasks of each workflow, and the execution status information of the executed tasks records which tasks in the workflow have been successfully executed. Before writing, the execution status information of the executed tasks of the first workflow is read from the centralized cache. For example, if the target task is Task 2, the execution status information of the executed tasks of the first workflow read from the centralized cache is that Task 1 is successfully executed. Then the target task execution success information (i.e., Task 2 is successfully executed) is written to the centralized cache, and the execution status information of the executed tasks of the first workflow is read again. If the execution status information of the executed tasks is still Task 1 is successfully executed, it means that in the process of writing the target task execution success information, no other service node modifies the execution status information of the executed tasks, and then Task 2 is successfully executed and added to the execution status information of the executed tasks of the first workflow.
[0088] If the execution status information of the executed tasks of the first workflow read twice is inconsistent, for example, the execution status information of the executed tasks read again is that Task 1 is executed successfully and Task 3 is executed successfully, it means that in the process of writing the target task execution success information, a service node has written other task execution success information, and this write fails. It is necessary to re-read the execution status information of the executed tasks of the first workflow from the centralized cache, for example, if the execution status information of the executed tasks is that Task 1 is executed successfully and Task 3 is executed successfully, write the target task execution success information (i.e., Task 2 is executed successfully) into the centralized cache, and read the execution status information of the executed tasks of the first workflow again. If the execution status information is still that Task 1 is executed successfully and Task 3 is executed successfully, then Task 2 is executed successfully and added to the execution status information of the executed tasks of the first workflow. At this time, the execution status information of the executed tasks is that Task 1 is executed successfully, Task 2 is executed successfully, and Task 3 is executed successfully.
[0089] See also Figure 6As shown, a schematic diagram of the working process of the counter is shown. After the executor notifies the counter that the target task is executed successfully, the counter reads the execution status information of the executed tasks of the first workflow from the centralized cache, and writes the target task execution success information into the centralized cache. Determine whether the execution status information of the executed tasks of the first workflow has changed, that is, read the execution status information of the executed tasks of the first workflow again. If the execution status information of the executed tasks of the first workflow has not changed, the execution success information is added to the execution status information of the executed tasks of the first workflow. If the execution status information of the executed tasks of the first workflow has changed, the execution status information of the executed tasks of the first workflow is re-read, and the target task execution success information is written into the centralized cache until the target task execution success information is added to the execution status information of the executed tasks of the first workflow.
[0090] Compared with the pessimistic lock that locks the centralized cache to prohibit other service nodes from writing before writing the target task execution success information into the centralized cache, the counter in the embodiment of the present application adopts an optimistic lock method, which allows other service nodes to operate the centralized cache when writing the target task execution success information. This reduces the locking time and improves the writing efficiency. At the same time, by comparing the execution status information before and after writing, the safety of concurrent writing is guaranteed.
[0091] See also Figure 7 As shown, a schematic diagram of the implementation process of a task scheduling method for multi-tenant isolation on the cloud in an embodiment of the present application is shown. The embodiment of the present application can be applied to a workflow scheduling system, which can be composed of four modules: a memory, a scheduler, an executor, and a counter.
[0092] In actual applications, when a tenant requests a service provider to create, delete or schedule resources, the service provider will automatically generate multiple tasks according to the tenant's needs through the management and control software. The management and control software submits the task to the workflow scheduling system, which generates a workflow based on the task workflow scheduling system, triggers the storage to use cache and persistent storage to save the workflow information composed of task-related information, and returns a successful response to the management and control software.
[0093] The control software or the workflow scheduling system itself creates a counter, and the counter returns a successful response. The control software triggers the workflow to run, and the scheduler adds the task to the distributed message queue and returns a successful response.
[0094] The scheduler in each service node obtains the tasks in the workflow from the distributed message queue, obtains the workflow information from the storage, and adds the tasks to the exclusive message queue of the tenant corresponding to the task. The tasks are read from the tenant's exclusive message queue, and the executor executes the tasks after the concurrency check. The executor also needs to obtain the workflow information from the storage before executing the task. After the executor completes the task execution, the counter is triggered to modify the execution status information of the executed tasks of the workflow in the centralized cache.
[0095] After receiving feedback that the counter modification is successful, the executor notifies the scheduler to schedule the next task and adds the task to the exclusive message queue of the tenant corresponding to the task. The task is read from the tenant's exclusive message queue and executed by the executor after the concurrency check. The executor also needs to obtain workflow information from the storage before executing the task. After the executor completes the task execution, it triggers the counter to modify the execution status information of the executed task of the workflow in the centralized cache.
[0096] The executor can also verify the completion status of the workflow through the counter. After determining that the workflow execution is completed, the scheduler sets the workflow execution success and clears the workflow information in the memory.
[0097] The embodiments of the present application can greatly improve the efficiency and fairness of workflow task scheduling for multiple tenants on the cloud. When sudden requests from certain tenants cause cloud computing services to be overloaded, dynamic scheduling can be performed to effectively ensure that the execution time of workflow tasks for other tenants does not increase significantly, thereby significantly improving the stability of cloud computing services.
[0098] Based on the task scheduling method for multi-tenant isolation on the cloud provided by the above method embodiment, the embodiment of the present application also provides a task scheduling device for multi-tenant isolation on the cloud, which will be described below in conjunction with the accompanying drawings.
[0099] See also Figure 8 As shown, this figure is a structural diagram of a task scheduling device for multi-tenant isolation on the cloud provided by an embodiment of the present application. Figure 8 As shown, the task scheduling device for multi-tenant isolation on the cloud includes:
[0100] A scheduling unit 801 is used to receive a first workflow of a first tenant on the cloud, load balance multiple service nodes in a service cluster through a distributed message queue, and schedule tasks of the first workflow to a target service node;
[0101] A storage unit 802, configured to store the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node;
[0102] The control unit 803 is configured to control the execution of the tasks of the first workflow through the target service node based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant.
[0103] In a possible implementation manner, the device further includes:
[0104] A first determining unit, configured to determine whether the service cluster has a workflow overload according to the number of remaining tasks in the exclusive message queue of each tenant in the service cluster and the remaining exclusive concurrency of the tenant;
[0105] A second determining unit is used to determine the abnormal task type according to the execution time of multiple tasks of the same task type when there is a workflow overload;
[0106] The first adjustment unit is used to adjust the task concurrency corresponding to the abnormal task type.
[0107] In a possible implementation manner, the device further includes:
[0108] A first determining unit, configured to determine whether the service cluster has a workflow overload according to the number of remaining tasks in the exclusive message queue of each tenant in the service cluster and the remaining exclusive concurrency of the tenant;
[0109] A third determining unit, configured to determine an abnormal tenant according to the number of tasks corresponding to the tenant if there is a workflow overload;
[0110] The second adjustment unit is used to adjust the exclusive concurrency of the abnormal tenant.
[0111] In a possible implementation manner, the control unit is specifically configured to:
[0112] Reading tasks from the dedicated message queue of the first tenant in sequence by the target service node based on the dedicated concurrency configured for the first tenant and the task concurrency corresponding to each task type, so that the total number of tasks read is less than or equal to the dedicated concurrency configured for the first tenant and satisfies the task concurrency corresponding to each task type;
[0113] The read task of the first workflow is executed through the target service node.
[0114] In a possible implementation manner, the device further includes:
[0115] An acquisition unit is used to acquire the workflow information of the first workflow from the multi-level cache based on the priority order of the multi-level cache; the multi-level cache includes a local cache, a centralized cache and a persistent storage sorted in descending order of priority.
[0116] In a possible implementation manner, the acquiring unit is specifically configured to:
[0117] Acquire the workflow information of the first workflow from the local cache of the target service node;
[0118] If the workflow information of the first workflow does not exist in the local cache of the target service node, obtain the workflow information of the first workflow from the centralized cache, and save the workflow information of the first workflow to the local cache of the target service node;
[0119] If the workflow information of the first workflow does not exist in the centralized cache, the workflow information of the first workflow is obtained from the persistent storage of the target service node, and the workflow information of the first workflow is saved in the local cache of the target service node.
[0120] In a possible implementation manner, the device further includes:
[0121] A reading unit, configured to read execution status information of executed tasks of the first workflow from a centralized cache after the current task of the first workflow is successfully executed;
[0122] A writing unit, used for writing the current task execution success information of the first workflow into the centralized cache;
[0123] The reading unit is further configured to read the execution status information of the executed tasks of the first workflow from the centralized cache again;
[0124] an adding unit, configured to add the current task execution success information of the first workflow to the execution status information of the executed tasks of the first workflow if the execution status information of the executed tasks of the first workflow read twice is consistent;
[0125] The triggering unit is used to trigger the reading unit to repeatedly read the execution status information of the executed tasks of the first workflow from the centralized cache if the execution status information of the executed tasks of the first workflow read twice is inconsistent.
[0126] In addition, an embodiment of the present application also provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the task scheduling method for multi-tenant isolation on the cloud as described in any of the above items.
[0127] Based on the task scheduling method for multi-tenant isolation on the cloud provided by the above method embodiments, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the task scheduling method for multi-tenant isolation on the cloud described in any of the above embodiments.
[0128] Reference below Fig. 9 , which shows a schematic diagram of the structure of an electronic device 1300 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0129] like Fig. 9 As shown, the electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1306 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 are also stored. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.
[0130] Typically, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1306 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9The electronic device 1300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0131] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1309, or installed from the storage device 1306, or installed from the ROM 1302. When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present application are executed.
[0132] The electronic device provided in the embodiment of the present application and the task scheduling method for multi-tenant isolation on the cloud provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0133] Based on the task scheduling method for multi-tenant isolation on the cloud provided by the above method embodiment, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the task scheduling method for multi-tenant isolation on the cloud as described in any of the above embodiments is implemented.
[0134] It should be noted that the computer-readable medium in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0135] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0136] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.
[0137] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the task scheduling method for multi-tenant isolation on the cloud.
[0138] The computer program code for performing the operation of the embodiment of the present application can be written in one or more programming languages or a combination thereof, and the above-mentioned programming languages include but are not limited to object-oriented programming languages-such as Java, Smalltalk, C++, and also include conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0140] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the unit / module does not, in some cases, constitute a limitation on the unit itself.
[0141] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0142] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0144] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0145] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0146] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0147] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A task scheduling method for multi-tenant isolation on the cloud, characterized in that: The method comprises: Receive a first workflow of a first tenant on the cloud, perform load balancing on multiple service nodes in a service cluster through a distributed message queue, and schedule tasks of the first workflow to a target service node; Storing the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node; The target service node controls the execution of tasks of the first workflow based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant.
2. The method according to claim 1, characterized in that The method further comprises: Determining whether the service cluster has workflow overload according to the number of remaining tasks in the exclusive message queue of each tenant in the service cluster and the remaining exclusive concurrency of the tenant; If there is a workflow overload, determine the abnormal task type based on the execution time of multiple tasks of the same task type; Adjust the task concurrency corresponding to the abnormal task type.
3. The method according to claim 1, characterized in that The method further comprises: Determining whether the service cluster has workflow overload according to the number of remaining tasks in the exclusive message queue of each tenant in the service cluster and the remaining exclusive concurrency of the tenant; If there is a workflow overload, determine the abnormal tenant according to the number of tasks corresponding to the tenant; The exclusive concurrency of the abnormal tenant is adjusted.
4. The method according to any one of claims 1 to 3, characterized in that: The controlling the task execution of the first workflow by the target service node based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant includes: Reading tasks from the dedicated message queue of the first tenant in sequence by the target service node based on the dedicated concurrency configured for the first tenant and the task concurrency corresponding to each task type, so that the total number of tasks read is less than or equal to the dedicated concurrency configured for the first tenant and satisfies the task concurrency corresponding to each task type; The read task of the first workflow is executed through the target service node.
5. The method according to claim 1, characterized in that The method further comprises: The workflow information of the first workflow is obtained from the multi-level cache based on the priority order of the multi-level cache, where the multi-level cache includes a local cache, a centralized cache, and a persistent storage sorted in descending order of priority.
6. The method according to claim 5, characterized in that The acquiring the workflow information of the first workflow from the multi-level cache based on the priority order of the multi-level cache includes: Acquire the workflow information of the first workflow from the local cache of the target service node; If the workflow information of the first workflow does not exist in the local cache of the target service node, obtain the workflow information of the first workflow from the centralized cache, and save the workflow information of the first workflow to the local cache of the target service node; If the workflow information of the first workflow does not exist in the centralized cache, the workflow information of the first workflow is obtained from the persistent storage of the target service node, and the workflow information of the first workflow is saved in the local cache of the target service node.
7. The method according to claim 1, characterized in that The method further comprises: After the current task of the first workflow is successfully executed, reading the execution status information of the executed tasks of the first workflow from the centralized cache; Writing the current task execution success information of the first workflow into the centralized cache; Reading the execution status information of the executed tasks of the first workflow from the centralized cache again; If the execution status information of the executed tasks of the first workflow read twice is consistent, adding the current task execution success information of the first workflow to the execution status information of the executed tasks of the first workflow; If the execution status information of the executed tasks of the first workflow read twice is inconsistent, the execution status information of the executed tasks of the first workflow read from the centralized cache and subsequent steps are repeated.
8. A task scheduling device for multi-tenant isolation on the cloud, characterized in that: The device comprises: A scheduling unit, configured to receive a first workflow of a first tenant on the cloud, load balance a plurality of service nodes in a service cluster through a distributed message queue, and schedule tasks of the first workflow to a target service node; A storage unit, configured to store the tasks of the first workflow in a dedicated message queue of the first tenant in the target service node; A control unit is used to control the task execution of the first workflow through the target service node based on the dedicated concurrency configured for the first tenant and the dedicated message queue of the first tenant.
9. A task scheduling device for multi-tenant isolation on the cloud, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for scheduling tasks for multi-tenant isolation on the cloud as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the task scheduling method for multi-tenant isolation on the cloud as described in any one of claims 1-7.
Citation Information
Patent Citations
The invention discloses a mMethod and a system for achievingrealizing multi-cluster multi-tenant resource isolation based on a Yarn queue
CN109614241A
Distributed workflow scheduling method and device, storage medium and electronic equipment
CN111950988A
Queue scheduling method, device and system based on multi-tenant technology
CN113986497A
Resource scheduling method and device, equipment and storage medium
CN114489978A
Distributed task scheduling method and device, electronic equipment and storage medium
CN114625533A
