A multi-tenant-oriented task scheduling method and related device

Abstract

This application discloses a multi-tenant task scheduling method and related apparatus, relating to the field of cloud computing technology. It receives tasks to be scheduled and their metadata, determines dynamic priority information, and sends the task to a target priority queue if the tenant corresponding to the task has an effective quota. When a tenant has a concurrent consumption token in the target priority queue, it acquires the token, executes the task, and then releases the token. This application sets different dynamic priorities for tasks to be scheduled, ensuring that tasks are distributed to resource-isolated independent queues, guaranteeing priority processing of each task, and improving resource utilization. Before task publication, sliding window rate limiting is implemented, and tenant-level concurrent consumption tokens are distributed during task consumption, isolating tasks between tenants. In complex multi-tenant, high-concurrency production scenarios, this enables intelligent task adjustment, significantly improving resource utilization and scheduling efficiency.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a task scheduling method and related apparatus for multi-tenancy. Background Technology

[0002] With the maturity of cloud computing technology, Software as a Service (SaaS) has become the mainstream method for enterprise application deployment. In a multi-tenant SaaS environment, numerous tenants share a single set of application instances and underlying computing resources. This requires the platform to have efficient, fair, and reliable task scheduling capabilities to handle various tasks such as automated workflows and asynchronous jobs from different tenants.

[0003] Currently, common task scheduling solutions mostly adopt a message queue-based first-in-first-out mechanism, or supplemented by simple static priority settings and fixed-frequency rate limiting strategies. However, when facing complex multi-tenant, high-concurrency production scenarios, these solutions cannot intelligently adjust strategies to prioritize critical business operations, resulting in low resource utilization and scheduling efficiency. Summary of the Invention

[0004] In view of the above problems, this application provides a multi-tenant task scheduling method and related apparatus to achieve high resource utilization and high scheduling efficiency. The specific solution is as follows:

[0005] The first aspect of this application provides a multi-tenant-oriented task scheduling method, including:

[0006] Receive the tasks to be scheduled and their task metadata.

[0007] The dynamic priority information of the task to be scheduled is determined based on the task metadata.

[0008] If the tenant corresponding to the task to be scheduled has an effective quota, the task to be scheduled will be sent to the target priority queue corresponding to the dynamic priority information; the effective quota is determined by sliding window rate limiting processing of the tenant and the preset priority queue information;

[0009] When a tenant has a concurrent consumption token in the target priority queue, acquire the concurrent consumption token, execute the scheduled task, and then release the concurrent consumption token.

[0010] In one possible implementation, the task to be scheduled and its task metadata are received, including:

[0011] Receive the task to be scheduled and its context metadata;

[0012] Context metadata includes at least the task identification information, the tenant identification information corresponding to the task to be scheduled, the task type, the task scenario identification information, and the task importance identification information;

[0013] Context metadata is the same as task metadata.

[0014] In one possible implementation, the dynamic priority information of the task to be scheduled is determined based on the task metadata, including:

[0015] Based on the tenant identifier information in the task metadata, query the preset service level of the tenant corresponding to the task to be scheduled;

[0016] Based on the task importance identifier and preset service level in the task metadata, the target dynamic priority tag is retrieved from the preset dynamic priority tag table. The target dynamic priority tag is the dynamic priority information corresponding to the task to be scheduled.

[0017] In one possible implementation, the process of determining the effective quota includes:

[0018] A unique rate limiting identifier key is determined using the tenant identification information and dynamic priority information corresponding to the task to be scheduled.

[0019] The default actual quota for the tenant corresponding to the task to be scheduled is determined based on the unique rate limiting identifier key;

[0020] Check the preset Redis cache to see if there is a tenant-level dynamic rate limiting quota or a global dynamic rate limiting quota for the tenant.

[0021] When a tenant-level dynamic rate limiting quota exists and is valid, the tenant-level dynamic rate limiting quota will be used as the effective quota.

[0022] When there is no tenant-level dynamic rate limiting quota, but there is a global dynamic rate limiting quota and the global dynamic rate limiting quota is valid, the global dynamic rate limiting quota will be used as the effective quota.

[0023] When neither tenant-level dynamic rate limiting quotas nor global dynamic rate limiting quotas exist, the actual quota will be used as the effective quota by default.

[0024] In one possible implementation, before sending the task to be scheduled to the target priority queue corresponding to the dynamic priority information, the following is also included:

[0025] If the tenant corresponding to the task to be scheduled does not have an effective quota, the time-slice scheduling algorithm is invoked to calculate the executable time point of the task to be scheduled.

[0026] The scheduled tasks are sent to the target priority queue corresponding to the dynamic priority information according to the executable time point.

[0027] In one possible implementation, acquiring a concurrent consumption token includes:

[0028] A unique token identifier is determined using the tenant identification information corresponding to the task to be scheduled and the target priority queue.

[0029] If a task scenario identifier exists in the task metadata of the task to be scheduled, determine whether the task to be scheduled belongs to a regular scenario or a batch scenario based on the task scenario identifier.

[0030] When the task to be scheduled belongs to a regular scenario, a unique token identifier is written into the first concurrent token set to obtain a scenario-level concurrent consumption token, which is the concurrent consumption token.

[0031] When the task to be scheduled belongs to a batch scenario, the unique token identifier is written into the second concurrent token set to obtain the group-level concurrent consumption token, which is the concurrent consumption token.

[0032] One possible implementation also includes:

[0033] Get the current number of tasks being processed, concurrent token usage rate, and task execution metrics;

[0034] Determine the system load status based on the current number of tasks being processed, concurrent token usage rate, and task execution metrics;

[0035] The current limiting parameters are dynamically adjusted based on the system load status.

[0036] The process of determining the effective quota based on dynamically adjusted rate limiting parameters and the process of obtaining concurrent consumption tokens.

[0037] One possible implementation also includes:

[0038] In response to the detection of a scheduled task failing to execute, determine the failure type of the scheduled task;

[0039] If the failure type is retryable and the maximum number of retries has not been exceeded, the scheduled task will be retried according to the preset retry interval; the time intervals corresponding to the preset retry intervals increase exponentially.

[0040] A second aspect of this application provides a multi-tenant task scheduling apparatus, comprising:

[0041] The receiving unit is used to receive the task to be scheduled and the task metadata of the task to be scheduled.

[0042] The determination unit is used to determine the dynamic priority information of the task to be scheduled based on the task metadata.

[0043] The sending unit is used to send the task to be scheduled to the target priority queue corresponding to the dynamic priority information, provided that the tenant corresponding to the task to be scheduled has an effective quota; the effective quota is determined by sliding window rate limiting processing of the tenant and the preset priority queue information;

[0044] The execution unit is used to acquire a concurrent consumption token when a tenant has a concurrent consumption token in the target priority queue, execute the scheduled task, and then release the concurrent consumption token.

[0045] A third aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multi-tenant-oriented task scheduling method described in the first aspect or any implementation thereof.

[0046] By employing the above technical solution, this application provides a task scheduling method and related apparatus for multi-tenant environments. The method receives a task to be scheduled and its task metadata, determines the dynamic priority information of the task based on the metadata, and sends the task to a target priority queue corresponding to the dynamic priority information if the tenant has an effective quota. After determining that the tenant has a concurrent consumption token in the target priority queue, the method acquires the token, executes the task, and then releases the token. This application sets different dynamic priorities for tasks to be scheduled, ensuring that tasks are distributed to resource-isolated independent queues, guaranteeing the priority processing of each task, and improving resource utilization. Furthermore, by implementing sliding window rate limiting before task publication and distributing tenant-level concurrent consumption tokens during task consumption, the method effectively isolates tasks between tenants. This enables intelligent task adjustment in complex multi-tenant, high-concurrency production scenarios, significantly improving resource utilization and scheduling efficiency. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0048] Figure 1 A flowchart illustrating the multi-tenant-oriented task scheduling method provided in this application;

[0049] Figure 2 This application provides a schematic diagram of the structure of a multi-tenant-oriented task scheduling device. Detailed Implementation

[0050] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0051] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0052] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0053] The inventors discovered that existing task scheduling schemes have many problems: uneven resource allocation and lack of SLA (Service Level Agreement) guarantees; poor resistance to sudden traffic surges, which can easily lead to system avalanche; rigid scheduling strategies that cannot adapt dynamically; and crude task failure handling mechanisms that may increase system burden.

[0054] The specific manifestations of the above problems are as follows:

[0055] Uneven resource allocation and lack of SLA guarantees: Single or static priority queue models cannot effectively distinguish the criticality of tasks and the SLA level of their respective tenants. This results in high-priority core business tasks (such as transaction payments) being blocked by a large number of low-priority batch tasks, failing to guarantee the task processing latency and success rate required by critical tenants, and posing a risk of SLA default.

[0056] For example, during a major e-commerce promotion, a tenant's (large brand) inventory deduction and physical trigger workflow task after successful order payment (high SLA, P1 level) and another tenant's historical data archiving and report generation workflow task (low SLA, P3 level) simultaneously influx. Under a FIFO (First-In First-Out) queue, a time-consuming report task may block hundreds of subsequent order tasks that urgently require a response within seconds, directly leading to shipping delays and customer complaints.

[0057] Poor ability to withstand sudden traffic surges can easily lead to system avalanche: When business peaks or a particular "noisy tenant" generates a massive number of tasks, the task queue will accumulate rapidly, and the consumption capacity cannot keep up with the production speed, causing the overall system latency to soar, or even crashing due to resource exhaustion, affecting the service of all tenants.

[0058] For example, a tenant's configuration error triggers tens of thousands of "data synchronization" tasks instantly. In a non-isolated system, these tasks would quickly exhaust all worker threads, causing all normal approval and notification tasks of other tenants in the same system to stop, resulting in a global service interruption.

[0059] The scheduling strategy is rigid and cannot adapt dynamically: the rate limiting and scheduling strategy based on fixed rules cannot perceive the real-time load of the system (such as queue depth and processing latency) and changes in tenant behavior, resulting in idle resources when the load is low and the strategy cannot be intelligently adjusted to prioritize the critical business under high load or sudden situations, resulting in a decline in resource utilization and scheduling efficiency.

[0060] For example, during the off-peak hours of nighttime business, a large amount of system resources are idle, but the batch processing tasks in the background are still executed slowly according to the conservative concurrent configuration during the day, which cannot make full use of idle resources and results in wasted efficiency.

[0061] A crude task failure handling mechanism may increase system load: It employs simple, fixed retry strategies for failed tasks, lacking intelligent retry mechanisms. This can not only trigger a "retry avalanche" under high concurrency, exacerbating the impact on downstream dependent services, but may also lead to the continuous consumption of valuable resources due to the repeated execution of individual abnormal tasks, thus blocking the smooth execution of the entire task flow.

[0062] For example, a brief failure of an external service causes a large number of tasks that depend on that service to fail simultaneously. If retries are performed at fixed intervals (such as retrying after a uniform 5 minutes), a larger retry surge will occur after 5 minutes than the original traffic, further overwhelming the newly recovered external service.

[0063] To address the aforementioned issues, this application provides a multi-tenant-oriented task scheduling method and related apparatus.

[0064] Optional, see Figure 1 This application provides a flowchart of a multi-tenant-oriented task scheduling method.

[0065] like Figure 1 As shown, this multi-tenant-oriented task scheduling method includes the following steps:

[0066] Step 101: Receive the task to be scheduled and its task metadata.

[0067] It should be noted that the application environment of this specific embodiment can be a multi-tenant SaaS environment, and the execution entity can be the workflow engine in that environment. The task to be scheduled can be a specific job that has been submitted to the system and is queuing to be allocated computing resources for execution. For example, the task to be scheduled could be sending a notification or calling an API (Application Programming Interface). Furthermore, a tenant in a multi-tenant SaaS environment is an independent enterprise or customer instance. The system needs to provide services to multiple tenants and ensure data, performance, and configuration isolation between them.

[0068] The task metadata for a task to be scheduled can specifically be the context metadata of that task. This context metadata includes at least the task identifier, the tenant identifier corresponding to the task, the task type, the task scenario identifier, and the task importance identifier. Specifically, the context metadata can include the task ID, the tenant ID to which the task belongs, the task type, the task scenario identifier, and the business importance identifier inherited from the process definition. The task scenario identifier defines whether the task belongs to a regular scenario or a batch scenario; the business importance identifier indicates whether the task is for order fulfillment or log archiving, etc.

[0069] Optionally, the workflow engine receives the task to be scheduled and the context metadata of the task to be scheduled. The context metadata includes at least the task ID, the tenant ID to which the task to be scheduled belongs, the task type, the task scenario identifier, and the business importance identifier inherited from the process definition.

[0070] Step 102: Determine the dynamic priority information of the task to be scheduled based on the task metadata.

[0071] It should be noted that the dynamic priority information can be a specific dynamic priority label corresponding to the task to be scheduled. This dynamic priority label is mainly determined by the tenant identification information and the task importance identification information.

[0072] The process for determining dynamic priority labels is as follows:

[0073] Optionally, based on the business importance identifier in the context metadata and the service level agreement corresponding to the tenant ID, the target dynamic priority label is queried from the preset dynamic priority label table based on the combination of the business importance identifier and the service level agreement.

[0074] In this application, the Service Level Agreement (SLA) specifically refers to the level of business commitment regarding the timeliness and success rate of automated workflow task execution, and is the core basis for prioritizing tasks.

[0075] In summary, in the specific embodiments of this application, the dynamic priority label is determined through a self-designed mapping logic, specifically based on the mapping relationship between the business importance identifier and the service level agreement and the target dynamic priority label. Each dynamic priority label in the preset dynamic priority label table can be P0-urgent, P1-high, P2-medium, and P3-low. The specific mapping process can be as follows:

[0076] First, read the business importance identifier of the task to be scheduled. Then, query the SLA configuration of the tenant to which the task belongs to obtain the service level agreement of the tenant. Finally, based on the combination of the business importance identifier and the service level agreement, query the dynamic priority label of the same service level from the preset dynamic priority label table.

[0077] It should also be noted that the dynamic priority label can change dynamically, or it can be understood as configurable, for the following reasons:

[0078] The dynamic priority label can be dynamically assigned primarily because the tenant SLA configuration supports runtime updates. When an administrator modifies a tenant's SLA configuration, subsequent new tasks from that tenant entering the scheduling system will automatically adapt to the new priority label without requiring a system restart or code modification. Therefore, the dynamic priority label in this application has dynamic adaptability. Furthermore, this dynamic priority label serves as the primary basis for all subsequent scheduling decisions.

[0079] Step 103: If the tenant corresponding to the task to be scheduled has an effective quota, send the task to be scheduled to the target priority queue corresponding to the dynamic priority information; the effective quota is determined by sliding window rate limiting processing of the tenant and the preset priority queue information.

[0080] It's important to note that different dynamic priority tags correspond to different priority queues. A priority queue is a special queue architecture where tasks of different priorities are allocated and stored in physically isolated message queues, such as high-priority queues, medium-priority queues, or low-priority queues. Each priority queue has independent storage space and consumption resources. Therefore, different priority queues can achieve resource independence for tasks of different priorities through physical isolation, preventing low-priority tasks from blocking high-priority tasks.

[0081] In addition, before sending the task to be scheduled to the target priority queue, it is necessary to determine whether the task has an effective quota. The effective quota is determined by sliding window rate limiting processing of the task to be scheduled and the preset priority queue information.

[0082] Sliding window rate limiting is a rate limiting algorithm that divides events into multiple sliding windows and counts the number of requests within each window. Compared to fixed windows, it can control traffic more smoothly and prevent traffic surges at window boundaries. In the multi-tenant shared resource architecture of this application, the sliding window rate limiting algorithm is mainly used to impose a constraint on the task release rate of individual tenants, thereby effectively preventing any tenant from experiencing a sudden surge of tasks, such as tens of thousands of data synchronization tasks triggered instantaneously due to configuration errors, which would quickly fill the queue capacity and crowd out the available resources of other tenants, causing a global service interruption. The process of determining the effective quota can be as follows:

[0083] 1> The unique rate limiting identifier key is determined by using the tenant identification information and dynamic priority information corresponding to the task to be scheduled.

[0084] Specifically, a rate limiting key is constructed for the task to be scheduled, and a unique rate limiting identifier key is generated using the queue scenario of the target priority queue corresponding to the tenant ID and dynamic priority label.

[0085] 2> Determine the default actual quota for the tenant corresponding to the task to be scheduled based on the unique rate limiting identifier key.

[0086] Read the publishing rate limiting quota for the tenant ID from the system configuration, for example, 1000 times per minute. Then check if the tenant has a current limit level. The current limit level is divided into standard level and half limit level. When the tenant's current limit level is standard level, the default actual quota is the above publishing rate limiting quota. When the tenant's current limit level is half limit level, the default actual quota is half of the publishing rate limiting quota, that is, 500 times per minute.

[0087] 3> Query the preset Redis cache to see if there is a tenant-level dynamic rate limiting quota or a global dynamic rate limiting quota for the tenant.

[0088] Next, a dynamic rate limiting strategy is used to determine the effective quota for the tenant corresponding to the task to be scheduled. This dynamic rate limiting strategy is a mechanism that supports dynamic adjustment of rate limiting parameters at runtime. It mainly includes global dynamic control and tenant-level dynamic control. Global dynamic control is mainly used to adjust the rate limiting threshold for all tenants, while tenant-level dynamic control is mainly used to adjust the rate limiting threshold for specific tenants. This mechanism is mainly used to deal with sudden traffic or system overload.

[0089] First, check the default Redis cache to see if there is a dynamic rate limiting policy for this tenant or globally.

[0090] When a tenant-level dynamic rate limiting quota exists and is valid, the tenant-level dynamic rate limiting quota will be used as the effective quota. When no tenant-level dynamic rate limiting quota exists, but a global dynamic rate limiting quota exists and is valid, the global dynamic rate limiting quota will be used as the effective quota. When neither a tenant-level dynamic rate limiting quota nor a global dynamic rate limiting quota exists, the default actual quota will be used as the effective quota.

[0091] In summary, during the determination of effective quotas, there is a three-tier priority coverage relationship between dynamic rate limiting strategies and the default actual quotas mentioned above. Tenant-level dynamic rate limiting quotas have the highest priority, followed by global dynamic rate limiting quotas, and the default actual quotas have the lowest priority.

[0092] The effective quota is one of the following: the default actual quota, the tenant-level dynamic rate limiting quota, or the global dynamic rate limiting quota, and the number of deliveries made by the tenant has not reached the quota limit.

[0093] Specifically, the sliding window rate limiter implemented based on Redis counts the number of tasks delivered by the tenant to which the task to be scheduled belongs within its corresponding current sliding window. If the quota limit has not been reached, it is determined that the tenant to which the task to be scheduled belongs has an effective quota, allowing delivery and incrementing the count by 1.

[0094] If it is determined that the number of tasks delivered by the tenant to which the task to be scheduled belongs within its corresponding current sliding window has reached the quota limit, then it is determined that the tenant to which the task to be scheduled belongs does not have an effective quota. A delayed delivery event is calculated, the system calls the time-slice scheduling algorithm to calculate the next executable time point, sets the task to be scheduled to a delayed delivery state, and resumes the delivery process after the executable time point is reached.

[0095] Specifically, if the tenant corresponding to the task to be scheduled does not have an effective quota, the time slice scheduling algorithm is invoked to calculate the executable time point of the task to be scheduled, and the task to be scheduled is sent to the target priority queue corresponding to the dynamic priority information according to the executable time point.

[0096] The time-slice scheduling algorithm is mainly used to schedule tasks to a future event slice when they exceed concurrency limits and cannot be executed immediately. Each time slice has a fixed duration and maximum capacity. Tasks exceeding the capacity of the current time slice are automatically scheduled to the next time slice, achieving smoothing of the load over time.

[0097] In summary, the operation of determining whether the tenant to which the task belongs has an effective quota before delivering the task can control the load generated by any tenant per unit time from the source, effectively preventing sudden traffic surges from impacting the system.

[0098] Optionally, if the tenant corresponding to the task to be scheduled has an effective quota, the task to be scheduled will be sent to the target priority queue corresponding to the dynamic priority information.

[0099] Step 104: When a tenant has a concurrent consumption token in the target priority queue, acquire the concurrent consumption token, execute the task to be scheduled, and then release the concurrent consumption token.

[0100] It's important to note that concurrent consumption tokens are a token-based concurrency control mechanism. The scheduling system allocates a fixed number of concurrent tokens to each tenant in each priority queue. A token must be acquired before task execution and released after completion. Specifically, distributed token management can be implemented using Redis sorted sets, supporting automatic token release upon timeout.

[0101] First, determine whether a concurrent consumption token exists in the target priority queue of the tenant to which the task to be scheduled belongs. If a concurrent consumption token exists, acquire it. The acquisition process can be as follows:

[0102] 1> Generate token identifier.

[0103] A unique random UUID (Universally Unique Identifier) ​​is generated for the task to be scheduled using the tenant ID and target priority queue information. This token identifier can also be understood as a placeholder. This unique random UUID serves as the sole identity credential for this concurrency permission. Throughout the entire lifecycle of the task, including acquisition, holding, release, and timeout cleanup, the scheduling system will use this UUID to uniquely identify and operate the concurrency consumption token.

[0104] Then, using the tenant ID and the queue scenario where the task resides as input parameters, a unique string key, also known as the token pool storage key, is generated according to preset rules. This string key is the key name of a sorted set in Redis, uniquely identifying the storage address of all active concurrent tokens belonging to this tenant in that queue. The generated token identifier UUID will also be stored as a member of this sorted set.

[0105] 2> If the task meta information of the task to be scheduled contains a task scenario identifier, determine whether the task to be scheduled belongs to a regular scenario or a batch scenario based on the task scenario identifier.

[0106] Then, it is determined whether there is a task scenario identifier in the task metadata of the task to be scheduled. If there is, the two-layer concurrent token check strategy needs to be called and executed. This strategy mainly defines the priority and logical dimensions of resource checks.

[0107] When a task to be scheduled carries a task scenario identifier, two levels of quota checks need to be performed sequentially.

[0108] The first layer involves scenario-level quota checks. The scheduling system checks the overall quota corresponding to the scenario to which the task to be scheduled belongs, as indicated by the task scenario identifier. Quotas are usually allocated proportionally. When the task scenario identifier indicates that the task to be scheduled belongs to a regular scenario, 80% of the concurrent consumption token quota is generally allocated; when the task scenario identifier indicates that the task to be scheduled belongs to a batch scenario, 20% of the concurrent consumption token quota is generally allocated. This ensures that resources are prioritized for regular and high-priority businesses, preventing time-consuming and high-traffic batch tasks from crowding out resources and impacting regular businesses.

[0109] The second layer is group-level quota checking. After passing the scenario-level check, the scheduling system further checks whether the specific group to which the task belongs has any remaining quota. This step enables finer-grained control over different business lines or task sources within the same scenario.

[0110] 3> When the task to be scheduled belongs to a regular scenario, the unique token identifier is written into the first concurrent token set to obtain the scenario-level concurrent consumption token, which is the concurrent consumption token;

[0111] When the task to be scheduled belongs to a batch scenario, the unique token identifier is written into the second concurrent token set to obtain the group-level concurrent consumption token, which is the concurrent consumption token.

[0112] The previous step's concurrent consumption token check primarily aimed to determine if the quota was met. This step's concurrent consumption token request primarily aims to write placeholder identifiers into the set. It's important to note that the check and request can be encapsulated within the same Redis Lua script's atomic operation, ensuring that over-issuance does not occur under high concurrency. The specific process can be as follows:

[0113] First, expired tokens are cleaned up. The script first cleans up all timed-out placeholders in the token pool (the Redis sorted set mentioned above) based on the current timestamp. This step mainly releases tokens that failed to be released normally due to task crashes or other reasons, and is a key self-healing mechanism to prevent token leakage from leading to resource exhaustion.

[0114] Then, a two-level quota determination is performed. The number of tokens used in the current scenario is calculated and it is determined whether it is less than the quota. The number of tokens used in the current group is calculated and it is determined whether it is less than the quota. If both are less than the quota, the token placeholder and the current timestamp are written into the ordered set of the corresponding token pool, and the concurrent consumption token is successfully obtained.

[0115] In addition, to prevent token leakage due to abnormal task termination, a timeout period for the token needs to be set, such as a timeout period of 4 minutes, after which the token will be automatically released.

[0116] It's easy to understand that there are cases where concurrent token acquisition fails. If concurrent consumption token acquisition fails, it means that the tenant's current concurrency has reached its limit. The scheduling system will also call the time slice scheduling algorithm to calculate the next retry time and delay the execution of the task.

[0117] Optionally, obtain the concurrent consumption token issued by the tenant corresponding to the task to be scheduled in the target priority queue, and release the concurrent consumption token after the task is executed.

[0118] In summary, this step ensures that within the same priority queue, the task consumption capabilities of different tenants are isolated and limited, avoiding the situation where a single tenant occupies all consumption resources.

[0119] In this step, acquiring concurrent consumption tokens, and in the previous step, acquiring effective quotas, both present the possibility that tasks cannot be executed immediately due to exceeding the concurrent consumption token quota and the rate limiting quota. The scheduling system uses a time-slice scheduling algorithm to calculate the next execution time. The specific implementation process is as follows:

[0120] 1> Obtain the scheduling policy.

[0121] The system configuration center retrieves the core parameters of the time-slice scheduling strategy: time-slice duration and maximum scheduling capacity per time slice. The time-slice duration is the length of each scheduling time window, for example, 60 seconds. This is the basic unit for dividing continuous time into discrete processing batches. The maximum scheduling capacity per time slice is the maximum number of tasks allowed to be scheduled within a single time slice, for example, 200. This parameter is crucial for controlling the task deployment rate and achieving smooth distribution.

[0122] 2> Calculate the start point of the next time slice.

[0123] Obtain the current precise system timestamp, and based on the time slice duration defined in the previous step, round the current timestamp up to the nearest complete time slice boundary. For example, if the current time is 12:00:35 and the time slice is 60 seconds, then the start point of the next time slice is calculated as 12:01:00. This ensures that all scheduling decisions are based on a unified, aligned time window.

[0124] 3> Query the current scheduling status.

[0125] To query the current system's scheduling progress status in an atomic manner, typically using distributed storage such as Redis, two key values ​​need to be obtained: the current maximum valid time slice number and the count of tasks scheduled in that time slice.

[0126] The current maximum valid time slice number is the time slice number to which the system is currently scheduling or the next task to be scheduled belongs; the time slice scheduled task count is the number of tasks that have been successfully allocated within the current active time slice. This count is used to determine whether the current time slice is saturated.

[0127] 4> Perform atomic scheduling and allocation.

[0128] This step is typically implemented using Redis Lua scripts, and the decision logic is as follows:

[0129] If the current time slice is not saturated (i.e., the number of scheduled tasks in the current time slice is less than the maximum capacity), the current task is assigned to the current time slice, and the scheduled task count for that time slice is incremented by 1 using an atomic operation. If the current time slice is saturated (i.e., the number of scheduled tasks in the current time slice is greater than or equal to the maximum capacity), it means that the current time slice is full. The system increments the maximum time slice number by 1, i.e., switches to the next time slice, and assigns the current task to this new next time slice, while initializing its scheduled count for that new time slice to 1.

[0130] 5> Calculate the precise execution time.

[0131] Once a task is assigned to a specific time slice (e.g., time slice number N), a precise execution time point needs to be determined for it within that time slice window. The calculation logic is: Task execution time = start timestamp of time slice N + (task's sequence number within the time slice * fixed millisecond offset).

[0132] The purpose of this step is to ensure that even if a large number of tasks are assigned to the same time slice, their actual execution times are dispersed by calculating a uniformly distributed, millisecond-level small offset for different tasks within the same time slice, thereby avoiding execution spikes at the beginning of the time slice.

[0133] 6> Update status and entry delay.

[0134] The precise next execution timestamp calculated in the previous step is updated in the task message's metadata. Then, the task is submitted to a delayed queue or timer, putting it into a delayed waiting state until the specified execution time arrives before being retrieved and executed by the consumer.

[0135] The time-slice scheduling algorithm proposed in this application solves the problem that concurrent consumption tokens cannot be executed immediately due to exceeding the concurrent consumption token quota and the rate limiting quota during the acquisition of concurrent consumption quota and effective quota. It realizes the smooth distribution of hyperconcurrency tasks in the time dimension and avoids system overload caused by concentrated retries.

[0136] In summary, this application provides a multi-tenant task scheduling method that receives a task to be scheduled and its task metadata, determines the dynamic priority information of the task to be scheduled based on the task metadata, and sends the task to the target priority queue corresponding to the dynamic priority information if the tenant corresponding to the task to be scheduled has an effective quota. After determining that the tenant has a concurrent consumption token in the target priority queue, the concurrent consumption token is acquired, the task to be scheduled is executed, and then the concurrent consumption token is released. This application sets different dynamic priorities for the tasks to be scheduled, thereby ensuring that tasks are distributed to resource-isolated independent queues, guaranteeing the priority processing rights of each task, and improving resource utilization. Furthermore, by implementing sliding window rate limiting before task publication and distributing tenant-level concurrent consumption tokens during task consumption, tasks between tenants are effectively isolated. This enables intelligent task adjustment in complex multi-tenant, high-concurrency production scenarios, greatly improving resource utilization and scheduling efficiency.

[0137] Optionally, the above-mentioned multi-tenant-oriented task scheduling method also includes a scheduling parameter adjustment strategy and an exponential backoff retry strategy.

[0138] It should be noted that the scheduling parameter adjustment strategy can be implemented by an independent adaptive scheduler, which periodically collects the system's operating status and adjusts the scheduling parameters.

[0139] The scheduling parameter adjustment strategy includes the following steps:

[0140] 1> Get the number of currently processed tasks, concurrent token usage rate, and task execution metrics.

[0141] The number of currently processed tasks can be referred to as priority queue information, which can specifically be a queue depth indicator. In this application, the number of currently processed tasks specifically refers to the cumulative number of tasks waiting to be consumed and processed in a certain priority queue at a certain moment. It is a key system load indicator that directly reflects the congestion level of the queue. For example, a high-priority queue with a depth of 5 indicates that there are currently 5 high-priority tasks waiting to be executed. By monitoring this indicator, the system can determine whether load balancing, backpressure triggering, or consumption capacity adjustment is necessary.

[0142] Concurrent token utilization rate can also be referred to as concurrent token occupancy information. This concurrent token utilization rate can characterize the concurrent consumption token occupancy of each tenant and each queue when querying the rate limiting system.

[0143] Task execution information mainly includes task execution metrics, specifically metrics such as average processing latency, success rate, and failure rate obtained from the monitoring system.

[0144] 2> Determine the system load status based on the current number of tasks being processed, concurrent token usage rate, and task execution metrics.

[0145] Based on the above information, determine whether the current system is in a normal, warning, or overload state.

[0146] When the system is determined to be in an overload state, the overload protection and adaptive degradation mechanism will be triggered immediately.

[0147] The execution process of overload protection and adaptive degradation mechanism can be as follows:

[0148] System overload detection:

[0149] The system continuously monitors the core load metrics of each physically isolated queue. When at least one of the following conditions is met—excessive queue depth or excessive waiting time—an overload judgment is triggered. Excessive queue depth is defined as the number of backlogged tasks in any priority queue exceeding a preset high-water mark threshold; excessive waiting time is defined as the average waiting time of tasks in any priority queue exceeding a preset latency threshold.

[0150] Tighten upstream entry flow restrictions:

[0151] Upon overload determination, the adaptive scheduler immediately sends a dynamic adjustment command to the sliding window rate limiting layer located at the traffic inlet in step 103. This command temporarily and dynamically lowers the task release rate limiting quota for the global or specific high-traffic tenants. The core function of this command is to reduce the inflow of new tasks from the source, thereby reducing the system load.

[0152] Perform internal resource degradation scheduling:

[0153] While tightening the entry point, the adaptive scheduler adjusts its internal resource allocation strategy. Specifically, this includes dynamically reducing the concurrent consumption token quota of low-priority queues, thereby reducing the computing resources allocated to them; concentrating the saved resources on high-priority queues to ensure that tasks in the core business chain can be scheduled and executed in a timely manner; and coordinating anti-starvation rules to adhere to preset anti-starvation rules during the degradation process, reserving a minimum guaranteed concurrent quota for low-priority queues to prevent them from being completely deprived of resources and falling into a starvation state, thus maintaining the basic service fairness of the system.

[0154] Continuous monitoring and automatic recovery:

[0155] During the overload protection period, the system continuously monitors load metrics such as queue depth and waiting time. When all metrics consistently drop below the preset low water level, it indicates that the overload risk has been eliminated. At this point, the system will gradually and smoothly restore the inlet flow limit quotas tightened during the tightening of upstream inlet flow limits, as well as the low-priority queue resource quotas reduced during the execution of internal resource degradation scheduling, until the system fully recovers to normal scheduling policy parameters. This process ensures that the system can automatically return to normal after the pressure subsides, forming a complete adaptive closed loop.

[0156] The overload protection and adaptive degradation mechanism enables the system to protect itself under unexpected loads and proactively degrade to protect core business operations.

[0157] 3> Determine and dynamically adjust the current limiting parameters based on the system load status.

[0158] The system executes corresponding adjustment strategies based on the load status. If a tenant is found to have a severe backlog of tasks or an abnormally high failure rate, the system writes the tenant's dynamic rate limiting policy into the rate limiting system and reduces its published rate limiting quota. If the global load is found to be too high, the system writes a global dynamic rate limiting policy and reduces the rate limiting quota of all tenants. When the load returns to normal, the corresponding dynamic rate limiting policy is deleted and the default quota is restored.

[0159] 4> The process of determining the effective quota based on dynamically adjusted rate limiting parameters and the process of obtaining concurrent consumption tokens.

[0160] The rate limiting system stores dynamic rate limiting policies, including global configurations and tenant-level configurations. These policies are read when rate limiting judgments are performed in steps 103 and 104 above.

[0161] In summary, this scheduling parameter adjustment strategy achieves closed-loop feedback control from monitoring to decision-making to execution.

[0162] In addition, the consumer adjusts parameters such as the number of concurrent tokens dynamically adjusted according to the scheduling parameters mentioned above, and executes the task of obtaining tokens in step 104. The specific execution process may include the following steps:

[0163] 1> Lock the task.

[0164] After retrieving a task from the message queue, the task is first distributed and locked by setting a locker identifier or a lock time to prevent duplicate consumption.

[0165] 2> Execute business logic.

[0166] The corresponding task processor is invoked to execute the specific business logic.

[0167] 3> Record the execution results.

[0168] If execution is successful, update the task status to "success", record the execution completion time, and set the task recycling time; if execution fails, enter the retry process in the exponential backoff retry strategy according to the task type.

[0169] 4> Release concurrent consumption tokens.

[0170] Regardless of whether the execution is successful or not, the acquired concurrent consumption token must be released.

[0171] 5> Report performance indicators.

[0172] The execution results, execution time, error codes, and other metrics are reported to the monitoring system for use by the adaptive scheduler in the scheduling parameter adjustment strategy. These performance metrics are the core input data for the adaptive scheduler to enter the next round of decision-making, thus forming a continuous optimization loop.

[0173] For the tasks that fail to execute as described above, this application adopts an exponential backoff retry strategy to isolate tasks that have exceeded the maximum number of retries.

[0174] The index retreat and retry strategy includes the following steps:

[0175] 1> In response to the detection of a scheduled task failing to execute, determine the failure type of the scheduled task.

[0176] Read the error codes and distinguish between retryable and non-retryable errors.

[0177] For example, retryable errors include network timeouts and service being temporarily unavailable; non-retryable errors include business logic errors and parameter validation failures.

[0178] For non-retryable errors, directly set the task status to failure, set the task recycling time, and stop retrying.

[0179] 2> If the failure type is retryable and the maximum number of retries has not been exceeded, the scheduled task will be retried according to the preset retry interval; the time intervals corresponding to the preset retry intervals increase exponentially.

[0180] For retryable errors, calculate the current number of retries. Specifically, read the number of retries (retry times) and the number of delays (delay times) of the task, and calculate the current actual number of retries. The calculation formula can be: Current actual number of retries = max(retry times - delay times, 0) + 1.

[0181] Then, check if the maximum number of retries has been exceeded. If the current number of retries exceeds the maximum number of retries, set the task status to expired, stop retries, and set the task recycling time.

[0182] If the maximum number of retries has not been exceeded, the corresponding backoff interval is obtained by looking up the table based on the current number of retries: 1st retry: 5 minutes later; 2nd retry: 15 minutes later; 3rd retry: 1 hour later; 4th retry: 2 hours later.

[0183] Add the backoff interval to the current time to get the next retry time. Then, increment the retry count by 1, adding 1 to the number of retries (retry_times) for the task.

[0184] At this point, the task enters a delayed waiting state. After the next retry time is reached, the compensation task will scan and re-deliver the message to the message queue.

[0185] The exponential backoff retry strategy adopted in this embodiment can effectively avoid retry storms and effectively prevent faulty tasks from continuously consuming system resources through the expiration isolation mechanism.

[0186] It should also be noted that, in the specific embodiments of this application, anti-starvation rules are embedded in all scheduling decisions to ensure minimum throughput.

[0187] Specifically, the adaptive scheduler and overload protection and adaptive degradation mechanisms in the scheduling parameter adjustment strategy both pre-define and enforce an anti-starvation rule: regardless of the system load, a minimum, non-preemptible concurrent consumption token quota (e.g., at least one) is reserved for each physically isolated priority queue (especially low-priority queues). This is implemented by ensuring that when adjusting the concurrent consumption token quota, the adjusted value is never less than the pre-determined minimum. This rule ensures that low-priority tasks are never completely "starved," guaranteeing basic fairness in long-term system operation and the eventual completion of all business flows.

[0188] In summary, the multi-tenant task scheduling method provided in this application addresses the issues of uneven resource allocation and lack of SLA guarantees by assigning dynamic priority labels and routing tasks to physically isolated target priority queues. It dynamically maps the SLA requirements of the business layer into specific, schedulable priority labels, and distributes tasks to resource-isolated independent queues based on these labels. This breaks the traditional single FIFO queue model, ensuring that high-SLA tasks do not physically compete with low-SLA tasks for the same channel, thus architecturally guaranteeing their right to be processed first.

[0189] To address the issue of poor resilience to sudden traffic surges and the potential for system avalanches, a tenant-level sliding window rate limiting and tenant-level concurrent consumption tokens are employed. Tenant-level sliding window rate limiting is implemented at the task publishing entry point, and tenant-level concurrent token isolation is implemented during task consumption. These two features constitute a double layer of protection against "noisy tenants": firstly, limiting the number of tasks they can publish per unit of time at the source; secondly, limiting the number of processing resources they can simultaneously consume, thereby effectively isolating tenants and preventing local overload from escalating into a global avalanche. For hyperconcurrency issues, a time-slice scheduling algorithm is used. Tasks exceeding the concurrency limit are not simply rejected or immediately retried, but are allocated to future time slices for execution. Each time slice is 60 seconds, accommodating a maximum of 200 tasks; tasks exceeding this capacity are automatically postponed. This feature achieves smooth load over time, avoiding system jitter caused by instantaneous peaks. For extreme overload issues, a triggered backpressure mechanism is used. A system load threshold is set, and backpressure is automatically triggered during overload to suppress task inflow and internally degrade non-core services. This feature gives the system the ability to "self-protect" under unexpected loads, proactively degrading to protect core business operations rather than passively crashing.

[0190] To address the issue of rigid scheduling strategies that cannot adapt dynamically, an adaptive scheduler is employed to dynamically adjust parameters and incorporate performance metrics based on consumer feedback. This establishes an independent, periodically running scheduling decision-making loop. This loop takes real-time collected system metrics (queue depth, latency, success rate, etc.) as input and dynamically calculates and outputs adjustment instructions for scheduling parameters (such as global or tenant-level release rate limiting quotas). This transforms the scheduling strategy from a fixed number in a configuration file into a "live" strategy that evolves in real-time based on actual system conditions, thereby optimizing resource utilization and handling load fluctuations. Furthermore, by embedding anti-starvation rules, a minimum amount of processing resources is reserved for low-priority queues during dynamic adjustments and backpressure degradation. This feature resolves the "starvation" problem that can arise from purely dynamic scheduling, ensuring the long-term fairness and robustness of the scheduling system and avoiding new risks introduced by overly flexible strategies.

[0191] To address the issue of a crude task failure handling mechanism potentially increasing system load, an exponential backoff retry strategy is employed. This strategy uses exponentially increasing retry intervals (5 minutes → 15 minutes → 1 hour → 2 hours) for failed tasks, instead of fixed intervals. This effectively distributes retry requests across time, preventing a "retry storm" caused by a large number of failed tasks simultaneously triggering retries, and providing a buffer time for system recovery. Expired task isolation is also implemented. Tasks exceeding the maximum number of retries (4) are marked as "expired" and retrying is stopped. This prevents faulty tasks from continuously consuming system resources while retaining task records for subsequent manual analysis and compensation.

[0192] This application provides a multi-tenant-oriented task scheduling method with interlocking links, multi-layered defense, and closed-loop feedback, systematically constructing an intelligent and robust scheduling system. It not only solves individual problems such as priority differentiation, tenant isolation, elastic scaling, and failure retries, but more importantly, it organically integrates these solutions through a complete logical chain of "perception-decision-execution-protection," forming a technical solution with overall performance superior to existing technologies.

[0193] The above describes a multi-tenant-oriented task scheduling method provided by the embodiments of this application. The following describes the apparatus for executing the above-described multi-tenant-oriented task scheduling method.

[0194] Please see Figure 2 , Figure 2 This application provides a structural schematic diagram of a multi-tenant-oriented task scheduling device. Figure 2 As shown, the device includes:

[0195] The unit comprises a receiving unit 10, a determining unit 20, a sending unit 30, and an execution unit 40; wherein:

[0196] The receiving unit 10 is used to receive the task to be scheduled and the task metadata of the task to be scheduled.

[0197] The determining unit 20 is used to determine the dynamic priority information of the task to be scheduled based on the task metadata.

[0198] The sending unit 30 is used to send the task to be scheduled to the target priority queue corresponding to the dynamic priority information when the tenant corresponding to the task to be scheduled has an effective quota; the effective quota is determined by sliding window rate limiting processing of the tenant and the preset priority queue information;

[0199] Execution unit 40 is used to acquire a concurrent consumption token when a tenant has a concurrent consumption token in the target priority queue, execute the scheduled task, and then release the concurrent consumption token.

[0200] In one embodiment, the receiving unit 10 is specifically used for:

[0201] Receive the task to be scheduled and its context metadata;

[0202] Context metadata includes at least the task identification information, the tenant identification information corresponding to the task to be scheduled, the task type, the task scenario identification information, and the task importance identification information;

[0203] Context metadata is the same as task metadata.

[0204] In one embodiment, the determining unit 20 is specifically used for:

[0205] Based on the tenant identifier information in the task metadata, query the preset service level of the tenant corresponding to the task to be scheduled;

[0206] Based on the task importance identifier and preset service level in the task metadata, the target dynamic priority tag is retrieved from the preset dynamic priority tag table. The target dynamic priority tag is the dynamic priority information corresponding to the task to be scheduled.

[0207] In one embodiment, the process of determining the effective quota in the sending unit 30 includes:

[0208] A unique rate limiting identifier key is determined using the tenant identification information and dynamic priority information corresponding to the task to be scheduled.

[0209] The default actual quota for the tenant corresponding to the task to be scheduled is determined based on the unique rate limiting identifier key;

[0210] Check the preset Redis cache to see if there is a tenant-level dynamic rate limiting quota or a global dynamic rate limiting quota for the tenant.

[0211] When a tenant-level dynamic rate limiting quota exists and is valid, the tenant-level dynamic rate limiting quota will be used as the effective quota.

[0212] When there is no tenant-level dynamic rate limiting quota, but there is a global dynamic rate limiting quota and the global dynamic rate limiting quota is valid, the global dynamic rate limiting quota will be used as the effective quota.

[0213] When neither tenant-level dynamic rate limiting quotas nor global dynamic rate limiting quotas exist, the actual quota will be used as the effective quota by default.

[0214] In one embodiment, the multi-tenant-oriented task scheduling device further includes a time scheduling unit before sending the task to be scheduled to the target priority queue corresponding to the dynamic priority information.

[0215] This time scheduling unit is specifically used for:

[0216] If the tenant corresponding to the task to be scheduled does not have an effective quota, the time-slice scheduling algorithm is invoked to calculate the executable time point of the task to be scheduled.

[0217] The scheduled tasks are sent to the target priority queue corresponding to the dynamic priority information according to the executable time point.

[0218] In one embodiment, the execution unit 40 is specifically used for:

[0219] A unique token identifier is determined using the tenant identification information corresponding to the task to be scheduled and the target priority queue.

[0220] If a task scenario identifier exists in the task metadata of the task to be scheduled, determine whether the task to be scheduled belongs to a regular scenario or a batch scenario based on the task scenario identifier.

[0221] When the task to be scheduled belongs to a regular scenario, a unique token identifier is written into the first concurrent token set to obtain a scenario-level concurrent consumption token, which is the concurrent consumption token.

[0222] When the task to be scheduled belongs to a batch scenario, the unique token identifier is written into the second concurrent token set to obtain the group-level concurrent consumption token, which is the concurrent consumption token.

[0223] In one embodiment, the multi-tenant-oriented task scheduling device further includes a scheduling parameter adjustment unit;

[0224] This scheduling parameter adjustment unit is specifically used for:

[0225] Get the current number of tasks being processed, concurrent token usage rate, and task execution metrics;

[0226] Determine the system load status based on the current number of tasks being processed, concurrent token usage rate, and task execution metrics;

[0227] The current limiting parameters are dynamically adjusted based on the system load status.

[0228] The process of determining the effective quota based on dynamically adjusted rate limiting parameters and the process of obtaining concurrent consumption tokens.

[0229] In one embodiment, the multi-tenant-oriented task scheduling apparatus further includes an exponential backoff retry unit;

[0230] This index-based backoff and retry unit is specifically used for:

[0231] In response to the detection of a scheduled task failing to execute, determine the failure type of the scheduled task;

[0232] If the failure type is retryable and the maximum number of retries has not been exceeded, the scheduled task will be retried according to the preset retry interval; the time intervals corresponding to the preset retry intervals increase exponentially.

[0233] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the multi-tenant-oriented task scheduling methods provided in this application.

[0234] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0235] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0236] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0237] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A task scheduling method for multi-tenancy, characterized in that, include: Receive the task to be scheduled and the task metadata of the task to be scheduled; The dynamic priority information of the task to be scheduled is determined based on the task metadata. If the tenant corresponding to the task to be scheduled has an effective quota, the task to be scheduled will be sent to the target priority queue corresponding to the dynamic priority information. The effective quota is determined by applying sliding window rate limiting to the tenant and the preset priority queue information. When the tenant has a concurrent consumption token in the target priority queue, the concurrent consumption token is acquired, the scheduled task is executed, and then the concurrent consumption token is released.

2. The task scheduling method for multi-tenancy as described in claim 1, characterized in that, The process of receiving the task to be scheduled and the task metadata of the task to be scheduled includes: Receive the task to be scheduled and its context metadata; The context metadata includes at least the task identification information, the tenant identification information corresponding to the task to be scheduled, the task type, the task scenario identification, and the task importance identification information; The context metadata is the task metadata.

3. The task scheduling method for multi-tenancy as described in claim 2, characterized in that, Determining the dynamic priority information of the task to be scheduled based on the task metadata includes: Based on the tenant identifier information in the task metadata, query the preset service level of the tenant corresponding to the task to be scheduled; Based on the task importance identification information in the task metadata and the preset service level, the target dynamic priority tag is queried from the preset dynamic priority tag table. The target dynamic priority tag is the dynamic priority information corresponding to the task to be scheduled.

4. The task scheduling method for multi-tenancy as described in claim 1, characterized in that, The process for determining the effective quota includes: A unique rate limiting identifier key is determined using the tenant identification information and dynamic priority information corresponding to the task to be scheduled. The default actual quota for the tenant corresponding to the task to be scheduled is determined based on the unique rate limiting identifier key. Query the preset Redis cache to see if the tenant-level dynamic rate limiting quota or global dynamic rate limiting quota exists for the tenant; When the tenant-level dynamic rate limiting quota exists and the tenant-level dynamic rate limiting quota is valid, the tenant-level dynamic rate limiting quota shall be used as the effective quota; When there is no tenant-level dynamic rate limiting quota, but there is a global dynamic rate limiting quota and the global dynamic rate limiting quota is valid, the global dynamic rate limiting quota will be used as the effective quota. When neither the tenant-level dynamic rate limiting quota nor the global dynamic rate limiting quota exists, the default actual quota will be used as the effective quota.

5. The task scheduling method for multi-tenancy as described in claim 1, characterized in that, Before sending the task to be scheduled to the target priority queue corresponding to the dynamic priority information, the method further includes: If the tenant corresponding to the task to be scheduled does not have an effective quota, the time-slice scheduling algorithm is invoked to calculate the executable time point of the task to be scheduled. The task to be scheduled is sent to the target priority queue corresponding to the dynamic priority information at the specified executable time point.

6. The task scheduling method for multi-tenancy as described in claim 1, characterized in that, Obtaining the concurrent consumption token includes: A unique token identifier is determined using the tenant identification information corresponding to the task to be scheduled and the target priority queue; If a task scenario identifier exists in the task metadata of the task to be scheduled, determine whether the task to be scheduled belongs to a regular scenario or a batch scenario based on the task scenario identifier. When the task to be scheduled belongs to a regular scenario, the unique token identifier is written into the first concurrent token set to obtain a scenario-level concurrent consumption token, which is the concurrent consumption token. When the task to be scheduled belongs to a batch scenario, the unique token identifier is written into the second concurrent token set to obtain a group-level concurrent consumption token, which is the concurrent consumption token.

7. The multi-tenant-oriented task scheduling method according to claim 1, characterized in that, Also includes: Get the current number of tasks being processed, concurrent token usage rate, and task execution metrics; The system load status is determined based on the current number of tasks being processed, the concurrent token usage rate, and the task execution metrics. The current limiting parameters are dynamically adjusted based on the system load status. The process of determining the effective quota based on the dynamically adjusted rate limiting parameters and the process of obtaining the concurrent consumption token.

8. The multi-tenant-oriented task scheduling method according to claim 1, characterized in that, Also includes: In response to detecting that a scheduled task has failed to execute, the failure type of the scheduled task is determined; If the failure type is a retryable type and the maximum number of retries has not been exceeded, the scheduled task will be retried according to the preset retry interval; the time intervals corresponding to the preset retry intervals increase exponentially.

9. A task scheduling device for multi-tenancy, characterized in that, include: The receiving unit is used to receive the task to be scheduled and the task metadata of the task to be scheduled; The determining unit is used to determine the dynamic priority information of the task to be scheduled based on the task metadata. The sending unit is configured to send the task to be scheduled to the target priority queue corresponding to the dynamic priority information, provided that the tenant corresponding to the task to be scheduled has an effective quota. The effective quota is determined by applying sliding window rate limiting to the tenant and the preset priority queue information. An execution unit is configured to acquire a concurrent consumption token when the tenant has a concurrent consumption token in the target priority queue, execute the scheduled task, and then release the concurrent consumption token.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the multi-tenant-oriented task scheduling method as described in any one of claims 1 to 8.

Images