Method and system for recovery from interruption based on ehpc cluster deployment

Abstract

The present disclosure provides an EHPC cluster deployment-based interruption recovery method and system. The EHPC cluster deployment-based interruption recovery method of an embodiment comprises: based on tenant identification of a plurality of deployment tasks, aggregating deployment tasks belonging to the same tenant into batch task groups, and performing service priority classification on each deployment task within the batch task groups; based on the grouped deployment tasks, constructing a layered token bucket of a resource and interface two-dimensional four-layer structure and dividing platform resources used for executing the deployment tasks into mutually isolated reserved core resource pools and shared resource pools; in response to detecting a deployment service restart, determining the admission qualification of the reserved core resource pools according to the service priority within the corresponding batch task group and allocating tokens of the layered token bucket; and performing an interruption recovery process of the deployment tasks allocated through the tokens. The present disclosure embodiment can ensure normal interruption recovery of core tasks under an extreme concurrent scenario.

Description

Technical Field

[0001] This disclosure relates to the field of computers. More specifically, it relates to an interrupt recovery method based on EHPC cluster deployment, an interrupt recovery system based on EHPC cluster deployment, a computer device, and a computer-readable storage medium. Background Technology

[0002] Automated deployment of Elastic High-Performance Computing (EHPC) clusters is a complex process involving multiple ordered steps, typically including compute instance creation, network configuration, script distribution and execution, and shared storage mounting. In existing deployment schemes, the deployment process relies on the continuous operation of backend management services. If these backend services are interrupted and restarted due to version upgrades, failures, or maintenance operations, all ongoing deployment tasks will be forced to stop. Currently, there is a lack of effective recovery mechanisms, resulting in interrupted tasks either being marked as failed, requiring manual intervention or re-initiation, or automatically expiring after a timeout. This not only delays the deployment process and fails to fully utilize created computing resources but also significantly increases operational complexity and the risk of resource waste. Especially in high-concurrency scenarios, a large number of tasks simultaneously attempt to recover after a service restart; disordered resource competition may further exacerbate system load and even trigger service avalanches, making it difficult to guarantee the recovery time target (SLA) for core business operations.

[0003] Therefore, how to orderly, efficiently, and fairly restore the execution of a large number of deployment tasks after a backend service interruption, while ensuring that core business tasks are completed first, has become an urgent technical problem to be solved. Summary of the Invention

[0004] One aspect of this disclosure provides an interruption recovery method based on a resilient high-performance computing cluster deployment, comprising:

[0005] Retrieve multiple deployment tasks created within a preset time period;

[0006] Based on the tenant identifiers of the multiple deployment tasks, deployment tasks belonging to the same tenant are aggregated into a batch task group, and the business priority of each deployment task in the batch task group is classified.

[0007] Based on the grouped deployment tasks, a hierarchical token bucket with a four-layer structure of resources and interfaces is constructed, and the platform resources used to execute deployment tasks are divided into mutually isolated reserved core resource pools and shared resource pools.

[0008] In response to the detection of a deployment service restart, the system iterates through each batch task group, determines the access eligibility of the reserved core resource pool based on the business priority within the batch task group, and allocates tokens to the hierarchical token bucket.

[0009] Execute the interruption recovery process for deployment tasks assigned via tokens and release their holdings on resources and tokens.

[0010] Optionally, the step of aggregating deployment tasks belonging to the same tenant into a batch task group based on the tenant identifiers of the multiple deployment tasks, and further classifying the business priority of each deployment task within the batch task group, includes:

[0011] Extract the tenant identifier, creation timestamp, and business line tag corresponding to each deployment task;

[0012] All deployment tasks are initially grouped using tenant identification as the core aggregation dimension;

[0013] Based on the creation timestamp and the preset sliding time window, the deployment tasks of the same tenant after the initial grouping are filtered a second time, and the deployment tasks within the time window are aggregated into a batch task group and a unique batch number is generated.

[0014] Based on the business line tag of each deployment task, all deployment tasks in the batch task group are classified according to their business type, and the classification results and batch numbers are synchronously written into the metadata of the deployment tasks.

[0015] Optionally, the step of constructing a hierarchical token bucket with a two-dimensional, four-layer structure of resources and interfaces based on the grouped deployment tasks, and dividing the platform resources used to execute the deployment tasks into mutually isolated reserved core resource pools and shared resource pools, further includes:

[0016] The platform resources and open interface call quotas are statistically analyzed to determine the total global token capacity and generation rate in order to initialize the global layer token bucket. The platform resources include the total platform computing resources and memory resources.

[0017] Based on the tenant's level, the task volume of the batch task group, and the business priority distribution, initialize the tenant layer token bucket, the task group layer token bucket, and the single task layer token bucket to complete the construction of a two-dimensional four-layer structure of layered token buckets.

[0018] The platform's total computing resources are divided into a reserved core resource pool and a shared resource pool according to a preset ratio. Independent token quotas are configured for the reserved core resource pool and the shared resource pool respectively. The reserved core resource pool is only open to deployment tasks with the highest business priority, while the shared resource pool is open to deployment tasks with all business priorities.

[0019] Optionally, the initialization of tenant-level token buckets, task group-level token buckets, and single-task-level token buckets based on the tenant's level, the task volume of batch task groups, and the distribution of business priorities, to complete the construction of a two-dimensional, four-layered token bucket structure, further includes:

[0020] Configure a corresponding global quota limit for the shared resource pool for each tenant, and allocate basic token capacity and generation rate to the tenant-level token bucket based on the quota limit. The total number of tokens in the tenant-level token bucket shall not exceed the global quota limit of the reserved core resource pool.

[0021] Obtain token quota from the corresponding tenant layer token bucket, initialize the task group layer token bucket for each batch task group, and reserve token quota for core transaction tasks according to a preset ratio;

[0022] Obtain token quota from the corresponding task group layer token bucket, initialize a single task layer token bucket for each deployment task in the batch task group, and allocate a corresponding number of internal resource tokens and interface call tokens based on the task's business priority level. The smallest unit of internal resource tokens is a fixed value of CPU cores, memory space, and task concurrency quota, and the smallest unit of interface call tokens is the quota for a single open interface call.

[0023] Optionally, the interruption recovery process for the deployment task assigned via token, and the release of its holdings in the resources and tokens, further includes:

[0024] Read the execution status of the deployment task and the index information of the completed deployment steps from the cache queue associated with the deployment service hostname;

[0025] In response to the deployment task being executed and the creation time being within a preset valid time threshold, a list of remaining deployment tasks is generated from the index position of the completed deployment steps;

[0026] Start an independent background process and execute the sub-steps in the remaining deployment task list in sequence. After each sub-step is completed, update the task status and step index in the cache queue. The sub-steps are idempotent sub-steps.

[0027] After all sub-steps are completed, the execution status of the deployment task will be updated to successful to complete the interruption recovery process.

[0028] Optionally, a caching middleware is used to store the task metadata of each deployment task. The caching middleware is configured with an index-associated storage architecture, which includes a first-level index and a second-level index. The first-level index includes a single task status structure, and the second-level index includes a task group metadata structure.

[0029] Optionally, the step of determining the access eligibility of the reserved core resource pool and allocating tokens to the tiered token bucket based on the business priority within the batch task group further includes:

[0030] Based on the business priority level of the deployment task, the resource pool access verification is performed. Only core transaction-type deployment tasks are eligible to be reserved in the core resource pool, while other deployment tasks are only allowed to access the shared resource pool.

[0031] Verify whether the resources used by the tenant to which the deployment task belongs exceed the global quota limit and whether the number of concurrent requests used by the batch task group to which it belongs exceeds the single batch limit. In response to the two verifications passing, allocate a token from the corresponding single task layer token bucket to the deployment task.

[0032] When a core transaction deployment task runs out of tokens, it will seize the self-idle tokens that have been applied for but not used by low-priority tasks within the same tenant to supplement the tokens required for task execution.

[0033] A second aspect of this disclosure provides an interruption recovery system based on a flexible high-performance computing cluster deployment, comprising:

[0034] The acquisition unit retrieves multiple deployment tasks created within a preset time period;

[0035] The task aggregation and grading unit aggregates deployment tasks belonging to the same tenant into a batch task group based on the tenant identifier of the multiple deployment tasks, and grades the business priority of each deployment task in the batch task group.

[0036] The resource management unit constructs a layered token bucket with a four-layer structure based on the grouped deployment tasks, and divides the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools.

[0037] The scheduling admission unit, in response to the detection of a service restart, traverses each batch task group, determines the admission qualification of the reserved core resource pool based on the business priority within the batch task group, and allocates tokens to the hierarchical token bucket.

[0038] The interrupt recovery execution unit executes the interrupt recovery process for deployment tasks assigned via tokens and releases the resources and tokens they hold.

[0039] A third aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the interrupt recovery method based on a flexible high-performance computing cluster deployment as described above.

[0040] The fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the interrupt recovery method based on the elastic high-performance computing cluster deployment described above.

[0041] The beneficial effects of this disclosure are as follows:

[0042] The interruption recovery method and system, computer equipment, and computer-readable storage medium based on EHPC cluster deployment described in this disclosure aggregate deployment tasks belonging to the same tenant into batch task groups based on the tenant identifiers of multiple deployment tasks. Furthermore, it prioritizes each deployment task within the batch task group according to its business priority. Based on the grouped deployment tasks, it constructs a hierarchical token bucket with a four-layer structure encompassing both resources and interfaces, and divides the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools. This ensures that the core tasks of all tenants can achieve their recovery time targets when a large number of deployment tasks are interrupted. It particularly safeguards rigid requirements under peak extreme concurrency scenarios, ensuring efficient and orderly recovery of deployment tasks, and has broad application prospects. Attached Figure Description

[0043] The specific embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0044] Figure 1 A flowchart illustrating an interruption recovery method based on an EHPC cluster deployment according to an embodiment of this disclosure is shown.

[0045] Figure 2 A flowchart illustrating an embodiment of the present disclosure based on an EHPC cluster deployment is shown.

[0046] Figure 3 A flowchart illustrating an interruption recovery method based on an EHPC cluster deployment according to another embodiment of this disclosure is shown.

[0047] Figure 4 A flowchart illustrating an interruption recovery method based on an EHPC cluster deployment according to another embodiment of this disclosure is shown.

[0048] Figure 5 A flowchart illustrating an interruption recovery method based on an EHPC cluster deployment according to another embodiment of this disclosure is shown.

[0049] Figure 6 A flowchart illustrating an interruption recovery method based on an EHPC cluster deployment according to another embodiment of this disclosure is shown.

[0050] Figure 7 A flowchart illustrating an interruption recovery method based on an EHPC cluster deployment according to another embodiment of this disclosure is shown.

[0051] Figure 8 This diagram illustrates an interruption recovery system based on an EHPC cluster deployment, according to another embodiment of the present disclosure.

[0052] Figure 9 A schematic diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0053] To more clearly illustrate this disclosure, the following description, in conjunction with embodiments and accompanying drawings, provides further insight. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of this disclosure.

[0054] When the backend service is interrupted, the relevant EHPC cluster deployment method cannot effectively resume task execution. For example, it causes the ongoing deployment steps to stop directly and no longer actively handle the remaining process. Even if the function of resuming the breakpoint of a single task is provided, in order to avoid blocking caused by the concentrated recovery of a large number of deployed tasks on multiple nodes in the cluster, the preset timeout mechanism is used to set the task to a timeout failure state. This approach is problematic in extremely high-concurrency cluster deployment scenarios, such as e-commerce promotions where peak deployments occur in a short period with stringent availability requirements. Top tenants (e.g., e-commerce platforms, leading merchants) may create dozens or even hundreds of cluster tasks in bulk to support order transactions, inventory management, risk control, and logistics. This indiscriminate competition for platform resources can block the core business of smaller merchants. Furthermore, in such high-concurrency scenarios, interruptions due to version upgrades, failures, or maintenance are frequent, while the workload of restarting and recovery tasks is enormous. Non-core business processes within the same tenant may compete indiscriminately for resources with core transaction processes, causing core processes to time out. The retrying of failed tasks in such high-concurrency scenarios creates a positive feedback loop, ultimately leading to a platform-wide service collapse. Therefore, there is currently no effective solution for interruption recovery in extremely high-concurrency cluster deployment scenarios.

[0055] In view of this, refer to Figure 1 As shown, one embodiment of this disclosure provides an interruption recovery method based on EHPC cluster deployment, including:

[0056] Step S1: Obtain multiple deployment tasks created within a preset time period;

[0057] Step S2: Based on the tenant identifiers of multiple deployment tasks, aggregate deployment tasks belonging to the same tenant into a batch task group, and classify the business priority of each deployment task in the batch task group.

[0058] Step S3: Based on the grouped deployment tasks, construct a layered token bucket with a two-dimensional, four-layer structure of resources and interfaces, and divide the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools;

[0059] Step S4: In response to the detection of a deployment service restart, traverse each batch task group, determine the access qualification of the reserved core resource pool according to the business priority within the batch task group, and allocate tokens to the hierarchical token bucket.

[0060] Step S5: Execute the interruption recovery process for the deployment task assigned by the token and release its holdings of resources and tokens.

[0061] In this embodiment, deployment tasks belonging to the same tenant are aggregated into batch task groups based on the tenant identifiers of multiple deployment tasks. The deployment tasks within each batch task group are then classified by business priority. Furthermore, a hierarchical token bucket with a four-layer structure of resources and interfaces is constructed based on the grouped deployment tasks. The platform resources used to execute deployment tasks are divided into mutually isolated reserved core resource pools and shared resource pools. This ensures that the core tasks of all tenants can achieve the recovery time target when a large number of deployment tasks are interrupted. In particular, it guarantees rigid requirements under peak extreme concurrency scenarios and ensures the efficient and orderly recovery of deployment tasks.

[0062] The following detailed description, using specific embodiments, illustrates the detailed process of the interruption recovery method based on EHPC cluster deployment according to this disclosure. It should be noted that EHPC refers to a high-performance computing technology based on dynamically scalable cloud resources. Using Elastic Compute Service (ECS) cloud servers as the basic computing resource, it rapidly builds large-scale parallel computing clusters to solve problems such as complex scientific computing, engineering simulation, and commercial big data processing. The interruption recovery method based on EHPC cluster deployment runs in a cloud server cluster environment, relying on caching middleware to achieve persistent storage of task states, and combining cloud-native container orchestration technology to achieve high-availability deployment of backend services.

[0063] The application scenarios of this embodiment include, but are not limited to, batch elastic high-performance computing cluster deployment during e-commerce promotional periods, batch expansion of computing resources during financial settlement cycles, and large-scale simulation task cluster deployment in research institutions—all extremely high-parallel scenarios requiring simultaneous processing of multiple deployment tasks and high service availability. For ease of description, the following embodiments will all use batch elastic high-performance computing cluster deployment during e-commerce promotional periods as the scenario.

[0064] In step S1, multiple deployment tasks created within a preset time period are obtained.

[0065] In this embodiment, the preset time period is set as an effective recovery time threshold. For example, if the effective recovery time threshold for interruption recovery is 1 hour, it is set by default to 1 hour prior to the service restart time. It should be understood that this preset time period can be adaptively adjusted according to the needs of different business scenarios. For example, for different business scenarios, the adaptive adjustment rule for this preset time period can be as follows: In a high-concurrency scenario during e-commerce promotions, when the query per second (QPS) rate of platform deployment task creation exceeds the preset threshold, the preset time period can be extended to 2 hours to ensure that all tasks being executed during the promotion can be included in the recovery scope; in a low-concurrency daily operation scenario, the preset time period can be shortened to 30 minutes to further filter invalid historical tasks. For another example, for long-cycle deployment scenarios such as financial settlement and scientific research simulation, the preset time period can be extended to 4 hours to adapt to the recovery needs of long-cycle tasks.

[0066] However, it should be noted that the maximum length of the preset time period is no more than 24 hours. When the caching middleware is Redis, combined with Redis's TTL (Time to Live) expiration mechanism, all deployed task storage data will be set to expire in 24 hours. Task data exceeding 24 hours will be automatically cleaned up by Redis to avoid the continuous expansion of Redis storage resources and ensure the stable operation of the caching middleware.

[0067] Specifically, a deployment task refers to a request submitted by a user through the cloud platform management interface or open API to create an elastic high-performance computing cluster. Each deployment task corresponds to an independent elastic high-performance computing cluster instance.

[0068] For example, refer to Figure 2 As shown, the deployment task in this embodiment may include multiple sub-steps, including: creating an instance and successfully detecting the cloud server status, configuring the network, preparing the deployment package and scripts, pre-deployment preparation (node ​​interconnection, etc.), node deployment, mounting shared storage, cluster initialization, and successful cluster deployment.

[0069] Specifically, the service backend creates the ECS instance based on the ECS creation request initiated by the user and queries until the ECS creation is successful. Next, it performs network configuration, including security group policies, network interface cards (NICs), and secondary NICs, to ensure ECS network service availability. Then, it prepares the deployment package and deployment scripts, sending the relevant files to the master ECS operation node created in the ECS creation step. The master node will then complete the relevant operations within the cluster. Afterward, the service backend sends pre-deployment preparation instructions to the master node, which performs key configuration and node communication operations. The service backend initiates cluster deployment instructions through the master ECS operation node to begin deploying each node. The service backend queries the deployment progress until successful. The service backend initiates shared storage mount instructions through the master ECS operation node to configure storage. The service backend initiates cluster initialization instructions through the master ECS operation node to perform relevant initialization operations on the EHPC. Finally, the EHPC cluster deployment is completed.

[0070] It should be noted that the above steps are merely illustrative simplified processes in the overall deployment process of an EHPC cluster. They are examples of non-critical processes in the complete process that have been reduced and merged. They are intended to describe and facilitate understanding of the implementation process of interrupted recovery of deployment tasks in the embodiments of this disclosure, and are not intended to limit the specific steps of actual deployment tasks to this.

[0071] It should be understood that the interruption described in this disclosure refers to a deployment interruption caused by version upgrades, failures, or maintenance operations before all the steps of the above task deployment are completed. The interruption may occur at any step of the above multiple sub-step processes. The interruption recovery in this disclosure embodiment can resume the interrupted sub-steps and complete the subsequent deployment sub-steps after restarting.

[0072] Specifically, the interrupt recovery method of this disclosure utilizes a buffer middleware to cache the task metadata related to the execution of each of the above-mentioned multiple sub-steps. This ensures that upon interruption restart, the task deployment status can be queried, the task group can be globally managed, and the interruption resumption position of the task can be obtained based on the task metadata stored in the buffer middleware. For example, the buffer middleware can be Redis. However, it should be understood that the buffer middleware described in this disclosure is not limited to this; other databases with data caching functions, such as in-memory databases, are also within the scope of protection of this disclosure.

[0073] For example, the acquisition of deployment tasks described in this embodiment may include: Redis connection initialization, global task index reading, initial filtering by time dimension, secondary filtering by the Pod, and loading of the task list. After the EHPC backend service Pod starts, it establishes a long connection with Redis to verify the connection status, data read / write permissions, and cluster shard consistency, ensuring that the service can read and write task metadata normally to complete the Redis connection initialization. In the global task index reading step, it reads the global deployment task index table from Redis to obtain the basic metadata of all stored deployment tasks, including the unique task ID, tenant identifier, creation timestamp, business line tag, index of completed deployment steps, current execution status, and hostname of the service to which it belongs (Pod hostname), etc., without having to read the complete details of each task, which greatly reduces the IO overhead of data reading. It performs an initial filtering by time dimension to remove historical tasks outside the preset time period. In the second filtering by Pod, based on the fixed hostname of the current service Pod, it performs a second filtering on the task list after the initial filtering, retaining only the deployment tasks managed by the current Pod and removing the tasks belonging to other Pods. By loading the complete task list into the service memory and creating a corresponding metadata index for each task, it is used for subsequent batch aggregation, classification, and recovery processing.

[0074] By obtaining deployment tasks through the above step-by-step filtering method, most invalid tasks can be eliminated during the data reading stage, and only the valid tasks that need to be processed can be loaded into memory. This reduces the IO pressure of the cache and the memory usage of the service, ensuring the execution efficiency of the recovery process in high-concurrency scenarios.

[0075] In step S2, based on the tenant identifiers of multiple deployment tasks, deployment tasks belonging to the same tenant are aggregated into a batch task group, and the business priority of each deployment task within the batch task group is classified.

[0076] Reference Figure 3 As shown, in step S21, the tenant identifier, creation timestamp, and business line tag corresponding to each deployment task are extracted.

[0077] In an optional embodiment, Redis, acting as a caching middleware, is configured with an index-associated storage architecture. Specifically, this index-associated storage architecture includes a first-level index and a second-level index. The first-level index includes a single-task status structure, specifically a task list for each single task, which includes the execution status of all sub-steps. The single-task status storage structure also includes three extended fields: batch number, business priority, and tenant level. The second-level index includes a hash table `batch_task_group`, where the key is "tenant ID + batch number" and the value is a task group metadata structure containing core metadata fields such as batch number, tenant ID, tenant level, time window, task ID list, number of tasks at each level, and concurrency limit.

[0078] In other words, in the embodiments of this disclosure, a hash table representing the second-level index is constructed, and corresponding fields related to the task group are extracted and written. That is, by setting up an index-associated storage architecture, it is possible to achieve single-task status query for interrupted recovery and global management of batch tasks.

[0079] Continuing with this step, first extract the core metadata fields used for aggregation and tiering from Redis, including: Tenant ID, Creation Timestamp, and Business Line Tag. The Tenant ID is a unique identifier assigned by the platform to the user to whom the task belongs. The Creation Timestamp is the task creation time, accurate to milliseconds. The Business Line Tag is a tag field representing the business type corresponding to the deployment task; it can be automatically extracted by the platform from the task name, description, and cluster configuration template through keyword matching, or it can be manually filled in by the user when creating the task.

[0080] It should be noted that a combination of keyword-based fuzzy matching and exact matching can be used to automatically extract business line tags. The platform's built-in standardized business keyword library can cover most business types in e-commerce, finance, scientific research and other scenarios. Automatic tag extraction can be completed without manual configuration. At the same time, it supports user-defined keyword libraries and tag mapping rules.

[0081] In step S22, all deployment tasks are initially grouped using tenant identifier as the core aggregation dimension.

[0082] After extracting the core metadata fields, all valid deployment tasks are initially grouped using the tenant identifier as the core aggregation dimension. Deployment tasks with the same tenant identifier are grouped into the same initial group, while tasks with different tenant identifiers are assigned to independent initial groups.

[0083] Initial grouping based on tenant identification is the foundation for achieving resource isolation and fair management among cloud platform tenants. It ensures that subsequent tenant-level quota management and batch task scheduling can accurately cover each tenant, avoiding resource crowding and mutual interference between different tenants.

[0084] However, it is important to note that the initial grouping only achieves coarse isolation at the tenant level and does not aggregate tasks within the same tenant at the time level. Tasks created by the same tenant at different times often correspond to different business batches. For example, tasks created in the morning correspond to daily operation scenarios, while tasks created before a major promotion in the afternoon correspond to promotion support scenarios. If all tasks are aggregated into the same batch task group, the granularity of concurrency control will be too coarse and will not be able to adapt to the differentiated needs of different business batches.

[0085] Therefore, in step S23, based on the creation timestamp and the preset sliding time window, the deployment tasks of the same tenant after the initial grouping are filtered a second time, and the deployment tasks within the time window are aggregated into a batch task group and a unique batch number is generated.

[0086] It should be noted that the preset sliding time window setting rules apply to concurrent scenarios such as e-commerce business scenarios. For example, when the platform's global deployment task creation QPS exceeds a preset threshold (e.g., 100 by default), it is determined to be a high-concurrency scenario, and the preset sliding time window is fixed at 5 minutes; when the platform's global task creation QPS is lower than the preset threshold, it is determined to be a daily low-concurrency scenario, and the preset sliding time window is fixed at 30 minutes. This application does not impose any restrictions on this.

[0087] By setting a sliding time window, the granularity of aggregation can be reduced in high-concurrency scenarios to achieve fine-grained control of batch tasks and avoid the failure of concurrency control caused by excessive batch task volume; in low-concurrency scenarios, the aggregation granularity can be increased to reduce the number of task groups, reduce the scheduling overhead of the platform, and balance control accuracy and operating efficiency.

[0088] In this specific example, the timestamp of the earliest created task within the initial group of the same tenant is used as the starting time of the sliding time window, which then covers the preset time window range. All tasks within this window are included in the same batch task group. If the tenant still has newly created tasks after the first time window ends, a new sliding time window and a new batch task group are generated starting from the timestamp of the first task after the first window ends. This process is repeated to complete the batch aggregation of all tasks of the tenant.

[0089] After aggregation, a globally unique batch number is generated for each batch task group. The batch number can be in the format of "tenant identifier + window start timestamp + 6-digit random sequence number". This setting ensures global uniqueness and allows for quick location of the corresponding tenant and task creation time through the batch number, facilitating subsequent operation and maintenance management, problem investigation, and audit traceability.

[0090] Next, an index-associated storage architecture is used, leveraging the Redis caching middleware to associate batch task groups with individual tasks. As mentioned above, the index-associated storage architecture includes a primary index and a secondary index. By adding a hash table `batch_task_group` to the secondary index, global management of batch task groups is achieved using the key of this hash table. Furthermore, by adding two extended fields, batch number and business priority, to the single-task status structure of each deployed task, the task group metadata and single-task metadata are synchronously written to Redis, ensuring data consistency in the index and providing efficient support for single-task status queries and global management of batch task groups.

[0091] In step S24, based on the business line tag of each deployment task, all deployment tasks in the batch task group are classified according to their business type, and the classification results and batch number are synchronously written into the metadata of the deployment task.

[0092] After the batch task groups are aggregated, for all deployment tasks within each batch task group, based on the task's business line tags and using preset keyword matching rules, the business priority is classified, the classification results are written into the deployment task's metadata, and simultaneously updated to the hash table of the corresponding batch task group's secondary index.

[0093] For example, business priorities can be divided into three fixed levels, from highest to lowest: P1 level core transaction categories, P2 level business support categories, and P3 level non-core categories. For instance, core business types in P1 level core transaction categories may include: order transactions, inventory management, payment settlement, traffic risk control, price management, and account systems. Matching keywords may include: transaction, order, payment, inventory, risk control, settlement, price, and account. This category has the highest core protection level and is given priority, granting access to the core resource pool. P2 level business support categories may include: logistics scheduling, user profiling, intelligent customer service, after-sales service, membership management, and marketing activities. Matching keywords may include: logistics, user, profiling, customer service, after-sales, membership, and marketing. This category has the second highest core protection level and is given priority scheduling, granting access to the shared resource pool. P3 level non-core categories may include: log analysis, offline computing, data backup, testing and verification, and statistical reports. Matching keywords may include: log, analysis, offline, backup, testing, and statistics. This category has the lowest core protection level and is only scheduled when available resources, granting access to the shared resource pool.

[0094] It should be noted that the keyword matching for priority grading can adopt a rule of prioritizing exact matching and supplementing with fuzzy matching: if a business line tag contains an exact keyword of a certain level, it will be directly assigned to the corresponding priority; if no exact keyword is matched, the business type will be identified through fuzzy matching to complete the grading; at the same time, the platform allows users to manually specify the business priority of a task when creating a task, and also allows administrators to dynamically adjust the priority of a task during task execution to supplement the coverage of the automatic matching rules, ensuring that the grading results are completely consistent with the actual business needs. This application does not impose any restrictions on this.

[0095] By prioritizing business operations, we can ensure that core business operations are prioritized and that internal tasks within the same tenant are kept in an orderly manner. Prioritization ensures that the most critical transaction-related tasks within the same tenant receive priority access to resources and token quotas, and that interruption recovery processes are executed first. This prevents non-core tasks from monopolizing resources for core business operations, thus guaranteeing the SLA (Service Level Agreement) of core business operations during high-concurrency scenarios such as major promotional events.

[0096] However, it should be noted that the business priority classification only takes effect within the same batch of tasks of the same tenant. The task priority between different tenants is controlled by the tenant level and resource pool quota to avoid situations where P3-level non-core tasks of top tenants have higher priority than P1-level core tasks of small and medium-sized tenants, thereby ensuring the fairness of resource allocation across the entire platform.

[0097] After the hierarchical classification is completed, the business priority of each task is written into its metadata structure and synchronously updated to the hash table of the secondary index of the batch task group. The number of deployment tasks of each priority within the group is also counted to provide data basis for subsequent token bucket initialization and quota reservation.

[0098] In step S3, a hierarchical token bucket with a two-dimensional four-layer structure of resources and interfaces is constructed based on the grouped deployment tasks, and the platform resources used to execute the deployment tasks are divided into a mutually isolated reserved core resource pool and a shared resource pool.

[0099] It's worth noting that since API calls (such as Open APIs) are essential to every step of EHPC cluster deployment, exhausting API quotas will directly lead to deployment task failures. This is a major reason for frequent task failures in high-concurrency scenarios. To prevent hundreds of tasks from simultaneously initiating requests after a service restart, instantly exceeding the global total threshold of the Open API and causing core business API requests to be rejected, resource management not only targets internal computing resources such as CPU and memory but also considers both resource and API call quotas to build a two-dimensional token bucket. This ensures that during deployment task interruption recovery, resources and API calls are included in the same management system, avoiding recovery failures caused by resource mismatch. Furthermore, based on task grouping and group priority, a four-layer token bucket system is constructed for resource pool management to ensure the recovery of core business processes during high-concurrency scenarios such as e-commerce promotions. This core concept will be explained in detail below.

[0100] In an alternative embodiment, refer to Figure 4 As shown, step S3 includes the following specific steps.

[0101] In step S31, platform resources and open interface call quotas are statistically analyzed, and the total global token capacity and generation rate are determined to initialize the global layer token bucket. Platform resources include total platform computing resources and memory resources.

[0102] Specifically, the first step is to measure platform resources and open interface call quotas. Platform resources include total platform computing resources (number of CPU cores) and total memory resources.

[0103] Specifically, the global token bucket is the top layer of the entire layered token bucket architecture. It is the sole source of all tokens, controls the total resource quota and API call quota of the platform, and is the first line of defense to ensure the overall stability of the platform.

[0104] For example, the global token bucket contains two completely independent sub-token buckets: an internal resource global token bucket and an API call global token bucket, corresponding to the dual-dimensional control of resources and interfaces. Specifically, the initialization of the global token bucket includes: the initialization of the internal resource global token bucket and the initialization of the API call global token bucket.

[0105] For example, during the initialization of the internal resource global token bucket, the smallest unit of the internal resource token is a fixed value of CPU cores, memory space, and task concurrency quota. Specifically, the smallest unit of the internal resource token is: 1 internal resource token = 0.5 CPU cores (a single CPU core generally includes two threads; 0.5 cores correspond to one independent thread) + 1GB of memory + 1 task process concurrency quota, that is, one smallest token unit corresponds to the smallest resource unit required for the execution of a single deployment task. It should be noted that this minimum unit setting can be customized and adjusted according to the platform's hardware configuration and business characteristics to adapt to different deployment environments.

[0106] The total capacity of the internal resource global token bucket can be calculated based on the total number of available CPU cores and total memory resources of the platform. The smaller value between the total number of tokens that the CPU can support and the total number of tokens that memory can support is taken to ensure that the total number of tokens matches the actual physical resources of the platform and avoid resource over-selling. For example, if the platform has 160 available CPU cores and 320GB of available memory, then the total number of tokens that the CPU can support is 320, and the total number of tokens that memory can support is also 320. Therefore, the total capacity of the internal resource global token bucket is set to 320.

[0107] The token generation rate of the internal resource global token bucket can be set based on the platform resource release speed and the concurrency requirements of task execution. The default setting is 100 tokens per second, and the specific value can be dynamically adjusted according to the platform resource level. For example, when the platform CPU utilization exceeds 80%, the generation rate will be automatically reduced; while when the platform CPU utilization is below 30%, the generation rate will be automatically increased to achieve adaptive traffic adjustment.

[0108] In addition, from the internal resource global token bucket, an independent core token pool will be divided according to the proportion of the reserved core resource pool. The tokens in the core token pool and the shared resource pool are completely isolated, and their capacity and generation rate are calculated and determined independently. They are only open to P1 level core tasks.

[0109] For example, during the initialization of the global token bucket for API calls, the smallest unit of the API call token is the single open API call quota. Specifically, the smallest unit of the API call token is: 1 API call token = 1 cloud platform OpenAPI call quota, that is, one token corresponds to the permission for one OpenAPI call. During task execution, each time an OpenAPI call is initiated, one API call token is consumed.

[0110] The total capacity and generation rate of the global token bucket for API calls can be set based on the global flow control threshold of the cloud platform's OpenAPI. For example, the global flow control threshold can be set to 80% of the standard flow control threshold, reserving a 20% buffer to absolutely avoid triggering the platform's global flow control threshold. For instance, if the global flow control threshold for the cloud platform's OpenAPI is 1000 calls per second, then the total capacity of the global token bucket for API calls can be set to 2000 tokens, and the token generation rate can be set to 800 tokens per second.

[0111] Consistent with the internal resource token bucket, the global token bucket for API calls also has a separate core token pool, which is completely isolated from the tokens in the shared resource pool, ensuring that the API call requirements of core tasks are not squeezed out by non-core tasks.

[0112] In step S32, based on the tenant's level, the task volume of the batch task group, and the business priority distribution, the tenant layer token bucket, the task group layer token bucket, and the single task layer token bucket are initialized to complete the construction of the layered token bucket with a two-dimensional four-layer structure.

[0113] It needs to be further clarified that the "dual dimensions" mentioned in this disclosure refer to the internal resource dimension and the open interface call dimension. Each dimension corresponds to an independent token system, and the token bucket hierarchy of the two dimensions is completely consistent, with synchronized initialization, synchronized quota allocation, and synchronized token deduction and return. The "four-layer structure" mentioned in this disclosure, in addition to the top-level global token bucket mentioned above, includes, in sequence, the tenant layer token bucket, the task group layer token bucket, and the single task layer token bucket. That is, based on the task groups and priorities of the deployed tasks, a four-layer tree structure is formed from the top to the bottom. The upper-level token bucket allocates quotas to the lower-level token buckets according to the task type and priority. The total token capacity of the lower-level token bucket must not exceed the quota limit allocated by the upper layer, thereby achieving orderly business assurance for interruption recovery.

[0114] Specifically, in this step, based on the level of each tenant, a corresponding global quota limit for the shared resource pool is configured for each tenant. Based on the quota limit, a basic token capacity and generation rate are allocated to the tenant-level token bucket. The total number of tokens in the tenant-level token bucket does not exceed its global quota limit, thus completing the initialization of the tenant-level token bucket.

[0115] For example, tenant tiers include: core tenants, general enterprise tenants, and individual / test tenants. Core tenants are, for example, core clients of leading e-commerce platforms and large financial institutions, whose shared resource pool quota can be set to no more than 30% of the total shared resource pool capacity to prevent a single tenant from consuming the vast majority of shared resources. General enterprise tenants are, for example, mainstream clients of the platform such as small and medium-sized e-commerce merchants and SMEs, whose shared resource pool quota can be set to no more than 10% of the total shared resource pool capacity. Individual / test tenants are, for example, individual developers and test users, whose shared resource pool quota can be set to no more than 5% of the total shared resource pool capacity.

[0116] By setting quota limits for each tenant tier, it is possible to avoid single-tenant monopolization of resources and ensure that even core tenants only occupy a maximum of 30% of the quota in the shared resource pool. The remaining 70% of the shared resources can be allocated to other small and medium-sized tenants, thus ensuring fairness in interruption recovery.

[0117] It should be noted that in the embodiments disclosed herein, the core tokens reserved in the core resource pool are not included in the tenant-level quota control. All tenants’ P1-level core transaction tasks can directly apply for tokens from the core token pool without being subject to the tenant quota limit.

[0118] This configuration ensures that even if the shared resource pool quota for small and medium-sized tenants is exhausted, their core transaction tasks can still obtain resources from the core resource pool to complete deployment recovery. This prevents them from being blocked by non-core tasks from top tenants, thus guaranteeing the SLA for the core business of small and medium-sized tenants. Importantly, to prevent top tenants' P1-level tasks from excessively consuming the core token pool, the core tokens used by a single tenant's P1-level tasks must not exceed 40% of the total capacity of the core token pool, ensuring that small and medium-sized tenants still have sufficient tokens available for their core tasks.

[0119] In the embodiments disclosed herein, each tenant's tenant-layer token bucket also includes two independent sub-buckets: an internal resource token bucket and an API call token bucket. The quota ratio of the two sub-buckets is consistent with that of the global layer, ensuring that the tenant's internal resources and API quotas are fully matched, thus avoiding resource mismatch problems such as sufficient CPU but exhausted API quotas or sufficient API quotas but full CPU usage.

[0120] Specifically, in this step, based on the task volume of the batch task group, the token quota is obtained from the corresponding tenant layer token bucket, the task group layer token bucket is initialized for each batch task group, and the token quota is reserved for core transaction tasks according to a preset ratio to complete the initialization of the task group layer token bucket.

[0121] It should be noted that the maximum concurrent execution limit for a single batch of tasks can also be set based on the tenant level to match the tenant quota rules. For example, the maximum concurrent execution limit for a single batch of tasks is 20 for core tenants; 10 for ordinary enterprise tenants; and 5 for individual / test tenants. The total token capacity of the token bucket at the task group level corresponds to the number of tokens required for the single batch concurrent execution limit, ensuring the implementation of concurrency control rules.

[0122] In this embodiment, the core reservation rule for the task group-level token bucket is as follows: When the task group-level token bucket is initialized, 70% of the token quota is reserved for P1-level core transaction tasks within the group, and the remaining 30% is reserved. Only after the token requirements of all P1-level tasks within the group are met will the remaining quota be allocated to P2 and P3-level tasks. This reservation rule prevents low-priority tasks within the same group from preempting token quotas after service restart, thus avoiding the inability of core tasks to obtain resources in a timely manner and ensuring the priority recovery of core tasks.

[0123] Meanwhile, the token capacity of the task group layer token bucket is not fixed. If the tenant releases token quota after other batches of tasks are completed, and there is available spare capacity in the tenant layer token bucket, the capacity of the task group layer token bucket can be dynamically increased to improve the concurrent execution limit of the task group and realize the dynamic adjustment and efficient utilization of quota.

[0124] The task group layer token bucket is located one level below the tenant layer token bucket. Each batch task group corresponds to an independent task group layer token bucket, which is used to control the token allocation and concurrent execution limit of tasks within the group, and to achieve fine-grained control of different batches of tasks within the same tenant.

[0125] Specifically, in this step, based on the business priority distribution of tasks, and further based on the business priority level of the tasks, cluster size, and estimated execution time, token quotas are obtained from the corresponding task group layer token bucket. A single task layer token bucket is initialized for each deployment task in the batch task group. Based on the business priority level of the tasks, a corresponding number of internal resource tokens and interface call tokens are allocated to complete the initialization of the single task layer token bucket.

[0126] Specifically, the token allocation rules of the single-task layer token bucket are deeply tied to business priorities. Specifically, P1-level core transaction tasks are allocated a sufficient amount of tokens. Based on the cluster size and estimated execution time, the number of tokens required for full-speed execution is calculated and fully allocated to ensure uninterrupted full-speed execution and meet the emergency recovery needs of core businesses. P2-level business support tasks are allocated a basic amount of tokens, 70% of the amount required for full-speed execution, ensuring normal task execution while avoiding excessive resource consumption. P3-level non-core tasks are allocated the minimum amount of tokens, 30% of the amount required for full-speed execution, and are only executed when platform resources are sufficient to avoid crowding out core business resources.

[0127] It should be noted that the single-task layer token bucket also includes two sub-buckets: internal resource tokens and API call tokens. The number of tokens in the two sub-buckets is matched and allocated according to a fixed ratio based on the number of steps in the task and the frequency of API calls. This ensures that the tokens in the two dimensions are always synchronized during task execution, and there will be no problem of task interruption due to the exhaustion of one type of token.

[0128] It's worth noting that a task must simultaneously obtain sufficient internal resource tokens and API call tokens from the single-task layer token bucket before entering the interruption recovery process. Otherwise, it will enter a waiting queue, waiting for tokens to be released and re-applying. This setting enables fine-grained control over resource consumption and API call rates for each task, avoiding traffic spikes and resource congestion issues after service restarts, and ensuring platform stability.

[0129] This completes the construction of a two-dimensional, four-layered token bucket structure, forming a comprehensive management and control system from global to single-task levels, thus providing core support for subsequent task access verification, token allocation, and hierarchical scheduling.

[0130] In step S33, the total computing resources of the platform are divided into a reserved core resource pool and a shared resource pool according to a preset ratio, and independent token quotas are configured for the reserved core resource pool and the shared resource pool respectively.

[0131] Specifically, based on a preset ratio, the platform's total computing resources and memory resources are divided into a reserved core resource pool and a shared resource pool that are logically isolated from each other. At the same time, independent internal resource token quotas and interface call token quotas are configured for the two resource pools respectively. The quotas of the two resource pools are independent of each other and cannot be occupied across pools.

[0132] It should be noted that the embodiments disclosed herein use logical isolation to divide the resource pools, thus eliminating the need for physical partitioning of underlying hardware resources. This can be achieved solely through software-level token quota management, offering extremely high flexibility. The ratio of the two resource pools can be dynamically adjusted according to changes in business scenarios without requiring service restarts or modifications to underlying configurations. In particular, logical isolation achieves complete resource isolation between core and non-core tasks while avoiding resource fragmentation caused by physical partitioning, thereby significantly improving the overall utilization rate of platform resources.

[0133] Optionally, the proportion of reserved core resource pool to total platform resources can be dynamically adjusted according to the periodic rhythm of high-concurrency scenarios. For example, in daily operation scenarios, the reserved core resource pool accounts for 30% of the total platform resources, and the shared resource pool accounts for 70% of the total platform resources; 24 hours before the peak of e-commerce promotions, the proportion of reserved core resource pool is automatically adjusted to 50% to reserve sufficient resources for the upcoming core business deployment needs; during the peak of e-commerce promotions, the proportion of reserved core resource pool is automatically adjusted to 60% to maximize the resource needs of core transaction businesses; 24 hours after the peak of e-commerce promotions, the proportion of reserved core resource pool is automatically reduced to 45%, gradually releasing resources to the shared resource pool; the platform supports administrators to manually adjust the resource pool allocation ratio to adapt to the needs of special scenarios such as financial settlement and batch processing of government affairs.

[0134] It should be noted that, in order to avoid resource waste caused by excessive reservation, the proportion of reserved core resource pool shall not exceed 70% of the platform's total resources, to ensure that the shared resource pool has basic resource capacity and to guarantee the normal execution of non-core business.

[0135] Importantly, the reserved core resource pool is only open to deployment tasks with the highest business priority, while the shared resource pool is open to deployment tasks with all business priorities.

[0136] Specifically, the reserved core resource pool is only open to P1-level core transaction deployment tasks. Regardless of the tenant's level, as long as it is a P1-level core transaction task, it is eligible to enter the core resource pool and apply for core token quotas. The reserved core resource pool is used to provide an independent resource channel for the core transaction business of all tenants across the platform. No matter how many tasks top merchants have, the core transaction tasks of small and medium-sized merchants will not be blocked, ensuring that the core business SLAs of all tenants across the platform can be fulfilled.

[0137] Specifically, the shared resource pool is open to all priority deployment tasks across all business operations, including P1, P2, and P3 level tasks. However, it enforces strict tenant-level quota control to prevent single-tenant resource monopoly. The shared resource pool is used to support non-core tasks for leading merchants and business support tasks for small and medium-sized merchants, ensuring the fairness of platform resource allocation and improving overall resource utilization.

[0138] By coordinating task groups for deployment tasks and setting up mutually isolated reserved core resource pools and shared resource pools at different levels, and by setting different access qualifications based on business types, it is possible to avoid non-core tasks crowding out core business interruption recovery resources in high-parallel scenarios, and also to avoid waste caused by excessive resource reservation, thus balancing SLA assurance and resource utilization.

[0139] In step S4, in response to the detection of a deployment service restart, each batch task group is traversed, and the access qualifications for the reserved core resource pool are determined according to the business priority within the batch task group, and tokens for the hierarchical token bucket are allocated.

[0140] Specifically, refer to Figure 5 As shown, step S4 further includes:

[0141] Step S41: Based on the business priority level of the deployment task, perform resource pool access verification. Only core transaction-type deployment tasks are eligible to be reserved in the core resource pool, while other deployment tasks are only allowed to access the shared resource pool.

[0142] Step S42: Verify whether the resources used by the tenant to which the deployment task belongs exceed the global quota limit and whether the number of concurrent requests used by the batch task group to which it belongs exceeds the single batch limit. In response to the two verifications passing, allocate tokens from the corresponding single task layer token bucket to the deployment task.

[0143] Step S43: When a core transaction deployment task has insufficient tokens, it preempts the self-idle tokens that have been applied for but not used by low-priority tasks within the same tenant to supplement the tokens required for task execution.

[0144] For example, the service can first complete the association mapping between deployment tasks and task groups, and then traverse all associated batch task groups. After restarting, the deployment service hostname remains fixed. After the service starts, it first completes the initialization of the hierarchical token bucket and the division of the resource pool according to the above steps, and retrieves the cache queue associated with itself from Redis based on its fixed hostname, such as the Redis Key queue, thereby obtaining the list of all deployment tasks stored in the queue. At the same time, it reads the hash table of the secondary index of the batch task groups in Redis, thereby completing the association mapping between each deployment task and the batch task group, ensuring that each task can be accurately assigned to the corresponding tenant and task group.

[0145] Optionally, batch task groups can be globally sorted based on preset rules to ensure that more urgent tasks are prioritized for admission verification and token allocation. For example, the first sorting dimension is tenant level: task groups of core tenants take precedence over task groups of ordinary enterprise tenants, and ordinary enterprise tenants take precedence over individual / test tenants; the second sorting dimension is the proportion of P1-level tasks within the task group: for task groups of tenants of the same level, the higher the proportion of P1-level tasks, the higher the ranking; the third sorting dimension is the urgency of the task SLA: for task groups with the same proportion of P1-level tasks, the lower the average time to SLA deadline for tasks within the group, the higher the ranking.

[0146] This sorting process ensures that the most critical and urgent tasks receive resources and tokens first, thereby ensuring timely recovery of core business operations during high-concurrency scenarios.

[0147] For example, based on the business priority level of deployment tasks, resource pool access verification is performed according to the rule that only core transaction-type deployment tasks are eligible to reserve the core resource pool, while other deployment tasks are only allowed to access the shared resource pool. This achieves resource isolation between core tasks and non-core tasks from the bottom layer.

[0148] For example, after the resource pool access verification in step S41, a secondary verification in step S42 is performed. This involves real-time statistics to determine if the number of shared resources currently used by a tenant exceeds the tenant's global quota limit. If the verification fails, the allocation of shared resource tokens to new deployment tasks for that tenant is prohibited; otherwise, the verification passes. Furthermore, the number of tasks currently being executed in the batch task group is counted. If the used concurrency reaches the single-batch concurrency execution limit, the verification fails, and token allocation to new tasks within the task group is prohibited; otherwise, the verification passes. It should be noted that tokens from the core relay token pool are not included in the tenant quota statistics, ensuring that core tasks are not limited by tenant quotas.

[0149] By simultaneously verifying tenant quota checks and task group concurrency limits, we can prevent a single tenant from occupying shared resources beyond their quota, ensuring the fairness of platform resource allocation. At the same time, we can prevent a single batch of tasks from occupying all of the tenant's quota, while limiting the concurrency of a single batch of tasks, preventing instantaneous traffic spikes from breaking through the interface control threshold, and ensuring the normal recovery of all core businesses.

[0150] Optionally, when core transaction deployment tasks lack sufficient tokens, a token preemption mechanism can be triggered for P1-level core transaction services to preempt unused self-idle tokens applied for by lower-priority tasks within the same tenant, supplementing the token quota required for task execution. Generally, a single preemption rule can be configured, that is, when P1-level services lack tokens, idle tokens from P2 and P3-level services are preempted proportionally. However, this can easily lead to P1-level tasks preempting any temporarily available idle tokens from P2 and P3-level tasks. In this case, P2 and P3-level tasks may starve due to insufficient self-idle tokens, making it impossible to ensure the normal operation of P1 tasks while preventing the starvation of P2 and P3-level tasks.

[0151] Based on this, this application provides a new token preemption mechanism. Specifically, the token preemption mechanism includes: only P1-level core transaction tasks can trigger the token preemption mechanism when token application fails; P2 and P3-level tasks have no preemption permission; it can only preempt idle tokens that have been applied for but not used by low-priority tasks (e.g., P2 and P3-level tasks) within the same tenant, and cannot preempt tokens across tenants or tokens already used by tasks that are currently being executed; the number of tokens preempted in a single instance cannot exceed 50% of the total quota required by the core task, for example, if 30% of the total quota is preempted in a single instance, the remaining quota must wait for the tokens to be released naturally to avoid excessive preemption; after the preemption is completed, it must be ensured that the remaining number of tokens for the preempted low-priority task is not less than the minimum quota required for its start, ensuring that the low-priority task will not be completely unable to execute due to preemption, and avoiding the situation of task "starvation"; the preempted tokens can only be used for the execution of the current task. After the task is completed, unused preempted tokens must be returned to the token bucket of the preempted low-priority task first, and then returned to the upper-level token bucket. With this setting, when resources are scarce during peak sales events, core tasks can quickly obtain the necessary resources by seizing idle tokens from low-priority tasks, thus meeting the emergency recovery SLA requirements in extreme scenarios without having to wait for low-priority tasks to finish executing and release resources.

[0152] Furthermore, in this embodiment of the application, in order to ensure the normal progress of task P1 while avoiding the starvation problem of tasks P2 and P3, it was found that this problem can be solved by using a conventional counter approach. Specifically, the system maintains a preemption counter corresponding to the task P1 and contribution counters corresponding to tasks P2 and P3. After each preemption, the preemption counter X1 (corresponding to the task P1) is incremented by one, and at the same time, the contribution counter X2 corresponding to the preempted task P2 is incremented by one, and / or the contribution counter X3 corresponding to the task P3 is incremented by one. Then, when task P1 preempts task P2 and / or task P3, it is necessary to compare the preemption counter X1 of task P1 with the contribution counters X2 and X3 corresponding to tasks P2 and P3. If it is determined whether to preempt, specifically, X1 needs to be greater than X2 or X3. For example, if X1 is greater than X2, then a P1-level task can preempt the token of a P2-level task. If X1 is less than X3, then a P1-level task cannot preempt the token of a P3-level task. In this way, based on the configuration of the contribution counter and the preemption counter, the higher the contribution counter, the more times it will be preempted, and the more preemption counters, the more times it will be preempted. By only allowing preemption of low-contribution-level tasks, the continuous preemption of low-level tasks is avoided. This can reduce the problem of tasks starving due to the continuous preemption of high-contribution-count tasks, while ensuring that P1-level tasks can obtain tokens to the maximum extent. Thus, the normal operation of P1 tasks is ensured while avoiding the starvation of P2-level and P3-level tasks.

[0153] In step S5, the interruption recovery process of the deployment task assigned by the token is executed, and its occupied resources and tokens are released.

[0154] Combination Figure 6 and Figure 7 As shown, step S5 further includes the following steps, wherein Figure 7 An exemplary interrupt execution flow is shown for step S5.

[0155] In step S51, the execution status of the deployment task and the index information of the completed deployment steps are read from the cache queue associated with the deployment service hostname.

[0156] Specifically, the Redis cache queue stores a complete task list for each deployment task. This task list includes the task status and step index of all sub-steps in the deployment process. The execution status of each sub-step is divided into four types: Pending, Started, Successful, and Failed. By reading the execution status of the deployment task and the index information of the completed deployment steps, the exact location where the deployment task was interrupted can be pinpointed, allowing execution to resume from the breakpoint and improving interruption recovery efficiency.

[0157] Before this step, the deployment service can be started and the ordered queue set of the deployment process, such as CreateEcs or ConfigNetwork, initialized. Then, the queue to which the deployment service hostname belongs is retrieved from the database (e.g., Redis) acting as a buffer middleware, and the cached task list is loaded. Next, a careful traversal is performed on all retrieved task lists to obtain the execution status of the deployment tasks and the index information of completed deployment steps. It is determined whether the execution status of the deployment task is in the "Started" state. The "Started" state indicates that the deployment has begun but has not yet ended. When the execution status is in this state, step S52 continues.

[0158] In step S52, in response to the deployment task being executed and the creation time being within a preset valid time threshold, a list of remaining deployment tasks is generated from the index position of the completed deployment steps.

[0159] For example, the preset effective time threshold can be 1 hour, and this can be extended to 2 hours in high-concurrency scenarios such as e-commerce promotions. For tasks that exceed the effective time threshold, they are determined to be unrecoverable, their execution status is set to Fail, the reason for the timeout is recorded, and the recovery process is no longer executed. That is, only tasks whose execution status is Started and whose creation time is within the effective time threshold are determined to be valid interrupted tasks and enter the subsequent recovery process.

[0160] In step S53, an independent background process is started to execute the sub-steps in the remaining deployment task list in sequence. After each sub-step is completed, the task status and step index in the cache queue are updated. The sub-steps are idempotent sub-steps.

[0161] It should be noted that in the interrupt recovery method of this embodiment, each sub-step in the deployment task is set as an idempotent sub-step, so that the result of repeated execution is completely consistent with single execution. Thus, during interrupt recovery, the remaining deployment tasks can be completed independently in any sub-step without restarting the entire deployment process.

[0162] For example, upon completion of each sub-step, the execution status and index of completed steps in the Redis cache queue are immediately updated, the current step is marked as completed, and the index of the next step to be executed is updated. It should be noted that the execution result of each sub-step is synchronously written to Redis. Even if the service is interrupted and restarted during the execution of the remaining steps, execution can continue from the latest index of completed steps after the service recovers, achieving seamless recovery from multiple interruptions.

[0163] In step S54, after all sub-steps are completed, the execution status of the deployment task will be updated to successful to complete the interruption recovery process.

[0164] In this example, refer to Figure 7 As shown, after generating the list of remaining deployment tasks, subsequent remaining steps 1 to n are executed sequentially, where n is an integer greater than or equal to 1. The execution status and the index of completed steps are updated in the Redis cache queue. When all sub-steps in the remaining deployment task list have been executed, the execution status of the deployment task is updated to Success, completing the entire interruption recovery process. At the same time, a cluster deployment completion notification event is generated and synchronized to the platform management interface and the user notification center.

[0165] It should be noted that during interrupt recovery, the interface call tokens are deducted in real time to control the interface request rate within the token generation rate range. After each terminal recovery task is completed, the occupied resources and unused tokens are immediately released and returned to the token bucket for use by tasks in the waiting queue. Those skilled in the art should understand that the interrupt recovery execution of each deployment task follows the token allocation method described above, and the deployment task that allocates the token executes the task. The interrupt recovery of this deployment task is completed after all deployment tasks in the deployment task list have been traversed.

[0166] With the above settings, it can be ensured that in high-parallelism scenarios, both small and medium-sized tenants and top tenants can complete the recovery of all core tasks in a short period of time, and subsequent low-priority tasks can also be completed step by step in order of priority, with no task timeouts or failures throughout the process.

[0167] Based on the same inventive concept, referring to Figure 8 As shown, another aspect of this disclosure provides an interruption recovery system based on an EHPC cluster deployment, comprising:

[0168] Get unit 101 to get multiple deployment tasks created within a preset time period;

[0169] The task aggregation and classification unit 102, based on the tenant identifiers of the multiple deployment tasks, aggregates deployment tasks belonging to the same tenant into a batch task group, and classifies the business priority of each deployment task within the batch task group.

[0170] Resource management unit 103 constructs a layered token bucket with a four-layer structure of resources and interfaces based on the grouped deployment tasks, and divides the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools.

[0171] The scheduling admission unit 104, in response to the detection of a deployment service restart, traverses each batch task group, determines the admission qualification of the reserved core resource pool based on the business priority within the batch task group, and allocates tokens to the hierarchical token bucket.

[0172] Interruption recovery execution unit 105 executes the interrupt recovery process of the deployment task assigned by the token and releases the resources and tokens it holds.

[0173] By using tenant identifiers for multiple deployment tasks, deployment tasks belonging to the same tenant are aggregated into batch task groups. Each deployment task within a batch task group is then prioritized according to its business function. Furthermore, a layered token bucket with a four-layer structure encompassing both resources and interfaces is constructed based on the grouped deployment tasks. The platform resources used to execute deployment tasks are divided into mutually isolated reserved core resource pools and shared resource pools. This ensures that the core tasks of all tenants can achieve their recovery time targets even when a large number of deployment tasks are interrupted. In particular, it safeguards rigid requirements under peak extreme concurrency scenarios, ensuring the efficient and orderly recovery of deployment tasks.

[0174] It is worth noting that specific embodiments of the interruption recovery system based on EHPC cluster deployment in this disclosure can be found in the interruption recovery method based on EHPC cluster deployment in the foregoing embodiments, and will not be repeated here.

[0175] Another embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon that is implemented when executed by a processor:

[0176] Retrieve multiple deployment tasks created within a preset time period;

[0177] Based on the tenant identifiers of multiple deployment tasks, deployment tasks belonging to the same tenant are aggregated into batch task groups, and business priority is classified for each deployment task within the batch task group.

[0178] Based on the grouped deployment tasks, a hierarchical token bucket with a four-layer structure of resources and interfaces is constructed, and the platform resources used to execute deployment tasks are divided into mutually isolated reserved core resource pools and shared resource pools.

[0179] In response to the detection of a deployment service restart, the system iterates through each batch task group, determines the access qualifications for the reserved core resource pool based on the business priority within the batch task group, and allocates tokens from the hierarchical token bucket.

[0180] Execute the interruption recovery process for deployment tasks assigned via tokens and release their holdings on resources and tokens.

[0181] In practical applications, the computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0182] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0183] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0184] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0185] like Figure 9As shown, another embodiment of this disclosure provides a structural schematic diagram of a computer device. Figure 9 The computer device 12 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0186] like Figure 9 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0187] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0188] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0189] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 8 (Not shown; for example, it is referred to as a "hard drive"). Although Figure 9 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0190] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules 42 exemplary perform the functions and / or methods described in the embodiments of this disclosure.

[0191] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 20. Figure 9 As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 9 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0192] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing an interruption recovery method based on EHPC cluster deployment as described in the above embodiments of this disclosure.

[0193] In the description of this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0194] Obviously, the above embodiments of this disclosure are merely examples for clearly illustrating this disclosure, and are not intended to limit the implementation of this disclosure. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all implementation methods here. Any obvious variations or modifications derived from the technical solutions of this disclosure are still within the protection scope of this disclosure.

Claims

1. An interruption recovery method based on EHPC cluster deployment, characterized in that, include: Retrieve multiple deployment tasks created within a preset time period; Based on the tenant identifiers of the multiple deployment tasks, deployment tasks belonging to the same tenant are aggregated into a batch task group, and the business priority of each deployment task in the batch task group is classified. Based on the grouped deployment tasks, a hierarchical token bucket with a four-layer structure of resources and interfaces is constructed, and the platform resources used to execute deployment tasks are divided into mutually isolated reserved core resource pools and shared resource pools. In response to the detection of a deployment service restart, the system iterates through each batch task group, determines the access eligibility of the reserved core resource pool based on the business priority within the batch task group, and allocates tokens to the hierarchical token bucket. Perform the interruption recovery process for deployment tasks assigned via tokens, and release the resources and tokens they hold. The hierarchical token bucket structure, which is based on a two-dimensional, four-layer architecture of resources and interfaces, and which divides the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools, further includes: The platform resources and open interface call quotas are statistically analyzed to determine the total global token capacity and generation rate in order to initialize the global layer token bucket. The platform resources include the total platform computing resources and memory resources. Based on the tenant's level, the task volume of the batch task group, and the business priority distribution, initialize the tenant layer token bucket, the task group layer token bucket, and the single task layer token bucket to complete the construction of a two-dimensional four-layer structure of layered token buckets. The platform's total computing resources are divided into a reserved core resource pool and a shared resource pool according to a preset ratio, and independent token quotas are configured for each of the reserved core resource pool and the shared resource pool. Specifically, the reserved core resource pool is only open to deployment tasks with the highest business priority, while the shared resource pool is open to deployment tasks with all business priorities. The process of initializing tenant-level token buckets, task group-level token buckets, and single-task-level token buckets based on tenant levels, task volume, and business priority distribution in batch task groups to complete the construction of a two-dimensional, four-layered token bucket structure further includes: Configure a corresponding global quota limit for the shared resource pool for each tenant, and allocate basic token capacity and generation rate to the tenant-level token bucket based on the quota limit. The total number of tokens in the tenant-level token bucket shall not exceed the global quota limit of the reserved core resource pool. Obtain token quota from the corresponding tenant layer token bucket, initialize the task group layer token bucket for each batch task group, and reserve token quota for core transaction tasks according to a preset ratio; Obtain token quota from the corresponding task group layer token bucket, initialize a single task layer token bucket for each deployment task in the batch task group, and allocate the corresponding number of internal resource tokens and interface call tokens based on the task's business priority level; The smallest unit of the internal resource token is a fixed value of CPU cores, memory space, and task concurrency quota, while the smallest unit of the interface call token is the quota for a single open interface call.

2. The interruption recovery method according to claim 1, characterized in that, The step of aggregating deployment tasks belonging to the same tenant into a batch task group based on the tenant identifiers of the multiple deployment tasks, and further classifying the business priority of each deployment task within the batch task group, includes: Extract the tenant identifier, creation timestamp, and business line tag corresponding to each deployment task; All deployment tasks are initially grouped using tenant identification as the core aggregation dimension; Based on the creation timestamp and the preset sliding time window, the deployment tasks of the same tenant after the initial grouping are filtered a second time, and the deployment tasks within the time window are aggregated into a batch task group and a unique batch number is generated. Based on the business line tag of each deployment task, all deployment tasks in the batch task group are classified according to their business type, and the classification results and batch numbers are synchronously written into the metadata of the deployment tasks.

3. The interruption recovery method according to claim 1, characterized in that, The interruption recovery process for the deployment task assigned via token, and the release of its holdings in the resources and tokens, further includes: Read the execution status of the deployment task and the index information of the completed deployment steps from the cache queue associated with the deployment service hostname; In response to the deployment task being executed and the creation time being within a preset valid time threshold, a list of remaining deployment tasks is generated from the index position of the completed deployment steps; Start an independent background process and execute the sub-steps in the remaining deployment task list in sequence. After each sub-step is completed, update the task status and step index in the cache queue. The sub-steps are idempotent sub-steps. After all sub-steps are completed, the execution status of the deployment task will be updated to successful to complete the interruption recovery process.

4. The interruption recovery method according to claim 1, characterized in that, The task metadata of each deployment task is stored using a caching middleware, which is configured with an index-associative storage architecture, including a first-level index and a second-level index. The first-level index includes a single-task status structure, and the second-level index includes a task group metadata structure.

5. The interruption recovery method according to claim 1, characterized in that, The step of determining the access eligibility of the reserved core resource pool and allocating tokens to the tiered token bucket based on the business priority within the batch task group further includes: Based on the business priority level of the deployment task, the resource pool access verification is performed. Only core transaction-type deployment tasks are eligible to be reserved in the core resource pool, while other deployment tasks are only allowed to access the shared resource pool. Verify whether the resources used by the tenant to which the deployment task belongs exceed the global quota limit and whether the number of concurrent requests used by the batch task group to which it belongs exceeds the single batch limit. In response to the two verifications passing, allocate a token from the corresponding single task layer token bucket to the deployment task. When a core transaction deployment task runs out of tokens, it will seize the self-idle tokens that have been applied for but not used by low-priority tasks within the same tenant to supplement the tokens required for task execution.

6. An interruption recovery system based on EHPC cluster deployment, characterized in that, include: The acquisition unit retrieves multiple deployment tasks created within a preset time period; The task aggregation and grading unit aggregates deployment tasks belonging to the same tenant into a batch task group based on the tenant identifier of the multiple deployment tasks, and grades the business priority of each deployment task in the batch task group. The resource management unit constructs a layered token bucket with a four-layer structure based on the grouped deployment tasks, and divides the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools. The scheduling admission unit, in response to the detection of a service restart, traverses each batch task group, determines the admission qualification of the reserved core resource pool based on the business priority within the batch task group, and allocates tokens to the hierarchical token bucket. The interrupt recovery execution unit performs the interrupt recovery process for the deployment task assigned via token, and releases the resources and tokens it holds. The hierarchical token bucket structure, which is based on a two-dimensional, four-layer architecture of resources and interfaces, and which divides the platform resources used to execute deployment tasks into mutually isolated reserved core resource pools and shared resource pools, further includes: The platform resources and open interface call quotas are statistically analyzed to determine the total global token capacity and generation rate in order to initialize the global layer token bucket. The platform resources include the total platform computing resources and memory resources. Based on the tenant's level, the task volume of the batch task group, and the business priority distribution, initialize the tenant layer token bucket, the task group layer token bucket, and the single task layer token bucket to complete the construction of a two-dimensional four-layer structure of layered token buckets. The platform's total computing resources are divided into a reserved core resource pool and a shared resource pool according to a preset ratio, and independent token quotas are configured for each of the reserved core resource pool and the shared resource pool. Specifically, the reserved core resource pool is only open to deployment tasks with the highest business priority, while the shared resource pool is open to deployment tasks with all business priorities. The process of initializing tenant-level token buckets, task group-level token buckets, and single-task-level token buckets based on tenant levels, task volume, and business priority distribution in batch task groups to complete the construction of a two-dimensional, four-layered token bucket structure further includes: Configure a corresponding global quota limit for the shared resource pool for each tenant, and allocate basic token capacity and generation rate to the tenant-level token bucket based on the quota limit. The total number of tokens in the tenant-level token bucket shall not exceed the global quota limit of the reserved core resource pool. Obtain token quota from the corresponding tenant layer token bucket, initialize the task group layer token bucket for each batch task group, and reserve token quota for core transaction tasks according to a preset ratio; Obtain token quota from the corresponding task group layer token bucket, initialize a single task layer token bucket for each deployment task in the batch task group, and allocate the corresponding number of internal resource tokens and interface call tokens based on the task's business priority level; The smallest unit of the internal resource token is a fixed value of CPU cores, memory space, and task concurrency quota, while the smallest unit of the interface call token is the quota for a single open interface call.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the interrupt recovery method based on EHPC cluster deployment as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the interrupt recovery method based on EHPC cluster deployment as described in any one of claims 1-5.

Images