A backend concurrent scheduling method and system based on capacity credentials

Abstract

This application relates to the field of computer technology and provides a backend concurrent scheduling method based on capacity credentials. The method includes: abstracting each executable backend as a capacity node; scheduling requests for each service; filtering capacity nodes based on capacity requirements and constraints to generate a candidate node set; applying for capacity credentials for each candidate node in the candidate node set; determining a target capacity node and performing routing for the target capacity node; issuing execution requests for the scheduled requests based on the capacity credentials; and after obtaining the execution results, performing a release operation and generating usage events written to an asynchronous buffer; and when the asynchronous buffer reaches a preset flushing condition, performing global mutual exclusion and batch settlement. This application abstracts the concurrent occupancy of the backend as capacity credentials, without relying on specific model protocols or vendor interfaces, and can uniformly support multiple remote execution backends.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a backend concurrent scheduling method and system based on capacity credentials. Background Technology

[0002] With the widespread application of large models and heterogeneous computing resources, systems typically need to simultaneously connect to multiple inference engines and execution backends, and complete request scheduling, concurrency control, and metering settlement under high concurrency, dynamic topology, and multi-instance deployment conditions. Existing systems generally implement functional modules such as request admission, backend routing, concurrency control, and metering settlement in a decentralized manner. For example, using independent rate limiting components to control inbound traffic, using load balancers for backend routing, using distributed locks or semaphores to manage concurrent usage, and relying on independent metering services to complete asynchronous or synchronous settlement, this decentralized architecture can basically meet the requirements under low concurrency or static topology conditions.

[0003] However, the aforementioned distributed implementation methods have significant shortcomings in dynamic, high-concurrency scenarios. First, the lack of unified constraints between request ingress rate limiting and backend concurrency control makes local overload easily propagate downstream, triggering cascading failures. Second, when backend nodes dynamically go online or offline, existing solutions often use a full-scale routing pool replacement method, resulting in transient jitter during topology refresh and affecting the stability of request routing. Third, the lack of a global serial guarantee mechanism when multiple instances refresh the topology in parallel leads to inconsistent state updates among instances, easily causing state drift and disrupting the system's consistent view. Furthermore, when requests carry capacity constraints or compliance requirements, existing routing strategies cannot strictly and predictably meet such constraints. Finally, if the metering and billing logic is executed synchronously in a blocking manner, it will significantly increase the latency at the tail of the main link, while if an asynchronous method is used, it faces problems such as unstable billing policy switching in a multi-instance environment and the possibility of duplicate or lost billing records.

[0004] Therefore, there is an urgent need to provide a unified scheduling kernel that can achieve capacity occupancy management, topology smooth convergence, and settlement consistency without blocking the main execution link, so as to overcome the above-mentioned defects caused by the decentralized implementation in the existing technology. Summary of the Invention

[0005] This application provides a backend concurrent scheduling method and system based on capacity credentials, which can solve one of the above-mentioned problems in the prior art.

[0006] In a first aspect, embodiments of this application provide a backend concurrent scheduling method based on capacity credentials, including: Each executable backend is abstracted as a capacity node. For request scheduling of each business, capacity nodes are filtered according to capacity requirements and constraints to generate a set of candidate nodes. Apply for capacity credentials for each candidate node in the candidate node set, determine the target capacity node and perform routing for the target capacity node, the capacity credentials are used to measure the capacity occupancy relationship between the capacity node and the request scheduling; Based on the capacity certificate, the execution request for the request scheduling is issued, and after obtaining the execution result, a release operation is performed, and a usage event is generated and written to the asynchronous buffer. When the asynchronous buffer reaches the preset flushing condition, global mutual exclusion and batch settlement are performed.

[0007] Furthermore, the capacity certificate includes a certificate number, a request identifier, a backend identifier, a capacity unit, a lease expiry date, an occupancy sequence number, and a certificate status.

[0008] Furthermore, following the abstraction of each executable backend as a capacity node, the following is included: Each capacity node is encapsulated into a capacity node set, and within a preset detection period, the node heartbeat is dynamically updated. This is combined with differential convergence to dynamically update the capacity node set; and... Maintain a backend capability directory, which is a registration table of the capabilities and constraints of each capacity node. Each directory entry includes the capacity node's requested capability requirements and backend constraints.

[0009] Furthermore, the request scheduling for each service involves filtering capacity nodes based on capacity requirements and constraints to generate a candidate node set, including: Request information is extracted from the request scheduler, and the request information includes identity information, capability requirements, and constraints. Based on the identity information, the request scheduling is verified, and entry access is determined in conjunction with the access information; When identity verification is successful and the entry admission rules meet the preset requirements, a candidate node set is obtained from the capacity node set based on the capability requirements and the constraints, combined with the backend capability catalog.

[0010] Furthermore, the step of applying for capacity credentials for each candidate node in the candidate node set, determining the target capacity node, and performing route selection for the target capacity node includes: The candidate nodes in the candidate node set are sorted according to a preset routing strategy; Based on the sorting results, it is determined whether each candidate node meets the preset occupancy standard in order to apply for a capacity certificate and determine the target capacity node. The request is scheduled and mapped to the target capacity node, and the routing result is output.

[0011] Furthermore, the phrase "meeting the preset occupancy criteria" means that at the time of requesting occupancy, the candidate node must simultaneously meet the first occupancy condition, the second occupancy condition, the third occupancy condition, the fourth occupancy condition, and the fifth occupancy condition. The first occupancy condition indicates that the candidate node is online and in a healthy state; The second occupancy condition indicates that the available capacity of the candidate node is greater than or equal to the capacity unit required for this request; The third occupancy condition indicates that the candidate node has passed the idempotency check; The fourth occupancy condition indicates that the atomic increment of the occupied capacity of the candidate node has been successfully written, and the incremented occupied capacity is ≤ the upper limit of capacity - the protection reserved capacity. The fifth occupancy condition indicates that the candidate node has successfully generated a capacity certificate containing the occupancy sequence number and the lease expiry time, and has written the certificate status as "occupied".

[0012] Furthermore, determining whether each candidate node meets the preset occupancy standard includes: If none of the candidate nodes meet the preset occupancy criteria, then differential convergence update is triggered. If, after triggering differential convergence update, none of the candidate nodes still meet the preset occupancy criteria, then the standardized degradation result is returned. If all candidate nodes do not meet either the first or the second occupancy condition, the standardized degradation result is returned directly. The standard degradation result is a controlled failure response in a uniform format returned by the scheduling process. The controlled failure response includes at least the result code, failure reason, current routing version, whether to recommend retry, recommended retry waiting time, and optional degradation type.

[0013] Furthermore, the step of performing a release operation after obtaining the execution result includes: Obtain the credential number, request identifier, backend identifier, and occupancy sequence number of the release request; verify whether the occupancy sequence number is the latest occupancy sequence number; and verify whether the credential status of the capacity credential corresponding to the release request is "occupied" and the lease has not been revoked. If the verification passes, the occupied capacity in the capacity certificate will be atomically decremented and the certificate status will be updated to "released". If the verification fails, the release request is rejected.

[0014] Furthermore, the execution of global mutual exclusion and batch settlement includes: Global mutual exclusion is achieved for each usage event in the asynchronous buffer through a distributed lock or lease mechanism; Within the preset settlement window, each usage event initiates a locking request to the distributed lock key. The usage event that successfully acquires the distributed lock is designated as the sole settlementr and obtains the lock identifier. The usage events corresponding to the unique biller are aggregated into billing batches according to the tenant, key, backend, and billing policy for billing, and a batch identifier is generated.

[0015] Secondly, embodiments of this application provide a backend concurrent scheduling system based on capacity credentials, comprising: The first processing module is used to abstract each executable backend into a capacity node, schedule requests for each service, filter capacity nodes according to capacity requirements and constraints, and generate a set of candidate nodes. The second processing module is used to apply for capacity credentials for each candidate node in the candidate node set, determine the target capacity node and perform routing for the target capacity node, wherein the capacity credentials are used to measure the capacity occupancy relationship between the capacity node and the requested scheduling. The third processing module is used to issue the execution request of the request scheduling based on the capacity certificate, and after obtaining the execution result, to perform the release operation and generate a usage event to write to the asynchronous buffer. The fourth processing module is used to perform global mutual exclusion and batch settlement when the asynchronous buffer reaches the preset flushing condition.

[0016] The beneficial effects of the embodiments in this application compared with the prior art are: This application discloses a backend concurrent scheduling method based on capacity credentials. By abstracting the concurrent occupancy of backends into capacity credentials, the method measures the capacity occupancy relationship of each backend in the global context and releases capacity through these credentials. This method does not rely on specific model protocols or vendor interfaces and can uniformly support various remote execution backends such as text inference, vector retrieval, reordering services, and image processing. It is suitable for model gateways, inference platforms, and heterogeneous computing orchestration platforms. By setting an idempotent key in the capacity credentials, idempotency verification is performed to avoid duplicate occupancy caused by concurrent re-entry. At the same time, differential convergence updates are used to dynamically update the capacity node set to prevent transient jitter during the refresh process, which could affect the stability of request routing and ensure the serialization of refresh operations in a multi-instance environment. In addition, by setting an entry admission layer to control the number of system requests, combined with backend capacity occupancy, overload propagation is suppressed. After the request scheduling and execution of each backend is completed, asynchronous buffering and global mutual exclusion and batch settlement are used to achieve stable scheduling, consistent occupancy release, and low-latency main link execution in high-concurrency scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a backend concurrent scheduling method based on capacity credentials provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a backend concurrent scheduling system based on capacity credentials provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] Please see Figure 1 As shown, this invention is a backend concurrent scheduling method based on capacity credentials, comprising the following steps: S100. Abstract each executable backend into a capacity node. For request scheduling of each service, filter capacity nodes according to capacity requirements and constraints to generate a set of candidate nodes. In some embodiments, the capacity certificate includes a certificate number, a request identifier, a backend identifier, a capacity unit, a lease expiry time, an occupancy sequence number, and a certificate status.

[0026] In this embodiment, the concurrency control of heterogeneous backends is uniformly abstracted into the invocation of each capacity node based on capacity credentials. That is, the occupancy of backend concurrency is abstracted into capacity credentials, and the capacity credentials are used to measure the capacity occupancy relationship between capacity nodes and request scheduling. This process does not rely on specific model protocols or vendor interfaces, and can uniformly support multiple remote execution backends such as text inference, vector retrieval, reordering service, and image processing. It is suitable for model gateways, inference middleware, and heterogeneous computing orchestration platforms.

[0027] Specifically, each executable backend is abstracted as a capacity node. For each capacity node, a corresponding capacity credential is generated based on the request scheduling that calls that capacity node. This credential is used to measure the capacity occupancy relationship of the capacity node globally. The capacity credential includes a credential number, a request identifier, a backend identifier, a capacity unit, a lease expiration time, an occupancy sequence number, and a credential status. The credential number is used to uniquely identify the capacity occupancy relationship globally; the request identifier is used to identify the request scheduling of the same business; the backend identifier is used to identify the target capacity node that is occupied; the capacity unit is used to represent the concurrent share, token quota, or standardized computing power share consumed by the current occupancy; the lease expiration time is used to trigger automatic reclamation in abnormal situations; the occupancy sequence number is used to identify the latest valid occupancy version of the same request scheduling on the same backend; and the credential status indicates whether the capacity credential is in the stages of pending creation, occupied, released, expired, release rejected, or settlement completed.

[0028] Furthermore, in the capacity credential, the request identifier and the backend identifier constitute the occupancy idempotent key, which is used to uniquely identify the correspondence between the request scheduling and the capacity node, that is, to identify the unique occupancy of the capacity credential. Specifically, the occupancy idempotent key K = (request identifier, backend identifier). During the request scheduling process, for repeated occupancy requests with the same occupancy idempotent key, the result of the first successful occupancy is returned, thereby avoiding duplicate occupancy caused by concurrent re-entry. At the same time, an idempotent record table is maintained, which records the occupancy idempotent key generated by the capacity node called by the request scheduling, and records the capacity credential for each occupancy idempotent key, which facilitates the execution of subsequent idempotent verification, thereby avoiding the repeated determination of the same capacity node for the same request scheduling and improving scheduling efficiency.

[0029] In some embodiments, after abstracting each executable backend as a capacity node, the following is included: Each capacity node is encapsulated into a capacity node set, and within a preset detection period, the node heartbeat is dynamically updated. This is combined with differential convergence to dynamically update the capacity node set; and... Maintain a backend capability directory, which is a registration table of the capabilities and constraints of each capacity node. Each directory entry includes the capacity node's requested capability requirements and backend constraints.

[0030] In this embodiment, all capacity nodes are encapsulated into a capacity node set, and differential convergence updates are performed on it within a preset detection period to dynamically update the capacity node set. Specifically, based on the registration center and configuration center, each backend is abstracted as a capacity node, and then the node heartbeat information of each capacity node is combined to generate a capacity node set. Within the preset detection period, the heartbeat dynamic update of the capacity node set is periodically performed to obtain node heartbeat information. Specifically, the scheduling kernel detects change events in the registration center or configuration center, and generates a target node snapshot based on the detection results. The target node snapshot and the capacity node set are compared and changed according to the node identifier to generate a differential set, which includes the intersection, the incremental set, and the expired set.

[0031] Specifically, based on the detection results, it can be determined whether a new capacity node has been added, a capacity node has been deleted, or the status of each capacity node has been updated, thereby forming the node heartbeat information of the corresponding capacity node. The node heartbeat information includes at least the backend identifier, capability tag, capacity limit, interface address, health status, version number, and most recent heartbeat time.

[0032] Furthermore, a change is determined when any of the following situations occur between the target node snapshot and the capacity node set: 1) a capacity node is added or deleted; 2) the capability label, capacity limit, interface address or static weight of a node changes; 3) the heartbeat of a node exceeds the timeout threshold and is not updated; 4) the health status check of a node fails for a preset number of consecutive times; 5) the version number or configuration summary changes.

[0033] Furthermore, when changes occur, after the debouncing time window ends, the capacity node set is updated in conjunction with differential convergence update to generate a new capacity node set, so as to ensure the real-time performance of each capacity node in the capacity node set, thereby improving the real-time performance and accuracy of subsequent candidate node selection. At the same time, a debouncing time window is set to avoid repeated refreshes caused by frequent jitter.

[0034] More specifically, for differential convergence, based on node identifiers, each capacity node is divided into an intersection set, an incremental set, and an expired set. Logically, the node set relationship between the target node snapshot and the capacity node set can be abstracted as: intersection node = target node snapshot ∩ capacity node set; incremental node = target node snapshot - capacity node set; expired node = capacity node set - target node snapshot. For intersection nodes in the intersection set, their capacity information, static node weight, health status, and version information are updated based on their node status updates without re-establishing connections with the capacity node set. For incremental nodes in the incremental set, they are preheated, health checked, and registered in the directory before being added to the capacity node set for subsequent candidate node selection. For expired nodes in the expired set, their node status is marked as unassignable, and they are removed from the capacity node set only after their pending release request is completed or their lease expires.

[0035] Furthermore, when updating the capacity node set, the process follows the sequence of "intersection update → incremental addition → expired deletion." After all differential set items have been processed, the routing version number is uniformly increased and a new capacity node set is published, thus achieving smooth convergence, preventing transient fluctuations during the refresh process, and ensuring the stability and reliability of the update. It is worth noting that the above sequential execution only describes the update order and does not limit the specific data structure or implementation method. The differential convergence update of the capacity node set is controlled by both in-process mutual exclusion and distributed mutual exclusion, ensuring that the refresh operation is serialized in a multi-instance environment, allowing only instances holding valid global mutual exclusion to submit refresh results.

[0036] Furthermore, the backend capability directory is a structured registration table for each capacity node, recording the service capabilities and currently schedulable attributes that each backend node can provide. Each capacity node corresponds to one directory entry, and each entry includes a backend identifier, a directory version number identifying the directory, a static node weight, and the request capability requirements and backend constraints for calling that capacity node. The request capability requirements at least include service type, engine type, model or algorithm identifier, capacity limit, and reserved protection capacity. Backend constraints at least include region or availability zone, compliance label, health status, most recent heartbeat time, and interface address. It's worth noting that each directory entry can be dynamically updated by the registry center, configuration center, or node heartbeats for subsequent candidate node and target capacity node selection.

[0037] In some embodiments, for request scheduling of each service, capacity nodes are screened based on capacity requirements and constraints to generate a candidate node set, including: Request information is extracted from the request scheduler, and the request information includes identity information, capability requirements, and constraints. Based on the identity information, the request scheduling is verified, and entry access is determined in conjunction with the access information; When identity verification is successful and the entry admission rules meet the preset requirements, a candidate node set is obtained from the capacity node set based on the capability requirements and the constraints, combined with the backend capability catalog.

[0038] In this embodiment, the system receives service request scheduling and obtains the corresponding request information, thereby filtering nodes in the capacity node set. Before node filtering, the request scheduling needs to be verified for identity and access control, which can effectively prevent unauthorized requests from entering the system, avoid malicious attacks and unauthorized access, and ensure that only legitimate requests that meet the access control conditions participate in subsequent node filtering, thereby improving system security and the rationality of resource utilization.

[0039] Specifically, the request information for scheduling includes identity information, namely the access key, signature, timestamp, random number and tenant identifier carried in the request. Identity verification is completed by verifying whether the signature is generated by a shared key or public-private key system, whether the timestamp falls within the allowed window, and whether the random number is reused.

[0040] More specifically, the access information is set by the entry control layer and is used to determine whether the request scheduling meets the entry access rules. The entry control layer is used to suppress sudden requests, ensure the rationality of the number of system requests, and prevent the system from being overloaded and paralyzed. The access information includes tenant quota, key status, blacklist and whitelist, number of requests per unit time, queuing depth, and total system load. The entry access judgment is completed by checking whether the corresponding information exceeds the preset threshold.

[0041] In some embodiments, a two-layer load control strategy involving both the entry point and the backend is adopted to regulate the scheduling of concurrent requests in the system. Specifically, the entry point load control layer suppresses sudden requests, while the backend load control layer limits the actual concurrent usage. That is, in the backend load control layer, capacity credentials are used to limit the concurrent usage of each heterogeneous backend. When the available capacity of each capacity node in the backend drops below a threshold, the entry access is dynamically tightened, that is, the preset thresholds corresponding to the access information are adjusted accordingly to reduce the number of requests entering the system.

[0042] Furthermore, the request scheduling will only proceed to the subsequent scheduling process when the identity verification is successful and the entry admission rules meet the preset requirements. For the selection of candidate nodes, the process is determined by querying the directory items in the backend capability directory based on the capability requirements and constraints of each capacity node. Specifically, the capability requirements include at least the service type, engine type, model or algorithm identifier, request size, and required capacity unit. The constraints include at least the availability zone, compliance label, tenant isolation level, cost level, latency level, or a specified backend set. In some embodiments, the above capability requirements and constraints are standardized into a unified constraint expression so that a computable match can be performed in the backend capability directory later.

[0043] Specifically, the process of filtering candidate nodes based on the backend capability directory is as follows: read the directory version number corresponding to the request scheduling to determine the backend capability directory to be filtered, and then filter according to service type, engine type, model identifier, region, compliance label and health status in sequence to obtain a set of candidate nodes that meet the capability constraints and are in a serviceable health status for subsequent filtering and occupancy processing of target capacity nodes.

[0044] S200: Apply for capacity certificates for each candidate node in the candidate node set, determine the target capacity node, and perform routing for the target capacity node; In some embodiments, step S200 above includes: The candidate nodes in the candidate node set are sorted according to a preset routing strategy; Based on the sorting results, it is determined whether each candidate node meets the preset occupancy standard in order to apply for a capacity certificate and determine the target capacity node. The request is scheduled and mapped to the target capacity node, and the routing result is output.

[0045] In this embodiment, for each candidate node in the candidate node set, the nodes need to be sorted first. When filtering the nodes, the candidate nodes that are ranked first apply for and occupy capacity certificates first, so as to improve the overall filtering efficiency.

[0046] Specifically, in the candidate node set, if any candidate node meets the preset occupancy criteria, it successfully applies for a capacity certificate and is identified as the target capacity node. An execution request is then sent to the backend corresponding to the target capacity node. During the screening process of the candidate node set, the occupancy criteria are judged sequentially according to the sorting results of each candidate node. If the preset occupancy criteria are met, the application for the capacity certificate is successful, and the candidate node is identified as the target capacity node. If the preset occupancy criteria are not met, it means that the application for the capacity certificate has failed, that is, the occupancy of the candidate node has failed. The next candidate node is then judged until success or the candidate node set is exhausted.

[0047] In one embodiment, the preset routing strategy takes into account at least the remaining capacity ratio of each candidate node, the static weight of the node, the nearest window latency, and the error rate to measure the overall availability of each candidate node.

[0048] It is worth noting that each capacity node maintains its capacity limit and occupied capacity. At the same time, to avoid instantaneous overprovision caused by the release delay of each node, a protective reserved capacity is also set to cover the short window during the release process. Thus, based on the capacity limit, occupied capacity, and protective reserved capacity, the available capacity can be calculated to determine whether the capacity node allows new execution to occupy it. The calculation relationship can be expressed as: available capacity = capacity limit - occupied capacity - protective reserved capacity. It can be understood that the above relationship is used to describe the capacity constraint logic, and in the specific implementation, it is maintained through atomic counting or distributed state storage.

[0049] Furthermore, for the selected target capacity node, routing must be performed to ensure that the current request is sent to a backend that is compatible in terms of capability, compliant in the region, healthy and available, and whose current capacity can handle it. This avoids sending requests that do not meet the constraints to general nodes, which could lead to compliance risks, execution failures, or uncertainty degradation. Specifically, the routing process is the process of deterministically mapping the request to the target capacity node while meeting the request capacity requirements and constraints, and outputting the routing result. The routing result can be expressed as: Routing result = (Target backend identifier, Routing version number, Target interface address, Capacity unit).

[0050] In some embodiments, meeting the preset occupancy criteria means that at the time of requesting occupancy, the candidate node must simultaneously meet the first occupancy condition, the second occupancy condition, the third occupancy condition, the fourth occupancy condition, and the fifth occupancy condition. The first occupancy condition indicates that the candidate node is online and in a healthy state; The second occupancy condition indicates that the available capacity of the candidate node is greater than or equal to the capacity unit required for this request; The third occupancy condition indicates that the candidate node has passed the idempotency check; The fourth occupancy condition indicates that the atomic increment of the occupied capacity of the candidate node has been successfully written, and the incremented occupied capacity is ≤ the upper limit of capacity - the protection reserved capacity. The fifth occupancy condition indicates that the candidate node has successfully generated a capacity certificate containing the occupancy sequence number and the lease expiry time, and has written the certificate status as "occupied".

[0051] Specifically, the evaluation criterion for successful capacity certificate application is whether the candidate node meets the preset occupancy criteria at the time of application. In a preferred embodiment of this application, the preset occupancy criteria include a first occupancy condition, a second occupancy condition, a third occupancy condition, a fourth occupancy condition, and a fifth occupancy condition. Only when all five occupancy conditions are met can the candidate node be considered to be occupied, selected as the target capacity node for the request scheduling, and a capacity certificate is generated to measure the capacity occupancy relationship between the capacity node and the request scheduling.

[0052] More specifically, for idempotency verification, the backend identifier of the candidate node and the request identifier of the request scheduling are used to construct a candidate idempotency key. The idempotency record table is queried. If the candidate idempotency key exists in the idempotency record table and its corresponding certificate status is "occupied" and the lease expiration time indicates that it has not expired, the capacity certificate of the corresponding record in the idempotency record table is directly returned. If there is no valid record in the idempotency record table, the current occupied capacity and the latest occupied sequence number of the candidate node are read. Under the premise that the available capacity is sufficient, the occupied capacity is increased by the capacity unit required by this request through an atomic comparison and exchange operation to generate a new occupied capacity of the candidate node, and the occupied sequence number is incremented to obtain a new occupied sequence number.

[0053] Specifically, the absence of a valid record indicates that the candidate idempotent key does not exist, or the credential status of the occupied idempotent key corresponding to the candidate idempotent key in the idempotent record table is an invalid record such as released, expired, release rejected, or solution completed, or the lease expiration time of the occupied idempotent key corresponding to the candidate idempotent key in the idempotent record table indicates that it has expired.

[0054] Furthermore, in the fourth occupancy condition, the new occupied capacity of the candidate node should not exceed "maximum capacity - protection reserved capacity", which means that after the application of this request call in the candidate node, its available capacity must be greater than zero, and then it is written into the capacity certificate of the candidate node.

[0055] Furthermore, in the fifth occupancy condition, the new occupancy sequence number of the candidate node needs to be written into the capacity certificate of the candidate node in sync with the lease expiration time, and the certificate status needs to be written as "occupied". At the same time, the contents of the above capacity certificate also need to be updated in the idempotency record table.

[0056] In some embodiments, when determining whether the preset occupancy criteria are met, the first occupancy condition, the second occupancy condition, the third occupancy condition, the fourth occupancy condition, and the fifth occupancy condition can be executed sequentially to update and generate the capacity unit, lease expiry time, occupancy sequence number, and certificate status of the capacity certificate. When all occupancy conditions meet the occupancy criteria, the capacity certificate of the candidate node has been formed accordingly. If any occupancy condition fails to meet its criteria during the sequential execution process, it indicates that the corresponding capacity unit, lease expiry time, occupancy sequence number, or certificate status has failed to be written, and the next candidate node is then tried.

[0057] In some embodiments, determining whether each candidate node meets the preset occupancy criteria includes: If none of the candidate nodes meet the preset occupancy criteria, then differential convergence update is triggered. If, after triggering differential convergence update, none of the candidate nodes still meet the preset occupancy criteria, then the standardized degradation result is returned. If all candidate nodes do not meet either the first or the second occupancy condition, the standardized degradation result is returned directly. The standard degradation result is a controlled failure response in a uniform format returned by the scheduling process. The controlled failure response includes at least the result code, failure reason, current routing version, whether to recommend retry, recommended retry waiting time, and optional degradation type.

[0058] In this embodiment, if the candidate node set is exhausted and no target capacity node is obtained, it means that all candidate nodes do not meet the preset occupancy standard. At this time, it is necessary to obtain the reason for the non-compliance of each candidate node. If the reason for the non-compliance of all candidate nodes is that they do not meet the occupancy standard of the first occupancy condition or the second occupancy condition, it means that the candidate nodes in the candidate node set do not meet the requirements of this request call and the quality of the candidate nodes is poor. Therefore, the standardized degradation result is returned directly.

[0059] Specifically, the standardized degradation result refers to the controlled failure response in a unified format returned by the scheduling kernel during the scheduling process. This controlled failure response includes at least the result code, the reason for failure, the current routing version, whether to suggest retrying, the suggested retry waiting time, and the optional degradation type. For failures caused by unmet constraints, a non-retry result can be returned; for failures caused by insufficient instantaneous capacity, a result suggesting retrying later or entering the queuing channel can be returned, thereby avoiding unbounded retries by the caller.

[0060] Furthermore, for cases where multiple types of non-compliance exist among the first, second, third, fourth, and fifth occupancy conditions, a differential convergence update is performed before executing the standardized degradation result. This involves dynamically updating the node heartbeat to determine whether a new executable capacity node has been added to the capacity node set as a candidate node to execute the occupancy criterion, or whether the status of a candidate node has changed to meet the occupancy criteria for each condition, thus enabling it to pass the capacity credential occupancy standard. If, after the differential convergence update, none of the candidate nodes in the candidate node set still meet the preset occupancy criteria, the standardized degradation result is returned, and a controlled failure response is recorded.

[0061] S300. Based on the capacity certificate, issue the execution request for the request scheduling, and after obtaining the execution result, perform the release operation and generate a usage event to write to the asynchronous buffer; In some embodiments, the step of performing a release operation after obtaining the execution result includes: Obtain the credential number, request identifier, backend identifier, and occupancy sequence number of the release request; verify whether the occupancy sequence number is the latest occupancy sequence number; and verify whether the credential status of the capacity credential corresponding to the release request is "occupied" and the lease has not been revoked. If the verification passes, the occupied capacity in the capacity certificate will be atomically decremented and the certificate status will be updated to "released". If the verification fails, the release request is rejected.

[0062] In this embodiment, after obtaining the capacity certificate of the target capacity node, an execution request for scheduling the request is issued. The backend corresponding to the target capacity node executes the service scheduled by the request, and after the execution is completed, a release operation is initiated to release the capacity occupied by the request scheduling.

[0063] It is worth noting that the release operation refers to the return of the previously occupied capacity share after the execution task corresponding to the target capacity node is completed normally, fails to execute, is actively canceled, timed out, or the calling instance recovers abnormally. The process does not require the corresponding capacity node to physically leave the capacity node cluster. Instead, it means that the capacity node has ended the execution task related to the scheduling of the request and no longer occupies the corresponding capacity. Instead, it releases the capacity occupied by the execution of the scheduling of the request and updates the capacity certificate, idempotency table, and capacity certificate record accordingly.

[0064] In this embodiment, to prevent duplicate releases caused by callbacks of old execution results, when the system receives a release request, it first reads the latest occupancy sequence number of the corresponding capacity certificate. Only when the occupancy sequence number carried by the release request is consistent with the latest occupancy sequence number and the certificate status is "occupied" will the release operation of atomically decrementing the occupied capacity and updating the certificate status to "released" be allowed. If the occupancy sequence number is outdated, the status has changed, or the lease has been reclaimed, the release write will be rejected.

[0065] Specifically, after execution, the execution side or scheduling side initiates a release request. This release request carries the credential number, request identifier, backend identifier, and occupancy sequence number. Based on the occupancy idempotent key formed by the request identifier and backend identifier, the capacity credential is read from the idempotent record table, and the latest occupancy sequence number, credential status, and lease expiration time of the capacity credential are obtained. The occupancy sequence number carried in this release request is compared with the latest occupancy sequence number of the capacity credential, and it is verified that the credential status is still "occupied" and the lease has not been reclaimed. Only when the above verification results are consistent will the occupied capacity be atomically decremented and the credential status updated. If the occupancy sequence number is less than the latest occupancy sequence number, it means that the release request corresponds to an old task or an old callback, and should be rejected directly to avoid the new occupancy being overwritten by the old release.

[0066] In this embodiment, after the release operation is performed, a usage event is generated for the release request, the corresponding capacity certificate, and the execution task. The execution result of the requested task and the release operation result are recorded. In addition, the usage event of each requested task is written to an asynchronous buffer without blocking the main link.

[0067] S400. When the asynchronous buffer reaches the preset flushing condition, global mutual exclusion and batch settlement are performed.

[0068] In this embodiment, the flushing condition refers to the threshold at which the usage events in the asynchronous buffer reach the point where they need to be written from memory or message queue to the settlement storage. Then, global mutual exclusion and batch calculation are performed, which significantly reduces the response latency of the main link.

[0069] Specifically, the criteria for meeting the flushing conditions can be set to satisfy any of the following conditions: the number of usage events is not less than the quantity threshold N; the cumulative usage in the buffer is not less than the usage threshold Q; the dwell time of the earliest unsettled event is not less than the time threshold T; or the usage event instance enters the graceful shutdown, policy switching, or periodic timer trigger state.

[0070] In some embodiments, the execution of global mutual exclusion and batch settlement includes: Global mutual exclusion is achieved for each usage event in the asynchronous buffer through a distributed lock or lease mechanism; Within the preset settlement window, each usage event initiates a locking request to the distributed lock key. The usage event that successfully acquires the distributed lock is designated as the sole settlementr and obtains the lock identifier. The usage events corresponding to the unique biller are aggregated into billing batches according to the tenant, key, backend, and billing policy for billing, and a batch identifier is generated.

[0071] In this embodiment, global mutual exclusion and batch settlement means that in a multi-instance deployment scenario, only one instance is allowed to hold the distributed lock to execute the aggregation, billing, account write-back and idempotent write of the current batch of events in the same settlement window, while other instances only continue to write to the asynchronous buffer without repeated settlement.

[0072] In this embodiment, global mutual exclusion can be obtained through distributed locks or lease mechanisms, such as using Redis-based SETNX with expiration time, etcd-based lease locks, or ZooKeeper-based temporary sequential nodes. The usage event instance that successfully obtains mutual exclusion is the sole settler and obtains a lock identifier. The usage events of the sole settler are aggregated into settlement batches according to tenant, key, backend, and billing policy, generating a batch identifier. This batch identifier is combined with the lock identifier to form "settlement batch identifier + lock identifier" as the sole execution credential for this batch settlement. This is used for cost calculation, account deduction, usage accumulation, and result writing back to prevent duplicate settlement or illegal operations. Usage event instances that do not obtain mutual exclusion do not perform settlement during this settlement process and only maintain buffer writes.

[0073] Specifically, taking Redis-based SETNX with expiration time as an example, its distributed lock key can be set to "lock:settlement_window_{window_id}", where {window_id} is a unique identifier for the settlement window, used to distinguish different settlement windows. During global mutual exclusion, multiple usage event instances start simultaneously and remain in a standby state, waiting to operate on the settlement window. Each instance has the ability to access the distributed storage system to acquire and release the distributed lock. Furthermore, one of the multiple instances first initiates a lock request for the distributed lock key corresponding to the settlement window. Specifically, this instance sends a lock request to the Redis server. If the Redis server returns 1, it indicates that the instance has successfully acquired the distributed lock. At this time, the system will generate a lock token or fence token for this lock acquisition operation as a lock identifier. The instance that successfully acquired the distributed lock is the sole settler and performs this round of batch settlement. In the settlement, the "settlement batch identifier + lock token" is stored as the unique execution credential for this batch settlement, and subsequent settlement operations need to verify this credential. If the Redis server returns 0, it indicates that other instances have already acquired the distributed lock, and the current instance cannot perform settlement operations, but only maintains buffer writes, waiting for the next opportunity to acquire the lock.

[0074] More specifically, once the sole settler completes all settlement operations, it proactively releases the distributed lock, allowing other instances to acquire it. To prevent duplicate settlements due to accidental lock acquisition, the distributed lock acquisition process preferably includes an expiration time and a fence token. The expiration time ensures that even if an instance crashes during settlement, preventing it from proactively releasing the lock, the distributed lock will automatically release after a certain period, avoiding deadlocks. The fence token further enhances the security of the distributed lock, ensuring that only instances holding the correct token can perform settlement operations.

[0075] Furthermore, other usage event instances that have not acquired the lock will attempt to acquire the lock again after the distributed lock is released, repeating the above process until all events in the settlement window have been processed. This ensures that in a multi-instance deployment scenario, only one instance performs the settlement operation for the same settlement window, avoiding issues such as duplicate settlement and data inconsistency.

[0076] Furthermore, after the settlement operation is completed, the account, key, and usage status are written back to facilitate the traceability of the operation process, thereby ensuring data consistency and accuracy.

[0077] In this embodiment, for concurrent scheduling of each request, observation metrics are generated, including at least the entry access pass rate, identity verification failure rate, candidate node screening time, route selection hit rate, capacity credential occupancy success rate, idempotency hit rate, release success rate, number of expired recyclings, number of times old occupied sequence number releases are rejected, topology refresh frequency, differential update time, buffer backlog, number of batch settlement batches, settlement success rate, account write-back time, number of standardized degradations, and distribution of various failure reasons. At the same time, the above metrics can be aggregated and output according to the dimensions of tenant, backend, region, model, and time window for policy feedback and capacity tuning.

[0078] Please see Figure 2 As shown, the present invention also provides a backend concurrent scheduling system based on capacity credentials, the system comprising: First processing module 201: used to abstract each executable backend as a capacity node, schedule requests for each service, filter capacity nodes according to capacity requirements and constraints, and generate a set of candidate nodes; The second processing module 202 is used to apply for capacity credentials for each candidate node in the candidate node set, determine the target capacity node and perform routing for the target capacity node, wherein the capacity credentials are used to measure the capacity occupancy relationship between the capacity node and the requested scheduling. The third processing module 203 is used to issue an execution request for the request scheduling based on the capacity certificate, and after obtaining the execution result, to perform a release operation and generate a usage event to write to the asynchronous buffer. The fourth processing module 204 is used to perform global mutual exclusion and batch settlement when the asynchronous buffer reaches the preset flushing condition.

[0079] It is understandable that, such as Figure 1 The content of the capacity credential-based backend concurrent scheduling method embodiments shown herein is applicable to this capacity credential-based backend concurrent scheduling system embodiment. The specific functions implemented in this capacity credential-based backend concurrent scheduling system embodiment are the same as those described above. Figure 1 The backend concurrent scheduling method based on capacity credentials shown is the same as the embodiment, and the beneficial effects achieved are the same as those described above. Figure 1 The beneficial effects achieved by the illustrated embodiment of the backend concurrent scheduling method based on capacity credentials are also the same.

[0080] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0082] Please see Figure 3 As shown, this embodiment of the invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, it implements the backend concurrent scheduling method based on capacity credentials as described in any of the above methods.

[0083] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0084] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0085] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0086] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the backend concurrent scheduling method based on capacity credentials as described in any of the above methods.

[0087] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0088] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A backend concurrent scheduling method based on capacity credentials, characterized in that, include: Each executable backend is abstracted as a capacity node. For request scheduling of each business, capacity nodes are filtered according to capacity requirements and constraints to generate a set of candidate nodes. Apply for capacity credentials for each candidate node in the candidate node set, determine the target capacity node and perform routing for the target capacity node, the capacity credentials are used to measure the capacity occupancy relationship between the capacity node and the request scheduling; Based on the capacity certificate, the execution request for the request scheduling is issued, and after obtaining the execution result, a release operation is performed, and a usage event is generated and written to the asynchronous buffer. When the asynchronous buffer reaches the preset flushing condition, global mutual exclusion and batch settlement are performed.

2. The method as described in claim 1, characterized in that, The capacity certificate includes certificate number, request identifier, backend identifier, capacity unit, lease expiration time, occupancy sequence number, and certificate status.

3. The method as described in claim 1, characterized in that, Following the abstraction of each executable backend as a capacity node, the following is included: Each capacity node is encapsulated into a capacity node set, and within a preset detection period, the node heartbeat is dynamically updated. This is combined with differential convergence to dynamically update the capacity node set; and... Maintain a backend capability directory, which is a registration table of the capabilities and constraints of each capacity node. Each directory entry includes the capacity node's requested capability requirements and backend constraints.

4. The method as described in claim 3, characterized in that, The request scheduling for each service involves filtering capacity nodes based on capacity requirements and constraints to generate a candidate node set, including: Request information is extracted from the request scheduler, and the request information includes identity information, capability requirements, and constraints. Based on the identity information, the request scheduling is verified, and entry access is determined in conjunction with the access information; When identity verification is successful and the entry admission rules meet the preset requirements, a candidate node set is obtained from the capacity node set based on the capability requirements and the constraints, combined with the backend capability catalog.

5. The method as described in claim 1, characterized in that, The step of applying for capacity credentials for each candidate node in the candidate node set, determining the target capacity node, and performing route selection for the target capacity node includes: The candidate nodes in the candidate node set are sorted according to a preset routing strategy; Based on the sorting results, it is determined whether each candidate node meets the preset occupancy standard in order to apply for a capacity certificate and determine the target capacity node. The request is scheduled and mapped to the target capacity node, and the routing result is output.

6. The method as described in claim 5, characterized in that, The phrase "meeting the preset occupancy criteria" means that at the time of requesting occupancy, the candidate node must simultaneously meet the first occupancy condition, the second occupancy condition, the third occupancy condition, the fourth occupancy condition, and the fifth occupancy condition. The first occupancy condition indicates that the candidate node is online and in a healthy state; The second occupancy condition indicates that the available capacity of the candidate node is greater than or equal to the capacity unit required for this request; The third occupancy condition indicates that the candidate node has passed the idempotency check; The fourth occupancy condition indicates that the atomic increment of the occupied capacity of the candidate node has been successfully written, and the incremented occupied capacity is ≤ the upper limit of capacity - the protection reserved capacity. The fifth occupancy condition indicates that the candidate node has successfully generated a capacity certificate containing an occupancy sequence number and a lease expiry time, and has written the certificate status as occupied.

7. The method as described in claim 6, characterized in that, The determination of whether each candidate node meets the preset occupancy standard includes: If none of the candidate nodes meet the preset occupancy criteria, then differential convergence update is triggered. If, after triggering differential convergence update, none of the candidate nodes still meet the preset occupancy criteria, then the standardized degradation result is returned. If all candidate nodes do not meet either the first or the second occupancy condition, the standardized degradation result is returned directly. The standard degradation result is a controlled failure response in a uniform format returned by the scheduling process. The controlled failure response includes at least the result code, failure reason, current routing version, whether to recommend retry, recommended retry waiting time, and optional degradation type.

8. The method as described in claim 1, characterized in that, The step of performing a release operation after obtaining the execution result includes: Get the credential number, request identifier, backend identifier, and occupancy sequence number of the release request; verify whether the occupancy sequence number is the latest occupancy sequence number; and verify whether the credential status of the capacity credential corresponding to the release request is occupied and the lease has not been revoked. If the verification passes, the occupied capacity in the capacity certificate will be atomically decremented and the certificate status will be updated to released. If the verification fails, the release request is rejected.

9. The method as described in claim 1, characterized in that, The execution of global mutual exclusion and batch settlement includes: Global mutual exclusion is achieved for each usage event in the asynchronous buffer through a distributed lock or lease mechanism; Within the preset settlement window, each usage event initiates a locking request to the distributed lock key. The usage event that successfully acquires the distributed lock is designated as the sole settlementr and obtains the lock identifier. The usage events corresponding to the unique biller are aggregated into billing batches according to the tenant, key, backend, and billing policy for billing, and a batch identifier is generated.

10. A backend concurrent scheduling system based on capacity credentials, characterized in that, include: The first processing module is used to abstract each executable backend into a capacity node, schedule requests for each service, filter capacity nodes according to capacity requirements and constraints, and generate a set of candidate nodes. The second processing module is used to apply for capacity credentials for each candidate node in the candidate node set, determine the target capacity node and perform routing for the target capacity node, wherein the capacity credentials are used to measure the capacity occupancy relationship between the capacity node and the requested scheduling. The third processing module is used to issue the execution request of the request scheduling based on the capacity certificate, and after obtaining the execution result, to perform the release operation and generate a usage event to write to the asynchronous buffer. The fourth processing module is used to perform global mutual exclusion and batch settlement when the asynchronous buffer reaches the preset flushing condition.

Images