Multi-model api request routing and scheduling method based on multi-dimensional state feedback

Abstract

The application discloses a multi-model API request routing scheduling method based on multi-dimensional state feedback and belongs to the field of artificial intelligence service interface scheduling and computer network resource management. The method encodes the request into a request constraint vector containing capability, data volume, complexity, streaming, time limit and cost constraint, reads the layered and versioned channel state and converts it into a success probability lower bound, a completion time delay upper bound and a cost upper bound, constructs channel nodes, fault domain nodes and primary and backup directed edges, generates a primary and backup routing execution plan with a planned state version under the double budget of remaining time and cost and the shared fault domain penalty; after the call fails, the source is attributed, the budget is updated and the version consistency is reviewed, and the state used for scoring, fusing and primary and backup relationship is atomically updated with the same incremental state version, so that the repeated hits, queuing and retry amplification of the fault channel are inhibited, and the tail delay, time limit violation and unit successful request cost are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence service interface scheduling and computer network resource management, and in particular to a multi-model API request routing and scheduling method based on multi-dimensional state feedback. Background Technology

[0002] Multi-model API aggregation platforms typically connect to multiple vendors, regions, accounts, or upstream computing clusters, and select the calling channel based on model capabilities, response latency, price, or historical success rate. As model types, request complexity, streaming output methods, input and output data volumes, and real-time loads continuously change, a single static weight or independent scoring metric is insufficient to reflect the feasibility of completing the current request within the given time and cost constraints.

[0003] In existing solutions, channel scoring, circuit breaker determination, and backup switching often read states from different sampling windows or different versions, making it difficult to convert queued request volume, in-transit request volume, first response latency, generation throughput, error source, and billing information into comparable boundaries under the same request dimension. The primary channel and backup channel may also share suppliers, regions, account quotas, or upstream clusters. If the shared fault correlation is not explicitly modeled, it is easy to repeatedly select the same source of faulty channel when the state changes abruptly.

[0004] Furthermore, if the time and cost already consumed are not continuously deducted during the standby switchover, and the status version, latest start time, and maximum number of attempts are not reviewed before the switchover, it may lead to amplified queuing and retrying, increased tail response latency, breach of deadline, decreased success rate, and an imbalance between cost and service quality. Summary of the Invention

[0005] One objective of this invention is to propose a multi-model API request routing and scheduling method based on multi-dimensional state feedback. Addressing the challenges of existing technologies that struggle to uniformly evaluate heterogeneous channels with consistent states, where primary and backup channels may share fault sources, and where backup switching lacks joint constraints on time and cost, this invention proposes a technical solution that includes request constraint encoding, hierarchical versioned state boundary transformation, channel-fault domain-primary / backup edge relationship modeling, residual dual-budget primary / backup plan solving, failure source attribution, pre-switch version verification, and atomic feedback of the same incremental state version. This ensures that scoring, fault isolation, and backup switching use consistent evidence and boundaries, reducing the amplifying effect of repeated fault hits and invalid retries on tail latency, completion time, and call costs.

[0006] This invention provides a multi-model API request routing and scheduling method based on multi-dimensional state feedback, including:

[0007] S1. Receive the model application programming interface request and encode the model capability requirements, requested resource amount, completion deadline and cost limit into a request constraint vector;

[0008] S2. Filter compatible channels according to the request constraint vector and read the channel status with version. Convert the channel status into a lower bound of success probability, an upper bound of completion delay, and an upper bound of cost. Construct a candidate state relationship structure that includes channel nodes, fault domain nodes, and primary / backup directed edges.

[0009] S3. Within the remaining time budget and remaining cost budget defined by the request constraint vector, based on the lower bound of the success probability, the upper bound of the completion delay, the upper bound of the cost, and the fault correlation represented by the candidate state relationship structure, jointly select the primary channel and at least one backup channel to generate a routing execution plan that includes the primary / backup order, switching conditions, and plan state version.

[0010] S4. Initiate a channel call according to the routing execution plan. If the call fails, determine the source of failure and deduct the time and cost. Switch to the next backup channel only when the upper bound of the completion delay and the upper bound of the cost of the next backup channel fall within the corresponding remaining budget and the state version before execution is consistent with the state version of the plan.

[0011] S5. Based on the call result and the source of failure when the call fails, the status evidence of this call is atomically projected to the channel node, fault domain node, primary / backup directed edge, candidate qualification and circuit breaker status according to the same incremental status version to form a unified routing status for subsequent requests.

[0012] Optionally, S1 includes: encoding the model identifier, context length, tool invocation capability, multimodal input type, and streaming output capability into model capability requirement fields, and generating a model type field based on the mapping relationship between model identifiers and model types in the preset model capability catalog;

[0013] A request resource quantity field is generated based on the number of input tokens, the amount of media data to be encoded, and the expected number of output tokens. The request resource quantity field includes the amount of input data and the expected amount of output data.

[0014] The request complexity is determined based on the prompt word structure, the number of tool calls, and the expected number of inference stages, and the request complexity is mapped to a preset complexity bucket;

[0015] The initial time budget is the difference between the completion deadline and the request acceptance time, and the initial cost budget is the upper limit of the cost. The request constraint vector is composed of the model capability requirement field, model type field, requested resource amount field, complexity bucket, streaming output requirement, initial time budget, initial cost budget and success probability threshold.

[0016] Optionally, S2 includes: reading channel status records according to the model type field and the complexity bucket, wherein the channel status records include queued request volume, in-transit request volume, first response latency quantile, generation throughput quantile, call result, error source, billing unit price, event time and status version;

[0017] The upper bound of the completion delay is determined based on the expected service start time, the first response delay quantile, the generation throughput quantile, and the expected output data volume.

[0018] The lower bound of the success probability is determined based on the lower endpoint of the confidence intervals for successful and failed invocation evidence;

[0019] The upper limit of the cost is determined based on the billing unit price, the amount of input data, the expected amount of output data, and the retry billing rules;

[0020] Records whose event time is later than the status reading time are excluded. The remaining records are sorted in ascending order by status version number, and the last record in the sorted list is taken as the current channel status record.

[0021] Furthermore, in the candidate state relationship structure, the channel node carries the first response latency quantile, generation throughput quantile, successful call evidence, failed call evidence, queued request volume, in-transit request volume, cost range, and state version of the corresponding channel.

[0022] The fault domain nodes represent suppliers, regions, account quotas, or upstream clusters, and carry shared anomaly evidence and status versions for the corresponding fault domains.

[0023] The starting point of the primary-backup directed edge is the primary channel node or the previous backup channel node, and the ending point is the next backup channel node. It carries the conditional common failure probability, the switching latency overhead measured in time, the switching cost overhead measured in billing amount, and the edge state version. The edge state version is the version field of the primary-backup directed edge in the candidate state relation structure, so as to determine the shared fault domain penalty and backup order in step S3.

[0024] Optionally, S3 includes: obtaining the remaining time budget by the difference between the completion deadline and the current time, and obtaining the remaining cost budget by subtracting the cost already consumed from the cost ceiling;

[0025] The shared fault domain penalty is determined based on the fault domain nodes jointly connected to the channel nodes at both ends of the primary and backup directed edges and the common failure probability of the conditions described above.

[0026] Using the lower bound of the success probability of the main channel and each backup channel, the upper bound of the completion delay, the upper bound of the cost, the shared fault domain penalty, the switching delay overhead, and the switching cost overhead as inputs for the plan solution, a sequence of channels to be inspected, including the main channel and all candidate backup channels, is formed according to the execution order from the main channel to the backup channel.

[0027] Set the switching delay overhead and switching cost overhead corresponding to the first main channel in the channel sequence to be inspected to zero. Starting from the second sequence position of the channel sequence to be inspected, the main and backup directed edge from the channel in the previous sequence position to the channel in the current sequence position is taken as the corresponding main and backup directed edge of the current sequence position.

[0028] The upper bound of the completion delay of each channel in the channel sequence to be inspected is added to the switching delay overhead corresponding to the current sequence position to obtain the upper bound of the cumulative delay of each sequence position. The upper bound of the cost of each channel is added to the switching cost overhead corresponding to the current sequence position to obtain the upper bound of the cumulative cost of each sequence position. This is used by step S3 to determine the executable prefix subject to dual budget constraints in the channel sequence to be inspected.

[0029] Furthermore, it also includes: for each backup channel, subtracting the upper bound of the completion delay of the backup channel and the switching delay overhead from the previous channel to the backup channel from the completion deadline to obtain the latest start time of the backup channel;

[0030] Starting from the first sequence position of the channel to be inspected, the upper bound of the cumulative delay and the upper bound of the cumulative cost of each sequence position are compared with the corresponding remaining budget. Sequence positions that continuously satisfy the condition that the upper bound of the cumulative delay does not exceed the remaining time budget and the upper bound of the cumulative cost does not exceed the remaining cost budget are formed into an executable prefix. The number of channels included in the executable prefix is ​​determined as the maximum number of attempts. The executable prefix includes all channels in the channel to be inspected when all sequence positions satisfy the dual budget.

[0031] Each time a channel call is initiated, the current attempt count is incremented by one; for the main channel call, this is the first attempt.

[0032] The following conditions are set as executable conditions for the corresponding primary and backup directed edges: the lower bound of the success probability of the backup channel reaches the success probability threshold, the current time is not later than the corresponding latest start time, the sequence position of the backup channel belongs to the executable prefix, and the edge state version of the primary and backup directed edges is equal to the plan state version.

[0033] Optionally, S4 includes: determining the source of failure includes: generating evidence of channel failure and evidence of shared failure domain failure based on response code, domain name resolution phase, connection establishment phase, transport layer security handshake phase, request sending phase, the location of the first stream data arrival, the location of stream interruption, gateway observation results, and abnormal records of other channels within the same time window.

[0034] When multiple channels sharing the same supplier, region, account quota, or upstream cluster experience the same connection stage or response code category failure within the same time window, the corresponding failure will be assigned to the corresponding fault domain node; otherwise, the corresponding failure will be assigned to the channel node where the call occurred.

[0035] Furthermore, before switching to the next backup channel, the status version of the backup channel node, the fault domain node connected to the backup channel node, and the edge status version of the corresponding primary and backup directed edge are reread.

[0036] The remaining time budget is recalculated based on the difference between the completion deadline and the reread completion time, and the costs incurred during the switching decision period are deducted from the remaining cost budget before the switching decision to obtain the updated remaining cost budget.

[0037] A switchover is initiated only if the reread channel node and fault domain node status versions and the edge status versions are all equal to the planned status version, the current number of attempts indicating the number of channel calls already initiated is less than the maximum number of attempts, and the executable condition still holds under the recalculated remaining time budget and the updated remaining cost budget. Otherwise, the routing execution plan is terminated and at least one of the following termination reasons is output: insufficient budget, version mismatch, attempt limit reached, or channel unavailable.

[0038] Optionally, S5 includes: reading the currently published status version before submitting the status evidence for this channel call, and allocating a target status version that is simultaneously greater than the corresponding planned status version and the currently published status version to the status evidence;

[0039] The channel's own fault evidence, shared fault domain fault evidence, call success evidence, changes in queued request volume, changes in in-transit request volume, measured first response latency, measured generation throughput, and measured cost are compiled into a versioned update record.

[0040] The versioned update record is projected to the corresponding channel node, fault domain node, candidate qualification, circuit breaker status and related primary and backup directed edges through an atomic commit. The edge state version of the related primary and backup directed edges is written as the target state version. The target state version is only published when the current published state version at the time of commit is still equal to the current published state version read when the target state version is allocated, so that the reader only uses the projected object that has been published and has the same version.

[0041] If the current published state version changes at the time of submission, the current published state version is reread and the target state version is reassigned.

[0042] Furthermore, a data plane consistency probe is executed on the channel in the circuit breaker state. The data plane consistency probe adopts the same domain name resolution, connection establishment, authentication, request transmission and response reception path as the model application interface request.

[0043] When the number of consecutive failures reaches the activation threshold, the circuit breaker is maintained in the circuit-breaker state; when the number of consecutive successes reaches the recovery threshold which is greater than the activation threshold, the circuit breaker state is released.

[0044] After the circuit breaker is lifted, the allowed concurrent call quota is increased step by step according to the preset incremental ladder. If a call fails at any level, the allowed concurrent call quota is rolled back to the previous level or the circuit breaker is re-entered. The probe results and quota level are updated to the corresponding channel node, fault domain node and primary / backup directed edge according to the target state version in step S5, and the edge state version of the primary / backup directed edge is written as the target state version.

[0045] The beneficial effects of this invention are:

[0046] 1. By converting the number of queued requests, the number of requests in transit, the first response latency quantile, the generation throughput quantile, the invocation evidence, and the billing information into the lower bound of the success probability of the current request, the upper bound of the completion latency, and the upper bound of the cost, different channels can be compared uniformly under the constraints of request capacity, complexity, completion time limit, and cost.

[0047] 2. By using channel nodes, fault domain nodes, and primary / backup directed edges to carry shared abnormal evidence, common failure probability, switching latency overhead, switching cost overhead, and edge state version, and incorporating shared fault domain penalties into the primary / backup joint selection, the duplicate selection of primary / backup channels based on shared suppliers, regions, account quotas, or upstream clusters can be reduced.

[0048] 3. By using executable prefixes, latest start time, maximum number of attempts, failure attribution, dual budget and version review before switching, and atomic feedback of the same incremental version, retries exceeding the remaining time or cost budget are limited, and subsequent scoring, circuit breaking, and standby switching reads are kept consistent. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 This is a flowchart of the multi-model API request routing and scheduling method based on multi-dimensional state feedback according to the present invention.

[0051] Figure 2 A flowchart for the S4 routing execution and handover control method. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0053] refer to Figures 1-2 A multi-model API request routing and scheduling method based on multi-dimensional state feedback includes:

[0054] S1. Receive the model application programming interface request and encode the model capability requirements, requested resource amount, completion deadline and cost limit into a request constraint vector;

[0055] S2. Filter compatible channels according to the request constraint vector and read the channel status with version. Convert the channel status into a lower bound of success probability, an upper bound of completion delay, and an upper bound of cost. Construct a candidate state relationship structure that includes channel nodes, fault domain nodes, and primary / backup directed edges.

[0056] S3. Within the remaining time budget and remaining cost budget defined by the request constraint vector, based on the lower bound of the success probability, the upper bound of the completion delay, the upper bound of the cost, and the fault correlation represented by the candidate state relationship structure, jointly select the primary channel and at least one backup channel to generate a routing execution plan that includes the primary / backup order, switching conditions, and plan state version.

[0057] S4. Initiate a channel call according to the routing execution plan. If the call fails, determine the source of failure and deduct the time and cost. Switch to the next backup channel only when the upper bound of the completion delay and the upper bound of the cost of the next backup channel fall within the corresponding remaining budget and the state version before execution is consistent with the state version of the plan.

[0058] S5. Based on the call result and the source of failure when the call fails, the status evidence of this call is atomically projected to the channel node, fault domain node, primary / backup directed edge, candidate qualification and circuit breaker status according to the same incremental status version to form a unified routing status for subsequent requests.

[0059] In this specific embodiment, S1 includes:

[0060] In this specific embodiment, the routing scheduling platform receives a multi-model API request with the request identifier req-20260812-0001. This request requests to call a dialogue model with image understanding, function tool calling and streaming text output capabilities. The request header carries the request identifier field, model identifier field, completion deadline field, cost limit microelement field, stream and tenant identifier fields. The request body carries the message array, the byte length of a JPEG image, two tool definitions and the expected maximum number of output tokens. When receiving the request, the access gateway simultaneously records the Unix millisecond reception time and the monotonic clock reception time, establishes a fixed conversion benchmark for this request based on the binary time, and associates the SHA-256 digest of the original message with the request identifier and stores it in the request registration table.

[0061] The encoder reads a record with the model identifier as the key from the 17th version of the model capability catalog. This record contains fields for model type, maximum context token, tool call allowed, receivable media type, streaming output allowed, effective time, and catalog version. The catalog is built by channel access testing and vendor capability declarations and is updated daily or when the model version changes. The model type of this record is vision-tool-chat, the maximum context length is 128,000 tokens, and image / jpeg input, tool calls, and streaming output are allowed. If the model identifier is not matched or the catalog effective time is later than the request reception time, the encoder does not infer the model type, but writes the request to the capability rejection queue and returns a structured reason why the capability catalog is unavailable.

[0062] The encoder calculates the number of input tokens for the message array according to the segmentation rules of the target model, converts the JPEG binary length into the amount of media data to be encoded after aligning it to 4 bytes, uses the maximum output token field as the expected number of output tokens, and writes the input token number field, media byte number field, expected output token number field, and payload byte number field into the requested resource amount field. In this case, they are set to 3200 tokens, 1887436 bytes, 1200 tokens, and 1904620 bytes, respectively. If the MIME type of the decoded image is inconsistent with the declaration or the total context exceeds the directory limit, the encoding is terminated before entering the channel filtering and the reason for the invalid resource is recorded.

[0063] The request complexity is jointly determined by the prompt word structure, the number of tool calls, and the expected number of inference stages. The specific calculation of the complexity score is as follows. ,symbol Represents the dimensionless request complexity score, with values ​​ranging from 0 to 1, and the symbol... This represents a dimensionless function that truncates the input to between 0 and 1. (symbol) This represents the number of prompt word structure nodes counted by the syntax parser, symbol [symbol]. This represents the normalized upper bound of the structure nodes in the bucket table for versioned complexity, with the current version taking 64. (Symbol) Indicates the tool definition number, symbol This indicates the normalized upper limit of the number of tools, which is 8 in this version. (symbol) The symbol represents the expected number of inference stages inferred from the branch constraints and toolchain in the prompt words. This represents the upper limit of normalization during the reasoning stage, and in this version it is 10. All three normalization terms are dimensionless, and the coefficients 0.4, 0.3 and 0.3 are fixed in the complexity bucket table of version 6 and their sum is 1.

[0064] The complexity bucketing table is labeled by the percentile of the processing time and the number of tokens generated for requests completed in the past 30 days. The fields include model type, lower bound of score, upper bound of score, bucket identifier, version, and effective time. Values ​​not less than 0 and not more than 0.30 are mapped to low complexity buckets, values ​​greater than 0.30 and not more than 0.65 are mapped to medium complexity buckets, and values ​​greater than 0.65 and not more than 1 are mapped to high complexity buckets. The internal boundaries are all taken as the range contained within. The table is looked up based on the latest effective version before the request receiving time. The complexity score for this request is 0.71, so it is written to the high complexity bucket. If the normalized upper bound is missing, the conservative high complexity bucket of the model type is used and a bucket rollback flag is written.

[0065] To clearly map the request complexity to the preset complexity bucket, the encoder uses the lower bound and upper bound of the score to perform a half-open and closed interval search for the current request complexity. Each bucket record also carries the model type, number of samples and effective time. After a hit, the bucket identifier and bucket version are written to the request constraint vector at the same time so that S2 can use the same bucket to read the channel status.

[0066] The budget encoder reads the absolute Unix millisecond deadline from the completion deadline field in the request header. Using the Unix millisecond reception time and monotonic clock reception time synchronously collected when receiving the request, it converts the deadline to a completion deadline under the same monotonic clock reference. Then, it subtracts the monotonic clock reception time from the converted completion deadline to obtain the initial time budget, which is 18,000 ms in this case. The conversion reference is fixed throughout the lifecycle of this request and is not affected by subsequent wall clock calibration. If the request header is missing a deadline, the deadline is earlier than the reception time, or the conversion result is not greater than zero, the invalid deadline reason is written and the request is terminated. The user-set 0.120000 yuan is used as the initial cost budget and internally converted to an integer of 120,000 micro-yuan to avoid floating-point cumulative deviation. The success probability threshold is 0.985, which is obtained from the 9th version of the tenant service level table by tenant identifier field and model type. This table is published by the tenant service level configuration, with a value range of 0 to 1 and is updated when the SLA changes. When no tenant is matched, the system conservative value of 0.990 is used and the service level default flag is marked.

[0067] The encoder ultimately forms a request constraint vector using ordered key-value records. The record fields are: request identifier, model capability requirement, model type, input token, media byte, expected output token, complexity bucket, streaming output requirement, completion deadline, initial time budget, cost ceiling, initial cost budget, success probability threshold, catalog version, and bucketed version. The encoder also calculates a constraint summary field from the serialized results sorted by field name. This request constraint vector is written to the request constraint storage area and serves as the unique filtering input for S2, allowing subsequent channel boundaries, routing plans, and execution feedback to be verified using the same request identifier and constraint summary.

[0068] In this specific embodiment, S2 includes:

[0069] In S2, the channel filter reads the model type, context length, multimodal type, tool capability, streaming requirements, and complexity bucket from the request constraint vector written in S1. Using the model type field and complexity bucket field as indexes, it generates an initial channel set from the version 23 channel capability table. Then, it retains compatible channels based on the predicates that the maximum context token field is not less than the sum of the input and expected output tokens, the receivable media type field includes image / jpeg, and the tool call allow field and streaming output allow field are both true. This time, six channels belonging to three suppliers, two regions, and four account quotas were obtained. If no compatible channels were generated, the no-compatible-channel-reason termination record was directly written without entering the plan solver.

[0070] The capability filter establishes a compatibility determination record for each initial channel. The fields include channel identifier, model type matching result, context margin, media type hit set, tool and streaming capability bitmap, and deletion reason. The sum of all Boolean conditions is used as a compatibility flag, so that the generation, deletion and output of the candidate set can be reproduced from the same record.

[0071] The status reader reads the channel status log for each compatible channel according to the channel identifier field, model type field, and high complexity bucket complexity bucket. The recorded fields include queued request volume, in-transit request volume, 95th quantile of first response latency, 10th quantile of generation throughput, call result field, error source field, unit price per thousand input tokens, unit price per thousand output tokens, unit price per megabyte of media, event millisecond time field, and status version field. Only records with event times no later than the current status read time are retained. The records are sorted in ascending order by unsigned 64-bit status version number, and the last digit is taken as the current channel status. If concurrent writing causes duplicate versions, the record with the published flag set to true and the event digest passing the verification is used. Channels with no available records are marked with a status missing flag and removed from the candidate set.

[0072] The upper bound of completion delay is calculated based on the current queue and in-transit counts of the channel and the historical conservative percentile. ,symbol Channel index The upper bound of the completion delay for this request is in milliseconds, and the symbol is... Indicates the compatibility channel number, symbol This represents the estimated latency, expressed in milliseconds, based on the number of queued requests, the number of requests in transit, and the service rate over the past 60 seconds. The symbol is... This represents the 95th quantile of the first response latency under the same model type and complexity bucket, in milliseconds. This represents the expected number of output tokens in the resource quantity requested by S1, expressed in tokens. The 10th percentile of the channel's generation throughput is represented in tokens / s. The constant 1000 represents the coefficient for converting seconds to milliseconds in ms / s. When the throughput value is not positive or the state age exceeds 30 seconds, the channel is marked as having no upper limit on latency.

[0073] The status reader counts the number of successes and failures from the last 500 completed calls of the same channel, model type, and complexity bucket, and calculates the lower bound of the success probability using the posterior lower quantile of the beta distribution prior. ,symbol Channel index The probability of success is a lower bound and a dimensionless value between 0 and 1, with the sign... The dimensionless quantile function representing the 5th percentile of the probability distribution is denoted by the symbol . Represents the beta probability distribution, symbol This indicates the number of pieces of evidence successfully retrieved within the window, and these numbers are non-negative integers. This represents the number of failed evidence calls within the window, and is a non-negative integer. Indicates the compatibility channel number, symbol This represents a dimensionless pseudo-count added to the success side and the failure side respectively. In this embodiment, the uniform beta prior specified in the 5th edition of the statistical boundary configuration table is adopted and both pseudo-counts are fixed at 1. When there are fewer than 20 valid pieces of evidence, the minimum value between the global conservative lower bound of the model type to which the channel belongs and the posterior lower quantile is taken.

[0074] The cost calculator bills on a micro-integer basis, rounding up the input token, expected output token, and media data to the billing granularity to calculate the upper bound of the cost. ,symbol Channel index The upper bound of the cost is in micro-element units, and the symbol is... Indicates the rounding up operation, sign This indicates the number of input tokens requested, expressed in tokens, with the symbol... Indicates the expected number of output tokens in the request, in units of tokens, symbol. Indicates the amount of media data to be encoded, in bytes, symbol This represents the price per thousand input tokens, expressed in micro-dollars per ktoken. (symbol) This represents the price per thousand output tokens, expressed in micro-dollars / ktoken, with the symbol... This indicates the price per megabyte of media, expressed in micro-dollars per MB (µA / MB). This represents the additional cost incurred under the worst-case scenario by the retry billing rule, expressed in micro-units. Indicates the compatibility channel number, symbol This represents the number of tokens contained in each ktoken and is used to convert the token count to a kilotoken billing granularity. (symbol) This indicates the number of bytes contained in each MB;

[0075] After the three types of conservative boundary calculations are completed, the channel state converter writes the original state version, calculation input summary, success probability lower bound, completion delay upper bound, and cost upper bound into the immutable boundary record. If any input field is missing, the unit is not convertible, or the quantile is invalid, the comparability flag of the record is set to false and the corresponding channel node is not created.

[0076] The candidate state relationship structure is stored as a versioned attribute directed graph. The channel node uses the channel identifier field as the primary key and carries the first response latency quantile, generation throughput quantile, success and failure evidence count, queue count, in-transit count, cost range, three conservative boundaries, and state version. The fault domain node uses the fault domain type field and the fault domain identifier field to form a composite key, which represents the supplier, region, account quota, or upstream cluster, respectively, and carries the shared anomaly evidence window, condition failure count, and state version. Channels and fault domains are associated edge by edge through membership relationship.

[0077] In the candidate graph, a primary and backup directed edge is generated for any two different channels. The starting point of the edge is a node that can act as the primary channel or the previous backup channel, and the ending point is the next backup channel node. The edge is retained only when the lower bound of the success probability of the ending point reaches the S1 threshold, the capability signatures of the two nodes are compatible, and the ending point is not in the circuit breaker enabled state. The edge fields include the starting channel field, the ending channel field, the conditional common failure probability field, the switching delay field, the switching cost field, the shared fault domain identifier set field, and the edge state version field. The switching delay is derived from the 95th quantile of the same authentication and connection reuse process in the past 7 days, and the switching cost is derived from the retry billing and connection establishment billing rules. When there are no direct historical edge samples, the conservative upper limit of the fault domain to which the ending channel belongs is used.

[0078] The construction of primary and backup directed edges also performs deduplication rules. For edges with the same start point, end point, and graph snapshot version, only the record with the newest event time and more evidence samples is retained. If they are still the same, the edge summary lexicographical order is used to decide. The deduplicated edge fields and the deletion reason are written to the graph construction log.

[0079] The graph builder stores the candidate graph and the state version field of each node and edge in the candidate graph storage area. It uses the largest published version seen in this state read as the graph snapshot version field and calculates the graph summary field for the graph. For nodes with no upper bound on latency, exceeded cost limit, missing state, or lower bound on success probability below the threshold, it writes the candidate qualification field = false and the specific reason for deletion. The final output candidate state relationship structure is only passed to S3 if it contains at least one available channel node; otherwise, the request is terminated and all boundary infeasibility evidence is retained.

[0080] In this specific embodiment, S3 includes:

[0081] In S3, the plan solver takes the request constraint vector of S1 and the candidate state relationship structure of S2 as input, reads the completion deadline, cost limit, time elapsed, cost elapsed, the three conservative boundaries of the channel node, and the primary and backup directed edge attributes, subtracts the current time of the monotonic clock from the deadline to obtain the remaining time budget, and subtracts the cost limit from the consumed infinitesimal integer to obtain the remaining cost budget. When any budget is not greater than zero, a budget exhaustion reason termination record is generated, and the selection of the primary channel is no longer attempted.

[0082] For each primary-backup directed edge, the solver takes the supplier, region, account quota, and upstream cluster fault domain nodes that are commonly connected by the channel nodes at both ends of the edge, and calculates the shared fault domain penalty. ,symbol Indicates from channel index Switch to Channel Index The dimensionless shared fault domain penalty has a value range of 0 to 1, and the symbol is... Indicates the channel number of the starting point of the directed edge, symbol Indicates the channel number of the destination edge, symbol Represents the set of fault domain nodes connected by both ends of the channel, symbol This represents the fault domain index in the set, symbol Indicates the index in the fault domain Channel Index under Conditions Channel Index The probability of common failure is a dimensionless value between 0 and 1. When the set is empty, the product is treated as 1 so that the penalty is 0.

[0083] The conditional co-failure probability comes from the 12th edition of the co-failure probability table. The fields include the starting channel field, the ending channel field, the fault domain type field, the fault domain identifier field, the co-failure count field, the starting failure count field, the probability upper limit field, the window start field, the window end field, and the version. This table is labeled with co-cause failure evidence within the same five-second window over the past 30 days. When there are fewer than 20 valid co-occurrence samples, the conservative probability upper limit of the fault domain is used. In an example record, the starting channel field = A-cn1, the ending channel field = B-cn2, the fault domain type field = account quota type, the probability upper limit field = 0.18, and the version = 12. The table is updated when the corresponding time window ends or the fault domain membership changes. When the table lookup fails, the global upper limit of the fault domain type is used as a penalty.

[0084] The solver first enumerates each available channel as the primary channel according to the lexicographical order of the channel identifier. Then, it generates a backup order by expanding position by position: after the previous channel has been determined, the next channel is selected only from the channels that have not been used and have corresponding primary and backup directed edges. The channel is then sorted by the following order: descending order of success probability, ascending order of completion delay, ascending order of cost, ascending order of shared fault domain penalty relative to the determined previous channel, and lexicographical order of the channel identifier to form a stable composite key. The first and second positions of the composite key are fixed each time before expanding to the next position. If the endpoint is in a circuit breaker state or the edge state version is inconsistent with the graph snapshot version, the action is deleted. Therefore, the same candidate graph will always get the same set of test sequences.

[0085] For a sorted sequence of channels to be inspected, the switching delay and switching cost of the first main channel are both set to zero. Starting from the second channel, the edge cost from the previous channel to the current channel is used, and the calculation is performed. Upper bound of cumulative delay and upper bound of cumulative cost for each sequence position ,symbol Indicates the position of the channel sequence to be inspected, and its value ranges from 1 to the length of the channel sequence to be inspected. ,symbol The total number of channels contained in the channel sequence to be inspected is a positive integer, and the symbol is [symbol missing]. Indicates as of the date The upper bound of the cumulative delay of bits and the unit is ms, symbol Indicates as of the date The upper bound of the cumulative cost of a unit is given, and the unit is micro-element, with the symbol... The symbol represents the sequence position index used for summation. Indicates the first The upper bound of the completion delay of the bit channel is in milliseconds, and the symbol is... Indicates entering the first The switching latency overhead before the channel is expressed in milliseconds, and the symbol is... Indicates the first The upper bound of the cost of a channel is expressed in micro-units, with the symbol [symbol missing]. Indicates entering the first The switching costs before the channel are expressed in micro-yuan;

[0086] For each backup channel, the solver establishes a candidate action record, which includes the previous channel, the current backup channel, the corresponding primary and backup directed edge, the lower bound of the success probability, the upper bound of the completion delay, the upper bound of the cost, the shared fault domain penalty, and the edge version. Only when the action record is complete is the channel sequence to be inspected allowed to be accessed.

[0087] The solver checks each position in the sequence to be checked, starting from the first position, to determine whether the upper bound of the cumulative delay does not exceed the remaining time budget and the upper bound of the cumulative cost does not exceed the remaining cost budget. When the first position that does not meet the requirements is encountered, the expansion is terminated. The previously consecutive positions that meet the requirements are formed into an executable prefix. The number of channels contained in the prefix is ​​written as the maximum number of attempts field. When all positions are met, the prefix contains the entire sequence. If even the first position does not meet the requirements, the sequence is discarded. When the main channel is actually called, the current number of attempts field is increased from 0 to 1. Each subsequent call to a backup channel is incremented by 1.

[0088] The determination of executable prefixes explicitly adopts the continuity rule, that is, starting from the first sequence position, only the sequence positions that continuously satisfy the upper bound of the cumulative delay not exceeding the remaining time budget and the upper bound of the cumulative cost not exceeding the remaining cost budget are formed into executable prefixes. If any intermediate position fails, the position is not skipped and the subsequent positions are used.

[0089] The state vector of the planned solution process includes the current sequence to be checked, the current sequence position, the upper bound of the cumulative delay, the upper bound of the cumulative cost, the executable prefix length, and the best verified sequence. The candidate actions are to access the next backup channel or to end the current sequence. After each action is completed, the dual budget and version constraints are immediately verified to avoid unverified sequences from entering the ranking.

[0090] For each backup channel in the executable prefix, the planner subtracts the upper bound of the channel's completion delay and the switching delay overhead of the previous channel switching into this channel from the completion deadline to obtain the latest start time field in absolute milliseconds. The conjunctive predicates that the success probability lower bound reaches the S1 success probability threshold, the current time is not later than the latest start time field, and the current sequence position belongs to the executable prefix and the edge state version field is equal to the planned state version are written as the executable condition field of the edge, where the planned state version is taken from the graph snapshot version field of S2.

[0091] For length of Each executable sequence The solver calculates the conservative combination success score. ,symbol Represents a sequence The dimensionless conservative combination success score, ranging from 0 to 1, is used only as a deterministic ranking measure and is not declared as a lower bound on the true combination success probability. The symbol is... This represents an ordered simple sequence consisting of a main channel and sequentially reserved backup channels, symbolized by... The length of the sequence is a positive integer, and the sign is... Indicates the current sequence position index and its value ranges from 1 to... ,symbol This represents the sequence position index before reaching the current position, with a value ranging from 1 to... ,symbol This indicates the sequence number read from the immutable boundary record of S2. The probability of success for a given channel is a dimensionless value between 0 and 1, with the subscript... Indicates the sequence number The corresponding channel identifier, symbol This indicates the calculation of the shared fault domain penalty formula from the first... Switching to the next channel The dimensionless penalty of the position channel takes a value between 0 and 1, and the symbol is... This indicates taking the maximum value among the listed real numbers, with the sign... The non-negative cutoff boundary of the score is represented by the symbol. The system represents a dimensionless, fully probabilistic baseline. The product terms form independent failure baselines based on the lower bounds of the success probabilities of each channel. The summation terms are weighted by the product of failures reaching the corresponding backup positions, minus the shared failure risk of adjacent edges. This structure is used to generate a repeatable conservative ranking of the same version inputs, rather than to infer unmodeled higher-order joint probabilities. The higher the score, the higher the priority, and the tie-breaking rules are determined by the smallest upper bound of cumulative delay, the smallest upper bound of cumulative cost, and the lexicographical order of the channel identifier array. When the plan takes longer than 20 ms, only the highest-ranked sequence that has passed both budget and version checks is returned.

[0092] The solver records a feasibility certificate for the sequence that passes the dual-budget verification. This certificate includes the remaining time budget, remaining cost budget, cumulative boundary for each sequence position, latest start time, source of shared fault domain penalty, and version set. The feasibility certificate is written to the routing execution plan at the same time, and S4 can use this to verify the source of the switching conditions item by item.

[0093] The final route execution plan is written to the route plan storage area. Fields include: request identifier, constraint summary, ordered channel identifier array, edge identifier array, latest start time array, executable condition array, cumulative delay upper bound array, cumulative cost upper bound array, maximum number of attempts, current number of attempts, plan status version, graph summary, plan creation time, and plan failure time. Before writing, the sequence is re-verified that the first element is the primary channel, each subsequent element contains at least one backup channel, and all edge versions are equal to the plan status version. If these conditions are not met, S2 is reread a maximum of three times at 50 ms, 100 ms, and 200 ms intervals. Solving continues only when a global published version change is detected, and the remaining time and cost budgets are both positive. If the reread limit, time limit, or cost limit is reached, or if no verified sequence is found within the 20 ms solution period, the reason for no feasible primary / backup plan and the specific reasons for edge missing, version mismatch, circuit breaker, insufficient budget, or solution timeout are output, and the request is terminated, without proceeding to S4. The verified plan serves as the sole source of S4's call order and switching conditions.

[0094] In this specific embodiment, S4 includes:

[0095] In S4, the executor reads the routing execution plan from S3 using the request identifier field. After verifying that the constraint digest field is consistent with the record in S1 and that the plan has not exceeded the plan expiration time field, it constructs a call context for the main channel that includes the request identifier field, the attempt sequence number field = 1, the plan status version field, and the idempotent key field. The idempotent key field is obtained by calculating a digest from the request identifier, the number of attempts, and the channel identifier. The executor records the start and end of DNS resolution, TCP connection establishment, TLS handshake, request sending, arrival of the first streaming data, the end position of the stream, and the response code in sequence. When the call is initiated, the current number of attempts field is atomically incremented and written to the attempt log.

[0096] When each call ends or is interrupted, the executor subtracts the start time from the end time of the monotonic clock to obtain the milliseconds consumed. Based on the number of output tokens received, the amount of input data sent, the media data, and the actual billing receipt, the cost consumed is obtained as a micro-element integer. The two are added to the milliseconds consumed and micro-element consumed fields in the execution budget record. If no billing receipt is received, it is temporarily recorded according to the upper limit of the retry billing rule of S2 and marked with a cost estimation flag. When a subsequent receipt arrives, the temporary value is replaced with the measured value under the same attempt identifier field.

[0097] The failure source determination in this step does not use an unexplained learning model, but instead uses the version 14 failure attribution rule table to perform deterministic matching. The rule table fields include connection stage, response code category, gateway observation type, co-occurrence channel lower limit, time window, and attribution object type. The table is labeled with historical failure records from gateway injection failure tests and manual review. The rule version and the hit rule identifier are written together into the failure evidence record.

[0098] The failure attributor categorizes response codes into authentication, quota, server, rate limiting, and request semantic categories. It also compiles failure evidence records by including fields from the domain name resolution phase, connection phase, transport layer security handshake phase, request sending phase, first stream block phase, stream interruption location, gateway observation, and abnormal records from other channels within 2.5 seconds before and after the failure. If a failure occurs during client parameter validation and the gateway does not send an upstream request, it is written as the reason for request semantic failure and the backup switch is terminated to prevent the same invalid request from being sent to multiple channels.

[0099] For a call with a response code of 429, if at least two channels under the same account quota simultaneously produce a rate-limiting response within a five-second window and the gateway observes that the remaining quota is zero, then the shared account quota fault rule is hit. If only the current channel produces the response and other channels in the same domain succeed, then the channel's own rate-limiting rule is hit. The two types of outputs are written to the fault domain evidence or the channel evidence, respectively.

[0100] For switchable failures, the attributor performs rule matching based on the failure stage field, response code category field, fault domain membership field, and a five-second co-occurrence window: when at least two channels sharing the same supplier, region, account quota, or upstream cluster experience the same connection stage or response code category failure within the same window, shared fault domain evidence with the shared fault domain fault field = true is generated and projected according to the matched specific fault domain identifier field; otherwise, channel-specific fault evidence with the channel's own fault field = true is generated. When multiple fault domains are satisfied simultaneously, all fault domain identifier fields and their respective co-occurrence sample summaries are retained.

[0101] Before any standby switchover, the executor initiates a snapshot read on the next channel node, all fault domain nodes connected to it, and the primary / standby edge pointing from the current channel to the next channel. It reads the channel status version field, fault domain status version set field, edge status version field, candidate qualification field, and circuit breaker status field, and requires that all these objects be marked with published=true. If a version jump occurs during the snapshot read, the old and new objects are not concatenated, but are reread. If they are still inconsistent, the version mismatch reason and termination reason are output.

[0102] After the snapshot is read, the remaining budget for the switching decision is recalculated. ,symbol Indicates the remaining time budget after rereading, in milliseconds (ms), symbol Represents the updated remaining cost budget in micro-units, with the symbol... This indicates the completion deadline of S1 converted to the fixed monotonic clock reference of this request, with the unit being ms. (symbol) This indicates the completion time of the current reread under the same monotonic clock reference, with the unit being milliseconds (ms). The symbol is... This represents the upper limit of cost in S1, expressed in micro-element units, with the symbol... This indicates that the cost has been accumulated before the switch judgment and the unit is micro-element, symbol This indicates the cost of snapshot rereading and the additional cost incurred after the snapshot read begins, expressed in micro-units. All time quantities use the same monotonic clock conversion base. or If the value is not greater than 0, the actuator will not perform a loose truncation on negative values, but will directly write the reason for insufficient budget and terminate the standby switch.

[0103] The executor initiates the next call only when the completion delay of the next channel plus the corresponding switching delay cost does not exceed the remaining time budget after rereading, the cost upper bound plus the switching cost cost does not exceed the remaining cost budget after updating, the channel node version being reread, the versions of all related fault domain nodes and edge status are equal to the planned status version field, the current number of attempts field is less than the maximum number of attempts field, the candidate qualification field is true, the circuit breaker status field is not open, and the executable condition field written by S3 is still true under the new budget. In other cases, no condition is relaxed.

[0104] To ensure that a switchover decision initiates only one upstream call, the executor establishes a unique index for the request identifier, plan status version, and attempt sequence number. If the replayed switchover command after a network timeout hits an existing index, the in-process or final state record of the original attempt is returned, the budget is not deducted again, and the idempotent result is synchronously written to the execution result record of S5.

[0105] For streaming output failures, the executor uses whether the first response byte has been submitted to the client as the branching condition: if no response bytes have been submitted, it can switch according to the aforementioned budget, version, and candidate conditions; if a partial response has been submitted, and both the current channel and the backup channel provide resume cursors verified by the channel capability table, it records the SHA-256 digest of the submitted prefix, the last acknowledgment cursor, and the output sequence number, requiring the backup response to carry the same prefix digest and resume from the next sequence number, discarding duplicate sequence numbers, and terminating the splicing of missing or out-of-order sequence numbers; if a partial response has been submitted but there is no verifiable resume capability, or if the prefix digest, cursor, or sequence number verification fails, it does not switch to the backup channel generated from scratch, but closes the stream and outputs the partial response failure status, writing the number of submitted bytes, the last cursor, the termination reason, and the prefix digest into the S5 execution result record;

[0106] Before each upstream call, the executor also checks whether the model identifier, media digest, tool definition digest, and streaming flag of the actual sent message are consistent with the constraint digest of S1. If the retry adapter changes the business semantic field, the alternative call is not sent and the termination reason for the request semantic inconsistency is output. Only the authentication header, supplier path, and idempotent key are allowed to be changed according to the channel adaptation rules.

[0107] In this specific call, the main channel A-cn1 returned a rate-limited response 1800 ms after the request was sent. Within the same five-second window, the executor found that A-cn3 with the same account quota also had a rate-limited response. Therefore, the failure was classified as an account quota fault domain. After rereading, the remaining time budget was 15840 ms and the remaining cost budget was 104300 microyuan. The upper limit of the completion delay plus the switching overhead of the backup channel B-cn2 was 7200 ms, and the upper limit of the cost plus the switching overhead was 46800 microyuan. Moreover, the node, fault domain, and edge version were all equal to the planned version. The current number of attempts was 1 and less than the maximum number of attempts of 3. Therefore, the executor initiated the B-cn2 call. If any of the above data exceeded the budget or the version was inconsistent, the switch was not executed and the corresponding termination reason was written.

[0108] When the switchover is successful, the executor writes the next channel identifier, current number of attempts, reread version set, updated dual budget, fault evidence summary, and switchover time into the switchover decision log. Then, it uses the new attempt sequence number field and idempotent key field to call the backup channel. When the switchover fails, the routing execution plan is terminated and at least one of the following reasons is output according to the actual unmet items: insufficient budget, version mismatch, attempt limit reached, or channel unavailability. The execution result record is composed of successful response, final failure, time and cost of each attempt, channel fault evidence, shared fault domain evidence, and termination reason and is passed to S5.

[0109] In this specific embodiment, S5 includes:

[0110] In S5, the status feedback service receives the execution result record from S4, verifies the request identifier field, attempt identifier field, plan status version field, and evidence digest, and combines the channel's own fault evidence, shared fault domain evidence, successful call evidence, changes in queued request volume, changes in in-transit request volume, measured first response latency, measured generation throughput, measured cost, termination reason, and event time into a versioned update record. Each latency is recorded in milliseconds, throughput in tokens / seconds, and cost in micro-integers. When the same attempt identifier field is submitted repeatedly, the evidence digest is used to remove duplicates.

[0111] The feedback service cleans events when assembling versioned update records, using request identifiers, attempt identifiers, channel identifiers and event summaries to form deduplication keys, truncating negative queued or in-transit requests to zero, marking non-positive generation throughput as invalid and not participating in quantile updates, retaining missing failure sources as evidence to be attributed, and not misprojecting incomplete records to the shared fault domain.

[0112] The status feedback service updates the measured metrics in a fixed window. The first response latency and generation throughput retain the most recent 500 bucket samples of the same model type and complexity. Records of canceled requests, client-initiated disconnection, and missing output token counts are marked with independent flags and are not mixed into normal generation throughput samples. After the window is full, the oldest sample is deleted according to the event time, and the window sample summary is included in the target version.

[0113] Before committing, the feedback service consistently reads the currently published state version and assigns the target state version. ,symbol This represents the target state version of the evidence to be submitted and is an unsigned 64-bit incrementing integer. This function represents the maximum value among multiple integers, with the symbol […]. This indicates the plan state version carried by the S3 routing execution plan and is an unsigned 64-bit integer. This indicates the currently published state version read during allocation, and is an unsigned 64-bit integer. It represents a version increment and is a dimensionless integer constant. If a 64-bit overflow occurs during the calculation, the release will be stopped and the version generation migration process will begin.

[0114] Within the same transaction, the feedback service first updates the success or failure evidence count, queued and in-transit count, actual first response latency, actual generation throughput, and actual test cost of the channel nodes affected by this evidence, based on the channel identifier field. Then, it updates the candidate qualification field and circuit breaker status field of the nodes hitting the fault domain, the relevant channels, and the relevant primary / backup directed edges, based on the fault domain type field and the fault domain identifier field. For channel nodes, fault domain nodes, candidate qualifications, circuit breaker status, and primary / backup directed edges not affected by this evidence, the transaction copies the content of the currently published version read during allocation or writes an immutable inheritance record pointing to that content, so that each object required by the candidate graph has a target state version record. All target version records are kept published=false before commit.

[0115] After completing the full snapshot materialization, the transaction compares the current published version field at the time of commit with the compare-and-swap rule to see if it is still equal to the published version read when the target version was allocated. Only when they are equal and the target version object list covers all channel nodes, fault domain nodes, candidate qualifications, circuit breaker status, and primary / standby directed edges, the published version of the objects in the list is set to true, the state version field of all edges is written to the target state version, and the global published version pointer is moved to the target state version. The reader can use this pointer as the snapshot key to obtain the complete same version projection composed of the content update object and the content inheritance object. Unchanged objects will not remain in the old version due to local evidence projection.

[0116] If compare-and-swap finds that the published version has changed during the commit, the transaction rolls back the temporary projection with published=false, rereads the latest published version and reallocates the target state version according to the aforementioned rules, and then replays the projection with the original evidence digest. It can be retried up to 3 times. In the event of consecutive conflicts, the update record is written to the state commit retry queue and no part of the state is published. It will still be replayed idempotently according to the request identifier and evidence digest later.

[0117] The feedback transaction also maintains a list of projection objects, which is arranged in a fixed order of channel nodes, fault domain nodes, candidate qualifications, circuit breaker status and primary / backup directed edges. Each object records the version before projection, the target version, the evidence digest and the write result. If any object fails to write, the entire transaction is rolled back, and no locally published nodes or edges are left behind.

[0118] After a successful atomic commit, the feedback service generates a release receipt containing the request identifier, target version, set of projected object identifiers, commit time, and transaction summary. This receipt is also written to the append-only audit log and the routing state change message queue. The queue consumer only treats the receipt as a notification of invalid cache and still needs to reread the state with the released version snapshot. The nodes or edges are not overwritten based on the order of message arrival.

[0119] The circuit breaker manager periodically initiates data plane consistency probes to channels in the open state. The probes use the same DNS resolution, TCP connection establishment, TLS handshake, account authentication, request transmission, and streaming response reception path as the real model API requests, but use a dedicated probe request with a minimum input token and a limit of 1 output token. The probe record includes probe identifier field, channel identifier field, path stage time set field, response code field, success, consecutive failure count field, consecutive success count field, and quota level field. The probes do not bypass quotas, authentication, or upstream gateways.

[0120] The circuit breaker hysteresis threshold is stored in the version 8 circuit breaker policy table. The table fields include channel category, activation failure threshold, recovery success threshold, probe interval, quota ladder, version, and effective time. In this specific implementation, the activation threshold is 3 consecutive failures, the recovery threshold is 5 consecutive successes that are strictly greater than the activation threshold, the probe interval is 5000 ms, and the quota ladder is 10%, 25%, 50%, and 100% of the standard concurrency. This policy is calibrated by the fault recovery time in the past 30 days and the impact of the probe request, and is updated when the channel type or SLA changes. When the policy record is not hit, the circuit breaker continues to be activated instead of using the lenient default value.

[0121] After the circuit breaker is released from the recovery threshold after a number of consecutive successful calls, the circuit breaker manager increases the allowed concurrent call quota in increments of 10%, 25%, 50%, and 100% according to preset increments. It will only move to the next level if there are no failures in the 30-second observation window of the current level. The increment, observation window, and current quota level are all written to the same target state version.

[0122] When the consecutive failure count reaches the opening threshold, the circuit breaker manager keeps the state open and sets the candidate qualification field to false. When the consecutive success count reaches the recovery threshold, the state is changed to half-open and 10% concurrency is allowed. Every 30-second failure-free observation window increments to the next level. If a call failure occurs at any level, it falls back to the previous level. When the 10% level fails or the consecutive failures reach the opening threshold again, it re-enters the open state. When it reaches the 100% level and an observation window has no failures, it changes to closed and the candidate qualification is restored.

[0123] The subsequent route reader uses the globally published version as the snapshot key to read the channel boundary, fault domain evidence, primary and backup edge attributes, candidate qualifications, circuit breaker status and quota level of that version in one go. If any object version is inconsistent or not published, the snapshot is discarded and reread from the latest published version, so that the output of this step is used as the unified route status source for new requests after S1.

[0124] To avoid repeated circuit breaking and recovery oscillations within a short window, the circuit breaker manager will accumulate the failure count and the success count separately. Any successful probe will reset the consecutive failure count to zero, and any failed probe will reset the consecutive success count to zero. After re-entering the circuit breaker, the quota level will return to 0. All counts, quota levels, and the starting point of the observation window are atomically released with respect to the target state version.

[0125] Each probe result, circuit breaker status change, and quota level change constitutes an independent versioned update record. According to the target state version allocation and atomic submission rules of this step, the corresponding channel node, its fault domain node, candidate qualification, circuit breaker status, and all related primary and backup directed edges are updated simultaneously, and the edge state version is written to the same target version. Finally, the published candidate graph of the same version, boundary indicators, fault evidence, candidate qualification, and circuit breaker quota constitute a unified routing state that can be read at once in subsequent requests.

[0126] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0127] This invention incorporates the reliability, latency, and cost conservative boundaries related to the request, along with the shared failure relationships between channels, into the routing plan solution. This ensures that the primary channel selection, backup order, latest start time, and maximum number of attempts are all constrained by the same remaining time and cost budgets. Consequently, backup switching is no longer triggered by a single failure, but rather occurs when the failure source, plan version, current budget, and candidate channel boundaries are all satisfied.

[0128] This invention synchronously projects the invocation evidence to channel nodes, fault domain nodes, candidate qualifications, circuit breaker status, and related primary and backup directed edges by atomic comparison and publication of the target state version. Combined with data plane consistency probes, hysteresis thresholds, and tiered quota recovery, this invention enables state updates and fault recovery to be read by subsequent requests with a consistent version. From a technical structure perspective, this invention suppresses repeated hits and retry amplification caused by asynchronous feedback.

Claims

1. A multi-model API request routing and scheduling method based on multi-dimensional state feedback, characterized in that, include: S1. Receive the model application programming interface request and encode the model capability requirements, requested resource amount, completion deadline and cost limit into a request constraint vector; S2. Filter compatible channels based on the request constraint vector and read the channel status with version. Convert the channel status into a lower bound of success probability, an upper bound of completion delay, and an upper bound of cost. Construct a candidate state relationship structure that includes channel nodes, fault domain nodes, and primary / backup directed edges. S3. Within the remaining time budget and remaining cost budget limited by the request constraint vector, based on the lower bound of the success probability, the upper bound of the completion delay, the upper bound of the cost, and the fault correlation represented by the candidate state relationship structure, jointly select the primary channel and at least one backup channel to generate a routing execution plan that includes the primary / backup order, switching conditions, and plan state version. S4. Initiate channel calls according to the routing execution plan. If a call fails, determine the source of the failure and deduct the time and cost. Switch to the next backup channel only when the upper bound of the completion delay and the upper bound of the cost of the next backup channel fall within the corresponding remaining budget and the state version before execution is consistent with the state version of the plan. S5. Based on the call result and the source of failure when the call fails, the status evidence of this call is atomically projected to the channel node, fault domain node, primary and backup directed edge, candidate qualification and circuit breaker status according to the same incremental status version, forming a unified routing status for subsequent requests.

2. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 1, characterized in that, S1 includes: The model identifier, context length, tool invocation capability, multimodal input type, and streaming output capability are encoded as model capability requirement fields, and a model type field is generated according to the mapping relationship between model identifiers and model types in the preset model capability catalog; a requested resource quantity field is generated according to the number of input tokens, the amount of media data to be encoded, and the expected number of output tokens, and the requested resource quantity field includes the amount of input data and the expected amount of output data. The request complexity is determined based on the prompt word structure, the number of tool calls, and the expected number of inference stages, and the request complexity is mapped to a preset complexity bucket. The difference between the completion deadline and the request reception time is used as the initial time budget, and the cost ceiling is used as the initial cost budget. The request constraint vector is composed of the model capability requirement field, model type field, request resource quantity field, complexity bucket, streaming output requirement, initial time budget, initial cost budget, and success probability threshold.

3. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 2, characterized in that, S2 includes: The channel status record is read according to the model type field and the complexity bucket. The channel status record includes the number of queued requests, the number of requests in transit, the first response latency quantile, the generation throughput quantile, the call result, the error source, the billing unit price, the event time, and the status version. The upper bound of the completion latency is determined based on the expected service start time, the first response latency quantile, the generation throughput quantile, and the expected output data volume. The lower bound of the success probability is determined based on the lower endpoint of the confidence interval of successful and failed call evidence; the upper bound of the cost is determined based on the billing unit price, input data volume, expected output data volume, and retry billing rules; records with event times later than the status reading time are excluded, the remaining records are sorted in ascending order by status version number, and the last record in the sorted list is taken as the current channel status record.

4. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 3, characterized in that, In the candidate state relation structure, the channel node carries the first response latency quantile, generation throughput quantile, successful call evidence, failed call evidence, queued request volume, in-transit request volume, cost range, and state version of the corresponding channel; the fault domain node represents the supplier, region, account quota, or upstream cluster, and carries the shared anomaly evidence and state version of the corresponding fault domain; the starting point of the primary-backup directed edge is the primary channel node or the previous backup channel node, and the ending point is the next backup channel node, and carries the conditional common failure probability, switching latency overhead measured in time, switching cost overhead measured in billing amount, and edge state version. The edge state version is the version field of the primary-backup directed edge in the candidate state relation structure, so as to determine the shared fault domain penalty and backup order in step S3.

5. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 4, characterized in that, S3 includes: obtaining the remaining time budget by the difference between the completion deadline and the current time; obtaining the remaining cost budget by subtracting the consumed cost from the cost ceiling; determining the shared fault domain penalty based on the fault domain nodes jointly connected to the channel nodes at both ends of the primary and backup directed edges and the common failure probability of the conditions; using the lower bound of the success probability of the primary channel and each backup channel, the upper bound of the completion delay, the upper bound of the cost, the shared fault domain penalty, the switching delay overhead, and the switching cost overhead as the inputs for the plan solution, forming a channel sequence to be inspected that includes the primary channel and all candidate backup channels according to the execution order from the primary channel to the backup channel; and cutting off the primary channel corresponding to the first channel in the channel sequence to be inspected. Both the switching delay overhead and the switching cost overhead are set to zero. Starting from the second sequence position of the channel sequence to be inspected, the primary and backup directed edges from the channel of the previous sequence position to the channel of the current sequence position are taken as the corresponding primary and backup directed edges of the current sequence position. The upper bound of the completion delay of each channel in the channel sequence to be inspected is added to the switching delay overhead corresponding to the current sequence position to obtain the upper bound of the cumulative delay of each sequence position. The upper bound of the cost of each channel is added to the switching cost overhead corresponding to the current sequence position to obtain the upper bound of the cumulative cost of each sequence position. These are used by step S3 to determine the executable prefix subject to dual budget constraints in the channel sequence to be inspected.

6. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 5, characterized in that, Step S3 further includes: for each backup channel, subtracting the completion delay upper bound of the backup channel and the switching delay overhead from the previous channel to the backup channel from the completion deadline to obtain the latest start time of the backup channel; starting from the first sequence position of the channel to be inspected sequence, comparing the cumulative delay upper bound and the cumulative cost upper bound of each sequence position with the corresponding remaining budget, forming an executable prefix with the sequence positions that continuously satisfy the cumulative delay upper bound not exceeding the remaining time budget and the cumulative cost upper bound not exceeding the remaining cost budget, and determining the number of channels included in the executable prefix as the maximum number of attempts, the executable prefix includes all channels in the channel to be inspected sequence when all sequence positions satisfy the dual budget; each time a channel call is initiated, the current number of attempts is incremented by one, the main channel call is the first attempt; setting the success probability lower bound of the backup channel reaching the success probability threshold, the current time not later than the corresponding latest start time, the sequence position of the backup channel belonging to the executable prefix, and the edge state version of the main and backup directed edge equal to the plan state version as the executable conditions for the corresponding main and backup directed edge.

7. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 6, characterized in that, Step S4, determining the source of failure, includes: generating channel-specific failure evidence and shared failure domain failure evidence based on response codes, domain name resolution phase, connection establishment phase, transport layer security handshake phase, request sending phase, the location of the first stream data arrival, the stream interruption location, gateway observation results, and abnormal records of other channels within the same time window; when multiple channels sharing the same supplier, region, account quota, or upstream cluster experience failures of the same connection phase or response code category within the same time window, the corresponding failure is assigned to the corresponding failure domain node; otherwise, the corresponding failure is assigned to the channel node where the call occurred.

8. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 7, characterized in that, Before switching to the next backup channel, reread the status version of the backup channel node, the fault domain node connected to the backup channel node, and the edge status version of the corresponding primary-backup directed edge. The remaining time budget is recalculated based on the difference between the completion deadline and the reread completion time. The incurred costs during the switching decision period are deducted from the remaining cost budget before the switching decision to obtain the updated remaining cost budget. The switching is initiated only when the status versions of the reread channel nodes and fault domain nodes, as well as the edge status versions, are equal to the planned status version, the current number of attempts indicating the number of channel calls already initiated is less than the maximum number of attempts, and the executable condition is still met under the recalculated remaining time budget and the updated remaining cost budget. Otherwise, the routing execution plan is terminated and at least one of the following termination reasons is output: insufficient budget, version mismatch, attempt limit reached, or channel unavailable.

9. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 8, characterized in that, S5 includes: before submitting the status evidence for this channel call, reading the currently published status version, and allocating a target status version that is simultaneously greater than the corresponding planned status version and the currently published status version to the status evidence; forming a versioned update record by combining the channel's own fault evidence, shared fault domain fault evidence, call success evidence, changes in queued request volume, changes in in-transit request volume, measured first response latency, measured generation throughput, and measured cost; projecting the versioned update record to the corresponding channel node, fault domain node, candidate qualification, circuit breaker status, and related primary and backup directed edges through an atomic commit, writing the edge status version of the related primary and backup directed edges as the target status version, and publishing the target status version only when the currently published status version at the time of submission is still equal to the currently published status version read when allocating the target status version, so that the reader only uses the published projection object with the same version; when the currently published status version at the time of submission changes, rereading the currently published status version and reallocating the target status version.

10. The multi-model API request routing and scheduling method based on multi-dimensional state feedback according to claim 9, characterized in that, For channels in a circuit breaker state, a data plane consistency probe is executed. The data plane consistency probe uses the same domain name resolution, connection establishment, authentication, request transmission, and response reception path as the model application interface request. When the number of consecutive failures reaches the activation threshold, the circuit breaker state is maintained. When the number of consecutive successes reaches the recovery threshold greater than the activation threshold, the circuit breaker state is released. After the circuit breaker state is released, the allowed concurrent call quota is increased step by step according to the preset incremental ladder. If a call failure occurs at any level, the allowed concurrent call quota is rolled back to the previous level or the circuit breaker state is re-entered. The probe results and quota level are updated to the corresponding channel node, fault domain node and primary / backup directed edge according to the target state version in step S5, and the edge state version of the primary / backup directed edge is written as the target state version.

Images