Multi-vendor large model unified scheduling management method and system

Abstract

This application discloses a unified scheduling and management method and system for multi-vendor large-scale models. The method includes: collecting the first token response latency of streaming requests from each vendor's interface and storing it in a corresponding sliding window in chronological order; calculating the window mean and the trend increment determined by the difference between the mean values ​​of the first and second halves of the sliding window based on the observations within the window; calculating the real-time service capability score of each vendor by combining a pre-calibrated baseline first token response latency, the window mean, and the amplified positive trend increment; determining the routing weight based on the real-time service capability score and a cost coefficient, and normalizing the routing weight into an allocation probability, thereby allocating service requests to the corresponding vendor's interface. This invention can dynamically adjust request allocation based on changes in the response status of vendor interfaces, improving the scheduling adaptability of multi-vendor large-scale model services.

Description

Technical Field

[0001] This application relates to the field of large model service scheduling technology, specifically a unified scheduling and management method and system for large models from multiple vendors. Background Technology

[0002] As large language model services are gradually opened to business systems via cloud APIs, enterprise applications simultaneously connecting to multiple model vendors has become a common deployment method to improve model selection flexibility, service availability, and cost control. Different vendors differ in interface protocols, authentication methods, model capabilities, call prices, and service performance. Therefore, in practical applications, interfaces from various vendors are typically encapsulated through a unified interface layer or AI gateway, and request allocation is performed using methods such as round-robin, fixed weights, rule-based routing, fault retries, or backup vendor switching.

[0003] For large model services using streaming output, the user's wait time from sending a request to receiving the first token is a crucial performance indicator affecting the user experience. This time is influenced by factors such as network transmission, server-side queuing, processing of prompts, and model inference, and may change with the instantaneous request load on the vendor's side. However, the caller typically cannot directly obtain the internal queuing depth, computing resource utilization, and other operational status of the third-party model service; they can only judge the service status based on external information such as interface response results and response times. Existing scheduling methods based on fixed weights or preset rules are difficult to adapt to the dynamic changes in vendor service capabilities in a timely manner, while methods that rely on switching after request timeouts or interface errors have a certain lag, easily allowing some requests to continue entering service channels that have already experienced performance degradation. Therefore, how to utilize the operational data available to the caller to timely assess the actual service status of different vendors, and dynamically adjust request allocation while taking into account call costs, remains a problem that needs to be solved in scenarios involving unified access to large models from multiple vendors. Summary of the Invention

[0004] The purpose of this application is to provide a unified scheduling and management method and system for large multi-vendor models to solve the problems mentioned in the background art.

[0005] According to one aspect of this application, a unified scheduling and management method for large-scale multi-vendor models is provided, comprising the following steps: For streaming requests to the large model interfaces of various vendors, the time interval from the time the request is sent to the time of receiving the first token in the streaming response is collected as the first token response delay, and the first token response delay is stored in the corresponding vendor's sliding window in chronological order. Based on the first token response delay observations within each vendor's sliding window, the sliding window mean and trend increment for each vendor are calculated. The trend increment is the difference between the mean of the observations in the second half of the sliding window and the mean of the observations in the first half. Based on the benchmark first token response latency pre-calibrated by each vendor, the average value of the sliding window, and the trend increment, calculate the real-time service capability score of each vendor: take the larger of the trend increment and zero, amplify it by the trend amplification factor, and add it to the average value of the sliding window to obtain the adjusted latency value. The ratio of the benchmark first token response latency to the adjusted latency value is used as the real-time service capability score. Based on the real-time service capability scores and cost coefficients of each vendor, the routing weight of each vendor is calculated, the routing weight is normalized into an allocation probability, and the arriving service requests are allocated to the corresponding vendor interfaces according to the allocation probability.

[0006] Preferably, the trend increment is calculated as follows: the observed values ​​in the sliding window are divided into a first half and a second half according to the storage time order, the arithmetic mean of the first half and the arithmetic mean of the second half are calculated respectively, and the result of subtracting the arithmetic mean of the first half from the arithmetic mean of the second half is used as the trend increment.

[0007] Preferably, in the calculation of the real-time service capability score, the real-time service capability score is subjected to a scoring sensitivity index and then exponentially calculated, and then multiplied by the cost coefficient to obtain the routing weight.

[0008] Preferably, the cost coefficient is determined by taking the maximum value of the billing price per thousand tokens among all access providers, dividing the maximum value by the billing price per thousand tokens of each provider, and using the resulting ratio as the cost coefficient of the corresponding provider.

[0009] Preferably, the method further includes vendor removal, which specifically involves removing a vendor from the candidate pool for routing weight calculation when the real-time service capability score of a certain vendor falls below a certain threshold for a certain number of consecutive preset times. Subsequent service requests will no longer be assigned to that vendor.

[0010] Preferably, the method further includes vendor recovery, which specifically involves: sending probe requests to the removed vendors at fixed time intervals and collecting the first token response delay of the probe requests; when the average first token response delay of multiple consecutive probes is lower than a preset multiple of the vendor's benchmark first token response delay, the vendor is reinstated into the candidate pool to participate in the routing weight calculation.

[0011] Preferably, the calibration method for the benchmark first token response latency is as follows: filter the first token response latency observations corresponding to the time periods when the total request rate is lower than a preset proportion of the historical peak, take the median of these observations as the benchmark first token response latency of the corresponding vendor, and recalibrate and update it according to a preset period.

[0012] Preferably, the capacity of the sliding window is determined based on the average request frequency of the corresponding vendor channel: the number of samples corresponding to the number of requests within a preset time period is taken as the window capacity, and the sliding window is updated in a first-in-first-out order.

[0013] Preferably, the trend amplification coefficient is determined in the following way: on historical request data containing known vendor performance degradation events, the trend amplification coefficient is taken one by one within a preset range in steps, the scheduling process is simulated for each value and the weighted sum of the request timeout rate and the number of unnecessary switching is calculated, and the value that minimizes the weighted sum is selected as the trend amplification coefficient.

[0014] In another aspect, this application also provides a unified scheduling and management system for large-scale multi-vendor models, including: The first token response delay acquisition module is used to collect the time interval from the time the request is sent to the time of receiving the first token in the streaming response for streaming requests of large model interfaces of various vendors. This time interval is used as the first token response delay, and the first token response delay is stored in the corresponding vendor's sliding window in chronological order. The statistical calculation module is used to calculate the sliding window mean and trend increment of each vendor based on the first token response delay observation value within the sliding window of each vendor. The trend increment is the difference between the mean of the observation values ​​in the second half of the sliding window and the mean of the observation values ​​in the first half of the sliding window. The service capability scoring module is used to calculate the real-time service capability score of each vendor based on the pre-calibrated benchmark first token response latency, the sliding window mean, and the trend increment. The larger of the trend increment and zero is taken, amplified by a trend amplification factor, and added to the sliding window mean to obtain the adjusted latency value. The ratio of the benchmark first token response latency to the adjusted latency value is used as the real-time service capability score. The routing allocation module is used to calculate the routing weight of each vendor based on the real-time service capability score and the cost coefficient of each vendor, normalize the routing weight into an allocation probability, and allocate the arriving service requests to the corresponding vendor interface according to the allocation probability.

[0015] This application continuously collects the first token response latency of each vendor's large-scale model interface and combines it with the sliding window average and trend increments reflecting recent changes to perform real-time evaluation of vendor service status. This allows scheduling criteria to be dynamically updated according to the actual response status of the interfaces. Real-time service capability scores and call costs are incorporated into the routing weight calculation, and the flow of business requests is adjusted with continuously changing allocation probabilities. This allows vendors whose performance begins to decline to gradually reduce their request load before significant timeouts occur, while smoothly transferring requests to other vendors with better current service capabilities. As a vendor's response status recovers, its routing weight can be increased accordingly. Combined with removal and probe recovery mechanisms during continuous degradation, this creates an adaptive scheduling process among multiple vendors that adjusts according to service status changes. This reduces waiting overhead caused by requests continuously entering congestion channels, improving the response stability and overall scheduling efficiency of the unified service for a multi-vendor large-scale model while considering call costs. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a unified scheduling and management method for a large multi-vendor model provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of dynamic routing and scheduling for a large multi-vendor model provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of a unified scheduling and management system for a large multi-vendor model provided in an embodiment of this application.

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] According to embodiments of this disclosure, this method is applicable to the following scenarios: the system simultaneously accesses the API interfaces of at least two large model service providers, and each provider's interface supports streaming response mode. The system is deployed as middleware between the business application layer and the APIs of each provider, undertaking request routing and response forwarding. During operation, the server-side load status of each provider is not visible to the caller, and the system cannot directly obtain any provider's internal queuing depth or resource consumption indicators. The system needs to infer the current actual service capacity of each provider based on the observable response characteristics on the client side and dynamically adjust request allocation accordingly. The system encapsulates the differences between providers in request format, authentication method, and response structure through a unified abstract interface layer. Upper-layer business code calls the unified interface to initiate requests, and the scheduling layer completes provider selection and forwarding within the interface, transparent to the business code.

[0023] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the multi-vendor large-scale unified scheduling and management method described in this application. It should be noted that these embodiments are merely for explaining this application and not for limiting its scope of protection. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the concept of this application should be included within the scope of protection of this application.

[0024] like Figure 1 As shown in the diagram, this application discloses a unified scheduling and management method for a large multi-vendor model, including the following steps: Step S1: Collect the response delay of the first token and store it in the corresponding vendor's sliding window.

[0025] Step S2: Calculate the sliding window mean and trend increment for each vendor based on the observed first token response delay within the sliding window; Step S3: Calculate the real-time service capability score of each vendor based on the baseline first token response latency, sliding window mean, and trend increment. Step S4: Calculate the routing weight based on the real-time service capability score and cost coefficient, and allocate service requests according to the allocation probability.

[0026] Figure 2 This is a schematic diagram of dynamic routing and scheduling for a large multi-vendor model provided in an embodiment of this application, such as... Figure 2As shown, after a service request enters the unified scheduling layer, the system collects the first token response latency for the streaming responses of each vendor's large model interface and writes the corresponding observations into each vendor's sliding window. Based on the observations within the sliding window, the window mean and trend increment are calculated, where the trend increment reflects the change in latency in the latter half relative to the former half. Subsequently, combining the benchmark first token response latency of the corresponding vendor and the trend increment after trend amplification, a real-time service capability score is calculated. The real-time service capability score is then combined with a cost coefficient to obtain the routing weight for each vendor, and the routing weight is normalized to obtain the allocation probability. New service requests are allocated to the corresponding vendor's API according to the allocation probability. After the vendor returns a streaming response, the system continues to collect new first token response latency, thereby updating the sliding window and subsequent scheduling parameters.

[0027] Specifically, in some embodiments, for step S1, for streaming requests to the large model interfaces of each vendor, the time interval from the time the request is sent to the time of receiving the first token in the streaming response is collected as the first token response delay, and the first token response delay is stored in the corresponding vendor's sliding window in chronological order.

[0028] In large-scale streaming response scenarios, the response process for a single request consists of two phases: the phase of waiting for the first token to be returned, and the phase of continuously outputting subsequent tokens. The first token response latency refers to the time interval, measured in milliseconds (ms), between the moment the system sends a request to a vendor's API and the moment it receives the first valid token in the vendor's streaming response. The waiting time before the first token is returned is primarily determined by the request queuing time on the vendor's server, reflecting the vendor's current internal queue depth. Once inference starts and the first token is returned, the generation rate of subsequent tokens mainly depends on the model inference speed and has a weaker correlation with the queuing status. Therefore, the first token response latency is the most time-leading observable indicator of the vendor's current load status on the client side.

[0029] Each time a streaming request is sent to a vendor, the timestamp of the request is recorded; when the first data block of the streaming response is received, the timestamp of that moment is recorded; the difference between the two is the first token response delay of this request.

[0030] Each connected vendor maintains a fixed-capacity sliding window to store observations of the response latency of the first token in the most recent requests. The sliding window is updated in a first-in, first-out (FIFO) order: when a new observation is added to the tail of the window, the oldest observation is removed from the head of the window. The window capacity is denoted by N, where N is a positive integer.

[0031] In one embodiment, regarding window capacity The method for determining this is based on the average request frequency of the system on that vendor's channel. This is set to cover the number of samples corresponding to the number of requests within a preset time period. Specifically, if the vendor's channel receives an average of [number] requests per minute... This request has a preset coverage duration of 2 to 3 minutes. The range of values ​​is to Round up to the nearest integer. For example, if a vendor's channel has an average request frequency of 15 times per minute, It can be configured to 35. If the window size is too small, the evaluation results will be affected by a single outlier; if it is too large, the perception of trend changes will be delayed. The above value range strikes a balance between the two.

[0032] In some embodiments, for step S2, based on the first token response delay observation value within each vendor's sliding window, the sliding window mean and trend increment of each vendor are calculated respectively, wherein the trend increment is the difference between the mean of the observation value in the second half of the sliding window and the mean of the observation value in the first half of the sliding window.

[0033] Specifically, the manufacturer The sliding window is currently stored The observations are recorded in chronological order of their entry time. ( As the earliest, (This is the latest value; all values ​​are in milliseconds.) Sliding window mean. All within the window Arithmetic mean of the observations:

[0034] in For the first in the window The response latency observation for each first token is expressed in milliseconds. For window capacity; The result is in milliseconds.

[0035] In one embodiment, the trend increment is used to reflect the direction and magnitude of the change in the first token response latency within a window period. This is applied to the sliding window... Each observation is divided into a first half and a second half according to its storage time. The arithmetic mean of the first half and the arithmetic mean of the second half are calculated separately. The result of subtracting the arithmetic mean of the first half from the arithmetic mean of the second half is taken as the trend increment. :

[0036] in express Divide by 2 and round down; the first half includes the first to the last. The first observation, the second half contains the first observation. To the One observation value. The unit is milliseconds. A positive value indicates that the vendor's first token response latency is trending upwards within the window period, and service capacity is declining; a zero or negative value indicates that the state is stable or recovering. Compared to point-by-point differencing, comparing the means of the first and second halves is less sensitive to single outlier observations and can more stably reflect the overall trend.

[0037] For example, suppose a manufacturer's sliding window capacity The 10 observations in the current window are 320, 330, 310, 340, 350, 400, 420, 450, 470, and 500 (in milliseconds), with the mean of the first half being... ms, the mean of the second half is ms, then the trend increment ms.

[0038] In some embodiments, for step S3, the real-time service capability score of each vendor is calculated based on the benchmark first token response latency pre-calibrated by each vendor, the sliding window mean, and the trend increment: the larger of the trend increment and zero is taken, amplified by the trend amplification factor, and added to the sliding window mean to obtain the adjusted latency value. The ratio of the benchmark first token response latency to the adjusted latency value is used as the real-time service capability score.

[0039] Specifically, the baseline first token response latency Indicates manufacturer The typical first-token response latency under normal load conditions is used as the benchmark for scoring calculation, measured in milliseconds. The calibration method involves selecting first-token response latency observations corresponding to periods when the system's total request rate is lower than a preset percentage of historical peak values, and taking the median of these observations as the benchmark. The preset ratio here is, for example, 30%, meaning that only data from periods when the total system request rate does not exceed 30% of the peak value are selected for calibration to exclude interference from high-load periods. The median is chosen instead of the mean because the median is not sensitive to occasional extreme values. The system is recalibrated and updated according to a preset cycle, for example, recalculated every 24 hours based on low-load data from the most recent 7 days.

[0040] Optionally, when the system is first started, there is no historical data. At this time, a small number of probe requests (e.g., 10 short text generation requests) are sent to each vendor, and the median of the first token response delay of these probe requests is taken as the initial baseline value.

[0041] By combining the baseline value, the moving window mean, and the trend increment, the manufacturer's... Real-time service capability score Specifically, the calculation rule is: take the trend increment. The larger of the two (within zero) is amplified by a trend factor. After magnification, the mean of the sliding window Add them together to get the adjusted delay value; use the base first token to respond to the delay. The ratio of this adjusted latency value to the real-time service capability score is used as the score.

[0042] This can be expressed by the formula:

[0043] in The baseline first token response latency is in milliseconds; This is the mean of the sliding window, in milliseconds. This represents the trend increment, measured in milliseconds. This is the trend amplification factor. Operations take The larger of the two, zero and zero, is used so that the trend term is included in the denominator calculation only when the delay is increasing.

[0044] Approaching 1.0, the manufacturer's current performance is close to the normal benchmark; A value significantly less than 1.0 indicates a decline in service capacity; A value greater than 1.0 indicates that the current state is better than the historical benchmark.

[0045] In the denominator The technical function of this item is that when the vendor's initial token latency just begins to rise, Because there are still many normal values ​​within the window, the change is slow, but The trend has turned positive and is gradually increasing. The trend increment is amplified and added to the denominator, causing the score to begin decreasing in the early stages of latency increases, thus providing scheduling decisions with a predictive window of several seconds to tens of seconds. In large-scale API services, the increase in first-token latency is an external mapping of internal queue backlogs within the vendor; this signal typically precedes the occurrence of a complete response timeout. This trend-based prediction-driven early traffic diversion is the key difference between this method and traditional passive timeout switching solutions.

[0046] In one embodiment, the trend amplification factor The sensitivity of the control system to the upward trend of the first token delay. Too small, and it will not respond adequately to early signs of degradation; Excessive fluctuations, even small fluctuations within the normal range, can trigger unnecessary routing adjustments.

[0047] in The value is determined by performing a grid search on historical request data containing known vendor performance degradation events. Specifically, values ​​are selected one by one within a preset range (e.g., 1.0 to 3.0) in fixed steps (e.g., 0.1). For each value, a scheduling process is simulated, and the weighted sum of the request timeout rate and the number of unnecessary handovers (i.e., situations where the target vendor's latency is actually higher after a handover) during the simulation is calculated. The value that minimizes this weighted sum is selected as the target vendor's ... In the weighted sum, the weights for timeout rate and unnecessary switchovers can be assigned separately based on the business's tolerance for latency and preference for stability. For example, the weight for timeout rate could be 0.7, and the weight for unnecessary switchovers could be 0.3. In the initial deployment phase without sufficient historical data, The default starting value is 2.0, and it will be updated as data accumulates using the method described above.

[0048] As an example, if the historical data contains 3 known degradation events, for The simulation yielded a timeout rate of 4.2% and 12 unnecessary switches. The simulation yielded a timeout rate of 2.1% and 18 unnecessary switches. The simulation yielded a timeout rate of 1.8% and 31 unnecessary switches, with weighted sums of... , , Then select In actual deployments, grid search involves finer step sizes and larger data volumes; this is only for illustrative purposes.

[0049] In some embodiments, for step S4, the routing weight of each vendor is calculated based on the real-time service capability score and the cost coefficient of each vendor, the routing weight is normalized into an allocation probability, and the arriving service requests are allocated to the corresponding vendor interface according to the allocation probability.

[0050] When a new business request arrives at the scheduling layer, the system calculates the routing weight based on the real-time service capability score and cost coefficient of each vendor in the current candidate pool, normalizes it into an allocation probability, and selects the target vendor accordingly.

[0051] Specifically, cost coefficient For quantitative manufacturers The relative advantage in terms of cost, assuming the current candidate pool of the system has a total of There are 10 available vendors, and the billing price per thousand tokens for each vendor is as follows: The unit is RMB per thousand tokens. The highest billing price per thousand tokens is taken from all vendors. Divide this maximum value by the respective billing price per thousand tokens of each vendor, and use the resulting ratio as the cost coefficient for the corresponding vendor:

[0052] in The unit is yuan / thousand tokens. The unit is RMB per thousand tokens, and the most expensive vendor's... Manufacturers with lower costs The larger the value. When the billing unit price of various vendors is adjusted. and each Synchronized updates.

[0053] For each vendor in the candidate pool First, score the real-time service capability. Applying a rating sensitivity index Perform exponentiation, then combine with cost coefficient. Multiply to obtain the routing weight. :

[0054] in To score real-time service capabilities, For rating sensitivity index, This is the cost coefficient (dimensionless). This represents the routing weight.

[0055] Rating Sensitivity Index Control the response magnitude of route allocation to the score differences between vendors, among which Time weighting and scoring show a linear relationship, and traffic differences among vendors are moderate. As the rating increases, manufacturers with higher ratings gain a larger market share, while manufacturers with lower ratings experience a faster decline in market share. The method of determination and Similarly, a grid search is performed on historical data ranging from 2.0 to 4.0 to select the value that minimizes the overall average response latency of the system. The default value is used during initial deployment. .

[0056] Normalize the routing weights of each vendor into allocation probabilities :

[0057] in The number of available vendors in the current candidate pool, ranked by allocation probability. Assign the currently arriving service requests to the vendor. The interface. In the specific implementation, a... The target manufacturer is determined by using a cumulative interval of uniformly distributed random numbers within the interval, based on the probability distribution of each manufacturer.

[0058] As an example, suppose the system connects to three vendors, A, B, and C, with billing prices of respectively. Yuan / thousand tokens Yuan / thousand tokens Yuan / thousand tokens. Then The cost coefficients are respectively , , The benchmark first-token response latency for each vendor is calibrated as follows: ms ms ms.

[0059] During normal operation, the latency of the first token for each vendor is close to the baseline value, and the scores are all close to 1.0. At this point, [the following is a possible interpretation:] Take... The routing weights are respectively , , The normalized distribution probabilities are respectively , , Most requests went to vendor A, which had lower costs.

[0060] When manufacturer A experiences queue congestion during peak hours, its sliding window mean rises to ms, trend increment ms (the latter half is significantly higher than the first half), take When the adjusted delay value is MS, rating Manufacturer B, on the other hand, remained normal. At this point, the routing weight... , , After normalization , , Vendor A's allocation ratio decreased from 76% to 26%, and traffic was automatically transferred to Vendor B.

[0061] In some embodiments, the following describes the vendor removal, restoration, and negative feedback adjustment processes during the scheduling process.

[0062] In actual operation, when a vendor is in a state of severe degradation, even if its routing weight has dropped to an extremely low level, a small number of requests will still hit that vendor by probability. To avoid unnecessary loss of these requests, the system is equipped with a removal mechanism.

[0063] The removal condition is that when the manufacturer Real-time service capability score The number of consecutive preset times is less than the removal threshold When this happens, the vendor will be removed from the candidate pool for route weight calculation, and subsequent service requests will no longer be assigned to that vendor.

[0064] Regarding the removal threshold The meaning and value of , specifically, according to the scoring formula, equal At that time, the equivalent first token response latency of the corresponding vendor is its baseline first token response latency. times. A value between 0 and 1 is used. For example, a value of 0.3 means that the removal decision is triggered when the vendor's first token latency inflates to approximately 3.3 times the baseline value. This value is based on historical operational statistics of mainstream large-scale APIs, where when the first token latency inflates by more than 3 times, the overall request timeout rate has significantly increased, and the benefit of continuing to allocate requests is lower than the waiting cost. It can be adjusted within the range of 0.2 to 0.4 depending on the business's tolerance for latency.

[0065] Regarding the value of the preset number of times, specifically, the number of consecutive low scores is set to prevent erroneous removal due to a single or a few abnormal observations. The value range is 3 to 5 times, meaning that the score is lower than the preset number of times for 3 to 5 consecutive window update cycles. The removal action is triggered.

[0066] Regarding the vendor recovery process, specifically, for removed vendors, the system sends probe requests at fixed time intervals (e.g., 30 seconds). These probe requests use minimal input prompts (e.g., requesting short text completion for a single word), aiming solely to measure the initial token response latency without consuming a large number of tokens. The system records the initial token response latency value for each probe. When the average initial token response latency of multiple consecutive probes (e.g., 3 consecutive probes) is lower than a preset multiple (e.g., 1.5 times) of the vendor's baseline initial token response latency, the system determines that the vendor has recovered and reintegrates it into the candidate pool for routing weight calculation.

[0067] When the vendor is reintegrated, its sliding window is initialized with the latency value collected during the probing period, and the score is calculated normally according to the aforementioned steps. Since the vendor's score is close to 1.0 rather than an abnormally high value after reintegration, traffic flows back according to the normal weight ratio, and there will be no sudden surge of requests.

[0068] In some embodiments, there is an inherent negative feedback regulation characteristic during continuous operation. When a vendor's initial token response latency increases due to an increase in external requests, the sliding window mean and trend increment calculated in the aforementioned steps increase accordingly, resulting in a decrease in the real-time service capability score. Consequently, the vendor's routing weight decreases, and the number of requests allocated to the vendor decreases. After the request volume decreases, the queuing depth of the vendor's server gradually decreases, the initial token response latency falls back, the sliding window mean and trend increment return to normal levels, the score recovers, the routing weight recovers, and the allocation ratio recovers accordingly.

[0069] The key difference between this process and the traditional circuit breaker approach is that a traditional circuit breaker completely cuts off traffic when "opening" and fully restores it when "closing." The switching process is discrete and abrupt, and the sudden surge in traffic during recovery may trigger overload again. In this method, the weights and allocation probabilities change continuously, and both the reduction and recovery of traffic are gradual, avoiding the secondary overload problem caused by abrupt changes. During this adjustment process, each vendor tends towards dynamic equilibrium, and the actual allocation share is automatically adjusted according to their respective service status, without the need for manual intervention in routing configuration.

[0070] In some embodiments, the scheduling logic is embedded within the factory method of the unified abstract interface layer. When the business code obtains a vendor instance through the factory method, the factory internally performs the weight calculation and probability selection steps described above, returning the Provider instance of the selected vendor. When each vendor's Provider instance executes a streaming response, it reports the timestamp to the scheduling layer upon receiving the first token, and the scheduling layer updates the sliding window for the corresponding vendor in the aforementioned steps accordingly.

[0071] Optionally, in scenarios where the business layer marks requests as urgent, the system can temporarily escalate requests that are close to their timeout deadline. Value (e.g., promoted to) This allows such requests to be more concentrated on the vendor with the highest current rating.

[0072] In practice, the sliding window and scoring state are maintained using in-memory data structures, and thread safety is ensured for each requesting thread's writes to the window through atomic operations or read-write locks. The computational complexity of the aforementioned steps is all... level, With latency in the tens of units, the impact on overall request processing latency is negligible, making it suitable for deployment in online services that are sensitive to response time.

[0073] Optionally, in scenarios where system operating budgets are limited, the cost coefficient of high-cost vendors can be adjusted. By further multiplying by a budget constraint factor of less than 1, traffic is concentrated in low-cost channels when no degradation occurs, and automatically overflows to high-cost vendors only when low-cost vendors degrade, thereby achieving a dynamic balance between cost constraints and service quality.

[0074] Therefore, this method uses the inherent first-token response latency in the large-scale streaming response model as a basis for inferring vendor service capabilities. Combined with trend-incremental amplification and extrapolation, it enables forward-looking scheduling decisions, initiating gradual traffic distribution in the early stages of vendor degradation. Simultaneously, continuous changes in routing weights achieve negative feedback regulation, bringing the load of each vendor towards dynamic balance. Compared to traditional passive switching schemes based on timeout detection, this reduces the waiting time for requests in congested channels, lowering the overall system request timeout rate and average response latency.

[0075] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0076] Please see Figure 3 , Figure 3 This application provides a unified scheduling and management system for a large-scale multi-vendor model. The system embodiments are similar to... Figure 1 Corresponding to the illustrated method embodiments, this system can be specifically applied to various electronic devices. The system specifically includes: The first token response delay acquisition module 301 is used to collect the time interval from the time the request is sent to the time of receiving the first token in the streaming response for the streaming request of the large model interface of each vendor, as the first token response delay, and store the first token response delay in the corresponding vendor's sliding window in chronological order. The statistical calculation module 302 is used to calculate the mean and trend increment of the sliding window for each manufacturer based on the first token response delay observation value within the sliding window of each manufacturer. The trend increment is the difference between the mean of the observation value in the second half of the sliding window and the mean of the observation value in the first half of the sliding window. The service capability scoring module 303 is used to calculate the real-time service capability score of each vendor based on the benchmark first token response latency pre-calibrated by each vendor, the sliding window mean, and the trend increment; wherein, the larger of the trend increment and zero is taken, amplified by a trend amplification factor, and added to the sliding window mean to obtain the adjusted latency value, and the ratio of the benchmark first token response latency to the adjusted latency value is used as the real-time service capability score. The routing allocation module 304 is used to calculate the routing weight of each vendor based on the real-time service capability score and the cost coefficient of each vendor, normalize the routing weight into an allocation probability, and allocate the arriving service requests to the corresponding vendor interface according to the allocation probability.

[0077] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0078] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0079] Figure 4 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input section 406, an output section 407, a communication section 409, and an input / output (I / O) interface 405 are also connected to the bus 404.

[0080] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 401, it performs the functions defined in the methods of this application.

[0081] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0082] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of electronic devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A unified scheduling and management method for a large multi-vendor model, characterized in that, Includes the following steps: For streaming requests to the large model interfaces of various vendors, the time interval from the time the request is sent to the time of receiving the first token in the streaming response is collected as the first token response delay, and the first token response delay is stored in the corresponding vendor's sliding window in chronological order. Based on the first token response delay observations within each vendor's sliding window, the sliding window mean and trend increment for each vendor are calculated. The trend increment is the difference between the mean of the observations in the second half of the sliding window and the mean of the observations in the first half. Based on the benchmark first token response latency pre-calibrated by each vendor, the average value of the sliding window, and the trend increment, calculate the real-time service capability score of each vendor: take the larger of the trend increment and zero, amplify it by the trend amplification factor, and add it to the average value of the sliding window to obtain the adjusted latency value. The ratio of the benchmark first token response latency to the adjusted latency value is used as the real-time service capability score. Based on the real-time service capability scores and cost coefficients of each vendor, the routing weight of each vendor is calculated, the routing weight is normalized into an allocation probability, and the arriving service requests are allocated to the corresponding vendor interfaces according to the allocation probability.

2. The method according to claim 1, characterized in that, The trend increment is calculated as follows: the observations within the sliding window are divided into a first half and a second half according to the storage time order, the arithmetic mean of the first half and the arithmetic mean of the second half are calculated respectively, and the result of subtracting the arithmetic mean of the first half from the arithmetic mean of the second half is used as the trend increment.

3. The method according to claim 1, characterized in that, In the calculation of the real-time service capability score, the score is subjected to a scoring sensitivity index and then exponentially calculated before being multiplied by the cost coefficient to obtain the routing weight.

4. The method according to claim 1, characterized in that, The cost coefficient is determined as follows: take the maximum value of the billing price per thousand tokens among all access providers, divide the maximum value by the billing price per thousand tokens of each provider, and use the resulting ratio as the cost coefficient of the corresponding provider.

5. The method according to claim 1, characterized in that, The vendor removal also includes the following: when a vendor's real-time service capability score falls below a certain threshold for a certain number of consecutive preset times, the vendor will be removed from the candidate pool for participating in the routing weight calculation, and subsequent service requests will no longer be assigned to that vendor.

6. The method according to claim 5, characterized in that, The vendor recovery process includes: sending probe requests to removed vendors at fixed time intervals and collecting the first token response latency of the probe requests; when the average first token response latency of multiple consecutive probes is lower than a preset multiple of the vendor's baseline first token response latency, the vendor will be reinstated into the candidate pool to participate in the routing weight calculation.

7. The method according to claim 1, characterized in that, The calibration method for the baseline first token response latency is as follows: filter the observation values ​​of the first token response latency corresponding to the time period when the total request rate is lower than the preset proportion of the historical peak, take the median of these observation values ​​as the baseline first token response latency of the corresponding vendor, and recalibrate and update it according to the preset period.

8. The method according to claim 1, characterized in that, The capacity of the sliding window is determined based on the average request frequency of the corresponding vendor channel: the number of samples corresponding to the number of requests within a preset coverage period is taken as the window capacity, and the sliding window is updated in a first-in-first-out order.

9. The method according to claim 1, characterized in that, The trend amplification coefficient is determined in the following way: on historical request data containing known vendor performance degradation events, the trend amplification coefficient is taken one by one within a preset range in steps. For each value, the scheduling process is simulated and the weighted sum of the request timeout rate and the number of unnecessary switching is calculated. The value that minimizes the weighted sum is selected as the trend amplification coefficient.

10. A unified scheduling and management system for a multi-vendor, large-scale model, characterized in that, include: The first token response delay acquisition module is used to collect the time interval from the time the request is sent to the time of receiving the first token in the streaming response for streaming requests of large model interfaces of various vendors. This time interval is used as the first token response delay, and the first token response delay is stored in the corresponding vendor's sliding window in chronological order. The statistical calculation module is used to calculate the sliding window mean and trend increment of each vendor based on the first token response delay observation value within the sliding window of each vendor. The trend increment is the difference between the mean of the observation values ​​in the second half of the sliding window and the mean of the observation values ​​in the first half of the sliding window. The service capability scoring module is used to calculate the real-time service capability score of each vendor based on the pre-calibrated benchmark first token response latency, the sliding window mean, and the trend increment. The larger of the trend increment and zero is taken, amplified by a trend amplification factor, and added to the sliding window mean to obtain the adjusted latency value. The ratio of the benchmark first token response latency to the adjusted latency value is used as the real-time service capability score. The routing allocation module is used to calculate the routing weight of each vendor based on the real-time service capability score and the cost coefficient of each vendor, normalize the routing weight into an allocation probability, and allocate the arriving service requests to the corresponding vendor interface according to the allocation probability.

Images