Interface fault-tolerant routing method based on flow perception and attenuation score

Abstract

The application discloses an interface fault-tolerant routing method based on streaming perception and attenuation scoring, which comprises the following steps: obtaining streaming inference response, historical request record and error type; extracting real-time Token generation rate, correcting the basic backoff interval by exponential mapping according to the deviation of the real-time Token generation rate and the overload threshold, obtaining the dynamic backoff interval to perceive implicit overload; combining the load indication and the error type to determine the basic load factor, and differentiating and aggregating the dynamic backoff interval to calculate the retry waiting time, thereby realizing macroscopic scene adaptive backoff; performing exponential attenuation weighted calculation on the delay and state of the historical request based on the time difference, thereby generating a predictive health score that emphasizes recent performance; performing failover on the candidate backends according to the score, and updating the state of the failed backend at an interval that increases exponentially with the number of failures. The application realizes double adaptive of backoff strategy and accurate prediction of failover, and greatly improves the system high availability and response timeliness.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an interface fault-tolerant routing method based on streaming awareness and attenuation scoring. Background Technology

[0002] With the widespread application of distributed systems and streaming inference services such as large language models, the interface calls between the client and the inference backend face complex network environments and backend load fluctuations. Interface fault tolerance and dynamic routing have become key technologies to ensure high availability of services.

[0003] Regarding retry mechanisms, existing technologies typically employ fixed-interval or simple exponential backoff strategies. However, these methods have two significant drawbacks: First, they lack real-time micro-awareness of streaming inference response data. When the backend computing queue begins to back up but before an explicit error is triggered, the token generation rate of the streaming output has already decreased significantly. At this point, the fixed backoff strategy cannot detect this implicit overload and dynamically adjust the backoff strength, which can easily lead to a continuous influx of requests and trigger a system avalanche. Second, they do not differentiate between error types and backend load indicators. For example, frequency-limited errors usually mean that the backend queue is saturated and requires a long emptying period, while brief connection rejections only require a very short retry. Existing uniform backoff strategies are insufficient in rate-limited scenarios, exacerbating backend pressure, and excessive in transient interruption scenarios, needlessly prolonging the wait. Summary of the Invention

[0004] To address these issues, this invention provides an interface fault-tolerant routing method based on streaming awareness and attenuation scoring, thereby resolving the aforementioned problems in the prior art.

[0005] To achieve the above objectives, this invention provides an interface fault-tolerant routing method based on stream awareness and attenuation scoring, comprising:

[0006] Step S1: Obtain the streaming inference response data of the currently active inference backend, the historical request record sequence of the candidate inference backend, and the error type parameter of the current request. The historical request record sequence includes timestamps, request delays, and request status.

[0007] Step S2: Extract features from the streaming inference response data to obtain the real-time token generation rate, and perform exponential mapping correction on the basic backoff interval based on the deviation between the real-time token generation rate and the preset overload threshold to obtain the dynamic backoff interval.

[0008] Step S3: Obtain the load indication information of the current request, determine the basic load factor according to the error type parameter and the load indication information, and perform differential aggregation calculation based on the dynamic backoff interval and the basic load factor to obtain the retry waiting time;

[0009] Step S4: Extract the time difference parameter between the timestamp of each record in the historical request record sequence and the current time; perform exponential decay weighted calculation on the request delay and request status based on the time difference parameter to generate a predictive health score.

[0010] Step S5: Based on the predictive health score, the candidate inference backends are sorted and selected to perform failover, and the failed backends are probed at a probe interval that increases exponentially with the number of probe failures to update the backend availability status.

[0011] Furthermore, the process of step S2 includes:

[0012] The number of tokens received in the streaming inference response data is counted according to a preset sliding time window, and the real-time token generation rate is calculated.

[0013] The difference between the preset overload threshold and the real-time token generation rate is calculated as the deviation value;

[0014] The product of the deviation value and the preset scaling factor is used as an exponent for exponential operation to obtain the micro-avoidance factor;

[0015] The product of the micro-avoidance factor and the basic avoidance interval is determined as the dynamic avoidance interval.

[0016] Furthermore, the process of step S3 includes:

[0017] Extract the load status parameters returned by the server in the current request;

[0018] Determine the backoff adjustment coefficient that matches the current abnormal scenario based on the error type parameter and the load status parameter;

[0019] Based on the dynamic backoff interval, the backoff adjustment coefficient, and the current retry order, an aggregation operation is performed to obtain the candidate waiting time;

[0020] The candidate waiting time is compared with a preset maximum waiting threshold, and the smaller value is taken as the retry waiting time.

[0021] Furthermore, the process of determining the backoff adjustment coefficient matching the current abnormal scenario based on the error type parameter and the load state parameter includes:

[0022] When the error type parameter indicates a frequency limit error, and the load status parameter contains a valid retry wait time indication or the remaining request quota is zero, the backoff adjustment coefficient is set to a first value.

[0023] When the error type parameter indicates a connection rejection, the backoff adjustment coefficient is set to a second value that is less than the first value;

[0024] When the error type parameter indicates a server error and the load status parameter does not indicate anything, the backoff adjustment coefficient is set to an intermediate value.

[0025] When the error type parameter indicates a timeout exception, the backoff adjustment coefficient is set to a third value between the intermediate value and the first value.

[0026] Furthermore, the process of performing aggregation calculations based on the dynamic backoff interval, the backoff adjustment coefficient, and the current retry order to obtain the candidate waiting time includes:

[0027] ,

[0028] Where T represents the waiting time, D represents the dynamic backoff interval, n is the current retry order, K is the backoff adjustment coefficient, and C is the dynamic exponent base, where C = 1 + K.

[0029] Furthermore, the process of step S4 includes:

[0030] Based on the request status of each record in the historical request record sequence, assign a differentiated decay constant to it;

[0031] Calculate the time difference parameter between the timestamp of each record and the current time. Use the negative of the product of the time difference parameter and the differential decay constant as the exponent to calculate the power of the natural constant, and obtain the differential decay weight of each record.

[0032] Using the differentiated attenuation weight, the weighted average delay is calculated by weighting the request delay in the historical request record sequence, the weighted delay fluctuation coefficient is calculated by weighting the variance of the request delay, and the weighted success rate is calculated by weighting the request status.

[0033] Extract the historical health gradient of the historical request record sequence within adjacent historical time windows, perform exponential smoothing and cumulative calculation on the historical health gradient, and generate a trend momentum factor;

[0034] The availability modulation coefficient is generated by strengthening the weighted success rate by a preset power greater than 1; the predictive health score is generated by correlating the availability modulation coefficient, the normalized reciprocal of the weighted average delay, the weighted delay fluctuation coefficient, and the trend momentum factor.

[0035] Furthermore, the differentiated attenuation constant includes a first attenuation constant and a second attenuation constant, with the first attenuation constant corresponding to the record representing failure and the second attenuation constant corresponding to the record representing success.

[0036] Further, the process of generating the predictive health score by performing correlation calculations based on the availability modulation coefficient, the normalized reciprocal of the weighted average delay, the weighted delay fluctuation coefficient, and the trend momentum factor includes:

[0037] ,

[0038] Where H is the predictive health score, S is the weighted success rate, L is the normalized value of the weighted average delay, V is the weighted delay volatility coefficient, and M is the trend momentum factor. and γ is a preset weighting coefficient, and γ is a preset exponent greater than 1.

[0039] Compared with existing technologies, the beneficial effects of this invention are as follows: It constructs a multi-dimensional perception base by integrating real-time streaming data, historical time series, and error types, eliminating single-point information blind spots; it dynamically lengthens the backoff interval by utilizing the decrease in token generation rate, giving congested backends breathing time to digest backlogged requests, effectively preventing request avalanches from overwhelming the system; it tailors retry waiting times according to error types and load information, providing long wait times for rate limiting overload and short retries for connection interruptions, neither adding to the problems of unhealthy backends nor causing users to wait idly; it uses time decay to calculate health scores, allowing the negative impact of historical faults to naturally fade over time, ensuring the score truly reflects the current actual performance of the backend to improve routing accuracy; based on accurate scores, it directs requests to the optimal node and probes faulty nodes at gradually increasing intervals, significantly reducing resource overhead caused by invalid probes, while ensuring timely traffic recovery when nodes recover, balancing high system availability and low resource consumption. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the interface fault-tolerant routing method based on streaming awareness and attenuation scoring provided by the present invention.

[0041] Figure 2 This is a flowchart illustrating step S2 in the interface fault-tolerant routing method based on streaming awareness and attenuation scoring provided by the present invention.

[0042] Figure 3 This is a flowchart illustrating step S3 in the interface fault-tolerant routing method based on streaming awareness and attenuation scoring provided by the present invention. Detailed Implementation

[0043] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0044] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0045] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0046] Please see Figure 1 As shown, this invention provides an interface fault-tolerant routing method based on stream awareness and attenuation scoring, including:

[0047] Step S1: Obtain the streaming inference response data of the currently active inference backend, the historical request record sequence of the candidate inference backend, and the error type parameter of the current request. The historical request record sequence includes timestamps, request delays, and request status.

[0048] Specifically, the system monitors the streaming inference response chain returned by the currently active inference backend based on the OpenAI compatible protocol, the Ollam native inference protocol, or the general HTTP API protocol. It intercepts and parses the acquired streaming data shards in real time to extract streaming inference response data. This streaming inference response data includes at least the first character delay, shard arrival timestamp, shard token quantity, and stream interruption flag. Based on this streaming inference response data, the system perceives the real-time throughput performance of the currently active inference backend when processing requests in office scenarios such as document continuation and polishing. It reads the historical request record sequence maintained in the plugin memory of each backend in the candidate inference backend pool within a preset sliding time window. This historical request record sequence is a fixed-length time-series data set stored using a circular buffer mechanism, where each record includes at least: the timestamp of the request initiation, the request delay representing the response time, and the request status representing the request result. The request status is divided into success status, rate-limited rejection status (e.g., HTTP 429 RateLimit), and server-side abnormal status (e.g., HTTP 429 RateLimit). 5xx), network transient status and timeout status; when an exception occurs in the current request, capture the exception event, parse and infer the HTTP response status code returned by the backend, the load indicator field in the response header (including the Retry-After header and the X-RateLimit-Remaining header) or the underlying network exception type; based on the parsing result, map the error type parameter of the current request to a preset error classification space adapted to the fault tolerance requirements of the office scenario; the error classification space includes at least: a first error type corresponding to backend overload or frequency limit (backoffset needs to be doubled to avoid continuous triggering of rate limiting), a second error type corresponding to network jitter or connection rejection (backoffset needs to be shortened to reduce user waiting interruption), and a third error type corresponding to server internal exception or timeout.

[0049] Step S2: Extract features from the streaming inference response data to obtain the real-time token generation rate, and perform exponential mapping correction on the basic backoff interval based on the deviation between the real-time token generation rate and the preset overload threshold to obtain the dynamic backoff interval.

[0050] Specifically, such as Figure 2 As shown, the process of step S2 includes:

[0051] Step S21: Count the number of tokens received in the streaming inference response data according to a preset sliding time window, and calculate the real-time token generation rate;

[0052] Specifically, according to a preset sliding time window (e.g., the last 2 or 3 seconds), the number of tokens contained in the received streaming data fragments in the streaming inference response data is continuously counted; the ratio of the number of tokens to the duration of the preset sliding time window is determined as the real-time token generation rate; this real-time token generation rate directly reflects the real-time throughput capability of the current inference backend for generating content for Office documents.

[0053] Step S22: Calculate the difference between the preset overload threshold and the real-time token generation rate as the deviation value;

[0054] Specifically, the preset overload threshold is the minimum speech rate (e.g., 15 tokens / s) to ensure a smooth reading experience for users in office scenarios; the difference between the preset overload threshold and the real-time token generation rate is calculated, and the difference is determined as the deviation value; when the deviation value is greater than zero, it indicates that although the backend has not disconnected, it is in a state of implicit overload due to insufficient computing power, and the magnitude of the deviation value is positively correlated with the severity of the implicit overload; when the deviation value is less than or equal to zero, it indicates that the backend throughput is normal.

[0055] Step S23: The product of the deviation value and the preset scaling factor is used as an exponent for exponential operation to obtain the micro-avoidance factor;

[0056] Specifically, the preset scaling factor is used to adjust the backoff sensitivity, and the preset base is a constant greater than 1 (such as the natural constant e or 2); when the deviation value is greater than zero, the micro backoff factor increases exponentially with the deepening of the implicit overload; when the deviation value is less than or equal to zero, the micro backoff factor is less than or equal to one.

[0057] Step S24: The product of the micro-avoidance factor and the basic avoidance interval is determined as the dynamic avoidance interval.

[0058] Specifically, the basic backoff interval is the baseline retry waiting time set in the configuration table (e.g., 100 milliseconds). Through the above-mentioned exponential mapping correction, when encountering implicit overload of slow speech, the system can automatically lengthen the dynamic backoff interval, giving the backend sufficient computing power recovery window, avoiding cascading pressure caused by invalid retries at fixed intervals, thereby improving the overall request success rate while ensuring the continuity of document editing.

[0059] Step S3: Obtain the load indication information of the current request, determine the basic load factor according to the error type parameter and the load indication information, and perform differential aggregation calculation based on the dynamic backoff interval and the basic load factor to obtain the retry waiting time;

[0060] Specifically, such as Figure 3 As shown, the process of step S3 includes:

[0061] Step S31: Extract the load status parameters returned by the server in the current request;

[0062] Specifically, the load status parameters include a retry wait time indicator field (such as the Retry-After header) and a remaining request quota field (such as the X-RateLimit-Remaining header) in the HTTP response header; the retry wait time indicator field is used to characterize the server-suggested cooldown time, and the remaining request quota field is used to characterize the remaining number of times the current API key is available within the rate limiting window.

[0063] Step S32: Determine the backoff adjustment coefficient that matches the current abnormal scenario based on the error type parameter and the load state parameter;

[0064] Specifically, the process of determining the backoff adjustment coefficient matching the current abnormal scenario based on the error type parameter and the load state parameter includes:

[0065] When the error type parameter indicates a frequency limit error, and the load status parameter contains a valid retry wait time indication or the remaining request quota is zero, the backoff adjustment coefficient is set to a first value.

[0066] Specifically, when the error type parameter indicates a frequency limit error (such as HTTP 429 RateLimit), and the load status parameter contains a valid retry wait time indication (such as a Retry-After field with a value greater than zero in the HTTP response header) or the remaining request quota is zero (such as the X-RateLimit-Remaining field value being 0), it indicates that the current inference backend is in an explicit overload state triggered by high-frequency calls. Sufficient cooling and recovery windows must be provided to avoid triggering a backend avalanche effect. At this time, the backoff adjustment coefficient is set to the first value (e.g., 2.0) to double the backoff interval in aggregation operations, ensuring that the AI ​​function will not be completely paralyzed due to blind retries during peak Office hours.

[0067] When the error type parameter indicates a connection rejection, the backoff adjustment coefficient is set to a second value that is less than the first value;

[0068] Specifically, when the error type parameter indicates a connection refused, it means that the backend service has only experienced a brief network interruption or the port is not ready, and it is not due to a depletion of computing power. In order to minimize the lag perceived by Office users when continuing to write documents, the connection should be re-established as soon as possible. At this time, the backoff adjustment coefficient is set to a second value (e.g., 0.5) that is less than the first value.

[0069] When the error type parameter indicates a server error and the load status parameter does not indicate anything, the backoff adjustment coefficient is set to an intermediate value.

[0070] Specifically, when the error type parameter indicates a server-side error (such as an HTTP 5xx series error) and the load status parameter does not indicate one (i.e., there is neither Retry-After nor X-RateLimit-Remaining information in the response header), it indicates that there is an internal anomaly in the backend but no load reference is actively provided. In this case, a conservative baseline strategy is adopted, and the backoff adjustment coefficient is set to an intermediate value (e.g., 1.0), and standard backoff is performed.

[0071] When the error type parameter indicates a timeout exception, the backoff adjustment coefficient is set to a third value between the intermediate value and the first value.

[0072] Specifically, when the error type parameter indicates a timeout exception, it means that the request has been sent but no response has been received within the specified time. This usually means that the network link is congested or the backend processing queue is backlogged. It requires a longer time than the standard backoff to wait for the queue to be cleared, but does not require an extremely long cooling-off period as in the rate limiting scenario. In this case, the backoff adjustment coefficient is set to a third value (e.g., 1.5) between the intermediate value and the first value, in order to appropriately extend the backoff interval in the aggregation operation to cope with network congestion.

[0073] Step S33: Perform aggregation calculation based on the dynamic backoff interval, the backoff adjustment coefficient, and the current retry order to obtain the candidate waiting time;

[0074] Specifically, the process of performing aggregation calculations based on the dynamic backoff interval, the backoff adjustment coefficient, and the current retry order to obtain the candidate waiting time includes:

[0075] ,

[0076] Where T represents the waiting time, D represents the dynamic backoff interval, n is the current retry order, K is the backoff adjustment coefficient, and C is the dynamic exponent base, where C = 1 + K.

[0077] Specifically, in frequency-limited scenarios (such as 429 RateLimit), K takes the first value (e.g., 2.0), and the dynamic exponent base C = 1 + 2.0 = 3.0; in aggregation operations, the retry interval will increase rapidly by powers of 3 (e.g., 3... 0 ,3 1 ,3 2 Combined with the direct multiplier amplification effect of K=2.0, it can provide the backend with an extremely sufficient cooling and recovery window during peak Office hours, effectively avoiding continuous flow restriction and avalanche effects caused by insufficient backoff.

[0078] When in a connection rejection scenario (brief network interruption), K takes the second value (e.g., 0.5), in which case the dynamic exponent base C = 1 + 0.5 = 1.5; in aggregation operations, the retry interval will increase gradually as a power of 1.5 (e.g., 1.5). 0 1.5 1 1.5 2 Combined with the direct multiplier shrinking effect of K=0.5, the time interval between multiple retries is extremely compact, thereby minimizing the waiting time for Office users when faced with brief network jitter, and achieving near-imperceptible rapid recovery;

[0079] When a server-side error occurs, K takes an intermediate value (e.g., 1.0), and the dynamic exponent base C = 1 + 1.0 = 2.0; the backoff interval increases regularly with a standard binary exponent, and regular backoff is performed.

[0080] By deeply fusing the microscopic streaming features (D) extracted in step S2 with the macroscopic explicit load (K) extracted in step S3 through the above aggregation operation, not only is the starting benchmark of backoff determined, but the ramping slope of the backoff time array is also dynamically changed, and finally the candidate waiting time adapted to the severity of the current Office office scenario is output.

[0081] Step S34: Compare the candidate waiting time with the preset maximum waiting threshold, and take the smaller value as the retry waiting time.

[0082] Specifically, the candidate waiting time T is compared with the preset maximum waiting threshold, and the smaller of the two is taken as the final retry waiting time. The preset maximum waiting threshold (e.g., 5000 milliseconds) is the maximum backoff limit set in the configuration table. It is used to prevent the calculated waiting time from increasing indefinitely after extreme load or multiple failed retries, which would cause Office users to experience meaningless long-term lag. This ensures that the system can promptly provide feedback to the user or trigger failover in the event of an unrecoverable failure.

[0083] Step S4: Extract the time difference parameter between the timestamp of each record in the historical request record sequence and the current time; perform exponential decay weighted calculation on the request delay and request status based on the time difference parameter to generate a predictive health score.

[0084] Specifically, step S4 includes the following process:

[0085] Based on the request status of each record in the historical request record sequence, assign a differentiated decay constant to it;

[0086] Specifically, the differentiated attenuation constant includes a first attenuation constant. Second attenuation constant The record representing failure (such as triggering rate limiting, server error) corresponds to the first decay constant. The record indicating success corresponds to a second attenuation constant greater than the first attenuation constant. (Right now In an Office environment, failures often indicate deep overload or faults in the backend, with a long recovery period. Therefore, a smaller first decay constant causes the weight of failure records to decay more slowly over time, thus having a sustained inhibitory effect on the health score over a longer period of time. On the other hand, the weight of success records decays faster, so as to more sensitively capture the recent performance recovery status of the backend.

[0087] Calculate the time difference parameter between the timestamp of each record and the current time. Use the negative of the product of the time difference parameter and the differential decay constant as the exponent to calculate the power of the natural constant, and obtain the differential decay weight of each record.

[0088] Specifically, the time difference parameter between the timestamp of each record and the current time (e.g., the time span in seconds) is calculated. The negative of the product of this time difference parameter and the corresponding record's differential decay constant is used as the exponent to calculate the power of the natural constant, thus obtaining the differential decay weight for each record (i.e., ...). ,in, It is the attenuation constant. (Time difference).

[0089] Using the differentiated attenuation weight, the weighted average delay is calculated by weighting the request delay in the historical request record sequence, the weighted delay fluctuation coefficient is calculated by weighting the variance of the request delay, and the weighted success rate is calculated by weighting the request status.

[0090] Specifically, the weighted average latency is calculated by using the differentiated attenuation weight W to perform a weighted average calculation on the request latency in the historical request record sequence. The weighted delay fluctuation coefficient V is obtained by taking the square root of the square of the difference between the request delay and the weighted average delay. This coefficient reflects the jitter of the backend response. In the Office streaming output scenario, excessive jitter means that text generation will be stuttered. At the same time, the weighted success rate is obtained by taking the weighted average of the request status (success is recorded as 1, failure is recorded as 0).

[0091] Extract the historical health gradient of the historical request record sequence within adjacent historical time windows, perform exponential smoothing and cumulative calculation on the historical health gradient, and generate a trend momentum factor;

[0092] Specifically, the historical health gradient of the historical request record sequence is extracted within adjacent historical time windows (e.g., the previous minute and the current minute), and the historical health gradient is exponentially smoothed and accumulated to generate a trend momentum factor M. This factor is used to quantify the future evolution trend of the backend state. If the health continues to rise recently, M is positive, indicating that the backend is recovering; if it continues to decline, M is negative, providing an early warning that the backend is about to be overloaded.

[0093] The availability modulation coefficient is generated by strengthening the weighted success rate by a preset power greater than 1; the predictive health score is generated by correlating the availability modulation coefficient, the normalized reciprocal of the weighted average delay, the weighted delay fluctuation coefficient, and the trend momentum factor.

[0094] Specifically, the weighted success rate S is enhanced by raising the calculation to a preset power γ greater than 1 to generate the availability modulation coefficient. Because Office scenarios have extremely high availability requirements, the exponent γ (e.g., γ=2) amplifies the penalty for low success rates, preventing the system from routing requests to backends that are occasionally fast but frequently fail.

[0095] Specifically, the process of generating the predictive health score by performing correlation calculations based on the availability modulation coefficient, the normalized reciprocal of the weighted average delay, the weighted delay fluctuation coefficient, and the trend momentum factor includes:

[0096] ,

[0097] Where H is the predictive health score, S is the weighted success rate, L is the normalized value of the weighted average delay, V is the weighted delay volatility coefficient, and M is the trend momentum factor. and γ is a preset weighting coefficient, and γ is a preset exponent greater than 1.

[0098] Specifically, in desktop Office add-in scenarios, availability is the top priority. A backend that is occasionally fast but frequently fails will cause repeated interruptions in document writing, resulting in a terrible user experience. Therefore, the introduction of the exponent γ amplifies the penalty for low success rates (e.g., when S=0.6, S...). 2 =0.36, the score was drastically reduced), effectively preventing the system from routing requests to a backend that is "occasionally fast but frequently fails" during failover, ensuring that the switch target has an absolute availability baseline; calculation ;in, This represents the reciprocal of the weighted average delay normalized value; the lower the delay, the higher the score for this item. As a penalty term for the weighted latency fluctuation coefficient, in streaming inference scenarios, an excessively large fluctuation coefficient V indicates that the backend is in a state of computing power preemption or unstable scheduling, which will cause stuttering in the Office interface.

[0099] The larger the value of V, the lower the score for that sub-item. Used to adjust the weights of the overall module for latency and stability. As a stability bias constant, it ensures that the module still has a reasonable benchmark score when the fluctuation is extremely small (V≈0), avoiding singular values ​​in the scoring formula; calculation The trend momentum factor M is calculated exponentially based on the health gradient of adjacent time windows and represents the future evolution trend of the backend health status. When M>0, it indicates that the backend is in the recovery and rise phase (such as just ending the rate limiting). At this time, (1+M)>1 positively amplifies the health score and encourages the system to tilt requests to the backend that is recovering. When M<0, it indicates that although the backend may not have completely collapsed yet, various indicators are continuously deteriorating (such as the queue backlog is intensifying). At this time, (1+M)<1 suppresses the health score in advance, so that the system can avoid the backend "truly crashing" in advance and achieve forward-looking intelligent failover.

[0100] Step S5: Based on the predictive health score, the candidate inference backends are sorted and selected to perform failover, and the failed backends are probed at a probe interval that increases exponentially with the number of probe failures to update the backend availability status.

[0101] Specifically, when the currently active inference backend fails after exhausting the dynamic adaptive backoff retries, intelligent failover is triggered. A set of candidate inference backends is obtained from the fallback list in the configuration table, first excluding backends already marked as "temporarily unavailable". For the remaining candidate inference backends, their predictive health scores H are calculated in real-time using the method described in step S4, and sorted in descending order of H values. The candidate inference backend with the highest H value is selected as the optimal failover target, and the runtime protocol stack and connection are replaced through the model hot-switching engine, transferring the subsequent A... The system requests routing to the optimal transfer target while simultaneously marking the original faulty backend as "temporarily unavailable." Through this mechanism, in heterogeneous multi-backend environments (such as local Ollam and cloud-based DeepSeek coexisting), the system can accurately avoid nodes experiencing degradation, ensuring a seamless experience for continuing Office document writing. For faulty backends marked "temporarily unavailable," an adaptive liveness detection process is initiated, sending lightweight liveness detection requests (such as model list queries or minimalist prompt requests). Based on the number of consecutive failed detections, the time interval for the next liveness detection is dynamically calculated using the following formula: ;in, For the next liveness detection interval, The preset basic liveness detection interval (e.g., 5 seconds). This represents the cumulative number of consecutive failed probes. During peak Office usage periods, backend crashes due to overload often require a long recovery period. This exponentially increasing mechanism effectively avoids sending frequent invalid probe requests while the fault remains unresolved, saving local network and computing resources for the plugin and preventing invalid probes from triggering secondary rate-limiting penalties on the cloud API. If the probe request returns successfully, it is determined that the faulty backend has recovered, its availability status is updated to "normal," and its consecutive probe failure count is reset. The value is set to 0, and the device is re-added to the healthy backend pool for subsequent failover scoring and ranking; if the liveness probe request fails again, the number of consecutive probe failures is incremented. And the detection interval increases exponentially according to the calculated result. Schedule the next detection until the maximum detection interval limit is reached or the detection is successful.

[0102] Specifically, this invention constructs a multi-dimensional perception base by integrating real-time streaming data, historical time series, and error types to eliminate single-point information blind spots; it dynamically lengthens the backoff interval by utilizing the decrease in token generation rate, giving congested backends breathing time to digest backlogged requests and effectively preventing request avalanches from overwhelming the system; it tailors retry waiting times according to error type and load information, with long wait times for rate limiting overload and short retries for connection interruptions, neither adding to the problems of unhealthy backends nor causing users to wait idly; it uses time decay to calculate health scores, allowing the negative impact of historical faults to naturally fade over time, making the score truly reflect the actual performance of the backend and improving routing accuracy; based on accurate scores, it directs requests to the optimal node and probes faulty nodes at gradually increasing intervals, significantly reducing the resource overhead caused by invalid probes, while ensuring that traffic can be promptly recovered when nodes recover, balancing high availability and low consumption of the system.

[0103] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fault-tolerant routing method for interfaces based on stream sensing and attenuation scoring, characterized in that, include: Step S1: Obtain the streaming inference response data of the currently active inference backend, the historical request record sequence of the candidate inference backend, and the error type parameter of the current request. The historical request record sequence includes timestamps, request delays, and request status. Step S2: Extract features from the streaming inference response data to obtain the real-time token generation rate, and perform exponential mapping correction on the basic backoff interval based on the deviation between the real-time token generation rate and the preset overload threshold to obtain the dynamic backoff interval. Step S3: Obtain the load indication information of the current request, determine the basic load factor according to the error type parameter and the load indication information, and perform differential aggregation calculation based on the dynamic backoff interval and the basic load factor to obtain the retry waiting time; Step S4: Extract the time difference parameter between the timestamp of each record in the historical request record sequence and the current time; perform exponential decay weighted calculation on the request delay and request status based on the time difference parameter to generate a predictive health score. Step S5: Based on the predictive health score, the candidate inference backends are sorted and selected to perform failover, and the failed backends are probed at a probe interval that increases exponentially with the number of probe failures to update the backend availability status.

2. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 1, characterized in that, The process of step S2 includes: The number of tokens received in the streaming inference response data is counted according to a preset sliding time window, and the real-time token generation rate is calculated. The difference between the preset overload threshold and the real-time token generation rate is calculated as the deviation value; The product of the deviation value and the preset scaling factor is used as an exponent for exponential operation to obtain the micro-avoidance factor; The product of the micro-avoidance factor and the basic avoidance interval is determined as the dynamic avoidance interval.

3. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 2, characterized in that, The process of step S3 includes: Extract the load status parameters returned by the server in the current request; Determine the backoff adjustment coefficient that matches the current abnormal scenario based on the error type parameter and the load status parameter; Based on the dynamic backoff interval, the backoff adjustment coefficient, and the current retry order, an aggregation operation is performed to obtain the candidate waiting time; The candidate waiting time is compared with a preset maximum waiting threshold, and the smaller value is taken as the retry waiting time.

4. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 3, characterized in that, The process of determining the backoff adjustment coefficient matching the current abnormal scenario based on the error type parameter and the load state parameter includes: When the error type parameter indicates a frequency limit error, and the load status parameter contains a valid retry wait time indication or the remaining request quota is zero, the backoff adjustment coefficient is set to a first value. When the error type parameter indicates a connection rejection, the backoff adjustment coefficient is set to a second value that is less than the first value; When the error type parameter indicates a server error and the load status parameter does not indicate anything, the backoff adjustment coefficient is set to an intermediate value. When the error type parameter indicates a timeout exception, the backoff adjustment coefficient is set to a third value between the intermediate value and the first value.

5. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 4, characterized in that, The process of performing aggregation calculations based on the dynamic backoff interval, the backoff adjustment coefficient, and the current retry order to obtain candidate waiting times includes: ， Where T represents the waiting time, D represents the dynamic backoff interval, n is the current retry order, K is the backoff adjustment coefficient, and C is the dynamic exponent base, where C = 1 + K.

6. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 5, characterized in that, The process of step S4 includes: Based on the request status of each record in the historical request record sequence, assign a differentiated decay constant to it; Calculate the time difference parameter between the timestamp of each record and the current time. Use the negative of the product of the time difference parameter and the differential decay constant as the exponent to calculate the power of the natural constant, and obtain the differential decay weight of each record. Using the differentiated attenuation weight, the weighted average delay is calculated by weighting the request delay in the historical request record sequence, the weighted delay fluctuation coefficient is calculated by weighting the variance of the request delay, and the weighted success rate is calculated by weighting the request status. Extract the historical health gradient of the historical request record sequence within adjacent historical time windows, perform exponential smoothing and cumulative calculation on the historical health gradient, and generate a trend momentum factor; The availability modulation coefficient is generated by strengthening the weighted success rate by a preset power greater than 1; the predictive health score is generated by correlating the availability modulation coefficient, the normalized reciprocal of the weighted average delay, the weighted delay fluctuation coefficient, and the trend momentum factor.

7. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 6, characterized in that, The differentiated attenuation constant includes a first attenuation constant and a second attenuation constant. Records that represent failures correspond to the first attenuation constant, while records that represent successes correspond to the second attenuation constant, which is greater than the first attenuation constant.

8. The interface fault-tolerant routing method based on stream sensing and attenuation scoring according to claim 7, characterized in that, The process of generating the predictive health score by performing correlation calculations based on the availability modulation coefficient, the normalized reciprocal of the weighted average delay, the weighted delay fluctuation coefficient, and the trend momentum factor includes: ， Where H is the predictive health score, S is the weighted success rate, L is the normalized value of the weighted average delay, V is the weighted delay volatility coefficient, and M is the trend momentum factor. and γ is a preset weighting coefficient, and γ is a preset exponent greater than 1.

Images