Request failover method and system under distributed AI reasoning scene

Abstract

The invention relates to the technical field of AI reasoning, and provides a request failover method for a distributed AI reasoning scene, which comprises the following steps: after receiving a user reasoning request by an API (Application Program Interface) service, forwarding the request to a reasoning engine instance through a gateway, encoding an original text prompt into input tokens, executing Prefilling processing, and entering a Decoding stage to continuously generate tokens after a result is generated. In the process, the system detects the running state of the instance in real time, if reasoning interruption is caused by network fluctuation, hardware faults or video memory OOM or other abnormalities, generated tokens can be obtained and spliced with original input tokens into newininput tokens, and the newininput tokens are forwarded to the healthy instance through a scheduling module. According to the new instance, Prefilling is carried out on the newputtokens by utilizing a KV cache technology, the remaining tokens are continuously pushed based on a Prefilling result, and then the newly generated tokens are decoded into a text chunk and are returned to a client in a streaming manner until an inference session is completed. According to the method, user perception reasoning abnormity is avoided, retry time consumption is reduced, the service SLA is effectively improved, and continuity and high efficiency of the reasoning process are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of AI reasoning, and in particular to a request failover method and system in a distributed AI reasoning scenario. Background Art

[0002] In a distributed scenario, the LLM inference gateway is located on the central cloud, while the LLM inference instances running on GPUs are located in distributed data centers. Therefore, a user inference request must first be received by the gateway and then forwarded to the corresponding LLM inference instance. The processing time for an inference request typically ranges from a few seconds to tens of seconds, or even minutes. During this time, network fluctuations between the central cloud and the distributed data centers may cause inference interruptions. Furthermore, hardware failures, inference engine bugs, and memory out-of-memory (OOM) errors can cause the inference process to terminate abnormally. This significantly impacts the user experience, forcing the user to either abandon the inference process or repeatedly retry until the inference request is successfully processed.

[0003] The LLM reasoning process generally consists of two phases: prefilling and decoding. In a distributed environment, user-requested reasoning may be interrupted midway due to various anomalies.

[0004] For example, due to network complexity and fluctuations, the inference process may occasionally be interrupted during the Prefilling or Decoding phases. Prefilling takes a relatively short time, typically a few hundred milliseconds to a few seconds, so the probability of an interruption is low. However, Decoding takes a longer time span, typically tens of seconds or even minutes. Therefore, the probability of encountering an exception during the Decoding phase is slightly higher than during the Prefilling phase. Whether during the Prefilling or Decoding phase, network fluctuations or inference instance anomalies can cause the entire inference request to be interrupted. For example, if an inference engine instance experiences a failure during the token return process (including network failure, instance failure, hardware failure, and so on), it will be unable to return the generated content. The API service will detect this and notify the client through an exception code.

[0005] The traditional approach to these exceptions is to simply retry. This means requiring the client to re-initiate the inference request so that the new inference request can be dispatched to a healthy inference instance. While this is logically feasible, it has many drawbacks: (1) For the client, it is necessary to add exception detection and retry logic (2) Retrying means that the previously partially completed inference calculation results (including tokens generated by Prefilling and partial Decoding) are discarded, resulting in the need to recalculate these tokens for subsequent re-inference.

[0006] (3) This significantly increases the overall inference latency. For example, a normal inference request takes 20 seconds to complete, but an error occurs at 19 seconds. Retrying the request requires the entire inference process to be repeated, which takes another 20 seconds. Therefore, from the user's perspective, the total inference time is 19 seconds + 20 seconds = 39 seconds. This significantly degrades the user experience. Summary of the Invention

[0007] To address the above issues, the present invention aims to provide a method and system for request failover in distributed AI reasoning scenarios. This method cleverly utilizes the two-stage prefilling and decoding process of LLM reasoning to combine the inferred tokens from the first failed reasoning with the original input to form new input tokens, which are then passed to the second instance of reasoning to achieve a continued reasoning effect. The breakthrough of this invention lies in not only avoiding the user's perception of anomalies during the reasoning process, but also avoiding the time consumption caused by users re-initiating reasoning requests, and further improving service level agreements.

[0008] The above-mentioned object of the present invention is achieved through the following technical solutions: A method for request failover in a distributed AI inference scenario includes the following steps: S1: The API service receives an inference request from a user client and forwards it to the inference engine instance through the gateway. The service then encodes the input prompt using the tokenizer to generate a sequence of input_tokens. The service then performs prefilling, generating a prefill result and a unique request identifier (request_id). The service then enters the decoding phase to continuously generate tokens. S2: In the Decoding phase, the running status of the inference engine instance is detected in real time. When an abnormal situation including network fluctuation, inference engine instance abnormality, hardware failure or video memory OOM is detected, resulting in inference interruption, the generated tokens are obtained; S3: The input_tokens sequence is spliced ​​with the generated tokens to form new new_input_tokens; S4: The new_input_tokens are dispatched to the healthy inference engine instance through the scheduling module in the API service; S5: The healthy inference engine instance performs Prefilling processing on the new_input_tokens, uses KV cache technology to avoid repeated calculations, and continues to execute the Decoding phase based on the Prefill results to generate the remaining tokens; S6: The newly generated tokens are decoded into text chunks through the tokenizer and returned to the client until all tokens are generated and the inference session is completed.

[0009] Furthermore, in step S1, the Prefilling process is performed to generate the Prefill result and request_id, specifically: All tokens in the input_tokens sequence are processed in parallel to generate attention states and hidden states, and a KV cache is constructed and stored in the session state. The KV cache is used to reuse historical calculation results in the Decoding stage. This processing process is computationally intensive, and the Prefilling stage takes less time than the Decoding stage.

[0010] Furthermore, in step S2, the operating status of the inference engine instance is detected in real time, specifically: When detecting inference interruptions through exception codes, the detection mechanism is integrated into the API service layer. It can identify network fluctuations, inference engine instance anomalies, hardware failures, or video memory OOM of the inference engine instance, and record the generated token sequence based on the token generation progress at the time of the interruption. The detection and recording process does not rely on additional client logic.

[0011] Furthermore, in step S3, the input_tokens sequence is concatenated with the generated tokens to form new_input_tokens, specifically: When storing generated tokens through session state, the generation order of tokens and the update status of the KV cache are recorded synchronously. When splicing, the input_tokens and the generated tokens are merged in the order of the original input + historical generation sequence to form a new new_input_tokens containing complete context dependencies, avoiding repeated calculation of the attention status of the generated tokens.

[0012] Furthermore, in step S4, the new_input_tokens are dispatched to the healthy inference engine instance through the dispatch module in the API service, specifically: The scheduling module screens similar inference instances through a health check mechanism, gives priority to the inference engine instances with sufficient video memory resources and balanced load, and realizes seamless transfer of request context through hot migration technology. The scheduling process takes time controlled in milliseconds and does not affect the real-time interaction of the client.

[0013] Furthermore, in step S5, the healthy inference engine instance performs Prefilling processing on new_input_tokens, specifically: When the new inference engine instance performs the Prefilling process, it builds an incremental KV cache based on new_input_tokens and only calculates the attention state and hidden state of the newly added tokens; During the decoding phase, when the remaining tokens are pushed through the autoregressive method, the combined data of the historical KV cache and the newly added KV cache are reused to achieve seamless continuation of the inference process. The continued push process does not repeatedly calculate the hidden states of the generated tokens.

[0014] Furthermore, in step S6, the newly generated tokens are decoded into text chunks by the tokenizer and returned to the client until all tokens are generated and the inference session is completed. Specifically: The newly generated tokens are decoded into text chunks by the tokenizer through streaming, and are returned to the client in the order they were generated. The text sequence is consistent with the timing of the normal reasoning process. The new reasoning engine instance maintains the KV cache state based on the spliced ​​new_input_tokens to ensure that the semantic coherence of the decoded text chunk is consistent with the normal reasoning process, completing the reasoning session closed loop.

[0015] A request failover system in a distributed AI inference scenario using the request failover method in the distributed AI inference scenario described above includes: In the request initialization and preprocessing module, the API service receives inference requests initiated by the user client, forwards the inference requests to the inference engine instance through the gateway, encodes the input original text prompt through the tokenizer to obtain the input_tokens sequence, performs prefilling processing, generates the prefill result and the request unique identifier request_id, and then enters the decoding phase to continuously generate tokens; The running status detection and exception capture module detects the running status of the inference engine instance in real time during the decoding phase. When an abnormality such as network fluctuation, inference engine instance abnormality, hardware failure, or memory OOM is detected, resulting in an interruption of inference, the module obtains the generated tokens. The input data reorganization module concatenates the input_tokens sequence with the generated tokens to form new_input_tokens; An instance scheduling and migration module, which schedules new_input_tokens to healthy inference engine instances through the scheduling module in the API service; Inference continuation and cache reuse module: A healthy inference engine instance performs prefilling on new_input_tokens, uses KV caching technology to avoid repeated calculations, and continues to execute the decoding phase based on the prefill results to generate the remaining tokens; The result decoding and session closure module decodes the newly generated tokens into text chunks through the tokenizer and returns them to the client until all tokens are generated and the inference session is completed.

[0016] A computer device includes a memory and one or more processors, wherein the memory stores computer code, and when the computer code is executed by the one or more processors, the one or more processors execute the above method.

[0017] A computer-readable storage medium stores computer code. When the computer code is executed, the above method is performed.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) User experience optimization Imperceptible failure: When an inference request fails, the system automatically loads the request to a new inference instance. Users are unaware of the anomaly during the inference process, and the inference experience is consistent with normal successful inference.

[0019] No additional client logic is required: Users do not need to add exception handling and retry logic for failures in requests, reducing usage complexity.

[0020] (2) Improved computing efficiency Preserve valid inference results: Avoid discarding intermediate results of failed inferences (such as generated tokens and prefilling calculation results) in traditional solutions. Only calculate the parts that were not completed due to exceptions, eliminating the need for repeated inference calculations.

[0021] No increase in the total inference time: During failover, technologies such as KV caching and input splicing are reused to ensure that the total inference time is not significantly different from the original time.

[0022] (3) Enhanced service reliability Improve request success rate: Automatically transfer faulty requests to healthy instances for continued reasoning, reducing request failures due to exceptions and significantly improving request success rate.

[0023] Optimized SLA indicators: The system automatically handles instance exceptions and completes fault migration, significantly improving service-level agreement (SLA) indicators from the user's perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the normal reasoning process of the present invention; Figure 2 This is a schematic diagram of an abnormality in the decoding stage of the present invention; Figure 3 This is a schematic diagram of the traditional exception handling method of the present invention; Figure 4 This is a flow chart of the new exception handling method of the present invention; Figure 5 A data flow diagram of the novel exception handling method of the present invention; Figure 6 This is an overall flow chart of the request failover method in the distributed AI reasoning scenario of the present invention; Figure 7 This is the overall structure diagram of the request failover system in the distributed AI reasoning scenario of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0027] This invention attempts to leverage the internal principles of LLM reasoning to automatically redirect the request to a new reasoning instance when a user's reasoning request fails, preventing the user from noticing an anomaly during the reasoning process. Furthermore, the reasoning result remains the same as a normal successful reasoning.

[0028] In the LLM reasoning process, there are generally two stages: Prefilling and Decoding.

[0029] (1) Prefilling is the first stage of LLM reasoning, which mainly includes: Processing input context: In this phase, the model processes the entire input text (prompt) provided by the user; Parallel computing: The model processes all tokens in the input simultaneously and generates attention states and hidden states for each token; Build a KV cache: Calculate key-value pairs for each token in the input sequence and store them in the KV cache in preparation for the subsequent decoding stage; Computationally intensive: Since a large number of tokens need to be processed in parallel, this stage is usually computationally intensive, especially when the input is long.

[0030] (2) Decoding is the second stage of LLM reasoning, which occurs after Prefilling: Autoregressive generation: The model generates one token at a time, and each new token depends on all previously generated tokens; Leverage KV caching: Use the KV cache built in the Prefilling phase to avoid recalculating the representation of previous tokens; Iterative process: Each time a new token is generated, it is added to the sequence, the KV cache is updated, and then the next token is generated; Memory intensive: This phase is usually memory intensive due to the need to store and access the KV cache. Sampling strategy: Various sampling strategies (such as temperature sampling, top-k, top-p, etc.) are applied at this stage to determine the next token.

[0031] Therefore, a normal reasoning process can be simply expressed as Figure 1 express.

[0032] In an actual distributed environment, the reasoning requested by the user may be terminated halfway due to various abnormal factors.

[0033] For example, due to the complexity and fluctuation of the network, the inference process may occasionally be interrupted during the Prefilling or Decoding phase.

[0034] Prefilling takes a relatively short time, typically a few hundred milliseconds to a few seconds, so the probability of anomalies is low. However, decoding takes a longer time span, typically tens of seconds or even minutes. Therefore, the probability of encountering anomalies during the decoding phase is slightly higher than during the prefilling phase.

[0035] Whether in the Prefilling or Decoding phase, if network fluctuations or inference instance exceptions occur during this period, the entire inference request will be abnormally interrupted.

[0036] Take the exception in the Decoding stage as an example, Figure 2 If an inference engine instance experiences a failure (including network failure, instance failure, hardware failure, etc.) while returning a token, it will be unable to return the generated content. The API service will detect this and notify the client through an exception code.

[0037] The traditional approach to these exceptions is to simply retry. This means requiring the client to re-initiate the inference request so that the new inference request can be dispatched to a healthy inference instance. While this is logically feasible, it has many drawbacks: (1) For the client, it is necessary to add exception detection and retry logic (2) Retrying means that the previously partially completed inference calculation results (including tokens generated by Prefilling and partial Decoding) are discarded, resulting in the need to recalculate these tokens for subsequent re-inference.

[0038] (3) This results in a significant increase in the overall inference latency. For example, a normal inference request takes 20 seconds to complete, but an error occurs at 19 seconds. At this point, after retrying, the entire inference process needs to be repeated, which takes 20 seconds. Therefore, from the user's perspective, this inference takes a total of 19 seconds + 20 seconds = 39 seconds. The user experience is greatly reduced. This process is as follows Figure 3 shown.

[0039] Based on the above-mentioned abnormal situation, this paper proposes a new solution. Instead of retrying at the user request level, the inference results of the current request are retained. Based on the principle of LLM reasoning, the original input is combined with the inferred token to form a new input. The new input is then scheduled to a new inference instance to continue the inference.

[0040] This method cleverly preserves the valid inference results of the current exception reasoning and avoids the need to recalculate these inference results, saving time. It also allows the reasoning transferred to the new reasoning instance to fully resume at the previously interrupted reasoning point.

[0041] From the client user's perspective, the entire inference process is not interrupted and does not fail. The user experience is exactly the same as a complete and successful inference process. Figure 4 and 5 shown.

[0042] Compared with the traditional request-level retry solution, this invention has the following advantages: (1) Efficient switching: quickly transferring the failure to the new instance without significantly increasing the total inference time of user requests.

[0043] (2) Users do not need to add additional exception handling logic and retry logic for failures in requests.

[0044] (3) Improve SLA. Since the system automatically performs fault migration when an instance is abnormal, the SLA perceived by users has been greatly improved.

[0045] Of course, the present invention also has some prerequisites: (1) There are multiple similar reasoning instances in the system, and at least one instance is healthy and serviceable.

[0046] (2) In the API service, there needs to be a scheduling module that is responsible for immediately scheduling requests to another healthy instance when an instance fails.

[0047] The technical solution of the present invention is described below through specific embodiments: First embodiment like Figure 6 As shown, this embodiment provides a request failover method in a distributed AI reasoning scenario, including the following steps: S1: The API service receives the inference request initiated by the user client, forwards the inference request to the inference engine instance through the gateway, encodes the input original text prompt through the tokenizer to obtain the input_tokens sequence, and performs prefilling processing to generate the prefill result and the request unique identifier request_id. Then it enters the decoding phase to continuously generate tokens.

[0048] In this embodiment, step S1 is specifically as follows: All tokens in the input_tokens sequence are processed in parallel to generate attention states and hidden states, and a KV cache is constructed and stored in the session state. The KV cache is used to reuse historical calculation results in the Decoding stage. This processing process is computationally intensive, and the Prefilling stage takes less time than the Decoding stage.

[0049] In distributed AI inference scenarios, step S1 is the core initialization step of the inference process. Essentially, it builds the basic computing environment required for inference through preprocessing. Specifically, when a user client initiates an inference request to the API service (for example, inputting "write a popular science article about quantum computing"), the API service first forwards the request through a gateway to the inference engine instance in the distributed data center. The system then performs token segmentation on the raw text prompt, using, for example, a BPE tokenizer to convert the text into a sequence of input_tokens such as ["write", "a", "about", "quantum", "computing", "of", "popular science", "article"]. The inference engine instance then performs prefilling on all input_tokens, computing the attention state (key) and hidden state (value) of each token in parallel using the GPU. These intermediate results are then constructed into a key-value cache and stored in the session state. Finally, a prefill result containing a reference to the key-value cache and a unique request identifier (request_id) is generated, laying the foundation for token generation in the subsequent decoding phase.

[0050] The core technology of the Prefilling stage lies in the combination of parallel computing and the KV cache mechanism. Because this stage requires processing the attention mechanism calculations for all input_tokens simultaneously, it is a typical computationally intensive operation. Taking the processing of 1024 tokens as an example, GPU parallel computing can complete the generation of the K / V states of all tokens in 1-2 seconds, while serial computing may take hundreds of times longer. The constructed KV cache is essentially a "memory" of historical calculation results. For example, when the input_tokens is "The weather is sunny today", the K / V state of each word (token) will record its semantic association information in the context. This design means that when a new token is generated in the Decoding stage, there is no need to repeatedly calculate the states of previously processed tokens such as "today" and "weather". Instead, it can be directly read and reused from the KV cache, greatly improving inference efficiency. In terms of time consumption, the prefilling stage usually takes only a few hundred milliseconds to several seconds, while the decoding stage requires token-by-token autoregressive generation (for example, generating 1,000 tokens takes about 50 milliseconds each), so the total time consumption often reaches tens of seconds or even longer. The difference in time consumption between the two is significant.

[0051] For example, consider the inference request "Generate AI development trend analysis." Assume the input prompt is segmented into 200 input_tokens. In the S1 phase, the inference engine instance uses the GPU to parallelize the key-value (K / V) states of these 200 tokens, build a KV cache, and store them. The entire prefilling process takes approximately 0.8 seconds. The system then enters the decoding phase, where it generates analysis token by token based on the KV cache. For example, it might output, "Driven by breakthroughs in computing power and data accumulation, AI technology is ushering in its third wave of development..." If the inference instance experiences a network failure while generating the 150th token, the KV cache for the previous 150 tokens will be retained. After failing over to a new instance, the new instance receives a concatenation of the original 200 input tokens and the 150 generated tokens. Since the Prefilling phase only requires calculating the K / V status of the newly added 150 tokens and reusing the KV cache of the original 200 tokens, the time taken can be controlled within approximately 0.5 seconds, which is nearly 60% less than the time taken to recalculate 350 tokens in the traditional solution (approximately 1.4 seconds), fully demonstrating the efficiency improvement effect of KV cache reuse.

[0052] S2: In the decoding phase, the running status of the inference engine instance is detected in real time. When an abnormal situation such as network fluctuation, inference engine instance abnormality, hardware failure or video memory OOM is detected, resulting in inference interruption, the generated tokens are obtained.

[0053] In this embodiment, step S2 is specifically as follows: When detecting inference interruptions through exception codes, the detection mechanism is integrated into the API service layer. It can identify network fluctuations, inference engine instance anomalies, hardware failures, or video memory OOM of the inference engine instance, and record the generated token sequence based on the token generation progress at the time of the interruption. The detection and recording process does not rely on additional client logic.

[0054] During the decoding phase of distributed AI inference, step S2 is responsible for real-time monitoring of inference instance status and detecting faults. As the system generates science fiction text token by token based on the KV cache generated in step S1 (e.g., continuing the plot from "In 2077, humanity first discovered faster-than-light travel"), the API service layer continuously monitors the inference engine instance using a built-in detection module. This detection mechanism is non-intrusively integrated into the underlying API service, requiring no modifications to client code or inference engine logic. It automatically identifies four key anomalies: network link timeouts between the central cloud and distributed data centers (e.g., TCP connection interruptions), inference engine process crashes due to software bugs, computational errors caused by GPU hardware failure, and out-of-memory (OOM) events caused by exceeding memory usage thresholds. Upon detecting an anomaly, the system immediately identifies the fault type using a pre-set exception code (e.g., 404 for network failures, 503 for memory OOM events). A progress counter is generated based on tokens (e.g., 300 tokens have been generated) to extract the complete token sequence, ensuring that inference context is preserved during subsequent failovers.

[0055] The technical characteristics of this detection mechanism are its fully automated and client-invisible design. With the API service layer as the monitoring hub, it monitors instance status by sending heartbeat packets every 50 milliseconds. If three consecutive heartbeat packets go unanswered or an exception code is received, the fault capture process is immediately triggered. For example, during the decoding phase of generating a science fiction novel, if an inference instance encounters an out-of-memory (OOM) error due to processing a complex scenario (such as a quantum physics calculation involving a spaceship crossing a black hole's event horizon), the API service layer will receive the OOM exception code returned by the instance in real time, extract the generated token sequence from the session state (e.g., "The spaceship's engines emit a sharp beep, and the spacetime structure outside the window takes on a twisted spiral..."), and simultaneously record the corresponding KV cache state. The entire detection and recording process takes less than 100 milliseconds, requiring no additional exception handling logic on the client side (e.g., the novel creation terminal); it is completely automated by the underlying system.

[0056] Taking the S1 stage of science fiction novel generation as an example, suppose a user input prompt enters the Decoding stage after S1 processing, generating 200 tokens of text (e.g., "Astronaut Li Ming enters a command in the console. The antimatter engine begins charging, and a three-dimensional star map of the target galaxy appears on the bridge screen..."). If the GPU node hosting the inference instance crashes due to hardware overheating, the detection module at the API service layer detects the failure through the server heartbeat interrupt signal and the instance disconnection exception code (e.g., ECONNREFUSED). It then extracts the sequence of 200 generated tokens from the session state and marks the location of the interruption (the 200th token). Because the detection mechanism is integrated into the API service layer, client-side authoring tools do not need to write retry logic. The system automatically saves the inference state at the time of the interruption, providing complete context for the subsequent S3 step to splice in new input. This ensures that after failover, the inference can seamlessly resume from the subsequent plot of "Star Map" without recalculating the semantic representation of the 200 generated tokens. S3: Concatenate the input_tokens sequence with the generated tokens to form new_input_tokens.

[0057] In this embodiment, step S3 is specifically as follows: When storing generated tokens through session state, the generation order of tokens and the update status of the KV cache are recorded synchronously. When splicing, the input_tokens and the generated tokens are merged in the order of the original input + historical generation sequence to form a new new_input_tokens containing complete context dependencies, avoiding repeated calculation of the attention status of the generated tokens.

[0058] The core of the S3 step is to semantically coherently concatenate the original input sequence with the sequence of tokens generated at the time of the failure, forming a new input token that can be continued. In its implementation, the system uses session state to store the generation order of generated tokens (e.g., the time sequence from the 1st to the Nth token) and incremental updates to the KV cache (e.g., the attention state and hidden state corresponding to each token) in real time. The concatenation strictly adheres to the order of "original input tokens + historically generated tokens." For example, if the original input is ["in", "2077", ",","human"] and the generated tokens are ["for the first time", "discovered", "achieved", "faster than the speed of light"], then the new input tokens will be ["in", "2077", ",",","human", "for the first time", "discovered", "achieved", "faster than the speed of light"]. This concatenation ensures that the new input sequence retains complete contextual dependencies, preventing repeated calculation of the attention state of previously generated tokens in new inference instances.

[0059] The session state storage module records three types of key information to support splicing operations: (1) Tokens generation order index: Mark the position of each generated token in the sequence (such as the 150th token is "engine") to ensure accurate splicing timing; (2) KV cache update log: records the new content of the KV cache after each token is generated (such as the key-value pair corresponding to the generation of "faster than light speed"), so that new instances can reuse historical calculation results; (3) Context-dependent tagging: This tagging method labels the semantic associations between generated tokens and the original input (e.g., the attention weights for “faster than the speed of light” and “navigation technology”). When an inference instance fails, the new instance can directly reuse the KV cache of the original input_tokens (such as the pre-populated result described above) and the KV cache increment of the generated tokens by reading the session state, without having to recalculate the intermediate state of the first N tokens, significantly reducing computing power consumption.

[0060] Taking the science fiction novel generation in the S1 stage as an example, the user input prompt is "In 2077, humanity first discovered faster-than-light travel technology, and astronaut Li Ming embarked on a journey to Cygnus X-1." After processing in S1, the input_tokens sequence is generated. In the decoding stage, the text of the first 200 tokens is output: "...As the spacecraft passes through the wormhole, the star trails outside the porthole are stretched into golden bands of light. Li Ming enters commands on the console, and the antimatter engine begins charging..." If the inference instance is interrupted due to a hardware failure at this time, the system obtains the 200 generated tokens and their key-value cache status through S2 and performs the following operations in S3: (1) Extraction of original input_tokens: Read the prompt-encoded token sequence from the session state, such as ["in","2077","",","humanity","first","discovered","superseded the speed of light","of","navigation technology",...]; (2) Append generated tokens: Append the 200 tokens generated in the decoding phase (such as ["spaceship", "travel", "wormhole", "time", ..., "antimatter", "engine", "start", "charge"]) to the original input in the order of generation; (3) Formation of new_input_tokens: After the new instance receives the concatenated sequence, it only needs to perform prefilling on the tokens to be generated later (such as "a three-dimensional star map of the target galaxy is displayed on the bridge screen"), reusing the KV cache of the first 200 tokens to avoid repeated calculation of the attention state of content such as "the antimatter engine starts charging", and the subsequent push efficiency is improved by about 60%.

[0061] This splicing mechanism ensures that when the new instance processes the "bridge screen," it can directly access the hidden state of the "antimatter engine" through the KV cache, maintaining semantic coherence between "engine charging" and "screen display," and avoiding the context gap caused by discarding historical calculation results in traditional retry solutions. S4: The dispatch module in the API service dispatches new_input_tokens to the healthy inference engine instance.

[0062] In this embodiment, step S4 is specifically as follows: The scheduling module screens similar inference instances through a health check mechanism, gives priority to the inference engine instances with sufficient video memory resources and balanced load, and realizes seamless transfer of request context through hot migration technology. The scheduling process takes time controlled in milliseconds and does not affect the real-time interaction of the client.

[0063] (1) Functional positioning and health check mechanism of the scheduling module The core of step S4 is to implement failover of inference tasks through the scheduling module in the API service. Its essence is an intelligent resource allocation process. The scheduling module has a built-in health check mechanism to monitor the operating status of all inference engine instances in the distributed computer room in real time, including: Hardware resource status: video memory usage (less than 80% is considered sufficient), GPU core temperature, memory utilization, etc. Service load indicators: number of requests currently being processed, average response time, instance throughput, etc. Network connectivity: Network latency and packet loss between the central cloud and instances. When a failure is detected in an inference instance (such as a network outage or memory out-of-memory error), the scheduling module immediately selects candidate targets from the list of healthy instances to ensure that the new instance has the resources to take over the inference task.

[0064] (2) Target instance screening strategy and priority mechanism The scheduling module selects the target instance based on the principle of "resource priority and load balancing": Prioritize video memory resources: Prioritize instances with a larger available video memory space than the KV cache capacity required by the current task (for example, if a task requires 10GB of video memory, the instance must have at least 12GB of available video memory) to avoid OOM failures during subsequent pushes. Load balancing strategy: Use weighted round-robin or minimum connection number algorithms to select instances with the fewest requests currently being processed and low CPU / GPU utilization to ensure balanced task distribution. Isomorphism requirement: Select instances with the same hardware configuration as the original faulty instance (such as the same GPU model and memory capacity) to avoid inconsistent inference results due to hardware differences. For example, if the original instance is an A100 GPU, the scheduling module will prioritize nodes with sufficient graphics memory and low load from other A100 instances to ensure computing power matching.

[0065] (3) Hot migration technology and millisecond-level scheduling implementation The scheduling module uses hot migration technology to achieve seamless transfer of request context, including: State data packaging: Package new_input_tokens, KV cache status of generated tokens, inference progress index, and other data into a migration package; Zero-interruption transmission: Use high-speed network channels (such as RDMA) to transmit the migration package to the target instance, maintaining the client connection during the transmission process; Fast loading mechanism: After receiving the migration package, the target instance directly reuses the KV cache data, eliminating the need to recalculate the attention state of previously generated tokens, ensuring that push resumes immediately from the breakpoint. The entire scheduling process takes milliseconds (e.g., 50-100ms). Clients (such as science fiction writing tools) are barely aware of the migration process and continue to receive streamed text chunks, without affecting the real-time interactive experience.

[0066] (4) Scheduling examples of science fiction scene generation Taking the science fiction generation in the S1 stage as an example, when the inference instance generates the paragraph "The antimatter engine begins to charge" due to a video memory OOM failure, the scheduling module performs the following operations: Health check: The three A100 instances in the distributed data center were scanned. Instance B had 15GB of video memory remaining (meeting the 10GB requirement) and a current load of 0.3 (below the threshold of 0.7). Instance C had 8GB of video memory remaining (not meeting the requirement), and instance D had a load of 1.2 (too high). Instance B was ultimately selected. Hot migration execution: The new_input_tokens (original input_tokens + 200 generated tokens) and the corresponding KV cache state (about 12GB) are transferred to instance B via the RDMA network, which takes about 80ms. Restart: After receiving the migration package, instance B directly generates the subsequent text, "A red alert suddenly illuminates on the bridge screen, and the image of Cygnus X-1 begins to distort...", based on the KV cache. The client authoring tool continues to receive the text stream, making the entire failover process imperceptible to the user. The delay in resuming the push only increases by approximately 80ms, without affecting the authoring experience. S5: The healthy inference engine instance performs prefilling on new_input_tokens, utilizing KV caching to avoid repeated computation. It then proceeds to the decoding phase based on the prefill results to generate the remaining tokens.

[0067] In this embodiment, step S5 is specifically as follows: When the new inference engine instance performs the Prefilling process, it builds an incremental KV cache based on new_input_tokens and only calculates the attention state and hidden state of the newly added tokens; During the decoding phase, when the remaining tokens are pushed through the autoregressive method, the combined data of the historical KV cache and the newly added KV cache are reused to achieve seamless continuation of the inference process. The continued push process does not repeatedly calculate the hidden states of the generated tokens.

[0068] (1) Incremental Prefilling Processing and KV Cache Reuse Principle The core of the S5 step is that the healthy inference instance performs efficient pre-population processing on new_input_tokens. Its technical essence is to avoid repeated computing power consumption through incremental calculation. When the new instance receives new_input_tokens (the concatenation of the original input_tokens and the generated tokens), it performs the following operations based on KV cache reuse technology: Incremental KV cache construction: Identify the already generated tokens in new_input_tokens (their KV cache has already been calculated in the original instance) and perform parallel calculations on only the newly added tokens (i.e., the ungenerated portion), generating the corresponding attention state (Key) and hidden state (Value). For example, if new_input_tokens contains 200 already generated tokens and 100 newly added tokens, the new instance only calculates the KV cache for the 100 newly added tokens, reusing the historical KV cache for the first 200 tokens.

[0069] Computational optimization mechanism: By comparing new_input_tokens with the index of generated tokens in the session state, repeated processing of the already calculated portion is skipped. This process reduces the computational complexity of the Prefilling phase from O(N) to O(M) (N is the total number of tokens, M is the number of newly added tokens), significantly improving efficiency.

[0070] (2) Seamless push in the decoding phase In the decoding phase, the new instance combines the historical KV cache with the newly added KV cache to achieve continuous generation of the reasoning process: Autoregressive generation continuation: Based on the KV cache of newly added tokens, the remaining tokens are generated one by one in an autoregressive manner. Each newly generated token depends on the KV cache of all previously generated tokens (including the previously generated ones and the newly added ones) to ensure semantic consistency.

[0071] Cache combination strategy: The historical KV cache (the state of generated tokens) and the newly added KV cache (the state of newly added tokens) are merged in sequential order to form a complete context dependency chain. For example, when generating "displayed on the bridge screen," the model will simultaneously reference the historical KV cache of "antimatter engine charging" and the newly added KV cache of "bridge screen" to maintain the attention association between "engine charging" and "screen display."

[0072] Zero-duplicate calculation guarantee: Through the cache index mark in the session state, secondary calculation of the KV cache of generated tokens is strictly avoided, ensuring the optimization of computing power during the subsequent push process.

[0073] (3) Practical application of science fiction scene generation Taking the science fiction novel generation in the S1 stage as an example, when the new instance receives new_input_tokens (including the original prompt code and the 200 generated tokens): Incremental Prefilling: The new instance detects that the KV cache for the first 200 tokens already exists and only performs parallel computation on the subsequent 100 newly added tokens (e.g., "A red alarm suddenly lights up on the bridge screen") to generate the corresponding KV cache, which takes about 0.5 seconds (the traditional solution requires recalculating 300 tokens, which takes 1.5 seconds).

[0074] Decoding and push-forwarding: When generating a "red alert," the model reuses the historical KV cache for "Antimatter Engine Begins Charging" (storing the semantic state of the engine charging) and the newly added KV cache for the "bridge screen" to generate the logically coherent text, "A red alert suddenly illuminates on the bridge screen. The image of Cygnus X-1 begins to distort, indicating that the wormhole structure is about to become unstable..." Throughout the push-forwarding process, the KV cache for the first 200 tokens is not recalculated, improving push-forwarding efficiency by 67%.

[0075] Context consistency assurance: Due to the combined use of KV cache, the "wormhole structure instability" of new instance generation and the "faster-than-light travel technology" mentioned above form a semantic closed loop, avoiding context gaps caused by recalculation in traditional retry solutions.

[0076] S6: The newly generated tokens are decoded into text chunks by the tokenizer and returned to the client until all tokens are generated and the inference session is completed.

[0077] In this example, step S6 is specifically as follows: The newly generated tokens are decoded into text chunks by the tokenizer through streaming, and returned to the client segment by segment in the order of generation; the text sequence is consistent with the timing of the normal reasoning process, and the new reasoning engine instance maintains the KV cache state based on the spliced ​​new_input_tokens to ensure that the semantic coherence of the decoded text chunk is consistent with the normal reasoning process, completing the reasoning session closed loop.

[0078] (1) Streaming decoding transmission mechanism and real-time interaction design The core of step S6 is to convert the newly generated tokens of the inference engine instance into human-readable text in real time and return it to the client in a streaming manner, thus building a complete closed loop of the inference session. In specific implementation, the system ensures the continuity of interaction through the following mechanisms: Incremental decoding processing: Newly generated tokens (such as ["ship","bridge","screen","curtain","on"]) are decoded into text chunks (such as "on the bridge screen") in sequence by the tokenizer. After each decoding is completed, it is immediately transmitted to the client through the network channel, rather than waiting for all tokens to be generated and returned in batches.

[0079] Streaming protocols: Use HTTP / 2 server push or WebSocket protocols to maintain a persistent connection, ensuring that text chunks arrive at the client in segments with millisecond delays. For example, in a science fiction writing tool, users can watch the word-for-word generation process of "a red alarm suddenly lights up on the bridge screen" in real time.

[0080] (2) Temporal consistency and semantic coherence guarantee To avoid text interruption caused by failover, S6 uses a dual mechanism to ensure output quality: Timing synchronization control: The order in which tokens are generated by the new instance strictly follows the original reasoning logic. By maintaining the token generation index (for example, tokens 201 to 300), the decoded text chunk is output in the logical order of "antimatter engine starts charging → bridge screen lights up alarm", which is completely consistent with the normal reasoning process.

[0081] KV cache state continuity: The new instance maintains a complete KV cache based on the concatenated new_input_tokens (including the KV state of historically generated tokens and the KV state of newly added tokens). During decoding, the attention mechanism is used to trace back the context. For example, when generating "alarm", the model will reference the KV cache of "antimatter engine charging" to ensure the causal relationship between "alarm" and the previous context, avoiding semantic discontinuities such as "the coffee cup suddenly rang on the bridge screen."

[0082] (3) A Complete Closed Loop for Sci-Fi Novel Scene Generation Taking the science fiction novel generation in the S1 phase as an example, after the new instance completes the generation of the remaining tokens: Streaming return process: The newly generated token sequence is ["suddenly", "suddenly", "lit", "start", "red", "color", "alarm", "report"], which is decoded by the tokenizer into "suddenly lit red alarm" and streamed back to the client authoring tool in sections. The user sees the text expansion in real time. Session state management: The system continuously monitors the progress of token generation. When the cumulative number of generated tokens reaches a preset length (such as 500) or the end symbol (such as <end>), the streaming transmission is automatically terminated and the complete session identifier is returned, indicating that the inference session has ended normally; User experience optimization: Throughout the entire process, the text stream received by the client is no different from normal reasoning, and the user does not need to be aware of the fault transition. For example, the transition from "the antimatter engine begins to charge" to "the bridge screen suddenly lights up a red alarm, indicating that the wormhole stability has decreased" is natural, with semantic coherence and no perceived delay.

[0083] Second embodiment like Figure 7 As shown, this embodiment provides a request failover system in a distributed AI reasoning scenario that adopts the request failover method in the distributed AI reasoning scenario in the first embodiment, including: Request initialization and preprocessing module 1: The API service receives the inference request initiated by the user client, forwards the inference request to the inference engine instance through the gateway, encodes the input original text prompt through the tokenizer to obtain the input_tokens sequence, performs prefilling processing, generates the prefill result and the request unique identifier request_id, and then enters the decoding phase to continuously generate tokens; Running status detection and exception capture module 2, in the decoding stage, detects the running status of the inference engine instance in real time, and obtains the generated tokens when an abnormal situation such as network fluctuation, inference engine instance abnormality, hardware failure or video memory OOM is detected, which leads to inference interruption; Input data reorganization module 3, concatenates the input_tokens sequence with the generated tokens to form new_input_tokens; Instance scheduling and migration module 4, which schedules new_input_tokens to the healthy inference engine instance through the scheduling module in the API service; Inference continuation and cache reuse module 5: The healthy inference engine instance performs prefilling processing on new_input_tokens, uses KV cache technology to avoid repeated calculations, and continues to execute the decoding phase based on the prefill results to generate the remaining tokens; Result decoding and session closure module 6 decodes the newly generated tokens into text chunks through the tokenizer and returns them to the client until all tokens are generated and the inference session is completed.

[0084] A computer-readable storage medium stores computer code. When the computer code is executed, the above-described method is performed. Those skilled in the art will appreciate that all or part of the steps in the various methods of the above-described embodiments can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0085] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

[0086] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.< / end>

Claims

1. A request failover method in a distributed AI reasoning scenario, characterized in that: The following steps are involved: S1: The API service receives an inference request from a user client and forwards it to the inference engine instance through the gateway. The service then encodes the input prompt using the tokenizer to generate a sequence of input_tokens. The service then performs prefilling, generating a prefill result and a unique request identifier (request_id). The service then enters the decoding phase to continuously generate tokens. S2: In the decoding phase, the running status of the inference engine instance is detected in real time. When an abnormality such as network fluctuation, inference engine instance abnormality, hardware failure, or memory OOM is detected, resulting in an interruption of inference, the generated tokens are obtained. S3: concatenate the input_tokens sequence with the generated tokens to form new_input_tokens; S4: dispatching new_input_tokens to the healthy inference engine instance through the dispatch module in the API service; S5: The healthy inference engine instance performs Prefilling on new_input_tokens, uses KV caching technology to avoid repeated calculations, and continues to perform Decoding based on the Prefill results to generate the remaining tokens; S6: The newly generated tokens are decoded into text chunks by the tokenizer and returned to the client until all tokens are generated and the inference session is completed.

2. The request failover method in a distributed AI reasoning scenario according to claim 1, characterized in that: In step S1, the Prefilling process is performed to generate the Prefill result and request_id, specifically: All tokens in the input_tokens sequence are processed in parallel to generate attention states and hidden states, and a KV cache is constructed and stored in the session state. The KV cache is used to reuse historical calculation results in the Decoding stage. This processing process is computationally intensive, and the Prefilling stage takes less time than the Decoding stage.

3. The request failover method in a distributed AI reasoning scenario according to claim 1, characterized in that: In step S2, the operating status of the inference engine instance is detected in real time, specifically: When detecting inference interruptions through exception codes, the detection mechanism is integrated into the API service layer. It can identify network fluctuations, inference engine instance anomalies, hardware failures, or video memory OOM of the inference engine instance, and record the generated token sequence based on the token generation progress at the time of the interruption. The detection and recording process does not rely on additional client logic.

4. The request failover method in a distributed AI reasoning scenario according to claim 1, characterized in that: In step S3, the input_tokens sequence is concatenated with the generated tokens to form new_input_tokens, specifically: When storing generated tokens through session state, the generation order of tokens and the update status of the KV cache are recorded synchronously. When splicing, the input_tokens and the generated tokens are merged in the order of the original input + historical generation sequence to form a new new_input_tokens containing complete context dependencies, avoiding repeated calculation of the attention status of the generated tokens.

5. The request failover method in a distributed AI reasoning scenario according to claim 1, characterized in that: In step S4, the dispatch module in the API service dispatches new_input_tokens to the healthy inference engine instance, specifically: The scheduling module screens similar inference instances through a health check mechanism, gives priority to the inference engine instances with sufficient video memory resources and balanced load, and realizes seamless transfer of request context through hot migration technology. The scheduling process takes time controlled in milliseconds and does not affect the real-time interaction of the client.

6. The request failover method in a distributed AI reasoning scenario according to claim 1, characterized in that: In step S5, the healthy inference engine instance performs Prefilling processing on new_input_tokens, specifically: When the new inference engine instance performs the Prefilling process, it builds an incremental KV cache based on new_input_tokens and only calculates the attention state and hidden state of the newly added tokens; During the decoding phase, when the remaining tokens are pushed through the autoregressive method, the combined data of the historical KV cache and the newly added KV cache are reused to achieve seamless continuation of the inference process. The continued push process does not repeatedly calculate the hidden states of the generated tokens.

7. The request failover method in a distributed AI reasoning scenario according to claim 1, characterized in that: In step S6, the newly generated tokens are decoded into text chunks by the tokenizer and returned to the client until all tokens are generated and the inference session is completed. Specifically: The newly generated tokens are decoded into text chunks by the tokenizer through streaming, and are returned to the client in the order they were generated. The text sequence is consistent with the timing of the normal reasoning process. The new reasoning engine instance maintains the KV cache state based on the spliced ​​new_input_tokens to ensure that the semantic coherence of the decoded text chunk is consistent with the normal reasoning process, completing the reasoning session closed loop.

8. A request failover system in a distributed AI reasoning scenario using the request failover method in a distributed AI reasoning scenario according to any one of claims 1 to 7, characterized in that: include: In the request initialization and preprocessing module, the API service receives inference requests initiated by the user client, forwards the inference requests to the inference engine instance through the gateway, encodes the input original text prompt through the tokenizer to obtain the input_tokens sequence, performs prefilling processing, generates the prefill result and the request unique identifier request_id, and then enters the decoding phase to continuously generate tokens; The running status detection and exception capture module detects the running status of the inference engine instance in real time during the decoding phase. When an abnormality such as network fluctuation, inference engine instance abnormality, hardware failure, or memory OOM is detected, resulting in an interruption of inference, the module obtains the generated tokens. The input data reorganization module concatenates the input_tokens sequence with the generated tokens to form new_input_tokens; An instance scheduling and migration module, which schedules new_input_tokens to healthy inference engine instances through the scheduling module in the API service; Inference continuation and cache reuse module: A healthy inference engine instance performs prefilling on new_input_tokens, uses KV caching technology to avoid repeated calculations, and continues to execute the decoding phase based on the prefill results to generate the remaining tokens; The result decoding and session closure module decodes the newly generated tokens into text chunks through the tokenizer and returns them to the client until all tokens are generated and the inference session is completed.

9. A computer device comprising a memory and one or more processors, wherein the memory stores computer code, and when the computer code is executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing computer code, wherein when the computer code is executed, the method according to claim 1 is performed.

Images