An automatic recovery method and system for large language model API calls

Abstract

The application relates to the technical field of artificial intelligence, and particularly discloses an automatic recovery method and system for large language model API calling, which comprises the following steps: S1, when a large language model API returns a flow limiting state code, exponential backoff retry is performed, and a calling alternative model is switched when a degradation condition is met; S2, when all models in a model degradation chain fail to be called, a preset buffer time is superimposed on a next quota reset window time to generate a task recovery time point; S3, task execution of a current intelligent agent is suspended, and a memory snapshot is generated; S4, the memory snapshot, the task recovery time point and a session identifier are stored as a recovery request record in a storage system, and a state of the recovery request record is marked as to be recovered; S5, the storage system is polled at a preset period; and S6, based on a session identifier in a queried recovery request record, an intelligent agent automatically triggers context history-based continuous execution of an unfinished task.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an automatic recovery method and system for API calls of large language models. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, LLM-based agents have been widely used in areas such as automated office work, intelligent customer service, and complex task planning. However, due to limitations in computing power and server load, mainstream LLM service providers (such as OpenAI, Anthropic, and domestic large model vendors) generally adopt strict API call rate limit mechanisms, typically billing and limiting requests / tokens per minute (TPM) or hour (TPM).

[0003] While existing technologies have alleviated the problem of API call failures to some extent, they still have the following significant drawbacks when dealing with complex long-running tasks in LLM Agents: Lack of a linkage mechanism between model degradation and retry: Existing solutions often only perform repeated retries or simple switching for a single model when handling rate limiting. In scenarios with multiple vendors and multiple models, each model has independent quotas and different consumption rates. Existing technologies have failed to establish an effective multi-level degradation link, resulting in the system being unable to intelligently call upon alternative models even if they still have sufficient quotas once the primary model is rate-limited, leading to low overall resource utilization.

[0004] Automatic recovery across time periods is not possible: LLM API rate limits are typically reset on an hourly basis (e.g., 60 times per hour). Existing solutions mostly perform intensive retries within the same hour, and once the limit is exceeded, the task is declared a failure or the user is required to manually retry. They cannot intelligently identify the "next quota window," resulting in task interruption and a very poor user experience.

[0005] Lack of memory state persistence mechanism: LLM Agents rely on multi-turn dialogue context (Memory / Context) and tool call state when executing tasks. Existing solutions, when a task is paused, only record the task ID without saving a complete memory snapshot. When resuming execution, the context must be reconstructed from scratch, resulting in high token consumption and wasted time. In some cases, the task may even fail to continue due to context length limitations.

[0006] Poor user experience: After a rate limit is implemented, users are often in a "black box" state, unable to know the task progress, the reason for the rate limit, and the estimated recovery time. They can only wait passively or refresh frequently, which can easily cause anxiety and operational errors. Summary of the Invention

[0007] To address the aforementioned issues, the purpose of this invention is to provide an automatic recovery method and system for API calls of large language models, which significantly improves API availability, ensures task continuity, greatly reduces resource consumption, improves recovery accuracy, and optimizes user experience.

[0008] This invention provides an automatic recovery method for API calls of large language models, comprising: Step S1: When the large language model API returns a rate-limiting status code, exponential backoff retries are performed within the preset maximum number of retries, and when the degradation conditions are met, the alternative models are switched and called according to the preset model degradation chain order. Step S2: When all models in the model degradation chain fail to be invoked, calculate the next quota reset window time closest to the current time, and add a preset buffer time to the next quota reset window time to generate the task recovery time point; Step S3: Pause the current agent's task execution, obtain the complete memory state of the current agent and perform serialization processing to generate a memory snapshot; Step S4: Establish a mapping relationship between the memory snapshot and the session identifier of the current session; store the memory snapshot, the task recovery time point, and the session identifier as a recovery request record in the storage system; and mark the status of the recovery request record as pending recovery. Step S5: Poll the storage system at a preset period to query recovery request records whose status is pending recovery and whose task recovery time is less than or equal to the current time; Step S6: Based on the session identifier in the queried recovery request record, locate and read the associated memory snapshot, deserialize the memory snapshot into context history, and automatically trigger the agent to continue executing the unfinished task based on the context history.

[0009] In one possible implementation, the model degradation chain includes: qwen3.7-plus, qwen3.6-plus, qwen3.5-plus, glm-5, and kimi-k2.5.

[0010] In one possible implementation, the step of switching and calling alternative models according to a preset model degradation chain order when the degradation conditions are met includes: When the current model fails to retry a preset number of times, and there are alternative models that have not yet been called in the model degradation chain, the alternative models are switched and called according to the preset model degradation chain order.

[0011] In one possible implementation, calculating the next quota reset window time closest to the current time and overlaying it with a preset buffer time includes: Get the next hour closest to the current time; The task recovery time point is obtained by adding a preset buffer duration to the hourly time.

[0012] In one possible implementation, the storage system includes any of the following: a relational database, a message queue middleware, or an in-memory database; When the storage system is a message queue middleware, the recovery request record is enqueued in the form of a message, and polling the storage system at a preset period is the consumption or subscription of the message queue.

[0013] In one possible implementation, the following is included after step S4: Push a rate limiting notification event to the front-end client; the rate limiting notification event includes a formatted task recovery time point, used to show the user the estimated recovery time.

[0014] In one possible implementation, calculating the next quota reset window time closest to the current time further includes: Collect historical data on the occurrence time of traffic throttling and quota reset patterns, and use machine learning models to predict the next quota reset window time closest to the current time.

[0015] In one possible implementation, when the interval between the next quota reset window time predicted by the machine learning model and the current time is less than the preset buffer duration, the current time plus the preset buffer duration is used as the task recovery time point.

[0016] In one possible implementation, the following is included after step S6: If the task recovery is successful, the status of the recovery request record will be updated to "responded". If a rate-limiting status code is detected again during the resumption of execution, a preset duration is added to the original task resumption time point to update the task resumption time point recorded in the resumption request, and the pending resumption status is maintained to wait for the next round of scanning.

[0017] This invention also provides an automatic recovery system for API calls of large language models, which performs any of the above methods, including: The multi-level retry and model degradation module is used to perform exponential backoff retries within a preset maximum number of retries when a rate-limiting status code is detected from the large language model API, and to switch and call alternative models according to the preset model degradation chain order when the degradation conditions are met. The task recovery time point generation module is used to calculate the next quota reset window time closest to the current time when all models in the model degradation chain fail to be called, and to add a preset buffer time to the next quota reset window time to generate the task recovery time point. The memory snapshot generation module is used to pause the current agent's task execution, obtain the current agent's complete memory state and perform serialization processing to generate a memory snapshot; A recovery request record generation module is used to establish a mapping relationship between the memory snapshot and the session identifier of the current session, store the memory snapshot, the task recovery time point and the session identifier as a recovery request record in the storage system, and mark the status of the recovery request record as pending recovery; The periodic query module is used to poll the storage system at a preset period to query recovery request records whose status is pending recovery and whose task recovery time is less than or equal to the current time. The continuation module is used to locate and read the associated memory snapshot based on the session identifier in the queried recovery request record, and deserialize the memory snapshot into context history, so as to automatically trigger the agent to continue to execute the unfinished task based on the context history.

[0018] The automatic recovery method and system for API calls of large language models provided by this invention can significantly improve API availability, ensure task continuity, greatly reduce resource consumption, improve recovery accuracy, and optimize user experience. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the automatic recovery method and system provided in the embodiments of the present invention. Detailed Implementation

[0020] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0021] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0022] First, the terminology involved in this invention will be explained.

[0023] LLM: Large Language Model, such as GPT-4, Qwen, etc. 429: HTTP status code, indicating that the requests are being made too frequently and the server is throttling the requests. SSE: Server-Sent Events, used for one-way data push from the server to the client. HITL: Human-In-The-Loop, a human-machine collaborative mechanism that refers to automated processes that require human intervention. Exponential backoff is a retry strategy where the waiting time for each retry increases exponentially. Model fallback: Automatically switches to a backup model when the primary model becomes unavailable. Memory Snapshot: A snapshot of the agent's complete dialogue context state at a specific point in time. ReAct: Reasoning + Acting, an agent architecture that allows LLMs to alternate between reasoning and acting. @Scheduled: A Spring framework annotation used to mark scheduled task methods. `fixedDelay`: A Spring scheduled task parameter that indicates a fixed delay after the previous execution before the next execution. Figure 1 A flowchart illustrating the automatic recovery method and system provided for embodiments of the present invention is shown below. Figure 1 As shown, this invention provides an automatic recovery method for API calls of large language models, including: Step S1: When the large language model API returns a rate-limiting status code, exponential backoff retries are performed within the preset maximum number of retries, and when the degradation conditions are met, the alternative models are switched and called according to the preset model degradation chain order. Step S2: When all models in the model degradation chain fail to be invoked, calculate the next quota reset window time closest to the current time, and add a preset buffer time to the next quota reset window time to generate the task recovery time point; Step S3: Pause the current agent's task execution, obtain the complete memory state of the current agent and perform serialization processing to generate a memory snapshot; Step S4: Establish a mapping relationship between the memory snapshot and the session identifier of the current session, store the memory snapshot, task recovery time point and session identifier as a recovery request record in the storage system, and mark the status of the recovery request record as pending recovery; Step S5: Poll the storage system at a preset period to query recovery request records whose status is pending recovery and whose task recovery time is less than or equal to the current time; Step S6: Based on the session identifier in the queried recovery request record, locate and read the associated memory snapshot, deserialize the memory snapshot into context history, and use it to automatically trigger the agent to continue executing the unfinished task based on the context history.

[0024] S1 S4 is executed synchronously by the foreground business thread when rate limiting occurs. S5 S6 is executed asynchronously by an independent background timer service, and the two are decoupled through the recovery request records in the storage system.

[0025] The following details step S1.

[0026] Step S1 aims to maximize the instantaneous throughput success rate when a single rate-limiting event occurs, avoiding blind waiting or considering only a single model.

[0027] In one possible implementation, the model degradation chain includes: qwen3.7-plus, qwen3.6-plus, qwen3.5-plus, glm-5, and kimi-k2.5.

[0028] In one possible implementation, when the degradation conditions are met, switching and calling the alternative model according to the preset model degradation chain order includes: when the current model fails to retry for a certain number of consecutive times (e.g., 2 times), and there is an alternative model in the model degradation chain that has not yet been called, switching and calling the alternative model according to the preset model degradation chain order.

[0029] Multi-level retry and model degradation linkage: 10 exponential backoff retries × 5 model degradation chains (qwen3.7-plus → qwen3.6-plus → qwen3.5-plus → glm-5 → kimi-k2.5), simultaneously evaluating model degradation conditions during the retry process.

[0030] In one example, when the Agent calls the LLM API, it first initiates a request through the RetryableLLMCall component (in a preferred embodiment, a wrapper unit implementing this logic). If a 429 rate-limiting response is received, a retry loop is entered: Maximum number of retries: 10 Withdrawal strategy: Exponential withdrawal, increasing from 1 second to a maximum of 60 seconds. Degradation conditions: The current model fails to retry twice, and there are alternative models in the degradation chain. Downgrade order: qwen3.7-plus → qwen3.6-plus → qwen3.5-plus → glm-5 → kimi-k2.5 The retry counter is reset after each downgrade to ensure that each model has sufficient chance to try, and to avoid wasting the overall quota due to a temporary failure of a model.

[0031] The following details step S2.

[0032] Step S2 aims to accurately predict the timing of quota release, providing a time anchor for cross-period recovery.

[0033] In one possible implementation, calculating the next quota reset window time closest to the current time and overlaying it with a preset buffer time includes: obtaining the next hour closest to the current time; and adding the preset buffer time to the hour to obtain the task recovery time point.

[0034] Since most LLM service providers reset quotas on an hourly basis, the next quota reset window is preferably the next hour. To avoid a large number of concurrent requests at the top of the hour causing secondary rate limiting, buffering needs to be added.

[0035] In one possible implementation, the next quota reset window time closest to the current time is calculated by: collecting historical data on the occurrence time of rate limiting and quota reset patterns, and using a machine learning model to predict the next quota reset window time closest to the current time.

[0036] In one possible implementation, when the interval between the next quota reset window time predicted by the machine learning model and the current time is less than the preset buffer duration, the current time plus the preset buffer duration is used as the task recovery time point.

[0037] In one example, the automatic recovery time calculation across time periods is: calculateNextHourBoundary() + 120_000L (next hour + 2-minute buffer), which accurately predicts the quota reset time.

[0038] Once all models have exhausted their quota (10 retries × 5 models = maximum 50 calls), the rate limiting pause process is triggered. Calculate the next hour's time: calculateNextHourBoundary() returns the next hour (e.g., if the current time is 14:32, it returns 15:00:00). Add buffer time: +120_000 milliseconds (2 minutes) to avoid a surge of requests at the top of the hour causing rate limiting to be applied again. Final recovery time: 15:02:00 The following details step S3.

[0039] Step S3 aims to freeze and preserve the agent.

[0040] In this embodiment, the agent's "complete memory state" refers to the multi-turn dialogue context, tool call records, and intermediate inference chain maintained by the agent at a certain moment (i.e., a memory snapshot). Serialization processing refers to converting it into a persistent text format (such as JSON).

[0041] Before pausing, serialize the Agent’s complete memory state into JSON (e.g., an array of conversation logs containing roles and content).

[0042] In one specific embodiment, the serialized text is saved to the agent state field (i.e., the agentState field) of the existing human-in-the-loop request entity, and the unique identifier (sessionId) of the current session is recorded for accurate association during recovery.

[0043] The following details step S4.

[0044] Step S4 aims to solidify all the context required to pause the task into a queryable record and transparently notify the user.

[0045] In one possible implementation, the storage system includes any of the following: a relational database, a message queue middleware, or an in-memory database; when the storage system is a message queue middleware, recovery request records are enqueued in the form of messages, and the storage system is polled at a preset period to consume or subscribe to the message queue.

[0046] Preferably, to reduce development costs, the data table structure in the existing human-in-the-loop (HITL) interaction infrastructure can be reused.

[0047] In one possible implementation, after step S4, the method further includes: pushing a rate limiting notification event to the front-end client; the rate limiting notification event includes a formatted task recovery time point, used to show the user the estimated recovery time.

[0048] In a preferred embodiment, creating a recovery request is manifested as: creating a HITLRequest record: type = HITLType.RATE_LIMIT status = HITLStatus.SCHEDULED_RESUME expiresAt = Calculated recovery time (15:02:00) agent = associated Agent entity agentState = Serialized memory snapshot JSON In other words, creating a recovery request in a preferred embodiment specifically manifests as follows: the system generates a human-in-the-loop request record, assigns the type attribute of the record to the rate-limited recovery type, the status attribute to the planned recovery status, the expiration time attribute to the recovery time point calculated above (e.g., 15:02:00), the associated proxy attribute points to the agent object of the current session, and stores the serialized memory snapshot in its proxy status field.

[0049] SSE Real-time Notification: Pushes a rate_limited event to the front end, carrying a retryAtDisplay (HH:mm format) to display the estimated recovery time.

[0050] The `rate_limited` event is pushed to the front end via Server-Sent Events: users can see the estimated recovery time on the front end interface without manual operation.

[0051] The following details step S5.

[0052] In a preferred embodiment, scheduled scanning and automatic triggering are achieved by using @Scheduled(fixedDelay = 300_000) (a Spring framework annotation, where fixedDelay indicates a fixed delay before the next execution after the previous execution) to scan for expired recovery requests every 5 minutes, automatically triggering the Agent to continue execution.

[0053] RateLimitResumeService performs a scan every 5 minutes using @Scheduled(fixedDelay = 300_000): Query criteria: type = RATE_LIMIT AND status = SCHEDULED_RESUME AND expiresAt <= now() Call the resumeOne() method for each expired record. The resumeOne() method performs the following operations: 1. Deserialize memory snapshots: Parse the memorySnapshot array from the agentState JSON. 2. Construct the recovery message: "[System Automatic Recovery] The previous task was forced to pause due to LLM API rate limiting (429). The rate limiting window has now been reset and the API quota has been restored. Please continue executing the previously incomplete task directly without reconfirmation." 3. Call agentService.chatWithAgent(), passing the deserialized memory as the history parameter. 4. Since the original SSE connection has been broken, a non-streaming execution mode is used (i.e., the background retrieves the complete result at once, instead of pushing it word by word). In other words, the scheduled scanning and automatic triggering mechanism, in a preferred embodiment, is implemented through a scheduled task scheduling annotation (e.g., Spring's @Scheduled annotation, with a fixed delay parameter set to 300,000 milliseconds, meaning it will execute again after a five-minute interval following the previous execution). A component named "Rate Limiting Recovery Service" scans for expired requests every five minutes. During scanning, this service queries the storage system for all records of type "Rate Limiting Recovery," status "Pending Planned Recovery," and expiration time no greater than the current time. For each matching record, the service performs a recovery operation: First, it parses the memory snapshot from the agent status field to restore the context history; second, it constructs a system recovery prompt stating, "The task was previously paused due to rate limiting, but the quota has now been restored. Please continue execution directly without reconfirmation"; third, it calls the intelligent agent dialogue service and passes in the historical context; fourth, since the original push connection may have been broken, the system adopts a non-streaming execution mode, that is, it calculates and returns the complete result in the background all at once, rather than pushing it word-by-word to the client.

[0054] The following details step S6.

[0055] Step S6 involves state machine transition based on the result of the recovery execution.

[0056] In one possible implementation, the following is included after step S6: If the task recovery is successful, the status of the recovery request record will be updated to "responded" and an automatic recovery flag will be added. If the rate-limiting status code (429) is detected again during the recovery process, a preset duration (e.g., 70 minutes) is added to the original task recovery time point to update the task recovery time point recorded in the recovery request, maintaining the pending recovery status to wait for the next round of scanning. If other errors occur, the status is set to expired and retrying stops.

[0057] In one example, the HITLRequest status is updated based on the result of the recovery execution: Success: status = ANSWERED, userAnswer = "[AUTO-RESUMED] Rate limitrecovered" Rate throttling again (Error 429): expiresAt += 70 minutes, maintaining SCHEDULED_RESUME state, waiting for the next scan. Other errors: status = EXPIRED, stop retrying. Example 1 Assuming the agent starts executing the task at 14:28 and receives a 429 response when calling the LLM API at 14:32: 14:32:06 - First retry (waiting 1 second) 14:32:07 - Downgraded to qwen3.6-plus 14:32:08~14:33:00 - Continue retrying and downgrading, for a total of 10 times × 5 models. 14:33:00 - All model quotas exhausted, triggering a pause. 14:33:01 - Calculate recovery time: 15:02:00 (15:00 + 2min buffer) 14:33:02 - Save memory snapshot, create HITLRequest 14:33:03 - SSE pushes the rate_limited event, and the front end displays "15:02". 15:00:00 - Scheduled task scan, expiresAt>now, do not restore 15:05:00 - Scheduled task scan, expiresAt<= now, trigger recovery. 15:05:01 - Deserialization memory snapshot 15:05:02 - Send recovery message 15:05:30 - Agent continues execution, task completed. In other words, the agent started the task at 14:28, encountered the rate limit for the first time at 14:32; from 14:32:06, it began the first retry and gradually downgraded until all model quotas were exhausted at 14:33:00; the system then calculated the recovery point at 15:02:01 at 14:33:01, saved a snapshot and created a record at 14:33:02, and pushed a front-end notification at 14:33:03; if the point was not reached during polling at 15:00:00, recovery would not proceed; if the polling reached the target at 15:05:00, recovery would be triggered; from 15:05:01, the memory was restored, prompts were sent, and the task was completed at 15:05:30.

[0058] This invention also provides an automatic recovery system for API calls of large language models, which performs any of the above methods, including: The multi-level retry and model degradation module is used to perform exponential backoff retries within a preset maximum number of retries when a rate-limiting status code is detected from the large language model API, and to switch and call alternative models according to the preset model degradation chain order when the degradation conditions are met. The task recovery time point generation module is used to calculate the next quota reset window time closest to the current time when all models in the model degradation chain fail to be called, and to add a preset buffer time to the next quota reset window time to generate the task recovery time point. The memory snapshot generation module is used to pause the current agent's task execution, obtain the current agent's complete memory state and perform serialization processing to generate a memory snapshot; The module for generating recovery request records is used to establish a mapping relationship between memory snapshots and the session identifier of the current session. It stores the memory snapshot, the task recovery time point, and the session identifier as a recovery request record in the storage system and marks the status of the recovery request record as pending recovery. The periodic query module is used to poll the storage system at preset intervals to query recovery request records whose status is pending recovery and whose task recovery time is less than or equal to the current time. The continuation module is used to locate and read the associated memory snapshot based on the session identifier in the queried recovery request record, and deserialize the memory snapshot into context history, so as to automatically trigger the agent to continue to execute the unfinished task based on the context history.

[0059] Compared with the prior art, the present invention has significant beneficial effects: API availability has been significantly improved: Through a 5-level model degradation linkage mechanism, the bottleneck of API availability of a single model has been broken, and the overall API call success rate has been increased from about 60% to more than 95%, greatly enhancing the robustness of the system.

[0060] Task continuity assurance: The cross-time automatic recovery mechanism ensures that long-cycle tasks are no longer interrupted by hourly rate limits, and the automatic task completion rate has increased from about 70% to over 99%, truly achieving unattended automated operation.

[0061] Resource consumption is significantly reduced: By using precise time window calculations and complete memory snapshot recovery, invalid retry requests and context rebuilds are avoided. Real-world testing shows that this solution can save approximately 30% of unnecessary token consumption and reduce context rebuild wait time by approximately 60 seconds.

[0062] High recovery accuracy: The algorithm of "hourly + 2-minute buffer" effectively avoids the instantaneous concurrency limit of the API server, making the success rate of cross-time period recovery reach 98%.

[0063] User experience optimization: The SSE real-time notification mechanism eliminates users' uncertainty and anxiety, improving user satisfaction by about 40%, while also reducing invalid duplicate submissions caused by mistakenly believing that the task is stuck.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An automatic recovery method for API calls of large language models, characterized in that, include: Step S1: When the large language model API returns a rate-limiting status code, exponential backoff retries are performed within the preset maximum number of retries, and when the degradation conditions are met, the alternative models are switched and called according to the preset model degradation chain order. Step S2: When all models in the model degradation chain fail to be invoked, calculate the next quota reset window time closest to the current time, and add a preset buffer time to the next quota reset window time to generate the task recovery time point; Step S3: Pause the current agent's task execution, obtain the complete memory state of the current agent and perform serialization processing to generate a memory snapshot; Step S4: Establish a mapping relationship between the memory snapshot and the session identifier of the current session; store the memory snapshot, the task recovery time point, and the session identifier as a recovery request record in the storage system; and mark the status of the recovery request record as pending recovery. Step S5: Poll the storage system at a preset period to query recovery request records whose status is pending recovery and whose task recovery time is less than or equal to the current time; Step S6: Based on the session identifier in the queried recovery request record, locate and read the associated memory snapshot, deserialize the memory snapshot into context history, and automatically trigger the agent to continue executing the unfinished task based on the context history.

2. The automatic recovery method for API calls of large language models according to claim 1, characterized in that, The model degradation chain includes: qwen3.7-plus, qwen3.6-plus, qwen3.5-plus, glm-5, and kimi-k2.

5.

3. The automatic recovery method for API calls of large language models according to claim 1, characterized in that, When the degradation conditions are met, switching and calling alternative models according to the preset model degradation chain order includes: When the current model fails to retry a preset number of times, and there are alternative models that have not yet been called in the model degradation chain, the alternative models are switched and called according to the preset model degradation chain order.

4. The automatic recovery method for API calls of large language models according to claim 1, characterized in that, The calculation of the next quota reset window time closest to the current time, and the overlay of a preset buffer time on it, includes: Get the next hour closest to the current time; The task recovery time point is obtained by adding a preset buffer duration to the hourly time.

5. The automatic recovery method for API calls of large language models according to claim 1, characterized in that, The storage system includes any one of the following: relational database, message queue middleware, or in-memory database; When the storage system is a message queue middleware, the recovery request record is enqueued in the form of a message, and polling the storage system at a preset period is the consumption or subscription of the message queue.

6. The automatic recovery method for API calls of large language models according to claim 1, characterized in that, The process after step S4 also includes: Push a rate limiting notification event to the front-end client; the rate limiting notification event includes a formatted task recovery time point, used to show the user the estimated recovery time.

7. The automatic recovery method for API calls of large language models according to claim 1, characterized in that, The calculation of the next quota reset window time closest to the current time also includes: Collect historical data on the occurrence time of traffic throttling and quota reset patterns, and use machine learning models to predict the next quota reset window time closest to the current time.

8. The automatic recovery method for API calls of large language models according to claim 7, characterized in that, Also includes: When the interval between the next quota reset window time predicted by the machine learning model and the current time is less than the preset buffer time, the current time plus the preset buffer time is used as the task recovery time point.

9. The automatic recovery method for API calls of large language models as described in claim 8, characterized in that, The process after step S6 also includes: If the task recovery is successful, the status of the recovery request record will be updated to "responded". If a rate-limiting status code is detected again during the resumption of execution, a preset duration is added to the original task resumption time point to update the task resumption time point recorded in the resumption request, and the pending resumption status is maintained to wait for the next round of scanning.

10. An automatic recovery system for API calls of large language models, characterized in that, Performing the method according to any one of claims 1-9, comprising: The multi-level retry and model degradation module is used to perform exponential backoff retries within a preset maximum number of retries when a rate-limiting status code is detected from the large language model API, and to switch and call alternative models according to the preset model degradation chain order when the degradation conditions are met. The task recovery time point generation module is used to calculate the next quota reset window time closest to the current time when all models in the model degradation chain fail to be called, and to add a preset buffer time to the next quota reset window time to generate the task recovery time point. The memory snapshot generation module is used to pause the current agent's task execution, obtain the current agent's complete memory state and perform serialization processing to generate a memory snapshot; A recovery request record generation module is used to establish a mapping relationship between the memory snapshot and the session identifier of the current session, store the memory snapshot, the task recovery time point and the session identifier as a recovery request record in the storage system, and mark the status of the recovery request record as pending recovery; The periodic query module is used to poll the storage system at a preset period to query recovery request records whose status is pending recovery and whose task recovery time is less than or equal to the current time. The continuation module is used to locate and read the associated memory snapshot based on the session identifier in the queried recovery request record, and deserialize the memory snapshot into context history, so as to automatically trigger the agent to continue to execute the unfinished task based on the context history.

Images