A method and system for reasoning stateless multi-request sharing agent architecture

Abstract

The application belongs to the technical field of agent architecture optimization, and discloses a multi-request shared agent architecture method and system for reasoning statelessness, which comprises the following steps: obtaining a request, querying a session mapping storage with a session identifier to obtain a workspace identifier corresponding to the current request; querying a workspace mapping storage with the workspace identifier corresponding to the current request to obtain a sandbox instance identifier, a sandbox endpoint and a sandbox-level access token corresponding to the current request; creating a corresponding agent request state for the current request in a stateless reasoning runtime service process; executing an agent cycle based on the agent request state until a cycle termination condition is met to complete the update of a session state in the agent request state; and writing the session state in the agent request state into a session state storage based on an incremental writing strategy after the agent cycle is terminated. The application can improve resource utilization, decouple the life cycle, prevent credential exposure and enhance portability.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent architecture optimization technology, and in particular to a method and system for a stateless reasoning multi-request shared intelligent agent architecture. Background Technology

[0002] AI agents are a rapidly developing form of artificial intelligence application in recent years. Their typical working method is to repeatedly call the Large Language Model (LLM) for reasoning, and then execute commands, read and write files, call tools, etc. in an isolated execution environment (sandbox) based on the reasoning results, forming a cycle of reasoning-execution-observation-re-reasoning until the user task is completed.

[0003] Mainstream intelligent agent platforms and open-source frameworks typically adopt an agent-in-sandbox architecture, where a separate agent process is launched for each user session. This process handles all responsibilities, including inference loop orchestration, large model invocation, and tool execution, and is bound to a unique sandbox. Examples include OpenDevin / OpenHands, Claude Code local version, Manus, and self-built standalone intelligent agents based on LangChain / LangGraph.

[0004] However, existing technologies suffer from low resource utilization. The traditional agent-in-sandbox model dedicates a complete agent process to each user session. In enterprise deployments, the agent process itself needs to load large language model SDKs, tool definitions, runtime dependencies, etc., with single-process memory usage typically ranging from 500MB to 2GB. When the number of concurrent online users reaches hundreds or even thousands, this "one user, one process" allocation method results in a severe waste of memory resources.

[0005] In existing technologies, the sandbox lifecycle is bound to the inference process and cannot be managed independently. After the inference process and the sandbox are bound one by one, the lifecycle events of the sandbox, such as creation, destruction, and state maintenance, must follow the inference process, resulting in: (1) the sandbox being lost when the inference process crashes unexpectedly; (2) the inability to switch between different inference processes on the same sandbox workspace (e.g., upgrading the inference SDK version); and (3) multiple sessions being unable to share the same sandbox workspace to achieve collaboration.

[0006] In existing technologies, LLM credentials are exposed to the user's code execution environment. Since the inference logic and tool execution reside in the same process or sandbox, large language model API credentials (such as OpenAI API Keys and Anthropic API Keys) must be accessible to this process. This means that the user's code execution environment can read these credentials, posing a risk of credential leakage under malicious user attacks or prompting attacks.

[0007] In existing technologies, SDK upgrades require rebuilding the sandbox image. When the inference SDK, user code, and sandbox environment are packaged in the same image, SDK fixes or upgrades must be rebuilt and the image redistributed, taking several minutes to tens of minutes. Furthermore, existing online user sessions must be completely interrupted and the sandbox rebuilt before they can use the new SDK.

[0008] It should be noted that although some existing solutions propose decoupling of AI inference services, such as separating CPU processing and GPU inference to independent nodes, the object of decoupling is the hardware resources of LLM inference itself (CPU node vs. GPU node), which solves the resource utilization problem of the LLM serving service itself.

[0009] Therefore, how to provide a stateless reasoning multi-request shared intelligent agent architecture method and system is an urgent problem to be solved. Summary of the Invention

[0010] This invention provides a method and system for a stateless, multi-request shared intelligent agent architecture to address the problems in the prior art.

[0011] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0012] According to a first aspect of the present invention, a method for a stateless, multi-request shared agent architecture is provided.

[0013] In one embodiment, a reasoning-stateless multi-request shared agent architecture method includes: Retrieve the request carrying the session identifier, query the session mapping store with the session identifier as the primary key, and obtain the workspace identifier corresponding to the current request; Use the workspace identifier corresponding to the current request as the primary key to query the workspace mapping storage and obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. Within the stateless inference runtime service process, create the corresponding agent request state for the current request; Based on the agent request state, the agent loop is executed. The large language model API credentials held internally by the inference runtime service are used to call the external large language model to obtain the inference result. Based on the inference result, a tool execution request is sent to the sandbox endpoint. The execution result returned by the sandbox is received and provided as an observation value to the external large language model to start the next round of inference. This continues until the loop termination condition is met to complete the update of the session state in the agent request state. After the agent loop terminates, the session state in the agent request state is written to the session state storage based on the incremental write strategy, and the agent request state corresponding to the current request is released. No state of the current request remains in the service process during inference runtime.

[0014] In one embodiment, to obtain a request carrying a session identifier, the session mapping store is queried using the session identifier as the primary key to obtain the workspace identifier corresponding to the current request, including: Retrieve requests carrying session identifiers, query session mapping storage using session identifiers as primary keys, and if a single record is returned, take the value of the workspace identifier field in that single record as the workspace identifier corresponding to the current request; If zero rows are returned, the workspace identifier is calculated according to the preset workspace allocation rules and written to the session mapping storage through insert or update operations to obtain the workspace identifier corresponding to the current request.

[0015] In one embodiment, querying the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key to obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request includes: Query the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key. If a single record is returned and the corresponding sandbox instance is alive, then take the sandbox instance identifier, sandbox endpoint, and sandbox-level access token from that single record as the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. If zero rows are returned or the corresponding sandbox instance has been destroyed, a sandbox allocation request is initiated to the sandbox management service. The sandbox management service allocates a new sandbox instance for the workspace identifier, generates the sandbox endpoint and sandbox-level access token corresponding to the sandbox instance and returns them; and writes the mapping relationship between the workspace identifier and the newly allocated sandbox instance identifier into the workspace mapping store.

[0016] In one embodiment, the inference runtime service includes: The inference runtime service process is expanded, shrunk, or restarted independently of the sandbox. The lifecycle of the sandbox instance is maintained independently by the sandbox management service and is not affected by changes to the inference runtime service.

[0017] In one embodiment, the inference runtime service process performing scaling up, scaling down, or rolling restart operations independently of the sandbox includes: When the inference runtime service process is scaled up, the newly added inference runtime service instance completes startup and health checks and is added to the load balancer pool before receiving and processing new requests. When the inference runtime service process is scaled down or rolled back, the inference runtime service instance to be stopped is removed from the load balancer pool and stops receiving new requests; it continues to execute the currently processed agent loop until the agent loop completes naturally; if there are still unfinished agent loops after the graceful shutdown timeout specified in the deployment configuration, the unfinished agent loops are canceled, the generated partial session state is written to the session state storage and then the process exits.

[0018] In one embodiment, an agent loop is executed based on the agent's request state. The loop uses the large language model API credentials held internally by the inference runtime service to call an external large language model to obtain the inference result. Based on the inference result, a tool execution request is initiated to the sandbox endpoint. The execution result returned by the sandbox is received, and this execution result is provided as an observation to the external large language model to start the next round of inference. This process continues until the loop termination condition is met, thus completing the update of the session state in the agent's request state. Based on the dialogue history and agent configuration in the agent's request state, the request body is assembled according to the large language model application interface protocol. The large language model API credentials held by the inference runtime service are used to call the external large language model to obtain the inference result. If the inference result contains a tool invocation instruction, a tool execution request is sent to the sandbox endpoint based on the sandbox-level access token; Receive the execution results returned by the sandbox after performing tool operations in the isolated execution environment, and add the execution results to the dialogue history of the agent's request state in the tool message format; The updated dialogue history is used as input, and the external large language model is called with the large language model API credentials to start the next round of reasoning. The above process is repeated until the loop termination condition is met, then the agent loop is terminated.

[0019] In one embodiment, based on the dialogue history and agent configuration in the agent's request state, a request body is assembled according to the Large Language Model API protocol. The external Large Language Model is then invoked using the Large Language Model API credentials held internally by the inference runtime service to obtain the inference result, which includes: Generate a message array based on the dialogue history in the agent's request state; Serialize the list of tools registered in the current agent configuration into a tool array; The message array, tool array, and hyperparameters in the agent version configuration are assembled into a request body according to the large language model application programming interface protocol. Based on the large language model API credentials carried in the authorization header of the request, an external large language model is called via HTTPS protocol, and the returned inference results are received. If the returned inference result contains a text message, the text message is extracted as an assistant message and added to the dialogue history; if the returned inference result contains a tool call, the tool call instruction is extracted.

[0020] In one embodiment, if the inference result contains tool invocation instructions, initiating a tool execution request to the sandbox endpoint based on a sandbox-level access token includes: Select the corresponding communication protocol and service interface based on the type of tool invocation command; Once the sandbox endpoint verifies the sandbox-level access token, the corresponding command or file operation is executed.

[0021] In one embodiment, selecting the corresponding communication protocol and service interface based on the type of tool invocation instruction includes: If the tool invocation command is of the type of command execution, then the sandbox endpoint's Process.Start remote procedure call is invoked via the sandbox daemon's gRPC protocol to assemble the tool command, parameters, and environment variables into a process configuration and send it. If the tool call command is a file operation, then the file endpoint of the sandbox endpoint is called through the HTTP REST interface to read the file content with a GET request or write the file content with a POST request.

[0022] In one embodiment, the loop termination condition includes: The inference results returned by the large language model do not contain tool call commands, only text messages; The agent loop has reached the maximum number of iterations preset in the agent configuration; The cumulative processing time for the current request exceeds the preset request timeout threshold in the deployment configuration.

[0023] In one embodiment, after the agent loop terminates, writing the session state from the agent's request state to the session state storage based on an incremental write strategy includes: During the agent request state initialization phase, message fingerprints are calculated for each message in the persistent dialogue history stored in the session state storage to generate a baseline fingerprint array. After the agent loop terminates, the current message list in the agent's request state is compared and verified against the baseline fingerprint array one by one; If the current number of messages is greater than the baseline number of messages, and the message fingerprints of the preceding messages corresponding to the baseline number of messages are completely consistent with the baseline fingerprint array, then it is determined to be an append-type change, and only the messages added in this round of the loop are written to the session state storage through the append write interface; If the current message count is less than or equal to the baseline message count, or if the message fingerprints do not match, it is determined to be a replacement change, and the current message list is completely replaced and written to the session state storage via the full replacement interface.

[0024] According to a second aspect of the present invention, a multi-request shared intelligent agent architecture system with stateless reasoning is provided.

[0025] In one embodiment, a stateless reasoning multi-request shared intelligent agent architecture system includes: The session resolution module is used to obtain requests carrying session identifiers, query the session mapping storage using the session identifier as the primary key, and obtain the workspace identifier corresponding to the current request. The sandbox location module is used to query the workspace mapping storage with the workspace identifier corresponding to the current request as the primary key, and obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. The state creation module is used to create the corresponding agent request state for the current request within the stateless inference runtime service process. The loop execution module is used to execute the agent loop based on the agent's request state. It uses the large language model API credentials held internally by the inference runtime service to call the external large language model to obtain the inference result. Based on the inference result, it sends a tool execution request to the sandbox endpoint, receives the execution result returned by the sandbox, and provides the execution result as an observation to the external large language model to start the next round of inference, until the loop termination condition is met to complete the update of the session state in the agent's request state. The state persistence module is used to write the session state in the agent's request state to the session state storage based on the incremental write strategy after the agent loop terminates, and release the agent request state corresponding to the current request. No state of the current request remains in the service process during inference runtime.

[0026] According to a third aspect of the present invention, a computer device is provided.

[0027] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0028] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0029] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0030] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: 1. This invention solves the resource utilization problem by using request-level agent state objects and a shared Runtime loop. A single Runtime process can simultaneously serve agent loops for dozens or even hundreds of concurrent users. Each request only occupies a small amount of memory (usually tens of MB) required by the request-level agent state object. Compared with the traditional agent-in-sandbox mode, which consumes hundreds of MB to GB of memory per user, the memory resources saved on the inference side can reach an order of magnitude.

[0031] 2. This invention solves the lifecycle binding problem through stateless design and independent scaling capabilities. Runtime process crashes or upgrades do not affect the sandbox, which is maintained as an independent resource by the sandbox management service; the sandbox state is stored in persistent storage outside the Runtime and can be recovered across processes.

[0032] 3. This invention solves the credential exposure problem by exposing only basic primitives in a sandbox and retaining credentials within the runtime process. The sandbox running user code cannot obtain large language model API credentials through any channel, and even if the user code is malicious or contains vulnerabilities, the credentials cannot be leaked.

[0033] 4. This invention solves the problem of needing to rebuild the image for SDK upgrades by leveraging the independent upgrade capability of the Runtime. Upgrading the inference SDK only requires restarting the Runtime process (within seconds), without needing to rebuild any sandbox image; existing online user sessions automatically use the new SDK after the Runtime restart is complete, without the user's awareness.

[0034] 5. The Runtime and Sandbox of this invention communicate through a standardized RPC protocol, allowing the sandbox backend to be replaced with different implementations, such as the Kubernetes sandbox within the platform, user local devices, and third-party cloud sandbox services, thereby improving the portability of the architecture.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] Figure 1 This is a flowchart illustrating a stateless reasoning multi-request shared agent architecture method according to an exemplary embodiment; Figure 2 This is a block diagram illustrating a stateless reasoning multi-request shared intelligent agent architecture system according to an exemplary embodiment; Figure 3 This is a schematic diagram of the system structure in a stateless reasoning multi-request shared intelligent agent architecture method according to an exemplary embodiment; Figure 4 This is a flowchart comparing the methods in a stateless reasoning multi-request shared agent architecture method according to an exemplary embodiment. Figure 5 This is a schematic diagram of a key mechanism in a stateless reasoning multi-request shared agent architecture method according to an exemplary embodiment; Figure 6 This is a flowchart illustrating a stateless reasoning multi-request shared agent architecture method according to an exemplary embodiment. Detailed Implementation

[0038] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0039] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] Figure 1 An embodiment of the inference-stateless multi-request shared agent architecture method of the present invention is shown.

[0042] In this optional embodiment, the inference-stateless multi-request shared agent architecture method includes: S101. Obtain the request carrying the session identifier, query the session mapping storage with the session identifier as the primary key, and obtain the workspace identifier corresponding to the current request. In this optional embodiment, the request carrying the session identifier is obtained by querying the session mapping storage using the session identifier as the primary key to obtain the workspace identifier corresponding to the current request, including: Retrieve requests carrying session identifiers, query session mapping storage using session identifiers as primary keys, and if a single record is returned, take the value of the workspace identifier field in that single record as the workspace identifier corresponding to the current request; If zero rows are returned, the workspace identifier is calculated according to the preset workspace allocation rules and written to the session mapping storage through insert or update operations to obtain the workspace identifier corresponding to the current request.

[0043] S102. Query the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key to obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request (bound to sandbox_id, reused within the sandbox lifecycle). In this optional embodiment, querying the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key to obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request includes: Query the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key. If a single record is returned and the corresponding sandbox instance is alive, then take the sandbox instance identifier, sandbox endpoint, and sandbox-level access token from that single record as the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. If zero rows are returned or the corresponding sandbox instance has been destroyed, a sandbox allocation request is initiated to the sandbox management service. The sandbox management service allocates a new sandbox instance for the workspace identifier, generates the sandbox endpoint and sandbox-level access token corresponding to the sandbox instance and returns them; and writes the mapping relationship between the workspace identifier and the newly allocated sandbox instance identifier into the workspace mapping store.

[0044] S103. In the stateless inference runtime service process, create the corresponding agent request state for the current request; In this optional embodiment, the inference runtime service includes: The inference runtime service process is expanded, shrunk, or restarted independently of the sandbox. The lifecycle of the sandbox instance is maintained independently by the sandbox management service and is not affected by changes to the inference runtime service.

[0045] In this optional embodiment, the inference runtime service process performs scaling up, scaling down, or rolling restart operations independently of the sandbox, including: When the inference runtime service process is scaled up, the newly added inference runtime service instance completes startup and health checks and is added to the load balancer pool before receiving and processing new requests. When the inference runtime service process is scaled down or rolled back, the inference runtime service instance to be stopped is removed from the load balancer pool and stops receiving new requests; it continues to execute the currently processed agent loop until the agent loop completes naturally; if there are still unfinished agent loops after the graceful shutdown timeout specified in the deployment configuration, the unfinished agent loops are canceled, the generated partial session state is written to the session state storage and then the process exits.

[0046] S104. Based on the agent request state, execute the agent loop, use the large language model API credentials held internally by the inference runtime service to call the external large language model to obtain the inference result, initiate a tool execution request to the sandbox endpoint according to the inference result, receive the execution result returned by the sandbox, and provide the execution result as an observation value to the external large language model to start the next round of inference, until the loop termination condition is met to complete the update of the session state in the agent request state. In this optional embodiment, the agent loop is executed based on the agent request state. The large language model API credentials held internally by the inference runtime service are used to call an external large language model to obtain the inference result. Based on the inference result, a tool execution request is initiated to the sandbox endpoint. The execution result returned by the sandbox is received, and this execution result is provided as an observation to the external large language model to start the next round of inference. This process continues until the loop termination condition is met, thus completing the update of the session state in the agent request state. Based on the dialogue history and agent configuration in the agent's request state, the request body is assembled according to the large language model application interface protocol. The large language model API credentials held by the inference runtime service are used to call the external large language model to obtain the inference result. If the inference result contains a tool invocation instruction, a tool execution request is sent to the sandbox endpoint based on the sandbox-level access token; Receive the execution results returned by the sandbox after performing tool operations in the isolated execution environment, and add the execution results to the dialogue history of the agent's request state in the tool message format; The updated dialogue history is used as input, and the external large language model is called with the large language model API credentials to start the next round of reasoning. The above process is repeated until the loop termination condition is met, then the agent loop is terminated.

[0047] In this optional embodiment, based on the dialogue history and agent configuration in the agent's request state, a request body is assembled according to the Large Language Model Application Programming Interface (API) protocol. The external Large Language Model is then invoked using the Large Language Model API credentials held internally by the inference runtime service to obtain the inference result, which includes: Generate a message array based on the dialogue history in the agent's request state; Serialize the list of tools registered in the current agent configuration into a tool array; The message array, tool array, and hyperparameters in the agent version configuration are assembled into a request body according to the large language model application programming interface protocol. Based on the large language model API credentials carried in the authorization header of the request, an external large language model is called via HTTPS protocol, and the returned inference results are received. If the returned inference result contains a text message, the text message is extracted as an assistant message and added to the dialogue history; if the returned inference result contains a tool call, the tool call instruction is extracted.

[0048] In this optional embodiment, if the inference result contains a tool invocation instruction, initiating a tool execution request to the sandbox endpoint based on the sandbox-level access token includes: Select the corresponding communication protocol and service interface based on the type of tool invocation command; Once the sandbox endpoint verifies the sandbox-level access token, the corresponding command or file operation is executed.

[0049] In this optional embodiment, selecting the corresponding communication protocol and service interface based on the type of tool invocation instruction includes: If the tool invocation command is of the type of command execution, then the sandbox endpoint's Process.Start remote procedure call is invoked via the sandbox daemon's gRPC protocol to assemble the tool command, parameters, and environment variables into a process configuration and send it. If the tool call command is a file operation, then the file endpoint of the sandbox endpoint is called through the HTTP REST interface to read the file content with a GET request or write the file content with a POST request.

[0050] In this optional embodiment, the loop termination condition includes: The inference results returned by the large language model do not contain tool call commands, only text messages; The agent loop has reached the maximum number of iterations preset in the agent configuration; The cumulative processing time for the current request exceeds the preset request timeout threshold in the deployment configuration.

[0051] S105. After the agent loop terminates, the session state in the agent request state is written to the session state storage based on the incremental write strategy, and the agent request state corresponding to the current request is released. No state of the current request remains in the service process during inference runtime.

[0052] In this optional embodiment, after the agent loop terminates, writing the session state from the agent's request state to the session state storage based on an incremental write strategy includes: During the agent request state initialization phase, message fingerprints are calculated for each message in the persistent dialogue history stored in the session state storage to generate a baseline fingerprint array. After the agent loop terminates, the current message list in the agent's request state is compared and verified against the baseline fingerprint array one by one; If the current number of messages is greater than the baseline number of messages, and the message fingerprints of the preceding messages corresponding to the baseline number of messages are completely consistent with the baseline fingerprint array, then it is determined to be an append-type change, and only the messages added in this round of the loop are written to the session state storage through the append write interface; If the current message count is less than or equal to the baseline message count, or if the message fingerprints do not match, it is determined to be a replacement change, and the current message list is completely replaced and written to the session state storage via the full replacement interface.

[0053] The following examples illustrate the inference-stateless multi-request shared intelligent agent architecture method of the present invention.

[0054] This invention decouples agent inference orchestration from tool execution into two independent, deployable service layers: the inference layer (Agent Runtime) is stateless and shared by multiple tenants, responsible for large language model invocation and agent loop orchestration; the execution layer (Sandbox) is exclusively used by a single tenant, bound by a workspace identifier (workspace_id) rather than a session identifier (session_id), allowing multiple sessions to share the same execution environment. `session_id` refers to the session identifier, `workspace_id` refers to the workspace identifier, and `sandbox_id` refers to the sandbox instance identifier.

[0055] Specifically, the service layer includes inference runtime service, sandbox management service, session state storage, and workspace mapping storage. The inference runtime service (hereinafter referred to as Runtime) is deployed in a stateless manner, with a single process handling multiple user requests in an agent loop simultaneously. Runtime internally holds the Large Language Model API credentials and is responsible for initiating calls to the external LLM API. Runtime does not persist any session state. The Sandbox Manager service (hereinafter referred to as Sandbox Manager) is responsible for the allocation, destruction, and state management of sandboxes. Sandboxes are managed using workspace_id as the key. Sandboxes do not contain an LLM API. Session state storage is a persistent database that saves the dialogue history and intermediate states of each session. Runtime writes this data at the end of each agent loop. The workspace mapping store stores the mapping relationship from workspace_id to sandbox_id, supporting multiple session_ids bound to the same workspace_id. Runtime and Sandbox communicate through standardized protocols, including the sandbox daemon gRPC interface (such as command execution Exec) and HTTP REST interface (such as file read / write ReadFile / WriteFile, corresponding to GET / POST / files endpoints). The sandbox only exposes basic primitives and does not contain agent-specific logic. Furthermore, the lifecycle of a sandbox is independent of any Runtime instance; restarting a Runtime instance does not affect a running sandbox. `Exec` refers to the basic primitive used to execute commands via the sandbox's gRPC interface. `ReadFile` refers to the basic primitive used to read the contents of a file within the sandbox via the sandbox's HTTP REST interface (GET / files?path=...). `WriteFile` refers to the basic primitive used to write byte content to a file within the sandbox via the sandbox's HTTP REST interface (POST / files?path=..., body is a file byte stream).

[0056] The architecture process specifically includes the following steps: Step 1: Receive the upper-layer request, which carries the session_id and the request-level context variable targets (or a similar field that identifies the sandbox location); if not carried, query the workspace_id bound to the session_id, and then query the corresponding sandbox_id from the workspace mapping storage.

[0057] Specifically, the process of querying the workspace_id bound to a session_id is as follows: (1.a) Query the session mapping storage (workspace_id column in the session table of the relational database) with session_id as the primary key. If a single row is returned, take the value of its workspace_id field. (1.b) If 0 lines are returned (first session), the upper-level caller calculates the workspace_id according to the workspace allocation rules agreed upon by the IM channel side or the API caller and writes it to the session mapping storage via upsert.

[0058] The process of querying the corresponding sandbox_id based on workspace_id is as follows: (1.c) Query the workspace mapping storage (sandbox_id column in the workspace_sandbox table of the relational database) with workspace_id as the primary key. If a single row is returned, take the value of its sandbox_id field. (1.d) If 0 lines are returned or the corresponding sandbox has been destroyed (is_alive=false), proceed to step 2.

[0059] Step 2: If no results are found in the query, a new sandbox is allocated for the workspace_id through the sandbox management service and the mapping is recorded; the allocation action is completed independently by the sandbox management service, and the runtime process does not hold the allocation lock.

[0060] The specific process of sandbox allocation is as follows: After receiving an allocation request, the sandbox management service first atomically selects a sandbox with a ready status from the sandbox preheating pool (the warm_pool_pods table); if the preheating pool is empty, it falls back to the path where the sandbox is created immediately. The entire allocation process is completed within the sandbox management service process. The runtime process only acts as the allocation requester, waiting for the result, and does not participate in the allocation decision or hold an allocation lock.

[0061] Step 3: The Runtime creates an independent Agent instance for the current request within the process. This instance holds a per-request state object (request-level agent state object), isolated from the request-level agent state objects of other concurrent requests. The request-level agent state object is a runtime concept, referring to a request-level state object instantiated within the Runtime process for a single request. It is provided by the Agent SDK and carries runtime variables and dialogue history specific to that request. The Sandbox is an isolated environment concept, referring to a Linux container isolated by gVisor or similar mechanisms, hosting the execution of user tool calls. The workspace is a logical grouping concept, referring to an execution domain shared by a group of session_ids, physically corresponding to a sandbox instance. The relationship among the three is as follows: the request-level agent state object is created within the Runtime process; the Runtime process accesses the Sandbox container via the RPC protocol; and the Sandbox container is bound to the workspace_id.

[0062] The specific process of creating an independent request-level state within the runtime process is as follows: The runtime loads the corresponding agent configuration based on the project_id and version_id in the request. The Agent SDK then instantiates a request-level agent state object specific to the current request. This object carries the runtime variables and dialogue history of this request and does not share memory objects with the request-level agent state objects of other concurrent requests in the same process. Runtime variables are not written to the process-level os.environ, thus avoiding mutable state pollution between requests from an architectural perspective.

[0063] Step 4: The Agent is a logical concept, referring to the execution specification composed of prompts, a tool list, and model configuration, described by a YAML file; Runtime is a deployment unit concept, referring to a stateless inference service process. A single process can handle multiple concurrent requests simultaneously and does not persist any session state; the Runtime executes the Agent loop as follows: (4.1) The Runtime calls the external large language model API and carries the API credentials held by the Runtime process in the call; When the runtime calls the large language model API, it assembles the request body according to the OpenAI Chat Completions protocol (or a similar protocol). The specific process is as follows: 1) The messages field: serializes the dialogue history (system prompts and historical user / assistant / tool ​​messages) in the request-level agent state object into a JSON array in the format {role,content} or {role,tool_call_id,content}; 2) tools field: The list of tools registered in the current agent configuration is serialized into a JSON array in the format {type:"function",function:{name,description,parameters:JSONSchema}}. The parameters describe the tool parameter constraints in the JSON Schema Draft 7 format. 3) Hyperparameters such as model, temperature, and max_tokens are derived from the agent version configuration; 4) The Authorization header carries the large language model API key held within the Runtime process (not from the structured variable model object).

[0064] (4.2) The LLM returns the inference result (usually a shell command, file operation, or tool invocation instruction to be executed); the specific process of the LLM returning the inference result is as follows: The Runtime assembles the request body according to the OpenAI Chat Completions protocol or a similar protocol, based on the current message sequence and the description of the tool list in the configuration, and calls the external large language model API via HTTPSPOST; the response message returned by the LLM may contain tool invocation instructions (including tool name and parameters), or only contain the final natural language response; the Runtime determines the next action (routing to sandbox execution / terminating the loop) based on the existence of the returned fields. The large language model API returns two types of content: 1) Text message (finish_reason="stop"): Extract choices[0].message.content as the assistant message for this round and append it to the conversation history; 2) Tool Calls (finish_reason="tool_calls"): Extract the array choices[0].message.tool_calls, each item contains {id,type,function:{name,arguments}}; Runtime deserializes arguments (JSON string), searches the tool registry by name and executes the corresponding tool, appends the execution result to the dialogue history in the format {role:"tool",tool_call_id:id,content:result}, and enters the next round of reasoning.

[0065] (4.3) The Runtime initiates a call to the sandbox endpoint corresponding to the sandbox_id. The call is divided into two protocol paths according to the operation type: (i) Command execution (Exec) calls Process.StartRPC through the sandbox daemon gRPC protocol (i.e., the general sandbox open specification, encapsulated by HTTPConnect), and the request includes ProcessConfig (cmd, args, envs, cwd); (ii) File reading and writing (ReadFile / WriteFile) accesses the sandbox endpoint / files endpoint (GET for reading, POST for writing) through the HTTP REST interface, instead of gRPC; Both types of calls carry the same sandbox-level access token (bound to sandbox_id and reused within the sandbox lifecycle). The sandbox-level access token is issued by SandboxManager when allocating sandboxes, bound to a specific sandbox_id, and reused within the sandbox lifecycle; after the sandbox endpoint verifies the token, it executes the corresponding primitive and returns the result. The sandbox command execution daemon refers to the standardized RPC protocol followed by the sandbox daemon (the sandbox daemon gRPC protocol is an open specification of the e2b sandbox), and the sandbox-level access token refers to the sandbox-level access token (bound to sandbox_id and reused within the sandbox lifecycle) issued by the sandbox management service for accessing a specific sandbox endpoint. Specifically, the Runtime and the sandbox communicate via the gRPC protocol through the sandbox daemon. The sandbox command execution daemon is a gRPC server (port 49983) running within the sandbox Pod, exposing two types of services: 1) Process Service: Responsible for executing command processes within the sandbox. The core RPC is Start(StartRequest) → streamStartResponse, where StartRequest contains ProcessConfig (cmd string, args string array, envs key-value pair mapping, cwd working directory) and optional PTY terminal configuration. The response is streaming output, with each frame containing fragments of process stdout / stderr content or exit codes.

[0066] 2) Filesystem Service: Responsible for file operations within the sandbox, exposed externally via an HTTP REST interface (not gRPC). The main endpoint is the ` / files` path: `GET / files?path={path}` reads the contents of a file at the specified path, `POST / files?path={path}` adds an `application / octet-stream` request body to write to a file, and `DELETE / files?path={path}` deletes a file. Directory operations (listing directories, creating directories) are implemented through the gRPCFilesystem service: `ListDir(ListDirRequest{path,depth}) → ListDirResponse` lists directory entries; `MakeDir(MakeDirRequest{path}) → MakeDirResponse` creates a directory.

[0067] Before invoking the sandbox daemon gRPC, the runtime appends `Authorization:Bearer{access_token}` to the HTTP request header, carrying the sandbox-level access token returned by the SandboxManager when allocating the sandbox (the sandbox-level access token is bound to the sandbox_id and reused throughout the sandbox's lifecycle, and is also bound to the sandbox instance). The sandbox daemon server verifies this token before allowing execution. The specific call flow is as follows: The runtime receives the tool call request → retrieves the endpoint corresponding to the current sandbox_id (e.g., https: / / {sandbox_id}.sandbox.AgentSDK.cloud:49983) and access_token from the request-level agent state object → establishes a connection and appends `Authorization:Bearer{access_token}` to the request header → calls `Process.Start`, assembling the tool command, parameters, and required environment variables into a `ProcessConfig` and sending it → streams the response frames, aggregates them into the tool execution result, and returns it to the Agent loop.

[0068] When multiple sessions are bound to the same workspace_id (i.e., multiple sessions share the same sandbox), there is a risk of concurrent write conflicts if multiple sessions concurrently send tool execution requests to the sandbox. The platform supports concurrent tool invocation capabilities but does not have a built-in dedicated locking mechanism. Concurrent conflicts for the same resource within the sandbox are managed and handled by the caller at the application layer.

[0069] (4.4) The sandbox executes the operation in an isolated execution environment and returns the result; (4.5) The Runtime returns the results as observations to the LLM to determine the next action, until the loop terminates. The criteria for loop termination include: 1) The response returned by LLM does not contain tool invocation commands, but only natural language replies; 2) The agent loop has reached the maximum number of iterations declared in the agent configuration; 3) The cumulative processing time of the current request exceeds the request timeout specified in the deployment configuration; if any of the above conditions are met, the loop will terminate and the process will proceed from step 5 to the status write-back process.

[0070] Step 5: When the loop terminates, the Runtime writes the current request's session state (dialogue history, intermediate states) to the session state storage, and then releases the request-level agent state object. The current request leaves no residual state within the Runtime process.

[0071] The specific process of writing session state to the session state storage is as follows: Several newly generated messages (including user input, several assistant / tool ​​message pairs, and the final assistant reply) from the current loop are appended to the message table corresponding to the session_id using an incremental append method. This step does not modify or rewrite any existing historical messages for that session_id. Specifically, the runtime writes the session state to the session state storage after each round of agent loop, using an incremental write strategy instead of a full overwrite to improve storage efficiency. 1) Baseline fingerprint mechanism: When the request-level agent state object is initialized, it calculates the fingerprint of each message in the persistent historical message list (message fingerprint value = sha256(key sort compact serialization(message.structure serialization()))) and stores it as an array of baseline message fingerprint lists.

[0072] 2) Incremental Decision: At the end of each loop, the message list in memory is compared with the baseline message fingerprint list one by one. If the current number of messages is greater than or equal to the baseline number, and the fingerprints of the first N messages (N = len(baseline message fingerprint list)) match completely, it is determined to be an append-type change (is_append_only=True), and only the newly added messages are incrementally appended to the interface, which then appends them to the storage; otherwise, it is determined to be a replacement-type change (such as the tool call result being corrected), and a full replacement is performed on the interface.

[0073] 3) Effect: For typical multi-turn dialogue scenarios, only 2-3 new messages (user messages, LLM responses and tool results) are added in each turn, and the storage write volume is reduced from O (total history length) to O (number of new messages in this turn).

[0074] Step 6: The runtime process can be expanded, shrunk, or restarted in a rolling fashion at any time: (6.1) When scaling up, the new Runtime instance is added to the load balancing pool and begins to process new requests; (6.2) When scaling down or restarting: The Runtime instance is first removed from the load balancer pool to stop accepting new requests; the existing executing agent loop continues to execute until it terminates naturally (LLM returns the final response or the loop converges); if the graceful shutdown timeout specified in the deployment configuration is exceeded, the Runtime process actively cancels the unfinished loop, that is, it sends a cancel signal to the LLM call and sandbox RPC, and writes the generated partial session state to the session state storage before exiting; (6.3) Runtime changes do not affect the allocated sandboxes, and the sandbox status is maintained independently by the sandbox management service.

[0075] Request-level state isolation is achieved by instantiating an independent request-level agent state object for each request. Multiple concurrent requests within a single runtime process each hold an independent request-level agent state object and do not share mutable states.

[0076] Large Language Model (LLM) API credentials are isolated by placing the credentials only in the runtime process's environment variables or memory; the runtime does not pass the credentials when communicating with the sandbox. When multiple LLM provider credentials exist in the deployment configuration (such as configuring OpenAI and Anthropic simultaneously), the runtime routes to the corresponding credential based on the model field declared in the agent YAML; each LLM call uses only a single credential, without mixing them.

[0077] like Figure 3The diagram shown is a system architecture diagram of this invention, illustrating the component relationships between Runtime / Sandbox Manager / Sandbox Cluster / Storage, specifically including the following components and their connections: Upper-layer callers include IM channel gateways, Web Playground, API clients, etc.; the inference runtime service (hereinafter referred to as Runtime) is stateless, multi-tenant, and horizontally scalable; the large language model API is an external LLM service, such as OpenAI and Anthropic; the sandbox management service (hereinafter referred to as Sandbox Manager) is responsible for sandbox allocation; the sandbox cluster includes multiple sandbox instances, each sandbox is bound to a workspace_id; the session state storage is a persistent database that stores dialogue history; the workspace mapping storage stores the mapping from workspace_id to sandbox_id; the session mapping storage stores the binding from session_id to workspace_id.

[0078] Data flow relationship: The upper-layer caller sends a request to the Runtime with the session_id; the Runtime retrieves the workspace_id from the session mapping store, and then retrieves the sandbox_id from the workspace mapping store; the Runtime creates a request-level agent state object within the process and initiates an agent loop: it calls the Large Language Model API (carrying the LLM credentials within the Runtime process) to obtain the inference result, sends a tool execution request to the corresponding sandbox via the RPC protocol, receives the execution result, until the loop terminates; when the loop terminates, the Runtime writes the session state back to the session state store.

[0079] The key interfaces are defined as follows: "Runtime for the upper-level chat interface (HTTP):" POST / v1 / chat Request body: {session_id, query, variables?, targets?} Response: Streamed agent loop events Runtime's gRPC interface to Sandbox (via the Sandbox daemon protocol): Exec(command, args, envs, user): ExecResult / / Sandbox daemon gRPCProcess.Start ReadFile(path): bytes / / HTTP GET / files?path= <path> WriteFile(path, bytes): void / / HTTP POST / files?path= <path> Sandbox Manager's interface for allocating Runtime: AllocateSandbox(workspace_id, template_id) → (sandbox_id, endpoint,access_token) ReleaseSandbox(sandbox_id)".

[0080] Specifically, Allocate Sandbox refers to the sandbox allocation interface exposed by the sandbox management service, and ReleaseSandbox refers to the sandbox release interface exposed by the service.

[0081] The pseudo-code for runtime multi-tenant concurrent processing is as follows: "async function handle_chat(request):session_id = request.session_idworkspace_id = session_to_workspace_store.get(session_id)if not workspace_id:workspace_id = create_workspace(session_id)sandbox = workspace_to_sandbox_store.get(workspace_id)if notsandbox or not sandbox.is_alive():sandbox = await sandbox_manager.allocate(workspace_id)workspace_to_sandbox_store.set(workspace_id, sandbox)agent_state = Request-level agent state object (session_id=session_id)try:async for event inrun_agent_loop(request.query, sandbox, agent_state):yield eventfinally:session_store.save(session_id, agent_state.history)del agent_state# Request-level state is released with the end of the requestasync function run_agent_loop(query, sandbox, agent_state):messages = [{role: "user", content: query}]while not agent_state.done:llm_response = await llm_client.call(messages, api_key=RUNTIME_LLM_KEY)if llm_response.is_tool_call:tool_result = await sandbox.exec(llm_response.command,token=get_one_time_token())messages.append({role: "tool", content: tool_result})else:messages.append({role: "assistant", content: llm_response.text})agent_state.done = Trueyield FinalResponseEvent(llm_response.text)"..

[0082] Runtime hot restart has no impact on the sandbox. Specifically, the Runtime service is deployed with three replicas: replica-A, replica-B, and replica-C. At a certain time, the operations and maintenance team upgrades the Runtime image: t=0s: Kubernetes rolling update starts, replica-A enters the terminating state and stops receiving new requests; t=0-30s: replica-A waits for ongoing requests to complete; t=30s: replica-A releases the distributed lock and exits; t=30s: The new version of replica-A-new starts up, and after completing the health check, it is added to the load balancer pool; Throughout the process: the sandbox cluster was unaffected, all sandboxes remained operational, and were rerouted to other Runtime replicas to continue providing services.

[0083] LLM credentials isolate evidence. The runtime process's startup environment variables contain the LLM_API_KEY, which is only used by the LLM client within the runtime process. The gRPC call parameters for communication between the runtime and the sandbox do not include the LLM_API_KEY. The sandbox container's environment variables and startup command do not contain the LLM_API_KEY. User code within the sandbox cannot read the LLM_API_KEY using methods such as printenv and / proc / self / environ. The LLM_API_KEY refers to the large language model API credentials stored in the runtime process's startup environment variables.

[0084] Example 1: Sharing Runtime to support high concurrency.

[0085] A certain service runtime is deployed with 5 replicas, each with 2GB of memory. In the traditional agent insandbox mode, each replica supports approximately 10 concurrent users (50 users in total). After adopting this invention, each replica can simultaneously support approximately 80 concurrent agent loops (400 users in total), improving resource efficiency by 8 times.

[0086] Example 2: Team workspace multi-session sharing.

[0087] A five-person team collaborates in a Lark group. Using this invention's architecture, all members share the same workspace_id "ws_team_devops", corresponding to the same sandbox_team. Member Alice deploys a script to / workspace / deploy.sh within the sandbox using an agent. Member Bob then asks in the same group, "Help me run deploy.sh," and the agent finds and executes the script within the same sandbox, enabling real-time collaboration among team members.

[0088] like Figure 4 The diagram shown is a comparison of the architecture of this invention, namely, comparing the traditional agent in sandbox single-tenant mode with the shared runtime multi-tenant mode of this invention.

[0089] like Figure 5 The diagram shown illustrates the decoupling relationship between workspace_id and session_id, illustrating the mapping model for multiple sessions sharing a workspace.

[0090] like Figure 6 The diagram shown is a loop flowchart of the intelligent agent, illustrating the complete path from Runtime calling LLM, routing to Sandbox, receiving the result, and continuing the loop. Figure 4-6 In this context, `session_alice` represents session A, simulating an independent session request initiated by user A; `session_bob` represents session B, simulating an independent session request initiated by user B; `workspace_id` is the workspace identifier, a logical unit identifier bound to the sandbox lifecycle; `sandbox_id` is the sandbox instance identifier, a unique identifier for a specific sandbox container; `session_id` is the session identifier, a unique identifier for a single dialogue context; `Agent State` is the agent request state, a memory-level temporary state object for a single HTTP request; `loop` is a sequence diagram where the loop interaction box represents the cyclically executed interaction sequence; and `alt` is a sequence diagram where the optional branch box represents the two-branch interaction sequence of condition judgment.

[0091] Figure 2 An embodiment of the inference-stateless multi-request shared agent architecture system of the present invention is shown.

[0092] In this optional embodiment, the inference-stateless multi-request shared intelligent agent architecture system includes: The session parsing module 201 is used to obtain requests carrying session identifiers, query the session mapping storage with the session identifier as the primary key, and obtain the workspace identifier corresponding to the current request. The sandbox location module 202 is used to query the workspace mapping storage with the workspace identifier corresponding to the current request as the primary key, and obtain the sandbox instance identifier, sandbox endpoint and sandbox-level access token (bound to sandbox_id, reused within the sandbox lifecycle) corresponding to the current request. The state creation module 203 is used to create a corresponding agent request state for the current request within the stateless inference runtime service process. The loop execution module 204 is used to execute the agent loop based on the agent request state. It uses the large language model API credentials held internally by the inference runtime service to call the external large language model to obtain the inference result. Based on the inference result, it initiates a tool execution request to the sandbox endpoint, receives the execution result returned by the sandbox, and provides the execution result as an observation value to the external large language model to start the next round of inference, until the loop termination condition is met to complete the update of the session state in the agent request state. The state persistence module 205 is used to write the session state in the agent request state to the session state storage based on the incremental writing strategy after the agent loop terminates, and release the agent request state corresponding to the current request. No state of the current request remains in the service process during inference runtime.

[0093] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0094] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0096] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.< / path> < / path>

Claims

1. A method of reasoning stateless multi-request sharing agent architecture, comprising: The method includes: Retrieve the request carrying the session identifier, query the session mapping store with the session identifier as the primary key, and obtain the workspace identifier corresponding to the current request; Use the workspace identifier corresponding to the current request as the primary key to query the workspace mapping storage and obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. Within the stateless inference runtime service process, create the corresponding agent request state for the current request; Based on the agent request state, the agent loop is executed. The large language model API credentials held internally by the inference runtime service are used to call the external large language model to obtain the inference result. Based on the inference result, a tool execution request is sent to the sandbox endpoint. The execution result returned by the sandbox is received and provided as an observation value to the external large language model to start the next round of inference. This continues until the loop termination condition is met to complete the update of the session state in the agent request state. After the agent loop terminates, the session state in the agent request state is written to the session state storage based on the incremental write strategy, and the agent request state corresponding to the current request is released. No state of the current request remains in the service process during inference runtime.

2. The method of claim 1, wherein, The process of obtaining a request carrying a session identifier involves querying the session mapping storage using the session identifier as the primary key to obtain the workspace identifier corresponding to the current request, including: Retrieve requests carrying session identifiers, query session mapping storage using session identifiers as primary keys, and if a single record is returned, take the value of the workspace identifier field in that single record as the workspace identifier corresponding to the current request; If zero rows are returned, the workspace identifier is calculated according to the preset workspace allocation rules and written to the session mapping storage through insert or update operations to obtain the workspace identifier corresponding to the current request.

3. The method of claim 1, wherein, The step of querying the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key to obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request includes: Query the workspace mapping storage using the workspace identifier corresponding to the current request as the primary key. If a single record is returned and the corresponding sandbox instance is alive, then take the sandbox instance identifier, sandbox endpoint, and sandbox-level access token from that single record as the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. If zero rows are returned or the corresponding sandbox instance has been destroyed, a sandbox allocation request is initiated to the sandbox management service. The sandbox management service allocates a new sandbox instance for the workspace identifier, generates the sandbox endpoint and sandbox-level access token corresponding to the sandbox instance and returns them; and writes the mapping relationship between the workspace identifier and the newly allocated sandbox instance identifier into the workspace mapping store.

4. The method of claim 1, wherein, The inference runtime service includes: The inference runtime service process is expanded, shrunk, or restarted independently of the sandbox. The lifecycle of the sandbox instance is maintained independently by the sandbox management service and is not affected by changes to the inference runtime service.

5. The method of claim 4, wherein, The inference runtime service process can be scaled up, scaled down, or rolled back independently of the sandbox, including: When the inference runtime service process is scaled up, the newly added inference runtime service instance completes startup and health checks and is added to the load balancer pool before receiving and processing new requests. When the inference runtime service process is scaled down or rolled back, the inference runtime service instance to be stopped is removed from the load balancer pool and stops receiving new requests; it continues to execute the currently processed agent loop until the agent loop completes naturally; if there are still unfinished agent loops after the graceful shutdown timeout specified in the deployment configuration, the unfinished agent loops are canceled, the generated partial session state is written to the session state storage and then the process exits.

6. The method of claim 1, wherein, The process of executing an agent loop based on the agent's request state, using the large language model API credentials held internally by the inference runtime service to call an external large language model to obtain inference results, initiating a tool execution request to the sandbox endpoint based on the inference results, receiving the execution results returned by the sandbox, and providing these execution results as observations to the external large language model to start the next round of inference, continues until the loop termination condition is met to complete the update of the session state in the agent's request state, including: Based on the dialogue history and agent configuration in the agent's request state, the request body is assembled according to the large language model application interface protocol. The large language model API credentials held by the inference runtime service are used to call the external large language model to obtain the inference result. If the inference result contains a tool invocation instruction, a tool execution request is sent to the sandbox endpoint based on the sandbox-level access token; Receive the execution results returned by the sandbox after performing tool operations in the isolated execution environment, and add the execution results to the dialogue history of the agent's request state in the tool message format; The updated dialogue history is used as input, and the external large language model is called with the large language model API credentials to start the next round of reasoning. The above process is repeated until the loop termination condition is met, then the agent loop is terminated.

7. The method of claim 6, wherein, The dialogue history and agent configuration based on the agent's request state are assembled into a request body according to the large language model application programming interface protocol. The large language model API credentials held internally by the inference runtime service are used to call an external large language model, resulting in the following inference results: Generate a message array based on the dialogue history in the agent's request state; Serialize the list of tools registered in the current agent configuration into a tool array; The message array, tool array, and hyperparameters in the agent version configuration are assembled into a request body according to the large language model application programming interface protocol. Based on the large language model API credentials carried in the authorization header of the request, an external large language model is called via HTTPS protocol, and the returned inference results are received. If the returned inference result contains a text message, the text message is extracted as an assistant message and added to the dialogue history; if the returned inference result contains a tool call, the tool call instruction is extracted.

8. The method of claim 6, wherein, If the inference result contains a tool invocation instruction, the tool execution request is initiated to the sandbox endpoint based on the sandbox-level access token, including: Select the corresponding communication protocol and service interface based on the type of tool invocation command; Once the sandbox endpoint verifies the sandbox-level access token, the corresponding command or file operation is executed.

9. The method of claim 8, wherein, The step of selecting the corresponding communication protocol and service interface based on the type of tool invocation command includes: If the tool invocation command is of the type of command execution, then the sandbox endpoint's Process.Start remote procedure call is invoked via the sandbox daemon's gRPC protocol to assemble the tool command, parameters, and environment variables into a process configuration and send it. If the tool call command is a file operation, then the file endpoint of the sandbox endpoint is called through the HTTP REST interface to read the file content with a GET request or write the file content with a POST request.

10. The method of claim 6, wherein, The loop termination condition includes: The inference results returned by the large language model do not contain tool call commands, only text messages; The agent loop has reached the maximum number of iterations preset in the agent configuration; The cumulative processing time for the current request exceeds the preset request timeout threshold in the deployment configuration.

11. The method of claim 1, wherein, After the agent loop terminates, writing the session state from the agent's request state to the session state storage based on the incremental write strategy includes: During the agent request state initialization phase, message fingerprints are calculated for each message in the persistent dialogue history stored in the session state storage to generate a baseline fingerprint array. After the agent loop terminates, the current message list in the agent's request state is compared and verified with the baseline fingerprint array one by one; If the current number of messages is greater than the baseline number of messages, and the message fingerprints of the preceding messages corresponding to the baseline number of messages are completely consistent with the baseline fingerprint array, then it is determined to be an append-type change, and only the messages added in this round of the loop are written to the session state storage through the append write interface; If the current message count is less than or equal to the baseline message count, or if the message fingerprints do not match, it is determined to be a replacement change, and the current message list is completely replaced and written to the session state storage via the full replacement interface.

12. A reasoning stateless multi-request sharing agent architecture system, comprising: The system includes: The session resolution module is used to obtain requests carrying session identifiers, query the session mapping storage using the session identifier as the primary key, and obtain the workspace identifier corresponding to the current request. The sandbox location module is used to query the workspace mapping storage with the workspace identifier corresponding to the current request as the primary key, and obtain the sandbox instance identifier, sandbox endpoint, and sandbox-level access token corresponding to the current request. The state creation module is used to create the corresponding agent request state for the current request within the stateless inference runtime service process. The loop execution module is used to execute the agent loop based on the agent's request state. It uses the large language model API credentials held internally by the inference runtime service to call the external large language model to obtain the inference result. Based on the inference result, it sends a tool execution request to the sandbox endpoint, receives the execution result returned by the sandbox, and provides the execution result as an observation to the external large language model to start the next round of inference, until the loop termination condition is met to complete the update of the session state in the agent's request state. The state persistence module is used to write the session state in the agent's request state to the session state storage based on the incremental writing strategy after the agent loop terminates, and release the agent request state corresponding to the current request. No state of the current request remains in the service process during inference runtime.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images