Model invocation control method and apparatus, device, and medium

Abstract

The application relates to a model calling control method and device, equipment and a medium, the method comprising: in response to a streaming inference request sent via a gateway server, creating a calling record for initiating streaming inference calling of a large language model, the calling record comprising an abort flag initialized as not requested to be aborted and an execution flag initialized as not completed execution; starting the streaming inference calling; in response to a control handle reaching event, writing the control handle to the calling record, and detecting whether the abort flag represents having been requested to be aborted, if yes, directly executing the control handle to abort the calling; in response to a user abort instruction relayed via the gateway server, setting the abort flag to represent having been requested to be aborted, and when the execution flag represents not having been completed execution and the control handle has been written, executing the control handle and setting the execution flag to represent having been completed execution. The application realizes low delay, high reliability and precise one-time streaming inference task cancellation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model invocation control method and its apparatus, device, and medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are widely deployed in various applications. To improve user experience, modern applications commonly employ streaming technology to push the content generated by large language models to users in real time, avoiding long waiting times. In this streaming inference scenario, users can typically initiate a stop operation at any time during the inference process to halt the model's continued output. How to efficiently and reliably implement the termination of such user-driven streaming inference tasks has become a key problem that needs to be solved in distributed system design.

[0003] Traditional streaming inference task termination schemes primarily rely on network connection context cancellation mechanisms. Specifically, after a streaming connection is established between the client and server via a remote procedure call, this connection is typically bound to a context object to manage the request's lifecycle. When the user clicks the stop button, the client disconnects the network connection. Upon detecting the connection disconnection event, the server terminates the ongoing inference task through context cancellation. This scheme tightly couples the inference task's lifecycle with the network connection's lifecycle. However, in practical applications, users may experience brief connection drops due to network jitter, browser refreshes, or page switching, but these situations do not necessarily indicate that the user intends to terminate the inference task. Once the connection is restored, the inference task has been mistakenly terminated, and the user cannot see the complete inference results, requiring a re-initiation of the request, resulting in a poor user experience. Furthermore, in distributed deployment scenarios, client requests are load-balanced and distributed to different service instances. The connection disconnection event can only notify the service instance currently holding the connection, failing to reliably transmit the cancellation signal to other service instances actually holding the inference task.

[0004] Another common approach is to implement task cancellation through database polling. When the server starts an inference task, it records the task status in the database and starts a background thread to periodically query the cancellation flag in the database. When a user initiates a cancellation request, the cancellation flag in the database is set to "cancelled." The background thread detects this flag change during the next poll, thus terminating the inference task. While this approach decouples the inference task from the network connection, it suffers from significant latency issues. Setting the polling interval presents a dilemma: too short an interval increases the database query load, potentially leading to a database performance bottleneck when there are many inference tasks; too long an interval prolongs the cancellation response time, requiring users to wait several seconds after clicking stop before seeing the inference actually stop, severely impacting user experience. More importantly, in this approach, during the brief time window when the inference task has just started and the database state initialization is not yet complete, the cancellation signal may arrive before the task status record, resulting in the cancellation signal being lost, and subsequent stop clicks by the user will not take effect.

[0005] Both of the above solutions share a common drawback: they cannot guarantee that the cancellation operation will be executed exactly once. In a distributed environment, cancellation requests may arrive at the same inference task via multiple paths. For example, a user may click the stop button multiple times, network retry mechanisms may cause duplicate requests, or load balancing may distribute cancellation requests to multiple service instances simultaneously. In these cases, the cancellation operation may be executed multiple times, triggering repeated resource release operations, state updates, and billing callbacks, leading to system state chaos and business anomalies.

[0006] Therefore, how to achieve low-latency, highly reliable, and exactly-once inference task cancellation in distributed streaming inference services, while maintaining the decoupling of the inference task lifecycle from the network connection lifecycle, has become an urgent technical problem to be solved. Summary of the Invention

[0007] The purpose of this application is to solve at least one of the above-mentioned problems by providing a model call control method and corresponding apparatus, devices, non-volatile readable storage media, and computer program products.

[0008] According to one aspect of this application, a model invocation control method is provided, comprising: In response to a streaming inference request forwarded via the gateway server, a call record is created to initiate a streaming inference call to the large language model. The call record includes an abort flag initialized as not requested to be aborted and an execution flag initialized as not completed. Initiate the streaming inference call to provide the streaming inference result data of the large language model as feedback to the streaming inference request; In response to the arrival event of the control handle of the streaming inference call, the control handle is written to the call record, and the abort flag is checked to determine whether it indicates that abort has been requested. If so, the control handle is executed directly to abort the current call. In response to a user abort command relayed via the gateway server, the abort flag is set to indicate that abort has been requested, and when the execution flag indicates that execution has not been completed and has been written to the control handle, the control handle is executed and the execution flag is set to indicate that execution has been completed.

[0009] According to another aspect of this application, a model recall control device is provided, comprising: The inference response module is configured to respond to streaming inference requests forwarded by the gateway server and create a call record for initiating streaming inference calls to the large language model. The call record includes an abort flag initialized as not requested to be aborted and an execution flag initialized as not completed. The inference call module is configured to initiate the streaming inference call to provide the inference result data of the streaming output of the large language model in response to the streaming inference request. The arrival response module is configured to respond to the arrival event of the control handle of the streaming inference call, write the control handle to the call record, and detect whether the abort flag indicates that it has been requested to be aborted. If so, the control handle is executed directly to abort the current call. The abort response module is configured to respond to a user abort command relayed via the gateway server by setting the abort flag to indicate that abort has been requested, and when the execution flag indicates that execution has not been completed and has been written to the control handle, execute the control handle and set the execution flag to indicate that execution has been completed.

[0010] According to another aspect of this application, an electronic device is provided, including a central processing unit and a memory, wherein the central processing unit is configured to invoke and run a computer program stored in the memory to perform the steps of the method described in this application.

[0011] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the model invocation control method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0012] According to another aspect of this application, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method.

[0013] Compared to traditional technologies, the technical solution provided in this application achieves several beneficial effects through the coordinated operation of the abort flag and execution flag in the call record, combined with the delayed binding of the control handle and the write-time review mechanism. This solution effectively solves the coupling problem between the inference task lifecycle and the network connection lifecycle. The user's abort intention is persistently recorded in local memory. Even if the network is briefly disconnected due to jitter, the recorded abort flag will not be lost, and the inference task will not be mistakenly aborted. Simultaneously, this solution significantly reduces cancellation response latency. The call record is stored in the local registry, and status reads and writes are completed in memory, eliminating the need for database polling and achieving near real-time cancellation responses. Furthermore, this solution fundamentally eliminates the timing race problem of cancellation signals. By pre-creating call records, a reliable storage medium for cancellation signals is provided, ensuring that the cancellation operation is correctly triggered regardless of the timing of the arrival of the cancellation signal and the control handle. Finally, this solution uses the execution flag as an idempotent lock to ensure that the cancellation operation is executed precisely once, avoiding resource release chaos and business anomalies caused by repeated cancellations. Attached Figure Description

[0014] Figure 1 This is an exemplary network architecture for this application; Figure 2 This is a flowchart illustrating one embodiment of the model invocation control method of this application; Figure 3 This is a schematic diagram of the control device for the model in this application; Figure 4 This is a schematic diagram of the structure of an electronic device used in this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0016] This application can be applied to various application scenarios that require providing streaming inference services from large language models to users. In these scenarios, users submit streaming inference requests through a client, expecting the large language model to return inference results in real time, word-by-word output. During the inference process, users may wish to stop the model's continued output at any time, for example, when the model's generated answer deviates from expectations, or when the user has obtained sufficient information. However, in a distributed deployment environment, streaming inference requests are distributed to different service instances via load balancing, and the user's stop command may also be distributed to another service instance, causing the stop signal to be inaccurately transmitted to the service instance holding the inference task. In addition, the inference task requires establishing a connection with the large language model and obtaining a control handle, which has a certain time window. The user's stop command may arrive within this window, and traditional solutions struggle to handle this timing race condition. This application provides a model call control solution that can reliably, timely, and accurately stop streaming inference calls in such application scenarios.

[0017] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a model invocation control system provided in an embodiment of this application. The system may include a client 80, a gateway server 81, and multiple inference servers 82. The client 80 can be a computer device used by a user to submit streaming inference requests to an artificial intelligence service and receive inference result data from the streaming output of a large language model. It also provides a user interface for users to initiate stop commands. The gateway server 81 acts as a traffic scheduling hub, receiving streaming inference requests and user stop commands from the client 80, and distributing these requests and commands to a service instance in the inference server 82 cluster using a load balancing strategy. The inference server 82, as the execution body of the model invocation control method of this application, responds to streaming inference requests, creates invocation records, initiates streaming inference calls to the large language model, and performs stop operations upon receiving a user stop command. The client 80, gateway server 81, and inference servers 82 communicate via a network, which can be a wired or wireless network, such as a local area network, wide area network, or the Internet. The complete invocation chain of the streaming inference service is implemented through this three-way network communication.

[0018] In this embodiment, a streaming inference request refers to a service request submitted by a user through client 80, requiring the large language model to return results in a streaming manner. This request is forwarded by gateway server 81 and then processed by inference server 82. A call record is a data structure created by inference server 82 in its local registry for each streaming inference call, used to store key information such as the call's identifier, abort flag, execution flag, and control handle. The abort flag is a field in the call record used to indicate whether the user has initiated an abort operation; it is initialized to indicate that no abort request has been made. The execution flag is another field in the call record used to indicate whether the abort operation has been executed; it is initialized to indicate that execution has not been completed. The control handle is a function or interface obtained by inference server 82 after successfully establishing a connection with the large language model, used to abort the call. Executing this control handle triggers the large language model to abort the inference process. The user abort instruction is an instruction sent to inference server 82 via gateway server 81 after the user clicks the stop button through client 80, used to instruct the user to abort the ongoing streaming inference call.

[0019] The model invocation control method of this application can be implemented as a computer program running on the inference server 82. The inference server 82 executes this program independently or collaboratively with the client 80 and the gateway server 81 to implement the complete control flow of streaming inference invocation. Specifically, the front-end program running on the client 80 is responsible for providing the streaming inference request submission interface and the inference result display interface, and responding to user stop operations. The routing service program running on the gateway server 81 is responsible for distributing streaming inference requests and user stop commands to the inference server 82 cluster. The background service program running on the inference server 82 is responsible for executing each step of the method of this application, including creating invocation records, initiating streaming inference invocation, binding control handles, responding to stop commands, and executing stop operations to support reliable control of the streaming inference service. The client 80, gateway server 81, and inference server 82 communicate via network to collaboratively complete each step of the method of this application.

[0020] Taking an online intelligent question-answering platform as an example, when a user enters a question and submits a streaming inference request on client 80, the request is forwarded by gateway server 81 to a service instance in the inference server cluster 82. Inference server 82 first creates a call record for this call, then initiates a streaming inference call to the large language model, and feeds back the inference results data streamed from the model to client 80 in real time via gateway server 81. If the user clicks the stop button during inference, the user's stop instruction may reach another inference server 82 after being relayed by gateway server 81. This inference server 82 then transmits the stop signal to the inference server 82 holding the call record via distributed broadcast. Finally, the inference server 82 holding the call record executes the stop operation, stopping the model's continued output. It should be noted that the above online question-answering scenario is only an example. In actual applications, streaming inference services can be flexibly deployed on different platforms according to business characteristics, and this application does not impose any restrictions on this.

[0021] The technical solution of this application will be further described in detail below with reference to specific embodiments.

[0022] Please see Figure 2 According to the model invocation control method provided in this application, it can be implemented as a computer program product and deployed in electronic devices such as inference servers, including the following steps: Step S5100: In response to the streaming inference request forwarded by the gateway server, create a call record for initiating streaming inference calls to the large language model. The call record includes an abort flag initialized as not requested to be aborted and an execution flag initialized as not completed. In a typical application scenario, taking an online intelligent question-answering platform as an example, a user enters a question through the client and clicks the send button. The client encapsulates the question into a streaming inference request and sends it to the gateway server. After receiving the streaming inference request, the gateway server forwards the request to a service instance in the inference server cluster using a load balancing strategy. Upon receiving the streaming inference request forwarded by the gateway server, the inference server containing that service instance begins executing this step.

[0023] Streaming inference requests refer to service requests submitted by users that require a large language model to return results in a streaming manner. The "streaming" aspect here means that the large language model pushes the generated content to the client word by word and sentence by sentence in real time during the generation of inference results, rather than waiting for all content to be generated before returning it all at once. This streaming output capability is determined by the architectural characteristics of the large language model itself. Large language models are typically implemented based on the Transformer architecture, which uses an autoregressive generation method. During the inference process, it predicts the next token (or word element) one by one, and outputs it immediately after each token is generated, thus naturally supporting streaming output. Users can see part of the results before the large language model has completed all the inference, greatly reducing the perceived waiting time and improving the interactive experience.

[0024] After receiving a streaming inference request, the inference server creates a call record for the current call corresponding to that request. A call record is a data structure created by the inference server in the local registry for each streaming inference call, used to store key information about the call. In this application, the call record includes at least two fields: an abort flag and an execution flag. The abort flag indicates whether the user has initiated an abort operation; it is initialized to indicate that no abort request has been made, meaning that the user has not expressed an abort intention at the time of creation. The execution flag indicates whether the user-initiated abort operation has been completed; it is initialized to indicate that no execution has been completed, meaning that no abort operation has been executed at the time of creation.

[0025] In one embodiment, the call record can be implemented as a hash table, using the unique call identifier of the streaming inference request as the key and a structure containing abort flags, execution flags, and other necessary information as the value, stored in the local memory of the inference server. In another embodiment, the call record can also use other key-value storage structures, such as ordered dictionaries or tree structures, as long as they can support fast lookup and update using the call identifier as an index.

[0026] The call log can also be configured with other fields as needed, including but not limited to: a call identifier field, used to store a globally unique identifier assigned by the gateway server to this streaming inference request, so as to uniquely identify this call in a distributed environment; a handle field, used to store the control handle that arrives later, this field is initialized to a null value and filled by a write operation after the control handle arrives; a timestamp field, used to record the creation time of the call log, so as to facilitate subsequent log auditing and performance monitoring; and a source client identifier field, used to record the identity information of the client that initiated this streaming inference request, so as to accurately locate the target client when an interruption event is pushed. These fields, together with the abort status and the abort execution status, constitute the complete data structure of the call log, providing comprehensive information support for subsequent streaming inference call control.

[0027] The creation of the call record is completed before the streaming inference call officially starts, in order to complete the pre-placement. Through pre-placement, the call record already exists during the inference task initialization phase. Subsequently, whether it is the arrival of the control handle or the arrival of the user's abort command, it can be coordinated and managed through the unified data structure of the call record, so as to resolve timing race problems later.

[0028] Step S5200: Initiate the streaming inference call to provide the streaming inference result data of the large language model to the streaming inference request; After the call record is created, the streaming inference call can be initiated. The inference server sends a streaming inference request to the large language model and establishes a streaming data transmission channel so that the large language model can push the generated content to the inference server in real time during the inference result generation process. After receiving these streaming inference result data, the inference server forwards them to the client through the gateway server. The client then displays the results to the user word by word, allowing the user to see part of the results even before the model has completed all the inference.

[0029] In one embodiment, the inference server initiates a streaming inference call by invoking a software development kit (SDK) provided by the large language model. This SSD encapsulates operations such as establishing a connection with the large language model, sending inference requests, receiving streaming output results, and obtaining a control handle. The inference server calls the SSD's interface, passing in the user input and related parameters included in the streaming inference request. The SSD then establishes a connection with the large language model and initiates the inference process. After a successful connection establishment, the SSD returns a control handle, which is used to subsequently terminate the inference call.

[0030] The initiation process of streaming inference calls typically takes time because operations such as establishing a connection with the large language model and initializing the inference environment consume computational and network resources. During this time window while waiting for the large language model to begin real-time inference and return inference result data, the large language model has not yet started generating inference result data, and the inference server has not yet obtained the control handle. If the user clicks the stop button during this time window, the user's abort command may arrive at the inference server before the control handle. This application addresses this timing race scenario by using an abort flag in the call log; the specific handling method will be explained in detail in subsequent steps.

[0031] After the streaming inference call is initiated, the large language model begins generating inference result data and pushes it to the inference server word by word through the streaming data transmission channel. Upon receiving this data, the inference server forwards it to the client in real time via the gateway server. The client then displays the received content to the user word by word, allowing the user to instantly see the model's output progress. This streaming output method fully leverages the autoregressive generation characteristics of the large language model based on the Transformer architecture, outputting each generated token immediately without waiting for all content to be generated, thus significantly reducing the user's perceived waiting time.

[0032] Step S5300: In response to the control handle arrival event of the streaming inference call, write the control handle to the call record and check whether the abort flag indicates that it has been requested to be aborted. If so, execute the control handle directly to abort the current call. After the streaming inference call starts, the inference server establishes a connection with the large language model by calling the software development kit (SDK) of the large language model. Once the connection is successfully established, the SSD returns a control handle, which serves as the interface for subsequently terminating the inference call. The arrival of the control handle triggers a corresponding control handle arrival event, and the inference server, upon receiving this event, executes the operation in this step.

[0033] The inference server first writes the received control handle into the handle field of a pre-placed call record. The handle field is initialized to null when the call record is created; this write operation fills it with the ability to perform an abort operation. After writing the control handle, the inference server immediately checks the abort flag in the call record to determine if the flag has changed from indicating no abort request to indicating abort request.

[0034] As revealed earlier, the change of the abort flag may occur before the control handle arrives. In this case, after receiving the user's abort instruction, the inference server will set the abort flag in the call record to indicate that abort has been requested. However, since the control handle has not yet been written at this time, the abort operation cannot be executed immediately, and the abort flag is temporarily recorded.

[0035] When the control handle subsequently arrives and is written to the call record, the inference server detects that the abort flag indicates a requested abort, meaning the user had previously initiated an abort operation. In this case, the inference server directly executes the newly written control handle to abort the current streaming inference call without waiting for a subsequent user abort command to arrive. This mechanism ensures that in scenarios where the user initiates an abort operation before the control handle arrives, the abort operation is triggered as soon as the control handle becomes ready, preventing the cancellation signal from being lost due to timing issues.

[0036] In another scenario, if the user clicks the stop button after the control handle has been written to the call record, the abort flag will still retain its initial, unrequested abort state when the control handle arrives. If the inference server detects that the abort flag has not been changed, it will not perform an abort operation and will continue the streaming inference call normally, waiting for subsequent user abort commands to arrive before processing them. This processing method will be described in detail in step S5400.

[0037] The specific execution method of the control handle depends on the implementation of the large language model and its software development kit. In one embodiment, the control handle is a function pointer or callback function. The inference server calls this function to trigger the large language model to suspend the inference process. Upon receiving the suspension signal, the large language model stops generating new tokens and returns a context cancellation error, indicating that the inference was terminated due to cancellation. In another embodiment, the control handle is a context object. The inference server triggers suspension by calling the object's cancellation method. This cancellation method internally sends an interrupt signal to the large language model's inference process. In yet another embodiment, the control handle is a channel or semaphore. The inference server triggers suspension by sending a signal to this channel. Upon detecting this signal, the large language model's inference process actively exits the inference loop.

[0038] As can be seen from the above mechanism, regardless of whether the abort flag is set before or after the control handle arrives, the abort operation can be correctly triggered at the appropriate time, thus solving the timing race problem between the cancellation signal and the control handle in streaming inference calls.

[0039] Step S5400: In response to a user abort instruction relayed via the gateway server, set the abort flag to indicate that abort has been requested, and when the execution flag indicates that execution has not been completed and has been written to the control handle, execute the control handle and set the execution flag to indicate that execution has been completed.

[0040] After the streaming inference call is initiated, the inference server listens for both control handle arrival events and user stop command arrival events. While browsing the streaming inference results data of the large language model using the client, if the user believes they have obtained sufficient information or finds that the model-generated answer deviates from expectations, they can click the stop button at any time to initiate a stop operation. The client responds to the user's click, generates a user stop command, and sends it to the gateway server. Upon receiving the user stop command, the gateway server forwards it to a service instance within the inference server cluster using a load balancing strategy. It is worth noting that because the gateway server's load balancing strategy distributes streaming inference requests and user stop commands independently and without correlation, the user stop command may be forwarded to a different service instance than the inference server holding the call record. This step describes the scenario where the user stop command is forwarded to the inference server holding the call record itself; scenarios where the user stop command is forwarded to other service instances will be explained through subsequent embodiments.

[0041] After receiving a user's abort instruction relayed by the gateway server, the inference server first changes the abort flag in the call record from its initial state indicating no abort request to a state indicating abort request has been made. This setting operation records the user's abort intention, ensuring that the user's abort request is not lost regardless of whether the abort operation can be executed immediately afterward.

[0042] After setting the abort flag, the inference server checks the execution flag in the call record to determine if it still retains its initial state indicating that the operation has not been completed. The execution flag indicates whether the user-initiated abort operation has been completed, and it is initialized to indicate that the operation has not been completed. If the execution flag indicates that the operation has been completed, it means that the abort operation has already been executed once before, so it will not be executed again, and the current processing flow will end directly. If the execution flag still indicates that the operation has not been completed, the server continues to check whether a control handle has been written to the call record.

[0043] The determination of whether a control handle has been written is based on the handle field in the call record. The handle field is initialized to null when the call record is created, and is filled by a write operation when the control handle arrives in step S5300. If the handle field is still null, it means the control handle has not yet arrived. In this case, the abort operation cannot be executed immediately; the inference server only completes the setting of the abort flag, waiting for the control handle to arrive later before the detection mechanism in step S5300 triggers the abort operation. If the handle field has been filled, it means the control handle is ready, and the inference server immediately executes the control handle to abort the current streaming inference call.

[0044] After executing the control handle, the inference server changes the execution flag in the call record from indicating that execution was incomplete to indicating that execution was complete. This setting operation signifies that the abort operation has been fully executed. Even if subsequent user abort commands with the same call flag arrive, or cancellation signals triggered by other paths arrive, the abort operation will not be repeated because the execution flag now indicates that execution was complete, thus ensuring that the cancellation operation is executed precisely on the first attempt.

[0045] Execution of the control handle causes the large language model to abort the inference process. For example, upon receiving an abort signal, the large language model stops generating new tokens and returns a context cancellation error via the software development kit. Upon receiving this error, the inference server confirms that the inference call has been successfully aborted and can then perform cleanup operations, including releasing the computational and network resources associated with the call.

[0046] Thus, this step, together with step S5300, constitutes a complete timing race condition handling solution. Step S5300 checks the abort flag when the control handle arrives, handling the scenario where the user abort command arrives before the control handle. Step S5400 checks the existence of the control handle when the user abort command arrives, handling the scenario where the control handle arrives before the user abort command. The processing logic for the two scenarios complements each other, ensuring that the abort operation is correctly triggered at the appropriate time regardless of the arrival order of the user abort command and the control handle. Simultaneously, by checking the execution flag, precise one-time execution of the abort operation is ensured, avoiding resource release chaos and business anomalies caused by repeated cancellations.

[0047] As can be understood from the above embodiments, the technical solution provided in this application, through the coordinated operation of the abort flag and the execution flag in the call record, combined with the delayed binding of the control handle and the back-checking mechanism at the write time, achieves many beneficial effects in the distributed streaming inference call scenario, including but not limited to: First, this application effectively solves the coupling problem between the inference task lifecycle and the network connection lifecycle. By pre-creating call records and setting independent termination flags, the user's termination intention is persistently recorded in the local memory of the service instance, no longer dependent on the network connection's persistence. Even if the network connection is briefly disconnected due to jitter or page refresh, the recorded termination flag will not be lost, and the inference task will not be mistakenly terminated. When the network recovers or the user reconnects, the server can still accurately determine the user's cancellation intention based on the termination flag in the call record, thus correctly executing the termination operation. This decoupling mechanism fundamentally avoids the problem of mistaken termination of inference tasks due to network fluctuations in traditional solutions.

[0048] Secondly, this application significantly reduces the response latency for inference task cancellation. Call records are stored in the local registry of the service instance, and read / write operations on the abort and execution flags are performed in memory, eliminating the need for database polling or cross-network communication. After a user initiates an abort command, the service instance directly sets the abort flag in the local call record and checks the existence of the execution flag and control handle; the entire process is completed in microseconds. Compared to the database polling intervals of hundreds of milliseconds or even seconds in traditional solutions, this application achieves near real-time cancellation response. Users can immediately feel the abort of the inference task after clicking stop, greatly improving the user experience.

[0049] Furthermore, this application fundamentally eliminates the timing race problem of cancellation signals. In traditional schemes, cancellation signals may arrive before the inference task initialization is complete, at which point there is no effective cancellation mechanism to bind them, leading to signal loss. This application provides a reliable storage medium for cancellation signals by pre-creating call records and initializing abort flags. When the cancellation signal arrives before the control handle, the abort flag is set to indicate that abort has been requested. When the control handle arrives subsequently and checks this state at the write time, the abort operation is executed immediately. When the control handle arrives before the cancellation signal, the existence of the execution flag and control handle is checked after the cancellation signal arrives, and abort is executed when the conditions are met. This two-way guarantee mechanism ensures that the cancellation operation can be correctly triggered regardless of the timing of the arrival of the cancellation signal and the control handle, completely eliminating the risk of cancellation signal loss.

[0050] Furthermore, this application achieves precise one-time execution of the cancellation operation. The execution flag in the call record acts as an idempotent lock, checking the execution flag before the cancellation operation is executed. The cancellation operation is only performed and the execution flag is set if the flag indicates that the cancellation request has not been completed. This mechanism ensures that even in a distributed environment where cancellation requests arrive via multiple paths, or if the user clicks the stop button multiple times, the cancellation operation is executed only once. This avoids business problems such as duplicate resource release, chaotic state updates, and abnormal billing callbacks caused by repeated cancellations, guaranteeing the correctness and consistency of the system state.

[0051] Furthermore, this application achieves decoupling and flexible scheduling in a distributed architecture by leveraging the relay role of the gateway server. The gateway server is only responsible for forwarding streaming inference requests and user abort commands to backend service instances, and does not participate in the lifecycle management of inference tasks or the determination of cancellation logic. This architectural design allows backend service instances to independently manage the lifecycle of inference tasks through local call records, eliminating the need for the gateway server to maintain complex session states or task routing information. In a distributed deployment scenario, inference requests and abort commands are distributed independently and are unrelated to each other, reducing the burden on the gateway server and improving the system's scalability and fault tolerance.

[0052] Based on any embodiment of the method in this application, a call record for initiating streaming inference calls to a large language model is created, including: Step S5110: Set a handle field in the call record to store the control handle, and initialize the handle field to a null value; The handle field stores the control handle subsequently returned from the Large Language Model Software Development Kit. When the call record is created, this field is explicitly initialized to null, indicating that a control handle has not yet been obtained. Subsequent steps, when checking the handle field, can accurately determine whether the control handle is ready by checking if the field is null. In one embodiment, the handle field can be implemented as a pointer type and initialized to a null pointer. In another embodiment, the handle field can be implemented as an optional type, such as Optional or Nullable in programming languages, and initialized to a null state. The transition of the handle field from null to non-null signifies the arrival of the control handle, providing a basis for determining the execution condition for subsequent abort operations.

[0053] Step S5120: Set the abort flag in the call record, wherein the abort flag is initialized to indicate that no abort has been requested; The abort flag is used to record whether a user has initiated an abort operation. Its initial state is set to indicate that abort has not been requested, meaning that at the time the record is created, the user has not yet expressed an intention to abort. The abort flag can be implemented using a Boolean variable, for example, with a value of false indicating that abort has not been requested, and a value of true indicating that abort has been requested. In another embodiment, the abort flag can also be implemented using an enumeration type, for example, defining enumeration values ​​including two states: REQUESTED and NOT_REQUESTED. When the user's abort instruction arrives, the abort flag changes from the initial state of unrequested abort to state of requested abort. This state transition records the user's abort intention and provides a triggering condition for subsequent abort operations.

[0054] Step S5130: Set the execution flag in the call record, wherein the execution flag is initialized to indicate that the execution has not been completed; The execution flag is used to record whether an aborted operation has been completed. Its initial state is set to indicate that execution has not been completed, meaning that at the time the call record is created, no aborted operation has been executed. The execution flag can be implemented using a Boolean variable, for example, with a value of false indicating that execution has not been completed and a value of true indicating that execution has been completed. In another embodiment, the execution flag can also be implemented using an integer counter, for example, with a value of 0 indicating that execution has not been completed and a value of 1 indicating that execution has been completed. After the aborted operation is completed, the execution flag changes from the initial state of not being completed to the state of completed execution. This state transition ensures that the aborted operation is executed exactly once. Even if a subsequent aborted instruction with the same call flag arrives, it will not be executed again because the execution flag indicates that execution has been completed.

[0055] Step S5140: Store the call record in the local registry of the current service instance using the call identifier of the streaming inference request as the key.

[0056] The call identifier is a globally unique identifier assigned by the gateway server for each streaming inference request; for example, it can be in the format of a Universally Unique Identifier (UUID). Using the call identifier as the key, the call record, containing information such as handle fields, abort flags, and execution flags, is stored as the value in the local registry of the current service instance. The local registry can be a key-value storage structure in the service instance's local memory, and can be implemented using a hash table (HashMap) to provide O1-level lookup performance.

[0057] In one embodiment, the local registry can be implemented using a thread-safe concurrent hash table (ConcurrentHashMap) to support secure read and write operations in multi-threaded concurrent access scenarios. In another embodiment, the local registry can also be implemented using a locked hash table, employing fine-grained locking or segmented locking mechanisms to ensure concurrency safety. After storing the call records in the local registry, subsequent steps can quickly locate the corresponding call records using the call identifier to perform operations such as status checks, control handle writing, and abort flag setting.

[0058] The above embodiments reveal the complete creation process of a call record. By explicitly setting the initial states of the handle field, abort flag, and execution flag, and storing the call record in the local registry using the call identifier as the key, a structured data foundation is provided for subsequent streaming inference call control. The null initialization of the handle field reserves storage space for delayed binding of the control handle; the initialization state of the abort flag provides a judgment benchmark for recording subsequent user abort commands; the initialization state of the execution flag provides idempotency guarantees for the precise one-time execution of subsequent abort operations; and the storage method using the call identifier as the key provides index support for fast lookup in a distributed environment. The organic combination of these techniques ensures that the call record has a complete initial state at the beginning of its creation, laying a solid data structure foundation for subsequently solving timing race conditions and precise one-time execution problems.

[0059] Based on any embodiment of the method in this application, responding to a user termination instruction relayed via the gateway server includes: Step S5411: Receive the termination notification message broadcast by the external service instance. After receiving the user termination instruction forwarded by the gateway server, the external service instance generates the termination notification message carrying the call identifier corresponding to the user termination instruction and the instance identifier of the current service instance and broadcasts it through the cluster distributed channel. The preceding embodiments have already described a scenario where a user abort command is forwarded by the gateway server to the inference server itself, which holds the call record. However, because the gateway server's load balancing strategy distributes streaming inference requests and user abort commands independently and without correlation, user abort commands may, in some cases, be forwarded to other service instances in the inference server cluster, rather than to the inference server itself, which holds the call record. For this cross-instance scenario, the inference server can use a distributed broadcast mechanism to learn of the user abort command and execute the abort operation.

[0060] When a user initiates an abort operation by clicking the stop button on the client, the client generates a user abort command and sends it to the gateway server. Upon receiving the user abort command, the gateway server forwards it to an external service instance within the inference server cluster using a load balancing strategy. This external service instance is a different service instance from the inference server that holds the call records. After receiving the user abort command forwarded by the gateway server, the external service instance first parses the command to obtain the call identifier it carries. Based on this call identifier, it then searches its local registry for a corresponding call record. Since call records only exist in the inference server that initially processed the streaming inference request, the external service instance's local registry typically does not contain a call record corresponding to this call identifier.

[0061] After confirming that the corresponding call record does not exist locally, the external service instance generates an abort notification message. This abort notification message carries at least two key pieces of information: first, the call identifier corresponding to the user's abort command, used to identify the streaming inference call that needs to be aborted; and second, the instance identifier of the external service instance itself, used to identify the source of the abort notification message. The instance identifier is a unique identifier generated by each service instance at startup, for example, it can adopt the format of a universally unique identifier (UUID). This identifier remains unchanged throughout the process lifecycle of the service instance, ensuring that the instance identifier of each service instance in the cluster is globally unique.

[0062] After an external service instance generates a stop notification message, it broadcasts it to all service instances in the inference server cluster via a cluster-distributed channel. The cluster-distributed channel can be a message delivery mechanism supporting a publish-subscribe pattern, where all service instances subscribe to the channel, and when any service instance publishes a message to the channel, all other service instances in the cluster can receive the message. In one embodiment, the cluster-distributed channel can be implemented using Redis's publish-subscribe PubSub functionality, where service instances receive stop notification messages by subscribing to a specified channel and broadcast stop notifications by publishing messages to that channel. In another embodiment, the cluster-distributed channel can be implemented using a message queue system, such as Apache Kafka or RabbitMQ, where service instances receive messages by subscribing to specific topics and broadcast messages by publishing messages to those topics. In yet another embodiment, the cluster-distributed channel can also be implemented using a distributed consensus protocol, such as the broadcast mechanism of Raft or ZooKeeper, to distribute stop notification messages.

[0063] Through the aforementioned broadcast mechanism, even if the inference server holding the call record does not directly receive the user's abort instruction forwarded by the gateway server, it can still receive the abort notification message broadcast by the external service instance through the cluster's distributed channel, thereby learning about the user's abort intention.

[0064] Step S5412: Detect whether the abort notification message is sent by the current service instance based on the instance identifier. If the abort notification message is not sent by the current service instance, check whether there is a corresponding call record based on the call identifier. If there is, respond to the user abort instruction in the abort notification message. Upon receiving a stop notification message through the cluster's distributed channel, each service instance in the cluster first extracts the sender's instance identifier from the message and compares it with its own instance identifier to determine if the stop notification message originated from the current service instance itself. As mentioned earlier, when the gateway server directly forwards the user's stop instruction to the inference server holding the call record, the inference server, while executing the stop operation, also generates a stop notification message and broadcasts it through the cluster's distributed channel. In this case, the inference server holding the call record will also receive its own broadcast stop notification message. If this message is processed indiscriminately, the stop operation will be executed repeatedly. Therefore, filtering messages sent by the server itself through instance identifier comparison is a crucial measure to avoid loop-like duplication.

[0065] Instance identifier comparison can be achieved through string comparison or numerical comparison. The current service instance compares its own instance identifier with the sender instance identifier carried in the abort notification message. If they match, it means the abort notification message was sent by the current service instance itself and should be discarded without further processing. If they do not match, it means the abort notification message was sent by another service instance and requires further processing.

[0066] After confirming that the abort notification message was not sent by the current service instance, the current service instance extracts the call identifier from the abort notification message and searches its local registry for a corresponding call record based on this call identifier. The call record is created in step S5100 and stored in the local registry using the call identifier as the key; therefore, the corresponding call record can be quickly located using the call identifier. When the local registry uses a key-value storage structure, the time complexity of searching using the call identifier as the key is constant, enabling efficient search operations.

[0067] If a call record corresponding to the call identifier exists in the local registry, it indicates that the current service instance is the inference server that initially processed the streaming inference request, holding the control handle and related status information for the call. In this case, the current service instance responds to the user's abort instruction in the abort notification message. The specific response method is consistent with the processing logic described in step S5400, including setting the abort flag in the call record to indicate that abort has been requested, checking whether the execution flag indicates that execution has not been completed, checking whether the control handle has been written to the handle field, executing the control handle to abort the call when the conditions are met, and setting the execution flag to indicate that execution has been completed. Since this response process has been described in detail in step S5400, it will not be repeated here.

[0068] If no call record corresponding to the call identifier exists in the local registry, it indicates that the current service instance is not the inference server that initially processed the streaming inference request and does not hold any state information for the call. In this case, the current service instance does not need to perform any processing and can simply ignore the abort notification message.

[0069] Step S5413: When the abort notification message is sent by the current service instance, discard the abort notification message.

[0070] When the instance identifier comparison indicates that the abort notification message originated from the current service instance itself, the current service instance discards the message without any further processing to prevent broadcast loops. As mentioned earlier, when the gateway server forwards the user's abort command directly to the inference server holding the call record, the inference server, after executing the abort operation, will also generate an abort notification message and broadcast it through the cluster's distributed channel. Without a discarding mechanism, the inference server would trigger the abort operation again upon receiving its own broadcast message, causing the execution flag to indicate that the operation has been completed and cannot be executed again. Although this does not cause actual duplication of the abort operation, it still triggers unnecessary state checks and judgment logic, consuming computational resources. By discarding the message after instance identifier comparison, this unnecessary processing overhead can be completely avoided.

[0071] In one embodiment, the discard operation can be implemented by skipping subsequent processing steps after a conditional check. That is, when a matching instance identifier is detected, the current message processing function is directly returned or exited without performing any subsequent lookup and response operations. In another embodiment, the discard operation can also be implemented in the message receiving stage through a message filtering mechanism. That is, before the message enters the processing queue, it is pre-filtered based on the instance identifier, and messages sent by the same entity are directly excluded from the processing queue.

[0072] The above embodiments provide a reliable cross-instance solution to the problem of user abort commands not directly reaching the inference server holding the call record in a distributed scenario. Through the broadcast mechanism of the cluster's distributed channel, the user abort command can be propagated from any external service instance receiving the command to all service instances in the cluster, ensuring that the inference server holding the call record always receives the abort signal, thus solving the technical problem of cancellation signals not being able to be transmitted across instances in traditional solutions. By filtering messages sent by the instance itself through instance identifier comparison, the problem of repeated execution of the abort operation caused by broadcast loops is effectively avoided, forming a double idempotency guarantee together with the execution flag in the call record. By performing a constant-level fast lookup in the local registry using the call identifier, the abort operation is executed only on the service instance holding the corresponding call record, while other service instances are unaffected, achieving precise targeted abort. This technical solution enables the distributed streaming inference service to reliably achieve cross-instance user abort command transmission and execution even when the gateway server independently distributes requests and commands, significantly improving the system's robustness and availability.

[0073] Based on any embodiment of the method in this application, responding to a user termination instruction relayed via the gateway server includes: Step S5421: When the current service instance receives the user termination instruction forwarded by the gateway server, it finds the call record belonging to the current service instance according to the call identifier corresponding to the user termination instruction, and responds to the user termination instruction. When the current service instance receives a user abort command forwarded by the gateway server, it first extracts the call identifier from the command. This call identifier is written to the call record when the streaming inference request is created, and the call record is stored in the current service instance's local registry using this identifier as the key. Therefore, the current service instance can use this call identifier to perform a lookup operation in its local registry to locate the call record belonging to the current service instance.

[0074] After locating the corresponding call record, the current service instance responds to the user's abort instruction. The specific response method is consistent with the processing logic described in step S5400, including changing the abort flag in the call record from an initial representation that the request for abort has not been made to a representation that the request for abort has been made; checking whether the execution flag still retains the initial representation that the execution has not been completed; checking whether a control handle has been written to the handle field; and, if the conditions are met, executing the control handle to abort the current streaming inference call, and changing the execution flag from a representation that the execution has not been completed to a representation that the execution has been completed. This response process ensures that the user's abort instruction can be accurately executed on the service instance holding the call record, regardless of whether the instruction arrives before or after the arrival of the control handle, and that correct timing coordination can be achieved through the status field in the call record.

[0075] Step S5422: Generate an abort notification message carrying the call identifier corresponding to the user's abort instruction and the instance identifier of the current service instance, and broadcast the abort notification message to all service instances of the large language model through the cluster distributed channel.

[0076] After the current service instance responds to the user's abort command, it also needs to generate an abort notification message to inform other service instances in the cluster of the user's abort intention. This abort notification message carries at least two key pieces of information: first, the call identifier corresponding to the user's abort command, used to identify the streaming inference call that needs to be aborted; and second, the instance identifier of the current service instance, used to identify the source of the abort notification message. The instance identifier is a unique identifier generated at startup for each service instance, and can be in the format of a universally unique identifier (UUID). This identifier remains unchanged throughout the service instance's process lifecycle, ensuring that the instance identifier of each service instance in the cluster is globally unique.

[0077] After generating the abort notification message, the current service instance broadcasts it to all service instances of the large language model via the cluster distributed channel. The specific implementation of the cluster distributed channel can be found in the various embodiments described above, and will not be repeated here.

[0078] Through this broadcast mechanism, other service instances in the cluster, upon receiving the abort notification message, will perform instance identifier comparison and call record lookup as described in steps S5412 and S5413. Since the abort notification message carries the instance identifier of the current service instance, other service instances, upon receiving the message, can confirm that the message was not sent by themselves by comparing the instance identifier, and then check if a corresponding call record exists in their local registry based on the call identifier. Because the call record only exists in the current service instance that initially processed the streaming inference request, other service instances typically do not have a call record corresponding to this call identifier in their local registry, and therefore they will not perform any abort operation. The main function of this broadcast mechanism is to ensure that in a distributed environment, even if other service instances also hold the same call record (e.g., in abnormal recovery or state synchronization scenarios), they can still receive the abort notification and perform the corresponding processing.

[0079] The above embodiments constitute the complete processing flow after the current service instance directly receives the user's abort instruction. By generating and broadcasting an abort notification message carrying the instance identifier while executing the abort response locally on the current service instance, it ensures that the user's abort instruction can be executed immediately on the service instance holding the call record, and also spreads the abort intention to other service instances in the cluster through the broadcast mechanism, providing complete sender support for abort signal transmission in cross-instance scenarios. The current service instance carries its own instance identifier in the broadcast message, so that other service instances can filter their own messages by comparing the instance identifier after receiving the message, avoiding loop duplication. When this technical solution works in conjunction with the receiver processing flow described in steps S5411 to S5413, it constitutes a complete distributed abort signal transmission mechanism, ensuring that no matter which service instance in the cluster the user's abort instruction is forwarded to by the gateway server, the inference server holding the call record can ultimately reliably receive the abort signal and execute the abort operation.

[0080] Based on any embodiment of the method of this application, after executing the control handle, the following is included: Step S6100: Persistently save the generated streaming output content of this call to the database; After executing the control handle to abort the current streaming inference call, the inference server first persists the streaming output generated by this call to the database. Streaming output refers to the inference results data generated by the large language model and pushed to the inference server before the abort. This data may have been partially forwarded to the client via the gateway server, or some data may still be cached in the inference server's memory and not yet forwarded. In either case, this generated content needs to be completely saved so that users can see the partial results obtained before the abort when they review it later.

[0081] In one embodiment, the inference server can store the streaming output in text format in a relational database, such as MySQL or PostgreSQL, with each record containing fields such as a call identifier, content fragment number, content text, and timestamp. In another embodiment, the inference server can store the streaming output in document format in a non-relational database, such as MongoDB or Elasticsearch, using the call identifier as the document identifier and storing all generated content as a single document. In yet another embodiment, the inference server can store the streaming output in an object storage service, such as Amazon S3 or Alibaba Cloud OSS, using the call identifier as the filename and saving the content as a text file. This persistent storage ensures that even if the client reconnects after a network interruption, the user can still view the inference results generated before the interruption.

[0082] Step S6200: Update the status record of this call from the generating state to the interrupted state, the interrupted state being different from the completed state and the error state; After persistently saving the streaming output, the inference server updates the status record of this call from the "generating" state to the "interrupted" state. The status record is maintained by the inference server in the database to track the lifecycle of each streaming inference call, and its status field indicates the processing progress of the current call. When a streaming inference call starts, the status record is set to the "generating" state, indicating that inference is in progress. When inference completes normally, the status record is updated to the "completed" state. When inference terminates due to an exception, the status record is updated to the "error" state. This application introduces the "interrupted" state as a separate status value, specifically used to identify inference calls that are terminated due to user-initiated cancellation.

[0083] In one embodiment, the status field can be implemented using an enumeration type, defining enumeration values ​​including PENDING (pending processing), GENERATING (generating), COMPLETED (completed), FAILED (error), and INTERRUPTED (interrupted). In another embodiment, the status field can be implemented using an integer type, for example, 0 for generating, 1 for complete, 2 for error, and 3 for interrupted.

[0084] By clearly distinguishing between interrupted, completed, and error states, subsequent business logic can adopt different processing strategies based on different states. For example, for calls in the interrupted state, billing callbacks or subsequent processing flows will no longer be triggered, while for calls in the completed state, subsequent billing and notification flows can be triggered normally.

[0085] Step S6300: Push an interruption event to the client that originates the streaming inference request through the streaming data transmission channel. The interruption event carries the generated partial content so that the client stops the visual feedback of streaming output and displays the truncated result.

[0086] After updating the state record, the inference server pushes an interruption event to the client that initiated the streaming inference request via a streaming data transmission channel. The streaming data transmission channel refers to the network connection established between the inference server and the client for real-time transmission of inference result data, such as a bidirectional stream based on gRPC or a unidirectional stream based on Server-Sent Events. The inference server sends an interruption event to the client through this channel, carrying a portion of the content already generated in this call. In one embodiment, the interruption event can be encapsulated in JSON format, including an event type field (type) with the value "interrupted", a content field (content) containing the generated text string, and a call identifier field (request_id). Upon receiving the interruption event, the client first stops the visual feedback of the streaming output. Visual feedback of the streaming output refers to the dynamic effects displayed by the client during the reception of streaming data, such as a blinking cursor at the text insertion point, a progress rotating icon, or an animated prompt indicating input. The client stopping these visual feedbacks indicates that the inference process has ended. Simultaneously, the client extracts the generated portion of the content from the interruption event and displays it on the user interface, allowing the user to see the results obtained before the inference was interrupted, rather than displaying blank or error messages. This mechanism allows users to immediately see the inference stop after clicking the stop button and view the generated content, providing complete interactive feedback and improving the user experience.

[0087] The above embodiments, by persistently saving the generated streaming output content to the database, ensure that the partial results obtained after the user suspends inference are not lost, and the user can review this content at any time. By updating the call status record to an independent interrupted state, a clear distinction is made between different termination reasons, providing accurate status basis for subsequent billing, log auditing, and statistical analysis. By pushing an interruption event carrying the generated content to the client, the client can promptly stop the visual feedback of streaming output and display the truncated results, providing the user with immediate interactive feedback and a visualized termination result. The organic combination of these technical means enables this application to not only achieve low-latency, high-reliability, and accurate one-time termination operations, but also provide users with a complete post-termination processing experience, ensuring no data loss, distinguishable states, and interactive feedback, comprehensively improving the system integrity and user experience in user termination scenarios in streaming inference services.

[0088] Based on any embodiment of the method in this application, after initiating the streaming inference call, the following is included: Step S7100: Periodically check whether the abort flag in the call record has changed by polling. When the abort flag changes from indicating that it has not been requested to be aborted to indicating that it has been requested to be aborted, determine whether the control handle has been written into the call record. After the streaming inference call starts, in addition to responding to control handle arrival events and user abort command arrival events through an event-driven mechanism, the inference server can also start a separate polling thread as a supplementary detection method. This polling thread periodically accesses the call records at fixed time intervals to check if the abort flag has changed. The polling time interval can be configured according to the system's real-time requirements, for example, it can be set to 10 milliseconds, 50 milliseconds, or 100 milliseconds. A shorter time interval results in lower latency for canceling the response, but also consumes more CPU resources. In one embodiment, the polling thread is implemented using a timer, executing a detection task every 50 milliseconds. In another embodiment, the polling thread is implemented using a scheduled thread pool, executing the detection task at a fixed period.

[0089] Each time the polling thread executes a detection task, it reads the abort flag from the call record and compares it with the value read in the previous cycle. If the abort flag still retains the initial state indicating that abort has not been requested, it means that the user has not yet initiated an abort operation, and the polling thread continues to wait for the next detection cycle. If it detects that the abort flag has changed from indicating that abort has not been requested to indicating that abort has been requested, it means that the user has initiated an abort operation, and the polling thread proceeds to the next step of processing.

[0090] After confirming that the abort flag has been changed, the polling thread further determines whether a control handle has been written to the call record. This determination is based on checking the handle field in the call record. This field is initialized to a null value when the call record is created, and is filled through a write operation when the control handle arrives in step S5300. The polling thread reads the value of the handle field; if the field is not null, it indicates that the control handle is ready; if the field is still null, it indicates that the control handle has not yet arrived.

[0091] Step S7200: If the control handle has been written, execute the control handle to terminate this call; if it has not been written, continue to wait until the control handle is written before execution.

[0092] If the polling thread detects that a control handle has been written to the call record, it immediately executes the control handle to abort the current streaming inference call. The execution method of the control handle is consistent with that described in steps S5300 and S5400. Calling the control handle triggers the large language model to abort the inference process. After receiving the abort signal, the large language model will stop generating new tokens and return a context cancellation error. After executing the control handle, the polling thread will also change the execution flag in the call record from indicating that execution has not been completed to indicating that execution has been completed, to ensure that the abort operation is executed exactly once.

[0093] If the polling thread detects that the control handle has not yet been written to the call record, it cannot immediately perform the abort operation. In this case, the polling thread enters a waiting state and continuously monitors changes in the handle field. In one embodiment, the polling thread can repeat the detection operation in each subsequent detection cycle, that is, continue to periodically read the handle field until it detects that the control handle has been written. In another embodiment, the polling thread can adopt a conditional waiting mechanism. When it detects that the control handle has not been written, it suspends execution by blocking or using a semaphore mechanism until the control handle arrives and is written to the handle field in step S5300, at which point it is woken up by a notification mechanism to continue execution. Regardless of the implementation method, the polling thread will continue to wait until the control handle is ready before performing the abort operation, ensuring that the abort operation is not prevented from being performed because the control handle has not yet arrived.

[0094] The polling mechanism implemented in the above embodiments provides a supplementary detection method to the event-driven mechanism for the termination operation of streaming inference calls. The event-driven mechanism relies on the triggering of control handle arrival events and user termination command arrival events, and can efficiently handle termination operations under normal circumstances. However, in some extreme cases, such as lost event notifications, blocked event processing threads, or event processing delays due to high system load, the polling mechanism can serve as a fallback solution to ensure that the termination operation can still be reliably executed. The polling mechanism and the event-driven mechanism work together to form a double guarantee, further improving the reliability and robustness of this application in complex production environments.

[0095] Based on any embodiment of the method in this application, after initiating the streaming inference call, the following is included: Step S8100: Periodically check the connection status between the current service instance and the large language model through a heartbeat mechanism. When an abnormal connection status is detected, determine whether the current call has been completed based on the execution flag in the call record. After the streaming inference call is initiated, a streaming connection for transmitting inference result data is established between the inference server and the large language model. To ensure the continuous availability of this connection, the inference server starts a heartbeat detection thread, which periodically checks the connection status between the current service instance and the large language model through a heartbeat mechanism. The basic principle of heartbeat detection is that the inference server sends heartbeat probe signals to the large language model at fixed time intervals, such as once every 5 or 10 seconds, and waits for a response from the large language model. If no response is received within a preset timeout period, the connection status is considered abnormal. In one embodiment, heartbeat detection can be implemented by periodically performing ping operations on the streaming connection provided by the large language model software development kit. The software development kit internally maintains the health status of the connection and reports the connection status to the inference server through callback functions or status codes. In another embodiment, heartbeat detection can be implemented by establishing an independent health check connection between the inference server and the large language model. This connection is dedicated to detecting network connectivity and service availability and does not interfere with normal inference data transmission.

[0096] When the heartbeat detection thread detects an abnormal connection status, such as no response after three consecutive heartbeat probes, or the software development kit reporting a disconnected connection, the inference server needs to determine whether it should actively abort the current call. To do this, the inference server reads the execution flag in the call log to determine if the call has been completed. If the execution flag indicates completion, the call has been normally aborted or completed, requiring no further processing. If the execution flag still indicates incomplete execution, the call has not yet been aborted, and the inference server needs to actively perform a termination operation to prevent the inference task from continuing to run idle after the connection is broken, wasting computational and model service resources.

[0097] Step S8200: If the execution flag indicates that execution has not been completed, then the current call is terminated by executing the control handle and the execution flag is set to indicate that execution has been completed.

[0098] When the execution flag indicates that execution has not been completed, the inference server first checks whether a control handle has been written to the call record. The control handle, returned by the Large Language Model software development kit in step S5300 after successful connection establishment, is the operation interface used to abort the inference call. If the control handle has been written, the inference server directly executes it to abort the current streaming inference call. Execution of the control handle triggers the Large Language Model to abort the inference process. Upon receiving the abort signal, the Large Language Model stops generating new tokens and returns a context cancellation error. If the control handle has not been written, it means the connection was broken at the beginning of the inference call, and the inference server may not have obtained the control handle yet. In this case, the inference server cannot abort the call by executing the control handle, but can clean up the system resources occupied by this call by closing the streaming connection or releasing related resources.

[0099] After executing the control handle or performing resource cleanup operations, the inference server changes the execution flag in the call log from indicating that execution was incomplete to indicating that execution was complete. This setting operation signifies that the abort operation of this call has been fully executed, and even if other abort signals arrive subsequently, the abort operation will not be executed again because the execution flag now indicates that execution was complete.

[0100] Through the heartbeat mechanism and connection status detection implemented in the above embodiments, this application can automatically detect connection anomalies in unexpected situations such as network failures, model service malfunctions, or system crashes, and promptly suspend ongoing streaming inference calls, avoiding resource waste caused by inference tasks continuing to run idle after the connection is broken. Simultaneously, by detecting and setting execution flags, it ensures that the suspension operation in connection anomaly scenarios is consistent with the normal suspension operation initiated by the user in terms of state management. Both are guaranteed to execute precisely on the first attempt through execution flags, preventing state chaos or duplicate execution due to connection anomalies. Therefore, this application further enhances its robustness and self-healing capabilities in complex production environments.

[0101] Please see Figure 3According to one aspect of this application, a model invocation control device includes an inference response module 5100, an inference invocation module 5200, an arrival response module 5300, and a termination response module 5400. The inference response module 5100 is configured to respond to a streaming inference request forwarded via a gateway server, creating a invocation record for initiating a streaming inference invocation of a large language model. The invocation record includes a termination flag initialized as not requested to be terminated and an execution flag initialized as not completed. The inference invocation module 5200 is configured to initiate the streaming inference invocation to provide feedback to the large language model in response to the streaming inference request. The model stream outputs inference result data; the arrival response module 5300 is configured to, in response to the arrival event of the control handle of the streaming inference call, write the control handle to the call record and detect whether the abort flag indicates that abort has been requested. If so, the control handle is executed directly to abort the current call; the abort response module 5400 is configured to, in response to a user abort instruction relayed via the gateway server, set the abort flag to indicate that abort has been requested, and when the execution flag indicates that execution has not been completed and has been written to the control handle, execute the control handle and set the execution flag to indicate that execution has been completed.

[0102] Based on any embodiment of the apparatus in this application, the inference response module 5100 includes: a handle field setting module, configured to set a handle field in the call record for storing the control handle, wherein the handle field is initialized to a null value; an abort field setting module, configured to set an abort flag in the call record, wherein the abort flag is initialized to indicate that no abort has been requested; an execution field setting module, configured to set an execution flag in the call record, wherein the execution flag is initialized to indicate that execution has not been completed; and a call record storage module, configured to store the call record in the local registry of the current service instance using the call identifier of the streaming inference request as the key.

[0103] Based on any embodiment of the apparatus in this application, the abort response module 5400 includes: a notification receiving module, configured to receive an abort notification message broadcast by an external service instance, wherein after receiving a user abort instruction forwarded by the gateway server, the external service instance generates the abort notification message carrying a call identifier corresponding to the user abort instruction and an instance identifier of the current service instance and broadcasts it through a cluster distributed channel; a notification processing module, configured to detect whether the abort notification message was sent by the current service instance based on the instance identifier, and if the abort notification message was not sent by the current service instance, to search for a corresponding call record based on the call identifier, and if so, to respond to the user abort instruction in the abort notification message; and a loopback discarding module, configured to discard the abort notification message when it was sent by the current service instance.

[0104] Based on any embodiment of the apparatus in this application, the abort response module 5400 includes: a direct response module, configured to, when the current service instance receives a user abort instruction forwarded by the gateway server, find the call record belonging to the current service instance according to the call identifier corresponding to the user abort instruction, and respond to the user abort instruction; and an external broadcast module, configured to generate an abort notification message carrying the call identifier corresponding to the user abort instruction and the instance identifier of the current service instance, and broadcast the abort notification message to all service instances of the large language model through a cluster distributed channel.

[0105] Based on any embodiment of the device in this application, the device further includes: a content storage module, configured to persistently save the generated streaming output content of the current call to a database; a status update module, configured to update the status record of the current call from the generating state to the interrupted state, wherein the interrupted state is different from the completed state and the error state; and a terminal control module, configured to push an interruption event to the client that originates the streaming inference request through a streaming data transmission channel, wherein the interruption event carries the generated partial content, so that the client stops the visual feedback of streaming output and displays the truncated result.

[0106] Based on any embodiment of the device in this application, the device further includes: a fallback polling module, configured to periodically detect whether the abort flag in the call record has changed by polling; when the abort flag is detected to change from indicating that abort has not been requested to indicating that abort has been requested, it determines whether the control handle has been written into the call record; and a response processing module, configured to execute the control handle to abort the current call if the control handle has been written, and to wait until the control handle is written before execution if the control handle has not been written.

[0107] Based on any embodiment of the device in this application, the device further includes: a connection detection module, configured to periodically detect the connection status between the current service instance and the large language model through a heartbeat mechanism, and when an abnormal connection status is detected, determine whether the current call has been completed based on the execution flag in the call record; and an unexecuted processing module, configured to terminate the current call by executing the control handle and set the execution flag to indicate that the execution has been completed if the execution flag indicates that the execution has not been completed.

[0108] Another embodiment of this application also provides an electronic device. For example... Figure 4 The diagram shows the internal structure of an electronic device. This electronic device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database can store information sequences, and when executed by the processor, the computer-readable instructions enable the processor to implement a model-invoking control method.

[0109] The processor of this electronic device provides computing and control capabilities to support the operation of the entire device. The memory of this electronic device can store computer-readable instructions, which, when executed by the processor, cause the processor to execute the model call control method of this application. The network interface of this electronic device is used for communication with a terminal.

[0110] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0111] In this embodiment, the processor is used to execute... Figure 3 The specific functions of each module are described, and the memory stores the program code and various data required to execute the aforementioned modules or sub-modules. A network interface is used to enable data transmission between user terminals or servers. In this embodiment, the non-volatile readable storage medium stores the program code and data required to execute all modules in the model invocation control device of this application. The server can invoke the server's program code and data to execute the functions of all modules.

[0112] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the model invocation control method of any embodiment of this application.

[0113] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0114] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

Claims

1. A model invocation control method, characterized in that, include: In response to a streaming inference request forwarded via the gateway server, a call record is created to initiate a streaming inference call to the large language model. The call record includes an abort flag initialized as not requested to be aborted and an execution flag initialized as not completed. Initiate the streaming inference call to provide the streaming inference result data of the large language model as feedback to the streaming inference request; In response to the arrival event of the control handle of the streaming inference call, the control handle is written to the call record, and the abort flag is checked to determine whether it indicates that abort has been requested. If so, the control handle is executed directly to abort the current call. In response to a user abort command relayed via the gateway server, the abort flag is set to indicate that abort has been requested, and when the execution flag indicates that execution has not been completed and has been written to the control handle, the control handle is executed and the execution flag is set to indicate that execution has been completed.

2. The model invocation control method according to claim 1, characterized in that, Create a call log for initiating streaming inference calls to a large language model, including: A handle field is set in the call record to store the control handle, and the handle field is initialized to an empty value; The abort flag is set in the call record, and the abort flag is initialized to indicate that it has not been requested to be aborted; The execution flag is set in the call record, and the execution flag is initialized to indicate that the execution has not been completed. The call record is stored in the local registry of the current service instance using the call identifier of the streaming inference request as the key.

3. The model invocation control method according to claim 1, characterized in that, Responding to a user termination instruction relayed via the gateway server includes: The external service instance receives a termination notification message broadcast by an external service instance. After receiving the user termination instruction forwarded by the gateway server, the external service instance generates the termination notification message carrying the call identifier corresponding to the user termination instruction and the instance identifier of the current service instance, and broadcasts it through the cluster distributed channel. The system detects whether the abort notification message was sent by the current service instance based on the instance identifier. If the abort notification message was not sent by the current service instance, it searches for a corresponding call record based on the call identifier. If a record exists, it responds to the user abort command in the abort notification message. When the abort notification message is sent by the current service instance, the abort notification message is discarded.

4. The model invocation control method according to claim 1, characterized in that, Responding to a user termination instruction relayed via the gateway server includes: When the current service instance receives a user termination instruction forwarded by the gateway server, it finds the call record belonging to the current service instance based on the call identifier corresponding to the user termination instruction, and responds to the user termination instruction. Generate an abort notification message carrying the call identifier corresponding to the user's abort instruction and the instance identifier of the current service instance, and broadcast the abort notification message to all service instances of the large language model through a cluster distributed channel.

5. The model invocation control method according to claim 1, characterized in that, After executing the control handle, the following is included: The generated streaming output content of this call will be persistently saved to the database; The status record of this call is updated from the generating state to the interrupted state, which is different from the completed state and the error state. An interruption event is pushed to the client that originates the streaming inference request via a streaming data transmission channel. The interruption event carries the generated partial content so that the client stops the visual feedback of the streaming output and displays the truncated result.

6. The model invocation control method according to any one of claims 1 to 5, characterized in that, After initiating the streaming inference call, the following is included: The call record is periodically checked for changes in the abort flag by polling. When the abort flag changes from indicating that abort has not been requested to indicate that abort has been requested, it is determined whether the control handle has been written into the call record. If the control handle has already been written, then the control handle is executed to abort the current call; otherwise, the process continues to wait until the control handle is written before execution.

7. The model invocation control method according to any one of claims 1 to 5, characterized in that, After initiating the streaming inference call, the following is included: The connection status between the current service instance and the large language model is periodically checked through a heartbeat mechanism. When an abnormal connection status is detected, the execution flag in the call record is used to determine whether the call has been completed. If the execution flag indicates that execution has not been completed, the current call is aborted by executing the control handle and the execution flag is set to indicate that execution has been completed.

8. A model recall control device, characterized in that, include: The inference response module is configured to respond to streaming inference requests forwarded by the gateway server and create a call record for initiating streaming inference calls to the large language model. The call record includes an abort flag initialized as not requested to be aborted and an execution flag initialized as not completed. The inference call module is configured to initiate the streaming inference call to provide the streaming inference result data of the large language model as feedback to the streaming inference request; The arrival response module is configured to respond to the arrival event of the control handle of the streaming inference call, write the control handle to the call record, and detect whether the abort flag indicates that it has been requested to be aborted. If so, the control handle is executed directly to abort the current call. The abort response module is configured to respond to a user abort command relayed via the gateway server by setting the abort flag to indicate that abort has been requested, and when the execution flag indicates that execution has not been completed and has been written to the control handle, execute the control handle and set the execution flag to indicate that execution has been completed.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images