Connection keep-alive method and system
By sending JSON field fragments in segments and taking into account the time characteristics of the large language model generation process, the connection timeout problem caused by long response time was solved, achieving efficient connection keep-alive and streaming data transmission, thus improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING DEEP SEARCH ARTIFICIAL INTELLIGENCE BASIC TECH RES CO LTD
- Filing Date
- 2026-01-14
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies in large language model API services suffer from connection timeouts due to long response times. Traditional keep-alive methods increase network overhead and resource consumption, cannot effectively transmit generation progress information, and result in a poor user experience.
By segmenting and periodically sending data, and utilizing a global sending pool and message queue to manage field fragments, combined with the requirements for business data sending and connection keep-alive, characters are sent cyclically at fixed intervals using a first-in-first-out principle to avoid unnecessary resource consumption.
It reduces network bandwidth and computing resource waste, lowers connection interruption rates, improves user experience, and enables streaming processing and fast response.
Smart Images
Figure CN121887849A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer network communication technology, and in particular to a method and system for keeping a connection alive. Background Technology
[0002] When a Large Language Model (LLM) provides services via API, the client and server typically communicate using protocols such as HTTP. When generating long content, the total time can easily exceed the default connection timeout of the HTTP protocol or network devices. This means the connection may time out and be disconnected by the server, proxy, or intermediate network devices (such as firewalls) due to prolonged lack of data exchange, leading to request failure. This long response time, inherent to the model's characteristics, is the root cause of API service connection problems. Therefore, a connection keepalive mechanism is needed to maintain the active state of the underlying TCP connection for HTTP requests.
[0003] In existing technologies, common keep-alive methods involve periodically sending TCP heartbeats or HTTP SSE keep-alive events. While these methods effectively prevent connections from being closed due to idle timeouts, they also incur additional network overhead and resource consumption. In scenarios with large-scale concurrent requests, numerous heartbeats consume valuable network bandwidth and increase the computational burden on the server to handle these "invalid" requests. In the context of large language model API services, simply sending heartbeats periodically before generating a complete JSON response is purely for keep-alive purposes and consumes resources inefficiently. Furthermore, this approach fails to convey any effective information about the model generation progress to the client, leaving the user in a "blind wait" state, and the experience is not substantially improved. Therefore, there is an urgent need for an intelligent mechanism that can tightly integrate business data transmission with connection keep-alive requirements. Summary of the Invention
[0004] To overcome the above-mentioned shortcomings, the present invention aims to provide a method and system for connection keep-alive, which reduces resource consumption and maintains connection activity by segmenting and periodically sending data.
[0005] This invention achieves the above objective through the following solution: a method for maintaining connection liveness, comprising the following steps:
[0006] (1) Initialize a global send pool;
[0007] (2) Receive the request, create a dedicated message queue, and store the generated fields into the dedicated message queue in sequence;
[0008] (3) Schedule and send fields at regular intervals;
[0009] (4) The task response is completed and resources are released.
[0010] Preferably, the global sending pool consists of multiple independent message queues, each message queue having a dedicated data structure allocated to each independent request, and fields managed using a first-in, first-out (FIFO) principle.
[0011] Preferably, in step (2), the server parses the client request and creates an asynchronous task to call the large language model to generate the final response return object; the response return object may be in JSON format and may be decomposed into multiple independent field fragments, each field fragment containing a field name and a corresponding value.
[0012] Preferably, the fields in step (2) include instant fields and streaming generated fields. Whether it is an instant field or a part of a streaming generated field, it is encapsulated into an independent field fragment after generation and appended to the tail of the corresponding dedicated message queue.
[0013] Preferably, fields in the message queue are sent in a cyclical manner at fixed intervals.
[0014] Preferably, step (3) includes the following steps:
[0015] (301) Traverse the global send pool;
[0016] (302) Determine if there are fields to be sent in the message queue: If the message queue is empty, proceed to step 303; if the message queue has fields, proceed to step 304.
[0017] (303) Generate and send a keep-alive signal, and execute step 306;
[0018] (304) Determine whether all fields have been generated: If not, proceed to step 305; if all fields have been generated, proceed to step 307.
[0019] (305) Send single characters in sequence;
[0020] (306) After a fixed time interval, proceed to step 302;
[0021] (307) Send all fields in the message queue.
[0022] As a preferred approach, the first field fragment in the message queue should be popped from the message queue, and then the characters of that field fragment should be sent sequentially according to the character order within the field fragment, in order to achieve keep-alive.
[0023] Preferably, step 305 specifically includes the following steps:
[0024] (3051) First determine whether the field fragment that popped up has been sent. If it has not been sent, proceed to step 3052; if it has been sent, proceed to step 3053.
[0025] (3052) Continue sending one character of the field fragment in sequence;
[0026] (3053) If the previous popped field fragment has been sent, pop a new field fragment from the message queue and send the first character of the new field fragment.
[0027] Preferably, the keep-alive signal can be a space character, a predefined empty JSON object, or a heartbeat signal.
[0028] A connection keep-alive system includes: a field generation module, a queue management module, and a sending scheduling module; the field generation module is responsible for parsing the request task and calling a large language model to generate various field fragments in the JSON response feedback; the queue management module stores the generated field fragments into a message queue in sequence; the sending scheduling module is used to schedule the sending of fields, popping the earliest field fragment from the message queue and sending the characters to the client sequentially at fixed time intervals according to character order.
[0029] The beneficial effects of this invention are as follows: This invention utilizes the characteristic that different JSON fields are generated at different times during the large language model response generation process. By sending JSON resources in segments, it combines business data transmission with connection keep-alive requirements. By sending valid field content only when necessary, it avoids the waste of bandwidth and computing resources caused by continuously sending blank characters. By periodically sending generated field fragments, it maintains connection activity, reduces link interruptions due to timeouts, reduces resource consumption, and lowers the interruption rate. Simultaneously, the client can gradually receive partial field content, achieving streaming processing, shortening the overall perceived latency, and improving the user experience. Furthermore, this invention has strong compatibility and is applicable to various JSON-based API protocols without requiring modification to the client's parsing logic. Attached Figure Description
[0030] Figure 1 This is a flowchart illustrating the steps of a connection keep-alive method provided in an embodiment of the present invention;
[0031] Figure 2 This is a flowchart illustrating the timed scheduling and field sending steps provided in an embodiment of the present invention;
[0032] Figure 3 This is a flowchart illustrating the steps of sequentially sending single characters according to an embodiment of the present invention;
[0033] Figure 4 This is a schematic diagram of a connection and keep-alive system provided in an embodiment of the present invention. Detailed Implementation
[0034] The reasoning process of large language models is essentially a sequence generation task. Its core is a complex reasoning process based on deep neural networks, typically requiring significant computational resources and time. The model uses the input prompt as an initial condition, predicting the next most likely token one by one until a complete response is generated. The computational cost and time consumed in this process are closely related to the length of the generated content, the model's complexity, and hardware performance. When the generated content is long, the total time can easily exceed the default connection timeout of the HTTP protocol or network devices. This long response time, determined by the inherent characteristics of the model, is the root cause of API service connectivity problems.
[0035] Connection keep-alive is a technical mechanism designed to maintain a network connection between two network nodes (such as a client and a server) in an active and available state. Its core purpose is to prevent the connection from being actively disconnected by intermediate network devices or the peer system due to prolonged inactivity. Traditional keep-alive methods treat the entire response as an indivisible whole, requiring all content to be generated before sending. This "one-size-fits-all" approach not only leads to unnecessary connection timeouts but also consumes significant resources.
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0037] Example: Figure 1 As shown, a method for keeping a connection alive includes the following steps:
[0038] (1) Initialize a global send pool.
[0039] In one specific embodiment, upon server startup, a global message pool needs to be initialized. This global message pool is a server-side global management container responsible for coordinating the message sending schedule of all concurrent requests. The global message pool consists of multiple independent message queues. Each message queue is allocated a dedicated data structure for each independent request, managing field fragments using a first-in, first-out (FIFO) principle. Each message queue corresponds to one client request. After receiving a client request, the server allocates a dedicated message queue and initiates the corresponding parallel processing flow.
[0040] In one specific implementation, the global send pool can be implemented using a hash table or other thread-safe containers, with the key being the request ID (such as a UUID) and the value being the corresponding message queue. Each message queue uses a linked list or circular buffer to store field entries, and the entry structure includes metadata such as field name, field value, and generation timestamp.
[0041] (2) Receive the request, create a dedicated message queue, and store the generated fields sequentially at the end of the dedicated message queue.
[0042] In one specific embodiment, after receiving a request from the client, the server performs the following operations:
[0043] Generate a unique identifier: Assign a unique request ID (e.g., UUID) to the request.
[0044] Create a dedicated message queue: In the global send pool, create a new, empty message queue (Message Queue) using the request ID as the key.
[0045] The request refers to a complete call initiated by the client to the server in order to obtain the inference results of the large language model. It may include prompt words, user identifier, API endpoint, model identifier, temperature, maximum generation length, and other information required to trigger, configure, and control the large language model generation task.
[0046] In one specific embodiment, the server parses the client request and creates an asynchronous task to invoke the large language model to generate the final response object. The response object can be in JSON format and can be broken down into multiple independent field fragments, including object, request ID, model name, UNIX timestamp of the response object's creation, system fingerprint of the inference system that generated the response object, role, content, token usage, etc. Each field fragment contains a field name and its corresponding value.
[0047] Based on their generation speed and dependencies, fields can be categorized into two types: instantaneous fields and streaming fields. Instantaneous fields are those that can be generated immediately after a request is received, without requiring large model computations, such as request ID (id), object type (object), and model name (model). Streaming fields, on the other hand, are generated word-by-word (Token-by-Token) based on a large model, resulting in slower and continuous generation, such as the output content of the content model (content) and token usage (usage).
[0048] Once each field is generated, it needs to be entered into a message queue. In one specific embodiment, once a field, whether it is an immediate field or part of a stream-generated field, is ready, it is encapsulated into an independent field fragment, and then the entire field fragment is appended to the tail of the dedicated message queue corresponding to the request.
[0049] (3) Schedule and send fields at regular intervals.
[0050] For fields in the message queue, the server sends fields cyclically at fixed intervals. In one specific embodiment, the server internally runs a send scheduler, which is activated at fixed intervals and executes the following cyclic logic to send fields.
[0051] like Figure 2 As shown, it includes:
[0052] (301) Traverse the global send pool. In practice, each time the scheduler is activated, it will traverse all active requests in the global send pool, that is, traverse all existing message queues.
[0053] (302) Perform the following judgment on the message queue corresponding to each request: whether there are fields to be sent in the message queue. If the message queue is empty, it means that no fields have been generated yet. Then, execute step 303 to generate a keep-alive signal and send it. If there are fields in the message queue, it means that the request task has started to return generated fields. Then, execute step 304 to further judge whether all fields have been generated.
[0054] (303) Generate and send a keep-alive signal, and execute step 306 at fixed intervals.
[0055] If the message queue is empty, meaning no field requests have been generated, a keep-alive signal is generated and sent. This keep-alive signal can be a space character, a predefined empty JSON object (such as {}), or a heartbeat signal. The purpose is to maintain a live connection and prevent interruption.
[0056] (304) Determine whether all fields have been generated: If not all fields have been generated, proceed to step 305, retrieve the earliest field fragment stored in the message queue, and send the single character in sequence; if all fields have been generated, proceed to step 307 to send all fields in the message queue.
[0057] (305) Send single characters in sequence.
[0058] The order refers to the character order within the field fragment. In a specific embodiment, step 305 requires popping the first field fragment (the earliest stored field fragment) from the message queue, and then sending the characters of that field fragment sequentially via the network connection to the client, according to the character order within the field fragment, to achieve keep-alive. It is worth noting that a single character is sent in a fixed-time loop performed by the scheduler. For example... Figure 3 As shown, its specific implementation method is as follows:
[0059] (3051) First determine whether the field fragment that popped up has been sent. If it has not been sent, proceed to step 3052; if it has been sent, proceed to step 3053.
[0060] (3052) Continue sending one character of the field fragment in sequence;
[0061] (3053) If the previous popped field fragment has been sent, pop a new field fragment from the message queue and send the first character of the new field fragment.
[0062] (306) After a fixed interval, step 302 is executed to determine whether there are fields to be sent in the message queue.
[0063] The scheduler sets a fixed time interval. Since the typical timeout of network communication relay devices (such as routers and switches) on the Internet exceeds 15 seconds, the fixed time in this embodiment of the invention is 15 seconds. After that, the next round of the loop begins, and step 302 is executed to determine whether there are any fields that have been generated and are yet to be sent.
[0064] (307) Send all fields in the message queue.
[0065] Once the large language model has completed generating all content, and the last field (such as the complete content and usage) has been stored in the message queue, the field generation module marks the message queue as "completed." Then, it executes step 307, combining all remaining unsent fields in the message queue into a string conforming to the complete JSON suffix format and sending them all at once. Once the last field fragment has been sent, the sending process ends, completing the response flow for the request. Furthermore, during this sending process, the status of each field fragment being sent is recorded to ensure that no duplicate sending occurs.
[0066] (4) The task response is completed and resources are released.
[0067] After confirming that all data has been successfully sent to the client, the system removes the message queue corresponding to the request from the global sending pool, releases the relevant resources, and ends the complete lifecycle of this request.
[0068] Through the coordinated work of the above four steps, this invention utilizes the characteristic that different fields have different generation rates during the large language model generation process, combines business data transmission with connection keep-alive requirements, and achieves efficient, segment-by-segment transmission of the large language model generation results, avoiding the waste of invalid traffic, reducing network interruptions, and improving user experience and system reliability.
[0069] like Figure 4 As shown, a connection keep-alive system that implements the above method includes a field generation module, a queue management module, and a sending scheduling module.
[0070] The field generation module is responsible for parsing the request task and calling the large language model to generate various field fragments in the JSON response feedback. It mainly includes receiving the client request task, parsing it, creating an asynchronous task to call the large language model to generate the final response return object, and further decomposing the return object into multiple independent JSON field fragments. Each field fragment contains a field name and its corresponding value, and after all fields are generated, it is marked as a response completion status.
[0071] The queue management module stores the generated field fragments sequentially into message queues. This module includes a global sending pool, which consists of multiple independent message queues. Each message queue is a dedicated data structure allocated to each independent request, managing field fragments using a first-in, first-out (FIFO) principle. Each message queue corresponds to one client request. After receiving a client request, the server assigns a unique request ID (e.g., UUID) to the request and creates a new, empty message queue in the global sending pool using this request ID as the key. After each request returns, the prepared field fragments are written to the corresponding message queue.
[0072] The sending scheduling module is used to schedule the sending of fields. It pops the earliest field fragment from the message queue and sends the characters to the client sequentially at fixed time intervals, character by character. Only one character is sent per time interval. After all field fragments are sent, the next loop begins, popping a new field fragment from the sending queue and sending its first character. The sending scheduling module includes a built-in scheduler, which is activated at fixed intervals and executes the predetermined sending loop logic to send fields.
[0073] The system iterates through all message queues in the global send pool, checking the content to be sent. Based on the message queue status, it decides whether to send business data or a keep-alive signal. If a field fragment already exists in the message queue, the earliest field fragment in the queue is sent according to the first-in, first-out (FIFO) principle. If no field fragment has been generated yet, a keep-alive signal is sent. When the field generation module marks the response as complete, the send scheduling module combines the remaining unsent fields into a complete JSON and sends it, then releases the message queue resource.
[0074] The above description describes specific embodiments of the present invention and the technical principles employed. If changes are made in accordance with the concept of the present invention, and the resulting functions do not exceed the spirit covered by the specification and drawings, they should still fall within the protection scope of the present invention.
Claims
1. A method of connection keep-alive, the method comprising: Includes the following steps: (1) Initialize a global send pool; (2) Receive the request, create a dedicated message queue, and store the generated fields into the dedicated message queue in sequence; (3) Schedule and send fields at regular intervals; (4) Task response complete, release resources.
2. The method of claim 1, wherein, The global sending pool consists of multiple independent message queues. Each message queue is allocated a dedicated data structure for each independent request and manages fields using a first-in, first-out (FIFO) principle.
3. The method of claim 1-2, wherein, In step (2), the server parses the client request and creates an asynchronous task to call the large language model to generate the final response return object; the response return object may be in JSON format and may be decomposed into multiple independent field fragments, each field fragment containing a field name and a corresponding value.
4. The method of claim 1-3, wherein, The fields mentioned in step (2) include real-time fields and streaming generated fields. Whether it is a real-time field or a part of a streaming generated field, it is encapsulated into an independent field fragment after generation and appended to the tail of the corresponding dedicated message queue.
5. The method of claim 1-4, wherein, For fields in the message queue, send fields cyclically at fixed intervals.
6. The method of claim 1-5, wherein, Step (3) includes the following steps: (301) Traverse the global send pool; (302) Determine if there are fields to be sent in the message queue: If the message queue is empty, proceed to step 303; if the message queue has fields, proceed to step 304. (303) Generate and send a keep-alive signal, and execute step 306; (304) Determine whether all fields have been generated: If not, proceed to step 305; if all fields have been generated, proceed to step 307. (305) Send single characters in sequence; (306) After a fixed time interval, proceed to step 302; (307) Send all fields in the message queue.
7. The connection keeping-alive method according to claim 6, characterized in that, In step 305, the first field fragment in the message queue needs to be popped from the message queue, and then the characters of the field fragment are sent sequentially according to the character order within the field fragment, in order to achieve keep-alive.
8. The method of claim 6-7, wherein, Step 305 specifically includes the following steps: (3051) First determine whether the field fragment that popped up has been sent. If it has not been sent, proceed to step 3052; if it has been sent, proceed to step 3053. (3052) Continue sending one character of the field fragment in sequence; (3053) If the previous pop-up field fragment has been sent, pop a new field fragment from the message queue and send the first character of the new field fragment.
9. The method of claim 1-8, wherein, The keep-alive signal can be a space character, a predefined empty JSON object, or a heartbeat signal.
10. A system for connection keep-alive, characterized by include: The module consists of a field generation module, a queue management module, and a send scheduling module. The field generation module is responsible for parsing the request task and calling the large language model to generate various field fragments in the JSON response feedback; the queue management module stores the generated field fragments into the message queue in order; the sending scheduling module is used to schedule the sending of fields, pop the earliest field fragment from the message queue, and send the characters to the client in character order at fixed time intervals.