An AI throttling push method and device for streaming response messages, equipment and medium

Abstract

The application discloses a kind of AI stream response message's throttle push method, device, equipment and medium, the method includes: the stream response fragment of upstream AI service output is added to buffer area, and buffer text is obtained;The source type of upstream AI service is identified, and length threshold and time interval threshold are selected accordingly;When the accumulated length reaches length threshold or push interval reaches time interval threshold, generate push trigger signal;In response to the signal, take out buffer text and clean, obtain the message to be pushed;The message to be pushed is sent to downstream platform, empty buffer area and update push time when successful, and backfill to buffer area head when failed.The application is with buffer convergence and the double threshold trigger criterion of adaptive selection according to source type, combine high frequency fine granularity fragment into a small number of batches push, and with failure backfill guarantee content complete, while ensuring that message is timely visible, reduce the push frequency, avoid being discarded by being limited flow due to exceeding downstream platform rate upper limit.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for throttling and pushing AI streaming response messages. Background Technology

[0002] In recent years, AI services based on large language models have widely adopted streaming response methods to continuously output generated content. A single response is typically divided into a large number of fine-grained streaming response segments, each carrying only a few to dozens of characters, arriving sequentially at millisecond intervals. In applications such as enterprise instant messaging robots integrating AI programming command-line tools, customer service robots integrating large language models, and streaming question-and-answer push in educational training scenarios, AI services use robots within instant messaging platforms as the presentation entry point to end users. The streaming response segments output by upstream AI services need to be aggregated through a gateway before being sent to downstream platforms for end users to view.

[0003] In existing technologies, gateways typically process the aforementioned streaming response fragments by forwarding them as is. That is, each time a streaming response fragment is received, it is immediately pushed to the downstream platform as a message, with the push rhythm completely following the output rhythm of the upstream AI service.

[0004] However, downstream platforms such as instant messaging platforms generally impose sending rate limits on single sessions or single bots, for example, stipulating that a maximum number of messages can be sent within a certain number of seconds. Due to the small granularity and high arrival frequency of streaming response segments, forwarding streaming response segments one by one as is would cause the push frequency to far exceed the rate limit of the downstream platform. As a result, push requests are rate-limited, messages are dropped due to rate limiting, and end users experience message breaks and content omissions. The reliability of AI streaming response message push cannot be guaranteed. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for throttling and pushing AI streaming response messages, aiming to solve the technical problem in the prior art where forwarding streaming response fragments one by one as is causes the push frequency to exceed the rate limit of the downstream platform, resulting in messages being discarded due to rate throttling and causing content omissions.

[0006] In a first aspect, embodiments of the present invention provide a throttling push method for AI streaming response messages, including: Receive streaming response fragments output by upstream AI services, and append the streaming response fragments to the buffer in the order of arrival to obtain the accumulated buffered text; Identify the source type of the upstream AI service, and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type; A push trigger signal is generated when the cumulative length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold. In response to the push trigger signal, the buffered text is retrieved from the buffer, and a preset cleaning process is performed on the retrieved buffered text to obtain the message to be pushed. The message to be pushed is sent to the downstream platform and the sending result is obtained. When the sending result is successful, the buffer is cleared and the time of the last push is updated. When the sending result fails, the message to be pushed is filled back into the header of the buffer, forming a new buffered text with the subsequently arriving streaming response fragment.

[0007] Secondly, embodiments of the present invention provide a throttling and pushing device for AI streaming response messages, comprising: The receiving module is used to receive streaming response fragments output by the upstream AI service, and append the streaming response fragments to the buffer in the order of arrival to obtain the accumulated buffered text. The selection module is used to identify the source type of the upstream AI service and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type. The determination module is used to generate a push trigger signal when the cumulative length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold. The cleaning module is used to respond to the push trigger signal, retrieve the buffered text from the buffer, perform preset cleaning processing on the retrieved buffered text, and obtain the message to be pushed. The sending module is used to send the message to be pushed to the downstream platform and obtain the sending result. When the sending result is successful, the buffer is cleared and the time of the last push is updated. The backfill module is used to backfill the message to be pushed to the header of the buffer when the sending result is a failure, so that it forms a new buffered text with the subsequently arriving streaming response fragments.

[0008] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the throttling and pushing method for AI streaming response messages of the first aspect.

[0009] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the throttling and pushing method for AI streaming response messages of the first aspect.

[0010] This invention provides a method for throttling and pushing AI streaming response messages, comprising: receiving streaming response segments output by an upstream AI service; appending the streaming response segments to a buffer in arrival order to obtain accumulated buffered text; identifying the source type of the upstream AI service; selecting a corresponding length threshold and time interval threshold from a preset parameter set according to the source type; generating a push trigger signal when the accumulated length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold; responding to the push trigger signal, retrieving the buffered text from the buffer; performing a preset cleaning process on the retrieved buffered text to obtain a message to be pushed; sending the message to be pushed to a downstream platform and obtaining the sending result; when the sending result is successful, clearing the buffer and updating the time of the last push; when the sending result is unsuccessful, filling the header of the buffer with the message to be pushed, forming a new buffered text with the subsequently arriving streaming response segments. This invention buffers and aggregates streaming response fragments output by upstream AI services instead of forwarding them one by one to obtain accumulated buffered text. It adaptively selects length and time interval thresholds from a preset parameter set according to source type, matching the trigger parameters with the output characteristics of different upstream AI services. A push trigger signal is generated only when the accumulated length of the buffered text reaches the length threshold or the time interval since the last push reaches the time interval threshold. This merges frequently arriving fine-grained streaming response fragments into a small batch for push, constraining the push frequency within the downstream platform's rate limit while ensuring the visible delay of generated content does not exceed the time interval threshold. A preset cleaning process is performed before pushing to ensure the readability of the message to be pushed. If the sending result fails, the message to be pushed is backfilled into the buffer header and retransmitted with the next push, ensuring that the message content is not missing even if the downstream platform is temporarily unavailable. Therefore, this invention solves the technical problem in the prior art where forwarding streaming response fragments one by one as is causes the push frequency to exceed the rate limit of the downstream platform, resulting in messages being discarded due to rate limiting and causing content omissions. While ensuring timely visibility and complete delivery of messages, it reduces the number of push calls to the downstream platform and improves the reliability of AI streaming response message push.

[0011] This invention also provides a throttling and pushing device, computer equipment, and storage medium for AI streaming response messages, which also have the above-mentioned beneficial effects. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a throttling push method for AI streaming response messages provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a throttling push device for AI streaming response messages provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0016] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0017] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] Please see below. Figure 1 , Figure 1 The flowchart of a throttling push method for AI streaming response messages provided in an embodiment of the present invention specifically includes steps S101 to S106.

[0019] S101. Receive streaming response segments output by the upstream AI service, and append the streaming response segments to the buffer in the order of arrival to obtain the accumulated buffered text; S102. Identify the source type of the upstream AI service, and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type; S103. When the cumulative length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold, a push trigger signal is generated. S104. In response to the push trigger signal, the buffered text is retrieved from the buffer, and a preset cleaning process is performed on the retrieved buffered text to obtain the message to be pushed. S105. Send the message to be pushed to the downstream platform and obtain the sending result. When the sending result is successful, clear the buffer and update the time of the last push. S106. When the sending result fails, the message to be pushed is filled back into the header of the buffer, forming a new buffer text with the subsequently arriving streaming response fragment.

[0020] In step S101, "throttling push" refers to a processing method that controls the push frequency by merging frequently arriving content into smaller batches through buffer aggregation. "Upstream AI service" refers to artificial intelligence (AI) services that output generated content in a streaming manner, such as dialogue services based on large language models or AI programming services running as command-line tools. "Streaming response fragment" refers to content segments output by the upstream AI service in a single response process according to the generation progress. A single streaming response fragment typically carries only a few to tens of characters and arrives at millisecond intervals. "Buffer" refers to a storage structure used to temporarily store the text carried by the streaming response fragments. As each streaming response fragment is added in the order of arrival, the accumulated text in the buffer is the buffered text. This step aggregates the high-frequency, fine-grained output of the upstream AI service into buffered text before sending individual streaming response fragments, thus providing statistical and data collection objects for subsequent triggering determination and batch push. The execution entity of the method can be a server, a node in a server cluster, or a container instance deployed in a container orchestration platform. The program implementing the method can be implemented by a general object-oriented programming language and a reactive programming framework, and can be embedded as an independent component in an existing instant messaging gateway program.

[0021] In one embodiment, step S101 includes: A text buffer is created as the buffer when the session is established, and streaming response fragments generated by the upstream AI service are subscribed to; The streaming response fragment is received by subscribing; wherein the streaming response fragment is a structured message carrying a text content field. Extract the values ​​of the text content fields from the structured message, remove the control fields from the structured message, and discard the values ​​when the length of the extracted values ​​is zero, and treat the values ​​as valid text content when the length is non-zero. The valid text content is appended to the end of the text buffer in the order of arrival to obtain the buffered text.

[0022] In this embodiment, the text buffer refers to a memory object that temporarily stores text content in string form, such as a mutable character sequence object. When a session is established, the text buffer is initialized to an empty string, and the time of the last push is initialized to the time the session was established. Subscription refers to establishing a streaming relationship with the upstream AI service, ensuring that each streaming response fragment generated by the upstream AI service is delivered to the receiver. A structured message refers to a data unit organized according to a predetermined format and consisting of several fields. The text content field carries the text presented to the end user, while the control fields include auxiliary fields such as type fields and timestamp fields that are not used for presentation. Extracting the values ​​of the text content fields from the structured message and removing the control fields ensures that the content appended to the text buffer and the cumulative length of subsequent statistics are consistent with the number of characters visible to the end user, avoiding interference from control fields in triggering judgments. Discarding values ​​with a length of zero avoids empty content consuming processing overhead. Appending valid text content to the end of the text buffer in arrival order ensures that the character order of the buffered text is consistent with the upstream generation order. Furthermore, throttling can be performed at the session level. Each session corresponds to an independent text buffer and a last push time. Different sessions do not interfere with each other, allowing the same execution entity to handle multiple concurrent sessions simultaneously. When a session ends, a final push is performed, releasing the context resources such as the text buffer, threshold, and last push time corresponding to that session. When append and push actions in the same session may be executed by different threads, the same mutual exclusion control can be applied to the three types of actions: append, trigger judgment, and retrieval and clearing, to ensure the atomicity of the above actions and prevent newly arrived streaming response fragments from being accidentally cleared or their relative order from being disrupted during the push process.

[0023] For example, an end user asks a code generation question to a bot in an instant messaging client. The upstream AI service returns a response in a streaming manner. The first streaming response fragment is a structured message with the text content field set to "Hello, ," and the second streaming response fragment has the text content field set to "Sample code is given below". After the two arrive in sequence, the buffered text in the text buffer is "Hello, sample code is given below". If a structured message only carries a type field and a timestamp field, and its text content field is empty, then the structured message will not append any content to the buffered text.

[0024] Furthermore, in one embodiment, the structured message output by the upstream AI service can be matched against a preset keyword set to detect whether the response has ended. The preset keyword set includes at least two different completion identifier strings, each corresponding to a feature string used by a different upstream AI service in its message to indicate the end of the response. For example, one upstream AI service may indicate the end of the response with a message whose type field value is "done," while another may indicate the end with a message whose type field value is "claude-complete." The keywords are matched using an "OR" relationship. Therefore, it is not necessary to implement a separate end-determination logic for each upstream AI service. When a new upstream AI service is added, its feature string only needs to be added to the preset keyword set. It should be noted that the object of the matching is the structured message itself, not the extracted valid text content. This is because the completion identifier is usually located in the control field of the structured message, which has been removed during extraction. Matching the extracted text will not result in a match. At the same time, matching the structured message can also prevent the end user from mistakenly judging the response as ended when the same string as the completion identifier accidentally appears in their input. When a structured message hits any keyword in the preset keyword set, a final push is performed. This involves sending the remaining buffered text after cleaning, and then clearing the buffer after sending to prevent residual content from being sent repeatedly when the same session is reused. If the upstream AI service does not output any completion flag but its response stream ends normally, the same final push and clearing actions are performed by the response stream's end callback as a fallback, ensuring that the content in the buffer is not left behind under any circumstances. For upstream AI services that have output all presentable content before the end message, the content remaining in the buffer when the completion flag is hit is usually control information or repetitive summary information. For such upstream AI services, the buffer can be cleared directly upon hit without performing the final push to avoid repeatedly presenting the same content to the end user.

[0025] In step S102, the source type refers to the category obtained by classifying upstream AI services according to their output format and access mode. For example, upstream AI services that output via a HyperText Transfer Protocol (HTTP) streaming interface and upstream AI services that output line by line based on local subprocess standard output belong to different source types. The preset parameter set refers to the set of throttling parameters pre-configured for each source type. The length threshold refers to the lower limit of the cumulative length of buffered text required to trigger a push. For ease of description, the length threshold is denoted as L_min below. The time interval threshold refers to the upper limit of the allowed time interval between two adjacent pushes, denoted as T_min below. Upstream AI services of different source types differ in output granularity, arrival jitter, and pause patterns. This step selects the length threshold and time interval threshold according to the source type to match the throttling parameters with the output characteristics of the upstream AI services, providing a basis for the trigger determination in step S103.

[0026] In one embodiment, step S102 includes: Obtain the service identifier of the upstream AI service, as well as the access method of remote interface call or local subprocess; The service identifier and the access method are used as identification criteria to determine the source type of the upstream AI service; Query the threshold combination corresponding to the source type from the preset parameter table that stores the preset parameter set; When the source type registered in the preset parameter table is not matched by the identification criteria, a threshold combination of the default source type is used, and a length threshold and a time interval threshold are selected from the threshold combination respectively.

[0027] In this embodiment, the service identifier refers to the unique identifier registered when accessing the upstream AI service; the access method refers to the connection form between the upstream AI service and the execution entity, including remote interface calls and local subprocesses. Remote interface calls refer to accessing the remotely deployed upstream AI service through a network interface, while local subprocesses refer to running the upstream AI service as a subprocess on the device where the execution entity resides and reading its standard output. The preset parameter table refers to a key-value mapping with the source type identifier as the key and threshold combinations as the values. For example, it can be implemented as an in-process mapping table and loaded by an external configuration file or configuration center at startup. The threshold combinations include at least a length threshold and a time interval threshold. By using the service identifier and access method together as identification criteria, upstream AI services with similar output characteristics can be grouped into the same source type. Identification and query can be performed once when the session is established. The identification result and the selected threshold are stored together in the context of the session. Subsequent streaming response segments can be directly reused without repeated table lookups, making the processing overhead of a single streaming response segment constant. When identifying any source type that is not registered according to the preset parameter table, the threshold combination of the default source type is used to ensure that the throttling push can still work normally when a new upstream AI service is connected but its parameters have not yet been registered.

[0028] For example, the preset parameter table can register the following three types of source types and their threshold combinations: The first type of source type corresponds to upstream AI services based on HTTP streaming interfaces, with a length threshold L_min of 20 characters and a time interval threshold T_min of 500 milliseconds; the second type of source type corresponds to upstream AI services that output line by line based on the standard output of local child processes, with L_min of 30 characters and T_min of 1000 milliseconds; the third type of source type corresponds to upstream AI services that output through other long-connection streams, with L_min of 25 characters and T_min of 800 milliseconds. The time interval threshold for the second type of source type is greater than that for the first type of source type because upstream AI services that output line by line use lines as the smallest output unit. A single line is relatively large, and there are natural pauses between lines. Using a larger time interval threshold avoids splitting the same logical segment into multiple downstream messages, and its length threshold is correspondingly increased.

[0029] The above values ​​can be determined as follows: the time interval threshold T_min is derived from the rate limiting rules of the downstream platform. Assume the downstream platform allows a single session to send at most N messages within W milliseconds, where W is the statistical time window length for rate limiting (in milliseconds), and N is the number of messages allowed to be sent within that time window. Then, T_min ≥ W / N with a safety margin, ensuring the push frequency remains below the downstream platform's rate limit even in the worst-case scenario. For example, for the first type of source, T_min is set to 500 milliseconds, corresponding to a push frequency of at most 2 messages per second. The length threshold L_min should be larger than the typical granularity of a single streaming response segment, i.e., larger than several to tens of characters, so that a single streaming response segment does not independently become a downstream message. At the same time, L_min should not be too large; otherwise, when the upstream AI service output is slow, it will prolong the visibility time of the first message. The above values ​​are merely illustrative examples; those skilled in the art can adjust them according to the downstream platform's rate limiting rules and user experience requirements.

[0030] Furthermore, the preset parameter table supports runtime hot updates: after the threshold value corresponding to a new source type is written into the preset parameter table, the execution entity does not need to be restarted, and the latest value is selected when the subsequent session is established; hot updates are performed on a group-by-group basis, avoiding reading mismatched intermediate states. When the downstream platform adjusts its rate limiting rules or needs to access a new source type, it only needs to write the corresponding threshold combination into the preset parameter table, without modifying any logic in the remaining steps. As an alternative to identification, in addition to service identifiers and access methods, identification can also be based on the upstream connection's corresponding operating mode configuration items or the media type declared in the upstream response message. For example, event streaming media types and line-separated structured text media types belong to different source types.

[0031] In step S103, the push trigger signal refers to the internal signal used to indicate that a push should be executed immediately; the cumulative length refers to the total number of characters currently contained in the buffered text. For ease of description, the buffer is denoted as buf, and the cumulative length is denoted as len(buf), where len() represents the operation of taking the text length, and buf represents the buffered text in the buffer; the time of the last push refers to the moment when the most recent push was successfully completed, denoted as last, and the current moment is denoted as now. This step uses a dual trigger criterion of length and time: condition A is len(buf)≥L_min, that is, the cumulative length reaches the length threshold L_min; condition B is now-last≥T_min, that is, the time interval since the last push reaches the time interval threshold T_min, where now-last represents the difference between the current moment and the time of the last push; conditions A and B are "OR" relationships, and a push trigger signal is generated when either condition is met. Merging by length alone would result in prolonged delays in push notifications when the upstream AI service is paused, while merging by time alone would lead to excessively large batches when the upstream AI service is outputting rapidly. This step, however, ensures that push batches are not too small when the upstream AI service is continuously outputting, thanks to condition A, and that generated content is not delayed for extended periods when the upstream AI service is paused, thus constraining message visibility latency to a time interval threshold. The time baseline for condition B is the last successful push time (last), not the session start time or the arrival time of the current streaming response segment, thereby ensuring that the push interval is uniformly constrained throughout the entire session.

[0032] In one embodiment, step S103 includes: After each new streaming response fragment is written to the buffer, the cumulative length of the buffered text is calculated. The cumulative length is compared with a forced upper limit threshold in the preset parameter set that is greater than the length threshold. When the cumulative length reaches the forced upper limit threshold, the push trigger signal is generated regardless of the time interval threshold; When the cumulative length does not reach the mandatory upper limit threshold but reaches the length threshold, a push trigger signal is generated; When the time interval since the last push reaches the time interval threshold, a push trigger signal is also generated.

[0033] In this embodiment, the forced upper limit threshold refers to another length threshold in the preset parameter set that is greater than the length threshold. It is used to limit the maximum length of a single push batch, hereinafter denoted as L_max. The length threshold L_min is less than the forced upper limit threshold L_max. The triggering determination is event-driven rather than polling-driven. That is, a condition evaluation is performed once after each new streaming response fragment is written to the buffer, without introducing additional periodic scanning overhead. During the evaluation, the cumulative length len(buf) is first compared with the forced upper limit threshold L_max. If a match is found, a push trigger signal is immediately generated regardless of the time interval threshold, and no other determinations are performed. That is, the triggering determination priority of the forced upper limit is higher than the triggering determination priority of the length threshold and the time interval threshold. If a match is not found, conditions A and B are evaluated in sequence. Setting the forced upper limit threshold has at least the following effects: First, it limits the maximum length of a single push batch, so that the length of a single message on the end user side and the rendering time are controlled; Second, when the upstream AI service outputs at high speed, it switches the push from time-driven to length-driven, avoiding the accumulation of excessively long batches within a T_min period; Third, when the buffered text continues to grow for some reason, it ensures that the push is triggered as soon as possible, preventing the buffer from expanding indefinitely. The value of the mandatory upper limit threshold L_max is constrained by two factors: First, L_max should not exceed the length limit specified by the downstream platform for a single message to avoid triggering fragmentation or truncation on the downstream platform side. Second, the ratio of L_max to L_min can be no less than 3 and no greater than 10. The ratio of L_max to L_min is the value obtained by dividing L_max by L_min. If the ratio is too small, the mandatory upper limit determination will trigger too frequently before the time condition, degenerating into a single length threshold scheme. If the ratio is too large, the content of a single batch will be too long, causing the end user to see too much new text in a single refresh, resulting in a visual jump. For example, following the values ​​for the three source types mentioned above, L_max for the first source type can be 100 characters, L_max for the second source type can be 150 characters, and L_max for the third source type can be 120 characters. The corresponding L_max to L_min ratios for these three sets of values ​​are 5.0, 5.0, and 4.8, respectively, all falling within the above range.

[0034] For example, taking the values ​​of the first type of source as an example, namely L_min is 20 characters, T_min is 500 milliseconds, and L_max is 100 characters, the triggering process under three typical scenarios is as follows. Scenario 1, cumulative length first reaches the target: After the session is established, last is 0 milliseconds and buf is empty. The upstream AI service outputs a streaming response segment at a rate of about 8 characters per segment, approximately every 40 milliseconds. When t is 40 milliseconds, len(buf) is 8, and when t is 80 milliseconds, len(buf) is 16, neither of which meets any triggering condition; when t is 120 milliseconds, len(buf) is 24. At this time, now-last is 120 milliseconds, which is still less than T_min, but len(buf) has reached L_min. Condition A is met, and a push trigger signal is generated. In this scenario, the push is driven by the length condition to ensure that the push batch is not too small when the upstream AI service is continuously outputting. Scenario 2, Time Condition First: Immediately after the previous push, the upstream AI service enters a low-speed output state, followed by only one streaming response segment of length 6. This streaming response segment arrives 500 milliseconds after the previous push and triggers condition evaluation. At this time, len(buf) is still less than L_min, so condition A is not met. However, now-last has reached T_min, so condition B is met, and a push trigger signal is still generated. In this scenario, the push is driven by a time condition to ensure that generated content is not delayed for a long time when the upstream AI service pauses, thus ensuring the immediacy of the interaction. Scenario 3, High-Speed ​​Output Triggers Forced Upper Limit: The upstream AI service then enters a high-speed output state, arriving multiple streaming response segments consecutively within 60 milliseconds, causing len(buf) to rapidly increase from 30 to 104. After the streaming response segment that first makes len(buf) reach L_max is written, the forced upper limit determination is hit. At this time, even though now-last is still less than T_min, a push trigger signal is immediately generated regardless of the time interval threshold to avoid accumulating excessively long batches. The three scenarios mentioned above can alternate within the same session. Each arriving streaming response segment is processed according to the same decision process, without the need to pre-determine which scenario is currently in.

[0035] Furthermore, in a preferred embodiment, the length threshold L_min can serve not only as the triggering threshold for condition A, but also as the lower limit threshold for push batches. That is, condition B only takes effect if len(buf)≥L_min is met. When this preferred embodiment is adopted, when the upstream AI service outputs at a high frequency with a granularity much smaller than L_min, it can avoid generating a large number of push batches with extremely short content due to repeated expiration of time conditions, thereby further reducing the number of pushes.

[0036] It is worth noting that since condition evaluation is performed when a new streaming response segment arrives, condition B may not be evaluated if the upstream AI service experiences a prolonged pause without any new streaming response segments arriving. Therefore, in one alternative implementation, a timer that expires after T_min can be set after each successful push. When the timer expires, a condition evaluation is actively performed. If buf is not empty at this time, a push trigger signal is generated; if buf is empty, no action is taken and the system waits for the next streaming response segment, ensuring that content generated during the pause can still be pushed in a timely manner.

[0037] In step S104, the cleaning process refers to the formatting process performed on the buffered text before pushing. The message to be pushed refers to the text actually sent after the cleaning process. Streaming output is prone to generating extra line breaks and indentation whitespace at the boundaries of streaming response segments. If left unprocessed, these extra whitespaces will be rendered as large blank lines on the downstream platform, affecting the readability of the message. Therefore, this step performs a preset cleaning process on the retrieved buffered text before each push to standardize the presentation of the message to be pushed.

[0038] In one embodiment, step S104 includes: In response to the push trigger signal, all the buffered text is retrieved from the buffer as the text to be cleaned; Compress three or more consecutive newline characters in the text to be cleaned into two newline characters to obtain the compressed text. Delete consecutive spaces or tabs at the end of lines in the compressed text to obtain the cleaned text; The length of the cleaned text is calculated, and when the length of the text is zero, the current push is merged into the next push. When the content length is not zero, the cleaned text will be used as the message to be pushed.

[0039] In this embodiment, the text to be cleaned refers to all buffered text retrieved from the buffer at once in response to a push trigger signal; a newline character refers to a character in the text that indicates a newline, a tab character refers to a tab character used for alignment, and the end of a line refers to the position at the end of a line of text immediately before a newline character. Compressing three or more consecutive newline characters into two newline characters eliminates large blank lines while preserving the paragraph separation semantics expressed by the two newline characters; deleting consecutive spaces or tab characters at the end of a line while retaining the following newline character removes invisible redundant whitespace without disrupting the line break structure. These two processes can be implemented, for example, using preset regular expressions, which are expressions that describe character patterns with specific syntax and are used for matching and replacement. Content length refers to the total number of characters contained in the cleaned text. When the content length is zero, the current push is merged into the next push, that is, the current sending is not performed, the newly arrived content in the buffer is not cleared, and the time of the previous push is not updated, so as to avoid sending empty messages to downstream platforms and wasting their sending quota. When the content length is not zero, the cleaned text is handed over to step S105 as a message to be pushed.

[0040] For example, if the retrieved text to be cleaned contains the text "Code is as follows:", four consecutive newline characters, the text "Example ends", and several consecutive spaces at the end of the line, after compression, the four consecutive newline characters are replaced with two newline characters, resulting in compressed text. Then, after deletion, the consecutive spaces at the end of the line are removed while the newline characters are retained, resulting in cleaned text "Code is as follows:", two newline characters, and "Example ends" consecutively. Since the length of this cleaned text is not zero, it is used as the message to be pushed. Alternatively, if the buffered text retrieved at a certain trigger consists only of spaces and newline characters, and its length is zero after the above processing, this push will be merged into the next push, and subsequent content arriving in the buffer will be processed together with it.

[0041] As a supplementary solution to the cleaning process, whitespace characters at the beginning and end of the text to be cleaned can be removed based on the above processing, making the beginning and end of the message to be pushed more regular.

[0042] In step S105, the downstream platform refers to the platform that receives the message to be pushed and presents it to the end user. In this embodiment, it is a downstream instant messaging platform. An instant messaging platform is a platform that provides real-time message sending and receiving capabilities and displays messages to the end user in the form of a robot or similar format. The sending result refers to the response returned by the downstream platform indicating success or failure for a single transmission. This step actually sends the message to be pushed to the downstream platform and determines the subsequent processing based on the sending result. When the transmission is successful, the buffer is cleared and the time of the previous push is updated to the time of completion of this transmission, establishing a new time baseline for the next round of trigger determination.

[0043] In one embodiment, step S105 includes: A connection is established with the downstream instant messaging platform through a long-connection communication channel or a temporary callback address, and the message to be pushed is divided into at least one sub-message according to the maximum length of a single message of the downstream instant messaging platform. Each of the sub-messages is sent to the downstream instant messaging platform in the original order, and the sending result is received. When the sending result of each of the sub-messages is successful, the buffered text in the buffer is cleared, and the time of the last push is updated to the time of completion of this sending, and the system waits for the next streaming response segment.

[0044] In this embodiment, the long-connection communication channel refers to a pre-established and long-term bidirectional communication connection with the downstream instant messaging platform; the temporary callback address refers to a session-level callback address issued by the downstream instant messaging platform in a session, which has a validity period. This address can be used to send messages before the expiration date, and the long-connection communication channel will be used after the expiration date; the maximum length of a single message refers to the maximum number of characters allowed in a single message by the downstream instant messaging platform; a sub-message refers to each message obtained after the message to be pushed is segmented according to this maximum length. When the length of the message to be pushed does not exceed the maximum length of a single message, the segmentation result is one sub-message; if it exceeds this limit, it is segmented into multiple sub-messages according to the maximum length, and each sub-message is sent sequentially in its original order to ensure that the reading order on the end user's side is consistent with the upstream generation order; the push is considered successful only when the sending results of each sub-message are all successful. At this time, the buffered text in the buffer is cleared, and the time of the last push is updated to the time of completion of this push, then the system waits for the next streaming response segment. A failure to send a request is considered to be due to at least one of the following: the HTTP response code returned by the downstream instant messaging platform is not 2xx, where 2xx represents a success response code; the downstream instant messaging platform returns a business error code indicating that the frequency limit has been exceeded in the response body; the long-connection communication channel is abnormally closed; the request timed out and no response was received; or the temporary callback address has expired.

[0045] For example, if the maximum length of a single message on the downstream instant messaging platform is 100 characters, and the length of a certain message to be pushed is 120 characters, then it is divided into two sub-messages with lengths of 100 characters and 20 characters respectively. The former is sent first, followed by the latter. If both sub-messages are successfully sent, the buffer is cleared and the time of the last push is updated to the time of completion of this push. If the sending result of either sub-message fails, then proceed to step S106 for processing.

[0046] In step S106, backfilling refers to the operation of putting the failed push message back into the buffer. The head of the buffer refers to the first position in the buffer, and the index zero position is the position where the character index is 0. The index refers to the position number of the character in the buffer according to its sequence. In this step, when the sending result is a failure, the push message is not discarded, but is backfilled into the head of the buffer. This allows the failed content and the subsequently arriving streaming response fragments to form a new buffered text, which is then retransmitted as a whole with the next push. This ensures that even in the event of a weak network or a temporary unavailability of the downstream platform, the message content on the end user side is not missing.

[0047] In one embodiment, step S106 includes: If the sending result fails, the time of the last push remains unchanged, and the message to be pushed is retained; The retained messages to be pushed are inserted as a whole into the index zero position of the buffer to complete the backfilling of the header. After backfilling, the header of the buffer stores the content of the failed transmission, and the remaining positions store the streaming response fragments that arrive later. The backfilled content is merged with the subsequently arriving streaming response fragments to form a new buffered text; The new buffered text is used to generate the push trigger signal, and the failed-to-send content is retransmitted with the next push.

[0048] In this embodiment, the message to be pushed is inserted as a whole into the index zero position of the buffer instead of being appended to the end. This is because the content that failed to be sent was generated earlier than the existing content in the buffer, and inserting it into the head ensures that the text seen by the end user is consistent with the upstream generation order. The time of the last push is kept unchanged, so that the next trigger determination is still evaluated based on the original time baseline. That is, the time interval since the last push reaches the time interval threshold immediately or quickly after backfilling. At the same time, backfilling increases the cumulative length of the buffered text, making it easier to reach the length threshold or even the forced upper limit threshold. Both of these factors together ensure that retransmission is triggered in a timely manner. Compared with the scheme of setting up a separate retransmission queue outside the throttling process, the backfilling method in this embodiment does not require maintaining a second storage and a separate scheduling thread for the content to be retransmitted, resulting in low implementation overhead. The retransmitted content and the newly arrived streaming response fragments after backfilling are naturally merged into a batch in the next push, without increasing the number of push calls on the downstream platform, thus not conflicting with the throttling goal. The retransmission order is naturally guaranteed by the character order of the buffer, without the need for additional sequence numbers and sorting logic.

[0049] For example, if the message to be pushed is "ABCDE" and fails to be sent, and the new content "FG" has arrived in the buffer, then the buffered text in the buffer after being filled will be "ABCDEFG", where the header stores the failed content "ABCDE" and the remaining positions store the subsequently arrived content "FG"; when the next push is triggered, "ABCDEFG" is sent as a whole after being cleaned, and the failed content is retransmitted.

[0050] Furthermore, in one optional implementation, the number of consecutive refills for the same batch of content can be recorded, or a cumulative length limit can be set for the buffer. When the number of consecutive refills reaches the preset limit or the cumulative length reaches the cumulative length limit, it is determined that the downstream platform is in a continuously unavailable state. At this time, the earliest part of the content can be discarded, or a status prompt message can be sent to the end user instead, to prevent the buffer from continuously growing and consuming too much memory. Furthermore, considering the situation where the sending request has been received by the downstream platform but its response is lost on the way back, a deduplication flag can be attached to each push batch, so that the downstream platform can discard duplicate batches accordingly, avoiding the same content being presented repeatedly.

[0051] In summary, the AI ​​streaming response message throttling push method provided by this invention aggregates streaming response fragments output by upstream AI services using a buffer, adaptively selects length and time interval thresholds according to source type, generates a push trigger signal when either the accumulated length or time interval condition is met, and sends the processed messages in batches to downstream platforms. In case of sending failure, the message to be pushed is backfilled into the buffer header and retransmitted with the next push. First, by merging high-frequency, fine-grained streaming response fragments into a small batch, the push frequency is constrained within the rate limit of the downstream platform, avoiding rate limiting at the source. Second, the time interval threshold controls the visible delay of generated content, balancing the immediacy of interaction. Third, the adaptive selection of thresholds according to source type allows the same execution entity to simultaneously support multiple upstream AI services with different output characteristics. Fourth, the cleaning process eliminates redundant blanks at fragment boundaries in the streaming output, ensuring message readability. Fifth, failure backfilling ensures that failed-to-send content is retransmitted with the next push, guaranteeing no missing message content. Under the same hardware environment and network conditions, comparative tests were conducted using the same upstream AI service, the same prompt word, and the same downstream instant messaging platform session. The downstream push count was defined as the total number of message sending calls initiated to the downstream instant messaging platform within a single session. The longest visible latency was defined as the maximum difference between the moment a character was generated upstream and the moment the message containing that character became visible to the end user. In a typical scenario with approximately 5000 characters in the generated code response: the scheme of forwarding the streaming response fragment as is triggered 318 times, hitting the downstream instant messaging platform's rate limiting policy 7 times, resulting in message interruption on the end user side; the scheme based solely on a fixed-length threshold merging (i.e., pushing after accumulating 50 characters) triggered 102 downstream pushes, hitting rate limiting once, and failing to push in time when the upstream paused for more than 2 seconds, resulting in a perceived latency of more than 3 seconds for the end user; when using the method described in this embodiment of the invention and taking the aforementioned first type of source type as a parameter, 38 downstream pushes were triggered without triggering rate limiting, and the longest visible latency was 500 milliseconds, constrained by the time interval threshold. Therefore, it can be seen that the embodiments of the present invention can reduce the number of downstream pushes by 5 to 10 times compared with the original forwarding scheme. In this test, it was reduced from 318 times to 38 times. At the same time, the visible delay is controlled by the time interval threshold, and the length of a single batch is controlled by the forced upper limit threshold. The advantages of both rate limiting avoidance and visible delay control are not obtained by sacrificing the other.

[0052] Combination Figure 2 As shown, Figure 2 This is a schematic block diagram of a throttling and pushing device for AI streaming response messages provided in an embodiment of the present invention. The throttling and pushing device 200 for AI streaming response messages includes: The receiving module 201 is used to receive streaming response fragments output by the upstream AI service, and append the streaming response fragments to the buffer in the order of arrival to obtain the accumulated buffered text; The selection module 202 is used to identify the source type of the upstream AI service and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type; The determination module 203 is used to generate a push trigger signal when the cumulative length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold. The cleaning module 204 is used to respond to the push trigger signal, retrieve the buffered text from the buffer, perform preset cleaning processing on the retrieved buffered text, and obtain the message to be pushed. Sending module 205 is used to send the message to be pushed to the downstream platform and obtain the sending result. When the sending result is successful, the buffer is cleared and the time of the last push is updated. The backfill module 206 is used to backfill the message to be pushed to the header of the buffer when the sending result is a failure, so as to form a new buffered text with the subsequently arriving streaming response fragment.

[0053] In this embodiment, the receiving module 201 is used to receive streaming response fragments output by the upstream AI service and append the streaming response fragments to the buffer in the order of arrival to obtain accumulated buffered text; the selection module 202 is used to identify the source type of the upstream AI service and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type; the determination module 203 is used to generate a push trigger signal when the accumulated length of the buffered text reaches the length threshold or the time interval since the last push reaches the time interval threshold; the cleaning module 204 is used to, in response to the push trigger signal, retrieve the buffered text from the buffer and perform preset cleaning processing on the retrieved buffered text to obtain a message to be pushed; the sending module 205 is used to send the message to be pushed to the downstream platform and obtain the sending result. When the sending result is successful, the buffer is cleared and the time of the last push is updated; the backfilling module 206 is used to, when the sending result is unsuccessful, backfill the message to be pushed to the head of the buffer to form a new buffered text with the subsequently arriving streaming response fragments.

[0054] In one embodiment, the receiving module 201 is specifically used for: A text buffer is created as the buffer when the session is established, and streaming response fragments generated by the upstream AI service are subscribed to; The streaming response fragment is received by subscribing; wherein the streaming response fragment is a structured message carrying a text content field. Extract the values ​​of the text content fields from the structured message, remove the control fields from the structured message, and discard the values ​​when the length of the extracted values ​​is zero, and treat the values ​​as valid text content when the length is non-zero. The valid text content is appended to the end of the text buffer in the order of arrival to obtain the buffered text.

[0055] In one embodiment, the selection module 202 is specifically used for: Obtain the service identifier of the upstream AI service, as well as the access method of remote interface call or local subprocess; The service identifier and the access method are used as identification criteria to determine the source type of the upstream AI service; Query the threshold combination corresponding to the source type from the preset parameter table that stores the preset parameter set; When the source type registered in the preset parameter table is not matched by the identification criteria, a threshold combination of the default source type is used, and a length threshold and a time interval threshold are selected from the threshold combination respectively.

[0056] In one embodiment, the determination module 203 is specifically used for: After each new streaming response fragment is written to the buffer, the cumulative length of the buffered text is calculated. The cumulative length is compared with a forced upper limit threshold in the preset parameter set that is greater than the length threshold. When the cumulative length reaches the forced upper limit threshold, the push trigger signal is generated regardless of the time interval threshold; When the cumulative length does not reach the mandatory upper limit threshold but reaches the length threshold, a push trigger signal is generated; When the time interval since the last push reaches the time interval threshold, a push trigger signal is also generated.

[0057] In one embodiment, the cleaning module 204 is specifically used for: In response to the push trigger signal, all the buffered text is retrieved from the buffer as the text to be cleaned; Compress three or more consecutive newline characters in the text to be cleaned into two newline characters to obtain the compressed text. Delete consecutive spaces or tabs at the end of lines in the compressed text to obtain the cleaned text; The length of the cleaned text is calculated, and when the length of the text is zero, the current push is merged into the next push. When the content length is not zero, the cleaned text will be used as the message to be pushed.

[0058] In one embodiment, the sending module 205 is specifically used for: A connection is established with the downstream instant messaging platform through a long-connection communication channel or a temporary callback address, and the message to be pushed is divided into at least one sub-message according to the maximum length of a single message of the downstream instant messaging platform. Each of the sub-messages is sent to the downstream instant messaging platform in the original order, and the sending result is received. When the sending result of each of the sub-messages is successful, the buffered text in the buffer is cleared, and the time of the last push is updated to the time of completion of this sending, and the system waits for the next streaming response segment.

[0059] In one embodiment, the backfill module 206 is specifically used for: If the sending result fails, the time of the last push remains unchanged, and the message to be pushed is retained; The retained messages to be pushed are inserted as a whole into the index zero position of the buffer to complete the backfilling of the header. After backfilling, the header of the buffer stores the content of the failed transmission, and the remaining positions store the streaming response fragments that arrive later. The backfilled content is merged with the subsequently arriving streaming response fragment to form a new buffered text; the new buffered text is used to generate the push trigger signal, and the failed content is retransmitted with the next push.

[0060] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0061] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0062] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, a power supply, a graphics card, etc., to utilize the graphics card's performance to operate the model, such as for inference and training.

[0063] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0064] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for throttling and pushing AI streaming response messages, characterized in that, include: Receive streaming response fragments output by upstream AI services, and append the streaming response fragments to the buffer in the order of arrival to obtain the accumulated buffered text; Identify the source type of the upstream AI service, and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type; A push trigger signal is generated when the cumulative length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold. In response to the push trigger signal, the buffered text is retrieved from the buffer, and a preset cleaning process is performed on the retrieved buffered text to obtain the message to be pushed. The message to be pushed is sent to the downstream platform and the sending result is obtained. When the sending result is successful, the buffer is cleared and the time of the last push is updated. When the sending result fails, the message to be pushed is filled back into the header of the buffer, forming a new buffered text with the subsequently arriving streaming response fragment.

2. The method for throttling and pushing AI streaming response messages according to claim 1, characterized in that, The process of receiving streaming response fragments from the upstream AI service and appending these fragments to a buffer in arrival order yields accumulated buffered text, including: A text buffer is created as the buffer when the session is established, and streaming response fragments generated by the upstream AI service are subscribed to; The streaming response fragment is received by subscribing; wherein the streaming response fragment is a structured message carrying a text content field. Extract the values ​​of the text content fields from the structured message, remove the control fields from the structured message, and discard the values ​​when the length of the extracted values ​​is zero, and treat the values ​​as valid text content when the length is non-zero. The valid text content is appended to the end of the text buffer in the order of arrival to obtain the buffered text.

3. The throttling and push method for AI streaming response messages according to claim 1, characterized in that, The step of identifying the source type of the upstream AI service, and selecting a corresponding length threshold and time interval threshold from a preset parameter set based on the source type, includes: Obtain the service identifier of the upstream AI service, as well as the access method of remote interface call or local subprocess; The service identifier and the access method are used as identification criteria to determine the source type of the upstream AI service; Query the threshold combination corresponding to the source type from the preset parameter table that stores the preset parameter set; When the source type registered in the preset parameter table is not matched by the identification criteria, a threshold combination of the default source type is used, and a length threshold and a time interval threshold are selected from the threshold combination respectively.

4. The method for throttling and pushing AI streaming response messages according to claim 1, characterized in that, The generation of the push trigger signal includes: After each new streaming response fragment is written to the buffer, the cumulative length of the buffered text is calculated. The cumulative length is compared with a forced upper limit threshold in the preset parameter set that is greater than the length threshold. When the cumulative length reaches the forced upper limit threshold, the push trigger signal is generated regardless of the time interval threshold; When the cumulative length does not reach the mandatory upper limit threshold but reaches the length threshold, a push trigger signal is generated; When the time interval since the last push reaches the time interval threshold, a push trigger signal is also generated.

5. The method for throttling and pushing AI streaming response messages according to claim 1, characterized in that, The step of performing a preset cleaning process on the extracted buffered text to obtain the message to be pushed includes: In response to the push trigger signal, all the buffered text is retrieved from the buffer as the text to be cleaned; Compress three or more consecutive newline characters in the text to be cleaned into two newline characters to obtain the compressed text. Delete consecutive spaces or tabs at the end of lines in the compressed text to obtain the cleaned text; The length of the cleaned text is calculated, and when the length of the text is zero, the current push is merged into the next push. When the content length is not zero, the cleaned text will be used as the message to be pushed.

6. The method for throttling and pushing AI streaming response messages according to claim 1, characterized in that, The step of sending the message to be pushed to the downstream platform and obtaining the sending result, and when the sending result is successful, clearing the buffer and updating the time of the last push, includes: A connection is established with the downstream instant messaging platform through a long-connection communication channel or a temporary callback address, and the message to be pushed is divided into at least one sub-message according to the maximum length of a single message of the downstream instant messaging platform. Each of the sub-messages is sent to the downstream instant messaging platform in the original order, and the sending result is received. When the sending result of each of the sub-messages is successful, the buffered text in the buffer is cleared, and the time of the last push is updated to the time of completion of this sending, and the system waits for the next streaming response segment.

7. The method for throttling and pushing AI streaming response messages according to claim 1, characterized in that, The step of filling the header of the buffer with the message to be pushed, forming a new buffered text with the subsequently arriving streaming response fragments, includes: If the sending result fails, the time of the last push remains unchanged, and the message to be pushed is retained; The retained messages to be pushed are inserted as a whole into the index zero position of the buffer to complete the backfilling of the header. After backfilling, the header of the buffer stores the content of the failed transmission, and the remaining positions store the streaming response fragments that arrive later. The backfilled content is merged with the subsequently arriving streaming response fragments to form a new buffered text; The new buffered text is used to generate the push trigger signal, and the failed-to-send content is retransmitted with the next push.

8. A throttling and push device for AI streaming response messages, characterized in that, include: The receiving module is used to receive streaming response fragments output by the upstream AI service, and append the streaming response fragments to the buffer in the order of arrival to obtain the accumulated buffered text. The selection module is used to identify the source type of the upstream AI service and select the corresponding length threshold and time interval threshold from a preset parameter set according to the source type. The determination module is used to generate a push trigger signal when the cumulative length of the buffered text reaches the length threshold, or when the time interval since the last push reaches the time interval threshold. The cleaning module is used to respond to the push trigger signal, retrieve the buffered text from the buffer, perform preset cleaning processing on the retrieved buffered text, and obtain the message to be pushed. The sending module is used to send the message to be pushed to the downstream platform and obtain the sending result. When the sending result is successful, the buffer is cleared and the time of the last push is updated. The backfill module is used to backfill the message to be pushed to the header of the buffer when the sending result is a failure, so that it forms a new buffered text with the subsequently arriving streaming response fragments.

9. A computer device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the throttling push method for AI streaming response messages as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the throttling push method for AI streaming response messages as described in any one of claims 1 to 7.

Images