Route switching method and device based on LLM calling gateway, equipment and medium

Abstract

This application discloses a routing switching method, apparatus, device, and medium based on an LLM-based gateway call. The method includes: real-time acquisition of indicator data for multiple different preset indicators of a target model; when a model switching instruction is generated, stopping the forwarding of tokens generated by the target model to the client and performing an atomic snapshot to record the original request parameters of the inference request, a first token sequence, and a second token sequence; performing semantic boundary detection on the second token sequence to determine the segmentation position; forwarding tokens before the segmentation position to the client and updating the first token sequence, and deleting tokens after the segmentation position; generating a target context based on the original request parameters and the first token sequence, and initiating a continuation request for the inference request to a backup model of the target model based on the target context; reading tokens from the backup model and pushing them to the client.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a routing switching method, apparatus, device, and medium based on LLM-based gateway invocation. Background Technology

[0002] Streaming (SSE / Streaming) has become the mainstream delivery mode for Large Language Model (LLM) inference services. In streaming mode, the model pushes generated content to the client one token at a time, allowing users to see the generation process in real time, significantly improving the interactive experience.

[0003] Once streaming response is enabled, traditional call gateways (such as large-scale enterprise gateways) cannot switch models for established streaming connections. If the selected model experiences performance degradation, the call gateway can only wait for a timeout or be passively interrupted, resulting in the client receiving a truncated and incomplete response, which affects the user experience. Summary of the Invention

[0004] This application provides a routing switching method, apparatus, device, and medium based on LLM-based gateway calls, which can improve the user experience in large-model streaming response scenarios.

[0005] In a first aspect, embodiments of this application provide a route switching method based on an LLM call gateway, the method being applied to an LLM call gateway, the method comprising: During the streaming processing of inference requests by the target model, indicator data of multiple different preset indicators of the target model are collected in real time. When a model switching instruction is generated based on multiple indicator data and a preset model switching decision, the forwarding of the token generated by the target model to the client is stopped, and an atomic snapshot is performed on the current token stream state of the target model to record the original request parameters of the inference request, the first token sequence that has been pushed to the client, and the second token sequence that has been generated but not pushed to the client. Perform semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence; Forward the tokens before the split position in the second token sequence to the client and update the first token sequence, and delete the tokens after the split position in the second token sequence; A target context is generated based on the original request parameters and the first token sequence, and a continuation request for the inference request is initiated to the backup model of the target model based on the target context; The token is read from the backup model and pushed to the client.

[0006] In some embodiments, the LLM call gateway stores multiple preset index thresholds; the model switching instruction is generated based on the following steps: Determine whether there are any abnormal indicator data among the plurality of indicator data that are higher than the corresponding indicator threshold; If the abnormal indicator data exists, it is determined whether the target model meets multiple preset model switching sub-conditions. The multiple sub-conditions include the target model having a current generated number of tokens greater than a preset first token amount threshold, the target model having a remaining number of tokens for the inference request greater than a preset second token amount threshold, the target model having a historical switching number less than a preset switching number threshold, and the target model having a backup model in an available state. If the target model satisfies multiple preset model switching sub-conditions, then the model switching instruction is generated.

[0007] In some embodiments, after determining whether the target model satisfies multiple preset model switching sub-conditions, the method further includes: If the number of tokens currently generated is less than or equal to the first token quantity threshold, then the tokens generated by the target model are discarded, and a new model is selected for the inference request; If the remaining token amount for the task is less than or equal to the second token amount threshold, then the inference request will continue to be executed using the current target model. If the number of historical switching events is greater than or equal to the switching event threshold, or if all backup models of the target model are unavailable, a degradation strategy is triggered.

[0008] In some embodiments, performing semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence includes: Starting from the end of the second token sequence and tracing backwards, target symbol detection is performed, where the target symbols include sentence end symbols, paragraph separators, and clause end symbols; The segmentation position is determined based on the first detected target symbol.

[0009] In some embodiments, the LLM call gateway connects to the client via a client-side streaming channel and to the target model via a model-side streaming channel. The LLM call gateway deploys a monitoring probe in the inbound direction of the model-side streaming channel. The real-time collection of indicator data for multiple different preset indicators of the target model includes: The monitoring probe collects data on multiple preset indicators in real time.

[0010] In some embodiments, the LLM call gateway includes a buffer proxy layer; stopping the forwarding of the token generated by the target model to the client includes: Stop forwarding the Token in the buffer proxy layer to the client.

[0011] In some embodiments, generating a target context based on the original request parameters and the first token sequence, and initiating a continuation request for the inference request to a backup model of the target model based on the target context, includes: The original request parameters and the first Token sequence are concatenated to form the target context; The continuation request is generated based on the target context and preset additional explicit continuation instructions.

[0012] Secondly, embodiments of this application also provide a routing switching device based on an LLM call gateway, the device being configured in the LLM call gateway, comprising: The transceiver unit is used to collect indicator data of multiple different preset indicators of the target model in real time during the streaming processing of inference requests by the target model; when a model switching instruction is generated based on the multiple indicator data and the preset model switching decision, the forwarding of the token generated by the target model to the client is stopped. The processing unit is configured to perform an atomic snapshot of the current Token stream state of the target model to record the original request parameters of the inference request, the first Token sequence pushed to the client, and the second Token sequence generated but not pushed to the client; perform semantic boundary detection on the second Token sequence to determine the segmentation position at the end of the second Token sequence; forward the Tokens in the second Token sequence before the segmentation position to the client and update the first Token sequence, and delete the Tokens in the second Token sequence after the segmentation position; generate a target context based on the original request parameters and the first Token sequence, and initiate a continuation request for the inference request to the backup model of the target model based on the target context; The transceiver unit is also used to read the Token from the backup model and push it to the client.

[0013] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0015] This application provides a routing switching method, apparatus, device, and medium based on an LLM call gateway. The method, applied to an LLM call gateway, includes: during the streaming processing of inference requests by a target model, real-time acquisition of indicator data for multiple preset indicators of the target model; when a model switching instruction is generated based on the multiple indicator data and a preset model switching decision, stopping the forwarding of tokens generated by the target model to the client, and performing an atomic snapshot of the current token stream state of the target model to record the original request parameters of the inference request, a first token sequence already pushed to the client, and a second token sequence that has been generated but not pushed to the client; performing semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence; forwarding tokens before the segmentation position in the second token sequence to the client and updating the first token sequence, and deleting tokens after the segmentation position in the second token sequence; generating a target context based on the original request parameters and the first token sequence, and initiating a continuation request for the inference request to a backup model of the target model based on the target context; reading tokens from the backup model and pushing them to the client. Firstly, the LLM call gateway in this embodiment can monitor the performance degradation of the target model in real time based on multiple indicator data of the collected target model, and switch the current inference request to the backup model of the target model when performance degradation occurs, reducing the probability of the client receiving an incomplete response. Secondly, the backup model receives the continuation request based on the original request parameters of the current inference request and the first token sequence, and continues to generate the response of the inference request based on the first token sequence. The backup model will not generate content that is duplicated with the target model, thus avoiding the client receiving duplicate content. Thirdly, the end of the token sequence generated by the target model and sent to the client is truncated after semantic boundary detection, so that the client receives a complete and smooth output, improving the user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the route switching method based on LLM calling the gateway provided in this application embodiment; Figure 2A schematic diagram of a sub-process of the route switching method based on LLM calling the gateway provided in the embodiments of this application; Figure 3 This is another sub-process diagram of the route switching method based on LLM calling the gateway provided in the embodiments of this application; Figure 4 This is another sub-process diagram of the route switching method based on LLM calling the gateway provided in the embodiments of this application; Figure 5 A schematic block diagram of a routing switching device based on an LLM-based gateway provided in an embodiment of this application; Figure 6 A schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been obtained by authorized entities (who have known and consented) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] This application provides a method, apparatus, device, and medium for route switching based on LLM-based gateway invocation.

[0024] The execution entity of the LLM-based call gateway routing switching method can be the LLM-based call gateway routing switching device provided in the embodiments of this application, or a computer device that integrates the LLM-based call gateway routing switching device. The LLM-based call gateway routing switching device can be implemented in hardware or software, and the computer device is an LLM call gateway (such as a large model enterprise gateway).

[0025] Figure 1 This is a flowchart illustrating the route switching method based on LLM calling the gateway provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps S110-S160.

[0026] S110. During the streaming processing of inference requests by the target model, the indicator data of multiple different preset indicators of the target model are collected in real time.

[0027] In some embodiments, the LLM call gateway connects to the client via a client-side streaming channel and to the target model via a model-side streaming channel. The LLM call gateway deploys a monitoring probe in the inbound direction of the model-side streaming channel. The real-time collection of indicator data for multiple different preset indicators of the target model includes: The monitoring probe collects data on multiple preset indicators in real time.

[0028] Specifically, the multiple indicator data in this application embodiment include first-byte delay (Time-To-First-Token, TTFT) data, token generation rate (Tokens Per Second, TPS) and auxiliary indicator data, wherein the auxiliary indicator data includes at least one of the following: continuous silence duration (time since the last token received), cumulative error rate, and upstream connection jitter.

[0029] Where, the first byte delay TTFT: refers to the time difference from the time t0 when the request is forwarded to the main model until the time t1 when the first non-empty token pushed by the main model is received, TTFT = t1-t0; Token generation rate: refers to the instantaneous generation rate obtained by dividing the number of tokens generated by the target model within a configurable sliding window (default 2 seconds) by the length of the time window.

[0030] In one embodiment, after the LLM gateway receives the inference request sent by the client, it schedules the target model to start streaming inference. The target model generates response content token by token and continuously pushes it to the gateway through the streaming processing channel on the model end. At the same time, the gateway collects multi-dimensional preset indicators in real time, including but not limited to: first byte delay of the first token, token generation rate, continuous silent duration, and error output rate, continuously monitoring the real-time running status of the model and providing data basis for switching decisions.

[0031] In this embodiment of the application, the target model is a model that is currently processing inference requests based on LLM call gateway routing. The backend of LLM call gateway routing maintains multiple LLM models, and each LLM model has one or more backup models.

[0032] When the LLM gateway initially receives an inference request, it selects the primary model for that request. Specifically, the primary model is the preferred inference model selected by the LLM gateway based on the routing strategy at the initial request stage, responsible for the initial generation of the streaming response. The backup model is a pre-configured compatible alternative model of the primary model. When the primary model experiences performance degradation or anomalies, the gateway switches over and takes over the subsequent generation tasks. The primary and backup models follow a preferred / backup relationship, jointly completing the streaming output of a single request. During subsequent model switching, the LLM gateway can achieve seamless connection between the two through context snapshots and semantic boundary alignment, presenting a single continuous SSE stream to the outside world.

[0033] In this embodiment of the application, the target model can be the primary model (at this time, the inference request has not undergone model switching) or a backup model of the primary model (at this time, the inference request has undergone model switching).

[0034] S120. When a model switching instruction is generated based on multiple indicator data and a preset model switching decision, the forwarding of the token generated by the target model to the client is stopped, and an atomic snapshot is performed on the current token stream state of the target model to record the original request parameters of the inference request, the first token sequence that has been pushed to the client, and the second token sequence that has been generated but not pushed to the client.

[0035] In this embodiment of the application, the LLM call gateway will determine whether to generate a model switching instruction based on multiple real-time monitored indicator data and pre-set model switching decisions through multiple judgment dimensions.

[0036] In some embodiments, specifically, the LLM call gateway stores multiple preset metric thresholds, each corresponding to a different metric. See [link to relevant documentation]. Figure 2 Specifically, the model switching instruction is generated through the following steps: S1201. Determine whether there are any abnormal indicator data among the plurality of indicator data that are higher than the corresponding indicator threshold.

[0037] In this embodiment, different threshold values ​​are preset for each type of preset index. For example, the default TTFT threshold is 3 seconds and the TPS threshold is 5 t / s. When the index data of any preset index is higher than the index threshold, it is determined that the target model has experienced performance degradation.

[0038] Specifically, in some embodiments, when the data of any preset indicator is higher than the corresponding indicator threshold for a preset duration (e.g., 3 seconds), it is determined that the target model has experienced performance degradation.

[0039] S1202. If the abnormal indicator data exists, determine whether the target model meets the preset multiple model switching sub-conditions.

[0040] The sub-conditions include: the number of tokens currently generated by the target model for the inference request is greater than a preset first token amount threshold; the number of remaining tokens for the target model for the inference request is greater than a preset second token amount threshold; the number of historical switching times of the target model for the inference request is less than a preset switching time threshold; and the backup model of the target model is in an available state.

[0041] In some embodiments, the functions of each sub-condition are as follows: (a) The number of tokens generated so far exceeds the first token threshold: This indicates that valid content has been output and switching is meaningful; (b) Remaining task tokens > second token threshold: This indicates that a large amount of content still needs to be generated, making switching necessary; (c) Historical handover count < threshold: Avoid infinite cascading handovers; (d) Alternate model available: This indicates that a switchable object is available.

[0042] In this embodiment, when there is abnormal indicator data and the target model meets multiple preset model switching sub-conditions, the model switching action is executed subsequently, making the decision more robust and intelligent.

[0043] In some embodiments, the number of currently generated tokens is used to determine whether the current model is worth switching and the switching context size; if the number of currently generated tokens is less than or equal to the first token quantity threshold, the tokens generated by the target model are discarded, and a new model is selected for the inference request; In this embodiment, the content corresponding to the first token threshold is extremely short, and the gateway has not yet sent the generated token to the client. At this time, the benefit of switching is low, so it can be retried or abandoned directly.

[0044] In some embodiments, if the remaining token amount of the task is less than or equal to the second token amount threshold, the inference request is continued to be executed using the current target model. The remaining token amount of the task is calculated based on the request parameter max_tokens and the number of generated tokens. If the remaining amount is too small, the switch is not performed and the main model is waited for to finish. If the remaining amount is sufficient, the switch is performed to ensure smooth subsequent generation.

[0045] In some embodiments, if the number of historical switching events is greater than or equal to the number of switching events threshold (e.g., two), or if the backup models of the target model are all unavailable, a degradation strategy is triggered.

[0046] The downgrade strategies include: (a) Continue to wait for the final continuation model to complete; (b) If the model also degrades, push the generated content to the client and mark it as a partial response, prompting the user to retry; (c) Triggering the fuse provides short-term shielding for the deteriorated model.

[0047] In some embodiments, the LLM call gateway further includes a switching decision unit. In this case, the presence or absence of the abnormal indicator data is determined by the monitoring probe, the satisfaction of the model switching sub-condition is determined by the switching decision unit, and the monitoring probe immediately sends an alarm event to the switching decision unit when it detects that the indicator exceeds the threshold.

[0048] In some embodiments, the LLM call gateway includes a buffer proxy layer; stopping the forwarding of the token generated by the target model to the client includes: Stop forwarding the Token in the buffer proxy layer to the client.

[0049] The second token sequence includes tokens in the buffer proxy layer, or tokens being transmitted in the model-end streaming channel, or tokens in both the buffer proxy layer and the model-end streaming channel.

[0050] S1203. If the target model satisfies multiple preset model switching sub-conditions, then the model switching instruction is generated.

[0051] In this embodiment, when the abnormal indicator data of the target model is detected and multiple preset model switching sub-conditions are met, the model switching instruction is generated.

[0052] When a model switching instruction is generated, the gateway immediately stops forwarding the token generated by the target model to the client and immediately performs an atomic snapshot of the current token stream state of the target model in response to the inference request, including: (a) Record the original request parameters, including the complete Prompt, System directives, and all call parameters of the original request; (b) Record the first Token sequence that has been successfully pushed to the client (indexed by time order and generation order) and the second Token sequence that has been generated but not pushed to the client.

[0053] Specifically, when the target model meets the preset switching decision conditions, the gateway immediately stops forwarding tokens to the client within the buffer proxy layer and locks the current streaming link; the atomic snapshot refers to the instantaneous freezing of the entire state of the current token stream, which cannot be interrupted or tampered with, and synchronously records three types of core information: the original request parameters (prompt, generation configuration, etc.), the complete token sequence that has been successfully pushed to the client (first token sequence), and the token sequence that has been generated by the model but is temporarily stored in the buffer proxy layer and has not been pushed (second token sequence), and completely saves the context information before the switch.

[0054] The atomic snapshot in this application embodiment can realize the instantaneous freezing of the streaming state, completely save the context, avoid information loss during switching, and provide a data foundation for seamless continuity.

[0055] S130. Perform semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence.

[0056] In some embodiments, please refer to Figure 3 Specifically, the splitting position at the end of the second token sequence is determined through the following steps: S1301. Backtracking from the end of the second Token sequence, perform target symbol detection, where the target symbols include sentence end symbols, paragraph separators, and clause end symbols; S1302. Determine the segmentation position based on the first detected target symbol.

[0057] Specifically, the optimal segmentation position k' (k' ≤ k) is located by backtracking the nearest sentence terminator (period, question mark, exclamation mark and its multilingual variants), paragraph separator or clause terminator (comma, semicolon) in the generated token sequence, where k is the number of tokens in the token sequence; and all tokens before the segmentation position are retained as continuation context, while the last segment tokens in the range (k', k) are discarded to ensure semantic closure.

[0058] The gateway identifies semantic boundaries by tracing back from the end of the second token sequence within the buffer proxy layer, distinguishing between complete semantic segments and fragmented content, determining the optimal segmentation position, and avoiding the push of incomplete sentences.

[0059] Through the embodiments of this application, semantic boundaries can be accurately identified, complete sentences can be pushed, the client can avoid receiving fragmented content, and the interactive experience can be improved.

[0060] S140. Forward the tokens before the split position in the second token sequence to the client and update the first token sequence, and delete the tokens after the split position in the second token sequence.

[0061] In this embodiment, the gateway pushes the complete semantic fragment before the segmentation position to the client, and simultaneously merges the fragment into the first Token sequence, updating the issued content; fragments without valid semantics after the segmentation position are directly deleted and do not participate in subsequent continuation.

[0062] As can be seen, this application embodiment only pushes complete semantic fragments and discards invalid fragments, ensuring the fluency of client content, while updating the context to achieve the continuity of pushed content.

[0063] S150. Generate a target context based on the original request parameters and the first Token sequence, and initiate a continuation request for the inference request to the backup model of the target model based on the target context.

[0064] In this embodiment, the gateway concatenates the original request parameters with the updated first token sequence to construct a complete target context and initiates a continuation request to the pre-configured backup model. The backup model continues to generate based on the existing complete content, avoiding duplicate output of the previous text.

[0065] In some embodiments, specifically, please refer to Figure 4 Step S150 includes: S1501. Concatenate the original request parameters and the first Token sequence to form the target context; S1502. Generate the continuation request based on the target context and the preset additional explicit continuation instructions.

[0066] Specifically, the original request parameters are concatenated with the generated content up to position k' to form a new context: context_continued = original_prompt + tokens[1..k'], and a continuation request is sent to the backup model. Explicit continuation instructions (such as "Please continue writing directly from the following content, without repeating existing content") can be attached to the request body to maximize the backup model's understanding of the continuation scenario.

[0067] S160. Read the Token from the backup model and push it to the client.

[0068] In this embodiment, the gateway establishes a model-side streaming processing channel with the backup model. Through the connection of the model-side streaming processing channel, the gateway receives the token generated by the backup model in real time and continuously pushes it to the client streaming channel, ensuring that the user-side link is uninterrupted throughout the entire process.

[0069] In this embodiment, the gateway maintains a single, stable client streaming channel externally, hiding the details of primary / backup model switching from the client. Key implementation points: (a) Buffered proxy layer buffer: Immediately stop forwarding tokens from the target model to the client after the switching decision is triggered to avoid sending the tail token after k', which would cause duplication; (b) Stream continuation: After the backup model starts generating, the gateway reads the Token from the backup model stream and continues to push it to the client SSE channel starting from Token (k'+1); (c) Channel continuity: The client's fetch / EventSource connection does not reconnect or renegotiate throughout the entire process. Only a very short interval (sub-second level) is felt in the token timing. EventSource and fetch are two standard technical implementation methods for the client to receive LLM streaming output. They are used to establish a user-side streaming channel between the client and the gateway to receive content pushed by token in real time, achieving a typewriter effect.

[0070] In some embodiments, this application also provides switchover logs and auditing. Each switchover event generates a structured log recording: trigger timestamp, trigger metric snapshot (measured TTFT / TPS values, threshold), primary model (including target model) ID, standby model ID, switchover location token number k', request ID, tenant ID, and switchover decision time. The logs are simultaneously entered into a real-time monitoring system (for anomaly alerts and SLA observation) and a historical archiving system (for post-event auditing and routing strategy optimization).

[0071] In summary, firstly, the LLM call gateway in this embodiment can monitor the performance degradation of the target model in real time based on multiple collected indicator data of the target model, and switch the current inference request to the backup model of the target model when performance degradation occurs, reducing the probability of the client receiving an incomplete response; secondly, the backup model receives the continuation request based on the original request parameters of the current inference request and the first token sequence, and continues to generate the response of the inference request based on the first token sequence, so that the backup model will not generate content duplication with the target model, avoiding the client receiving duplicate content; thirdly, the end of the token sequence generated by the target model that needs to be sent to the client is truncated after semantic boundary detection, so that the client receives a complete and smooth output, improving the user experience.

[0072] In addition, this application has the following specific beneficial effects: (1) Fundamental improvement in service continuity: Achieve sub-second seamless switching when the target model performance deteriorates, reducing the user-perceptible streaming interruption rate to near zero; This is a fundamental improvement in the availability of streaming LLM services from interruption upon failure to smooth transition from failure, enabling enterprise-level streaming applications to support high SLA commitments. (2) Dual guarantee of content integrity: Through the context splicing mechanism of semantic boundary alignment, the semantic continuity of the content before and after the switch is guaranteed, so that there will be no repeated sending of the generated content (the problem of rough restart) or the content will be truncated (the problem of direct disconnection); the user receives a complete and smooth output and has no perception of the mid-switching. (3) Real-time switching decision in milliseconds: The end-to-end decision-making time from indicator degradation detection to switching execution is very small (usually less than 200ms). In addition, the short pause perceived by the user after the backup model API network round trip is usually very small (generally controlled within 500ms). This extremely fast switching response makes the impact of streaming interruption almost imperceptible. (4) Operational value of enhanced observability: Each switching event generates structured logs (trigger indicators, switching time, primary / standby model, switching location Token sequence number, request ID, etc.), providing refined streaming quality monitoring data for the gateway's observability system; the operations and maintenance team can use this data to analyze model performance trends, optimize switching thresholds, and identify potential problems. (5) Configurable multi-level thresholds: Both TTFT threshold and TPS threshold support dynamic configuration at runtime without restarting the gateway service; the operation and maintenance team can adjust the sensitivity according to the business scenario. Latency-sensitive services can be configured with lower thresholds (for faster switching), while stable services can be configured with higher thresholds (to avoid accidental switching). (6) Switching circuit breaker protection mechanism: Set the maximum number of switching times for a single request to prevent cascading switching loops in extreme scenarios from exhausting resources; when the number of switching times reaches the upper limit, the system will degrade to waiting for the target model to recover or returning the generated part of the content to ensure that there is no dead loop switching; (7) Multi-semantic boundary detection capability: The gateway detects multiple semantic boundary types, such as sentence end (period, question mark, exclamation mark and their multilingual variants), paragraph separator, and clause end (comma, semicolon), and selects the optimal segmentation position from them; this multi-level boundary detection enables the switch to find a suitable segmentation point in as many positions as possible; (8) Comprehensive judgment of auxiliary indicators: In addition to the main indicators of TTFT and TPS, the monitoring probe also collects auxiliary indicators such as continuous silent duration, cumulative error rate, and upstream connection jitter; the switching decision is made by combining multiple indicators to avoid excessive switching caused by false alarms of a single indicator; (9) Semantic boundary alignment context optimization: When switching, instead of simply switching from the latest token, it backtracks to the nearest semantic boundary (period / paragraph marker), retaining all tokens before the split position as the continuation context; discarding the tail tokens in the range of (k', k] to ensure semantic closure, so that the backup model faces a complete closed context rather than incomplete fragments when continuing; (10) Multidimensional information integration for switching decisions: Switching decisions not only rely on performance metrics, but also take into account (a) the number of tokens currently generated (to determine whether it is worthwhile to switch); (b) the estimated amount of remaining tasks; (c) the availability of backup models; and (d) the number of historical switching attempts for the current request. This multidimensional information integration makes the decision more robust and intelligent. (11) Complete transparency to the client SSE protocol: The gateway maintains a single and stable client streaming channel to the outside world, hiding the details of the primary and backup model switching from the client; the client's fetch / EventSource connection does not reconnect or renegotiation throughout the process, and only a very short interval (sub-second level) is felt in the Token timing. The entire switching is completely transparent to the client code and requires zero modification. (12) Graceful degradation in failure scenarios: If all backup models are unavailable (a rare but extreme scenario that needs to be considered), the system will automatically degrade to waiting for the target model to recover or return the generated partial content and mark it as a partial response, prompting the client to retry, thus avoiding service unresponsiveness.

[0073] Figure 5 This is a schematic block diagram of a routing switching device based on an LLM-based gateway call, provided in an embodiment of this application. Figure 5As shown, corresponding to the above-described route switching method based on an LLM call gateway, this application also provides a route switching device 500 based on an LLM call gateway. This route switching device 500 includes a unit for executing the above-described route switching method based on an LLM call gateway, and can be configured in an LLM call gateway. Specifically, please refer to... Figure 5 The routing switching device 500 based on LLM call gateway includes a transceiver unit 501 and a processing unit 502, wherein: The transceiver unit 501 is used to collect indicator data of multiple different preset indicators of the target model in real time during the streaming processing of inference requests by the target model; when a model switching instruction is generated based on the multiple indicator data and the preset model switching decision, the forwarding of the token generated by the target model to the client is stopped. Processing unit 502 is configured to perform an atomic snapshot of the current Token stream state of the target model to record the original request parameters of the inference request, the first Token sequence pushed to the client, and the second Token sequence generated but not pushed to the client; perform semantic boundary detection on the second Token sequence to determine the segmentation position at the end of the second Token sequence; forward the Tokens in the second Token sequence before the segmentation position to the client and update the first Token sequence, and delete the Tokens in the second Token sequence after the segmentation position; generate a target context based on the original request parameters and the first Token sequence, and initiate a continuation request for the inference request to the backup model of the target model based on the target context; The transceiver unit 501 is also used to read the Token from the backup model and push it to the client.

[0074] In some embodiments, the LLM call gateway stores multiple preset index thresholds corresponding to different preset indices; the processing unit 502 generates a model switching instruction based on the following steps: Determine whether there are any abnormal indicator data among the plurality of indicator data that are higher than the corresponding indicator threshold; If the abnormal indicator data exists, it is determined whether the target model meets multiple preset model switching sub-conditions. The multiple sub-conditions include the target model having a current generated number of tokens greater than a preset first token amount threshold, the target model having a remaining number of tokens for the inference request greater than a preset second token amount threshold, the target model having a historical switching number less than a preset switching number threshold, and the target model having a backup model in an available state. If the target model satisfies multiple preset model switching sub-conditions, then the model switching instruction is generated.

[0075] In some embodiments, after performing the step of determining whether the target model meets the preset multiple model switching sub-conditions, the processing unit 502 is further configured to: If the number of tokens currently generated is less than or equal to the first token quantity threshold, then the tokens generated by the target model are discarded, and a new model is selected for the inference request; If the remaining token amount for the task is less than or equal to the second token amount threshold, then the inference request will continue to be executed using the current target model. If the number of historical switching events is greater than or equal to the switching event threshold, or if all backup models of the target model are unavailable, a degradation strategy is triggered.

[0076] In some embodiments, when the processing unit 502 performs the step of performing semantic boundary detection on the second token sequence and determining the segmentation position of the tail of the second token sequence, it is specifically used for: Starting from the end of the second token sequence and tracing backwards, target symbol detection is performed, where the target symbols include sentence end symbols, paragraph separators, and clause end symbols; The segmentation position is determined based on the first detected target symbol.

[0077] In some embodiments, the LLM call gateway connects to the client via a client-side streaming channel and to the target model via a model-side streaming channel. The LLM call gateway deploys a monitoring probe in the inbound direction of the model-side streaming channel. When performing the step of real-time acquisition of indicator data for multiple different preset indicators of the target model, the transceiver unit 501 is specifically used for: The monitoring probe collects data on multiple preset indicators in real time.

[0078] In some embodiments, the LLM call gateway includes a buffer proxy layer; when the transceiver unit 501 executes the step of stopping the forwarding of the token generated by the target model to the client, it is specifically used for: Stop forwarding the Token in the buffer proxy layer to the client.

[0079] In some embodiments, when the processing unit 502 performs the step of generating a target context based on the original request parameters and the first token sequence, and initiating a continuation request for the inference request to the backup model of the target model based on the target context, it is specifically used for: The original request parameters and the first Token sequence are concatenated to form the target context; The continuation request is generated based on the target context and preset additional explicit continuation instructions.

[0080] In summary, firstly, the LLM call gateway in this embodiment can monitor the performance degradation of the target model in real time based on multiple collected indicator data of the target model, and switch the current inference request to the backup model of the target model when performance degradation occurs, reducing the probability of the client receiving an incomplete response; secondly, the backup model receives the continuation request based on the original request parameters of the current inference request and the first token sequence, and continues to generate the response of the inference request based on the first token sequence, so that the backup model will not generate content duplication with the target model, avoiding the client receiving duplicate content; thirdly, the end of the token sequence generated by the target model that needs to be sent to the client is truncated after semantic boundary detection, so that the client receives a complete and smooth output, improving the user experience.

[0081] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned routing switching device and its units based on LLM call gateway can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0082] The aforementioned routing switching device based on LLM calling the gateway can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0083] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 600 is an LLM call gateway.

[0084] See Figure 6The computer device 600 includes a processor 602, a memory, and a network interface 605 connected via a system bus 601. The memory may include a non-volatile storage medium 603 and internal memory 604.

[0085] The non-volatile storage medium 603 may store an operating system 6031 and a computer program 6032. The computer program 6032 includes program instructions that, when executed, cause the processor 602 to perform a routing switching method based on an LLM-based gateway call.

[0086] The processor 602 provides computing and control capabilities to support the operation of the entire computer device 600.

[0087] The internal memory 604 provides an environment for the execution of the computer program 6032 in the non-volatile storage medium 603. When the computer program 6032 is executed by the processor 602, the processor 602 can execute a routing switching method based on LLM calling the gateway.

[0088] This network interface 605 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 600 to which the present application is applied. The specific computer device 600 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0089] The processor 602 is used to run a computer program 6032 stored in the memory to perform the following steps: During the streaming processing of inference requests by the target model, indicator data of multiple different preset indicators of the target model are collected in real time. When a model switching instruction is generated based on multiple indicator data and a preset model switching decision, the forwarding of the token generated by the target model to the client is stopped, and an atomic snapshot is performed on the current token stream state of the target model to record the original request parameters of the inference request, the first token sequence that has been pushed to the client, and the second token sequence that has been generated but not pushed to the client. Perform semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence; Forward the tokens before the split position in the second token sequence to the client and update the first token sequence, and delete the tokens after the split position in the second token sequence; A target context is generated based on the original request parameters and the first token sequence, and a continuation request for the inference request is initiated to the backup model of the target model based on the target context; The token is read from the backup model and pushed to the client.

[0090] It should be understood that, in the embodiments of this application, the processor 602 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0091] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0092] Therefore, this application also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: During the streaming processing of inference requests by the target model, indicator data of multiple different preset indicators of the target model are collected in real time. When a model switching instruction is generated based on multiple indicator data and a preset model switching decision, the forwarding of the token generated by the target model to the client is stopped, and an atomic snapshot is performed on the current token stream state of the target model to record the original request parameters of the inference request, the first token sequence that has been pushed to the client, and the second token sequence that has been generated but not pushed to the client. Perform semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence; Forward the tokens before the split position in the second token sequence to the client and update the first token sequence, and delete the tokens after the split position in the second token sequence; A target context is generated based on the original request parameters and the first token sequence, and a continuation request for the inference request is initiated to the backup model of the target model based on the target context; The token is read from the backup model and pushed to the client.

[0093] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0094] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0096] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A route switching method based on LLM calling a gateway, characterized in that, The method is applied to an LLM call gateway, and the method includes: During the streaming processing of inference requests by the target model, indicator data of multiple different preset indicators of the target model are collected in real time. When a model switching instruction is generated based on multiple indicator data and a preset model switching decision, the forwarding of the token generated by the target model to the client is stopped, and an atomic snapshot is performed on the current token stream state of the target model to record the original request parameters of the inference request, the first token sequence that has been pushed to the client, and the second token sequence that has been generated but not pushed to the client. Perform semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence; Forward the tokens before the split position in the second token sequence to the client and update the first token sequence, and delete the tokens after the split position in the second token sequence; A target context is generated based on the original request parameters and the first token sequence, and a continuation request for the inference request is initiated to the backup model of the target model based on the target context; Read the token from the backup model and push it to the client; The LLM call gateway stores multiple preset metrics corresponding to different metric thresholds; the model switching instruction is generated based on the following steps: Determine whether there are any abnormal indicator data among the plurality of indicator data that are higher than the corresponding indicator threshold; If the abnormal indicator data exists, it is determined whether the target model meets multiple preset model switching sub-conditions. The multiple sub-conditions include the target model having a current generated number of tokens greater than a preset first token amount threshold, the target model having a remaining number of tokens for the inference request greater than a preset second token amount threshold, the target model having a historical switching number less than a preset switching number threshold, and the target model having a backup model in an available state. If the target model satisfies multiple preset model switching sub-conditions, then the model switching instruction is generated.

2. The method according to claim 1, characterized in that, After determining whether the target model meets multiple preset model switching sub-conditions, the method further includes: If the number of tokens currently generated is less than or equal to the first token quantity threshold, then the tokens generated by the target model are discarded, and a new model is selected for the inference request; If the remaining token amount for the task is less than or equal to the second token amount threshold, then the inference request will continue to be executed using the current target model. If the number of historical switching events is greater than or equal to the switching event threshold, or if all backup models of the target model are unavailable, a degradation strategy is triggered.

3. The method according to claim 1, characterized in that, The step of performing semantic boundary detection on the second token sequence to determine the segmentation position at the end of the second token sequence includes: Starting from the end of the second token sequence and tracing backwards, target symbol detection is performed, where the target symbols include sentence end symbols, paragraph separators, and clause end symbols; The segmentation position is determined based on the first detected target symbol.

4. The method according to claim 1, characterized in that, The LLM call gateway is connected to the client through the client streaming channel and to the target model through the model streaming channel. The LLM call gateway has a monitoring probe deployed in the inbound direction of the model streaming channel. The real-time acquisition of indicator data for multiple different preset indicators of the target model includes: The monitoring probe collects data on multiple preset indicators in real time.

5. The method according to claim 1, characterized in that, The LLM call gateway includes a buffer proxy layer; stopping the forwarding of the token generated by the target model to the client includes: Stop forwarding the Token in the buffer proxy layer to the client.

6. The method according to claim 1, characterized in that, The step of generating a target context based on the original request parameters and the first token sequence, and initiating a continuation request for the inference request to the backup model of the target model based on the target context, includes: The original request parameters and the first Token sequence are concatenated to form the target context; The continuation request is generated based on the target context and preset additional explicit continuation instructions.

7. A routing switching device based on LLM calling a gateway, characterized in that, The device is configured in an LLM call gateway and includes: The transceiver unit is used to collect indicator data of multiple different preset indicators of the target model in real time during the streaming processing of inference requests by the target model; when a model switching instruction is generated based on the multiple indicator data and the preset model switching decision, the forwarding of the token generated by the target model to the client is stopped. The processing unit is configured to perform an atomic snapshot of the current Token stream state of the target model to record the original request parameters of the inference request, the first Token sequence pushed to the client, and the second Token sequence generated but not pushed to the client; perform semantic boundary detection on the second Token sequence to determine the segmentation position at the end of the second Token sequence; forward the Tokens in the second Token sequence before the segmentation position to the client and update the first Token sequence, and delete the Tokens in the second Token sequence after the segmentation position; generate a target context based on the original request parameters and the first Token sequence, and initiate a continuation request for the inference request to the backup model of the target model based on the target context; The transceiver unit is also used to read the Token from the backup model and push it to the client; The LLM call gateway stores multiple preset index thresholds; the processing unit generates a model switching instruction based on the following steps: Determine whether there are any abnormal indicator data among the plurality of indicator data that are higher than the corresponding indicator threshold; If the abnormal indicator data exists, it is determined whether the target model meets multiple preset model switching sub-conditions. The multiple sub-conditions include the target model having a current generated number of tokens greater than a preset first token amount threshold, the target model having a remaining number of tokens for the inference request greater than a preset second token amount threshold, the target model having a historical switching number less than a preset switching number threshold, and the target model having a backup model in an available state. If the target model satisfies multiple preset model switching sub-conditions, then the model switching instruction is generated.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the routing switching method based on LLM call gateway as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the routing switching method based on LLM calling the gateway as described in any one of claims 1-6.

Images