A method, apparatus, device, and readable storage medium for intelligent routing scheduling.
By extracting task complexity features and service cost indicators for routing decisions, monitoring streaming response status and generating compensation data, the problems of unreasonable resource allocation and session interruption in existing technologies are solved, achieving efficient routing scheduling and business continuity.
Patent Information
- Application Number
- CN202611324940.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, API gateway routing decisions rely on network layer load and do not consider the matching between the task complexity of the request and the service cost and processing capacity of the channel, resulting in unreasonable resource allocation and poor cost controllability; health monitoring is lagging and performance fluctuations cannot be detected in real time; and the lack of session history context processing during channel switching leads to session interruption.
By extracting the task complexity features of large model call requests, combining service cost and processing capacity indicators to make routing decisions, monitoring streaming response status and triggering switching, and obtaining historical context data for format adaptation to generate compensation data, accurate matching and seamless continuation can be achieved.
It achieves precise matching of requests and channel resources, reduces service costs, improves system stability, ensures business continuity, and enhances user experience.
Smart Images

Figure CN123123451A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of routing optimization technology, specifically to an intelligent routing scheduling method, apparatus, device, and readable storage medium. Background Technology
[0002] With the rapid development of artificial intelligence technology, large language models have been widely applied in various intelligent service scenarios. As the core hub connecting clients and backend model services, the API gateway's scheduling performance directly affects service stability, cost-effectiveness, and user experience. In practical applications, the task complexity of different user requests varies greatly, and backend model channels exhibit heterogeneity in billing costs and processing capabilities. Therefore, how to achieve efficient routing and scheduling of large model call requests, and how to ensure session continuity in the event of channel failures, have become critical requirements that urgently need to be addressed in this field.
[0003] In existing technologies, API gateways typically use round-robin or network layer load-weighted algorithms to distribute requests to backend channels, and periodically send heartbeat packets or check HTTP status codes to determine the health status of the channels. When a channel failure is detected, the gateway usually forwards the request to another available channel for retry, or directly returns an error message to the client so that the client can re-initiate the request, thereby achieving basic load balancing and fault tolerance.
[0004] However, the aforementioned existing technologies have the following technical problems: First, routing decisions rely solely on network layer load, without considering the matching relationship between the task complexity of the request itself and the service cost and processing capacity of the channel, resulting in unreasonable resource allocation and poor cost controllability; second, health monitoring is based on heartbeats or status codes, which cannot detect performance fluctuations in the streaming response process in real time, leading to delayed fault detection; finally, the lack of a mechanism for processing session history context during channel switching results in the inability of the backup channel to rebuild the session state, causing dialogue interruption and seriously affecting user experience. Summary of the Invention
[0005] This application provides an intelligent routing scheduling method, apparatus, device, and readable storage medium, which can solve the technical problems of low routing matching degree, delayed fault detection, and session interruption caused by channel switching in the prior art.
[0006] In a first aspect, embodiments of this application provide an intelligent routing scheduling method, including: Receive large model call requests sent by the client and extract the task complexity features corresponding to the large model call requests; Obtain service cost and processing capacity metrics from multiple backend channels, and determine the target channel from multiple backend channels based on task complexity characteristics, service cost metrics, and processing capacity metrics. The large model call request is forwarded to the target channel, and the streaming response status of the target channel is monitored. When the streaming response status meets the preset switching conditions, the channel switching operation is triggered, and a backup channel is determined from multiple backend channels. Obtain the historical context data of the current session, perform format adaptation processing on the historical context data to generate compensation data, and send the compensation data to the backup channel so that the backup channel can reconstruct the session state based on the compensation data and continue the client's current conversation.
[0007] In conjunction with the first aspect, in one implementation, the task complexity features corresponding to large model call requests are extracted, including: Extract the text length and long text association density of large model call requests; The overall computational load is predicted based on the text length and the association density of long texts, and the task complexity features are determined based on the overall computational load.
[0008] In conjunction with the first aspect, in one implementation, a target channel is determined from multiple backend channels based on task complexity characteristics, service cost metrics, and processing capacity metrics, including: Construct a routing evaluation model, which should include at least cost evaluation factors, performance evaluation factors, and capacity matching evaluation factors; The weight coefficients of cost evaluation factors, performance evaluation factors, and capability matching evaluation factors are dynamically adjusted based on the characteristics of task complexity. The optimization algorithm is used to solve the routing evaluation model, and the back-end channel that produces the best routing evaluation score output by the routing evaluation model is determined as the target channel. Among them, the cost evaluation factor is determined based on the service cost index, the performance evaluation factor is determined based on the processing capacity index, and the capacity matching evaluation factor is determined based on the matching degree between the task complexity characteristics and the processing capacity index.
[0009] In conjunction with the first aspect, in one implementation, the weight coefficients of the cost evaluation factor, performance evaluation factor, and capability matching evaluation factor are dynamically adjusted based on the task complexity characteristics, including: Extract the text length of the large model call request; When the text length exceeds the preset long text threshold, the weight coefficient of the performance evaluation factor is increased.
[0010] In conjunction with the first aspect, in one implementation, monitoring the streaming response status of a target channel includes: Obtain the output delay between adjacent streaming data blocks, and construct a delay difference sequence based on the output delay; Determine the differential response delay volatility based on the delay difference sequence; When the number of times the differential response delay volatility continuously meets the preset baseline threshold reaches the state switching threshold, the streaming response state is determined to switch from the default healthy state to the decaying state. In the decay state, if the differential response delay volatility reaches the preset circuit breaker threshold, or triggers a system error, the streaming response state is determined to switch from the decay state to the circuit breaker state.
[0011] In conjunction with the first aspect, in one implementation, when the streaming response state meets preset switching conditions, a channel switching operation is triggered, including: When the streaming response is in a decay state, the flow quota of the target channel is reduced based on a preset ratio. When the streaming response is in circuit breaker mode, the flow quota of the target channel is adjusted to zero.
[0012] In conjunction with the first aspect, in one implementation, historical context data is format-adapted to generate compensation data, including: Based on a preset sliding window, the dialogue information from the most recent preset number of rounds is extracted from historical context data, and redundant data is removed. Identify the target data format corresponding to the backup channel, convert the filtered dialogue information into a target data format compatible with the backup channel, and obtain compensation data.
[0013] Secondly, embodiments of this application provide an intelligent routing scheduling device, comprising: The request receiving module is used to receive large model call requests sent by the client and extract the task complexity features corresponding to the large model call requests. The routing decision module is used to obtain service cost indicators and processing capacity indicators of multiple backend channels, and determine the target channel from multiple backend channels based on task complexity characteristics, service cost indicators, and processing capacity indicators. The status monitoring module is used to forward large model call requests to the target channel and monitor the streaming response status of the target channel. When the streaming response status meets the preset switching conditions, the channel switching operation is triggered, and a backup channel is determined from multiple backend channels. The context compensation module is used to obtain the historical context data of the current session, perform format adaptation processing on the historical context data to generate compensation data, and send the compensation data to the backup channel so that the backup channel can reconstruct the session state based on the compensation data and continue the client's current dialogue.
[0014] Thirdly, embodiments of this application provide an intelligent routing scheduling device, which includes a processor, a memory, and an intelligent routing scheduling program stored in the memory and executable by the processor, wherein when the intelligent routing scheduling program is executed by the processor, it implements the steps of the intelligent routing scheduling method as described above.
[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing an intelligent routing scheduler, wherein when the intelligent routing scheduler is executed by a processor, it implements the steps of the intelligent routing scheduling method described above.
[0016] The beneficial effects of the technical solutions provided in this application include: This application achieves precise matching of requests and channel resources by extracting task complexity features and combining service costs and processing capabilities for routing decisions, thereby reducing service costs. By monitoring the streaming response status to trigger switching, channel anomalies can be detected in a timely manner, improving system stability. Furthermore, by acquiring historical context data and performing format adaptation to generate compensation data, backup channels can rebuild session states and continue dialogues, ensuring business continuity. This effectively solves the technical problems of low routing matching accuracy, delayed fault detection, and session interruption caused by channel switching in existing technologies, and effectively improves user experience. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating an embodiment of the intelligent routing scheduling method of this application; Figure 2 This is a schematic diagram of the process for extracting task complexity features according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of determining the target channel according to an embodiment of this application; Figure 4 For this application Figure 3 A detailed flowchart of step S302; Figure 5 This is a flowchart illustrating the monitoring of streaming response status according to an embodiment of this application; Figure 6 This is a flowchart illustrating the channel switching operation in an embodiment of this application. Figure 7 This is a schematic diagram of the process for generating compensation data according to an embodiment of this application; Figure 8 This is a functional module diagram of an embodiment of the intelligent routing scheduling device of this application; Figure 9 This is a schematic diagram of the hardware structure of the intelligent routing and scheduling device involved in the embodiments of this application. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0020] In a first aspect, embodiments of this application provide an intelligent routing scheduling method.
[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the intelligent routing scheduling method of this application. Figure 1 As shown, the intelligent routing scheduling method includes: S101. Receive the large model call request sent by the client and extract the task complexity features corresponding to the large model call request; S102. Obtain the service cost indicators and processing capacity indicators of multiple backend channels, and determine the target channel from multiple backend channels based on task complexity characteristics, service cost indicators and processing capacity indicators. S103. Forward the large model call request to the target channel and monitor the streaming response status of the target channel. When the streaming response status meets the preset switching conditions, trigger the channel switching operation and determine the backup channel from multiple backend channels. S104. Obtain the historical context data of the current session, perform format adaptation processing on the historical context data to generate compensation data, and send the compensation data to the backup channel so that the backup channel can rebuild the session state based on the compensation data and continue the client's current dialogue.
[0022] Specifically, in the embodiment of the intelligent routing scheduling method, for a large model call request sent by the client, the embodiment allocates a target channel for each large model call request by using the task complexity characteristics of the large model call request and the service cost indicators and processing capacity indicators of each backend channel.
[0023] For each large model call request, the embodiment extracts its text length and long text association density, and then estimates the overall computational load of the task based on the text length and long text association density to determine the task complexity characteristics of the large model call request. For each backend channel, the embodiment determines the service cost index of the backend channel based on the unit token billing of the backend channel, and determines the processing capacity index of the backend channel based on the rolling average response latency maintained by the current gateway.
[0024] Then, the implementation example selects the target channel for processing the large model call request from multiple backend channels based on the task complexity characteristics of the large model call request, the service cost indicators of the backend channels, and the processing capacity indicators. In this example, a routing evaluation model is established, consisting of a cost evaluation factor, a performance evaluation factor, and a capability matching evaluation factor. The cost evaluation factor is determined based on the service cost indicator, the performance evaluation factor is determined based on the processing capacity indicator, and the capability matching evaluation factor is determined based on the matching degree between the task complexity characteristics and the processing capacity indicator.
[0025] Based on the established routing evaluation model, the implementation example can use an optimization algorithm to obtain the allocation scheme that maximizes the overall score, and then use this scheme to allocate the corresponding target channel for the large model call request to execute the dialogue task.
[0026] Furthermore, the weighting coefficients for the three dimensions—cost evaluation factor, performance evaluation factor, and capability matching evaluation factor—can be dynamically adjusted in this embodiment. Based on the text length of the current large model call request, if the text length exceeds a preset long text threshold (i.e., the large model call request is a long text prompt), the system will automatically increase the weight of the performance evaluation factor to ensure response speed. This directs traffic to a backend channel with high bandwidth and low initial latency, thereby improving user experience.
[0027] During data transmission, the embodiment uses differential response latency to measure channel link health and performs gradual traffic reduction when channel performance degrades. In the embodiment, the output delay between adjacent data blocks is recorded, a delay difference sequence is established based on this output delay, and then the differential response latency volatility is calculated. This differential response latency volatility is used to assess the current state of the channel link.
[0028] In this embodiment, the streaming response status of the target channel is defaulted to a healthy state. When the differential response delay volatility calculated multiple times (e.g., three times) exceeds a preset circuit breaker threshold, the embodiment determines that the streaming response status switches from a healthy state to a decaying state. At this time, the traffic quota of the target channel is reduced by a certain proportion (e.g., 50%). In the decaying state, if the differential response delay volatility continues to increase and reaches the preset circuit breaker threshold, or triggers a system error, the embodiment determines that the streaming response status switches from a decaying state to a circuit breaker state. At this time, the traffic quota of the target channel is adjusted to zero, and the traffic is completely isolated.
[0029] Furthermore, to ensure seamless client continuity during channel switching and eliminate service interruptions and repetitive first-character delays caused by context gaps in traditional solutions, this embodiment also incorporates a context-based prompt compensation mechanism. The embodiment encapsulates the historical dialogue context into an intermediate, generic JSON object, forming historical context data. Then, using a sliding window and anchored by the current timestamp, the embodiment extracts dialogue information from the most recent rounds of the historical context data. After trimming non-semantic auxiliary modifiers and redundant paragraphs in long texts, it converts the information into a target data format compatible with the backup channel, obtaining compensation data. When switching channels, the embodiment forwards the compensation data along with subsequent large model call requests to the backup channel, effectively avoiding session interruptions caused by channel switching and achieving seamless client switching.
[0030] In this embodiment, by extracting task complexity features and combining service cost and processing capacity for routing decisions, precise matching of requests and channel resources is achieved, reducing service costs. By monitoring the streaming response status to trigger switching, channel anomalies can be detected in a timely manner, improving system stability. Furthermore, by acquiring historical context data and performing format adaptation to generate compensation data, the backup channel can rebuild the session state and continue the conversation, ensuring business continuity. This effectively solves the technical problems of low routing matching degree, delayed fault detection, and session interruption caused by channel switching in the prior art, and effectively improves user experience.
[0031] Furthermore, in one embodiment, Figure 2 This is a schematic diagram of the process for extracting task complexity features according to an embodiment of this application, such as... Figure 2 As shown, the task complexity features corresponding to large model call requests are extracted, including: S201. Extract the text length and long text association density of large model call requests; S202. Predict the overall computational load based on the text length and the association density of long texts, and determine the task complexity characteristics based on the overall computational load.
[0032] Specifically, in this embodiment, after receiving a large model request initiated by a client, the gateway is first intercepted by the intelligent routing and traffic splitting module. The system uses a preset lexical analyzer or a lightweight embedding model to extract the text length and long text association density of the prompt words in the large model call request, and combines the max_tokens in the call parameters to estimate the overall computational load generated this time, and determines the task complexity characteristics of the large model call request based on the overall computational load.
[0033] In other embodiments, the lightweight embedding model can also be replaced by a rule-based keyword matcher or a tokenizer-based length counter, which can also achieve coarse-grained traffic splitting in terms of cost when high accuracy is not required.
[0034] Furthermore, in one embodiment, Figure 3 This is a schematic diagram of the process for determining the target channel according to an embodiment of this application, as shown below. Figure 3 As shown, based on task complexity characteristics, service cost metrics, and processing capacity metrics, the target channel is determined from multiple backend channels, including: S301. Construct a routing evaluation model, which shall include at least a cost evaluation factor, a performance evaluation factor, and a capacity matching evaluation factor. S302. Dynamically adjust the weight coefficients of cost evaluation factors, performance evaluation factors, and capability matching evaluation factors based on the characteristics of task complexity. S303. Solve the optimization algorithm for the routing evaluation model, and determine the back-end channel that gives the optimal routing evaluation score from the routing evaluation model as the target channel. Among them, the cost evaluation factor is determined based on the service cost index, the performance evaluation factor is determined based on the processing capacity index, and the capacity matching evaluation factor is determined based on the matching degree between the task complexity characteristics and the processing capacity index.
[0035] Furthermore, in one embodiment, Figure 4 For this application Figure 3 A detailed flowchart of step S302 is shown below. Figure 4 As shown, the weight coefficients of the cost evaluation factor, performance evaluation factor, and capability matching evaluation factor are dynamically adjusted based on the task complexity characteristics, including: S401. Extract the text length of the large model call request; S402. When the text length exceeds the preset long text threshold, increase the weight coefficient of the performance evaluation factor.
[0036] Specifically, in this embodiment, when a client initiates a request, the system optimizes the allocation of backend channels based on three dimensions: cost, performance, and the degree of matching between complexity and channel capacity. The current large model call request is represented as... The current set of alternative backend channels is For backend channels The unit token billing cost As a service cost metric, the rolling average response latency currently maintained by the gateway will be used. As an indicator of the processing capacity of the backend channel, the objective function of the routing evaluation model is defined as the sum of three factors: cost evaluation factor, performance evaluation factor, and capacity matching evaluation factor.
[0037] in, , and These are configurable dynamic weighting coefficients. This refers to the degree of matching between the channel's processing capacity and request complexity. To maximize billing cost, This represents the maximum rolling average response time. The underlying gateway solves for this by performing matrix iterations every 100 milliseconds. The smallest channel is the optimal target channel.
[0038] In this embodiment, the matching degree between the channel's processing capacity index and the request complexity can be quantitatively evaluated using the normalized difference method. For example, the processing capacity index of each channel and the request complexity of the large model call request can be normalized to the same range (such as the [0,1] range) based on the maximum and minimum values of the processing capacity index of the backend channel and the maximum and minimum values of the request complexity of the historical large model call request. Then, the matching degree can be calculated based on the normalized similarity between the two.
[0039] In addition, other quantitative evaluation methods can be used in other embodiments, such as ratio methods or weighted scoring methods, or hierarchical and classification evaluation methods. In some preferred embodiments, multiple indicators such as throughput, number of connections, bandwidth, and resource utilization can be introduced to evaluate the channel's processing capacity and construct a comprehensive evaluation system.
[0040] Furthermore, the embodiment also sets dynamically adjustable weight coefficients for each factor in the routing evaluation model. For large model call requests, if the text length exceeds a preset text threshold (i.e., the large model call request is a long text prompt), the system automatically increases the weight of the performance evaluation factor when matching channels to improve the response speed for such requests. This directs traffic to a backend channel with high bandwidth and low latency, thereby improving the user experience.
[0041] In this embodiment, through multi-dimensional channel matching, complex inference requests can be accurately directed to high-computing-power channels and simple question-and-answer requests can be directed to low-cost channels. Compared with traditional polling schemes, this can reduce token call costs and prevent high-cost channels from being filled by low-value traffic.
[0042] Furthermore, in one embodiment, Figure 5 This is a flowchart illustrating the monitoring of streaming response status according to an embodiment of this application, as shown below. Figure 5 As shown, the streaming response status of the target channel is monitored, including: S501. Obtain the output delay between adjacent streaming data blocks and construct a delay difference sequence based on the output delay; S502. Determine the differential response delay volatility based on the delay difference sequence; S503. When the number of times the differential response delay volatility continuously meets the preset baseline threshold reaches the state switching threshold, the streaming response state is determined to switch from the default healthy state to the decaying state. S504. In the decay state, if the differential response delay fluctuation rate reaches the preset circuit breaker threshold, or triggers a system error, the streaming response state is determined to switch from the decay state to the circuit breaker state.
[0043] Furthermore, in one embodiment, Figure 6 This is a flowchart illustrating the channel switching operation in an embodiment of this application, as shown below. Figure 6 As shown, when the streaming response status meets the preset switching conditions, a channel switching operation is triggered, including: S601. When the streaming response is in a decay state, reduce the flow quota of the target channel based on a preset ratio. S602. When the flow response status is in the circuit breaker state, adjust the flow quota of the target channel to zero.
[0044] Specifically, to improve the real-time performance of fault detection, the link health measurement does not rely on active heartbeat detection, but instead employs a non-intrusive bypass monitoring mechanism. In this embodiment, a time-sliding window W (e.g., the most recent 10 seconds) is defined, and the gateway monitors the channel... The streaming response is sampled and the first... In the streaming response of a request, the absolute timestamp of the token output. Define the output latency between tokens as... And construct a time delay difference sequence.
[0045] At this time, the passage The differential response delay volatility is defined as:
[0046] in, This represents the differential response delay volatility. Represents variance. These are the weighting coefficients.
[0047] In this embodiment, a built-in health decay state machine is used when... If the number of consecutive exceedances of the system-defined baseline threshold T_baseline reaches the state transition threshold (e.g., exceeding the baseline threshold three times consecutively), the state machine switches from a healthy state to a decaying state. At this time, the gateway does not disconnect the channel but instead reduces its traffic quota by a preset ratio (e.g., 50%) through the Inversion of Control (IoC) mechanism. In the decaying state, if the differential response latency fluctuation continues to increase, reaching the preset circuit breaker threshold, or triggering a system error, the state machine switches from the decaying state to the circuit breaker state, the traffic quota of the target channel is adjusted to zero, and the traffic is completely isolated.
[0048] In this embodiment, by non-intrusive bypass monitoring to calculate differential response latency volatility, the system can detect upstream computing power lag and network degradation in advance when the HTTP status code is still 200 (soft fault); through the tiered circuit breaker mechanism, the system performs gradual peak shaving instead of abrupt disconnection when channel performance degrades, which protects the user experience and avoids the avalanche effect caused by traffic surges.
[0049] Furthermore, in one embodiment, Figure 7 This is a schematic diagram of the process for generating compensation data according to an embodiment of this application, such as... Figure 7 As shown, historical context data is processed for format adaptation to generate compensation data, including: S701. Extract dialogue information from the most recent preset number of rounds from historical context data based on a preset sliding window, and remove redundant data. S702. Identify the target data format corresponding to the backup channel, convert the filtered dialogue information into a target data format compatible with the backup channel, and obtain compensation data.
[0050] Specifically, to ensure seamless connection for clients and eliminate service interruptions and repeated first-character delays caused by context gaps in traditional solutions, the embodiment also incorporates a context-based prompt word compensation mechanism during channel switching.
[0051] In this implementation, all historical dialogue contexts are uniformly encapsulated into an intermediate, generic JSON object structure (containing the system role "system", user input "user", and model response "assistant", along with an absolute timestamp). When the target channel fails and a switch to the backup channel is required, the compensation algorithm is activated. The implementation uses a sliding window, with the current timestamp as the anchor, to retain N dialogue rounds forward. The number of dialogue rounds N retained in the implementation can be dynamically adjusted based on the backup channel's context window size and billing requirements.
[0052] The implementation then uses the gateway's built-in tokenizer to physically trim auxiliary modifiers and redundant paragraphs in long texts that do not affect semantics. The trimmed text is then dynamically modified and concatenated according to the specific schema of the backup channel's API (e.g., converting from OpenAI's Messages format to Anthropic's Messages format). Finally, it is smoothly pushed to the new channel via asynchronous I / O, eliminating format incompatibility or context overflow anomalies caused by channel switching.
[0053] In this embodiment, by using sliding window clipping and format conversion, the core entities and the most recent N rounds of dialogue are preserved when switching between vendor channels. The client only needs to withstand millisecond-level jitter to seamlessly resume the connection, eliminating the business interruption and repeated first character delay caused by context breakage in traditional solutions, thus improving the user experience.
[0054] Secondly, embodiments of this application also provide an intelligent routing scheduling device.
[0055] In one embodiment, reference is made to Figure 8 , Figure 8 This is a functional module diagram of an embodiment of the intelligent routing scheduling device of this application. Figure 8 As shown, the intelligent routing scheduling device includes: The request receiving module 801 is used to receive large model call requests sent by the client and extract the task complexity features corresponding to the large model call requests. The routing decision module 802 is used to obtain the service cost indicators and processing capacity indicators of multiple backend channels, and determine the target channel from multiple backend channels based on task complexity characteristics, service cost indicators and processing capacity indicators; The status monitoring module 803 is used to forward large model call requests to the target channel and monitor the streaming response status of the target channel. When the streaming response status meets the preset switching conditions, it triggers the channel switching operation and determines the backup channel from multiple backend channels. The context compensation module 804 is used to obtain the historical context data of the current session, perform format adaptation processing on the historical context data to generate compensation data, and send the compensation data to the backup channel so that the backup channel can reconstruct the session state based on the compensation data and continue the client's current dialogue.
[0056] The functions of each module in the above-mentioned intelligent routing scheduling device correspond to the steps in the above-mentioned intelligent routing scheduling method embodiment, and their functions and implementation processes will not be described in detail here.
[0057] Thirdly, embodiments of this application provide an intelligent routing scheduling device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.
[0058] Reference Figure 9 , Figure 9 This is a schematic diagram of the hardware structure of the intelligent routing scheduling device involved in the embodiments of this application. In the embodiments of this application, the intelligent routing scheduling device may include a processor, a memory, a communication interface, and a communication bus.
[0059] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.
[0060] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the intelligent routing and scheduling device, as well as interfaces used for interconnecting the intelligent routing and scheduling device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.
[0061] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0062] The processor can be a general-purpose processor, which can call the intelligent routing scheduler stored in memory and execute the intelligent routing scheduling method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the intelligent routing scheduler is called can be referred to in the various embodiments of the intelligent routing scheduling method of this application, and will not be repeated here.
[0063] Those skilled in the art will understand that Figure 9 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0064] Fourthly, embodiments of this application also provide a computer-readable storage medium.
[0065] The present application provides a computer-readable storage medium storing an intelligent routing scheduler, wherein when the intelligent routing scheduler is executed by a processor, it implements the steps of the intelligent routing scheduling method described above.
[0066] The method implemented when the intelligent routing scheduler is executed can be referred to in various embodiments of the intelligent routing scheduling method of this application, and will not be repeated here.
[0067] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0068] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.
[0069] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.
[0070] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0071] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods of the various embodiments of this application.
[0073] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An intelligent routing scheduling method, characterized in that, include: Receive large model call requests sent by the client and extract the task complexity features corresponding to the large model call requests; Obtain service cost metrics and processing capacity metrics from multiple backend channels, and determine the target channel from the multiple backend channels based on the task complexity characteristics, the service cost metrics, and the processing capacity metrics; The large model call request is forwarded to the target channel, and the streaming response status of the target channel is monitored. When the streaming response status meets the preset switching conditions, the channel switching operation is triggered, and a backup channel is determined from multiple backend channels. Obtain the historical context data of the current session, perform format adaptation processing on the historical context data to generate compensation data, and send the compensation data to the backup channel so that the backup channel can reconstruct the session state based on the compensation data and continue the current dialogue of the client.
2. The intelligent routing scheduling method according to claim 1, characterized in that, The extraction of task complexity features corresponding to the large model call request includes: Extract the text length and long text association density of the large model call request; The overall computational load is predicted based on the text length and the long text association density, and the task complexity features are determined based on the overall computational load.
3. The intelligent routing scheduling method according to claim 1, characterized in that, The process of determining the target channel from multiple backend channels based on the task complexity characteristics, the service cost metrics, and the processing capacity metrics includes: Construct a routing evaluation model, which includes at least a cost evaluation factor, a performance evaluation factor, and a capacity matching evaluation factor; The weight coefficients of the cost evaluation factor, the performance evaluation factor, and the capability matching evaluation factor are dynamically adjusted based on the task complexity characteristics. The optimization algorithm is applied to the routing evaluation model to determine the backend channel that produces the optimal routing evaluation score from the model as the target channel. The cost evaluation factor is determined based on the service cost index, the performance evaluation factor is determined based on the processing capacity index, and the capacity matching evaluation factor is determined based on the matching degree between the task complexity characteristics and the processing capacity index.
4. The intelligent routing scheduling method according to claim 3, characterized in that, The step of dynamically adjusting the weight coefficients of the cost evaluation factor, performance evaluation factor, and capability matching evaluation factor based on the task complexity characteristics includes: Extract the text length of the large model call request; When the length of the text exceeds a preset long text threshold, the weight coefficient of the performance evaluation factor is increased.
5. The intelligent routing scheduling method according to claim 1, characterized in that, The monitoring of the streaming response status of the target channel includes: Obtain the output delay between adjacent streaming data blocks, and construct a delay difference sequence based on the output delay; The differential response delay volatility is determined based on the delay difference sequence. When the number of times the differential response delay volatility continuously meets the preset baseline threshold reaches the state switching threshold, it is determined that the streaming response state switches from the default healthy state to the decaying state. In the decay state, if the differential response delay volatility reaches the preset circuit breaker threshold, or triggers a system error, the streaming response state is determined to switch from the decay state to the circuit breaker state.
6. The intelligent routing scheduling method according to claim 5, characterized in that, When the streaming response state meets the preset switching conditions, a channel switching operation is triggered, including: When the streaming response state is in a decay state, the flow quota of the target channel is reduced based on a preset ratio; When the streaming response state is in the circuit breaker state, the flow quota of the target channel is adjusted to zero.
7. The intelligent routing scheduling method according to claim 1, characterized in that, The step of performing format adaptation processing on the historical context data to generate compensation data includes: Based on a preset sliding window, the dialogue information from the most recent preset number of rounds is extracted from the historical context data, and redundant data is removed. Identify the target data format corresponding to the backup channel, convert the filtered dialogue information into a target data format compatible with the backup channel, and obtain compensation data.
8. An intelligent routing scheduling device, characterized in that, include: The request receiving module is used to receive large model call requests sent by the client and extract the task complexity features corresponding to the large model call requests. The routing decision module is used to obtain service cost indicators and processing capacity indicators of multiple backend channels, and determine the target channel from multiple backend channels based on the task complexity characteristics, the service cost indicators and the processing capacity indicators. The status monitoring module is used to forward the large model call request to the target channel and monitor the streaming response status of the target channel. When the streaming response status meets the preset switching conditions, the channel switching operation is triggered, and a backup channel is determined from multiple backend channels. The context compensation module is used to obtain the historical context data of the current session, perform format adaptation processing on the historical context data to generate compensation data, and send the compensation data to the backup channel so that the backup channel can reconstruct the session state based on the compensation data and continue the current dialogue of the client.
9. An intelligent routing scheduling device, characterized in that, The intelligent routing scheduling device includes a processor, a memory, and an intelligent routing scheduling program stored in the memory and executable by the processor, wherein when the intelligent routing scheduling program is executed by the processor, it implements the steps of the intelligent routing scheduling method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an intelligent routing scheduler, wherein when the intelligent routing scheduler is executed by a processor, it implements the steps of the intelligent routing scheduling method as described in any one of claims 1 to 7.
