A language model-based dynamic allocation system based on cost budgeting

Abstract

A dynamic language model allocation system based on cost budgeting is disclosed. A task parsing module is connected to a receiving module, a cost and delay estimation module is connected to a historical database and the task parsing module respectively, and a model decision module is connected to the cost and delay estimation module. The historical database records historical task information for different task types, risk levels, and language models. The task parsing module parses the task information to generate task type and risk level. The cost and delay estimation module calculates task cost and delay estimates based on historical task information. Finally, the model decision module performs a comprehensive evaluation based on the total actual execution cost, risk level, task cost estimate, and task delay estimate to dynamically allocate a suitable language model for the task. This achieves the technical benefits of dynamically allocating suitable language models, effectively controlling model call costs, improving model scheduling efficiency and response speed, and enhancing the resource utilization and operational stability of enterprise AI platforms.

Claims

1. A language model dynamic allocation system based on cost budget, the system comprising: a historical database corresponding to a task type, a risk level, a language model, and historical task information, wherein the task information includes an input prompt, an output message, an input token, an output token, a context window size, and a response time; a receiving module for receiving task information, wherein the task information includes the input prompt and the input token; and a task parsing module connected to the receiving module for parsing the task information to generate the task type and the risk level. A cost and delay estimation module, connected to the historical database and the task parsing module, respectively, filters the corresponding language model and historical task information from the historical database based on the task type, risk level, input prompt words, and input lexical units. Based on the language model, it calculates an average input lexical unit, an average output lexical unit, and an average response time for the filtered historical task information. Using a cost and delay lookup table, it compares the state context window capacity, the average input lexical unit, the average output lexical unit, and the average response time to calculate a task cost estimate and a task delay estimate corresponding to the language model. A model decision module, connected to the cost and delay estimation module, calculates the total actual execution cost of tasks executed within a cost statistics time range. Then, based on a comprehensive evaluation of the total actual execution cost, the risk level, the task cost estimate, and the task delay estimate, it allocates the language model for executing the task information.

2. The language model dynamic allocation system based on cost budget as described in claim 1, wherein the task parsing module further determines the risk level based on the semantic classification result of one of the input prompt words, a user permission, and a task execution purpose, and adjusts the risk weight of one of the model decision modules according to the risk level, so that high-risk tasks are preferentially allocated to the language model with higher capabilities.

3. The language model dynamic allocation system based on cost budget as described in claim 1, wherein the cost and delay estimation module further calculates a word capacity occupancy ratio, the word capacity occupancy ratio being the sum of the average input words and the average output words divided by the state context window capacity, and calculates the task delay estimation based on the word capacity occupancy ratio and a capacity load coefficient.

4. The language model dynamic allocation system based on cost budget as described in claim 1, wherein the model decision module further executes an automatic lexical over-limit backup mechanism when the estimated task cost exceeds a preset cost budget or a preset lexical budget, in order to adjust the input prompt words, trim a state context data, limit an upper limit of an output lexical, segment the task information into multiple sub-tasks, or switch to the language model with lower cost.

Images