Method and device for managing consumption of large language models

Abstract

This invention provides a method and device for managing the consumption of large language models. The method includes: receiving a service request from a client; determining demand information based on the service request, the demand information including the number of input tokens, context length, task type information, and target model information; predicting the number of output tokens based on the demand information; and determining an estimated cost consumption based on the number of input tokens, the number of output tokens, the context length, and the target model information, wherein the estimated cost consumption is measured using a unified standard computing unit.

Description

Technical Field

[0001] This invention relates to the field of cloud computing, and in particular to a method and apparatus for managing the consumption of large language models in a multimodal, multi-vendor environment. Background Technology

[0002] With the widespread adoption of Large Language Model (LLM) technology, enterprises often use a mix of multiple model services. However, relying directly on the original billing rules of model service providers for cost management has significant drawbacks, primarily in the following aspects:

[0003] 1. Unknown "Anticipated" Costs: Service providers typically bill "post-event," meaning they only know how many tokens have been consumed after the model has generated all the text. However, businesses need "pre-event control" to prevent users with insufficient funds from initiating long text generation requests or to prevent exorbitant bills caused by program infinite loops. Current technology cannot predict the output length before a request is initiated, thus hindering precise budget circuit breaking.

[0004] 2. Inability to make horizontal comparisons of heterogeneous resources: The billing metrics for different models are completely different. For example, commercial models (such as DoubaoSeed) are billed by tokens, while self-hosted models (such as Qwen3) are billed by GPU inference time (milliseconds) or memory usage. Enterprises cannot directly compare the cost-effectiveness of "calling an API once" versus "running a self-built service for one second," lacking a unified value metric.

[0005] 3. Non-linear cost risk: In many long context scenarios, the inference cost increases non-linearly as the ContextWindow increases (e.g., KV Cache usage leads to exponential consumption of video memory), but ordinary billing linear models cannot smoothly reflect this hardware cost characteristic. Summary of the Invention

[0006] In view of the problems in the prior art, the present invention provides a method for managing the consumption of large language models, the method comprising:

[0007] Receive service requests from clients;

[0008] Based on the service request, the requirement information is determined, which includes the number of input tokens, context length, task type information, and target model information.

[0009] Predict the number of output tokens based on the aforementioned demand information;

[0010] The estimated cost consumption is determined based on the number of input tokens, the number of output tokens, the context length, and the target model information. The estimated cost consumption is measured using a unified standard computing unit.

[0011] Further, check the customer's account balance. If the balance can cover the estimated cost, forward the service request to the corresponding model provider and freeze the corresponding estimated amount. If the balance cannot cover the estimated cost, trigger an alarm.

[0012] Furthermore, after the service request is completed, the deviation between the estimated cost and the actual cost is calculated, and a compensation operation is performed based on the deviation.

[0013] Furthermore, the formula for calculating the actual cost consumption is as follows:

[0014]

[0015] in, This represents the actual cost incurred. This represents the actual number of tokens input. This represents the actual number of output tokens. This represents the total actual time spent on reasoning. , and They are respectively , and The weight, For model identifier, for The basic call overhead of the corresponding model, For context penalty function; and

[0016]

[0017] in, Total length of the context The context penalty coefficient, The context window threshold for free or standard price.

[0018] Furthermore, based on historical service requests and their results, a regression prediction model is trained to predict the number of output tokens based on the demand information.

[0019] Furthermore, the loss function of the regression prediction model is as follows:

[0020]

[0021] in, For the model parameter set, For the first The item outputs a predicted value for the number of tokens. For the first The item outputs the actual value of the number of tokens. This is a regularization term.

[0022] Furthermore, the formula for calculating the estimated cost consumption is as follows:

[0023]

[0024] in, To estimate cost consumption, This represents the actual number of tokens input. For the predicted number of output tokens, and They are respectively and The weight, The base cost for a single call. For context penalty function; and

[0025]

[0026] in, Total length of the context The context penalty coefficient, The context window threshold for free or standard price.

[0027] Furthermore, the unit price of the standard computing power unit corresponds to the minimum pre-deposit amount in the account.

[0028] The present invention also provides a device for managing the consumption of large language models, the device comprising:

[0029] Processor; and

[0030] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the operations described above.

[0031] The present invention also provides a computer-readable medium for storing instructions that, when executed, cause the system to perform the operations described above.

[0032] The method and apparatus for managing large language model consumption of the present invention have the following beneficial effects:

[0033] 1. Achieve "pre-visibility" of costs: Users can know the accurate estimated cost before the API call occurs, solving the problem of unpredictable costs caused by the "black box" of tokens.

[0034] 2. Unified measurement standard: Convert heterogeneous units such as Token and Time into SCU, enabling enterprises to compare the cost-effectiveness of different models and eliminate supplier lock-in.

[0035] 3. Provides a non-linear cost smoothing mechanism: Through a context penalty algorithm, the unit price of SCU is dynamically adjusted according to the context length to truly reflect the hardware cost and achieve fair billing. Attached Figure Description

[0036] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0037] Figure 1 A flowchart illustrating a method for managing the consumption of a large language model according to an embodiment of the present invention is shown.

[0038] Figure 2 The illustration shows a functional module of an exemplary system that can be used in various embodiments of the present invention.

[0039] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0040] The present application will now be described in further detail with reference to the accompanying drawings.

[0041] In a typical configuration of the present invention, the terminal, the device of the service network, and the trusted party all include one or more processors (e.g., a central processing unit (CPU)), input / output interfaces, network interfaces, and memory.

[0042] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.

[0043] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information that can be accessed by a computing device.

[0044] The devices referred to in this invention include, but are not limited to, user equipment, network devices, or devices formed by integrating user equipment and network devices through a network. The user equipment includes, but is not limited to, any mobile electronic product capable of human-computer interaction (e.g., via a touchpad), such as smartphones and tablets. These mobile electronic products can use any operating system, such as Android or iOS. The network device includes an electronic device capable of automatically performing numerical calculations and information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices. The network device includes, but is not limited to, computers, network hosts, single network servers, multiple network server clusters, or a cloud composed of multiple servers. Here, the cloud consists of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual supercomputer composed of a group of loosely coupled computer clusters. The network includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, and wireless ad hoc network. Preferably, the device can also be a program running on the user equipment, network device, or a device formed by integrating user equipment and network device, network device, touch terminal, or network device and touch terminal through a network.

[0045] Of course, those skilled in the art should understand that the above-described devices are merely examples, and other existing or future devices that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0046] In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0047] Figure 1 The diagram illustrates a flowchart of a method for managing the consumption of large language models according to an embodiment of the present invention. This method is executed by a corresponding system deployed in a Kubernetes cluster, and specifically includes the following steps:

[0048] Step S1: Receive the service request from the client.

[0049] In this step, the gateway receives HTTP POST requests sent by the user through the client.

[0050] Step S2: Determine the requirement information based on the service request.

[0051] In this step, the system first parses the API-Key in the HTTP POST request header to determine the tenant's identity and extracts parameters such as prompt, max_tokens, and model from the Body. The system then calculates the number of characters in the input prompt. It then calls a tokenizer that matches the target model to calculate the number of tokens in the input text, as well as determine the context length and task type information. The task type information is initially determined based on regular expressions or keyword matching (containing "Translation", "Summary", "CodeGeneration", "Chat") to determine the task type (Translation, Summary, Code Generation, Dialogue).

[0052] Step S3: Predict the number of tokens to be output based on the demand information.

[0053] In this step, a pre-trained regression prediction model is used to predict the number of output tokens based on the demand information determined in step S2.

[0054] The regression prediction model described above employs a lightweight model, preferably a gradient boosting tree regression model due to its extremely fast inference speed (<5ms). The model's training data is extracted from historical call record tuples in the system logs.

[0055]

[0056] in,

[0057] (Feature vector): contains That is, input the number of tokens, task type, model identifier (qwen3-max, qwen3-plus, llama-3-8b-local, etc.);

[0058] (Label): The actual number of output tokens generated.

[0059] The training objective of this regression prediction model is to minimize the mean squared logarithmic error to reduce the impact of long-tailed data (maximum token outputs) on the model. The loss function is as follows:

[0060]

[0061] in, For the model parameter set, For the first The item outputs a predicted value for the number of tokens. For the first The item outputs the actual value of the number of tokens. This is a regularization term.

[0062] Preferably, the system uses newly generated log data for incremental learning at predetermined intervals (e.g., every 24 hours) to adapt to changes in user behavior.

[0063] The regression prediction model obtained through the above training can predict the number of output tokens based on the number of input tokens, task type information, and target model information. The prediction formula is as follows:

[0064]

[0065] in, It is a regression prediction model. For the predicted number of output tokens, This represents the actual number of tokens input. For task types, such as Translation, Code Generation, Summary, and Chat, this type can be obtained by matching Prompt keywords or by training a lightweight classification model to classify the input Prompt. For model identifiers, such as qwen3-max, qwen3-plus, llama-3-8b-local, etc.

[0066] Step S4: Determine the estimated cost.

[0067] In this step, the estimated cost consumption is determined based on the actual number of input tokens, the predicted number of output tokens, the context length, and the target model information obtained in the previous steps. The estimated cost consumption is measured using a unified Standard Compute Unit (SCU). Each SCU corresponds to a fixed price. The various resource costs of using the target model are measured based on the SCU. Preferably, the price of one SCU can be limited to the minimum pre-deposited amount in the account. Based on this, a dynamically adjustable Cost Mapping Table is established to convert all resource costs into a unified measurement using Standard Compute Units, setting corresponding weights. Table 1 exemplarily shows the weight settings for various resources for the large language model "qwen3-max" and the locally deployed "local_gpu_cluster".

[0068] Table 1: Cost Mapping Table

[0069]

[0070] Estimated cost consumption The calculation formula (unit: SCU) is as follows:

[0071]

[0072] in, This represents the actual number of tokens input. For the predicted number of output tokens, and They are respectively and The weight, The base cost for a single call (to prevent malicious small packet attacks). For context penalty function; and

[0073]

[0074] in,

[0075] Total length of the context;

[0076] The context window threshold for free or standard price (e.g., some existing models set it to 8192 tokens).

[0077] λ is the context penalty coefficient, defined as the rate at which the cost increases with length. For example, λ = 0.0001, which ensures that when the total context length exceeds... In the future, with each additional token, its price will increase smoothly according to the square relationship, rather than jumping in steps.

[0078] For example, if = 8000, = 0.0001, when the total length of the user call context is 9000 tokens. This means that the unit price of extremely long text will be 101 times that of the standard unit price, thus reflecting the hardware nature of video memory resources being consumed quadratically with the length of the context.

[0079] Step S5: Atomization budget operation.

[0080] In this step, the system accesses a high-performance database to check the real-time balance of the customer's account. If the account balance can cover the estimated cost at this time If the service request in step S1 is forwarded to the corresponding model provider, an atomic operation will be performed to freeze the corresponding estimated limit. If the account balance cannot cover the estimated cost, the service request will be cancelled. If this occurs, an alarm and circuit breaker will be triggered.

[0081] Step S6: Service execution and final settlement.

[0082] In this step, the gateway forwards the service request to the corresponding model provider. After the request returns, the gateway extracts the actual number of output tokens and / or the total inference time from the response header or the end frame of the stream. (For the hourly billing model), and calculate the actual cost consumption. The calculation formula is as follows:

[0083]

[0084] in, This represents the actual number of tokens input. This represents the actual number of output tokens. This represents the total actual time spent on reasoning. , and They are respectively , and The weight, For model identifier, for The basic call overhead of the corresponding model, For context penalty function; and

[0085]

[0086] in, Total length of the context The context penalty coefficient, The context window threshold for free or standard price.

[0087] Finally, calculate the deviation. The compensation operation is performed in the ledger to cancel the estimated amount and deduct the actual amount. It should be noted that during the execution of the method in this embodiment, the deviation ∆ may always exist, but the magnitude of the deviation ∆ can be controlled within a small range. This is because if the deviation ∆ is large, a circuit breaker will be triggered in step S5. In other embodiments, a minimum predetermined balance is set for the account, thereby raising the threshold for circuit breaker occurrence in step S5 to some extent, so that even if a circuit breaker occurs in step S6... Slightly larger Customers are less likely to incur outstanding fees in their accounts.

[0088] above The calculation formula is a general formula that covers both token-based and time-based billing models. It can be used to calculate token-only billing models, time-based billing models, or a combination of both. and All tasks are measured using a unified standard computing unit, thus allowing for the calculation of costs based on the token billing method for tasks already completed using the time-based billing model. Calculated via token billing method Obtained by time billing This allows for a comparison, making it suitable for evaluating the cost-effectiveness of both.

[0089] This embodiment also provides a computer-readable storage medium storing computer code that, when executed, is performed as described in any of the preceding embodiments.

[0090] This embodiment also provides a computer program product that, when executed by a computer device, performs the method described in any of the preceding embodiments.

[0091] This embodiment also provides a computer device, the computer device comprising:

[0092] One or more processors;

[0093] Memory, used to store one or more computer programs;

[0094] When the one or more computer programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described in any of the preceding methods.

[0095] Figure 2 Exemplary systems that can be used to implement the various embodiments described in this invention are shown.

[0096] like Figure 2 As shown, in some embodiments, system 1000 can function as any of the user terminal devices described in each of the embodiments. In some embodiments, system 1000 may include one or more computer-readable media having instructions (e.g., system memory or NVM / storage device 1020) and one or more processors (e.g., one or more processors 1005) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thus perform the actions described in this invention.

[0097] In one embodiment, the system control module 1010 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1005 and / or any suitable device or component communicating with the system control module 1010.

[0098] The system control module 1010 may include a memory controller module 1030 to provide an interface to the system memory 1015. The memory controller module 1030 may be a hardware module, a software module, and / or a firmware module.

[0099] System memory 1015 may be used, for example, to load and store data and / or instructions for system 1000. In one embodiment, system memory 1015 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 1015 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0100] In one embodiment, the system control module 1010 may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device 1020 and (one or more) communication interfaces 1025.

[0101] For example, the NVM / storage device 1020 may be used to store data and / or instructions. The NVM / storage device 1020 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0102] NVM / storage device 1020 may include storage resources that are physically part of a device on which system 1000 is mounted, or that can be accessed by the device without necessarily being part of the device. For example, NVM / storage device 1020 may be accessed via a network through one or more communication interfaces 1025.

[0103] One or more communication interfaces 1025 may provide the system 1000 with an interface to communicate over one or more networks and / or with any other suitable device. The system 1000 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0104] In one embodiment, at least one of the processors 1005 may be logically packaged with one or more controllers of the system control module 1010 (e.g., memory controller module 1030). In one embodiment, at least one of the processors 1005 may be logically packaged with one or more controllers of the system control module 1010 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1005 may be integrated with the logic of one or more controllers of the system control module 1010 on the same die. In one embodiment, at least one of the processors 1005 may be integrated with the logic of one or more controllers of the system control module 1010 on the same die to form a system-on-a-chip (SoC).

[0105] In various embodiments, system 1000 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 1000 may have more or fewer components and / or different architectures. For example, in some embodiments, system 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0106] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including associated data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of the present invention can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0107] Furthermore, a portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0108] Communication media include media through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided transmission) media capable of propagating energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals in, for example, wireless media (such as carrier waves or similar mechanisms embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.

[0109] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or hereafter developed capable of storing computer-readable information / data for use by a computer system.

[0110] Hereinafter, an embodiment of the present invention includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run a method and / or technical solution based on the foregoing embodiments of the present invention.

[0111] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for managing consumption using a large language model, characterized in that, include: Receive service requests from clients; Based on the service request, the requirement information is determined, which includes the number of input tokens, context length, task type information, and target model information. Predict the number of output tokens based on the aforementioned demand information; The estimated cost consumption is determined based on the number of input tokens, the number of output tokens, the context length, and the target model information. The estimated cost consumption is measured using a unified standard computing unit.

2. The method according to claim 1, characterized in that, Check the customer's account balance. If the balance can cover the estimated cost, forward the service request to the corresponding model provider and freeze the corresponding estimated limit. If the balance cannot cover the estimated cost, trigger an alarm.

3. The method according to claim 2, characterized in that, After the service request is completed, the deviation between the estimated cost and the actual cost is calculated, and a compensation operation is performed based on the deviation.

4. The method according to claim 3, characterized in that, The formula for calculating the actual cost consumption is as follows: in, This represents the actual cost incurred. This represents the actual number of tokens input. This represents the actual number of output tokens. This represents the total actual time spent on reasoning. , and They are respectively , and The weight, For model identifier, for The basic call overhead of the corresponding model, For context penalty function; and in, Total length of the context The context penalty coefficient, The context window threshold for free or standard price.

5. The method according to claim 1, characterized in that, Based on historical service requests and their results, a regression prediction model is trained to predict the number of output tokens based on the demand information.

6. The method according to claim 5, characterized in that, The loss function of the regression prediction model is as follows: in, For the model parameter set, For the first The item outputs a predicted value for the number of tokens. For the first The item outputs the actual value of the number of tokens. This is a regularization term.

7. The method according to claim 1, characterized in that, The formula for calculating the estimated cost consumption is as follows: in, To estimate cost consumption, This represents the actual number of tokens input. For the predicted number of output tokens, and They are respectively and The weight, The base cost for a single call. For context penalty function; and in, Total length of the context The context penalty coefficient, The context window threshold for free or standard price.

8. The method according to claim 1, characterized in that, The unit price of the standard computing power unit corresponds to the minimum pre-deposit amount in the account.

9. A device for managing the consumption of a large language model, wherein, The device includes: Processor; and A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the operations of the method according to any one of claims 1 to 8.

10. A computer-readable medium storing instructions that, when executed, cause a system to perform operations according to any one of claims 1 to 8.

Images