A token-based dynamic pricing method based on real-time computing power costs
By combining hardware telemetry and thermodynamic prediction modules with a dynamic billing engine in a multi-tenant inference service architecture, fine-grained billing of hardware resources is achieved, solving the problem of the disconnect between billing objects and the underlying hardware power consumption in existing technologies, improving billing accuracy and preventing overheating risks.
Patent Information
- Application Number
- CN202611306646.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, when the utilization rate of hardware computing resources is improved in a multi-tenant inference service architecture, it is impossible to accurately attribute the improvement to the computational complexity characteristics of each tenant. This results in coarse billing granularity, failure to obtain the actual physical hardware energy consumption, and a disconnect between the billing object and the underlying hardware energy consumption.
By receiving tenant requests in the gateway module, executing operator fusion operations in the computing power scheduling module, activating hardware performance counters in the hardware telemetry module, collecting temperature data in the thermodynamic prediction module, and calculating real-time unit price and triggering circuit breaker mechanism in the dynamic billing engine, refined billing is achieved.
It achieves independent isolation of power consumption under multi-tenant concurrent conditions, accurately links billing objects with underlying physical power consumption, improves billing accuracy, and prevents unlimited spikes in computing costs due to overheating.
Smart Images

Figure CN123123047A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data center computing power billing and scheduling technology, specifically to a token-based dynamic pricing method for real-time computing power costs. Background Technology
[0002] With the application of large language model technology, cloud-based AI inference services have become an important business for data centers. In commercial operations, cloud service providers typically adopt a streaming billing model based on the number of tokens generated, providing on-demand computing resources and inference power services to multiple concurrently accessing tenants.
[0003] In existing multi-tenant inference service architectures, to improve the utilization of hardware computing resources, service providers generally adopt continuous batch processing technology to merge and compute inference requests from different tenants. In this case, the billing system usually relies on the request timestamps and the total amount of tokens output at the application software level to settle batch fees according to a pre-set fixed business unit price, and uses this as the basis for recovering the overall computing power cost of the system.
[0004] However, existing technologies suffer from a deficiency in accurately attributing hardware losses when handling computing power cost accounting in continuous batch processing. Because matrix multiplication operators from multiple tenants are executed in fusion within the underlying graphics processor, traditional pure software billing logic lacks real-time linkage with underlying hardware performance counters. This prevents fine-grained attribution and allocation of macroscopic physical losses based on the computational complexity characteristics of each tenant under concurrent conditions. This billing model results in coarse-grained billing, failing to capture the true physical hardware energy consumption down to the individual tenant level. Consequently, the billing objects at the business layer are disconnected from the actual physical computing power losses at the underlying level, making it difficult to accurately link the billing objects with the actual underlying hardware energy consumption. Summary of the Invention
[0005] To address the problems mentioned in the background, this invention provides a token-based dynamic pricing method for real-time computing power costs. This method is implemented using a token-based dynamic pricing system that utilizes real-time computing power costs and involves the following steps:
[0006] In step S100, the gateway module concurrently receives inference requests from multiple tenants, parses and obtains the tolerance marginal unit price parameter in the inference request, and then sends the inference request to the computing power scheduling module.
[0007] In step S200, the computing power scheduling module uses the continuous batch processing engine to extract the context state information of each inference request. The computing power scheduling module identifies the corresponding pre-filling stage operator or decoding stage operator based on the context state information. The computing power scheduling module performs operator fusion operation to generate a matrix multiplication computation graph containing the inference request. The computing power scheduling module sends the matrix multiplication computation graph to the graphics processor instruction queue of the computing node for execution. The specific implementation of the operator fusion operation is that the computing power scheduling module aligns and splices the tensor dimensions of multiple different tenants in the same execution stage to construct a unified batch processing computation kernel function instruction.
[0008] In step S300, during the cycle of the graphics processor executing the matrix multiplication computation graph, the hardware telemetry module activates the hardware performance counter in the graphics processor. At the end of the execution cycle, the hardware telemetry module accesses the hardware performance counter and extracts the physical hardware macro-loss data of the execution cycle. The physical hardware macro-loss data includes the total number of active cycles of the streaming multiprocessor and the total number of read and write bytes of high-bandwidth memory.
[0009] In step S400, synchronized with step S300, the thermodynamic prediction module performs asynchronous physical state acquisition in the out-of-band channel. The thermodynamic prediction module reads the discrete temperature measurement of the computing node and the power consumption data of the graphics processor core through the baseboard management controller. The thermodynamic prediction module obtains the current timestamp difference. The thermodynamic prediction module multiplies the power consumption data of the graphics processor core by a preset thermal conversion coefficient to obtain the temperature change gradient. The temperature change gradient is multiplied by the current timestamp difference and then superimposed on the discrete temperature measurement to calculate and output the transient core temperature prediction value in the current execution cycle.
[0010] In step S500, the dynamic billing engine receives the macroscopic loss data of physical hardware and the transient core temperature prediction value. The dynamic billing engine extracts the context theoretical computational complexity features of each inference request in the current batch. The context theoretical computational complexity features include the theoretical number of floating-point operations and the theoretical memory usage of each inference request. The dynamic billing engine divides the resource allocation weights according to the context theoretical computational complexity features. The dynamic billing engine maps the macroscopic loss data of physical hardware to the corresponding micro tenants according to the resource allocation weights and calculates the basic physical cost of each inference request.
[0011] In step S600, the dynamic billing engine matches the corresponding nonlinear thermodynamic penalty multiplier based on the transient core temperature prediction value. The dynamic billing engine multiplies the basic physical cost by the nonlinear thermodynamic penalty multiplier and calculates and outputs the real-time dynamic unit price of the single token generated by each inference request.
[0012] In step S700, the dynamic billing engine extracts the tolerance marginal unit price parameter obtained in step S100 and performs a numerical comparison and verification between the real-time dynamic unit price and the corresponding tolerance marginal unit price parameter.
[0013] In step S800, in response to the comparison result that the real-time dynamic unit price is less than or equal to the tolerance marginal unit price parameter, the dynamic billing engine sends a streaming amount deduction instruction to the corresponding tenant account, the computing node outputs the currently generated single token, and enters the next round of decoding loop operation.
[0014] In step S900, in response to the comparison result that the real-time dynamic unit price is greater than the tolerance marginal unit price parameter, the dynamic billing engine triggers the system circuit breaker mechanism. The dynamic billing engine sends a request reorganization instruction to the computing power scheduling module. The computing power scheduling module receives the request reorganization instruction and stops the operation of the inference request context that generated the out-of-bounds price in the current computing node. The computing power scheduling module transfers the stopped inference request context to the waiting queue of low-power cold state nodes in the system.
[0015] Preferably, in step S200, after the computing power scheduling module performs the operator fusion operation, it also includes dynamic memory allocation of computing power. Specifically, the computing power scheduling module pre-divides a fixed-size memory block pool in the global memory of the computing node; the computing power scheduling module extracts the context tensor of the memory blocks as needed and allocates them to different tenants, and records the memory virtual address offset of each memory block; when constructing the matrix multiplication computation graph, the computing power scheduling module directly hardcodes the memory virtual address offset of each memory block into the computation instructions, so that the computing node directly reads in parallel according to the memory virtual address offset.
[0016] Preferably, in step S300, the hardware telemetry module extracts the physical hardware macroscopic wear data of the execution cycle as follows: the hardware telemetry module inserts an asynchronous event recording instruction into the execution instruction stream of the matrix multiplication computation graph, and the hardware telemetry module listens to the execution completion status of the instruction stream through the asynchronous event recording instruction; in response to capturing the execution completion status, the hardware telemetry module accesses the hardware performance counter register of the graphics processor, and the hardware telemetry module reads the specific performance counter register address pre-mapped in the memory-mapped input / output space through the peripheral component interconnect bus, and extracts the total number of active cycles and the total number of read / write bytes.
[0017] Preferably, in step S400, when the thermodynamic prediction module obtains the transient core temperature prediction value, the specific implementation of asynchronous physical state acquisition is as follows: the thermodynamic prediction module stores the discrete temperature measurement and the graphics processor core power consumption data into a pre-set circular memory buffer; at the instant the thermodynamic prediction module receives the discrete physical state data in the circular memory buffer, it calls the high-precision event timer on the computing node motherboard to generate a local timestamp; the thermodynamic prediction module binds and stores the local timestamp with the discrete temperature measurement and the graphics processor core power consumption data; the thermodynamic prediction module extracts the latest local timestamp from the circular memory buffer, subtracts the latest local timestamp from the current high-frequency execution time, and calculates the timestamp difference.
[0018] Preferably, in step S500, the specific implementation of the dynamic billing engine extracting the context theoretical computational complexity features and dividing the resource allocation weights is as follows: the dynamic billing engine extracts the computational context tensor dimension data of the K concurrently executed inference requests in the current batch. The computational context tensor dimension data includes the input sequence length and hidden layer dimension of each inference request; the dynamic billing engine multiplies the input sequence length of the inference request by the square of the hidden layer dimension of the inference request to obtain the theoretical floating-point operation count of the corresponding inference request in the pre-filling stage; the dynamic billing engine multiplies the input sequence length of the inference request by the hidden layer dimension of the inference request, and combines it with the network layer number feature quantity preset by the model to obtain the key-value cache tensor capacity of the corresponding inference request in the decoding stage, and uses the key-value cache tensor capacity as the theoretical GPU memory usage of the corresponding inference request;
[0019] When the dynamic billing engine detects that an inference request is in the pre-filling stage, it assigns a first weight coefficient to the phase indicator factor; when it detects that an inference request is in the decoding stage, it assigns a second weight coefficient to the phase indicator factor, and the first weight coefficient is greater than the second weight coefficient. The dynamic billing engine calculates the resource allocation weight corresponding to each inference request based on the theoretical number of floating-point operations, the theoretical memory usage, and the phase indicator factor. The formula for calculating the resource allocation weight is as follows:
[0020]
[0021] in, Indicates the first The resource allocation weight of each inference request in the current batch; This indicates the total number of concurrent inference requests included in the current batch; Indicates the first The theoretical number of floating-point operations per inference request; Indicates the first The theoretical number of floating-point operations per inference request; Indicates the first Theoretical memory usage per inference request; Indicates the first Theoretical memory usage per inference request; This indicates the phase indicator factor.
[0022] Preferably, in step S500, the specific implementation of the dynamic billing engine calculating the basic physical cost of each inference request is as follows: The dynamic billing engine obtains the system's preset microarchitecture basic rate parameters, which include the baseline depreciation and power rate per unit stream multiprocessor active cycle, and the baseline energy consumption rate per unit high-bandwidth memory read / write byte; the dynamic billing engine multiplies the total number of active cycles by the baseline depreciation and power rate, and multiplies the total number of read / write bytes by the baseline energy consumption rate; the dynamic billing engine adds the two multiplication results to calculate the total macroscopic physical cost of the graphics processor within the execution cycle; the dynamic billing engine multiplies the total macroscopic physical cost by the resource allocation weight corresponding to the inference request, and divides the multiplication result by the number of tokens generated by the inference request within the execution cycle to calculate and output the basic physical cost of the inference request. The calculation formula for the basic physical cost is as follows:
[0023]
[0024] in, Indicates the first The basic physical cost of generating a single token for each inference request within the execution cycle; Indicates the first The resource allocation weight of each inference request in a batch; The baseline depreciation and electricity rate represent the active cycle of a single stream multiprocessor. This indicates the total number of active cycles for the streaming multiprocessor. The baseline energy consumption rate is expressed as a unit of high-bandwidth memory read / write bytes; This indicates the total number of bytes read and written to high-bandwidth memory; Indicates the first The number of tokens actually generated by each inference request during the execution cycle.
[0025] Preferably, in step S600, the specific implementation of the dynamic billing engine matching the nonlinear thermodynamic penalty multiplier is as follows: The dynamic billing engine reads the equipment serial number of the rack cooling system, queries the preset hardware specification mapping table in the system based on the equipment serial number, extracts the rated power peak and cooling energy efficiency ratio curve recorded in the hardware specification mapping table, performs a division operation on the rated power peak and cooling energy efficiency ratio curve, and outputs the cooling elasticity base coefficient; the dynamic billing engine subtracts the optimal energy efficiency temperature threshold from the transient core temperature prediction value to obtain the temperature offset; the dynamic billing engine subtracts the transient core temperature prediction value from the frequency reduction critical extreme temperature and adds the anti-overflow minimum normal number to obtain the frequency reduction temperature margin; the dynamic billing engine divides the temperature offset by the frequency reduction temperature margin to obtain the thermodynamic relative approximation, performs natural exponential operation, calculates the exponential characteristic value of the thermodynamic relative approximation, multiplies the exponential characteristic value by the cooling elasticity base coefficient, and adds the product result to the preset basic constant to calculate and output the nonlinear thermodynamic marginal penalty multiplier. The calculation formula of the nonlinear thermodynamic marginal penalty multiplier is as follows:
[0026]
[0027] in, This represents the nonlinear thermodynamic marginal penalty multiplier; Indicates the basic coefficient of refrigeration elasticity; This represents the predicted transient core temperature. This indicates the optimal energy efficiency temperature threshold; Indicates the critical extreme temperature for frequency reduction; This represents a minimum positive number to prevent overflow.
[0028] Preferably, in step S800, the specific implementation of the dynamic billing engine sending the streaming amount deduction instruction to the corresponding tenant account is as follows: the dynamic billing engine establishes an independent mapping billing accumulator for each inference request in the system memory; when the graphics processor completes the decoding output of each token, the dynamic billing engine calls the lock-free atomic addition instruction at the system's underlying level to directly accumulate the real-time dynamic unit price according to the virtual address offset of the video memory into the memory segment corresponding to the billing accumulator to perform the integral accumulation operation and obtain the cumulative consumption amount of the inference request; the dynamic billing engine extracts the reserved quota data of the tenant account corresponding to the inference request, compares and verifies the cumulative consumption amount with the reserved quota data, and in response to the comparison result that the cumulative consumption amount is less than or equal to the reserved quota data, sends a data packet release instruction to the gateway module.
[0029] Preferably, in step S900, the specific implementation of the dynamic billing engine triggering the system circuit breaker mechanism and the computing power scheduling module suspending the inference request context operation is as follows: the computing power scheduling module captures the scheduling-level billing interrupt signal, and controls the continuous batch processing engine to perform a context stripping operation on the inference request that generates an out-of-bounds price at the Kernel execution boundary at the end of the current decoding loop; in the batch construction of the next computing cycle, the continuous batch processing engine removes the sequence identifier corresponding to the inference request that generates an out-of-bounds price, and releases the occupied identifier of the video memory virtual address corresponding to the inference request in the global memory pool.
[0030] Preferably, the specific implementation of the computing power scheduling module transferring the context of aborted inference requests to the waiting queue of low-power cold state nodes in the system is as follows: For inference requests that generate out-of-bounds prices, the computing power scheduling module extracts the key-value cache tensor residing in the graphics processor's video memory and the already generated identifier sequence, serializes and packages the key-value cache tensor and the identifier sequence to generate a standard breakpoint resume status file, and transfers the inference request after generating the breakpoint resume status file to the economic queue in the system to wait; the computing power scheduling module sends a thermodynamic status query message to the global cluster management node in the data center, receives the global node temperature routing table returned by the global cluster management node, parses the global node temperature routing table, and obtains the network location information of idle graphics processor nodes whose core temperature is lower than the optimal energy efficiency temperature threshold; the computing power scheduling module reschedules the inference requests in the economic queue to the idle graphics processor nodes to resume decoding execution, loads the breakpoint resume status file into the video memory of the idle graphics processor nodes, and controls the idle graphics processor nodes to continue executing autoregressive generation according to the identifier sequence.
[0031] The present invention, by adopting the above technical solution, can bring the following beneficial effects:
[0032] 1. This invention uses the hardware telemetry module (40) to insert asynchronous event recording instructions to access the hardware performance counter to extract the total number of active cycles of the streaming multiprocessor and the total number of read and write bytes of high bandwidth memory. It also uses the dynamic billing engine to extract the context theoretical calculation complexity features to divide the resource allocation weights and map macroscopic physical losses to microscopic tenants. This achieves the effect of independent isolation of power consumption in multi-tenant concurrent state and accurate linkage between billing objects and underlying physical energy consumption.
[0033] 2. This invention uses a thermodynamic prediction module to read discrete temperature measurements from the substrate management controller and GPU core power consumption data from the out-of-band channel and store them in a buffer. It calls a high-precision timer on the computing node to generate a local timestamp to eliminate the risk of cross-hardware clock domain error calculation. It combines cooling power consumption with thermal conversion coefficient to obtain compensation amount and adds it to discrete measurements for interpolation calculation. This achieves the effect of aligning the calculation cycle with continuous temperature and providing high-precision thermal parameter input for billing.
[0034] 3. This invention compares and verifies the real-time dynamic unit price with the tolerance marginal unit price through a dynamic billing engine, and uses lock-free atomic addition instructions to perform memory accumulation. When the unit price exceeds the limit, the computing power scheduling module is triggered to lock the video memory block, strip the context data, extract the key-value cache tensor and identifier sequence to generate a status file and migrate it to an idle node. This achieves the effect of underlying hardware thermal status and thermal load closed-loop control loop based on price levers, as well as global cluster thermal load balancing scheduling. Attached Figure Description
[0035] Figure 1 This is a flowchart illustrating the overall process of the token-based dynamic pricing method for real-time computing power costs according to the present invention.
[0036] Figure 2 This is a block diagram of the token-based dynamic pricing system architecture for real-time computing power costs, as described in this invention.
[0037] Figure 3 This is a flowchart of the multi-tenant concurrent computing power decoupling and resource allocation weight calculation process of the present invention;
[0038] Figure 4 This is a flowchart illustrating the basic physical cost mapping and hardware macroscopic loss extraction process of this invention.
[0039] Figure 5 This is a flowchart of the out-of-band asynchronous acquisition of physical states and thermodynamic penalty multiplier prediction of the present invention;
[0040] Figure 6 This is a flowchart illustrating the real-time dynamic unit price generation of tokens and the account-based streaming deduction process of this invention.
[0041] Figure 7 This is a flowchart of the request circuit breaking and cross-node reorganization scheduling process based on the billing threshold of the present invention. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] Example 1
[0044] Please refer to the appendix. Figure 1 This invention provides a token-based dynamic pricing method for real-time computing power costs. It is implemented using a token-based dynamic pricing system that utilizes real-time computing power costs and executes the following steps:
[0045] In step S100, the gateway module 10 concurrently receives inference requests from multiple tenants, parses and obtains the tolerance marginal unit price parameter in the inference request, and sends the inference request to the computing power scheduling module 20.
[0046] In step S200, the computing power scheduling module 20 uses the continuous batch processing engine 21 to extract the context state information of each inference request. The computing power scheduling module 20 identifies the corresponding pre-filling stage operator or decoding stage operator based on the context state information. The computing power scheduling module 20 performs operator fusion operation to generate a matrix multiplication computation graph containing the inference request. The computing power scheduling module 20 sends the matrix multiplication computation graph to the instruction queue of the graphics processor 31 of the computing node 30 for execution. The computing power scheduling module 20 aligns and splices the tensor dimensions of multiple different tenants in the same execution stage to construct a unified batch processing computation kernel function instruction.
[0047] In step S300, during the cycle of the matrix multiplication calculation graph executed by the graphics processor 31, the hardware telemetry module 40 activates the hardware performance counter in the graphics processor 31. At the end of the execution cycle, the hardware telemetry module 40 accesses the hardware performance counter and extracts the physical hardware macro-loss data of the execution cycle. The physical hardware macro-loss data includes the total number of active cycles of the streaming multiprocessor 311 and the total number of read and write bytes of the high-bandwidth memory 312. By extracting the physical hardware macro-loss data, the billing system is provided with a basis for the real underlying hardware overhead that is independent of software simulation.
[0048] In step S400, synchronized with step S300, the thermodynamic prediction module 50 performs asynchronous physical state acquisition in the out-of-band channel. The thermodynamic prediction module 50 reads the discrete temperature measurement of the computing node 30 and the power consumption data of the graphics processor core through the substrate management controller 32. The thermodynamic prediction module 50 obtains the current timestamp difference. The thermodynamic prediction module 50 multiplies the power consumption data of the graphics processor core by a preset heat conversion coefficient to obtain the temperature change gradient. The temperature change gradient is multiplied by the current timestamp difference and then superimposed on the discrete temperature measurement to calculate and output the transient core temperature prediction value in the current execution cycle.
[0049] In step S500, the dynamic billing engine 60 receives the physical hardware macroscopic loss data and the transient core temperature prediction value. The dynamic billing engine 60 extracts the context theoretical computational complexity features of each inference request in the current batch. The context theoretical computational complexity features include the theoretical floating-point operation count and theoretical video memory usage of each inference request. The dynamic billing engine 60 divides the resource allocation weights according to the context theoretical computational complexity features. The dynamic billing engine 60 maps the physical hardware macroscopic loss data to the corresponding micro tenants according to the resource allocation weights and calculates the basic physical cost of each inference request.
[0050] In step S600, the dynamic billing engine 60 matches the corresponding nonlinear thermodynamic penalty multiplier based on the transient core temperature prediction value. The dynamic billing engine 60 multiplies the basic physical cost by the nonlinear thermodynamic penalty multiplier and calculates and outputs the real-time dynamic unit price of the single token generated by each inference request.
[0051] In step S700, the dynamic billing engine 60 extracts the tolerance marginal unit price parameter obtained in step S100 and performs a numerical comparison and verification between the real-time dynamic unit price and the corresponding tolerance marginal unit price parameter.
[0052] In step S800, in response to the comparison result that the real-time dynamic unit price is less than or equal to the tolerance marginal unit price parameter, the dynamic billing engine 60 sends a streaming amount deduction instruction to the corresponding tenant account, the computing node 30 outputs the currently generated single token, and enters the next round of decoding loop operation.
[0053] In step S900, in response to the comparison result that the real-time dynamic unit price is greater than the tolerance marginal unit price parameter, the dynamic billing engine 60 triggers the system circuit breaker mechanism. The dynamic billing engine 60 sends a request reorganization instruction to the computing power scheduling module 20. The computing power scheduling module 20 receives the request reorganization instruction and stops the operation of the inference request context that generated the out-of-bounds price in the current computing node 30. The computing power scheduling module 20 transfers the stopped inference request context to the waiting queue of the computing node in the system that is in a low-power cold state.
[0054] The above steps improve the parallel computing throughput of the graphics processor 31 by performing operator fusion operations; overcome the communication latency barrier caused by the mismatch between the low-frequency sampling rate of the underlying physical sensors and the extremely short inference cycle of the upper layer by performing asynchronous acquisition of out-of-band channels, timestamp alignment, and gradient-based temperature prediction; overcome the technical defect that the graphics processor 31 cannot directly perform power isolation detection of concurrent tenants at the physical circuit level in continuous batch processing by dividing and mapping resource allocation weights based on theoretical computational complexity characteristics; effectively transmit the physical cooling cost of the data center to the token generation unit price at the micro level by introducing a nonlinear thermodynamic penalty multiplier; and effectively avoid the risk of unlimited soaring computing costs caused by single-node overheating by triggering the system circuit breaker mechanism and issuing request reorganization instructions to build a feedback control loop between the underlying hardware thermodynamic state and the upper-layer business billing rules.
[0055] Example 2
[0056] Please refer to the appendix. Figure 2Based on Example 1, the above-mentioned token-based billing dynamic pricing system based on real-time computing power cost is deployed in a supercomputing data center, connecting external client requests with underlying computing nodes. The token-based billing dynamic pricing system based on real-time computing power cost includes a gateway module 10, a computing power scheduling module 20, a computing node 30, a hardware telemetry module 40, a thermodynamic prediction module 50, and a dynamic billing engine 60.
[0057] The gateway module 10 is configured to receive supercomputing inference requests from external multi-tenants. The gateway module 10 extracts the authorization credential parameters and tolerance marginal unit price parameters carried in the request header. By extracting the tolerance marginal unit price parameters through the gateway module 10, the token billing dynamic pricing system based on real-time computing power costs can obtain the tenant's expected cost threshold, thereby providing a trigger benchmark for subsequent thermodynamic sensing scheduling and billing circuit breakers, and preventing tenant settlement anomalies caused by unit price spikes.
[0058] The computing power scheduling module 20 is communicatively connected to the gateway module 10. The computing power scheduling module 20 has a built-in continuous batch processing engine 21. The continuous batch processing engine 21 divides each inference request into pre-filling stage operators and decoding stage operators according to the execution status of each inference request. The computing power scheduling module 20 reassembles multiple concurrent inference requests into a single matrix multiplication computation graph through an operator fusion mechanism. The computing power scheduling module 20 then sends the matrix multiplication computation graph to the computing node 30 for execution.
[0059] The computing power scheduling module 20 pre-divides a fixed-size memory block pool in the global memory of the computing node 30. The computing power scheduling module 20 extracts the memory blocks as needed and allocates them to the context tensors of different tenants, and records the memory virtual address offset of each memory block. When constructing the matrix multiplication computation graph, the computing power scheduling module 20 hardcodes the memory virtual address offset of each memory block directly into the computation instructions, so that the computing node 30 can directly read in parallel according to the memory virtual address offset. By introducing the above-mentioned dynamic memory allocation mechanism of hard-coding the memory virtual address offset, the memory access lock contention problem in the multi-threaded concurrent environment is avoided, and the computation graph execution throughput under multi-tenant concurrent requests is effectively improved.
[0060] The computing node 30 includes a graphics processor 31 and a baseboard management controller 32. The graphics processor 31 integrates a streaming multiprocessor 311 and a high-bandwidth memory 312. The streaming multiprocessor 311 executes the matrix operation logic in the matrix multiplication computation graph, and the high-bandwidth memory 312 provides storage space for reading and writing tensor data.
[0061] The hardware telemetry module 40 is deployed in the driver layer of the computing node 30. The hardware telemetry module 40 directly accesses the hardware performance counter register in the graphics processor 31. At the end of the operator execution cycle, the hardware telemetry module 40 reads and extracts the effective active cycle number of the streaming multiprocessor 311 and the number of read and write bytes of the high-bandwidth memory 312. The hardware telemetry module 40 outputs the physical hardware macro-consumption data of the previous execution cycle. By directly accessing the hardware performance counter register through the hardware telemetry module 40, the underlying real physical resource consumption can be accurately obtained without causing context switching delay or interrupting the normal inference pipeline, providing accurate hardware data support for micro token pricing.
[0062] The thermodynamic prediction module 50 establishes communication with the substrate management controller 32 through the out-of-band channel interconnected by peripheral high-speed components. The thermodynamic prediction module 50 runs independently of the kernel scheduling framework of the operating system. The thermodynamic prediction module 50 collects the core temperature data of the graphics processor 31 and the instantaneous power data of the rack cooling equipment. The thermodynamic prediction module 50 constructs a time-smoothing prediction model based on discrete low-frequency sampling data points and outputs a transient temperature prediction value aligned with the high-frequency inference cycle time. By constructing a smooth prediction model through the thermodynamic prediction module 50 and running it independently in the out-of-band channel, it avoids the monitoring task from crowding out the computing power of the main processor and eliminates the latency barrier caused by the mismatch between the low-frequency sampling limitation of physical sensors and the millisecond-level inference cycle.
[0063] The dynamic billing engine 60 establishes data connections with the hardware telemetry module 40 and the thermodynamic prediction module 50, respectively. The dynamic billing engine 60 acquires physical hardware macroscopic loss data and transient temperature prediction values. The dynamic billing engine 60 allocates computing resource weights based on the request sequence length of each tenant. The dynamic billing engine 60 splits and maps the physical hardware macroscopic loss data into the basic computing cost of a single tenant, and adds a thermodynamic penalty multiplier converted from the transient temperature prediction value to calculate the dynamic price of each tenant's current token. The dynamic billing engine 60 establishes a feedback control connection with the computing power scheduling module 20, and executes computing power reorganization intervention commands based on the dynamic price. Through multi-dimensional feature fusion calculation, the dynamic billing engine 60 realizes the accurate transmission of data center physical environment costs to the upper-layer business side. At the same time, through the feedback control connection, the system is given the ability to spontaneously adjust the cluster load distribution through price leverage when encountering local heat dissipation bottlenecks, preventing the risk of single computing node downtime.
[0064] Example 3
[0065] Please refer to the appendix. Figure 3Based on the first embodiment, in step S500, for the case where multi-tenant requests share underlying hardware resources in continuous batch processing, the dynamic billing engine 60 needs to decouple the computing power. Specifically, the dynamic billing engine 60 establishes data communication with the continuous batch processing engine 21 in the computing power scheduling module 20. The dynamic billing engine 60 extracts the computation context tensor dimension data of the K inference requests executed concurrently in the current batch. The computation context tensor dimension data includes the input sequence length and hidden layer dimension of each inference request.
[0066] Subsequently, the dynamic billing engine 60 calculates the theoretical number of floating-point operations and the theoretical memory usage of each inference request in the current batch based on the computation context tensor dimension data. The dynamic billing engine 60 multiplies the length of the input sequence of the inference request by the square of the hidden layer dimension of the inference request to obtain the theoretical number of floating-point operations of the corresponding inference request in the pre-filling stage. The dynamic billing engine 60 multiplies the length of the input sequence of the inference request by the hidden layer dimension of the inference request, and combines it with the network layer number feature of the model to obtain the key-value cache tensor capacity of the corresponding inference request in the decoding stage. The dynamic billing engine 60 uses the key-value cache tensor capacity as the theoretical memory usage of the corresponding inference request. By extracting the theoretical number of floating-point operations and the theoretical memory usage, the system transforms the multi-tenant concurrent computation loss that cannot be directly isolated at the physical circuit level into a measurable theoretical mathematical feature.
[0067] After obtaining the theoretical feature quantity, the dynamic billing engine 60 generates a phase indicator factor based on the current execution state of each inference request. The execution state is divided into a computationally intensive pre-filling stage and a memory-intensive decoding stage. When the dynamic billing engine 60 detects that the inference request is in the pre-filling stage, it assigns a first weight coefficient (e.g., 0.8) to the phase indicator factor to characterize that the pre-filling stage is highly dependent on the computational throughput of the streaming multiprocessor 311. When the dynamic billing engine 60 detects that the inference request is in the decoding stage, it assigns a second weight coefficient (e.g., 0.2) to the phase indicator factor, and the first weight coefficient is greater than the second weight coefficient to characterize that the decoding stage is highly dependent on the memory access bandwidth of the high-bandwidth memory 312.
[0068] Finally, the dynamic billing engine 60 calculates the resource allocation weight for each inference request based on the theoretical number of floating-point operations, the theoretical memory usage, and the phase indicator factor. The formula for calculating the resource allocation weight is as follows:
[0069]
[0070] in, Indicates the first The resource allocation weight of each inference request in the current batch; This indicates the total number of concurrent inference requests included in the current batch; Indicates the first The theoretical number of floating-point operations per inference request; Indicates the first Theoretical memory usage per inference request; Indicates the phase indicator factor;
[0071] By performing the above weight quantization calculation, the dynamic billing engine 60 constructs a resource ratio quantization channel based on the individual characteristics of multiple tenants under the fusion matrix multiplication calculation graph. While maintaining the high concurrent throughput of the graphics processor 31, the resource ratio quantization channel based on the individual characteristics of multiple tenants provides a theoretical computing power cost allocation basis accurate to a single inference request, overcoming the engineering obstacle that the physical hardware resource consumption cannot be measured independently under continuous batch processing mode.
[0072] Example 4
[0073] Please refer to the appendix. Figure 4-5 Based on Example 3, the resource allocation weights calculated above are combined with the actual hardware losses at the underlying level by the dynamic billing engine 60 to complete the microscopic mapping of the basic physical cost. At the same time, the thermodynamic prediction module 50 performs asynchronous thermal state prediction to obtain the transient core temperature prediction value.
[0074] The hardware telemetry module 40 inserts asynchronous event recording instructions into the execution instruction stream of the matrix multiplication computation graph. The hardware telemetry module 40 listens to the execution completion status of the instruction stream through the asynchronous event recording instructions. In response to capturing the execution completion status, the hardware telemetry module 40 accesses the hardware performance counter register of the graphics processor 31. The hardware telemetry module 40 reads and extracts the total number of active cycles of the streaming multiprocessor 311 and the total number of read and write bytes of the high-bandwidth memory 312. The hardware telemetry module 40 reads the specific performance counter register address pre-mapped in the memory-mapped input / output space through the peripheral component interconnect bus to extract the total number of active cycles and the total number of read and write bytes.
[0075] By reading the address of a specific performance counter register pre-mapped in the memory-mapped input / output space, the context switching overhead caused by calling the operating system kernel-level interrupt response is avoided, ensuring the continuity of instruction execution of the graphics processor 31 in continuous batch processing state. At the same time, by inserting asynchronous event recording instructions into the instruction stream, the reading action of the hardware performance counter register and the completion time of the calculation instruction are strictly aligned, avoiding the loss of data acquisition drift caused by the asynchronous execution characteristics of the graphics processor 31.
[0076] The dynamic billing engine 60 obtains the system's preset microarchitecture base rate parameters, which include the baseline depreciation and power rate per active cycle of a stream multiprocessor, and the baseline power consumption rate per unit of high-bandwidth memory read / write bytes. The dynamic billing engine 60 multiplies the total number of active cycles with the baseline depreciation and power rate, and multiplies the total number of read / write bytes with the baseline power consumption rate. The dynamic billing engine 60 adds the two multiplication results to calculate the total macroscopic physical cost of the graphics processor 31 within the execution cycle. By introducing the microarchitecture base rate parameters and combining them with the underlying total number of active cycles and total number of read / write bytes, the system's coarse billing mode based on usage time is transformed into a quantitative cost measurement mode based on the actual consumption of hardware microinstructions.
[0077] The dynamic billing engine 60 obtains the resource allocation weight of the inference request calculated in the aforementioned steps. The dynamic billing engine 60 multiplies the total macro physical cost by the resource allocation weight corresponding to the inference request, and divides the result of the multiplication by the number of tokens generated by the inference request during the execution period to calculate and output the basic physical cost of the inference request. The formula for calculating the basic physical cost is as follows:
[0078]
[0079] in, Indicates the first The basic physical cost of generating a single token for each inference request within the execution cycle; Indicates the first The resource allocation weight of each inference request in a batch; The baseline depreciation and electricity rate represent the active cycle of a single stream multiprocessor. This indicates the total number of active cycles for the streaming multiprocessor 311; The baseline energy consumption rate is expressed as a unit of high-bandwidth memory read / write bytes; This indicates the total number of bytes read and written to the high-bandwidth memory (312); Indicates the first The equivalent number of tokens processed per inference request within the execution cycle, for the pre-population phase. The sum of the input sequence length and the number of the first output tokens; for the decoding phase, The number of tokens generated within a single execution cycle;
[0080] By performing formula mapping calculations of basic physical costs, the dynamic billing engine 60 accurately breaks down the total macroscopic physical cost of the graphics processor 31 to each micro-tenant participating in concurrent calculations. Performing formula mapping calculations of basic physical costs overcomes the engineering defect that physical circuits cannot independently isolate the power consumption of multiple tenants in the operator fusion state, and realizes the accurate link between the billing object of the business layer and the underlying physical loss.
[0081] To address the technical issue of the mismatch between the sampling frequency of the underlying physical sensors and the execution frequency of the upper-layer business inference, the thermodynamic prediction module 50 executes an asynchronous thermal state prediction mechanism based on out-of-band low-frequency sampling to obtain the transient core temperature prediction value.
[0082] The thermodynamic prediction module 50 establishes a data communication connection with the substrate management controller 32 through the out-of-band interconnection channel of the peripheral high-speed components in the computing node 30, and sends a physical state reading command to the substrate management controller 32.
[0083] The thermodynamic prediction module 50 receives discrete physical state data returned by the baseboard management controller 32. The discrete physical state data includes discrete temperature measurements of the graphics processor 31 at low-frequency sampling times and graphics processor core power consumption data of the rack cooling equipment. The thermodynamic prediction module 50 stores the discrete temperature measurements and graphics processor core power consumption data in a pre-set circular memory buffer. To avoid hardware clock drift between the baseboard management controller 32 and the main system of the computing node 30, the thermodynamic prediction module 50 calls the high-precision event timer on the motherboard of the computing node 30 to generate a local timestamp identifier the moment it receives the discrete physical state data in the circular memory buffer. The thermodynamic prediction module 50 binds and stores the local timestamp identifier with the discrete temperature measurements and graphics processor core power consumption data. By calling the high-precision event timer on the motherboard to generate the local timestamp identifier, the time base of the underlying physical state data and the upper-layer business inference task is unified, avoiding the risk of time difference miscalculation caused by cross-hardware clock domains.
[0084] By using an out-of-band independent channel to read data and store it in a circular memory buffer, the thermodynamic prediction module 50 bypasses the kernel scheduling framework of the operating system in the process of acquiring the underlying physical measure, thus avoiding the occupation and interference of the monitoring task on the inference computing resources of the graphics processor 31.
[0085] When the dynamic billing engine 60 initiates a billing request, the thermodynamic prediction module 50 obtains the current high-frequency execution time when the dynamic billing engine 60 initiates the billing request, the thermodynamic prediction module 50 extracts the latest local timestamp identifier from the circular memory buffer, and the thermodynamic prediction module 50 subtracts the latest local timestamp identifier from the current high-frequency execution time to calculate and obtain the timestamp difference.
[0086] The thermodynamic prediction module 50 obtains the preset heat conversion coefficient within the system. The heat conversion coefficient is a hardware evaluation constant characterizing the rate of temperature rise caused by a unit heat dissipation of a specific model of graphics processor 31. The thermodynamic prediction module 50 multiplies the core power consumption data of the graphics processor by the heat conversion coefficient to calculate the temperature change gradient. The thermodynamic prediction module 50 multiplies the temperature change gradient by the timestamp difference to calculate the temperature change compensation amount. The thermodynamic prediction module 50 superimposes the temperature change compensation amount onto the discrete temperature measure to complete the linear interpolation operation across the time domain and calculates and outputs the transient core temperature prediction value.
[0087] By performing out-of-band low-frequency sampling and asynchronous thermal state prediction operations, the thermodynamic prediction module 50 converts the second-level discrete physical data provided by the substrate management controller 32 into continuous temperature feature quantities aligned with the millisecond-level matrix operation cycle. The asynchronous thermal state prediction mechanism solves the constraint of the underlying physical sensor hardware polling communication bottleneck on the real-time performance of micro billing, and provides reliable transient thermodynamic parameter input for the dynamic billing engine 60.
[0088] The dynamic billing engine 60 receives the transient core temperature prediction value output by the thermodynamic prediction module 50. The dynamic billing engine 60 obtains the system's preset optimal energy efficiency temperature threshold and the frequency reduction critical extreme temperature. The optimal energy efficiency temperature threshold is the minimum power consumption maintenance temperature of the graphics processor 31 under standard heat dissipation conditions. The frequency reduction critical extreme temperature is the hardware protection temperature at which the graphics processor 31 triggers the underlying basic input / output system to actively reduce the clock frequency.
[0089] Subsequently, the dynamic billing engine 60 obtains the cooling elasticity base coefficient and the anti-overflow minimum normal number. The cooling elasticity base coefficient represents the increase ratio of power consumption of the rack cooling system under unit temperature change. The dynamic billing engine 60 reads the equipment serial number of the rack cooling system, queries the pre-set hardware specification mapping table in the system based on the equipment serial number, extracts the rated power peak and cooling energy efficiency ratio curve recorded in the hardware specification mapping table, performs a division operation on the rated power peak and cooling energy efficiency ratio curve, and outputs the cooling elasticity base coefficient that matches the current physical environment. By dynamically obtaining the cooling elasticity base coefficient by querying the hardware specification mapping table, it ensures that the nonlinear pricing model can adapt to the heterogeneous cooling physical equipment of different data center computer rooms, improving the generalization ability and cost transmission accuracy of the model.
[0090] Next, the dynamic billing engine 60 subtracts the optimal energy efficiency temperature threshold from the transient core temperature prediction value to obtain the temperature offset. The dynamic billing engine 60 subtracts the transient core temperature prediction value from the frequency reduction critical extreme temperature and adds the anti-overflow minimum normal number to obtain the frequency reduction temperature margin.
[0091] Finally, the dynamic billing engine 60 divides the temperature offset by the frequency reduction temperature margin to obtain the thermodynamic relative approximation. The dynamic billing engine 60 performs natural exponent calculation to calculate the exponential characteristic value of the thermodynamic relative approximation. The dynamic billing engine 60 multiplies the exponential characteristic value by the refrigeration elasticity base coefficient and adds the product to a preset fundamental constant to calculate and output the nonlinear thermodynamic marginal penalty multiplier. The formula for calculating the nonlinear thermodynamic marginal penalty multiplier is as follows:
[0092]
[0093] in, ,and The value of this ensures that the denominator is always positive. This represents the nonlinear thermodynamic marginal penalty multiplier; This represents the system's preset marginal cost penalty coefficient for cooling; This represents the predicted transient core temperature. This indicates the optimal energy efficiency temperature threshold; Indicates the critical extreme temperature for frequency reduction; Indicates a minimum positive integer to prevent overflow;
[0094] By constructing a nonlinear thermodynamic marginal penalty multiplier model, the system establishes a nonlinear mapping relationship between the thermodynamic state of hardware and the price of computing resources. When the temperature of the 31-core graphics processor approaches the hardware protection limit, the marginal cost of cooling shows a nonlinear increasing trend. The system utilizes the characteristics of the natural exponential function to accurately calculate the price penalty for resources on computing nodes with high heat dissipation consumption, realizing the transmission of physical cooling energy consumption of the data center to the cost at the micro level, and providing a direct quantitative indicator for subsequent billing intervention.
[0095] Example 5
[0096] Please refer to the appendix. Figure 6-7 Based on Example 4, the dynamic billing engine 60 performs real-time pricing and streaming deduction operations for the token by calculating the basic physical cost and nonlinear thermodynamic marginal penalty multiplier, and coordinates with the computing power scheduling module 20 to perform request circuit breaking and reverse reorganization scheduling based on billing thresholds.
[0097] The dynamic billing engine 60 obtains the basic physical cost of the inference request and the matching nonlinear thermodynamic marginal penalty multiplier. The dynamic billing engine 60 multiplies the basic physical cost with the nonlinear thermodynamic marginal penalty multiplier to calculate and output the real-time dynamic unit price of the single token generated by the inference request.
[0098] The dynamic billing engine 60 establishes an independent mapping of billing accumulators for each inference request in system memory;
[0099] When the graphics processor 31 completes the decoding output of each token, the dynamic billing engine 60 writes the corresponding real-time dynamic unit price into the corresponding billing accumulator to perform the points accumulation operation and obtain the cumulative consumption amount of the inference request. The dynamic billing engine 60 calls the lock-free atomic addition instruction at the system bottom layer to directly accumulate the real-time dynamic unit price into the memory segment corresponding to the billing accumulator according to the virtual address offset of the video memory. By using the lock-free atomic addition instruction to operate the billing accumulator, memory read and write data competition in multi-tenant concurrent billing scenarios is avoided, the context switching delay caused by the traditional thread lock mechanism is eliminated, and the strict time synchronization between the billing settlement action and the high-frequency decoding cycle of the graphics processor 31 is guaranteed.
[0100] The dynamic billing engine 60 extracts the reserved credit limit data of the tenant account corresponding to the inference request, and compares and verifies the cumulative consumption amount with the reserved credit limit data.
[0101] In response to the comparison result that the cumulative consumption amount is less than or equal to the reserved quota data, the dynamic billing engine 60 sends a data packet release instruction to the gateway module 10. The gateway module 10 receives the data packet release instruction, encapsulates the generated single token into a network response message and sends it to the client.
[0102] In response to the comparison result that the cumulative consumption amount is greater than the reserved quota data, the dynamic billing engine 60 triggers the overdue payment blocking mechanism. The dynamic billing engine 60 sends an overdue payment interruption command to the computing power scheduling module 20. The computing power scheduling module 20 receives the overdue payment interruption command, forcibly unloads the context data of the inference request from the execution queue of the continuous batch processing engine 21, interrupts the resource residence of the inference request in the graphics processor 31, and returns an abnormal status code message of quota exhaustion to the client.
[0103] Meanwhile, during the execution cycle of the matrix multiplication computation graph, the dynamic billing engine 60 extracts the tolerance marginal unit price parameter from the inference request parsed and transmitted by the gateway module 10. During the execution cycle of each matrix multiplication computation graph, the dynamic billing engine 60 performs a real-time numerical comparison between the real-time dynamic unit price and the tolerance marginal unit price parameter.
[0104] In response to the comparison result that the real-time dynamic unit price is greater than the tolerance marginal unit price parameter, the dynamic billing engine 60 triggers the price circuit breaker mechanism based on thermodynamic marginal cost. The dynamic billing engine 60 sends a scheduling-level billing interrupt signal to the computing power scheduling module 20. By sending an asynchronous interrupt flag, the system can safely intercept and suspend subsequent decoding tasks without destroying the integrity of the current computing instructions.
[0105] The computing power scheduling module 20 captures the scheduling-level billing interrupt signal. The computing power scheduling module 20 controls the continuous batch processing engine 21 to perform a context stripping operation on the inference request that generates an out-of-bounds price at the Kernel execution boundary at the end of the current decoding loop. In the batch construction of the next computing cycle, the continuous batch processing engine 21 removes the sequence identifier corresponding to the inference request that generates an out-of-bounds price and releases the occupied identifier of the video memory virtual address corresponding to the inference request in the global memory pool. By performing the context stripping operation, the system physically removes the inference request that generates an out-of-bounds cost from the high-energy-consuming concurrent execution flow.
[0106] The computing power scheduling module 20 swaps the stripped context data from the high-bandwidth memory 312 of the graphics processor 31 to the system main memory of the computing node 30. For inference requests that generate out-of-bounds prices, the computing power scheduling module 20 extracts the key-value cache tensor and the generated identifier sequence residing in the video memory of the graphics processor 31. The computing power scheduling module 20 serializes and packages the key-value cache tensor and the identifier sequence to generate a standard breakpoint resume status file. By explicitly extracting and packaging the key-value cache tensor, the time-consuming pre-filled matrix multiplication calculation is avoided from being re-executed on the target node for inference requests that generate out-of-bounds prices, ensuring the continuity of computation and low latency performance in the cross-node scheduling process. The computing power scheduling module 20 transfers the inference requests after generating the breakpoint resume status file to the economic queue in the system to wait.
[0107] The computing power scheduling module 20 monitors other computing nodes in a cold state within the data center. The computing power scheduling module 20 sends a thermodynamic status query message to the global cluster management node within the data center. The computing power scheduling module 20 receives the global node temperature routing table returned by the global cluster management node. The computing power scheduling module 20 parses the global node temperature routing table to obtain the network location information of idle graphics processor nodes whose core temperature is lower than the optimal energy efficiency temperature threshold. By parsing the global node temperature routing table, the communication blind spot of the local computing power scheduling module 20 in cross-physical chassis temperature detection is eliminated, ensuring the accuracy of subsequent task migration targets.
[0108] The computing power scheduling module 20 reschedules the inference requests in the economic queue to the idle graphics processor nodes to resume decoding execution. The computing power scheduling module 20 loads the breakpoint resume status file into the video memory of the idle graphics processor nodes and controls the idle graphics processor nodes to continue to execute autoregressive generation according to the identifier sequence.
[0109] By executing the above-mentioned flow steps, the token-based billing dynamic pricing system based on real-time computing power costs constructs a closed-loop control loop for thermal load between the underlying hardware thermodynamic state and the upper-layer business billing rules based on price leverage. The billing logic reacts to the batch processing combination of the computing power scheduling module 20, avoiding the risk of unlimited surge in computing costs caused by single node overheating, and realizing automated thermal load balancing of the global cluster based on price leverage.
Claims
1. A token-based dynamic pricing method for real-time computing power costs, implemented using a token-based dynamic pricing system for real-time computing power costs, characterized in that... Perform the following workflow steps: In step S100, the gateway module (10) concurrently receives inference requests from multiple tenants, the gateway module (10) parses and obtains the tolerance marginal unit price parameter in the inference request, and the gateway module (10) sends the inference request to the computing power scheduling module (20). In step S200, the computing power scheduling module (20) uses the continuous batch processing engine (21) to extract the context state information of each inference request. The computing power scheduling module (20) identifies the corresponding pre-filling stage operator or decoding stage operator based on the context state information. The computing power scheduling module (20) performs operator fusion operation to generate a matrix multiplication computation graph containing the inference request. The computing power scheduling module (20) sends the matrix multiplication computation graph to the instruction queue of the graphics processor (31) of the computing node (30) for execution. The specific implementation of the operator fusion operation is that the computing power scheduling module (20) aligns and splices the tensor dimensions of multiple different tenants in the same execution stage to construct a unified batch processing computation kernel function instruction. In step S300, during the cycle of the matrix multiplication computation graph executed by the graphics processor (31), the hardware telemetry module (40) activates the hardware performance counter in the graphics processor (31). At the end of the execution cycle, the hardware telemetry module (40) accesses the hardware performance counter and extracts the physical hardware macro-loss data of the execution cycle. The physical hardware macro-loss data includes the total number of active cycles of the streaming multiprocessor (311) and the total number of read and write bytes of the high-bandwidth memory (312). In step S400, synchronized with step S300, the thermodynamic prediction module (50) performs asynchronous physical state acquisition in the out-of-band channel. The thermodynamic prediction module (50) reads the discrete temperature measurement of the computing node (30) and the power consumption data of the graphics processor core through the substrate management controller (32). The thermodynamic prediction module (50) obtains the current timestamp difference. The thermodynamic prediction module (50) multiplies the power consumption data of the graphics processor core by the preset heat conversion coefficient to obtain the temperature change gradient. The temperature change gradient is multiplied by the current timestamp difference and then superimposed on the discrete temperature measurement to calculate and output the transient core temperature prediction value in the current execution cycle. Step S500, the dynamic billing engine (60) receives the physical hardware macro-loss data and the transient core temperature prediction value. The dynamic billing engine (60) extracts the context theoretical computational complexity features of each inference request in the current batch. The context theoretical computational complexity features include the theoretical floating-point operation count and theoretical video memory usage of each inference request. The dynamic billing engine (60) divides the resource allocation weight according to the context theoretical computational complexity features. The dynamic billing engine (60) maps the physical hardware macro-loss data to the corresponding micro-tenant according to the resource allocation weight and calculates the basic physical cost of each inference request. In step S600, the dynamic billing engine (60) matches the corresponding nonlinear thermodynamic penalty multiplier based on the transient core temperature prediction value. The dynamic billing engine (60) multiplies the basic physical cost by the nonlinear thermodynamic penalty multiplier and calculates and outputs the real-time dynamic unit price of the single token generated by each inference request. In step S700, the dynamic billing engine (60) extracts the tolerance marginal unit price parameter obtained in step S100 and performs a numerical comparison and verification between the real-time dynamic unit price and the corresponding tolerance marginal unit price parameter. In step S800, in response to the comparison result that the real-time dynamic unit price is less than or equal to the tolerance marginal unit price parameter, the dynamic billing engine (60) sends a streaming amount deduction instruction to the corresponding tenant account, the computing node (30) outputs the currently generated single token, and enters the next round of decoding loop operation; In step S900, in response to the comparison result that the real-time dynamic unit price is greater than the tolerance marginal unit price parameter, the dynamic billing engine (60) triggers the system circuit breaker mechanism. The dynamic billing engine (60) sends a request reorganization instruction to the computing power scheduling module (20). The computing power scheduling module (20) receives the request reorganization instruction and stops the operation of the inference request context that generates the out-of-bounds price in the current computing node (30). The computing power scheduling module (20) transfers the stopped inference request context to the waiting queue of the computing node in the system that is in a low-power cold state.
2. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S200, after the computing power scheduling module (20) performs the operator fusion operation, it also includes dynamic memory allocation of computing power, specifically implemented as follows: The computing power scheduling module (20) pre-divides a fixed-size memory block pool in the global memory of the computing node (30); The computing power scheduling module (20) extracts the context tensors of the video memory blocks allocated to different tenants as needed, and records the video memory virtual address offset of each video memory block; When constructing the matrix multiplication computation graph, the computing power scheduling module (20) directly hardcodes the virtual address offset of each memory block into the computation instructions, so that the computing nodes (30) can directly read in parallel according to the virtual address offset of the memory.
3. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S300, the hardware telemetry module (40) extracts the physical hardware macroscopic loss data of the execution cycle as follows: The hardware telemetry module (40) inserts an asynchronous event recording instruction into the execution instruction stream of the matrix multiplication computation graph. The hardware telemetry module (40) listens to the execution completion status of the instruction stream through the asynchronous event recording instruction. In response to the capture of the execution completion status, the hardware telemetry module (40) accesses the hardware performance counter register of the graphics processor (31). The hardware telemetry module (40) reads the specific performance counter register address pre-mapped in the memory-mapped input / output space through the peripheral component interconnect bus and extracts the total number of active cycles and the total number of read / write bytes.
4. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S400, when the thermodynamic prediction module (50) obtains the transient core temperature prediction value, the specific implementation of asynchronous physical state acquisition is as follows: The thermodynamic prediction module (50) stores discrete temperature measurements and graphics processor core power consumption data into a pre-set circular memory buffer. Thermodynamic prediction module (50) calls the high-precision event timer on the motherboard of computing node (30) to generate a local timestamp identifier the instant the discrete physical state data is received in the ring memory buffer; The thermodynamic prediction module (50) binds and stores the local timestamp identifier with discrete temperature measurements and graphics processor core power consumption data; The thermodynamic prediction module (50) extracts the latest local timestamp identifier from the circular memory buffer, subtracts the latest local timestamp identifier from the current high-frequency execution time, and calculates the timestamp difference.
5. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S500, the specific implementation of the dynamic billing engine (60) extracting contextual theoretical computational complexity features and dividing resource allocation weights is as follows: The dynamic billing engine (60) extracts the computation context tensor dimension data of the K inference requests executed concurrently in the current batch. The computation context tensor dimension data includes the input sequence length and hidden layer dimension of each inference request. The dynamic billing engine (60) multiplies the length of the input sequence of the inference request by the square of the hidden layer dimension of the inference request to obtain the theoretical number of floating-point operations of the corresponding inference request in the pre-filling stage; the dynamic billing engine (60) multiplies the length of the input sequence of the inference request by the hidden layer dimension of the inference request, and combines it with the network layer number feature quantity preset by the model to obtain the key-value cache tensor capacity of the corresponding inference request in the decoding stage, and uses the key-value cache tensor capacity as the theoretical memory usage of the corresponding inference request; When the dynamic billing engine (60) detects that the inference request is in the pre-filling stage, it assigns a first weight coefficient to the phase indicator factor; when the dynamic billing engine (60) detects that the inference request is in the decoding stage, it assigns a second weight coefficient to the phase indicator factor, and the first weight coefficient is greater than the second weight coefficient. The dynamic billing engine (60) calculates the resource allocation weight for each inference request based on the theoretical number of floating-point operations, the theoretical memory usage, and the phase indicator factor. The formula for calculating the resource allocation weight is as follows: in, Indicates the first The resource allocation weight of each inference request in the current batch; This indicates the total number of concurrent inference requests included in the current batch; Indicates the first The theoretical number of floating-point operations per inference request; Indicates the first The theoretical number of floating-point operations per inference request; Indicates the first Theoretical memory usage per inference request; Indicates the first Theoretical memory usage per inference request; This indicates the phase indicator factor.
6. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S500, the dynamic billing engine (60) calculates the specific implementation of the basic physical cost of each inference request as follows: The dynamic billing engine (60) obtains the system's preset microarchitecture base rate parameters, which include the base depreciation and power rate per unit stream multiprocessor active cycle, and the base power consumption rate per unit high-bandwidth memory read / write byte. The dynamic billing engine (60) multiplies the total number of active cycles with the baseline depreciation and the electricity rate, and multiplies the total number of read / write bytes with the baseline energy rate. The dynamic billing engine (60) adds the two multiplication results together to calculate the total macroscopic physical cost of the graphics processor (31) within the execution cycle. The dynamic billing engine (60) multiplies the total macro physical cost by the resource allocation weight corresponding to the inference request, and divides the result by the number of tokens generated by the inference request during the execution period to calculate and output the basic physical cost of the inference request. The formula for calculating the basic physical cost is as follows: in, Indicates the first The basic physical cost of generating a single token for each inference request within the execution cycle; Indicates the first The resource allocation weight of each inference request in a batch; The baseline depreciation and electricity rate represent the active cycle of a single stream multiprocessor. This represents the total number of active cycles for the streaming multiprocessor (311). The baseline energy consumption rate is expressed as a unit of high-bandwidth memory read / write bytes; This represents the total number of bytes read and written to the high-bandwidth memory (312); Indicates the first The equivalent number of tokens processed per inference request within the execution cycle, for the pre-population phase. The sum of the input sequence length and the number of the first output tokens; for the decoding phase, The number of tokens generated within a single execution cycle.
7. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S600, the specific implementation of the dynamic billing engine (60) matching the nonlinear thermodynamic penalty multiplier is as follows: The dynamic billing engine (60) reads the equipment serial number of the rack heat dissipation system, queries the pre-set hardware specification mapping table in the system based on the equipment serial number, extracts the rated power peak and cooling energy efficiency ratio curve recorded in the hardware specification mapping table, performs a division operation on the rated power peak and cooling energy efficiency ratio curve, and outputs the cooling elasticity base coefficient. The dynamic billing engine (60) subtracts the optimal energy efficiency temperature threshold from the transient core temperature prediction value to obtain the temperature offset. The dynamic billing engine (60) subtracts the transient core temperature prediction value from the frequency reduction critical extreme temperature and adds the anti-overflow minimum normal number to obtain the frequency reduction temperature margin. The dynamic billing engine (60) divides the temperature offset by the frequency reduction temperature margin to obtain the thermodynamic relative approximation, performs natural exponent calculation, calculates the exponential characteristic value of the thermodynamic relative approximation, multiplies the exponential characteristic value by the refrigeration elasticity base coefficient, and adds the product result to the preset base constant to calculate and output the nonlinear thermodynamic marginal penalty multiplier. The calculation formula of the nonlinear thermodynamic marginal penalty multiplier is as follows: in, ,and The value of this ensures that the denominator is always positive. This represents the nonlinear thermodynamic marginal penalty multiplier; This represents the system's preset marginal cost penalty coefficient for cooling; This represents the predicted transient core temperature. This indicates the optimal energy efficiency temperature threshold; Indicates the critical extreme temperature for frequency reduction; This represents a very small positive integer to prevent overflow, and The value of is chosen to ensure that the denominator is always positive.
8. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S800, the specific implementation of the dynamic billing engine (60) sending a streaming amount deduction instruction to the corresponding tenant account is as follows: The dynamic billing engine (60) establishes an independent mapping of billing accumulators for each inference request in system memory; When the graphics processor (31) completes the decoding output of each token, the dynamic billing engine (60) calls the lock-free atomic addition instruction at the system bottom layer to directly add the real-time dynamic unit price to the memory segment corresponding to the billing accumulator according to the virtual address offset of the video memory to perform the points accumulation operation and obtain the cumulative consumption amount of the inference request. The dynamic billing engine (60) extracts the reserved credit data of the tenant account corresponding to the inference request, compares and verifies the cumulative consumption amount with the reserved credit data, and sends a data packet release instruction to the gateway module (10) in response to the comparison result that the cumulative consumption amount is less than or equal to the reserved credit data.
9. The token-based dynamic pricing method for billing based on real-time computing power cost according to claim 1, characterized in that, In step S900, the specific implementation of the dynamic billing engine (60) triggering the system circuit breaker mechanism and the computing power scheduling module (20) suspending the inference request context operation residency is as follows: The computing power scheduling module (20) captures the scheduling-level billing interrupt signal and manipulates the continuous batch processing engine (21) to perform a context stripping operation on the inference request that generates an out-of-bounds price at the Kernel execution boundary at the end of the current decoding loop. In the batch building of the next computation cycle, the continuous batch processing engine (21) removes the sequence identifier corresponding to the inference request that generates an out-of-bounds price and releases the memory virtual address corresponding to the inference request from the global memory pool.
10. A method for dynamic pricing of token-based billing based on real-time computing power cost according to claim 9, characterized in that, The specific implementation of the computing power scheduling module (20) in transferring the context of aborted inference requests to the waiting queue of low-power cold state nodes in the system is as follows: For inference requests that generate out-of-bounds prices, the computing power scheduling module (20) extracts the key-value cache tensor and the generated identifier sequence residing in the graphics processor (31) memory, serializes and packages the key-value cache tensor and the identifier sequence, generates a standard breakpoint resume status file, and transfers the inference request after generating the breakpoint resume status file to the economic queue in the system to wait. The computing power scheduling module (20) sends a thermodynamic status query message to the global cluster management node in the data center, receives the global node temperature routing table returned by the global cluster management node, parses the global node temperature routing table, and obtains the network location information of idle graphics processor nodes whose core temperature is lower than the optimal energy efficiency temperature threshold. The computing power scheduling module (20) reschedules the inference requests in the economic queue to the idle graphics processor node to resume decoding execution, loads the breakpoint resume status file into the video memory of the idle graphics processor node, and controls the idle graphics processor node to continue to execute autoregressive generation according to the identifier sequence.
