Intelligent computing cloud platform based on GPU computing power consumption value and inference token value computing power measurement method and device

Abstract

This invention provides a method and apparatus for measuring computing power on an intelligent computing cloud platform based on GPU computing power consumption and inference token values. It relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart computing cloud technologies, and includes the following steps: Step S1, collecting task data, GPU runtime data, and model and engine parameters during inference on the intelligent computing cloud platform; Step S2, extracting first feature information, second feature information, and third feature information from the task data, GPU runtime data, and model and engine parameters; Step S3, performing token fitting computing power calculation based on the first feature information, GPU model, and third feature information to obtain a first computing power consumption value; Step S4, performing GPU computing power consumption benchmark calculation based on the first, second, and third feature information to obtain a second computing power consumption value; Step S5, determining a reliable computing power consumption value based on the first and second computing power consumption values. This invention can significantly improve the accuracy of computing power measurement on intelligent computing center cloud platforms.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart cloud technologies, specifically to a method and apparatus for measuring computing power on an intelligent computing cloud platform based on GPU computing power consumption and inference token values. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] With the rapid development of large-scale model training and wide-area intelligent computing cloud platforms, intelligent computing cloud platforms have begun to offer GPU computing power leasing and large-scale model inference calling services to enterprise and individual users. Users initiate model inference requests through interfaces, and the platform charges based on the number of input and output tokens generated by the call. Simultaneously, the intelligent computing cloud platform needs to monitor, calculate, and schedule the physical computing power consumption of fixed-model GPUs such as NVIDIA A100, H100, and RTX 4090 in real time. However, inaccurate computing power measurement can easily lead to unfair billing and resource imbalance on the intelligent computing cloud platform. Unlike metering in other areas such as electricity, traffic, and storage, computing power metering cannot be simply statistically analyzed and faces multiple challenges due to the multidimensionality, dynamism, and complexity of computing power.

[0008] It is evident that since the emergence of intelligent computing centers, the lack of accurate and reliable computing power measurement solutions has made computing power measurement a key bottleneck restricting the stable, efficient, and large-scale deployment of intelligent computing cloud platforms. This problem has always been an urgent issue to be solved in this field. Summary of the Invention

[0009] This invention provides a method and apparatus for measuring computing power in an intelligent computing cloud platform based on GPU computing power consumption and inference token values, in order to solve the problem of low accuracy in computing power measurement in existing intelligent computing center cloud platforms.

[0010] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for measuring computing power on an intelligent computing cloud platform based on GPU computing power consumption and inference token value, comprising: Step S1: Collect task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform; Step S2: Extract features from the task data, GPU runtime data, model and engine parameters to obtain first feature information, second feature information and third feature information respectively. The first feature information includes at least one of the following in the task data: number of input tokens, number of output tokens, batch size, context length and KV cache on / off status. The second feature information includes at least one of the following in the GPU runtime data: peak GPU computing power, real-time GPU utilization, GPU memory bandwidth utilization, GPU power consumption and effective GPU runtime. The third feature information includes at least one of the following in the model and engine parameters: number of model parameters, number of quantization bits and inference engine type. Step S3: Calculate the first computing power consumption value by using a decision tree-based regression model based on the first feature information, GPU model, and the third feature information to fit the token. Step S4: Calculate the GPU computing power consumption benchmark based on the first feature information, the second feature information, and the third feature information to obtain the second computing power consumption value; Step S5: Determine the trusted computing power consumption value during inference on the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value.

[0011] In one embodiment, step S3 includes: Step S3.1: Perform vector encoding on at least one of the input token count, the output token count, and the batch size to obtain a first vector representation; Step S3.2: Perform vector encoding on at least one of the model parameters and the quantization bit depth to obtain a second vector representation; Step S3.3: Perform vector encoding on the GPU model to obtain a third vector representation; Step S3.4: Merge the first vector representation, the second vector representation, and the third vector representation to obtain a fused vector representation; Step S3.5: Call the pre-trained decision tree-based regression model to process the fusion vector representation, and use the decision tree-based regression model to perform token fitting computing power calculation to obtain the first computing power consumption value.

[0012] In one embodiment, step S4 includes: Step S4.1: Determine the theoretical total number of computational operations based on the model parameter count, the number of input tokens, the number of output tokens, and the batch size; Step S4.2: Determine the inference engine optimization coefficients based on the inference engine type; Step S4.3: Determine the quantization precision correction coefficient based on the quantization bit depth; Step S4.4: Determine the dynamic management overhead of KVCache based on the KV cache switch status, the context length, and the batch size.

[0013] Step S4.5: Calculate the GPU computing power consumption benchmark based on the theoretical total number of computational operations, the GPU real-time utilization rate, the GPU memory bandwidth utilization rate, the inference engine optimization coefficient, the quantization precision correction coefficient, the KV Cache dynamic management overhead, and the GPU effective runtime, and obtain the second computing power consumption value.

[0014] In one embodiment, step S4.1 includes: Step S4.1.1: Calculate the total number of tokens based on the input token count and the output token count; Step S4.1.2: Determine the first total number of computational operations based on the total number of tokens, the number of model parameters, and the batch size; Step S4.1.3: Determine the theoretical total number of calculation operations based on the first total calculation operation.

[0015] In one embodiment, step S4.5 includes: Step S4.5.1: Multiply the total theoretical computational operands, the GPU real-time utilization rate, the GPU memory bandwidth utilization rate, the inference engine optimization coefficient, and the quantization precision correction coefficient to obtain the first product value; Step S4.5.2: Summing the first product value with the dynamic management overhead of the KV Cache to obtain the first sum value; Step S4.5.3: Determine the quotient of the first sum divided by the effective runtime of the GPU as the second computing power consumption value.

[0016] In one embodiment, step S5 includes: Step S5.1: Determine the relative deviation between the first computing power consumption value and the second computing power consumption value; Step S5.2: Obtain the dynamic deviation threshold; Step S5.3: Based on the comparison result between the relative deviation and the dynamic deviation threshold, determine the reliable computing power consumption value during inference on the intelligent computing cloud platform.

[0017] In one embodiment, step S5.2 includes: Step S5.2.1: Based on the GPU model and the third feature information, obtain multiple sets of historical computing power consumption values ​​with the same GPU model and the same model specification, wherein each set of historical computing power consumption values ​​includes a historical first computing power consumption value and a historical second computing power consumption value; Step S5.2.2: Determine the historical deviation standard deviation based on the multiple sets of historical computing power consumption values; Step S5.2.3: Determine the second product value obtained by multiplying the historical deviation standard deviation by a preset confidence coefficient as the reference deviation threshold; Step S5.2.4: Determine the maximum value between the reference deviation threshold and the preset deviation threshold as the dynamic deviation threshold.

[0018] In one embodiment, step S5.3 includes at least one of the following: Step S5.3.1: If the relative deviation is less than or equal to the dynamic deviation threshold, calculate the weighted average of the first computing power consumption value and the second computing power consumption value as the trusted computing power consumption value, and synchronize the trusted computing power consumption value, the computing power consumption value deviation report, and the computing power consumption value deviation verification status to the cloud platform billing engine and resource scheduler. Step S5.3.2: If the relative deviation is greater than the dynamic deviation threshold, the fusion vector representation and the second computing power consumption value are stored in the pending buffer pool, and when the number of pending items in the pending buffer pool reaches a preset value, the closed-loop correction of the decision tree-based regression model is triggered. The fusion vector representation is determined by at least one feature information used to determine the first computing power consumption value, and the corrected decision tree-based regression model is used to determine the new first computing power consumption value.

[0019] In one embodiment, step S1 further includes at least one of the following: Step S1.1: Perform time alignment preprocessing on the task data, GPU running data, model and engine parameters using microsecond-level timestamps, and remove queuing periods, cold start periods and GPU Idle periods; Step S1.2: Perform interpolation or default value filling preprocessing on the missing parameters in the task data, GPU running data, model and engine parameters; Step S1.3: Perform outlier filtering preprocessing on the task data, GPU running data, model and engine parameters, wherein the outliers are processed separately or excluded from this calculation process.

[0020] Secondly, the present invention also provides a computing power metering device for an intelligent computing cloud platform based on GPU computing power consumption value and inference token value, comprising: The data acquisition module is used to collect task data, GPU runtime data, and model and engine parameters during inference on the intelligent computing cloud platform. The feature extraction module is used to extract features from the task data, GPU running data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, respectively. The first feature information includes at least one of the following in the task data: number of input tokens, number of output tokens, batch size, context length, and KV cache on / off status. The second feature information includes at least one of the following in the GPU running data: peak GPU computing power, real-time GPU utilization, GPU memory bandwidth utilization, GPU power consumption, and effective GPU runtime. The third feature information includes at least one of the following in the model and engine parameters: number of model parameters, quantization bit depth, and inference engine type. The first computing power calculation module is used to perform token fitting computing power calculation based on the first feature information, GPU model and the third feature information to obtain the first computing power consumption value. The second computing power calculation module is used to perform GPU computing power consumption benchmark calculation based on the first feature information, the second feature information and the third feature information to obtain the second computing power consumption value. The determination module is used to determine the trusted computing power consumption value during inference of the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value.

[0021] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the computing power measurement method based on GPU computing power consumption value and inference token value of the intelligent computing cloud platform as described in the first aspect above.

[0022] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the computing power measurement method for intelligent computing cloud platforms based on GPU computing power consumption value and inference token value as described in the first aspect above.

[0023] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the computing power measurement method for intelligent computing cloud platforms based on GPU computing power consumption value and inference token value as described in the first aspect above.

[0024] In this invention, step S1 involves collecting task data, GPU runtime data, and model and engine parameters during inference on an intelligent computing cloud platform; step S2 involves extracting features from the task data, GPU runtime data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, wherein the first feature information includes at least one of the following in the task data: input token count, output token count, batch size, context length, and KV cache on / off status; the second feature information includes the following in the GPU runtime data: GPU peak computing power, GPU real-time utilization, GPU memory bandwidth utilization, GPU power consumption, and GPU... At least one of the effective runtime, the third feature information includes at least one of the model parameter quantity, quantization bit depth, and inference engine type in the model and engine parameters; Step S3, calculate the first computing power consumption value by performing a token fitting computing power measurement based on the first feature information, GPU model, and the third feature information using a decision tree-based regression model; Step S4, calculate the second computing power consumption value by performing a GPU computing power consumption benchmark measurement based on the first feature information, the second feature information, and the third feature information; Step S5, determine the trusted computing power consumption value during inference on the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value. In this way, multiple feature information is first obtained by extracting features from the task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform. Then, based on these feature information, token fitting computing power measurement and GPU computing power consumption benchmark measurement are performed to obtain the first computing power consumption value and the second computing power consumption value. The reliable computing power consumption value during inference on the intelligent computing cloud platform is determined by combining the first computing power consumption value and the second computing power consumption value. This not only breaks the limitations of the traditional single computing power measurement path and realizes the two-way mapping between business token and physical hardware, but also solves the problem of deviation verification between the two types of measurement results, thereby solving the problem of low accuracy of computing power measurement in the existing technology. Attached Figure Description

[0025] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a computing power measurement method for an intelligent computing cloud platform based on GPU computing power consumption and inference token value, provided by the present invention. Figure 2 This is a flowchart of another intelligent computing cloud platform computing power measurement method based on GPU computing power consumption value and inference token value provided by the present invention; Figure 3 This is an overall flowchart of the intelligent computing cloud platform provided by the present invention, based on GPU computing power consumption value and inference token value; Figure 4 This is the data time alignment and parameter acquisition mapping diagram provided by the present invention; Figure 5 This is the logic diagram for dual-path measurement and deviation determination provided by the present invention; Figure 6 This is a diagram of the model incremental correction and anti-forgetting mechanism provided by the present invention; Figure 7 This is a structural diagram of a computing power metering device based on GPU computing power consumption value and inference token value provided by the present invention for an intelligent computing cloud platform; Figure 8 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0027] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The “computing power” mentioned in this invention refers to: the ability of computer equipment or computing / data center to process information; the ability of computer hardware and software to work together to perform a certain computing requirement; the computing power to achieve the target result output by processing information data; and a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, mainly providing services to society through computing power infrastructure.

[0029] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0030] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capability of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capability.

[0031] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), while the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB). Disaster recovery ratio is an important indicator of security and reliability.

[0032] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0033] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0034] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0035] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0036] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0037] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0038] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0039] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0040] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0041] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0042] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0043] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0044] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0045] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0046] The "large language model" mentioned in this invention refers to a large language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language, is trained with a large amount of text data, and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0047] The “GPU computing power consumption value” mentioned in this invention refers to the actual computing power of the GPU hardware occupied by the model during task execution (training / inference) per unit time, measured in FLOPS / TOPS. It integrates accuracy, video memory, bandwidth and utilization, and is a core indicator for computing power accounting, cost billing and resource scheduling.

[0048] The “Token” mentioned in this invention refers to the smallest information unit that a large model processes, also known as a word unit. It is the core carrier that connects human language and machine digital encoding and can be used as a basic discrete unit of characters, word fragments, symbols, or images / audio.

[0049] The “computing power running task” mentioned in this invention refers to a specific workload or job executed on computing power resources that requires a certain amount of computing power support, usually involving complex data processing, numerical calculation, model training or simulation scenarios.

[0050] The "task data" mentioned in this invention refers to task-level data (also known as task-level data) when the model actually processes computing power to run tasks, which is the content that the model "eats" and "outputs". Commonly included are: input content (such as: text prompt, images, audio, video frames, feature vectors, user request parameters), output content (such as: classification results, detection boxes, generated text, embedding, response probability, latency), and batch information (such as: batch size, sequence length, image resolution, number of tokens).

[0051] The "GPU runtime data" mentioned in this invention refers to real-time status indicators of GPU hardware during inference, which are hardware-level monitoring data. Common indicators include: GPU utilization, memory usage / free, temperature, power consumption, frequency, memory bandwidth, PCIe bandwidth, kernel execution time, and stream processor activity.

[0052] The "model and engine parameters" mentioned in this invention refer to the relevant parameters of the model and engine used during inference on the intelligent computing cloud platform. These parameters may include the model's inherent weights, biases, and structural parameters, as well as the configuration parameters used by the inference engine to control how the model runs.

[0053] The "first feature information" mentioned in this invention refers to one or more feature data obtained by feature extraction of task data that can be used in the subsequent computing power measurement process.

[0054] The "second feature information" mentioned in this invention refers to one or more feature data obtained by feature extraction of GPU running data that can be used in the subsequent computing power measurement process.

[0055] The "third feature information" mentioned in this invention refers to one or more feature data obtained by feature extraction of model and engine parameters that can be used in the subsequent computing power measurement process.

[0056] The “input token count” mentioned in this invention refers to the total number of tokens that the text, image, and other data input to the pre-trained neural network model during the model inference process are converted into tokens that the model can recognize after preprocessing (such as word segmentation and feature encoding). It is a core indicator that represents the scale of input data and computational volume of the inference task and directly affects the basic computing power consumption of a single inference.

[0057] The “output token count” mentioned in this invention refers to the total number of tokens corresponding to the output results (such as generated text, predicted labels, feature vectors, etc.) after the model completes the inference calculation and is decoded and converted. It reflects the output scale of the inference task. The number of tokens is related to the model decoding strategy and the output length limit, and will also affect the computing power consumption in the later stages of inference.

[0058] The "batch size" mentioned in this invention refers to the number of samples that are input into the model for computation in parallel in a single model inference task by the intelligent computing cloud platform (i.e., the number of input samples processed in batches in a single inference). It is a key parameter that affects GPU utilization, inference throughput and computing power consumption. The larger the batch size, the higher the overall computing power consumption of a single inference task.

[0059] The "context length" mentioned in this invention refers to the maximum amount of text that the model can process and reference simultaneously in a single inference, measured in tokens. It is the model's "working memory," determining the length of history that can be referenced in a dialogue, the maximum size of a single batch of documents, and the available length for generating responses.

[0060] The "KV cache switch status" mentioned in this invention refers to the status indicator (enabled / disabled) of whether the KV cache (Key-Value Cache) optimization strategy is enabled during the model inference process; wherein the KV cache is used to cache the Key and Value information generated during the inference process, reducing repeated calculations and thus reducing computing power consumption, and its switch status directly affects the actual computing power overhead and inference speed of a single inference.

[0061] The "GPU peak computing power" mentioned in this invention refers to the maximum computing power that GPU hardware used to perform model inference tasks in an intelligent computing cloud platform can provide under ideal working conditions (no resource bottlenecks, full load operation). It is usually measured in FLOPS (floating-point operations per second) and is a benchmark indicator for measuring the computing performance of GPU hardware, used to characterize the maximum potential of GPU to process model inference tasks.

[0062] The "GPU real-time utilization rate" mentioned in this invention refers to the actual proportion of GPU computing cores (stream processors) occupied at a certain moment during the model inference process, reflecting the real-time load of the GPU during the inference execution process, and its value range is 0%-100%. The higher the utilization rate, the more fully the GPU computing resources are occupied by the model inference task, which is directly related to the computing power consumption efficiency of a single inference.

[0063] The “GPU memory bandwidth utilization rate” mentioned in this invention refers to the ratio of the data transfer rate of GPU memory (video memory) to the theoretical maximum transfer rate of the memory during model inference, reflecting the busy level of memory data reading and writing. Input data, model parameters, intermediate calculation results, etc. during model inference all require memory bandwidth, and its utilization rate directly affects the stability of inference speed and computing power consumption.

[0064] The “GPU power consumption” mentioned in this invention refers to the electrical energy consumed by the GPU hardware per unit time during model inference, usually measured in watts (W). It is an important indicator of the resource investment of the GPU during inference operation. Its power consumption is related to GPU utilization, inference task complexity, and computational accuracy, and can indirectly reflect the actual computing power investment of the GPU.

[0065] The “GPU effective runtime” mentioned in this invention refers to the actual time consumed from the start of the model inference task (GPU starts inference computation) to the end of the execution (GPU completes inference and outputs results). It does not include the time consumed by GPU idle waiting, task queuing, and data preprocessing / postprocessing. It is the core time dimension indicator for accurately calculating the actual computing power consumption of GPU during the model inference process.

[0066] The “model parameter quantity” mentioned in this invention refers to the total number of learnable parameters such as weights and biases that a pre-trained neural network model used to perform inference tasks in an intelligent computing cloud platform learns and stores during the training process. It is a core indicator that characterizes the complexity of the model structure and its feature expression ability. The quantity of these parameters directly determines the basic computational load and computing power consumption scale during model inference.

[0067] The "quantization bit width" mentioned in this invention refers to the number of binary bits used when quantizing model parameters and intermediate calculation results during model inference. It is a key configuration parameter for controlling inference accuracy and computing power consumption. Common quantization bit widths include FP32 (32-bit floating point), FP16 (16-bit floating point), INT8 (8-bit integer), etc. The lower the quantization bit width, the lower the inference computing power consumption, but it may affect the inference accuracy.

[0068] The "inference engine type" mentioned in this invention refers to the specific type of software framework / tool ​​used in intelligent computing cloud platforms to load pre-trained models, schedule hardware resources, and perform forward inference computation of the model. It is an important configuration that affects inference efficiency and computing power consumption. Common inference engine types include TensorRT, ONNX Runtime, TorchServe, Triton, etc. Different types of inference engines have different optimization strategies (such as operator fusion and graph optimization), which will directly lead to differences in computing power consumption of the same model during inference.

[0069] The “GPU model” mentioned in this invention refers to the specific specification identifier of the GPU hardware used to perform model inference tasks in the intelligent computing cloud platform. It is associated with the core hardware parameters of the GPU (such as the number of computing cores, peak computing power, memory capacity, memory bandwidth, etc.) and performance level. Different GPU models have different computing power performance and resource configurations, which directly affect the computing power consumption efficiency and inference speed of model inference. It is an important reference parameter for token fitting computing power measurement.

[0070] The "Token Fitting Computing Power Measurement" described in this invention refers to the process of obtaining a first computing power consumption value that characterizes the theoretical computing power consumption of model inference by establishing a mapping relationship between the number of tokens and computing power consumption based on the first feature information of the model inference task (number of input tokens, number of output tokens, etc.), GPU model and third feature information (number of model parameters, number of quantization bits, etc.). Its core is to rely on the semantic features of the task, model configuration and hardware performance benchmarks, eliminate the interference of instantaneous hardware fluctuations, and realize standardized and traceable measurement of computing power consumption across models and hardware scenarios.

[0071] The "first computing power consumption value" mentioned in this invention refers to a quantitative value that characterizes the computing power consumption of the model inference theory, obtained by measuring the computing power through token fitting. It is calculated by fitting the first feature information (number of input tokens, number of output tokens, batch size, etc.), GPU model, and third feature information (number of model parameters, number of quantization bits, etc.). It is not affected by the instantaneous fluctuation of GPU hardware operation and is the core reference value that reflects the computing power requirements of the model inference theory under the correlation of task semantics, model configuration, and hardware performance benchmark.

[0072] The “GPU computing power consumption benchmark calculation” described in this invention refers to a calculation method that uses actual GPU hardware operation data as the core basis, combines task characteristics (first feature information) and model engine parameters (third feature information), and calculates the actual computing power consumed by the GPU at the physical level during model inference to obtain the second computing power consumption value.

[0073] The "second computing power consumption value" mentioned in this invention refers to a quantitative value that characterizes the actual physical computing power consumption of the GPU hardware during model inference, obtained by measuring the GPU computing power consumption benchmark. This second computing power consumption value is calculated by comprehensively utilizing the first feature information, GPU operating data (utilization rate, power consumption, effective runtime, etc.), and the third feature information. It can truly reflect the actual load and computing power input of the hardware when executing this inference task, and can be used as a measured benchmark value for cross-verification with the theoretical computing power consumption.

[0074] The "trusted computing power consumption value" mentioned in this invention refers to the final quantified value of computing power consumption that is highly accurate, robust, and traceable, determined by combining the theoretical first computing power consumption value obtained from the Token fitting computing power measurement and the measured second computing power consumption value obtained from the GPU computing power consumption benchmark measurement, after fusion, verification, and correction.

[0075] The "decision tree-based regression model" described in this invention refers to a regression prediction model (such as the LightGBM regression model) built on a decision tree-based structure. This model can fit the nonlinear mapping relationship between features and target values ​​based on the input multidimensional feature vector through multi-level feature partitioning and branch decision-making, and finally output a continuous prediction result.

[0076] The "theoretical total number of computational operations" mentioned in this invention refers to the total number of floating-point operations or arithmetic operations required to complete this model inference under an ideal computing environment, based solely on business and model parameters such as model structure, number of model parameters, number of input and output tokens, and batch size.

[0077] The "inference engine optimization coefficient" mentioned in this invention refers to a dimensionless coefficient used to characterize the degree of optimization of computing power consumption by different inference engines. Different inference engines (such as TensorRT, ONNX Runtime, Transformers, vLLM, TGI, etc.) have different optimization capabilities in operator fusion, memory reuse, kernel optimization, asynchronous execution, pipeline scheduling, etc., resulting in significant differences in the actual utilization efficiency of computing power. Therefore, this coefficient is used to correct the computing power measurement results to make them more closely reflect the actual operating efficiency.

[0078] The "quantization precision correction coefficient" mentioned in this invention refers to a normalized correction coefficient used to reflect the impact of different numerical precisions on computing power consumption and computing efficiency.

[0079] The "KV Cache dynamic management overhead" mentioned in this invention refers to the additional computing power and memory access overhead generated during the dynamic maintenance operations of KV cache allocation, reading, writing, updating, copying, eviction, and memory defragmentation during the large model inference process.

[0080] The "dynamic deviation threshold" mentioned in this invention refers to a non-fixed deviation judgment threshold that can be adaptively adjusted based on historical operating data and the current task environment, used to measure whether the deviation between the first computing power consumption value and the second computing power consumption value is within a reasonable and reliable range.

[0081] The "billing engine" mentioned in this invention refers to a service module in the intelligent computing cloud platform that is responsible for calculating costs, generating bills, and settling fees based on actual usage.

[0082] The "resource scheduler" mentioned in this invention refers to the core module in the intelligent computing cloud platform that is responsible for the unified allocation, scheduling, and management of hardware resources such as GPUs / video memory / computing nodes.

[0083] It is important to emphasize that the difficulty of measuring computing power is far greater than that of measuring other fields such as electricity, flow, and storage in existing technologies. The core reason is the multidimensionality, dynamism, and complexity of computing power.

[0084] For example, the difficulty of measuring computing power can be understood by comparing measurement methods in different fields: For electricity metering, only one core indicator is needed—how much work was done (kilowatt-hours). Due to the stable and predictable nature of work done, accurate measurement can be achieved with a single electricity meter, and its consistency and reliability can be guaranteed by physical laws. For network traffic metering, it is only necessary to determine how much data (bytes) was transmitted. Although the transmission may be affected by network conditions, the data packets are countable, the protocol is standard, the metering is relatively direct, and the metering results are reliable. As for computing power measurement, the purpose of measurement is "how many effective calculations have been completed". Measuring computing power is like a black box. Data is input into the black box, and the black box outputs results, but the process in between is extremely complex and difficult to measure with a single, universal indicator. Because the measurement process is complex, the more factors that need to be considered, the higher the difficulty of measurement.

[0085] Based on the applicant's repeated testing and experiments on the intelligent computing center's cloud platform, the following are some of the core reasons why computing power measurement is extremely difficult: 1. Computing power is "multidimensional" and "heterogeneous".

[0086] Computing power, since it is not a simple physical quantity, cannot be compared with homogeneous indicators such as electricity or network traffic.

[0087] For example, computing power can be provided by different types of accelerator cards, such as GPUs, NPUs, TPUs, and DPUs. Different accelerator cards have completely different architectures and are best suited for different tasks. Using floating-point operations per second, memory bandwidth, or memory capacity to measure dynamically changing computing power utilization is as unreasonable as using only "load capacity" to measure the overall performance of a truck, and it cannot be used to measure the performance of other types of vehicles (such as sports cars and sedans).

[0088] At the same time, due to the different types of computing tasks, such as scientific computing, AI model training, video rendering, and database querying, the "effective computing power" that the same accelerator card can provide can vary greatly when executing different types of computing tasks. For example, an accelerator card that scores highly in matrix operations may perform only moderately when handling complex logical operations and may not achieve the same effect as when handling matrix operations.

[0089] For the reasons mentioned above, the field of intelligent computing center cloud platforms still lacks a standard unit that is universally recognized by the industry as "kilowatt-hour" and can measure "effective computing workload" across platforms and tasks.

[0090] 2. The performance indicators of computing power consumption are "dynamic" and "fluctuating".

[0091] During the execution of computing power tasks, the computing power output by the intelligent computing center cloud platform is not a constant value, but a fluctuating value, which is affected by many dynamic factors.

[0092] For example, the hardware is affected by the software stack and drivers. Different operating systems, drivers, compilers, and math libraries can greatly influence the actual performance of the hardware; a minor driver update can lead to a significant performance improvement or decrease.

[0093] Computing power is also affected by workload characteristics. Even for the same task, the size, shape, and precision (such as FP32, FP16, INT8 precision) of the input data can lead to huge differences in computing efficiency.

[0094] The influence of the system environment, such as memory bandwidth, PCIe bandwidth, storage I / O, heat dissipation conditions, and other concurrent tasks, can compete for system resources, leading to unstable computing power output.

[0095] Furthermore, the computing power output suffers from the "barrel effect," meaning that the final performance of the intelligent computing center cloud platform depends on the slowest component in the entire system (memory, cache, bus, etc.), rather than just the peak computing power of the accelerator card.

[0096] 3. The “externalities (or intrusiveness)” and “costs (or consumption)” of the process of measuring computing power.

[0097] Measuring computing power is a complex systemic problem that relies on external devices or services, such as performance counters, instrumentation, and proxies, that are unrelated to the execution of computing power tasks.

[0098] Among them, performance counters are hardware configurations of the intelligent computing center and are an important source of data for resource parameters. However, performance counters themselves incur overhead when collecting or reading parameters, requiring a certain amount of computing power. Moreover, for resource parameters that are too low-level (such as the number of instructions or cache hit rate), it is difficult to directly map them to "effective computing workload," and it is necessary to additionally calculate the relationship between the parameter and "effective computing workload."

[0099] For instrumentation and proxying, a common practice is to run a standard benchmark program (such as MLPerf) as a proxy to estimate the system's computing power. However, this is a sampling method and cannot represent all real-world tasks, so its accuracy is generally low. Moreover, running benchmarks itself incurs overhead, requiring a certain amount of computing power.

[0100] Furthermore, it is difficult to measure computing power online or without loss, meaning that it is impossible to obtain an absolutely accurate computing power value in real time and without interrupting operations, just like reading an electricity meter.

[0101] 4. The ease with which fraud and cheating occur is the core issue of "computing power measurement".

[0102] In scenarios such as computing power leasing and trading, computing power measurement is directly related to economic interests, and therefore faces serious challenges of cheating.

[0103] For example, at the software level, computing power providers can deliberately provide under-optimized drivers or libraries, or obtain false high scores by "tuning" system parameters in benchmark tests, while performance drops sharply in actual customer tasks, and users cannot obtain services corresponding to the real computing power.

[0104] Hardware fraud, such as using inferior materials (e.g., disguising consumer-grade graphics cards as professional cards) and overselling virtualization resources, can lead to insufficient computing power due to hardware fraud.

[0105] In existing technologies, the verification cost is extremely high. In order to verify whether the computing power provided by the supplier is real, the demand side may need to deploy a complex test environment, which is costly and impractical.

[0106] 5. Fragmentation of standards and ecosystems.

[0107] There is a lack of an authoritative body for computing power measurement: Unlike the International Bureau of Weights and Measures (BIPM) which defines "meter" and "second", the computing power field lacks a globally recognized authoritative body to develop and certify measurement standards.

[0108] 6. Measuring cloud-based resources is far more difficult than measuring physical resources.

[0109] Physical resource metering relies on actual hardware measurements (fixed caliber), while cloud resource metering considers virtual allocation plus actual consumption (elastic and dynamic). The challenge lies in the metering granularity, elastic adaptation, and allocation rules.

[0110] In summary, the great difficulty in measuring computing power stems from the need to use one or a few simple, static numbers to describe the effective output of a highly complex, multidimensional, dynamic system that is strongly dependent on the environment under a specific task.

[0111] It is evident that computing power measurement is not merely a technical issue, but a complex systems engineering problem involving hardware architecture, software engineering, metrology, economics, and trust mechanisms. As AI computing power becomes a new productive force, resolving the issue of computing power measurement is crucial for building a fair and efficient computing power market and promoting the development of the digital economy. This invention precisely addresses this pressing problem in the field of computing power.

[0112] Please see Figure 1 , Figure 1 This is a flowchart of a computing power measurement method for an intelligent computing cloud platform based on GPU computing power consumption and inference token value, provided by the present invention. Figure 1 As shown, the method includes: Step S1: Collect task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform.

[0113] When the intelligent computing cloud platform of this invention is used for large-scale model inference services, the computational power consumption of model inference is greatly affected by multiple factors such as input scale, model structure, engine configuration, and runtime optimization. If billing is based solely on the number of calls, task duration, or simple resource percentage, it will lead to a mismatch between billing and actual resource consumption, resulting in unfairness among users and inaccurate cost accounting for the platform. If computational power metering cannot provide the intelligent computing cloud platform with a fine-grained profile of task computational power, the platform will be unable to dynamically allocate heterogeneous resources such as GPUs / NPUs reasonably, easily leading to resource idleness or excessive contention. For intelligent computing cloud platform operators, accurate computational power metering is the foundation for implementing pay-as-you-go billing, flexible billing, package pricing, cost sharing, and budget control. For model service providers and users, it allows for a clear understanding of inference costs, optimization of model structure and calling strategies, and reduction of overall computational overhead.

[0114] Therefore, intelligent computing cloud platforms have extremely high requirements for the accuracy of computing power measurement. Any deviation in computing power measurement will lead to revenue loss or waste of computing resources for the intelligent computing cloud platform operator. Improving the accuracy of computing power measurement of intelligent computing cloud platforms is an urgent problem to be solved.

[0115] In this embodiment of the invention, there is no limitation on the specific task type of the intelligent computing cloud platform model inference. For example, it can be a natural language processing inference task, a computer vision inference task, a speech and audio inference task, etc.

[0116] It should be noted that the specific content included in the task data, GPU running data, and model and engine parameters in this embodiment of the invention is not limited and can be flexibly adjusted according to the application scenario.

[0117] The beneficial technical effects achieved by step S1 are as follows: By synchronously collecting task data, GPU running data, and model and engine parameters during the model inference process, it is possible to comprehensively obtain information on business, hardware, and configuration dimensions that affect inference computing power consumption. This avoids distortion in subsequent computing power measurement due to a single data dimension, and lays a complete data foundation for achieving refined and highly reliable computing power measurement. At the same time, it is compatible with cloud platform scenarios with multi-tenancy, multi-model, and heterogeneous deployment.

[0118] Step S2: Extract features from the task data, GPU runtime data, model and engine parameters to obtain first feature information, second feature information and third feature information respectively. The first feature information includes at least one of the following in the task data: number of input tokens, number of output tokens, batch size, context length and KV cache on / off status. The second feature information includes at least one of the following in the GPU runtime data: peak GPU computing power, GPU real-time utilization, GPU memory bandwidth utilization, GPU power consumption and effective GPU runtime. The third feature information includes at least one of the following in the model and engine parameters: number of model parameters, number of quantization bits and inference engine type.

[0119] In one possible implementation, regular expressions can be used to extract features from task data, GPU runtime data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, respectively.

[0120] In another possible implementation, a pre-trained feature extraction model can be used to extract features from task data, GPU running data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, respectively.

[0121] It should be noted that in intelligent computing cloud platforms, task data, GPU running data, and model and engine parameters usually contain diverse related feature information. Different application scenarios may have personalized differences in their requirements for the categories of feature information. Therefore, feature extraction can be performed on task data, GPU running data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, which will be used in the subsequent process of measuring computing power consumption.

[0122] The beneficial technical effects achieved in step S2 are as follows: First, feature information is obtained by extracting features from task data. The number of input tokens and output tokens directly reflects the computational volume of the inference task; batch size and context length characterize the concurrency and computational complexity of the task; and the KV cache on / off status reflects the differences in computational overhead brought about by inference optimization strategies, thus accurately depicting the computational requirements of the inference task itself. Second, feature information is obtained by extracting features from GPU runtime data. GPU peak computing power, real-time utilization, and memory bandwidth utilization reflect the actual hardware load; and GPU power consumption and effective runtime quantify computational input from a physical execution perspective, thus truly reflecting the actual computational consumption state of the GPU during inference. Third, feature information is obtained by extracting features from model and engine parameters. The number of model parameters characterizes the structural complexity of the model itself; the number of quantization bits reflects the impact of computational precision on computational overhead; and the inference engine type reflects the optimization differences of different execution frameworks, thereby eliminating interference from differences in model and runtime configurations on computational power measurement.

[0123] Step S3: Calculate the first computing power consumption value by using a decision tree-based regression model to fit the token based on the first feature information, GPU model, and third feature information.

[0124] In one possible implementation, the first computing power consumption value can be obtained by linear weighted fitting based on task characteristics such as the number of input tokens, the number of output tokens, the batch size, and the context length, combined with the hardware computing power coefficient corresponding to the GPU model, the number of model parameters, and the model coefficient corresponding to the number of quantization bits.

[0125] In another possible implementation, a computing power benchmark mapping table indexed by Token size, GPU model, model parameter quantity, and quantization bit depth can be pre-built. Based on the current feature information, the corresponding first computing power consumption value can be obtained by matching or interpolating in the mapping table.

[0126] The beneficial technical effects achieved by step S3 are as follows: Based on task characteristics, GPU model and model engine parameters, token fitting computing power measurement can be carried out. It can establish the mapping relationship between task volume and computing power consumption from the theoretical calculation level, and is not affected by hardware instantaneous fluctuations. It can obtain a stable and reliable first computing power consumption value, realize standardized measurement of computing power consumption across models and hardware scenarios, and ensure the consistency and traceability of computing power measurement.

[0127] Step S4: Calculate the GPU computing power consumption benchmark based on the first feature information, the second feature information and the third feature information to obtain the second computing power consumption value.

[0128] In one possible implementation, the baseline computing power is obtained based on the number of input tokens, the number of output tokens, the number of model parameters, and the number of quantization bits. Then, the second computing power consumption value is obtained by combining GPU runtime data.

[0129] In another possible implementation, time-series data such as GPU real-time utilization and GPU power consumption can be integrated over time to obtain the cumulative computing power consumption as the second computing power consumption value.

[0130] The beneficial technical effects achieved by step S4 are as follows: By combining task characteristics, hardware operation characteristics and model engine parameters to perform benchmark calculation of GPU computing power consumption, it is possible to fully rely on the actual operation data of the GPU to calculate the actual computing power consumption from the perspective of physical execution, obtain the second computing power consumption value, effectively correct the deviation in theoretical fitting calculation, make the computing power measurement results fit the actual operation status of the cloud platform, and improve the authenticity of computing power measurement.

[0131] Step S5: Determine the trusted computing power consumption value during inference on the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value.

[0132] It should be noted that in the intelligent computing cloud platform, the first computing power consumption value is a business-oriented computing power consumption value obtained by token fitting computing power measurement based on the first feature information, GPU model, and third feature information. It reflects the theoretical computing power demand under ideal conditions and does not consider the actual fluctuations in hardware operation. The second computing power consumption value is a hardware-oriented computing power consumption value obtained by GPU computing power consumption benchmark measurement based on the first, second, and third feature information. It reflects the actual execution overhead of physical hardware. The two calculation bases are different, and GPUs have dynamic frequency adjustment and resource contention. The inference process includes non-computational overhead. In addition, inference engine optimization, instantaneous hardware fluctuations, and sampling errors can all cause the measured value to deviate from the theoretical fitted value. Therefore, the intelligent computing cloud platform urgently needs a more accurate and reliable computing power measurement method.

[0133] In one possible implementation, a reasonable range can be formed based on the first computing power consumption value and the second computing power consumption value, and the midpoint or constraint mean within the range can be taken as the reliable computing power consumption value.

[0134] In one possible implementation, one of the computing power consumption values, the first and the second, can be used as a benchmark, and the other computing power consumption value can be used to correct for the deviation.

[0135] The beneficial technical effects achieved by step S5 are as follows: by fusing the first computing power consumption value obtained by theoretical fitting with the second computing power consumption value obtained by hardware measurement, abnormal data is eliminated and the error of a single measurement method is offset through a dual verification mechanism. The final determined reliable computing power consumption value has both stability and authenticity, which can realize the accurate measurement of model inference computing power and provide objective and reliable data support for cloud platform billing and settlement, resource scheduling and cost control.

[0136] In this invention, step S1 involves collecting task data, GPU runtime data, and model and engine parameters during inference on an intelligent computing cloud platform; step S2 involves extracting features from the task data, GPU runtime data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, wherein the first feature information includes at least one of the following in the task data: input token count, output token count, batch size, context length, and KV cache on / off status; the second feature information includes the following in the GPU runtime data: GPU peak computing power, GPU real-time utilization, GPU memory bandwidth utilization, GPU power consumption, and GPU... At least one of the effective runtime, the third feature information includes at least one of the model parameter quantity, quantization bit depth, and inference engine type in the model and engine parameters; Step S3, calculate the first computing power consumption value by performing a token fitting computing power measurement based on the first feature information, GPU model, and the third feature information using a decision tree-based regression model; Step S4, calculate the second computing power consumption value by performing a GPU computing power consumption benchmark measurement based on the first feature information, the second feature information, and the third feature information; Step S5, determine the trusted computing power consumption value during inference on the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value. In this way, multiple feature information is first obtained by extracting features from the task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform. Then, based on these feature information, token fitting computing power measurement and GPU computing power consumption benchmark measurement are performed to obtain the first computing power consumption value and the second computing power consumption value. The reliable computing power consumption value during inference on the intelligent computing cloud platform is determined by combining the first computing power consumption value and the second computing power consumption value. This not only breaks the limitations of the traditional single computing power measurement path and realizes the two-way mapping between business token and physical hardware, but also solves the problem of deviation verification between the two types of measurement results, thereby solving the problem of low accuracy of computing power measurement in the existing technology.

[0137] In one embodiment, step S3 includes: Step S3.1: Encode at least one of the input token number, output token number, and batch size into a vector to obtain the first vector representation.

[0138] The beneficial technical effect achieved by step S3.1 is that by vector encoding at least one of the input token number, output token number and batch size, discrete or numerical task features can be uniformly converted into a vector form that the model can recognize, fully preserving feature information such as task scale and concurrency, and providing standardized task feature input for subsequent high-precision computing power fitting.

[0139] Step S3.2: Perform vector encoding on at least one of the model parameters and the number of quantization bits to obtain a second vector representation.

[0140] The beneficial technical effects achieved by step S3.2 are as follows: By vector encoding at least one of the model parameters and the number of quantization bits, key parameters such as model complexity and computational accuracy can be represented in a structured manner, highlighting the impact of different scale models and different quantization methods on computing power consumption, and making the computing power fitting more closely match the computational characteristics of the model itself.

[0141] Step S3.3: Perform vector encoding on the GPU model to obtain the third vector representation.

[0142] The beneficial technical effects achieved by step S3.3 are as follows: By vector encoding the GPU model, the computing power performance and architectural differences of different GPUs can be transformed into quantifiable and comparable vector features, so that the computing power differences of the same model on different GPUs can be accurately represented, thereby improving the consistency and accuracy of cross-hardware computing power measurement.

[0143] Step S3.4: Merge the first vector representation, the second vector representation, and the third vector representation to obtain the merged vector representation.

[0144] In one possible implementation, the first, second, and third vector representations can be directly concatenated end-to-end along the feature dimension to form a longer fused vector representation.

[0145] The beneficial technical effects achieved by step S3.4 are as follows: By fusing the first, second, and third vector representations, the joint representation of task features, model features, and hardware features can be realized, capturing the correlation and interaction between multiple types of features, forming a more comprehensive and discriminative integrated feature representation, and providing a foundation for high-precision regression prediction.

[0146] Step S3.5: Call the pre-trained decision tree-based regression model to process the fusion vector representation, and use the decision tree-based regression model to perform token fitting computing power calculation to obtain the first computing power consumption value.

[0147] In one possible implementation, the decision tree-based regression model is the LightGBM regression model.

[0148] The beneficial technical effects achieved by step S3.5 are as follows: by calling the pre-trained decision tree-based regression model to process the fusion vector representation for measurement, on the one hand, the strong fitting ability and high robustness of the decision tree model for nonlinear relationships are utilized to improve the accuracy of computing power prediction; on the other hand, the errors and limitations caused by manually designed formulas are avoided, realizing end-to-end, adaptive token fitting computing power measurement and stably outputting the first computing power consumption value.

[0149] In one embodiment, step S4 includes: Step S4.1: Determine the theoretical total number of computational operations based on the number of model parameters, the number of input tokens, the number of output tokens, and the batch size.

[0150] The beneficial technical effects achieved by step S4.1 are as follows: By determining the theoretical total number of computational operations based on the number of model parameters, the number of input tokens, the number of output tokens, and the batch size, the basic computational workload of model inference can be quantified at the algorithm level, providing a unified theoretical computational benchmark for subsequent GPU computing power measurements and ensuring the physical meaning and comparability of computing power measurements.

[0151] It should be noted that the following steps are used to determine the total theoretical number of operands: Step S4.1.1: Calculate the total number of tokens based on the input token count and the output token count.

[0152] In one possible implementation, the sum of the input tokens and the output tokens can be calculated as the total number of tokens.

[0153] Step S4.1.2: Determine the first total number of computation operations based on the total number of tokens, the number of model parameters, and the batch size.

[0154] In one possible implementation, the product of the total number of tokens, the number of model parameters, and the batch size can be calculated as the first total computation operand.

[0155] Step S4.1.3: Determine the theoretical total number of calculation operations based on the first total calculation operation.

[0156] In one possible implementation, the product of the first total computational operation and 2 can be calculated as the theoretical total computational operation number.

[0157] Step S4.2: Determine the inference engine optimization coefficients based on the inference engine type.

[0158] In one possible implementation, the inference engine optimization coefficients can be pre-calibrated constants (e.g., PagedAttention optimization ≈ 1.05~1.15, native HF ≈ 1.0), reflecting the efficiency improvement brought about by software optimization.

[0159] The beneficial technical effects achieved by step S4.2 are as follows: By determining the optimization coefficient of the inference engine according to the type of inference engine, the optimization benefits of different inference frameworks in terms of operator fusion, memory reuse, asynchronous execution, etc. can be reflected, so that the computing power measurement can be adapted to different execution environments and improve versatility and accuracy.

[0160] Step S4.3: Determine the quantization precision correction coefficient based on the quantization bit depth.

[0161] In one possible implementation, the correspondence between the quantization bit depth and the quantization precision correction factor is: FP16=1.0, INT8≈0.65, INT4≈0.35 (based on actual measurement calibration of the GPU TensorCore architecture, reflecting the impact of quantization on effective computing power capacity).

[0162] The beneficial technical effect achieved by step S4.3 is that by determining the quantization precision correction coefficient based on the number of quantization bits, the actual computing power consumption under different precisions such as FP32 / FP16 / INT8 can be differentiated and corrected, solving the problem of inconsistent impact of different numerical precisions on computing power consumption, and making the calculation more consistent with the actual hardware execution.

[0163] Step S4.4: Determine the dynamic management overhead of KV Cache based on the KV cache switch status, context length, and batch size.

[0164] The beneficial technical effects achieved by step S4.4 are as follows: By determining the dynamic management overhead of KV Cache based on the KV cache switch status, context length and batch size, the additional resource consumption of cache read, write, replacement and copy can be accurately characterized, making up for the deficiency of traditional computing power measurement ignoring cache overhead and improving the robustness of measurement in large context scenarios.

[0165] Step S4.5: Calculate the GPU computing power consumption benchmark based on the theoretical total number of computational operations, GPU real-time utilization, GPU memory bandwidth utilization, inference engine optimization coefficient, quantization precision correction coefficient, KV Cache dynamic management overhead, and GPU effective runtime, and obtain the second computing power consumption value.

[0166] The beneficial technical effects achieved by step S4.5 are as follows: By comprehensively considering multiple factors such as theoretical computational load, GPU core utilization, bandwidth efficiency, engine optimization, quantization correction, and cache overhead, the benchmark calculation of GPU computing power consumption can comprehensively and realistically reflect the actual computing power consumption at the physical level of the GPU, and obtain a stable, reliable, and highly credible second computing power consumption value, laying a solid foundation for subsequent integration with theoretical values.

[0167] It should be noted that the calculation of the second computing power consumption value is performed using the following steps: Step S4.5.1: Multiply the total theoretical computational operands, GPU real-time utilization, GPU memory bandwidth utilization, inference engine optimization coefficients, and quantization precision correction coefficients to obtain the first product value.

[0168] The beneficial technical effects achieved by step S4.5.1 are as follows: By multiplying the theoretical total number of computational operations, GPU real-time utilization, GPU memory bandwidth utilization, inference engine optimization coefficient and quantization precision correction coefficient, the combined impact of theoretical computational load, actual hardware load, memory access bottleneck, engine optimization and precision conversion on computing power consumption can be comprehensively reflected, thereby achieving joint modeling of multi-dimensional key factors and improving the accuracy and comprehensiveness of computing power measurement.

[0169] Step S4.5.2: Summing the first product value with the KV Cache dynamic management overhead to obtain the first sum value.

[0170] The beneficial technical effect achieved by step S4.5.2 is that by summing the first product value with the dynamic management overhead of KV Cache, additional system overhead such as cache read / write and data migration can be further included on the basis of theoretical calculation and running efficiency, making the computing power measurement closer to the actual GPU running scenario and effectively making up for the measurement deviation caused by only considering the computing load.

[0171] Step S4.5.3: Determine the quotient of the first sum divided by the effective runtime of the GPU as the second computing power consumption value.

[0172] The beneficial technical effect achieved by step S4.5.3 is that by dividing the first sum by the effective runtime of the GPU to obtain the second computing power consumption value, the cumulative computing overhead can be converted into computing power consumption per unit time, so that the result has a unified physical dimension and comparability, and finally a stable GPU measured computing power benchmark that can be directly used for trusted fusion is obtained.

[0173] In one embodiment, such as Figure 2 As shown, step S5 includes: Step S5.1: Determine the relative deviation between the first computing power consumption value and the second computing power consumption value.

[0174] In one possible implementation, the absolute value of the difference between the first computing power consumption value and the second computing power consumption value can be calculated, and the ratio of the obtained absolute value of the difference to the second computing power consumption value can be used as the relative deviation between the first computing power consumption value and the second computing power consumption value.

[0175] The beneficial technical effect achieved by step S5.1 is that by calculating the relative deviation between the first computing power consumption value and the second computing power consumption value, the degree of deviation between the theoretically fitted computing power and the actual hardware computing power can be quantitatively measured, and the difference between the two calculation methods can be intuitively reflected, providing a quantitative basis for subsequent deviation verification and reliable value screening.

[0176] Step S5.2: Obtain the dynamic deviation threshold.

[0177] In the intelligent computing cloud platform of this invention, the deviation fluctuations vary greatly under different GPUs, models, and loads. Static thresholds cannot adapt to all scenarios, which may not only lead to normal data being misjudged as abnormal, but also cause abnormalities to go undetected. As new models, new GPUs, and new versions of the inference engine are launched, the normal deviation range will continue to change, and static thresholds will gradually become ineffective. In complex scenarios such as large contexts, large batch sizes, and frequent changes in KV cache, the deviation is naturally larger, and static thresholds are very likely to trigger abnormalities falsely.

[0178] Therefore, intelligent computing cloud platforms urgently need a threshold that can dynamically change according to the usage scenario to compare with the relative deviation of the first computing power consumption value and the second computing power consumption value.

[0179] In one possible implementation, the deviation values ​​of the most recent N similar inference tasks can be taken, and the mean plus the standard deviation within the sliding window can be used as the dynamic deviation threshold.

[0180] In another possible implementation, task levels can be divided according to model size, token length, and batch size, with different dynamic deviation thresholds preset for each level, and the thresholds can be adaptively selected based on the current task level.

[0181] It should be noted that the dynamic deviation threshold is obtained through the following steps: Step S5.2.1: Based on the GPU model and the third feature information, obtain multiple sets of historical computing power consumption values ​​with the same GPU model and the same model specifications. Each set of historical computing power consumption values ​​includes a historical first computing power consumption value and a historical second computing power consumption value.

[0182] The beneficial technical effects achieved by step S5.2.1 are as follows: By filtering historical computing power consumption values ​​according to the same GPU model and the same model specifications, it is possible to ensure that the historical data is highly consistent with the hardware environment and model characteristics of the current task, eliminate data interference caused by heterogeneous environments, and provide highly comparable historical samples for subsequent deviation statistics.

[0183] Step S5.2.2: Determine the historical standard deviation based on multiple sets of historical computing power consumption values.

[0184] The beneficial technical effects achieved by step S5.2.2 are as follows: Based on multiple sets of historical computing power consumption values, the standard deviation of historical deviation is calculated, which can objectively quantify the degree of dispersion between theoretical and measured values ​​in historical scenarios, reflect the normal fluctuation range of computing power measurement, and provide a real, data-driven statistical basis for dynamic thresholds.

[0185] Step S5.2.3: Determine the second product value obtained by multiplying the historical deviation standard deviation by the preset confidence coefficient as the reference deviation threshold.

[0186] In one possible implementation, the default confidence coefficient is set to 2, which can be adjusted according to the required confidence level, such as 95%.

[0187] The beneficial technical effect achieved by step S5.2.3 is that by multiplying the historical deviation standard deviation by the preset confidence coefficient to obtain the reference deviation threshold, a reasonable controllable deviation range can be adaptively set according to the historical fluctuation level, so that the threshold is more in line with the actual operating rules of the platform and avoids the fixed threshold being too strict or too lenient.

[0188] Step S5.2.4: Determine the maximum value between the reference deviation threshold and the preset deviation threshold as the dynamic deviation threshold.

[0189] In one possible implementation, the preset deviation threshold is set to 5%.

[0190] The beneficial technical effects achieved by step S5.2.4 are as follows: by taking the larger value between the reference deviation threshold and the preset deviation threshold as the dynamic deviation threshold, a fallback protection can be superimposed on the data-driven adaptive threshold, which not only ensures the adaptability of the scenario, but also prevents the threshold from failing due to abnormal historical data, thereby improving the stability and security of the overall judgment.

[0191] Step S5.3: Based on the comparison results of the relative deviation and the dynamic deviation threshold, determine the reliable computing power consumption value during inference on the intelligent computing cloud platform.

[0192] The beneficial technical effects achieved by step S5.3 are as follows: by determining the reliable computing power consumption value during inference of the intelligent computing cloud platform based on the comparison results of relative deviation and dynamic deviation threshold, intelligent optimization or fusion can be performed on the basis of stable theoretical value and real measured value, eliminating abnormal fluctuation data, ensuring that the final computing power consumption value is accurate, reliable and traceable, and providing a reliable basis for platform billing and scheduling.

[0193] In one embodiment, step S5.3 includes at least one of the following: Step S5.3.1: If the relative deviation is less than or equal to the dynamic deviation threshold, calculate the weighted average of the first computing power consumption value and the second computing power consumption value as the trusted computing power consumption value, and synchronize the trusted computing power consumption value, the computing power consumption value deviation report, and the computing power consumption value deviation verification status to the cloud platform billing engine and resource scheduler.

[0194] In one possible implementation, the average of the first computing power consumption value and the second computing power consumption value can be calculated as the trusted computing power consumption value.

[0195] It should be noted that the computing power consumption deviation report is a structured record summarizing information such as the first computing power consumption value, the second computing power consumption value, the relative deviation between the two, the dynamic deviation threshold, and the deviation calculation process during this model inference. The computing power consumption deviation verification status is a status marker used to indicate whether the current computing power deviation is within a reasonable and reliable range. Synchronizing the reliable computing power consumption value, the computing power consumption deviation report, and the computing power consumption deviation verification status to the cloud platform billing engine and resource scheduler serves two purposes. First, it provides the billing engine with a true, reliable, and traceable basis for computing power measurement, enabling fair, transparent, and accurate pay-as-you-go billing, avoiding billing deviations due to abnormal computing power data, and improving the platform's billing reliability and user acceptance. Second, it provides the resource scheduler with real-time and accurate load and computing power consumption feedback, allowing the scheduler to rationally allocate hardware resources such as GPUs and video memory based on actual computing power consumption, optimize task queuing and node scheduling strategies, and improve the overall resource utilization and system stability of the cloud platform.

[0196] The beneficial technical effects achieved by step S5.3.1 are as follows: When the relative deviation is within a reasonable range, the weighted average is used to obtain the reliable computing power consumption value, which can take into account both the stability of the theoretical fitting value and the authenticity of the hardware measured value, thereby improving the reliability of the final computing power data; at the same time, the relevant data is synchronized to the billing engine and resource scheduler, which can realize the real-time linkage between computing power data and platform business processes, ensuring fair billing, accurate scheduling, and improving the operating efficiency of the cloud platform.

[0197] Step S5.3.2: If the relative deviation is greater than the dynamic deviation threshold, the fusion vector representation and the second computing power consumption value are stored in the pending buffer pool. When the number of pending items in the pending buffer pool reaches a preset value, the closed-loop correction of the decision tree-based regression model is triggered. The fusion vector representation is determined by at least one feature information used to determine the first computing power consumption value. The corrected decision tree-based regression model is used to determine the new first computing power consumption value.

[0198] In one possible implementation, the preset value could be 50, which should strike a balance between the timeliness of correction and the stability of training.

[0199] In the intelligent computing cloud platform of this invention, as inference task scenarios change, new models are introduced, and new GPUs are connected, the original regression model gradually becomes out of sync with actual computing power consumption patterns. Without closed-loop correction, prediction errors will increase, making it unsuitable for new scenarios. Without abnormal data collection and model iteration mechanisms, the system can only rely on initial training results and cannot continuously iterate based on real-world data. This leads to a significant decrease in computing power measurement accuracy over time, making it difficult to meet the needs of large-scale, multi-scenario cloud platforms. Once a model fails, manual retraining and updates are necessary, resulting in low intelligence and slow response times, failing to meet the automated operation and maintenance requirements of cloud platforms.

[0200] Therefore, intelligent computing cloud platforms have a high demand for closed-loop correction of decision tree-based regression models.

[0201] The beneficial technical effects achieved by step S5.3.2 are as follows: when the deviation exceeds the limit, abnormal data is stored in the pending buffer pool, and the model closed-loop correction is triggered after a certain amount is accumulated. This can promptly detect the prediction deviation of the computing power fitting model. By continuously iterating and optimizing the regression model based on the decision tree, the accuracy of the token fitting computing power measurement is continuously improved, realizing the self-supervision and self-optimization of the entire system and ensuring the accuracy of the trusted computing power consumption value in the long term.

[0202] In one embodiment, step S1 further includes at least one of the following: Step S1.1: Perform time alignment preprocessing on task data, GPU running data, model and engine parameters using microsecond-level timestamps, and remove queuing periods, cold start periods and GPU Idle periods.

[0203] The beneficial technical effects achieved by step S1.1 are as follows: By using microsecond-level timestamps to align the multi-source feature information in time and eliminating queuing, cold start and GPU idle periods, feature distortion caused by time sequence misalignment can be eliminated, interference data in non-inference effective periods can be excluded, and subsequent computing power calculations can be based on real, effective and time-consistent running segments, thereby improving the accuracy of benchmark values ​​and calculation results.

[0204] Step S1.2: Perform preprocessing by interpolating or filling in missing parameters in the task data, GPU running data, model and engine parameters with default values.

[0205] The beneficial technical effects achieved by step S1.2 are as follows: by interpolating or filling missing parameters with default values, it is possible to avoid process interruptions or calculation errors caused by abnormal acquisition or loss of sampling, ensure the integrity of feature dimensions and the uniformity of data structure, and improve the stability and compatibility of the system in complex acquisition environments.

[0206] Step S1.3: Perform outlier filtering preprocessing on the task data, GPU running data, model and engine parameters. Outliers are processed separately or excluded from this calculation process.

[0207] The beneficial technical effects achieved by step S1.3 are as follows: By filtering outliers and handling abnormal data separately, it is possible to eliminate the interference of invalid data such as spikes, jumps, and exceeding limits on the model and bias calculation, prevent abnormal samples from skewing the calculation results, and improve the reliability and robustness of the overall computing power calculation.

[0208] In summary, as described in the above embodiments, Figure 3 As shown, Figure 3 This is a flowchart illustrating the overall process of the intelligent computing cloud platform provided by this invention, based on GPU computing power consumption and inference token values. The entire process can be automatically executed by the intelligent computing cloud platform scheduling server, and includes the following steps: Step a: Multi-dimensional feature data acquisition and time-aligned preprocessing.

[0209] Collect task data via the inference API: Input the number of tokens. Output the number of tokens Batch size Context length KV buffer switch status; GPU operating data is collected via a hardware monitoring interface: peak GPU computing power. GPU real-time utilization GPU memory bandwidth utilization GPU power consumption, GPU effective runtime ; Collect model and engine parameters: Number of model parameters Quantization bit depth, inference engine type.

[0210] like Figure 4 As shown, Figure 4 This is the data time alignment and parameter acquisition mapping diagram provided by the present invention.

[0211] The key improvements in this step are: employing a microsecond-level timestamp alignment mechanism to strictly match the API request lifecycle with the GPU monitoring window; eliminating queuing, cold start, and GPU idle periods; interpolating or filling missing parameters with default values ​​(e.g., using historical averages); and filtering outliers (using the 3σ principle) from the collected data. Outlier data can be marked and processed separately or excluded from the current calculation.

[0212] Step b: Calculate computing power based on machine learning fitting of token consumption.

[0213] Preprocessed The feature vector (i.e., the fusion vector representation mentioned above) is input into the pre-trained LightGBM regression model (i.e., the decision tree-based regression model mentioned above).

[0214] This LightGBM regression model is trained using a historical high-confidence test dataset with the same GPU and architecture, and the feature weights are fixed. The first computational cost is output. (Unit: TFLOPS), representing the average computing power rate calculated in reverse using the business token.

[0215] Step c: Benchmark calculation of physical computing power consumption based on full GPU parameters Theoretical total number of operands .

[0216] Substitute the reconstructed physical parameters into the formula to calculate the second computing power consumption value. (Unit: TFLOPS)

[0217] in, These are the optimization coefficients for the aforementioned inference engine. This is the aforementioned quantization accuracy correction coefficient. This refers to the overhead of dynamic management of the aforementioned KV Cache.

[0218] All correction coefficients in the formula are pre-calibrated using the standard test set of the same type of GPU to ensure physical interpretability and scenario universality.

[0219] In one possible implementation, the KV cache dynamically manages overhead. Obtained using the following method: (1) Conduct experimental measurements: Environment preparation: Determine the GPU model, LLM model, and inference engine.

[0220] Controlled variable: fixed batch size and change the context length .

[0221] Measurement indicators: precise measurement in different The additional overhead introduced by the KV cache. This can be achieved through: GPU power consumption difference: Total power consumption - Pure computational power consumption.

[0222] Memory bandwidth usage: Memory bandwidth generated by KV Cache reads and writes.

[0223] Fine-grained profiling: Using tools such as Nsight Compute, analyze and quantify the actual or equivalent FLOPs of KV cache-related operations (such as cudaMemcpy, cudaMemset, gather, scatter, etc.). This is usually the most accurate but also the most complex approach.

[0224] Indirect method: While controlling for other variables, observe the change in unit token inference latency or FLOPs consumption. The growth trend, and attempts to separate it from... The relevant parts.

[0225] (2) Data collection: For a series of context lengths Collect corresponding additional KV overhead measurements. .

[0226] (3) Data fitting: Convert the data points to logarithmic coordinates: .

[0227] Perform linear regression and fit a straight line. :

[0228]

[0229] slope That is .

[0230] intercept That is .

[0231] (4) Calculation : Intercept obtained from fitting ,calculate .

[0232] Because it is in a fixed batch size The experiment was conducted below, and the formula is Therefore, the actual KV cost factor for: .

[0233] Step d: Comparison of computing power along two paths and calculation of dynamic deviation.

[0234] Calculate absolute deviation and relative deviation .

[0235] Introducing a dynamic threshold mechanism: ,in The historical standard deviation of the GPU model plus model specifications (including quantization bits, inference engine, etc.) is given. The confidence coefficient (default value is 2, which can be adjusted according to the required confidence level, such as 95%) is used to improve the robustness of judgment in complex scenarios and reduce misjudgments.

[0236] Reliability assessment and closed-loop correction of the fitted model, such as Figure 5 As shown, Figure 5 This is the logic diagram for dual-path measurement and deviation determination provided by the present invention.

[0237] Step e: If If it is determined to be abnormal, the current feature vector will be compared with... Stored in the pending review buffer pool.

[0238] Step f: If ,determination It is reliable and outputs directly to the billing and scheduling system.

[0239] Step g: When the accumulated amount of abnormal data in the pending buffer pool reaches... (For example, after 50 values, this value strikes a balance between the timeliness of correction and training stability), incremental learning is triggered: the Elastic Weight Consolidation (EWC) algorithm is used to fine-tune the leaf node weights of LightGBM, identifying and protecting model parameters important to old tasks to prevent catastrophic forgetting, thus completing model iteration, such as... Figure 6 As shown, Figure 6 This is a diagram of the model incremental correction and anti-forgetting mechanism provided by the present invention.

[0240] Step h: Output results, traceability archiving, and scheduling linkage.

[0241] Output the final trusted computing power value (if trusted, the value can be taken as follows). , The average value (or weighted average according to business strategy), deviation report, and verification status are synchronized to the cloud platform billing engine (billed on equivalent computing power) and resource scheduler (dynamically adjusting GPU queues and load balancing).

[0242] All logs (Token data, hardware metrics, dual-path results, deviations, and correction records) are encrypted and archived, and third-party audit API calls are supported to ensure data traceability and auditability.

[0243] Please see Figure 7 , Figure 7 This is a structural diagram of a computing power metering device for an intelligent computing cloud platform based on GPU computing power consumption and inference token value, provided by the present invention. Figure 7As shown, the computing power metering device 700 of the intelligent computing cloud platform, based on GPU computing power consumption and inference token value, includes: The acquisition module 701 is used to collect task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform. The feature extraction module 702 is used to extract features from the task data, GPU running data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, respectively. The first feature information includes at least one of the following in the task data: number of input tokens, number of output tokens, batch size, context length, and KV cache on / off status. The second feature information includes at least one of the following in the GPU running data: peak GPU computing power, real-time GPU utilization, GPU memory bandwidth utilization, GPU power consumption, and effective GPU runtime. The third feature information includes at least one of the following in the model and engine parameters: number of model parameters, quantization bit depth, and inference engine type. The first computing power measurement module 703 uses a decision tree-based regression model to perform token fitting computing power measurement based on the first feature information, GPU model and the third feature information to obtain the first computing power consumption value. The second computing power calculation module 704 is used to perform GPU computing power consumption benchmark calculation based on the first feature information, the second feature information and the third feature information to obtain the second computing power consumption value. The determination module 705 is used to determine the trusted computing power consumption value during inference of the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value.

[0244] In one embodiment, the first computing power calculation module 703 includes: The first encoding unit is used to perform vector encoding on at least one of the input token number, the output token number, and the batch size to obtain a first vector representation; The second encoding unit is used to perform vector encoding on at least one of the model parameters and the quantization bits to obtain a second vector representation; The third encoding unit is used to perform vector encoding on the GPU model to obtain a third vector representation; A vector fusion unit is used to fuse the first vector representation, the second vector representation, and the third vector representation to obtain a fused vector representation; The first computing power measurement unit is used to call the pre-trained decision tree-based regression model to process the fused vector representation, and to perform token fitting computing power measurement through the decision tree-based regression model to obtain the first computing power consumption value.

[0245] In one embodiment, the second computing power calculation module 704 includes: The first determining unit is used to determine the theoretical total number of computational operations based on the number of model parameters, the number of input tokens, the number of output tokens, and the batch size. The second determining unit is used to determine the inference engine optimization coefficients based on the inference engine type. The third determining unit is used to determine the quantization accuracy correction coefficient based on the quantization bit depth. The fourth determining unit is used to determine the dynamic management overhead of the KV Cache based on the KV cache switch status, the context length, and the batch size.

[0246] The second computing power measurement unit is used to perform a benchmark calculation of GPU computing power consumption based on the theoretical total number of computational operations, the real-time utilization rate of the GPU, the GPU memory bandwidth utilization rate, the inference engine optimization coefficient, the quantization precision correction coefficient, the dynamic management overhead of the KV Cache, and the effective runtime of the GPU, and obtain the second computing power consumption value.

[0247] In one embodiment, the first determining unit is specifically used for: The total number of tokens is calculated based on the number of input tokens and the number of output tokens. The first total number of computational operations is determined based on the total number of tokens, the number of model parameters, and the batch size; Based on the first total calculation operation, determine the theoretical total number of calculation operations.

[0248] In one embodiment, the second computing power measurement unit is specifically used for: The first product value is obtained by multiplying the total theoretical computational operations, the GPU real-time utilization rate, the GPU memory bandwidth utilization rate, the inference engine optimization coefficient, and the quantization precision correction coefficient. The first sum is obtained by summing the first product value with the dynamic management overhead of the KV Cache; The quotient of the first sum divided by the effective runtime of the GPU is determined as the second computing power consumption value.

[0249] In one embodiment, the determining module 705 includes: The fifth determining unit is used to determine the relative deviation between the first computing power consumption value and the second computing power consumption value; The acquisition unit is used to acquire the dynamic deviation threshold. The sixth determining unit is used to determine the reliable computing power consumption value during inference of the intelligent computing cloud platform based on the comparison result between the relative deviation and the dynamic deviation threshold.

[0250] In one embodiment, the acquiring unit is specifically used for: Based on the GPU model and the third feature information, multiple sets of historical computing power consumption values ​​with the same GPU model and the same model specifications are obtained, wherein each set of historical computing power consumption values ​​includes a historical first computing power consumption value and a historical second computing power consumption value. Based on the aforementioned multiple sets of historical computing power consumption values, the historical deviation standard deviation is determined; The second product value obtained by multiplying the historical deviation standard deviation by a preset confidence coefficient is determined as the reference deviation threshold. The maximum value between the reference deviation threshold and the preset deviation threshold is determined as the dynamic deviation threshold.

[0251] In one embodiment, the sixth determining unit is specifically used for: If the relative deviation is less than or equal to the dynamic deviation threshold, the weighted average of the first computing power consumption value and the second computing power consumption value is calculated as the trusted computing power consumption value, and the trusted computing power consumption value, the computing power consumption value deviation report, and the computing power consumption value deviation verification status are synchronized to the cloud platform billing engine and resource scheduler. If the relative deviation is greater than the dynamic deviation threshold, the fusion vector representation and the second computing power consumption value are stored in the pending buffer pool, and when the number of pending items in the pending buffer pool reaches a preset value, the decision tree-based regression model is closed-loop corrected. The fusion vector representation is determined by at least one feature information used to determine the first computing power consumption value, and the corrected decision tree-based regression model is used to determine the new first computing power consumption value.

[0252] In one embodiment, the acquisition module 701 further includes at least one of the following: The first preprocessing unit is used to perform time alignment preprocessing on the task data, GPU running data, model and engine parameters using microsecond-level timestamps, and to remove queuing periods, cold start periods and GPU Idle periods. The second preprocessing unit is used to perform interpolation or fill in the missing parameters in the task data, GPU running data, model and engine parameters; The third preprocessing unit is used to perform outlier filtering preprocessing on the task data, GPU running data, model and engine parameters, wherein the outliers are processed separately or excluded from the current calculation process.

[0253] In this way, multiple feature information is first obtained by extracting features from the task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform. Then, based on these feature information, token fitting computing power measurement and GPU computing power consumption benchmark measurement are performed to obtain the first computing power consumption value and the second computing power consumption value. The reliable computing power consumption value during inference on the intelligent computing cloud platform is determined by combining the first computing power consumption value and the second computing power consumption value. This not only breaks the limitations of the traditional single computing power measurement path and realizes the two-way mapping between business token and physical hardware, but also solves the problem of deviation verification between the two types of measurement results, thereby solving the problem of low accuracy of computing power measurement in the existing technology.

[0254] It should be noted that the computing power metering device of the intelligent computing cloud platform in this invention based on GPU computing power consumption value and inference token value can be a device, or it can be a component, integrated circuit or chip in an electronic device.

[0255] The present invention also provides an electronic device, see [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 801, a processor 802, and a program or instructions stored in the memory 801 that run on the processor 802. When the program or instructions are executed by the processor 802, they can achieve the following: Figure 1 The corresponding intelligent computing cloud platform's computing power measurement method based on GPU computing power consumption and inference token value, and any steps in the embodiment that achieve the same beneficial effect, will not be elaborated here.

[0256] The processor 802 can be a CPU, ASIC, FPGA, or GPU.

[0257] Those skilled in the art will understand that all or part of the steps of the above-described intelligent computing cloud platform computing power measurement method based on GPU computing power consumption value and inference token value can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0258] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1 The corresponding intelligent computing cloud platform uses any step in the computing power measurement method embodiment based on GPU computing power consumption value and inference token value, and can achieve the same technical effect. To avoid repetition, it will not be described again here. The storage medium mentioned is such as read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0259] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The corresponding intelligent computing cloud platform uses various processes based on the computing power measurement method of GPU computing power consumption value and inference token value, and can achieve the same technical effect. To avoid repetition, these processes will not be described in detail here.

[0260] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this invention indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0261] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0262] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of the present invention.

[0263] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for measuring computing power in an intelligent computing cloud platform based on GPU computing power consumption and inference token value, characterized in that, include: Step S1: Collect task data, GPU running data, and model and engine parameters during inference on the intelligent computing cloud platform; Step S2: Extract features from the task data, GPU runtime data, model and engine parameters to obtain first feature information, second feature information and third feature information respectively. The first feature information includes at least one of the following in the task data: number of input tokens, number of output tokens, batch size, context length and KV cache on / off status. The second feature information includes at least one of the following in the GPU runtime data: GPU peak computing power, GPU real-time utilization, GPU memory bandwidth utilization, GPU power consumption and GPU effective runtime. The third feature information includes at least one of the following in the model and engine parameters: number of model parameters, number of quantization bits and inference engine type. Step S3: Calculate the first computing power consumption value by using a decision tree-based regression model based on the first feature information, GPU model, and the third feature information to fit the token. Step S4: Calculate the GPU computing power consumption benchmark based on the first feature information, the second feature information, and the third feature information to obtain the second computing power consumption value; Step S5: Determine the trusted computing power consumption value during inference on the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value.

2. The method as described in claim 1, characterized in that, Step S3 includes: Step S3.1: Perform vector encoding on at least one of the input token count, the output token count, and the batch size to obtain a first vector representation; Step S3.2: Perform vector encoding on at least one of the model parameters and the quantization bit depth to obtain a second vector representation; Step S3.3: Perform vector encoding on the GPU model to obtain a third vector representation; Step S3.4: Merge the first vector representation, the second vector representation, and the third vector representation to obtain a fused vector representation; Step S3.5: Call the pre-trained decision tree-based regression model to process the fusion vector representation, and use the decision tree-based regression model to perform token fitting computing power calculation to obtain the first computing power consumption value.

3. The method as described in claim 1, characterized in that, Step S4 includes: Step S4.1: Determine the theoretical total number of computational operations based on the model parameter count, the number of input tokens, the number of output tokens, and the batch size; Step S4.2: Determine the inference engine optimization coefficients based on the inference engine type; Step S4.3: Determine the quantization precision correction coefficient based on the quantization bit depth; Step S4.4: Determine the dynamic management overhead of KV Cache based on the KV cache switch status, the context length, and the batch size; Step S4.5: Calculate the GPU computing power consumption benchmark based on the theoretical total number of computational operations, the GPU real-time utilization rate, the GPU memory bandwidth utilization rate, the inference engine optimization coefficient, the quantization precision correction coefficient, the KV Cache dynamic management overhead, and the GPU effective runtime, and obtain the second computing power consumption value.

4. The method as described in claim 3, characterized in that, Step S4.1 includes: Step S4.1.1: Calculate the total number of tokens based on the input token count and the output token count; Step S4.1.2: Determine the first total number of computational operations based on the total number of tokens, the number of model parameters, and the batch size; Step S4.1.3: Determine the theoretical total number of calculation operations based on the first total calculation operation.

5. The method as described in claim 3, characterized in that, Step S4.5 includes: Step S4.5.1: Multiply the total theoretical computational operands, the GPU real-time utilization rate, the GPU memory bandwidth utilization rate, the inference engine optimization coefficient, and the quantization precision correction coefficient to obtain the first product value; Step S4.5.2: Summing the first product value with the dynamic management overhead of the KV Cache to obtain the first sum value; Step S4.5.3: Determine the quotient of the first sum divided by the effective runtime of the GPU as the second computing power consumption value.

6. The method according to any one of claims 1-5, characterized in that, Step S5 includes: Step S5.1: Determine the relative deviation between the first computing power consumption value and the second computing power consumption value; Step S5.2: Obtain the dynamic deviation threshold; Step S5.3: Based on the comparison result between the relative deviation and the dynamic deviation threshold, determine the reliable computing power consumption value during inference of the intelligent computing cloud platform.

7. The method as described in claim 6, characterized in that, Step S5.2 includes: Step S5.2.1: Based on the GPU model and the third feature information, obtain multiple sets of historical computing power consumption values ​​with the same GPU model and the same model specification, wherein each set of historical computing power consumption values ​​includes a historical first computing power consumption value and a historical second computing power consumption value; Step S5.2.2: Determine the historical deviation standard deviation based on the multiple sets of historical computing power consumption values; Step S5.2.3: Determine the second product value obtained by multiplying the historical deviation standard deviation by a preset confidence coefficient as the reference deviation threshold; Step S5.2.4: Determine the maximum value between the reference deviation threshold and the preset deviation threshold as the dynamic deviation threshold.

8. The method as described in claim 6, characterized in that, Step S5.3 includes at least one of the following: Step S5.3.1: If the relative deviation is less than or equal to the dynamic deviation threshold, calculate the weighted average of the first computing power consumption value and the second computing power consumption value as the trusted computing power consumption value, and synchronize the trusted computing power consumption value, the computing power consumption value deviation report, and the computing power consumption value deviation verification status to the cloud platform billing engine and resource scheduler. Step S5.3.2: If the relative deviation is greater than the dynamic deviation threshold, the fusion vector representation and the second computing power consumption value are stored in the pending buffer pool, and when the number of pending items in the pending buffer pool reaches a preset value, the closed-loop correction of the decision tree-based regression model is triggered. The fusion vector representation is determined by at least one feature information used to determine the first computing power consumption value, and the corrected decision tree-based regression model is used to determine the new first computing power consumption value.

9. The method as described in claim 1, characterized in that, Step S1 further includes at least one of the following: Step S1.1: Perform time alignment preprocessing on the task data, GPU running data, model and engine parameters using microsecond-level timestamps, and remove queuing periods, cold start periods and GPU Idle periods; Step S1.2: Perform interpolation or default value filling preprocessing on the missing parameters in the task data, GPU running data, model and engine parameters; Step S1.3: Perform outlier filtering preprocessing on the task data, GPU running data, model and engine parameters, wherein the outliers are processed separately or excluded from this calculation process.

10. A computing power metering device for an intelligent computing cloud platform based on GPU computing power consumption and inference token value, characterized in that, include: The data acquisition module is used to collect task data, GPU runtime data, and model and engine parameters during inference on the intelligent computing cloud platform. The feature extraction module is used to extract features from the task data, GPU runtime data, and model and engine parameters to obtain first feature information, second feature information, and third feature information, respectively. The first feature information includes at least one of the following in the task data: number of input tokens, number of output tokens, batch size, context length, and KV cache on / off status. The second feature information includes at least one of the following in the GPU runtime data: peak GPU computing power, real-time GPU utilization, GPU memory bandwidth utilization, GPU power consumption, and effective GPU runtime. The third feature information includes at least one of the following in the model and engine parameters: number of model parameters, quantization bit depth, and inference engine type. The first computing power calculation module is used to perform token fitting computing power calculation based on the first feature information, GPU model and the third feature information to obtain the first computing power consumption value. The second computing power calculation module is used to perform GPU computing power consumption benchmark calculation based on the first feature information, the second feature information and the third feature information to obtain the second computing power consumption value. The determination module is used to determine the trusted computing power consumption value during inference of the intelligent computing cloud platform based on the first computing power consumption value and the second computing power consumption value.

11. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the program implements the steps of the intelligent computing cloud platform based on GPU computing power consumption value and inference token value as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent computing cloud platform's computing power measurement method based on GPU computing power consumption value and inference token value as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, implement the steps of the computing power measurement method based on GPU computing power consumption value and inference token value for the intelligent computing cloud platform as described in any one of claims 1 to 9.

Images