Efficiency data determination method, apparatus and computer device in inference framework

Abstract

The application relates to an efficiency data determination method, device and computer equipment in an inference framework. The method comprises the following steps: in an inference process performed by the inference framework, a target computing operator performs computation on a first input tensor, and the execution process is measured to obtain execution time consumption of the execution process; dimension information of the first input tensor is analyzed to obtain a first operator shape parameter of the first input tensor; based on the first operator shape parameter, the execution time consumption and a hardware maximum computing power, the computing efficiency is determined; and the first operator shape parameter and the computing efficiency are stored in association to form efficiency data used for inference performance simulation. The method can provide reliable data support for inference system simulation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and computer device for determining efficiency data in a reasoning framework. Background Technology

[0002] Current deep learning inference performance simulations typically employ offline speed testing. This method uses independent tools to collect operator performance data and builds a predefined efficiency library. During simulation, performance data is retrieved by looking up tables based on fixed hyperparameters and computational shapes. This approach relies on manually preset hyperparameters and offline speed testing, thus failing to cover the variable and dynamically changing input shapes in inference scenarios. Furthermore, it struggles to adapt to the dynamic coupling relationships caused by request length, concurrency, and scheduling strategies, resulting in insufficient accuracy in performance modeling and failing to provide reliable data support for inference system simulations. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, and computer device for determining efficiency data in a reasoning framework that can provide reliable data support for inference system simulation, addressing the aforementioned technical problems.

[0004] Firstly, this application provides a method for determining efficiency data within a reasoning framework, including:

[0005] During the inference process, the first input tensor is computed based on the target computation operator, and the execution process is measured to obtain the execution time of the execution process.

[0006] The dimension information of the first input tensor is parsed to obtain the first operator shape parameter of the first input tensor;

[0007] The computational efficiency of the target computational operator is determined based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware.

[0008] The shape parameters of the first operator are associated with the computational efficiency and stored to form efficiency data for inference performance simulation.

[0009] In one embodiment, determining the computational efficiency of the target computational operator based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware includes:

[0010] The theoretical floating-point computational complexity is determined based on the shape parameters of the first operator and the operator type of the target computational operator.

[0011] The computational efficiency of the target computation operator is determined based on the theoretical floating-point operation volume, the execution time, and the maximum computing power of the hardware.

[0012] In one embodiment, determining the theoretical floating-point computational complexity based on the first operator shape parameters and the operator type of the target computation operator includes:

[0013] Obtain the operator type of the target computation operator;

[0014] Based on the operator type and the shape parameters of the first operator, the theoretical floating-point operation quantity corresponding to the target computation operator is determined.

[0015] In one embodiment, the target calculation operator is an online velocity detection operator;

[0016] The online speed detection operator is used to replace the native computation operator in the inference framework, execute the computation function of the native computation operator, and measure the execution process.

[0017] In one embodiment, the first input tensor includes input tensors with various different operator shape parameters;

[0018] Specifically, for input tensors with various different operator shape parameters, each step as described in claim 1 is executed to obtain and store the computational efficiency of the target computation operator corresponding to the various different operator shape parameters.

[0019] In one embodiment, the method further includes:

[0020] During the inference performance simulation, the performance of the inference scenario is simulated based on the efficiency data and the maximum computing power of the hardware to obtain the prediction time and / or system throughput.

[0021] In one embodiment, the process of performing inference performance simulation, based on the efficiency data and the maximum computing power of the hardware, includes:

[0022] During the inference performance simulation, the second operator shape parameter of the second input tensor is obtained;

[0023] If the efficiency data contains a third operator shape parameter that is consistent with the second operator shape parameter, then the performance simulation of the inference scenario is performed based on the first computing efficiency stored in association with the third operator shape parameter and the maximum computing power of the hardware.

[0024] If there is no parameter in the efficiency data that is consistent with the shape parameter of the second operator, then the fourth operator shape parameter that is closest to the shape parameter of the second operator is determined from the efficiency data, and the performance simulation of the inference scenario is performed based on the second computing efficiency stored in association with the fourth operator shape parameter and the maximum computing power of the hardware.

[0025] In one embodiment, the fourth operator shape parameter is one or more, and the performance simulation of the inference scenario based on the second computational efficiency stored in association with the fourth operator shape parameter and the maximum computing power of the hardware includes:

[0026] For a single fourth operator shape parameter, the computational efficiency of storing the fourth operator shape parameter in association is taken as the second computational efficiency;

[0027] For multiple fourth operator shape parameters, the computational efficiency of the multiple fourth operator shape parameters stored separately is integrated and calculated by means calculation or similarity weighted fusion to obtain the second computational efficiency;

[0028] Based on the second computational efficiency and the maximum computing power of the hardware, a performance simulation of the inference scenario is performed.

[0029] Secondly, this application also provides an efficiency data determination device in a reasoning framework, comprising:

[0030] The computation execution module is used to perform computation on the first input tensor based on the target computation operator during the inference process of the inference framework, and to measure the execution process to obtain the execution time of the execution process;

[0031] The parsing module is used to parse the dimension information of the first input tensor to obtain the first operator shape parameter of the first input tensor;

[0032] An efficiency determination module is used to determine the computational efficiency of the target computational operator based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware.

[0033] The association module is used to associate and store the shape parameters of the first operator with the computational efficiency to form efficiency data for inference performance simulation.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as shown in the first aspect or any embodiment thereof.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as shown in the first aspect or any embodiment thereof.

[0036] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method as shown in the first aspect or any embodiment thereof.

[0037] The efficiency data determination method, apparatus, and computer equipment in the above-mentioned inference framework, during the inference process, perform calculations on the first input tensor based on the target computation operator and measure the execution process to obtain the execution process; parse the dimensional information of the first input tensor to obtain the first operator shape parameter of the first input tensor; determine the computation efficiency based on the first operator shape parameter, the execution time, and the maximum computing power of the hardware; and associate and store the first operator shape parameter with the computation efficiency to form efficiency data for inference performance simulation. This scheme utilizes a target computation operator to perform calculations on the input tensor and measures them in real time during the inference process. The computational efficiency is determined by combining the dimensional information of the input tensor, and the operator shape parameters are then associated with and stored to form effective data support. This effectively solves the shortcomings of existing offline speed measurement methods, such as reliance on manually preset hyperparameters, inability to cover dynamically changing input shapes, and difficulty in adapting to the dynamic coupling relationships caused by request length and concurrency. It breaks through the limitations of offline speed measurement, achieves coverage of dynamic input scenarios, improves the accuracy of inference performance modeling, and provides reliable data support for inference performance simulation. This solves the problems of insufficient performance modeling accuracy and inability to provide effective data support for simulation in existing technologies. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the efficiency data determination method within the inference framework of one embodiment.

[0040] Figure 2 This is a flowchart illustrating an efficiency data determination method within an inference framework in one embodiment.

[0041] Figure 3 This is a flowchart illustrating the process of determining computational efficiency in one embodiment;

[0042] Figure 4 This is a flowchart illustrating the performance simulation process in one embodiment;

[0043] Figure 5 A structural block diagram of an efficiency data determination device in an inference framework in one embodiment;

[0044] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] Currently, deep learning inference performance simulation typically employs offline speed testing. This method uses independent tools to collect operator performance data and build a predefined efficiency library. During simulation, performance data is obtained by looking up tables based on fixed hyperparameters and computational shapes. This approach relies on manually preset hyperparameters and offline speed testing, thus failing to cover the variable and dynamically changing input shapes in inference scenarios. Furthermore, it struggles to adapt to the dynamic coupling relationships caused by request length, concurrency, and scheduling strategies, resulting in insufficient accuracy in performance modeling and an inability to provide reliable data support for inference system simulation.

[0047] To address the aforementioned issues, embodiments of this application provide a method, apparatus, and computer device for determining efficiency data within a reasoning framework that can provide reliable data support for inference system simulation.

[0048] The efficiency data determination method in the inference framework provided in this application embodiment can be applied to an efficiency data determination device or computer device in the inference framework. The efficiency data determination device in the inference framework can be a functional module or functional entity within the computer device used to implement the efficiency data determination method in the inference framework.

[0049] The aforementioned computer equipment may include, but is not limited to: servers, edge computing devices (edge ​​servers and edge gateways for edge inference scenarios), terminal devices (intelligent terminals with inference computing capabilities, such as high-performance workstations, artificial intelligence terminals, and industrial terminals equipped with inference chips), and other computing devices with data processing, inference computing, and storage capabilities, capable of running deep learning inference frameworks and executing efficient data determination methods.

[0050] For example, servers may include standalone servers, cluster servers, cloud servers, etc., especially graphics processing unit (GPU) servers and artificial intelligence (AI) dedicated servers that are adapted to deploy deep learning inference frameworks.

[0051] The efficiency data determination method in the inference framework provided in this application embodiment involves the following flowchart: Figure 1 As shown. Among them, the efficiency data determination method in this inference framework mainly involves efficiency data determination stage 11 and performance simulation stage 12.

[0052] In the efficiency data determination stage 11, based on input tensors of various dimensions, calculations are performed by the target computation operator, and the execution process is measured. The dimensional information of the input tensors is analyzed, the computation efficiency is determined and associated with the storage, and the collection and storage of efficiency data is completed. In this way, real efficiency data under different input scenarios can be obtained, providing reliable support for subsequent simulations. At the same time, it can achieve comprehensive coverage of inputs of different dimensions and avoid the limitations of offline collection in existing technologies.

[0053] In performance simulation phase 12, simulation calculations can be performed based on the efficiency data acquired in the previous phase, combined with the maximum computing power of the hardware. This forms a complete technical link with the efficiency data acquisition in the previous phase, realizing a closed loop from data acquisition to simulation output.

[0054] In one exemplary embodiment, such as Figure 2 As shown, a method for determining efficiency data in an inference framework is provided, comprising the following steps 201 to 204. Wherein:

[0055] 201. During the inference process of the inference framework, the first input tensor is calculated based on the target computation operator, and the execution process is measured to obtain the execution time of the execution process.

[0056] The first input tensor can refer to a multidimensional array in the inference framework used to carry input data. It is the processing object of the operator calculation. In this embodiment, it specifically refers to the input tensor currently used to perform calculation and speed measurement, and is used to distinguish the input tensor in the subsequent simulation stage.

[0057] The aforementioned measurement of the execution process can refer to using a hardware timer to accurately count the time spent on the calculation while the operator is performing the calculation.

[0058] In some possible implementations, the target computation operator is an online velocity detection operator.

[0059] The online speed detection operator is used to replace the native computation operator in the inference framework, execute the computation function of the native computation operator, and measure the execution process.

[0060] The aforementioned online speed detection operator is an improved operator that has been integrated into the inference framework, combining native computational functions with the ability to measure the execution process. The native computational operator is the operator originally used in the inference framework to execute standard computational logic.

[0061] In this embodiment, the original computation operator can be transparently replaced by an online speed measurement detection operator. The operator completes the computation task normally, while the hardware timer records the execution time without interruption or change to the original data stream.

[0062] The aforementioned online speed measurement operator can be obtained by embedding logic for measuring the execution process, capturing shape information, and calculating efficiency into the execution logic of the original calculation operator, thus obtaining an online speed measurement operator that combines the original calculation and measurement functions.

[0063] For example, a dynamic replacement method using monkey-patching can be employed to deploy online speed testing operators. The specific process includes: without modifying the original source code of the inference framework, dynamically extending or adjusting the runtime code during program execution. First, the original implementations of the native computation operators in the inference framework modules are saved. Then, an online speed testing operator is defined, combining the functions of computational execution and measurement of the execution process. The inference framework's calls to the native computation operators are dynamically redirected to the newly defined online speed testing operator. This allows the inference process to automatically execute the speed testing logic of the online speed testing operator during runtime, while maintaining the original computational functions and inference flow unaffected, achieving a non-intrusive and transparent replacement. In this implementation, through transparent operator replacement and online speed testing, accurate and real operator execution times are obtained without affecting inference execution, providing foundational data for subsequent efficiency calculations.

[0064] In this embodiment, the inference framework is typically implemented based on a mature open-source inference engine, such as the Virtual Large Language Model (vLLM), which is suitable for high-concurrency inference of large language models. These inference frameworks possess robust task scheduling, memory management, batch processing, and token arrangement capabilities, enabling them to stably handle dynamically variable-length inputs from real-world business scenarios. In these frameworks, the data stream flows sequentially through various computational operators according to a pre-defined inference logic, completing feature extraction, linear transformation, attention calculation, normalization, and other processing steps in sequence, ultimately outputting the model inference result. This embodiment reuses this native data stream, eliminating the need to reconstruct the input, construct simulated data, or launch independent offline speed testing tools. Performance data collection can be completed synchronously within the real inference process, fundamentally eliminating dependence on offline tools.

[0065] Meanwhile, the inference framework integrates a scheduler. The scheduler dynamically determines the size of the current inference batch, the length of the input sequence, and the operator execution order based on the length of the currently incoming requests, the number of concurrent requests, the system load status, and cache usage. It then distributes the prepared input tensors to each computational operator. This embodiment utilizes the real inference tasks distributed by the scheduler to capture and measure the speed of real-world computation scenarios, rather than relying on manually preset hyperparameters. Therefore, it can accurately reflect the dynamic coupling relationships caused by changes in request length, fluctuations in concurrency, and adjustments to the scheduling strategy, thus overcoming the technical deficiency of traditional offline speed testing in simulating the real inference runtime environment.

[0066] In the inference process, the inference flow of a large language model can generally be divided into a prefill stage and an incremental decoding stage. The prefill stage processes the initial input sequence and completes feature encoding of the full context, while the incremental decoding stage generates new output word by word based on the generated context. The two stages differ in input shape, computation method, memory access characteristics, and computational efficiency. Traditional offline speed testing methods typically only collect efficiency data under fixed shapes, failing to distinguish the performance differences brought about by different inference stages, thus making it difficult to support high-precision simulations. However, the embodiments of this application can automatically identify the current inference stage during inference runtime, and, combined with the input tensor dimension information of the corresponding stage, collect operator shape parameters and computational efficiency for different stages. This allows the efficiency data to fully cover the performance characteristics of the prefill and incremental decoding stages, further improving the comprehensiveness and accuracy of the efficiency data.

[0067] 202. Parse the dimension information of the first input tensor to obtain the first operator shape parameter of the first input tensor.

[0068] The dimensional information can include various parameters that constitute the shape of the input tensor, such as batch size, sequence length, and feature dimensions.

[0069] The aforementioned first operator shape parameter can be obtained by parsing the dimension information of the first input tensor, and is used to uniquely identify the computational shape of the target computational operator.

[0070] For example, the attention mechanism operator of Llama-2 8B is parsed to obtain batch size, number of heads, head dimension, context length, etc., forming the corresponding operator shape parameters.

[0071] The Llama 2 8B mentioned above is an open-source large language model designed with a Transformer architecture. It possesses text understanding and generation capabilities and can be used in various natural language processing scenarios such as dialogue, text generation, and content creation. Its inference process involves multiple different types of computational operators. The execution time and computational efficiency of these operators directly affect the overall inference performance of the model. Therefore, it is necessary to collect efficiency data and conduct performance simulations for key operators in the model to optimize the inference scheduling strategy. The Attention operator is the core computational unit in the Transformer architecture. It models contextual information by calculating the association weights of elements at different positions in a sequence. In large language models like Llama 2 8B, the computational cost of this operator is strongly correlated with the input dimension. For example, changes in parameters such as batch size, context length, number of attention heads, and head dimension significantly affect its computational complexity and execution time, making it one of the key operators affecting the overall inference performance of the model.

[0072] In the above 202, dimensional information can be automatically extracted from the current input tensor and transformed into operator shape parameters that can be used for retrieval, storage, and matching. No manual preset of hyperparameters is required, and it can adapt to dynamic and variable input changes in inference scenarios.

[0073] During the input data processing process in the inference framework, the dimensionality of the input tensor dynamically changes with different user requests. For example, different text lengths, different numbers of users making simultaneous requests, and different batch sizes merged by the scheduler all directly lead to changes in the dimensions of the input tensor, resulting in different operator shape parameters. Since operator execution efficiency is strongly correlated with operator shape parameters, the computational efficiency of the same operator under different shapes can fluctuate significantly, especially in large language models. As the context length increases, cache access overhead increases, and computational efficiency shows a significant downward trend. Therefore, only by collecting dynamically changing operator shape parameters and their corresponding computational efficiencies can we construct efficiency data that closely reflects real-world inference scenarios.

[0074] In this embodiment, the system can automatically extract key dimensional features affecting computational efficiency from the input tensor. These dimensional features constitute the core content of the operator shape parameters, such as batch size, sequence length, number of attention heads, head dimension, hidden layer dimension, etc. These dimensions are also referred to as operator shape keywords. For linear operators, the system automatically extracts the input dimension (M, K) and weight dimension (K, N). In the pre-filling stage, M usually corresponds to the total number of tokens, and in the incremental decoding stage, M usually corresponds to the number of concurrent requests. Through the automatic parsing of these keywords, the system can accurately identify the operator shape under different computational scenarios and complete online modeling without manual intervention.

[0075] Furthermore, in large language model inference, the key-value cache (KV cache) is a crucial factor affecting operator execution efficiency. As the context length increases, the amount of data in the key-value cache grows, significantly increasing the pressure on GPU memory access and thus reducing operator computation efficiency. Traditional offline speed testing methods cannot detect the dynamic changes in the key-value cache, resulting in a significant deviation between the collected efficiency data and real-world inference scenarios. This application's embodiment, by directly reusing the native data stream of the inference framework, can naturally capture key information affecting efficiency, such as key-value cache size, context length, and sequence length. This allows the operator shape parameters to fully reflect performance influencing factors in a real-world inference environment, providing more reliable data support for subsequent performance simulations.

[0076] 203. Based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware, determine the computational efficiency of the target computational operator.

[0077] The aforementioned computational efficiency refers to the ratio of actual computing power to the maximum computing power of the hardware, used to characterize the actual execution performance of the operator.

[0078] The aforementioned actual computing power refers to the number of floating-point operations actually completed by the operator per unit time in a real reasoning scenario. It is determined by the shape parameters of the first operator and the running time, and can reflect the actual computing level of the operator in actual operation.

[0079] The aforementioned maximum hardware computing power refers to the peak computing capability that the hardware device can achieve under theoretically ideal conditions. It represents the total number of floating-point operations that the hardware can complete per unit time and is a core indicator for measuring hardware computing performance. This parameter is determined by the hardware's own specifications and is not affected by external factors such as inference scenarios, input shapes, and scheduling strategies. It can serve as a benchmark for the actual execution efficiency of computing operators. By comparing the actual computing power with the maximum hardware computing power, standardized computing efficiency can be obtained, thus providing a stable and unified hardware benchmark for inference performance simulation.

[0080] In step 203 above, the theoretical floating-point operation volume can be calculated first based on the operator type and operator shape parameters, then the actual computing power can be obtained by combining the execution time, and finally compared with the maximum computing power of the hardware to obtain the computing efficiency. This process transforms execution time, operator shape, and hardware capabilities into standardized computing efficiency that can be used for simulation, improving the accuracy of subsequent performance predictions.

[0081] Among them, the theoretical floating-point operation capacity parameter can be the theoretical computational capacity calculated based on the operator type and shape parameters.

[0082] 204. The shape parameters of the first operator are associated with the computational efficiency and stored to form efficiency data for inference performance simulation.

[0083] For example, associated storage could be storing the shape parameters of the first operator in key-value pairs with their corresponding computational efficiency.

[0084] The efficiency data mentioned above can be used to support inference performance simulation, such as a set of correspondences between the shape parameters of each operator and their corresponding computational efficiency.

[0085] In step 204 above, the operator shape parameters obtained from the analysis and the computational efficiency obtained from the calculation can be stored in the database in a one-to-one correspondence, forming efficiency data that can be directly called upon in the simulation stage. This construction of efficiency data that can be directly used for simulation achieves the connection between data acquisition and simulation processes, laying a data foundation for high-precision simulation.

[0086] After determining the operator shape parameters and computational efficiency, this embodiment employs an asynchronous storage method to associate the operator shape features, computational efficiency, and system load status into an efficiency database. The system load status characterizes the hardware's operating state during inference execution, such as GPU utilization, memory usage, power consumption limits, and temperature. This information further reflects the changes in operator efficiency under different loads, allowing subsequent simulations to select more realistic efficiency data based on system load. Through this triplet-based associated storage method, the efficiency data not only includes the correspondence between shape and efficiency but also operational environment information, greatly enhancing the reliability of the simulation results.

[0087] In this embodiment, to further expand the coverage of efficiency data and improve adaptability to dynamic input shapes, a high-density sampling strategy is adopted in the early stages of inference execution. Specifically, for a period of time after the inference framework starts, diverse input requests are received, ensuring that the input tensors cover different batch sizes, sequence lengths, context lengths, and inference stages. By performing intensive rate testing on operators of various shapes, a comprehensive and representative efficiency database is quickly constructed. High-density sampling effectively reduces the probability of encountering unsampled shapes during the simulation stage and provides richer reference data for the nearest neighbor backoff strategy, thereby reducing prediction errors and improving overall simulation accuracy.

[0088] Through the complete process of online acquisition, automatic parsing, real-time calculation, asynchronous storage, and high-density sampling described above, this application embodiment constructs an efficiency data generation mechanism that is completely different from traditional offline speed measurement methods. This mechanism relies entirely on the native inference process, reuses real inference data streams, requires no manual preset of hyperparameters, requires no independent offline tools, and does not interrupt normal inference business. Therefore, it can truly adapt to highly dynamic and highly uncertain input shapes in inference scenarios, solving the problems that traditional technologies cannot cover dynamic shape parameters and cannot adapt to the coupling relationship between concurrency and scheduling.

[0089] In the above embodiments, through online speed testing, automatic dimension parsing, efficiency calculation and associated storage within the inference framework, efficiency data collection in real inference scenarios is completed without relying on offline tools or manually presetting hyperparameters, effectively covering dynamically variable inputs and improving the authenticity and comprehensiveness of efficiency data.

[0090] In one exemplary embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating the process of determining computational efficiency in one embodiment. Step 203 above, which involves determining computational efficiency, may include steps 301 to 302. Wherein:

[0091] 301. Determine the theoretical floating-point computational complexity based on the shape parameters of the first operator and the operator type of the target computational operator.

[0092] The operator types mentioned above can be used to distinguish operator categories with different computational logics, such as Attention, Linear, and LayerNorm. Among them, the Linear operator is the most basic computational unit in deep learning models, mainly used to perform matrix multiplication and bias addition operations on input features, realizing linear transformation and dimensional mapping of features. It is the core operator for feature extraction and information mapping, and its computational cost is directly related to the input and output dimensions, making it one of the main operators affecting model inference performance. The LayerNorm operator is used to normalize input feature data, stabilizing the numerical range during model inference by adjusting the data distribution, thus improving computational stability and model convergence. This operator performs normalization calculations on a single sample or single channel basis and is widely used in models with transformer architectures. Its execution time and memory access overhead directly affect the overall inference efficiency of the model.

[0093] In step 301 above, the type of operator currently being executed can be identified first, and then the theoretical floating-point operation quantity for this calculation can be calculated based on the calculation formula corresponding to the operator type and the shape parameters of the first operator.

[0094] In some possible implementations, the theoretical floating-point operation quantity is determined based on the shape parameters of the first operator and the operator type of the target computation operator. This may include, but is not limited to: obtaining the operator type of the target computation operator; and determining the theoretical floating-point operation quantity corresponding to the target computation operator based on the operator type and the shape parameters of the first operator.

[0095] For example, for the self-attention operator of Llama-2 8B, the theoretical floating-point computation can be calculated according to the operator shape parameters such as batch size, number of heads, head dimension, and context length.

[0096] In step 301 above, the theoretical computational load can be accurately calculated based on the operator shape parameters and operator type, providing an accurate basis for subsequent efficiency calculations.

[0097] 302. Based on the theoretical floating-point operation volume, execution time, and maximum hardware computing power, determine the computational efficiency of the target computational operator.

[0098] For example, the actual computing power can be obtained by dividing the theoretical floating-point operation volume by the execution time, and then the ratio of the actual computing power to the maximum computing power of the hardware can be used as the standardized computing efficiency.

[0099] The above embodiments provide a process for determining computational efficiency. By identifying operator types, calculating theoretical computational load, and standardizing efficiency conversion, the efficiency data becomes more accurate and comparable, further improving the accuracy of subsequent simulations.

[0100] In one exemplary embodiment, the first input tensor includes input tensors with various different operator shape parameters.

[0101] Specifically, for input tensors with various operator shape parameters, the following operations are performed respectively: Figure 2 The steps shown are used to obtain and store the computational efficiency corresponding to various different operator shape parameters.

[0102] For example, input tensors with different operator shape parameters can refer to a set of input tensors with different batches, lengths, and dimensions. High-density, multi-scenario sampling can be achieved through such input tensors.

[0103] For example, in the early stages of inference, multiple input tensors of different dimensions can be input into the inference framework. For each input tensor, the steps of calculation, speed measurement, resolution of dimensions, computational efficiency, and associated storage are fully executed, so that the efficiency data can cover more operator shape parameters.

[0104] In the above embodiments, by densely sampling multi-dimensional input tensors, the coverage of the efficiency library can be greatly improved, which can cope with the dynamically changing input shape in the inference scenario, reduce the error of backoff prediction during simulation, and further improve the simulation accuracy.

[0105] In an exemplary embodiment, the efficiency data determination method in the above inference framework can also perform performance simulation of the inference scenario based on the efficiency data and the maximum computing power of the hardware during the inference performance simulation process to obtain the prediction time and / or the system throughput.

[0106] The prediction time can be the execution time of the operator or model predicted by the simulation. The system throughput mentioned above can refer to the number of requests or tokens that can be processed per unit time as predicted by the simulation.

[0107] For example, efficiency data can be loaded through a simulator, and the computational efficiency can be obtained by looking up a table based on the operator shape parameters of the inference scenario. The prediction time and throughput can then be calculated by combining the maximum computing power of the hardware.

[0108] In the above embodiments, simulations can be performed based on real online collected efficiency data, which can accurately reflect the dynamic characteristics of the inference scenario, improve the reliability of simulation results, and provide support for the scheduling and optimization of the inference system.

[0109] In one exemplary embodiment, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating a performance simulation process in one embodiment, which may include steps 401 to 404. Wherein:

[0110] 401. During the inference performance simulation, obtain the second operator shape parameters of the second input tensor.

[0111] The second input tensor mentioned above can refer to the tensor corresponding to the inference input to be predicted during the simulation phase.

[0112] The aforementioned second operator shape parameter is obtained after analyzing the second input tensor of the simulation input, and is the operator shape parameter to be searched and matched through simulation.

[0113] In step 401 above, when starting the simulation, the dimension information of the second input tensor is parsed to obtain the shape parameters of the second operator used to retrieve efficiency data.

[0114] 402. Determine whether there is an operator shape parameter in the efficiency data that is consistent with the shape parameter of the second operator.

[0115] Specifically, the second operator shape parameters required for simulation can be precisely matched with the records already stored in the efficiency data to determine whether there are consistent operator shape parameters.

[0116] 403. There exists a third operator shape parameter that is consistent with the shape parameter of the second operator. Based on the first computational efficiency stored in association with the shape parameter of the third operator and the maximum computing power of the hardware, the performance of the inference scenario is simulated.

[0117] The third operator shape parameter is the sampled operator shape parameter in the efficiency data that is completely consistent with the simulation target. The first computational efficiency is the measured computational efficiency in the efficiency data corresponding to the third operator shape parameter.

[0118] In step 403 above, if a completely matching record is found, that is, if there is a third operator shape parameter in the efficiency data that is consistent with the shape parameter of the second operator, the third operator shape parameter is directly used in combination with the maximum computing power of the hardware to perform simulation calculation and obtain the prediction result.

[0119] 404. There is no parameter that is consistent with the shape parameter of the second operator. Determine the fourth operator shape parameter that is closest to the shape parameter of the second operator from the efficiency data. Perform performance simulation on the inference scenario based on the second computational efficiency stored in association with the shape parameter of the fourth operator and the maximum computing power of the hardware.

[0120] The fourth operator shape parameter is the sampled parameter in the efficiency library that is closest to the shape of the simulation target. The second computational efficiency is the measured efficiency corresponding to the fourth operator shape parameter.

[0121] The fourth operator shape parameter can be single or multiple. The performance simulation of the inference scenario based on the second computational efficiency associated with the fourth operator shape parameter and the maximum computing power of the hardware may include, but is not limited to: for a single fourth operator shape parameter, using the computational efficiency associated with the fourth operator shape parameter as the second computational efficiency; for multiple fourth operator shape parameters, integrating and calculating the computational efficiency associated with each of the multiple fourth operator shape parameters through mean calculation or similarity weighted fusion to obtain the second computational efficiency; and then performing performance simulation of the inference scenario based on the second computational efficiency and the maximum computing power of the hardware.

[0122] In some embodiments, the fourth operator shape parameter can be a single sampled parameter in the efficiency library that has the highest similarity to the simulated target shape, or multiple sampled parameters whose similarity reaches a threshold can be selected; the second computational efficiency is the measured efficiency value associated with the corresponding matching parameter.

[0123] If a single parameter is selected, its associated measured efficiency is directly used as the second computational efficiency; if multiple parameters are selected, the corresponding measured efficiencies can be integrated through mean calculation, weighted fusion, etc., to finally obtain the appropriate second computational efficiency.

[0124] The above embodiments enable rapid and accurate simulation calculations using sampled real efficiency data when only a single optimal matching sample exists, ensuring both simulation efficiency and basic accuracy. Furthermore, in scenarios with multiple highly similar samples, they allow for the full utilization of measured data from multiple sets of similar working conditions. By using mean calculation or similarity-weighted fusion, the random errors of single-sampled data are mitigated, effectively improving the smoothness and robustness of computational efficiency predictions in scenarios with unknown operator shapes. Simultaneously, this matching and fusion method aligns with the characteristic that the efficiency of large model operators continuously changes with shape parameters, minimizing the performance deviation between unsampled and sampled shapes.

[0125] In step 404 above, if no perfect match is found, the fourth operator shape parameter with the closest feature is searched in the efficiency data. The fourth operator shape parameter is then used in conjunction with the maximum computing power of the hardware to perform simulation calculations and obtain the prediction result.

[0126] In step 404 above, the simulation problem of unsampled shapes is solved by the nearest neighbor backoff strategy, ensuring the continuous availability of the simulation process while minimizing prediction errors.

[0127] In this embodiment, "closest" refers to matching the second operator shape parameter with the stored operator shape parameters in the efficiency data within the feature space composed of operator shape parameters, based on dimensional features, numerical distribution, and computational logic relevance. The stored operator shape parameter that has the smallest difference from the second operator shape parameter in key dimensions affecting computational efficiency and has the most similar computational features is selected as the fourth operator shape parameter. Key dimensions include, but are not limited to, batch size, sequence length, context length, number of attention heads, head dimension, and hidden layer dimension—parameters that directly determine the computational load and execution time of the operator. This matching method minimizes simulation errors caused by inconsistencies in operator shapes, ensuring the accuracy and reliability of performance simulation results.

[0128] For example, Euclidean distance can be used for feature matching, that is, the Euclidean distance between the shape parameter of the second operator and the stored shape parameters of each operator in the key dimension is calculated, and the stored shape parameter of the operator with the smallest Euclidean distance is determined as the closest fourth operator shape parameter; Manhattan distance, Chebyshev distance or other distance metrics suitable for high-dimensional feature matching can also be used, and this application does not limit this.

[0129] During the performance simulation phase, when the simulator encounters a new operator shape parameter not collected in the efficiency database, this embodiment employs a nearest neighbor fallback strategy for efficiency prediction, also known as a smooth fallback mechanism. Specifically, the system maps the new operator shape parameter to the feature space and searches the existing efficiency data for the sampled shape that is closest to the parameter in terms of dimensionality distribution, numerical characteristics, and inference stage. The computational efficiency corresponding to this shape is used as the predicted efficiency, and the time consumption is estimated by combining this with the theoretical floating-point operation cost of the new shape. This method achieves performance prediction for any unknown shape without requiring re-measurement, stopping the simulation, or expanding the database, ensuring the continuity and stability of the simulation process.

[0130] The advantage of the nearest neighbor backoff strategy lies in its ability to maintain high prediction accuracy even when the operator shape space is large and full-coverage sampling is impossible. Since the operator efficiency of large language models exhibits continuous and regular variations with shape, operators with similar shapes typically have similar computational efficiency under the same hardware environment. Therefore, nearest neighbor matching can minimize errors, ensuring high reliability of simulation results. Traditional offline velocity measurement methods lack online data acquisition and backoff prediction capabilities. When encountering shapes outside the preset range, they can only rely on fixed empirical values, leading to a significant drop in simulation accuracy.

[0131] Furthermore, the performance simulation of this application embodiment can fully support high-precision simulation of complex inference scenarios, including multi-level key-value caching, dynamic scheduling latency, dynamic batch processing, and concurrent request jitter. Based on real online data in the efficiency database, combined with information such as maximum hardware computing power, operator theoretical floating-point operation volume, inference stage characteristics, and key-value cache overhead, the simulator simulates the real inference execution process layer by layer, operator by operator, and batch by batch, ultimately outputting multiple performance indicators such as end-to-end prediction time, system throughput, single-batch time, and operator-level time. These indicators can be directly used for scheduling optimization, resource allocation, performance evaluation, and capacity planning of the inference system, providing scientific and reliable data support for the inference system.

[0132] By establishing a complete closed loop between the online efficiency data determination stage and the performance simulation stage, this application embodiment realizes a full-link technical solution from "real inference data flow" to "online efficiency acquisition" and then to "high-precision performance simulation". Compared with the traditional offline method, it has many advantages such as no need for manual intervention, coverage of dynamic shapes, adaptation to scheduling and concurrent coupling, high simulation accuracy, low deployment cost, and no need to modify the core inference code. It can effectively solve the industry pain points of performance modeling difficulties, insufficient simulation accuracy, and unreliable data support in deep learning inference scenarios.

[0133] In the above embodiments, by combining precise matching with nearest neighbor backoff in the simulation process, high-precision prediction can be guaranteed in sampled scenarios, and reliable continuous simulation can be achieved in unsampled scenarios. This completely solves the modeling challenges caused by dynamic input and dynamic concurrency in inference scenarios, and significantly improves the applicability and accuracy of inference performance simulation.

[0134] For example, using the standard self-attention operator in the Llama-2 8B model as the simulation object, and considering the operational characteristics of the incremental decoding stage of large models, the method involved in the embodiments of this application is illustrated. During the inference process of large models, the memory access overhead and computational proportion of the self-attention operator fluctuate significantly with changes in sequence length. The method in the embodiments of this application can effectively improve the simulation accuracy of inference performance in scenarios with unknown shapes. The specific implementation process is as follows:

[0135] I. Real-time detection triggering and raw operation data capture.

[0136] When the scheduler of the vLLM inference framework issues inference batch tasks, the detection module in this application can intervene in the native inference data stream in real time to capture the actual running scenario and core operator shape parameters from the attention operator. This exemplary embodiment selects the incremental decoding stage of the Llama-2 8B model as the test scenario. The detection module captures the core parameters of this inference scenario: the inference batch size is 32, the number of attention heads is 32, and the dimension of a single attention head is 128.

[0137] Furthermore, the detection module performs dimensionality analysis on the second input tensor of the current inference input to extract the key shape features corresponding to this inference, namely, the batch dimension is 32 and the average context length of the sequence is 1024, thus completing the parameter capture work before simulation.

[0138] II. Quantitative storage of online speed measurement and operator computation efficiency.

[0139] This stage relies on the native reasoning process of the reasoning framework to complete the calculation of operator theoretical computational load, real-time time consumption acquisition, computational efficiency conversion, and structured data storage. The specific steps are as follows:

[0140] The first step is to calculate the theoretical floating-point operation volume of the operator. Based on the standard operation logic of the self-attention operator and combined with the operator shape parameters captured in this case, the theoretical floating-point operation volume of the core operation module of the self-attention operator in this incremental decoding scenario is calculated to be 0.016 trillion floating-point operations.

[0141] The second step was to collect the actual execution time of the operator. Using a high-precision hardware timer, the execution process of the self-attention operator was measured throughout the inference process, and the actual execution time of the operator under the current GPU hardware environment was found to be 0.4 milliseconds.

[0142] The third step is to convert the standardized computational efficiency. Based on the theoretical floating-point operation volume and the actual execution time obtained above, the actual computing power of this operator in the current scenario is calculated to be 40 trillion floating-point operations per second. Due to the large amount of video memory data reading and KV cache read / write operations in the incremental decoding stage of large models, the overall inference performance is limited by the video memory bandwidth bottleneck. The actual computing power of the operator is far lower than the theoretical peak computing power of the hardware device, which is in line with the characteristics of real inference operation.

[0143] The fourth step is to associate and store efficiency data. Key feature information collected in this study, such as operator type, inference stage, batch size, context length, and measured computational efficiency, is bound and associated, and stored in a pre-set efficiency data set in a structured form. This forms standardized efficiency sample data that can be used for subsequent simulation matching, enriching the efficiency data sample.

[0144] III. Simulation retreat matching and performance prediction for scenes with unknown shapes.

[0145] To verify the adaptability and simulation accuracy of the proposed solution to unsampled dynamic operator shape scenarios, this embodiment sets a new simulation requirement: predicting the inference execution time of the self-attention operator and the system throughput of the Llama-2 8B model in a scenario with a context length of 2048.

[0146] According to the efficiency data retrieval, the currently stored valid sample data only contains two sets of data: a context length of 1024, corresponding to a computational efficiency of 40 trillion floating-point operations per second, and a context length of 4096, corresponding to a computational efficiency of 35 trillion floating-point operations per second. There is currently no matching parameter with a context length of 2048, thus triggering the nearest neighbor backoff simulation strategy of this application.

[0147] Based on the high-dimensional feature space constructed by the operator shape parameters, the target simulation parameters and the sampled parameters in the library are matched for key dimensions similarity. The differences in core dimensions that affect the operator computation efficiency, such as batch size and context length, are comprehensively compared. The sampled parameters corresponding to the context length of 1024 are determined to be the fourth operator shape parameters with the smallest difference from the target features of this simulation and the most similar computational features. The second computational efficiency associated with it is retrieved as the benchmark efficiency of this simulation.

[0148] Based on the operator shape parameters of the target simulation scene, the theoretical floating-point operation volume of the self-attention operator is calculated to be 0.032 trillion floating-point operations when the context length is 2048. Combining the benchmark computational efficiency obtained through matching, and through the correspondence between computational volume, computing power and execution time, the prediction execution time of the operator in the unknown shape scene is calculated to be 0.8 milliseconds, thus completing the high-precision performance simulation prediction of the dynamic unsampled scene.

[0149] Compared to traditional offline simulation schemes that rely on offline tools and manually preset hyperparameters, this application achieves online real-time sampling through the native data stream of the inference framework. The standardized efficiency data constructed can accurately capture the true performance degradation patterns of self-attention operators at different inference stages and with different sequence lengths. For dynamic inference scenarios such as the decrease in operator computation efficiency caused by increasing context length and increased KV cache read / write overhead, this application effectively solves the technical shortcomings of traditional offline simulations—namely, their inability to adapt to dynamic operator shapes, their inability to fit real scheduling coupling relationships, and their large simulation errors—through a simulation strategy combining precise matching and nearest neighbor backoff. This achieves continuous, high-precision performance simulation of unknown dynamic inference scenarios, significantly improving the accuracy, versatility, and stability of large-model inference performance simulation, and providing reliable data support for scheduling optimization and capacity planning of inference systems.

[0150] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0151] Based on the same inventive concept, this application also provides an apparatus for determining efficiency data in a reasoning framework to implement the above-described method for determining efficiency data in a reasoning framework. The solution provided by this apparatus is similar to the solution described in the above-described method. Therefore, the specific limitations of one or more embodiments of the apparatus for determining efficiency data in a reasoning framework provided below can be found in the limitations of the method for determining efficiency data in a reasoning framework described above, and will not be repeated here.

[0152] In one exemplary embodiment, such as Figure 5 As shown, an efficiency data determination device in a reasoning framework is provided, comprising:

[0153] The computation execution module 501 is used to perform computation on the first input tensor based on the target computation operator during the inference process of the inference framework, and to measure the execution process to obtain the execution process.

[0154] The parsing module 502 is used to parse the dimension information of the first input tensor to obtain the first operator shape parameter of the first input tensor;

[0155] The efficiency determination module 503 is used to determine the computational efficiency of the target computational operator based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware.

[0156] The association module 504 is used to associate and store the shape parameters of the first operator with the computational efficiency to form efficiency data for inference performance simulation.

[0157] In one embodiment, the efficiency determination module 503 is specifically used for:

[0158] The theoretical floating-point computational complexity is determined based on the shape parameters of the first operator and the operator type of the target computational operator.

[0159] The computational efficiency of the target computation operator is determined based on the theoretical floating-point operation volume, the execution time, and the maximum computing power of the hardware.

[0160] In one embodiment, the efficiency determination module 503 is specifically used for:

[0161] Obtain the operator type of the target computation operator;

[0162] Based on the operator type and the shape parameters of the first operator, the theoretical floating-point operation quantity corresponding to the target computation operator is determined.

[0163] In one embodiment, the target calculation operator is an online velocity detection operator;

[0164] The online speed detection operator is used to replace the native computation operator in the inference framework, execute the computation function of the native computation operator, and measure the execution process.

[0165] In one embodiment, the first input tensor includes input tensors with various different operator shape parameters;

[0166] Specifically, for input tensors with various different operator shape parameters, each step as described in claim 1 is executed to obtain and store the computational efficiency of the target computation operator corresponding to the various different operator shape parameters.

[0167] In one embodiment, the device further includes: a simulation module for:

[0168] During the inference performance simulation, the performance of the inference scenario is simulated based on the efficiency data and the maximum computing power of the hardware to obtain the prediction time and / or system throughput.

[0169] In one embodiment, the simulation module is specifically used for:

[0170] During the inference performance simulation, the second operator shape parameter of the second input tensor is obtained;

[0171] If the efficiency data contains a third operator shape parameter that is consistent with the second operator shape parameter, then the performance simulation of the inference scenario is performed based on the first computing efficiency stored in association with the third operator shape parameter and the maximum computing power of the hardware.

[0172] If there is no parameter in the efficiency data that is consistent with the shape parameter of the second operator, then the fourth operator shape parameter that is closest to the shape parameter of the second operator is determined from the efficiency data, and the performance simulation of the inference scenario is performed based on the second computing efficiency stored in association with the fourth operator shape parameter and the maximum computing power of the hardware.

[0173] In one embodiment, the fourth operator shape parameter is one or more, and the simulation module is specifically used for:

[0174] For a single fourth operator shape parameter, the computational efficiency of storing the fourth operator shape parameter in association is taken as the second computational efficiency;

[0175] For multiple fourth operator shape parameters, the computational efficiency of the multiple fourth operator shape parameters stored separately is integrated and calculated by means calculation or similarity weighted fusion to obtain the second computational efficiency;

[0176] Based on the second computational efficiency and the maximum computing power of the hardware, a performance simulation of the inference scenario is performed.

[0177] The modules in the efficiency data determination device within the aforementioned reasoning framework can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0178] In one exemplary embodiment, a computer device is provided, the internal structure of which can be as shown in the figure. Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an efficient data determination method within an inference framework.

[0179] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0181] During the inference process, the first input tensor is computed based on the target computation operator, and the execution process is measured to obtain the execution time of the execution process.

[0182] The dimension information of the first input tensor is parsed to obtain the first operator shape parameter of the first input tensor;

[0183] The computational efficiency of the target computational operator is determined based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware.

[0184] The shape parameters of the first operator are associated with the computational efficiency and stored to form efficiency data for inference performance simulation.

[0185] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0186] The theoretical floating-point computational complexity is determined based on the shape parameters of the first operator and the operator type of the target computational operator.

[0187] The computational efficiency of the target computation operator is determined based on the theoretical floating-point operation volume, the execution time, and the maximum computing power of the hardware.

[0188] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0189] Obtain the operator type of the target computation operator;

[0190] Based on the operator type and the shape parameters of the first operator, the theoretical floating-point operation quantity corresponding to the target computation operator is determined.

[0191] In one embodiment, the target calculation operator is an online velocity detection operator;

[0192] The online speed detection operator is used to replace the native computation operator in the inference framework, execute the computation function of the native computation operator, and measure the execution process.

[0193] In one embodiment, the first input tensor includes input tensors with various different operator shape parameters;

[0194] Specifically, for input tensors with various operator shape parameters, the steps described above for obtaining efficiency data are executed respectively, so as to obtain and store the computational efficiency of the target computation operator corresponding to various operator shape parameters.

[0195] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0196] During the inference performance simulation, the performance of the inference scenario is simulated based on the efficiency data and the maximum computing power of the hardware to obtain the prediction time and / or system throughput.

[0197] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0198] During the inference performance simulation, the second operator shape parameter of the second input tensor is obtained;

[0199] If the efficiency data contains a third operator shape parameter that is consistent with the second operator shape parameter, then the performance simulation of the inference scenario is performed based on the first computing efficiency stored in association with the third operator shape parameter and the maximum computing power of the hardware.

[0200] If there is no parameter in the efficiency data that is consistent with the shape parameter of the second operator, then the fourth operator shape parameter that is closest to the shape parameter of the second operator is determined from the efficiency data, and the performance simulation of the inference scenario is performed based on the second computing efficiency stored in association with the fourth operator shape parameter and the maximum computing power of the hardware.

[0201] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0202] For a single fourth operator shape parameter, the computational efficiency of storing the fourth operator shape parameter in association is taken as the second computational efficiency;

[0203] For multiple fourth operator shape parameters, the computational efficiency of the multiple fourth operator shape parameters stored separately is integrated and calculated by means calculation or similarity weighted fusion to obtain the second computational efficiency;

[0204] Based on the second computational efficiency and the maximum computing power of the hardware, a performance simulation of the inference scenario is performed.

[0205] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps and processes shown in the above method embodiments.

[0206] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the various steps and processes shown in the above method embodiments.

[0207] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0208] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0209] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining efficiency data in a reasoning framework, characterized in that, The method includes: During the inference process, the first input tensor is computed based on the target computation operator, and the execution process is measured to obtain the execution time of the execution process. The dimension information of the first input tensor is parsed to obtain the first operator shape parameter of the first input tensor; The computational efficiency of the target computational operator is determined based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware. The shape parameters of the first operator are associated with the computational efficiency and stored to form efficiency data for inference performance simulation. During the inference performance simulation, the performance of the inference scenario is simulated based on the second operator shape parameter of the second input tensor, the efficiency data, and the maximum computing power of the hardware to obtain the prediction time and / or the system throughput. The second operator shape parameter is used to determine the target operator shape parameter with the smallest difference from the second operator shape parameter in the efficiency data, and the target computational efficiency corresponding to the target operator shape parameter is used to perform performance simulation on the inference scenario.

2. The method according to claim 1, characterized in that, Determining the computational efficiency of the target computational operator based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware includes: The theoretical floating-point computational complexity is determined based on the shape parameters of the first operator and the operator type of the target computational operator. The computational efficiency of the target computation operator is determined based on the theoretical floating-point operation volume, the execution time, and the maximum computing power of the hardware.

3. The method according to claim 2, characterized in that, The step of determining the theoretical floating-point computational complexity based on the shape parameters of the first operator and the operator type of the target computational operator includes: Obtain the operator type of the target computation operator; Based on the operator type and the shape parameters of the first operator, the theoretical floating-point operation quantity corresponding to the target computation operator is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The target calculation operator is an online velocity detection operator; The online speed detection operator is used to replace the native computation operator in the inference framework, execute the computation function of the native computation operator, and measure the execution process.

5. The method according to any one of claims 1 to 3, characterized in that, The first input tensor includes input tensors with various different operator shape parameters; Specifically, for input tensors with various different operator shape parameters, each step as described in claim 1 is executed to obtain and store the computational efficiency of the target computational operator corresponding to the various different operator shape parameters.

6. The method according to claim 1, characterized in that, The performance simulation of the inference scenario based on the shape parameters of the second operator, the efficiency data, and the maximum computing power of the hardware, to obtain the prediction time and / or system throughput, includes: Obtain the second operator shape parameter of the second input tensor; If the efficiency data contains a third operator shape parameter that is consistent with the shape parameter of the second operator, then the performance simulation of the inference scenario is performed based on the first computing efficiency stored in association with the third operator shape parameter and the maximum computing power of the hardware. If there is no parameter in the efficiency data that is consistent with the shape parameter of the second operator, then the fourth operator shape parameter that is closest to the shape parameter of the second operator is determined from the efficiency data, and the performance simulation of the inference scenario is performed based on the second computing efficiency stored in association with the fourth operator shape parameter and the maximum computing power of the hardware.

7. The method according to claim 6, characterized in that, The fourth operator shape parameter can be single or multiple. The second computational efficiency based on the storage associated with the fourth operator shape parameter, and the performance simulation of the inference scenario based on the maximum computing power of the hardware, include: For a single fourth operator shape parameter, the computational efficiency of storing the fourth operator shape parameter in association is taken as the second computational efficiency; For multiple fourth operator shape parameters, the computational efficiency of the multiple fourth operator shape parameters stored separately is integrated and calculated by means calculation or similarity weighted fusion to obtain the second computational efficiency; Based on the second computational efficiency and the maximum computing power of the hardware, a performance simulation of the inference scenario is performed.

8. An efficiency data determination device in a reasoning framework, characterized in that, The device includes: The computation execution module is used to perform computation on the first input tensor based on the target computation operator during the inference process of the inference framework, and to measure the execution process to obtain the execution time of the execution process; The parsing module is used to parse the dimension information of the first input tensor to obtain the first operator shape parameter of the first input tensor; An efficiency determination module is used to determine the computational efficiency of the target computational operator based on the shape parameters of the first operator, the execution time, and the maximum computing power of the hardware. The association module is used to associate and store the shape parameters of the first operator with the computational efficiency to form efficiency data for inference performance simulation. The simulation module is used to perform performance simulation of the inference scenario based on the shape parameters of the second operator, the efficiency data, and the maximum computing power of the hardware during the inference performance simulation process, so as to obtain the prediction time and / or the system throughput. The second operator shape parameter is used to determine the target operator shape parameter with the smallest difference from the second operator shape parameter in the efficiency data, and the target computational efficiency corresponding to the target operator shape parameter is used to perform performance simulation on the inference scenario.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images