System for predicting inference performance of large language models using multiple inference engines, and the control method thereof

Abstract

A multi-inference engine-based large-scale language model (LLM) inference performance prediction system is disclosed. The multi-inference engine-based large-scale language model (LLM) inference performance prediction system includes a user interface unit that selects parameter information for LLM inference performance prediction and visualizes and provides performance indicators accordingly, and a server that calculates LLM inference performance indicators based on parameter information and database information input through the user interface unit, quantifies implementation performance differences by inference engine and hardware, calculates engine descriptor vectors and kernel-specific correction coefficients for each inference engine based on this, and utilizes the calculated engine descriptor vectors and kernel-specific correction coefficients to reflect performance differences by inference engine in the LLM inference performance indicators to improve the accuracy of LLM inference performance. Accordingly, by utilizing engine descriptor vectors and kernel-specific correction coefficients for various inference engine and hardware combinations, the accuracy of TTFT / TBT / ITL / TPS / E2E predictions can be improved by combining actual measurement-based correction and latency estimation, reproducibility can be guaranteed with versioned metadata, and decision-making can be supported by providing visualization, timelines, and scoreboards.

Description

Technology Field

[0001] The present invention relates to a system for predicting large-scale language model inference performance based on multiple inference engines and a control method thereof. More specifically, it relates to a system for predicting large-scale language model inference performance based on multiple inference engines and a control method thereof, which predicts user-perceived metrics such as Time to First Token (TTFT), Inter-token Latency (TBT / ITL), Tokens Per Second (TPS), and End-to-End Latency (E2E) in advance by reflecting implementation differences between engines in a serving environment of a large-scale language model (LLM), and provides the prediction results and token output simulations through a user interface. Background Technology

[0002] Conventional large-scale language model performance prediction methods have predominantly relied on static approximations or Roofline coupled regression based on FLOPs and memory bandwidth, but they struggled to adequately reflect the influence of internal engine optimization and scheduling.

[0003] For example, due to differences in engine-specific optimization features such as vLLM’s PagedAttention and Continuous Batching, DeepSpeed ​​Inference’s model parallelization, and Prefill / Decode separation optimization, there are large performance variations between engines even with the same model and GPU, but there was no integrated tool to comprehensively predict this in advance.

[0004] Furthermore, existing services tended to limit themselves to VRAM capacity or approximate token speed estimations, failing to reflect variables such as engine version, scheduling, and kernel optimization, and lacked predictive support for IoT / edge devices.

[0005] In particular, as research and development have primarily focused on performance measurement for NVIDIA GPUs, there is a lack of performance prediction systems for individual devices such as IoT and edge devices. Furthermore, since the inference engines supported by specific GPUs and edge devices vary, users face the inconvenience of having to manually check support status through the websites and manuals of each individual inference engine.

[0006] Accordingly, there has been an increased need for a system to derive performance prediction results for various development environments. The problem to be solved

[0007] The objective of the present invention is to provide a multi-inference engine-based large-scale language model inference performance prediction system and a control method thereof, which predict user-perceived metrics such as TTFT (Time to First Token), TBT / ITL (Inter-token Latency), TPS (Tokens Per Second), and E2E (End-to-End Latency) in a serving environment of a large-scale language model (LLM) by reflecting implementation differences between engines, and provide the prediction results and token output simulation through a user interface. means of solving the problem

[0008] A multi-inference engine-based large-scale language model (LLM) inference performance prediction system according to one embodiment of the present invention for achieving such objectives includes a user interface unit that selects parameter information for LLM inference performance prediction and visualizes and provides performance indicators accordingly, and a server that calculates the LLM inference performance indicators based on the parameter information and database information input through the user interface unit, quantifies the difference in implementation performance by inference engine and hardware, calculates engine descriptor vectors and kernel-specific correction coefficients for each inference engine based on this, and utilizes the calculated engine descriptor vectors and kernel-specific correction coefficients to reflect the performance difference by each inference engine in the LLM inference performance indicators to improve the accuracy of LLM inference performance.

[0009] Here, the server comprises a storage unit for storing the database information, a processor for calculating the LLM inference performance indicator and reflecting the performance difference per inference engine in the LLM inference performance indicator using the calculated engine descriptor vector and kernel-specific correction coefficient, and a manager interface unit for managing the multi-inference engine-based large-scale language model inference performance prediction system, wherein the database information may include performance database information storing the implementation performance difference per inference engine and hardware calculated by the processor and the engine descriptor vector and kernel-specific correction coefficient, and device database information storing the hardware-specific specifications and supported inference engine information.

[0010] In addition, the processor can calculate the LLM inference performance indicator by calculating at least one of the Time to First Token (TTFT), Inter-token Latency (TBT / ITL), Tokens Per Second (TPS), and End-to-End Latency (E2E) based on the parameter information and database information.

[0011] Additionally, the processor can calculate the engine descriptor vector and kernel-specific correction coefficients by measuring at least one of the prefill / decode operation cost, scheduler policy and batch formation delay, sampler and post-processing overhead, multi-device communication and host-hardware data movement and tokenizer and driver or runtime influence.

[0012] In addition, the processor can calculate the LLM inference performance indicator based on at least one of prefill time, decode time per token, kernel relative efficiency, scheduler overhead, and wait delay estimate.

[0013] In addition, the above parameter information may include the type and number of the hardware, the type and version information of the inference engine, the type of the LLM, the size of the parameter information, the prompt length, the degree of simultaneous connection, and batch-related information.

[0014] Meanwhile, a control method for a large-scale language model (LLM) inference performance prediction system based on multiple inference engines according to an embodiment of the present invention comprises: receiving parameter information for predicting LLM inference performance from a user; calculating the LLM inference performance indicator based on the parameter information and database information; quantifying the difference in implementation performance by inference engine and hardware and calculating the engine descriptor vector and kernel correction coefficient for each inference engine based thereon; utilizing the calculated engine descriptor vector and kernel correction coefficient to reflect the performance difference by each inference engine in the LLM inference performance indicator to increase the accuracy of the LLM inference performance; and visualizing and providing the LLM inference performance indicator reflecting the performance difference by each inference engine.

[0015] Here, the database information may include performance database information storing the difference in implementation performance by the inference engine and hardware calculated by the processor, the engine descriptor vector, and the correction coefficient by kernel, and device database information storing the hardware specifications and supported inference engine information.

[0016] In addition, the step of calculating the LLM inference performance indicator may calculate the LLM inference performance indicator by calculating at least one of the Time to First Token (TTFT), Inter-token Latency (TBT / ITL), Tokens Per Second (TPS), and End-to-End Latency (E2E) based on the parameter information and database information.

[0017] In addition, the step of calculating the engine descriptor vector and kernel-specific correction coefficient per inference engine may calculate the engine descriptor vector and kernel-specific correction coefficient by measuring at least one of the prefill / decode operation cost, scheduler policy and batch formation delay, sampler and post-processing overhead, multi-device communication and host-hardware data movement and tokenizer and driver or runtime influence.

[0018] In addition, the step of improving the accuracy of LLM inference performance by reflecting it in the above LLM inference performance indicator can improve the accuracy of the LLM inference performance based on at least one of prefill time, decode time per token, kernel relative efficiency, scheduler overhead, and wait delay estimate.

[0019] Meanwhile, a computer-readable recording medium according to one embodiment of the present invention includes a program for executing a control method for a multi-inference engine-based large-scale language model (LLM) inference performance prediction system on a computer. Effects of the invention

[0020] According to various embodiments of the present invention as described above, by utilizing engine descriptor vectors and kernel-specific correction coefficients for various inference engine and hardware combinations, the accuracy of TTFT / TBT / ITL / TPS / E2E predictions can be improved by combining actual-based correction and latency estimation, reproducibility can be ensured with versioned metadata, and decision-making can be supported by providing visualization, timelines, and scoreboards. Brief explanation of the drawing

[0021] FIG. 1 is a block diagram illustrating the configuration of a large-scale language model (LLM) inference performance prediction system based on a multi-inference engine according to an embodiment of the present invention. FIG. 2 is a drawing illustrating the specific configuration of a server according to one embodiment of the present invention. FIG. 3 is a diagram illustrating a specific operation process for predicting the inference performance of a large-scale language model (LLM) based on a multi-inference engine according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a control method for a large-scale language model (LLM) inference performance prediction system based on a multi-inference engine according to an embodiment of the present invention. FIG. 5 is a block diagram illustrating the specific configuration of the server shown in FIG. 2. FIG. 6 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention. Specific details for implementing the invention

[0022] The present invention will be described in more detail below with reference to the drawings. Furthermore, in describing the present invention, detailed descriptions of related known functions or configurations are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the invention. Additionally, the terms described below are defined considering their functions in the present invention, and these may vary depending on the intentions or relationships of the user or operator. Therefore, their definitions should be based on the content throughout this specification.

[0023] FIG. 1 is a block diagram illustrating the configuration of a large-scale language model (LLM) inference performance prediction system based on a multi-inference engine according to an embodiment of the present invention.

[0024] Referring to FIG. 1, a multi-inference engine-based large-scale language model (LLM) inference performance prediction system (10) according to one embodiment of the present invention may include a user interface unit (100) and a server (200).

[0025] Here, the user interface unit (100) may include a user terminal device or a UI / UX displayed on the user terminal device, and may select parameter information for predicting LLM inference performance and provide visualized performance indicators accordingly.

[0026] In addition, parameter information may include the type and number of hardware, the type and version information of the inference engine, the type of LLM, the size of the parameter information, the prompt length, the degree of concurrent connection, and batch-related information.

[0027] Specifically, the user interface section (100) may provide a screen or user interface for selecting or receiving serving conditions including hardware information such as the type and number of hardware (GPU / edge / embedded, etc.), memory constraints, and connection structure (NVLink / PCIe / Ethernet, etc.), inference engine information such as the type and version of the engine (vLLM, DeepSpeed, TGI, TensorRT-LLM, etc.), batch policy (continuous / FCFS, etc.), precision / quantization (FP16 / BF16 / INT8 / FP8), LLM type information such as model type, parameter scale, and maximum context length, prompt length, expected / limited output length, concurrent connection, and batch-related information.

[0028] Additionally, the server (200) can calculate an LLM inference performance indicator based on parameter information and database information input through the user interface, quantify the difference in implementation performance by inference engine and hardware, calculate an engine descriptor vector and a correction coefficient by kernel based on the same, and perform an operation to increase the accuracy of LLM inference performance by reflecting the difference in performance by inference engine in the LLM inference performance indicator using the calculated engine descriptor vector and the correction coefficient by kernel.

[0029] FIG. 2 is a drawing illustrating the specific configuration of a server according to one embodiment of the present invention.

[0030] Referring to FIG. 2, the server (200) includes a storage unit (210), a processor (220), and an administrator interface unit (230). The storage unit (210) stores database information, the processor (220) calculates an LLM inference performance indicator, and uses the calculated engine descriptor vector and kernel-specific correction coefficients to reflect performance differences by inference engine in the LLM inference performance indicator. The administrator interface unit (230) can manage a large-scale language model inference performance prediction system (10) based on multiple inference engines.

[0031] Here, the database information may include performance database information that stores the difference in implementation performance per inference engine and hardware calculated by the processor (220), engine descriptor vectors, and correction coefficients per kernel, and a device database that stores hardware specifications and supported inference engine information.

[0032] Specifically, performance database information may store actual benchmark measurements, correction factors, engine descriptor vectors, predicted outputs, and version metadata, and may store information on changes resulting from engine updates, driver changes, or hardware replacements.

[0033] In addition, device database information records device-specific specifications (architecture, memory, computational performance, connection structure, power) and supported engines, models, and functions (precision / quantization / special functions) to provide compatibility determination and combination guides.

[0034] In addition, the administrator interface section (230) can perform data quality management (verification, correction, approval), scoreboard policy and version management, engine / model metadata update, and user participation benchmark (remote execution / collection) management.

[0035] Meanwhile, the processor (220) can calculate an LLM inference performance indicator by calculating at least one of the first token generation time (TTFT, Time to First Token), the inter-token generation time (TBT / ITL, Inter-token Latency), the number of tokens per second (TPS, Tokens Per Second), and the total execution time (E2E, End-to-End Latency) based on parameter information and database information.

[0036] Specifically, the processor (220) normalizes the parameter information (hardware type / number, inference engine / version, LLM type / parameter scale / max context, prompt length, expected / limited output length, concurrent connection, batch-related information, etc.) received from the user interface unit (100), and then queries the database information and performance database information of the storage unit (210). It determines combination compatibility (support engine / precision / connection structure / memory constraint) from the device database information and loads the set of correction coefficients (kernel relative efficiency, scheduler characteristic value, communication coefficient, etc.) and engine descriptor vector of the corresponding combination (or proximity combination) from the performance database information. If there is no actual value for the corresponding combination, it uses the default coefficient or nearest coefficient defined in the policy, and in this case, the reliability attribute is included in the result.

[0037] Additionally, the processor (220) can obtain a first approximation by separating the prefill and decode to reduce calculation errors.

[0038] Here, prefill time can be estimated by considering the computational amount based on the number of layers, hidden dimensions, head configuration, prompt length, and precision of the model, as well as the device's effective throughput / memory bandwidth, and fixed overheads such as tokenizer preprocessing and graph capture / compilation delays can be treated as a separate term.

[0039] In addition, the decode time per token can reflect dependencies on context length, precision, and batch size, while taking KV cache access, attention / MLP kernel latency, and sampler / post-processing costs as base terms.

[0040] And, the processor (220) can obtain the sum of the first token generation time (TTFT) and the inter-token generation time (TBT / ITL) and can calculate the number of tokens per second (TPS, Tokens Per Second) and the total execution time (E2E, End-to-End Latency) from this.

[0041] Here, since the first approximation does not sufficiently reflect the differences in engine implementation and environment, the processor (220) can apply the engine descriptor vector of the performance DB (attention kernel type / placement policy / precision·quantization / parallelization strategy / tokenizer·sampler implementation / engine·runtime·driver version, etc.) and kernel-specific correction coefficients (kernel relative efficiency·scheduler characteristic value·communication coefficient, etc.) to the first approximation.

[0042] In actual service, depending on the request arrival rate λ, concurrency, batch upper limit B, and output length distribution, batch formation delay and preemption / fairness policies can have a direct impact on TTFT and TBT. Accordingly, the processor (220) can use scheduler characteristics (batch formation time distribution, overhead by policy) of performance database information and combine input λ, B and output length statistics to approximate the waiting delay Wq and the change in decode efficiency according to the batch.

[0043] As a result, it can be refined into the following mathematical formula.

[0044]

[0045] Additionally, the processor (220) can perform distribution estimation of TTFT and TBT by retrieving the measurement distribution (variance / skewness of kernel delay, scheduler delay, and communication delay) from the performance DB or interpolating it with the basic distribution assumption, calculate the required percentile, and synthesize a token timeline (first token time + sequence of intervals between tokens) based on the delay value of the percentile. Furthermore, the processor (220) can provide a token output simulation along with the calculation of the indicator.

[0046] Meanwhile, when model parallelization (TP / PP, sharding) or a multi-GPU / node configuration is used, the processor (220) uses the communication coefficients (round-trip delay, effective bandwidth, pattern-based overhead) of the performance DB to T pre ,T dec Communication terms such as NVLink / PCIe / Ethernet can be corrected, and bottleneck terms vary depending on the interconnect type and topology, and the impact can be reflected mainly as an increase in TBT and E2E.

[0047] In addition, timeline samples of TTFT, TBT, TPS, and E2E, as well as the correction factors / engine descriptor vectors / version metadata used, can be mapped to request IDs and stored in the performance DB with version management, thereby enabling recalculation under the same conditions and tracking of changes resulting from engine / driver updates.

[0048] Meanwhile, the processor (220) can calculate the engine descriptor vector and kernel-specific correction coefficient by measuring at least one of the prefill / decode operation cost, scheduler policy and batch formation delay, sampler and post-processing overhead, multi-device communication and host-hardware data movement and tokenizer and driver or runtime influence.

[0049] Specifically, the processor (220) can collect finely detailed micro-indicators for combinations specified by engine, device, model, precision, and placement policy through a micro-benchmark executor (runner).

[0050] Here, the collected raw measurements are processed in the following order: (a) preprocessing and normalization, (b) outlier / noise removal, (c) calculation of summary values ​​by category, and (d) estimation of the engine descriptor vector (A) and kernel-specific correction coefficients (B). The obtained A and B are recorded in the performance database of the storage unit along with version, timestamp, and combined metadata, and are subsequently applied as product / additive correction values ​​for prefill, decode, scheduler, and communication terms during the performance prediction stage.

[0051] Specifically, the processor (220) can determine the target model (number of layers, hidden dimension, maximum context), engine (version, placement policy, kernel option), device (memory / bandwidth / interconnect), precision / quantization), prompt / output length, placement upper limit / concurrent connection.

[0052] And, the processor (220) can generate a set of micro workloads with a combination of prefill / decode separation, multiple context lengths, and different batch sizes / arrival rates, and for each workload, can measure prefill / decode, scheduler / batch, sampler / postprocessing, communication / transmission, and environmental factors on individual channels (including timestamp, frequency, variance / box values).

[0053] Here, prefill refers to execution time per layer, average latency and variability due to changes in prompt length, decode refers to latency per token (ITL), context length, and batch size sensitivity, collection methods refer to engine profiler / kernel timestamps, GPU events / timelines, and sample-level iterative measurements, and the summary is T pre Decomposition term (attention / MLP / memory), T dec It refers to the decomposition term and context sensitivity coefficient.

[0054] As described above, the processor (220) measures and quantifies each item to generate an engine descriptor vector (A) and a kernel-specific correction coefficient (B), and uses this as a performance prediction correction value so that the implementation differences between the engine and hardware can be accurately reflected in TTFT / TBT / TPS / E2E.

[0055] Meanwhile, the processor (220) can calculate an LLM inference performance indicator based on at least one of prefill time, decode time per token, kernel relative efficiency, scheduler overhead, and wait delay estimate.

[0056] That is, the processor (220) normalizes the parameter information (device type / number, engine / version, model / parameter scale / max context, prompt length, expected / limited output length, concurrent connection rate, batch limit, etc.) transmitted from the user interface unit (100), and then queries the device DB and performance DB of the storage unit (210). The device DB provides combination compatibility (supported engine / precision / connection structure / memory limit), and the performance DB can provide correction values ​​such as kernel relative efficiency, scheduler characteristic value, and communication coefficient of the corresponding combination (or close combination).

[0057] And, the processor (220) can obtain a first-order approximation by separating prefill and decode, and since the first-order approximation does not sufficiently reflect the difference in internal engine implementation, the processor (220) can correct the prefill / decode time by multiplying and adding using kernel relative efficiency accumulated in the performance DB, and reflect overhead and batch formation delay according to the engine's scheduler policy (continuous batch / FCFS / preemption / fairness, etc.) as correction terms, and the micro-benchmark measures and summarizes the batch formation time distribution, waiting time distribution, and overhead by policy (scheduler characteristic value), and the processor (220) can sum the TTFT and TBT based on this as shown in the following mathematical formula.

[0058]

[0059] Here, is the composite value of the scheduler overhead including the sampler and post-processing.

[0060] FIG. 3 is a diagram illustrating a specific operation process for predicting the inference performance of a large-scale language model (LLM) based on a multi-inference engine according to an embodiment of the present invention.

[0061] Referring to FIG. 3, the system includes a web interface (310) accessible to users and an administrator interface or management console (320) accessible to administrators, and through the web-based user interface and management console, data input, prediction execution, and result management can be performed.

[0062] The user can input parameter information for predicting LLM inference performance through the web interface (310) and check the predicted performance indicators (TTFT, TBT / ITL, TPS, E2E, etc.). The web interface (310) includes a scoreboard that provides performance comparison results, through which the relative performance rankings of various engine and hardware combinations can be checked.

[0063] In addition, the administrator can manage the operational status of the system through the management console (administrator interface section) (320), verify micro-benchmark results collected from users, manage the quality of performance data, update scoreboard policies, and manage engine / model version information.

[0064] Additionally, the performance database (330) stores actual data, correction coefficients, engine descriptor vectors, and performance prediction results (TTFT, TBT, TPS, E2E, etc.) collected from various inference engines and hardware combinations. The performance database (330) is referenced by the micro-benchmark module (360) and the performance prediction module (350), and records version control information of the correction coefficients and prediction results together.

[0065] Additionally, the device database (340) stores specification information (computational performance, memory capacity, interconnect structure, power, etc.) and supported inference engine information for each hardware (e.g., GPU, edge device, embedded device, etc.). The device database (340) is used as a standard to determine the compatibility of combinations when performing predictions or to guide the user to possible settings.

[0066] The performance prediction module (350) is a core processing unit that calculates LLM inference performance indicators by combining parameter information input by the user and information stored in the database (330, 340). This module performs the approximate calculation of prefill time and decode time per token, correction using engine descriptor vectors and kernel-specific correction coefficients, reflection of scheduler overhead and wait delay estimates, and calculation of final indicators (TTFT, TBT, TPS, E2E).

[0067] Additionally, the micro-benchmark module (360) is a component that quantitatively measures implementation differences for each inference engine and device to generate data necessary for correction. The module includes benchmark code and kernel code, and measures prefill / decode operation time, scheduler policy / batch formation delay, sampler and post-processing overhead, multi-device communication / host-hardware data movement, and the effects of tokenizers, runtimes, and drivers, converting them into numerical correction coefficients. The results measured by the micro-benchmark module (360) are stored in the performance database (330) and reflected in the correction process of the performance prediction module (350).

[0068] Meanwhile, the performance database (330), device database (340), performance prediction module (350), and micro benchmark module (360) may be included in the server (200), the web interface (310) corresponds to the user interface section (100), and the management console (320) corresponds to the administrator interface section (230).

[0069] FIG. 4 is a flowchart illustrating a control method for a large-scale language model (LLM) inference performance prediction system based on a multi-inference engine according to an embodiment of the present invention.

[0070] Referring to FIG. 4, a control method for a large-scale language model (LLM) inference performance prediction system based on a multi-inference engine according to an embodiment of the present invention may include the steps of: receiving parameter information for LLM inference performance prediction from a user (S110); calculating an LLM inference performance indicator based on the parameter information and database information (S120); quantifying the difference in implementation performance by inference engine and hardware and calculating an engine descriptor vector and a kernel correction coefficient based thereon (S130); utilizing the calculated engine descriptor vector and kernel correction coefficient to reflect the difference in performance by inference engine in the LLM inference performance indicator to increase the accuracy of the LLM inference performance (S140); and visualizing and providing an LLM inference performance indicator that reflects the difference in performance by inference engine (S150).

[0071] Here, the database information may include performance database information that stores the difference in implementation performance per inference engine and hardware calculated by the processor (220), engine descriptor vectors, and correction coefficients per kernel, and a device database that stores hardware specifications and supported inference engine information.

[0072] Additionally, the step (S120) of calculating LLM inference performance indicators can calculate LLM inference performance indicators by calculating at least one of the Time to First Token (TTFT), Inter-token Latency (TBT / ITL), Tokens Per Second (TPS), and End-to-End Latency (E2E) based on parameter information and database information.

[0073] Additionally, the step (S130) of calculating engine descriptor vectors and kernel-specific correction coefficients per inference engine can calculate engine descriptor vectors and kernel-specific correction coefficients by measuring at least one of prefill / decode operation costs, scheduler policy and batch formation delays, sampler and post-processing overhead, multi-device communication and host-hardware data movement and tokenizer and driver or runtime influences.

[0074] Additionally, the step (S140) of improving the accuracy of LLM inference performance by reflecting it in the LLM inference performance indicator can improve the accuracy of LLM inference performance based on at least one of prefill time, decode time per token, kernel relative efficiency, scheduler overhead, and wait delay estimate.

[0075] In addition, the operation process processed by the processor (220) described above can be equally applied to the control method of a multi-inference engine-based large-scale language model (LLM) inference performance prediction system according to one embodiment of the present invention.

[0076] FIG. 5 is a block diagram illustrating the specific configuration of the server shown in FIG. 2.

[0077] Referring to FIG. 5, the server (200) may include a storage unit (210), a processor (220), and an administrator interface unit (230), and the processor (220) may control the overall operation of the server (200).

[0078] Specifically, the processor (220) includes RAM (221), ROM (222), main CPU (223), graphics processing unit (224), first to n interfaces (225-1 to 225-n), and a bus (226).

[0079] RAM (221), ROM (222), main CPU (223), graphics processing unit (224), first to n interfaces (225-1 to 225-n), etc. can be connected to each other via a bus (226).

[0080] The first to n interfaces (225-1 to 225-n) are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.

[0081] The main CPU (223) accesses the storage unit (210) and performs booting using the O / S stored in the storage unit (210). Then, it performs various operations using various programs, content, data, etc. stored in the storage unit (210).

[0082] In particular, the main CPU (223) calculates the LLM inference performance indicator based on the parameter information and database information input through the user interface, quantifies the difference in implementation performance by inference engine and hardware, calculates the engine descriptor vector and kernel correction coefficient for each inference engine based on this, and utilizes the calculated engine descriptor vector and kernel correction coefficient to reflect the difference in performance by each inference engine in the LLM inference performance indicator, thereby increasing the accuracy of the LLM inference performance.

[0083] A set of instructions for booting the system is stored in the ROM (222). When a turn-on command is input and power is supplied, the main CPU (223) copies the O / S stored in the storage unit (210) to the RAM (221) according to the instructions stored in the ROM (222), and executes the O / S to boot the system. When booting is complete, the main CPU (223) copies various application programs stored in the storage unit (210) to the RAM (221), and executes the application programs copied to the RAM (221) to perform various operations.

[0084] The graphics processing unit (224) generates a screen containing various objects such as icons, images, and text using a calculation unit (not shown) and a rendering unit (not shown). The calculation unit (not shown) calculates attribute values ​​such as coordinate values, shape, size, and color for each object to be displayed according to the layout of the screen based on a received control command. The rendering unit (not shown) generates a screen of various layouts containing objects based on the attribute values ​​calculated by the calculation unit (not shown).

[0085] In particular, the graphics processing unit (224) can implement objects generated by the main CPU (223) into a GUI (Graphic User Interface), icon, user interface screen, etc.

[0086] Meanwhile, the operation of the above-described processor (220) can be performed by a program stored in the storage unit (210).

[0087] The storage unit (210) stores various data, such as an O / S (Operating System) software module for operating the server (200) and various multimedia content.

[0088] In particular, the storage unit (210) may include a software module for increasing the accuracy of LLM inference performance by calculating the LLM inference performance indicator based on the parameter information and database information input through the user interface unit, quantifying the difference in implementation performance by inference engine and hardware, calculating the engine descriptor vector and kernel correction coefficient for each inference engine based on the above, and reflecting the difference in performance by each inference engine in the LLM inference performance indicator using the calculated engine descriptor vector and kernel correction coefficient.

[0089] FIG. 6 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention.

[0090] Referring to FIG. 6, the storage unit (210) may include an LLM inference performance indicator calculation module (211), an engine descriptor vector and kernel-specific correction coefficient calculation module (212), and an LLM inference performance indicator reflection module (213).

[0091] Here, the LLM inference performance indicator calculation module (211) can calculate an LLM inference performance indicator based on parameter information and database information entered through the user interface.

[0092] In addition, the engine descriptor vector and kernel-specific correction coefficient calculation module (212) can quantify the difference in implementation performance by inference engine and hardware and calculate the engine descriptor vector and kernel-specific correction coefficient based thereon.

[0093] In addition, the LLM inference performance indicator reflection module (213) can improve the accuracy of LLM inference performance by utilizing the calculated engine descriptor vector and kernel-specific correction coefficient to reflect the performance difference between inference engines in the LLM inference performance indicator.

[0094] Meanwhile, a non-transitory computer-readable medium storing a program that sequentially performs the control method according to the present invention may be provided.

[0095] A non-transient readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on non-transient readable media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0096] In addition, although the bus is not depicted in the aforementioned block diagram for the server, a processor such as a CPU or microprocessor may be further included to calculate an LLM inference performance indicator based on parameter information and database information input through the user interface, quantify the implementation performance differences by inference engine and hardware, calculate engine descriptor vectors and kernel correction coefficients by inference engine based on this, and utilize the calculated engine descriptor vectors and kernel correction coefficients to reflect the performance differences by inference engine in the LLM inference performance indicator, thereby increasing the accuracy of LLM inference performance.

[0097] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0098] 10: Multi-inference Engine Based Large-Scale Language Model (LLM) Inference Performance Prediction System 100: User Interface Section 200: Server 210: Storage 220: Processor 230: Administrator Interface Section

Claims

Claim 1 A multi-inference engine-based large-scale language model (LLM) inference performance prediction system comprising: a user interface unit that selects parameter information for LLM inference performance prediction and provides visualized performance indicators accordingly; and a server that calculates the LLM inference performance indicators based on the parameter information and database information input through the user interface unit, quantifies the difference in implementation performance by inference engine and hardware, calculates engine descriptor vectors and kernel correction coefficients by inference engine based on the same, and utilizes the calculated engine descriptor vectors and kernel correction coefficients to reflect the difference in performance by inference engine in the LLM inference performance indicators to improve the accuracy of LLM inference performance. Claim 2 A multi-inference engine-based large-scale language model inference performance prediction system according to claim 1, wherein the server comprises: a storage unit for storing the database information; a processor for calculating the LLM inference performance indicator and reflecting the performance difference per inference engine in the LLM inference performance indicator using the calculated engine descriptor vector and kernel-specific correction coefficients; and an administrator interface unit for managing the multi-inference engine-based large-scale language model inference performance prediction system; wherein the database information comprises performance database information storing the implementation performance difference per inference engine and hardware calculated by the processor and the engine descriptor vector and kernel-specific correction coefficients, and device database information storing the hardware-specific specifications and supported inference engine information. Claim 3 A large-scale language model inference performance prediction system based on multiple inference engines, wherein, in paragraph 2, the processor calculates at least one of Time to First Token (TTFT), Inter-token Latency (TBT / ITL), Tokens Per Second (TPS), and End-to-End Latency (E2E) based on the parameter information and database information to calculate the LLM inference performance indicator. Claim 4 A large-scale language model inference performance prediction system based on multiple inference engines, wherein, in paragraph 3, the processor measures at least one of prefill / decode operation cost, scheduler policy and batch formation delay, sampler and post-processing overhead, multiple device communication and host-hardware data movement and tokenizer and driver or runtime influence to calculate the engine descriptor vector and kernel-specific correction coefficient. Claim 5 A large-scale language model inference performance prediction system based on multiple inference engines, wherein, in claim 4, the processor calculates the LLM inference performance indicator based on at least one of prefill time, decode time per token, kernel relative efficiency, scheduler overhead, and wait delay estimate. Claim 6 A large-scale language model inference performance prediction system based on multiple inference engines, wherein, in claim 5, the parameter information includes the type and number of the hardware, the type and version information of the inference engine, the type of the LLM, the size of the parameter information, the prompt length, the degree of concurrency, and batch-related information. Claim 7 A control method for a multi-inference engine-based large-scale language model (LLM) inference performance prediction system comprising: receiving parameter information for LLM inference performance prediction from a user; calculating the LLM inference performance indicator based on the parameter information and database information; quantifying the difference in implementation performance by inference engine and hardware and calculating the engine descriptor vector and kernel correction coefficient for each inference engine based thereon; utilizing the calculated engine descriptor vector and kernel correction coefficient to reflect the difference in performance by each inference engine in the LLM inference performance indicator to increase the accuracy of the LLM inference performance; and visualizing and providing the LLM inference performance indicator reflecting the difference in performance by each inference engine. Claim 8 A control method for a multi-inference engine-based large-scale language model (LLM) inference performance prediction system, wherein, in claim 7, the database information includes performance database information storing differences in implementation performance by inference engine and hardware, and correction coefficients by engine descriptor vector and kernel, and device database information storing specifications and supported inference engine information by hardware. Claim 9 A control method for a large-scale language model (LLM) inference performance prediction system based on multiple inference engines, wherein the step of calculating the LLM inference performance indicator is to calculate the LLM inference performance indicator by calculating at least one of the Time to First Token (TTFT), Inter-token Latency (TBT / ITL), Tokens Per Second (TPS), and End-to-End Latency (E2E) based on the parameter information and database information. Claim 10 A control method for a multi-inference engine-based large-scale language model (LLM) inference performance prediction system, wherein the step of calculating engine descriptor vectors and kernel-specific correction coefficients for each inference engine is to calculate engine descriptor vectors and kernel-specific correction coefficients by measuring at least one of prefill / decode operation costs, scheduler policy and batch formation delays, sampler and post-processing overheads, multi-device communication and host-hardware data movement, and tokenizer and driver or runtime influences. Claim 11 A control method for a multi-inference engine-based large-scale language model (LLM) inference performance prediction system, wherein the step of improving the accuracy of LLM inference performance by reflecting it in the above LLM inference performance indicator is to improve the accuracy of said LLM inference performance based on at least one of prefill time, decode time per token, kernel relative efficiency, scheduler overhead, and wait delay estimate. Claim 12 A computer-readable recording medium storing a program for executing on a computer a control method for a multi-inference engine-based large-scale language model (LLM) inference performance prediction system described in any one of claims 7 through 11.

Images