Deep learning reasoning service performance analysis method based on kernel function trajectory

Abstract

The invention provides a kernel function trajectory-based deep learning inference service performance analysis method, which comprises the following steps of: based on service indexes and hardware theoretical computing power acquired from a production cluster, defining floating point operation times per request (FPR) index to quantify service resource efficiency, and identifying high FPR hotspot services; positioning a reasoning iteration candidate boundary based on a GPU kernel function trajectory, verifying iteration integrity through fingerprint matching and chi-square test, and calculating a second reasoning iteration number IIPS and a model reasoning efficiency MIE; aiming at calculation-intensive operators on the key path, combining a dynamic Roofline model to estimate an operator theoretical performance upper limit, and based on actual execution time, calculating efficiency and a BottleScore index to identify a key bottleneck operator; and outputting targeted optimization suggestions according to analysis results of service efficiency analysis, model efficiency analysis and operator efficiency analysis. According to the method, the inference behavior pattern can be automatically identified from massive kernel trajectories, and the efficiency loss of each level is quantified.

Description

Technical Field

[0001] This invention relates to the field of performance optimization and observability technology for artificial intelligence systems. Specifically, it relates to a performance profiling method and system for large language model (LLM) inference services in large-scale GPU clusters, and in particular, a non-intrusive performance bottleneck diagnosis and optimization guidance method based on hierarchical collaborative analysis and kernel function trajectory modeling. Background Technology

[0002] With the widespread application of LLM in scenarios such as multi-turn dialogue, code generation, and document summarization, the demand for GPU computing resources in LLM inference services has increased dramatically. To support high-concurrency, low-latency online inference, the industry generally adopts dedicated inference frameworks such as vLLM, TensorRT-LLM, and sglang, and deploys them in large-scale clusters containing hundreds of heterogeneous GPUs. However, the LLM inference process is highly dynamic and cross-stack complex: its execution flow spans the service layer (request scheduling), the model layer (pre-filling / decoding stage), and the operator layer (GPU kernel), and is coupled with multiple factors such as model architecture, inference framework strategy, and hardware characteristics.

[0003] Generally, the complete inference process of an LLM consists of two main phases. First, in the Prefill (P) phase, the LLM processes and encodes the complete input sequence in parallel. During this phase, a large Key-Value Cache (KV Cache) is built and prefilled to support token generation in the next phase and reduce redundant computations in the attention module. Second, in the Decode (D) phase, the LLM iteratively predicts the output token using historical entries from the KV Cache, while incrementally updating the KV Cache. To manage operations in LLM inference to handle large numbers of queries, various inference frameworks are used in various MaaS applications, such as vLLM, sglang, TensorRT-LLM, and RTP-LLM. These frameworks typically use a PD separation strategy to reduce latency or a PD fusion strategy to improve GPU utilization.

[0004] In detail, each stage involves a large number of kernel instances running on the GPU. Each kernel performs a specific function called an operator. Generally, key operators in LLM inference can be divided into three main categories: Linear computation operators: These operators satisfy linearity (superposition and homogeneity). They provide LLM with capabilities such as feature transformation (e.g., translation and scaling), feature combination, and information routing, constituting the basic computational load of LLM. Common linear operators include matrix multiplication, convolution, and linear projection.

[0005] Nonlinear computation operators: They introduce nonlinear transformation capabilities. The powerful expressiveness of LLMs comes from the alternating stack of linear and nonlinear operators. Common nonlinear operators include typical activation functions (e.g., Softmax and GELU) and normalization functions.

[0006] Communication operators: These operators handle communication, data sharing, and synchronization between GPUs within the same node and across multiple nodes. This includes collective operations such as AllReduce, All-Gather, and ReduceScatter.

[0007] Current GPU profiling techniques evaluate the utilization of various GPU resources at multiple levels, such as stream multiprocessor (SM) occupancy, thread bundle (warp) execution efficiency, memory access coalescence, and interconnect bandwidth. Representative profiling methods are summarized as follows: System-level timeline profilers: Use GPU runtime hooks and CPU event sampling to correlate kernel launches, memory transfers, and host activities, ultimately building a timeline profile across multiple LLM inference life cycles. For example, NsightSystem uses the CUPTI Activity API to generate an execution timeline.

[0008] Kernel-level analyzers: Collect hardware counter metrics and cache behavior related to each kernel execution to diagnose stall and bottlenecks in its active phases. For example, HPCToolkit combines runtime callbacks and optional GPU program counter sampling to link performance issues to kernel source code.

[0009] Event-driven monitors: Capture selected events to evaluate various GPU performance metrics at different levels, usually with low runtime overhead. For example, DCGM polls NVML counters to record CPU and GPU events with sub-microsecond precision.

[0010] Full-stack heterogeneous profilers: Track execution on CPUs, GPUs, and interconnects by inserting interception layers or instrumentation into libraries or compilers during compilation, such as TAU.

[0011] There are three key limitations that hinder the widespread application of these GPU profiling systems on MaaS platforms. First, tools like Nsight Compute and TAU often cause high overhead and intrusive side effects during kernel execution, even potentially changing the behavior of the kernel in a parallel environment. Second, the data from high-level profilers is often too limited to help analyze bottlenecks in inference tasks or guide kernel optimization for LLMs. Third, most current GPU profiling methods ignore how business-level operations in LLM services affect the performance of the entire platform.

[0012] Although some works try to analyze operator performance through Roofline model, they do not consider the actual weight of different kinds of operators in the inference critical path, leading to deviation of optimization direction.

[0013] Therefore, there is an urgent need in the market for a lightweight, non-intrusive and multi-level correlation capable performance profiling framework that can automatically identify inference behavior patterns from massive kernel trajectory and quantify the efficiency loss at each level, thereby providing actionable optimization guidance for LLM inference services. SUMMARY

[0014] In view of the defects in the prior art, the purpose of the present application is to provide a kernel function trajectory-based deep learning inference service performance profiling method and system.

[0015] According to the kernel function trajectory-based deep learning inference service performance profiling method provided by the present application, the following steps are included: Data collection step: collect service indicators and GPU kernel trajectories from production clusters; Service efficiency analysis step: based on service indicators and hardware theoretical computing power, define the FPR (floating point operation per request) indicator to quantify service resource efficiency, and identify hot services with high FPR; Model efficiency analysis step: based on GPU kernel function trajectory, locate inference iteration candidate boundary, and verify iteration integrity through fingerprint matching and chi-square test, calculate IIPS (inference iteration per second) and MIE (model inference efficiency); Operator efficiency analysis step: for computation-intensive operators on the critical path, estimate the theoretical performance upper limit of the operator based on the dynamic Roofline model, and calculate the efficiency degree of the operator based on the actual execution time and identify key bottleneck operators with the BottleScore indicator; Optimization guidance generation step: according to the analysis results of service efficiency analysis, model efficiency analysis and operator efficiency analysis, output targeted optimization suggestions.

[0016] Preferably, the FPR (floating point operation per request) indicator has the following calculation formula:

[0017] Where F peak represents the theoretical peak floating point operation capability defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during service running; N GPU represents the number of GPU devices allocated to the service; QPS represents the query throughput of the service.

[0018] Preferably, the service efficiency analysis is to build a cumulative distribution function CDF of the floating point operation number per request FPR for each framework independently, and mark the service with FPR value exceeding P85 percentile as a hotspot service.

[0019] Preferably, the model efficiency analysis step comprises: An iterative boundary preliminary screening step: using the host-to-device and device-to-host memory copy events in the GPU kernel trace as the start and end anchors of the candidate iteration; A fingerprint matching and verification step: for each candidate iteration, calculating the observed execution time proportion of each operator type j , from the global operator distribution aggregated by the same model in the entire trace , and calculating the chi-square goodness-of-fit test statistic and the weighted mean square error of its execution fingerprint aggregated globally by the model:

[0020] By setting a threshold value p>0.05 to filter out incomplete iteration fragments and Loss as a complete inference iteration; A model efficiency quantification step: based on the successfully identified complete iterations, calculating the inference iteration number per second IIPS, defining the model inference efficiency MIE index, and the formula is:

[0021] Where, F peak represents the theoretical peak floating point operation capability defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during the service running; N GPU represents the number of GPU devices allocated to the service.

[0022] Preferably, the operator arithmetic intensity calculation formula in the operator efficiency analysis step is as follows:

[0023] Where, represents the total floating point operation amount of the operator, represents the data movement amount required by the operator; Based on the peak computing performance π and the memory bandwidth β of the target GPU, the theoretical performance upper limit of the operator is determined according to The size relationship of the ridge point , and the efficiency degree is calculated through the measured throughput , and the calculation formulas are as follows:

[0024] ​

[0025] wherein, represents the actual execution time.

[0026] Preferably, the BottleScore index calculation formula in the operator efficiency analysis step is: *

[0027] wherein, represents the operator efficiency degree, is the time proportion of the operator in the critical path, used for bottleneck influence weighted ranking of the operator.

[0028] Preferably, the output of the optimization guidance generation step includes service resource configuration adjustment, inference framework selection, operator-level parameter tuning suggestion, and decision basis for cluster-wide resource scheduling and hardware procurement.

[0029] According to the present application, a deep learning inference service performance profiling system based on kernel function trajectory is provided, comprising: a data collection module: collecting service indicators and GPU kernel trajectories from a production cluster; a service efficiency analysis module: based on service indicators and hardware theoretical computing power, defining the FPR index to quantify service resource efficiency, and identifying hot services with high FPR; a model efficiency analysis module: based on GPU kernel function trajectory, locating inference iteration candidate boundaries, and verifying iteration integrity through fingerprint matching and chi-square test, calculating IIPS and MIE; an operator efficiency analysis module: for computation-intensive operators on the critical path, combining dynamic Roofline model to estimate the theoretical performance upper limit of the operator, and calculating the efficiency degree based on the actual execution time and identifying key bottleneck operators with the BottleScore index; an optimization guidance generation module: according to the analysis results of service efficiency analysis, model efficiency analysis and operator efficiency analysis, outputting targeted optimization suggestions.

[0030] Preferably, the FPR index per request is calculated as:

[0031] wherein, F peak represents the theoretical peak floating point operation capability defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during service running; N GPUrepresents the number of GPU devices allocated to the service; QPS represents the query throughput of the service.

[0032] Preferably, the service efficiency analysis is to build a cumulative distribution function CDF of floating point operations per request FPR for each framework independently, and mark the service with FPR value exceeding P85 percentile as a hotspot service.

[0033] Compared with the prior art, the present application has the following beneficial effects: 1. The present application first establishes a three-layer unified index system (FPR / MIE / BottleScore) of service-model-operator, realizes end-to-end performance bottleneck attribution and model-hardware affinity matching; 2. The present application does not need to modify the model code or inject probes, but only relies on the memory transfer events and operator sequence fingerprints in the kernel track, so as to accurately identify the inference iteration boundary, and the accuracy reaches 98.7%; 3. The present application fuses the operator efficiency loss and key path weight through the BottleScore index, accurately locates the bottleneck operator with the greatest impact on end-to-end performance, and avoids invalid optimization; 4. The present application uses random sampling and lightweight tracking to control the performance overhead within 5%, which is suitable for continuous monitoring in production environment; 5. The present application successfully identifies the performance bottleneck of 265 services in a real 350 GPU cluster; in the Qwen2.5-7B optimization case, through framework-hardware collaborative optimization, the execution efficiency of the key operator is improved by 1.56 times, and the GPU utilization is improved by 1.12 times. BRIEF DESCRIPTION OF DRAWINGS

[0034] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings: Figure 1 The overall framework structure diagram of LLM-Prof proposed by the present application is shown in the figure; Figure 2 The flowchart of inference iteration period detection is shown in the figure; Figure 3 The schematic diagram of a single inference iteration period containing kernel functions is shown in the figure; Figure 4 The pseudocode for inference iteration period detection is shown in the figure; Figure 5 The bottleneck score box plot of different kinds of operators is shown in the figure. DETAILED DESCRIPTION

[0035] The application will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of the present application.

[0036] The entire process of the present application can be triggered on demand at service runtime without restarting the task, and the total overhead is controlled within 5%; the remaining service efficiency analysis SEA, model efficiency analysis MEA, operator efficiency analysis OEA and optimization decision can be executed offline, and the time of single complete profiling and evaluation is controlled within 3 minutes per case, ensuring its practicability in large-scale production environment.

[0037] Embodiment 1 According to the kernel function trajectory-based deep learning inference service performance profiling method provided by the present application, it is applied to a large-scale LLM inference cluster containing multiple servers and hundreds of GPUs, such as shown in Figure 1 The method comprises the following steps: Data acquisition step: service indicators and GPU kernel trajectories are collected from the production cluster. The Activity API of NVIDIACUPTI (CUDA Profiling Tools Interface) is used to capture complete GPU kernel function execution trajectories (Kernel Trace) with very low runtime overhead (<1%), and service-level indicators (such as QPS, GPU utilization) are obtained through monitoring systems such as Prometheus.

[0038] Service efficiency analysis step: based on service-level monitoring indicators (QPS, GPU utilization) and hardware theoretical computing power, define the FPR (Flops Per Request) indicator to quantify service resource efficiency and identify hot services with high FPR. This step is the first layer of analysis step of the present application, and its core goal is to establish a model-independent, horizontally comparable service-level efficiency measurement standard. Traditional indicators such as QPS cannot reflect the true resource consumption intensity. Therefore, the present application defines FPR (Flops Per Request) as the core indicator, and its calculation formula is:

[0039] Where, F peak represents the theoretical peak floating point operation capability defined by the GPU hardware architecture (for example, NVIDIA A100 is 312 TFLOPs); U GPU ∈[0,1] represents the average GPU utilization observed during service running; N GPUrepresents the number of GPU devices allocated to the service; QPS represents the query throughput of the service (unit: times / second). This step unifies hardware computing power, resource occupation, and business output in one dimension, effectively eliminating the natural differences caused by parameter quantity and sequence length in different models. The larger the FPR value, the more computing resources are consumed to process a single request, and the lower the service efficiency. To avoid misjudgment caused by differences in scheduling strategies of different inference frameworks (such as vLLM and RTP-LLM), this step independently builds the cumulative distribution function (CDF) of FPR for each framework, and marks services with FPR values exceeding the P85 percentile as hotspot services, thereby systematically identifying targets with high optimization potential.

[0040] Model efficiency analysis step: based on the GPU kernel function trajectory, the memory transfer event (memcpyHtoD / memcpyDtoH) is used to locate the inference iteration candidate boundary, and the operator execution sequence fingerprint matching and chi-square test are used to verify the iteration integrity, and then the inference iteration times per second (IIPS) and model inference efficiency (MIE) are calculated. The model efficiency analysis step is the second layer analysis of the present application, which aims to solve the fundamental problem of "fuzzy iteration boundary" in LLM inference. As shown in Figure 2 , the main process of MEA analysis is shown, including memory event retrieval, fingerprint detection, and cycle verification. Due to the widespread application of advanced scheduling strategies such as dynamic batching, PD fusion, and PD separation, the forward propagation process of a single inference lacks clear start and end markers in the kernel trajectory, making performance evaluation at the model level extremely difficult. The design purpose of MEA is to automatically and accurately restore complete inference iterations from the chaotic kernel function execution flow. Its workflow includes: Iteration boundary preliminary screening step: using the memory copy events of host to device (memcpyHtoD) and device to host (memcpyDtoH) in the GPU kernel trajectory as the start and end anchor points of the candidate iteration, which is based on the domain knowledge that LLM inference necessarily involves input / output data transmission.

[0041] Fingerprint matching and verification step: as shown in Figure 3 , a typical operator iteration fingerprint of LLM inference is shown, including the operator execution sequence (such as Attention, MLP, RMSNorm, Activation, and communication operators, etc.) in the complete single iteration cycle candidate segment. For each candidate iteration, we calculate the observed execution time proportion of each operator type j . The expected proportion It is derived from the global operator distribution aggregated throughout the entire tracking process using the same model. The chi-square goodness-of-fit test statistic and WMSE (weighted mean squared error) of its execution fingerprint, aggregated with the global model, are calculated.

[0042] Incomplete iteration fragments are filtered out by setting a threshold (p≥0.05), and Loss<τ indicates a high degree of consistency with the expected computation pattern. The specific iteration detection pseudocode is as follows: Figure 4 Segments that pass these two checks will be retained as valid reasoning iterations, ensuring that subsequent analysis is based on a semantically complete forward propagation process.

[0043] Model efficiency quantification steps: Based on a successful complete iteration, calculate the number of inference iterations per second (IIPS). Then, define the Model Inference Efficiency (MIE) metric, with the formula:

[0044] Based on the iterative characteristics of inference, MIE (Model Iteration) shifts the evaluation granularity from "requests" at the service layer down to "model iterations," accurately revealing the true execution efficiency of a specific model under a given hardware-framework combination. A higher MIE value indicates lower model efficiency. By comparing the MIE of the same model on different hardware, hardware selection and model deployment strategies can be effectively guided.

[0045] Operator efficiency analysis steps: For computationally intensive operators on the critical path, estimate the theoretical performance ceiling of the operator using a dynamic Roofline model, and calculate the efficiency based on the actual execution time. Furthermore, the BottleScore metric is introduced to comprehensively account for efficiency losses (1-). ) and time percentage ( This paper employs a weighted ranking of operators to identify key bottleneck operators. Operator efficiency analysis, the third layer of analysis in this invention, aims to penetrate the kernel black box, accurately pinpoint the microscopic root causes of performance bottlenecks, and provide interpretable quantitative evidence. Operator efficiency analysis includes two key innovations: first, the introduction of an efficiency degree to measure the performance potential of individual operators; and second, the proposal of a BottleScore metric for end-to-end weighted ranking of operators based on their impact.

[0046] Firstly, for any kernel function, according to its execution function, it is classified into different kinds of operators O, and the operator efficiency analysis module estimates its theoretical performance upper limit on the corresponding GPU hardware through the Roofline model. According to the input / output tensor shape of the operator, the system estimates the total floating point operation amount and the required data movement amount (unit: GB), and further calculates the arithmetic intensity: (unit: TFLOPs / Byte).

[0047] Combined with the peak computing performance π (unit: TFLOPs / s) and the memory bandwidth β (unit: GB / s) of the target GPU, the theoretical performance upper limit of the operator can be obtained:

[0048] , On this basis, the OEA uses the actual execution time in the kernel trajectory to calculate the measured throughput as: , and defines the efficiency degree as:

[0049] The efficiency degree ranges from 0 to 1, which directly reflects the gap between the operator and its theoretical performance limit. An operator with a very low value means that there is a significant optimization space, which may be caused by poor memory access mode, insufficient use of Tensor Core or poor software implementation.

[0050] However, the efficiency degree alone is not enough to guide the optimization priority. An operator with very low efficiency but small time consumption proportion has limited optimization benefits; on the contrary, an operator with acceptable efficiency but occupying a large amount of time may not be the best target. Therefore, the present application proposes a BottleScore index, which combines the intrinsic efficiency loss of the operator with its external weight in the end-to-end delay. The specific calculation formula is: *

[0051] wherein, is the time proportion of the operator O in the critical path. The higher the BottleScore value, the greater the drag of the operator on the overall performance. For example, Figure 5 ​As shown, the box plot of bottleneck scores of different kinds of operators is shown, which shows the difference in performance efficiency and bottleneck score contribution of several main operator types Linear, Attention operator. Through this index, OEA can generate an ordered bottleneck operator list to provide clear and operable optimization targets for developers, further guiding which types of operators need to be optimized as the priority optimization target. Specifically, considering the Linear several projection kernel functions of the computationally intensive operator (such as , , , , ) and Attention operator, blind optimization is avoided.

[0052] Optimization guidance generation step: According to the three-layer analysis results, output targeted optimization suggestions, including service resource configuration adjustment, inference framework selection, operator-level parameter tuning, etc. Convert the structured insights generated by three-layer analysis into quantifiable and implementable performance improvement. Not only support deep tuning for a single service, but also provide data-driven decision-making basis for resource scheduling, framework selection and hardware procurement within the cluster range through horizontal comparison analysis.

[0053] In the systematic evaluation of 265 LLM inference services in the production cluster, LLM-Prof showed strong diagnostic and guidance capabilities. First, in terms of framework-hardware affinity analysis, the same model (such as the Qwen series) under different inference frameworks (vLLM, SGLang, RTP-LLM) has significant differences in GPU resource utilization efficiency. For example, the SGLang framework generally exhibits higher GPU utilization than RTP-LLM when processing medium QPS load, which is attributed to the use of more efficient PD fusion strategies, kernel function implementations, and KV-Cache access strategies by the framework. The MEA module quantitatively confirms this observation by calculating the MIE index of each service and finds the sensitivity of the GPU architecture in the instance (such as Ampere vs Hopper), i.e., there is a three-way affinity problem of "framework-model-hardware".

[0054] Secondly, in the case analysis of fixed models using different frameworks and GPU types, a Qwen2.5-7B LLM inference service instance was identified as having poor performance. Layer 1 (SEA) analysis showed that its FPR value was significantly higher than other services using the same framework, initially indicating inefficient resource utilization. Layer 2 (MEA) further confirmed that the service's Model Inference Efficiency (MIE) was abnormal, revealing a mismatch between its inference framework and the currently deployed GPU hardware. Layer 3 (OEA) provided the most refined diagnosis: through efficiency and BottleScore analysis of key operators, the system precisely located the issue of… (The qkv projection operator in linear operators) is the highest priority bottleneck operator, and its efficiency is... With a score of only 0.347 and a GPU utilization of 0.822, it indicates that it is in a severely memory-constrained state, and its arithmetic strength is far from reaching the hardware's "bottom line".

[0055] Based on the above performance analysis metrics, the decision module provides optimization suggestions: On the one hand, adjust the configuration of the inference framework, switching from the vllm inference framework to the sglang inference framework, and using more aggressive operator fusion strategies and precision compression strategies to increase the computational density of GEMM operators and improve their arithmetic strength; on the other hand, migrate the service instance from the current GPU to a GPU with higher memory bandwidth to alleviate its memory bottleneck.

[0056] Based on the above suggestions, after implementing the above optimizations using different frameworks and GPU hardware, the bottleneck operator " "efficiency" The efficiency improved significantly from 0.347 to 0.540, achieving a 1.56x increase in operator efficiency. Thanks to this improved operator efficiency, GPU resources across the entire service were utilized more fully, with GPU utilization increasing from 82.2% to 91.9% (a 1.12x increase), while end-to-end inference latency was also significantly reduced. This case not only validates the accuracy of the LLM-Prof three-layer profiling framework but also demonstrates its powerful ability to guide real-world performance optimization.

[0057] This invention aims to overcome the problems of cross-stack fragmentation, ambiguous iteration boundaries, and inaccurate operator attribution in existing LLM inference profiling methods. It proposes a deep learning inference service performance profiling method and system based on kernel function trajectories. By constructing a hierarchical profiling mechanism of service layer, model layer, and operator layer, it achieves end-to-end performance bottleneck diagnosis from macro-level services to micro-level operators with extremely low overhead (<5%), and generates interpretable optimization suggestions to help analyze and make decisions to improve GPU resource utilization and LLM inference efficiency.

[0058] As an external performance profiling framework, the application does not depend on the source code of model training and deployment and the upper deep learning framework, has good universality and engineering landing property, and can be conveniently integrated into an existing distributed cluster management system.

[0059] Embodiment 2 The application also provides a deep learning inference service performance profiling system based on a kernel function trajectory, which can be implemented by performing the process steps of the deep learning inference service performance profiling method based on a kernel function trajectory, that is, the deep learning inference service performance profiling method based on a kernel function trajectory can be understood by those skilled in the art as a preferred embodiment of the deep learning inference service performance profiling system based on a kernel function trajectory.

[0060] According to the deep learning inference service performance profiling system based on a kernel function trajectory provided by the application, the constructed LLM-Prof performance profiling framework includes a data acquisition module, a service efficiency analyzer (SEA), a model efficiency analyzer (MEA), an operator efficiency analyzer (OEA) and an optimization decision module. Figure 1 As shown in the figure, the hierarchical architecture realizes accurate correlation from macro business indicators to micro hardware behavior through top-down collaborative analysis. The system first collects service indicators and GPU kernel trajectories from the production cluster; after the SEA screens the hot services, the MEA performs iterative identification and model efficiency evaluation on them; the OEA further analyzes the performance of the key operators; and finally the optimization decision module outputs the optimization scheme. Specifically, it includes: Data acquisition module: obtain service-level indicators through monitoring systems such as Prometheus, and capture GPU kernel function trajectories in a random sampling manner using CUPTIActivity API to ensure non-intrusiveness and low overhead.

[0061] Service efficiency analysis module (Service Efficiency Analyzer, SEA): calculate the floating point operation per request FPR indicator of each service, and select hot services with FPR higher than P85 percentile according to inference framework grouping. The floating point operation per request FPR indicator has the following formula:

[0062] Wherein, F peak represents the theoretical peak floating point operation capability defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during service running; N GPUdenotes the number of GPU devices allocated to the service; QPS denotes the query throughput of the service. The service efficiency analysis is to build a cumulative distribution function CDF of the floating point operations per request FPR for each framework independently, and mark the services with FPR values exceeding the P85 percentile as hotspot services.

[0063] Model Efficiency Analyzer (MEA): Identify complete inference iterations from kernel traces, compute IIPS and MIE, evaluate model-hardware matching. The model efficiency analysis module includes: Iteration boundary preliminary screening module: Use the host-to-device and device-to-host memory copy events in the GPU kernel trace as the start and end anchors of the candidate iterations; Fingerprint matching and verification module: For each candidate iteration, calculate the observed execution time proportion of each operator type j , from the global operator distribution aggregated from the same model throughout the trace , and calculate its chi-square goodness-of-fit test statistic and weighted mean square error with the execution fingerprint aggregated globally for the model:

[0064] By setting the threshold value p ≥ 0.05 to filter out incomplete iteration fragments and Loss as a complete inference iteration; Model efficiency quantification module: Based on the successfully identified complete iterations, calculate the inference iteration number per second IIPS, define the model inference efficiency MIE index, the formula is:

[0065] Where, F peak represents the theoretical peak floating point operation capability defined by the GPU hardware architecture; U GPU ∈ [0, 1] represents the average GPU utilization observed during the service running; N GPU denotes the number of GPU devices allocated to the service.

[0066] Operator Efficiency Analyzer (OEA): Roofline modeling is performed on key operators to calculate efficiency δO and BottleScore, and the ranking of bottleneck operators is output. The operator arithmetic intensity calculation formula in the operator efficiency analysis module is as follows:

[0067] Where, represents the total floating point operation of the operator, represents the data movement required by the operator; based on the peak computing performance π and memory bandwidth β of the target GPU, according to and ridge point The size relationship determination operator theory performance upper limit And through the measured throughput The calculation efficiency degree The calculation formula is as follows:

[0068]

[0069] Among them, The actual execution time is represented. The BottleScore index calculation formula in the operator efficiency analysis module is: *

[0070] Among them, The operator efficiency degree is represented, The time proportion of the operator in the critical path is used to sort the bottleneck influence of the operator.

[0071] The optimization guidance generation module integrates the three-layer analysis results to generate configuration optimization suggestions (such as BatchSize size, context and output input length, GPU card selection, framework parameter setting), and supports closed-loop verification. The output of the optimization guidance generation module includes service resource configuration adjustment, inference framework selection, operator-level parameter optimization suggestion, and decision basis for cluster-wide resource scheduling and hardware procurement.

[0072] Those skilled in the art know that in addition to implementing the system provided by the present application and each device, module and unit thereof in a pure computer readable program code manner, the system provided by the present application and each device, module and unit thereof can also be realized in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for realizing various functions can also be considered as structures within the hardware component; the devices, modules and units for realizing various functions can also be considered as both software modules realizing the method and structures within the hardware component.

[0073] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for performance profiling of deep learning inference services based on kernel function trajectories, characterized in that, include: Data acquisition steps: Collect service metrics and GPU kernel trajectories from the production cluster; Service efficiency analysis steps: Based on service metrics and theoretical hardware computing power, define the Floating Point Rate (FPR) metric to quantify service resource efficiency and identify hot services with high FPR. Model efficiency analysis steps: Locate candidate boundaries for inference iterations based on GPU kernel function trajectories, verify the completeness of the iterations through fingerprint matching and chi-square test, and calculate the number of inference iterations per second (IIPS) and model inference efficiency (MIE). Operator efficiency analysis steps: For computationally intensive operators on the critical path, estimate the theoretical performance ceiling of the operator using a dynamic Roofline model, and calculate the efficiency based on the actual execution time. Identify key bottleneck operators using the BottleScore metric; Optimization guidance generation steps: Based on the analysis results of service efficiency analysis, model efficiency analysis, and operator efficiency analysis, output targeted optimization suggestions.

2. The method for performance profiling of deep learning inference services based on kernel function trajectories according to claim 1, characterized in that, The formula for calculating the Floating Point Representation (FPR) index is as follows: Among them, F peak This represents the theoretical peak floating-point performance defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during service operation; N GPU This indicates the number of GPU devices allocated to the service; QPS indicates the query throughput of the service.

3. The method for performance profiling of deep learning inference services based on kernel function trajectories according to claim 1, characterized in that, Service efficiency analysis involves independently constructing a cumulative distribution function (CDF) for the number of floating-point operations per request (FPR) for each framework, and marking services with FPR values ​​exceeding the 85th percentile as hot services.

4. The method for performance profiling of deep learning inference services based on kernel function trajectories according to claim 1, characterized in that, The model efficiency analysis steps include: Iterative boundary screening step: Use host-to-device and device-to-host memory copy events in the GPU kernel trajectory as the start and end anchor points of candidate iterations; Fingerprint matching and verification steps: For each candidate iteration, calculate the proportion of observation execution time for each operator type j. The desired proportion is obtained from the global operator distribution aggregated from the same model throughout the tracking process. And calculate the chi-square goodness-of-fit test statistic and weighted mean square error of its execution fingerprint with the global aggregation of the model: Incomplete iteration fragments are filtered out by setting a threshold p≥0.05, and the loss is taken as a complete inference iteration; Model efficiency quantification steps: Based on a successfully identified complete iteration, calculate the number of inference iterations per second (IIPS), and define the model inference efficiency (MIE) metric, with the formula: Among them, F peak This represents the theoretical peak floating-point performance defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during service operation; N GPU This indicates the number of GPU devices allocated to this service.

5. The method for performance profiling of deep learning inference services based on kernel function trajectories according to claim 1, characterized in that, The formula for calculating the arithmetic strength of the operator in the operator efficiency analysis step is as follows: in, This represents the total number of floating-point operations performed by the operator. This represents the amount of data movement required by the operator; Based on the peak computing performance π and memory bandwidth β of the target GPU, according to With ridge point The size relationship determines the upper limit of the operator's theoretical performance. And through actual throughput computational efficiency The calculation formulas are as follows: in, Indicates the actual execution time.

6. The method for performance profiling of deep learning inference services based on kernel function trajectories according to claim 1, characterized in that, The formula for calculating the BottleScore index in the operator efficiency analysis step is as follows: * in, Indicates the operator efficiency. It represents the time percentage of the operator in the critical path, used to weight and rank operators based on their bottleneck impact.

7. The method for performance profiling of deep learning inference services based on kernel function trajectories according to claim 1, characterized in that, The output of the optimization guidance generation step includes service resource configuration adjustment, inference framework selection, operator-level parameter tuning suggestions, and decision-making basis for resource scheduling and hardware procurement within the cluster.

8. A performance profiling system for deep learning inference services based on kernel function trajectories, characterized in that, include: Data acquisition module: Collects service metrics and GPU kernel trajectories from the production cluster; Service Efficiency Analysis Module: Based on service metrics and theoretical hardware computing power, it defines the Floating Point Rate (FPR) metric to quantify service resource efficiency and identifies hot services with high FPR. Model efficiency analysis module: Based on the GPU kernel function trajectory, it locates the candidate boundary of inference iteration, verifies the integrity of the iteration through fingerprint matching and chi-square test, and calculates the number of inference iterations per second (IIPS) and the model inference efficiency (MIE). Operator efficiency analysis module: For computationally intensive operators on the critical path, it estimates the theoretical performance ceiling of the operator using a dynamic Roofline model and calculates the efficiency based on the actual execution time. Identify key bottleneck operators using the BottleScore metric; Optimization guidance generation module: Based on the analysis results of service efficiency analysis, model efficiency analysis, and operator efficiency analysis, it outputs targeted optimization suggestions.

9. The deep learning inference service performance profiling system based on kernel function trajectories according to claim 8, characterized in that, The formula for calculating the Floating Point Representation (FPR) index is as follows: Among them, F peak This represents the theoretical peak floating-point performance defined by the GPU hardware architecture; U GPU ∈[0,1] represents the average GPU utilization observed during service operation; N GPU This indicates the number of GPU devices allocated to the service; QPS indicates the query throughput of the service.

10. The deep learning inference service performance profiling system based on kernel function trajectories according to claim 8, characterized in that, Service efficiency analysis involves independently constructing a cumulative distribution function (CDF) for the number of floating-point operations per request (FPR) for each framework, and marking services with FPR values ​​exceeding the 85th percentile as hot services.

Images