A model inference performance optimization method, device and equipment and storage medium

Abstract

The application discloses a model inference performance optimization method and device, equipment and a storage medium, comprising: in response to receiving an inference request, identifying the structural characteristics of the target inference model, determining the type of the target inference model according to the structural characteristics, wherein the inference request comprises a request parameter, the target inference model is used to execute the inference request, determining the first resource allocation according to the type of the target inference model and the request parameter, obtaining system load data, determining the system load state level according to the system load data, determining the second resource allocation according to the first resource allocation and the system load state level, calculating the inference request according to the second resource allocation to obtain the inference result. Through the method, the resource utilization rate can be optimized and the system throughput can be improved.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a method, apparatus, device and storage medium for optimizing model inference performance. Background Technology

[0002] Current large-scale model inference services typically employ a hybrid architecture of prefill and decode, where both stages run on the same computing device. The prefill stage handles parallel computations of long sequences, which is a computationally intensive operation; while the decode stage requires frequent reads and writes to the key-value cache, making it a memory-intensive operation. Due to the significant differences in the computational characteristics of the two stages, this hybrid deployment model can easily lead to uneven GPU resource utilization, resulting in problems such as high first-token latency (TTFT) and limited throughput.

[0003] To optimize inference efficiency, a PD separation technique is proposed, decoupling the prefill and decode phases and deploying them on different devices. For example, by separating the two-stage computation tasks, computational resources are allocated selectively: the prefill phase prioritizes high-performance computing devices, while the decode phase is adapted to high-bandwidth memory devices. Open-source solutions such as vLLM improve throughput by optimizing key-value cache management. By independently scheduling the two-stage tasks and leveraging dedicated hardware to enhance overall performance, system throughput is increased.

[0004] Although the PD separation technology significantly improves inference efficiency by decoupling the pre-filling and decoding stages, the existing PD separation architecture adopts a static resource allocation strategy, which cannot dynamically adjust the computing resource allocation according to the real-time load. This can easily lead to resource mismatch in multi-task concurrent scenarios. Cross-stage data transmission relies on high-overhead serialization operations, resulting in a significant increase in the latency of the first token. Summary of the Invention

[0005] Therefore, it is necessary to provide a model inference performance optimization method, apparatus, device, and storage medium that can optimize resource utilization and improve system throughput in order to address the above-mentioned technical problems.

[0006] Firstly, a method for optimizing model inference performance is provided, including:

[0007] In response to receiving an inference request, the structural features of the target inference model are identified, and the type of the target inference model is determined based on the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request;

[0008] The first resource allocation is determined based on the type of the target inference model and the request parameters;

[0009] Obtain system load data and determine the system load status level based on the system load data;

[0010] The second resource allocation is determined based on the first resource allocation and the system load status level;

[0011] The inference request is calculated based on the second resource allocation to obtain the inference result.

[0012] Secondly, a model inference performance optimization apparatus is provided, applied to a model inference performance optimization method described in the first aspect, comprising:

[0013] The receiving module is used to respond to receiving an inference request, identify the structural features of the target inference model, and determine the type of the target inference model based on the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request;

[0014] The first allocation module is used to determine the first resource allocation based on the type of the target inference model and the request parameters;

[0015] The load level determination module is used to acquire system load data and determine the system load status level based on the system load data.

[0016] The second allocation module is used to determine the second resource allocation based on the first resource allocation and the system load status level.

[0017] The execution module is used to calculate the inference request based on the second resource allocation and obtain the inference result.

[0018] Thirdly, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the model inference performance optimization method described in the first aspect.

[0019] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the model inference performance optimization method described in the first aspect.

[0020] By implementing the aforementioned model inference performance optimization method, apparatus, device, and storage medium, this method receives an inference request containing a target inference model and request parameters, identifies the structural features of the target inference model to determine its type, formulates a first resource allocation based on the type and request parameters, evaluates the load status level in conjunction with system load data, and dynamically adjusts to a second resource allocation to adapt to load fluctuations in multi-task scenarios. Inference computation is then performed based on the second resource allocation, significantly improving inference performance and system efficiency. Through structural feature analysis, the model type can be accurately identified, ensuring that resource allocation matches model characteristics, thereby optimizing computation and memory usage and improving inference efficiency. The two-stage resource allocation, combined with real-time system load data, dynamically adjusts resources to adapt to load fluctuations, avoiding resource waste or performance bottlenecks caused by fixed allocation, and significantly improving GPU utilization and system throughput. The inference process is executed based on the second resource allocation, ensuring efficient utilization of computing resources and reducing latency. This significantly improves inference performance, resource utilization efficiency, and system adaptability. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a model inference performance optimization method provided in an embodiment of this application;

[0023] Figure 2 A structural block diagram of a model inference performance optimization device provided in an embodiment of this application;

[0024] Figure 3 A timing diagram of a model inference performance optimization method provided in an embodiment of this application;

[0025] Figure 4 This is a diagram showing the internal structure of a computer device in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] In one embodiment, such as Figure 1 As shown, a model performance optimization method is provided, including:

[0030] S100: In response to receiving an inference request, identify the structural features of the target inference model and determine the type of the target inference model based on the structural features; wherein the inference request includes request parameters and the target inference model is used to execute the inference request.

[0031] The target inference model refers to the large model (such as GPT, LLaMA, etc.) to be used for the inference task. Request parameters include: input data (such as user queries), inference configuration (such as maximum generation length, temperature parameters, etc.), and potential resource requirements (such as GPU computing power, memory usage, etc.). The target inference model is the machine learning model specified in the user request for performing the inference task. Structural features refer to the architectural characteristics of the target inference model, including but not limited to the number of layers, parameter size, type of computational operation (such as matrix multiplication, convolution), input / output dimensions, and attention mechanism configuration. These features reflect the model's computational complexity and resource requirements. Model type refers to classifying the target inference model into a specific category based on its structural features.

[0032] Specifically, by accurately analyzing the structural features of the model, such as the number of layers, parameter size, and type of computational operation, the model type can be accurately identified, ensuring that subsequent resource allocation is highly matched with the model's computation and memory requirements, thereby improving inference efficiency.

[0033] S200: Determine the first resource allocation based on the type of the target inference model and the request parameters.

[0034] The request parameters refer to the specific parameters related to task execution included in the inference request, such as the size of the input data, the length of the generated output, the batch size, latency requirements, and other user-defined constraints. The initial resource allocation refers to the computing resources initially allocated to the inference task based on the target inference model type and the request parameters, such as the number of GPU / CPU cores, the number of compute nodes, and memory resources.

[0035] Specifically, by clearly defining the model type and combining it with request parameters, computational and memory resource requirements can be accurately matched, avoiding resource over-allocation or under-allocation, thereby improving inference efficiency and resource utilization. By comprehensively considering request parameters, such as batch size and latency requirements, resource allocation is ensured to align with task objectives, supporting diverse inference scenarios, such as flexible adaptation to real-time inference and batch inference, thus enhancing system versatility.

[0036] S300: Obtain system load data and determine the system load status level based on the system load data.

[0037] System load data refers to real-time indicators reflecting the current operating status of the computing system. This data is collected in real-time through system monitoring tools or hardware counters, comprehensively describing the system's resource usage and load. System load status level refers to the grading result of system resource occupancy based on system load data analysis, used to guide resource allocation decisions.

[0038] Specifically, by collecting system load data in real time, the current operating status of the system can be accurately reflected, resource bottlenecks or idle periods can be identified in a timely manner, and data-driven decision support can be provided for subsequent resource adjustments. By converting load data into hierarchical load status levels, the complex system status analysis process is simplified, making resource allocation decisions more efficient and operable. Real-time monitoring of load data and precise classification of status levels ensure that resource allocation is highly matched with the actual operating status of the system.

[0039] S400: Determine the second resource allocation based on the first resource allocation and the system load status level.

[0040] The second resource allocation refers to the final resource allocation scheme obtained by dynamically adjusting the system load status level based on the first resource allocation.

[0041] Specifically, by comprehensively considering the first resource allocation and the system load status level, resource configuration can be dynamically adjusted to ensure that resource allocation is highly matched with the current system operating status and task requirements. The dynamic optimization mechanism of the second resource allocation enhances the system's adaptability in multi-tasking and high-concurrency scenarios, enabling it to flexibly respond to load fluctuations and ensure the stability and efficiency of inference tasks.

[0042] S500: Calculate the inference request based on the second resource allocation to obtain the inference result.

[0043] The reasoning result refers to the output result generated after performing calculations using the target reasoning model under the second resource allocation, based on the reasoning request.

[0044] Specifically, by receiving inference requests containing the target inference model and request parameters, the system identifies the structural characteristics of the target inference model to determine its type. Based on the type and request parameters, a first resource allocation is formulated. The system then dynamically adjusts the resource allocation to a second allocation based on system load data to adapt to load fluctuations in multi-tasking scenarios. Inference computation is performed based on this second resource allocation, significantly improving inference performance and system efficiency. Structural feature analysis accurately identifies model types, ensuring resource allocation matches model characteristics, thereby optimizing computation and memory usage and improving inference efficiency. Two-stage resource allocation based on real-time system load data dynamically adjusts resources to adapt to load fluctuations, avoiding resource waste or performance bottlenecks caused by fixed allocations, significantly improving GPU utilization and system throughput. The inference process is executed based on the second resource allocation, ensuring efficient use of computing resources and reducing latency. This significantly improves inference performance, resource utilization efficiency, and system adaptability.

[0045] In one embodiment, identifying the structural features of the target reasoning model and determining the type of the target reasoning model based on the structural features includes:

[0046] The architecture of the target inference model is read through model structure analysis, and the architectural features in the target inference model architecture are extracted.

[0047] If the architectural features match the preset architectural features, then the target inference model is a first-class model;

[0048] If the architectural features do not match the preset architectural features, the target inference model is a second type of model.

[0049] Among them, the preset architecture features refer to predefined feature templates or standards used to classify model types. The first type of model refers to a model category whose architecture features highly match the preset architecture features, and a specific type has clear computational and memory requirements (e.g., high memory requirements, suitable for parallel computing). The second type of model refers to a model category whose architecture features do not match the preset architecture features; these are other types (e.g., CNN, RNN) or hybrid architectures, indicating that their computational or memory requirements are significantly different from those of the first type of model.

[0050] Specifically, model structure analysis techniques are used to analyze the target inference model, such as configuration files or computation graphs, and extract its architectural features, including key attributes such as the number of layers, computational unit types, and parameter scales. These extracted architectural features are compared with a pre-defined architectural feature library, and the matching degree is evaluated based on feature similarity. If the architectural features highly match the pre-defined features, the target inference model is classified as a first-class model; otherwise, it is classified as a second-class model. This process is implemented through automated analysis and classification algorithms, ensuring the accuracy and efficiency of model type identification. Precise architectural feature extraction and type determination ensure that subsequent resource allocation is highly matched with the model's computational and memory requirements, thereby improving inference efficiency. The automated classification mechanism reduces manual intervention, improves the system's automation level and robustness, and adapts to diverse inference requests.

[0051] In one embodiment, determining the first resource allocation based on the type of the target inference model and the request parameters includes:

[0052] If the target inference model is a type 1 model, then the first resource allocation is determined according to the allocation strategy and request parameters corresponding to the type 1 model.

[0053] If the target inference model is a second type model, then the first resource allocation is determined according to the allocation strategy and request parameters corresponding to the second type model.

[0054] The allocation strategy refers to the method of allocating computing and memory resources based on the predefined resource allocation rules or algorithms of the model type and the request parameters.

[0055] Assuming the target inference model is a Type I model, the request parameters include: input text length of 512 words, batch size of 32, and inference precision requirement of FP16 (half-precision floating-point arithmetic, balancing performance and memory usage). The allocation strategy for Type I models prioritizes allocating high memory bandwidth resources for the first stage of computation to support the generation of key-value caches. Assuming the target inference model has a dimension of 1024, FP16 occupies 2 bytes, and the key-value cache occupies approximately 512 × 32 × 1024 × 2 ≈ 33.5 MB per layer, with a total of approximately 804 MB for 24 layers. Based on the allocation strategy, high memory bandwidth GPUs are prioritized. To meet the parallel computing requirements of a batch size of 32, 4 GPU cores are allocated to ensure efficient execution of the first stage of computation.

[0056] Assuming the target inference model is a Type II model, the requested parameters include: input image resolution of 224×224 pixels, batch size of 64, and inference precision requirement of INT8. The allocation strategy for Type II models prioritizes high-parallel computing resources to support the second-stage computation. Each image initially occupies approximately 224×224×3×1 bytes ≈ 150KB. Assuming the Type II model has 50 layers, the intermediate feature maps and fully connected layers require approximately 64×50×150KB ≈ 480MB of memory for processing. Based on the allocation strategy, high-parallel computing GPU cores are prioritized to meet the parallel computing requirements of a batch size of 64. Six GPU cores are allocated to share the computational tasks of convolution and fully connected layers, ensuring efficient execution of the second-stage computation.

[0057] Specifically, determining the initial resource allocation based on the target inference model type and request parameters lays the foundation for resource optimization in the inference process, yielding significant benefits. Targeted allocation strategies ensure precise matching of computational and memory resources with model characteristics and task requirements, thereby improving inference efficiency. Automated allocation mechanisms reduce manual intervention, enhance system automation and robustness, and adapt to diverse inference tasks. Model-type-based differentiated allocation simplifies resource planning, improves configuration efficiency, and provides a reliable basis for subsequent dynamic adjustments. This step ensures a high degree of alignment between resource allocation and task characteristics, significantly improving computational resource utilization, inference performance, and system throughput.

[0058] In one embodiment, determining the first resource allocation based on the type of the target inference model and the request parameters further includes:

[0059] The computation process of the inference request of the target inference model is divided into a first-stage computation and a second-stage computation. The first-stage computation is used to process the input data and generate intermediate values, and the second-stage computation is used to generate the output results based on the intermediate values.

[0060] Determine the initial resource allocation based on the request parameters;

[0061] If the target inference model is a type 1 model, then the resource allocation ratio for the first stage calculation of the inference request for the type 1 model is greater than the resource allocation ratio for the second stage calculation.

[0062] If the target inference model is a type II model, then the resource allocation ratio for the first stage calculation of the inference request for the type II model is less than the resource allocation ratio for the second stage calculation.

[0063] The intermediate value refers to the temporary data generated in the first stage of calculation, which serves as the input for the second stage of calculation; the initial resource allocation refers to the resource allocation scheme initially determined based on the request parameters, which serves as the basis for the first resource allocation.

[0064] Specifically, the computation process of an inference request is divided into a first-stage computation and a second-stage computation. Initial resource allocation is determined based on the request parameters. For the target inference model type, if it is a first-type model, more resources are allocated to the first-stage computation; if it is a second-type model, more resources are allocated to the second-stage computation. Through predefined allocation strategies and optimization algorithms, this ensures that resource allocation precisely matches the model type and task requirements. By splitting the inference request into two stages and determining the first resource allocation based on the model type and request parameters, this step provides refined support for resource optimization in the inference process, bringing significant benefits. The two-stage computation allows resource allocation to more accurately match the computational characteristics of the model, optimizing computational efficiency and resource utilization. The design of prioritizing the allocation of first-stage resources for first-type models and prioritizing the allocation of second-stage resources for second-type models ensures that resource configuration is highly aligned with the model's computational needs, thereby improving inference performance.

[0065] In one embodiment, acquiring system load data and determining the system load status level based on the system load data includes:

[0066] System load data includes: number of requests, computing resource utilization, storage resource status, and network latency;

[0067] Obtain historical system load data;

[0068] Create a predictive model based on historical system load data;

[0069] The system load status level is determined by analyzing the number of requests, computing resource utilization, storage resource status, and network latency using a predictive model.

[0070] Among them, the predictive model refers to the machine learning or statistical model built based on historical system load data, such as time series prediction model and regression model, which is used to predict future load trends or analyze current load data to determine the system status.

[0071] Specifically, the collection and analysis of real-time load data ensures that system status assessment accurately reflects current resource usage, while predictive models utilize historical data to capture load trends, improving the foresight and accuracy of status level determination, thereby optimizing resource allocation efficiency. Based on a comprehensive analysis of request volume, computing resource utilization, storage resource status, and network latency, the system load status level can comprehensively reflect the system's operating status, support targeted resource adjustments, reduce resource waste or performance bottlenecks, and significantly improve resource utilization.

[0072] In one embodiment, creating a predictive model based on historical system load data includes:

[0073] The inference requests and system load data are correlated to generate structured time data;

[0074] The structured time data is cleaned and extracted to obtain preset features;

[0075] Normalize the preset features to obtain the preset feature dataset;

[0076] Convert the preset feature dataset and structured time data into a time series input format;

[0077] Based on the time series input format, set up the prediction model.

[0078] Structured time data refers to an ordered dataset generated by associating inference requests and system load data by timestamps. This dataset includes time dimensions (such as timestamps) and corresponding request parameters and load metrics (such as request count and resource utilization), stored in tabular or database form. Data cleaning refers to the preprocessing of structured time data, including removing outliers, filling in missing values, eliminating duplicate data, or standardizing formats to improve data quality. Normalization refers to standardizing preset features, mapping feature values ​​of different dimensions (such as request count and network latency) to a unified range (such as [0, 1]) to eliminate dimensional differences. The time series input format refers to the input format that organizes the preset feature dataset and structured time data in chronological order, including time steps, feature vectors, and labels (such as load status levels), adapted to the time series prediction model.

[0079] Specifically, inference requests, such as request parameters, are correlated with system load data, such as request quantity and resource utilization, using timestamps to generate structured time data containing time dimensions and load metrics. The data is cleaned by removing outliers, filling in missing values, and standardizing the format, extracting preset features, such as the average number of requests and peak resource utilization. These preset features are then normalized to form a pre-defined feature dataset with uniform dimensions. The pre-defined feature dataset and structured time data are converted into a time-series input format, including time steps and feature vectors. Based on this time-series input format, a prediction model is configured, and model parameters are set to predict system load status or request trends. The generation of structured time data ensures that the correlation between inference requests and system load is fully explored, providing a comprehensive data foundation for subsequent analysis. Data cleaning and feature extraction improve data quality and feature representativeness, enhancing the accuracy of the prediction model. Normalization eliminates dimensional differences, thereby optimizing model training efficiency and stability. Adapting the time-series input format to meet dynamic load prediction needs supports the modeling of complex time dependencies. Predictive model-based settings can proactively analyze system load trends, guide resource allocation optimization, and reduce performance bottlenecks and resource waste.

[0080] In one embodiment, the system load status level is determined by analyzing the number of requests, computing resource utilization, storage resource status, and network latency using a predictive model, including:

[0081] The number of requests, computing resource utilization, storage resource status, and network latency are normalized to obtain a time series dataset.

[0082] Input the time series dataset into the prediction model to obtain the normalized load index;

[0083] The normalized load metric is denormalized to obtain readable raw data.

[0084] If the readable raw data exceeds a first preset load data threshold, the system load status level is high load.

[0085] If the readable raw data is less than the second preset load data threshold, the system load status level is low load, where the first preset load data threshold is greater than the second preset load data threshold.

[0086] The time-series dataset refers to normalized request counts, computing resource utilization, storage resource status, and network latency data, organized chronologically and including time steps and feature vectors, suitable as input for time-series prediction models. The normalized load metric refers to the normalized load prediction result output by the prediction model, generated based on the time-series dataset, representing a quantified value of system load within the normalized range (e.g., [0, 1]). Denormalization refers to the process of converting the normalized load metric back to its original dimensions, such as converting [0, 1] back to its original dimensions. [1] The range of indicators is restored to the original request quantity or latency value, generating readable raw data for easy interpretation and threshold comparison; readable raw data refers to load indicator data with original dimensions obtained after denormalization, such as: actual request quantity, percentage utilization or millisecond latency, which is convenient for comparison with preset thresholds; the first preset load data threshold refers to the predefined high load judgment standard, set based on historical data or experience, such as the request quantity greater than 1000 or the utilization rate greater than 80%, used to identify the high load state of the system; the second preset load data threshold refers to the predefined low load judgment standard, which is lower than the first preset load data threshold, such as the request quantity less than 200 or the utilization rate less than 30%, used to identify the low load state of the system.

[0087] Specifically, the accuracy and real-time nature of load status determination are ensured through automated data processing and model prediction. Normalization eliminates dimensional differences, ensuring that the time-series dataset fits the prediction model and improving the stability and accuracy of model training. The prediction model generates load indicators based on time-series data, capturing dynamic load trends and providing a forward-looking basis for status determination. Denormalization generates readable raw data, facilitating comparison with preset thresholds and ensuring that the determination results are intuitive and operable. Threshold-based load status classification (high load, low load) simplifies the system status assessment process.

[0088] In one embodiment, determining a second resource allocation based on a first resource allocation and a system load status level includes:

[0089] In response to a system load status level of high load and a target inference model of type 1, a second resource allocation is determined based on the first resource allocation corresponding to type 1 model and the allocation strategy corresponding to high load.

[0090] In response to a system load status level of high load and a target inference model of type 2, the second resource allocation is determined based on the first resource allocation corresponding to type 2 model and the allocation strategy corresponding to high load.

[0091] In response to the system load status level being low load and the target inference model being a first type model, the second resource allocation is determined based on the first resource allocation corresponding to the first type model and the allocation strategy corresponding to low load.

[0092] In response to a system load status level of low load and a target inference model of type 2, the second resource allocation is determined based on the first resource allocation corresponding to type 2 model and the allocation strategy corresponding to low load.

[0093] Specifically, the system load status is set to high load (1200 requests, 85% GPU utilization), the target inference model is Transformer (Type 1 model), and the first resource allocation is 4 high-memory-bandwidth GPUs. According to the high load allocation strategy, the second resource allocation is increased to 6 GPUs, and high-bandwidth memory is prioritized to support key-value cache generation, thereby reducing inference latency;

[0094] The system load level is high, the target inference model is CNN (type II model), and the first resource allocation is 3 parallel computing cores. Based on the high load allocation strategy, the second resource allocation is increased to 5 cores to optimize parallel computing capabilities and accelerate feature map processing and output generation.

[0095] The system load is low (150 requests, 25% GPU utilization), the target inference model is Transformer, and the first resource allocation is 4 GPUs. Based on the low-load allocation strategy, the second resource allocation is reduced to 2 GPUs to free up redundant resources and improve utilization.

[0096] The system load is low, the target inference model is CNN, and the first resource allocation is 3 cores. Based on the low-load allocation strategy, the second resource allocation is reduced to 1 core, and memory bandwidth is reduced to optimize resource utilization.

[0097] By combining the first resource allocation with the system load status level to determine the second resource allocation, this step provides dynamic and refined resource optimization support for the inference process, bringing significant benefits. The allocation strategy, which comprehensively considers model type and system load status, ensures that resource configuration is highly aligned with current task requirements and system operating status, thereby improving inference performance and resource utilization. For high-load scenarios, more resources are prioritized to alleviate performance bottlenecks; for low-load scenarios, redundant resources are reduced to avoid waste and optimize system efficiency. The automated adjustment mechanism reduces manual intervention, improving the system's automation level and robustness.

[0098] In one embodiment, determining the second resource allocation based on the first resource allocation and the system load status level further includes:

[0099] In response to a system load status level of high load and a target inference model of type I, idle resources are identified and allocated to the first stage of computation.

[0100] If the system load status level is low and the target inference model is a type 1 model, then idle resources are identified and allocated to the first stage of computation.

[0101] Specifically, the system load status is set to high load (1500 requests, 90% GPU utilization), the target inference model is Transformer (Type 1 model), and the first resource allocation is 4 high-memory-bandwidth GPUs. At this time, the system identifies 2 idle GPU cores, and the second resource allocation allocates these 2 idle cores to the first stage of computation to accelerate key-value cache generation and reduce inference latency;

[0102] The system load status is low (100 requests, 20% GPU utilization), the target inference model is Transformer (Type 1 model), and the first resource allocation is 2 GPUs. The system identified 3 idle GPU cores, and the second resource allocation allocates 2 of these idle cores to the first stage of computation, optimizing the computational efficiency of the pre-filling stage and improving resource utilization.

[0103] To address the high resource demands of the first stage of computation in the first type of model, idle resources are prioritized for allocation to ensure the efficient execution of critical computations such as key-value cache generation, thereby improving inference performance. Regardless of system load, dynamically utilizing idle resources optimizes resource utilization and reduces waste. In high-load scenarios, allocating idle resources alleviates performance bottlenecks; in low-load scenarios, fully utilizing idle resources improves system efficiency.

[0104] In one embodiment, determining the second resource allocation based on the first resource allocation and the system load status level further includes:

[0105] If the system load status level is high and the target inference model is a second type model, then idle resources are identified and allocated to the second stage of computation.

[0106] In response to a system load status level of low load and a target inference model of type 2, the idle resources are identified and allocated to the second stage of computation.

[0107] Specifically, the system load is set to high (1800 requests, 88% GPU utilization), the target inference model is CNN (second-class model), and the first resource allocation is 3 GPU cores. The system identifies one idle GPU core, and the second resource allocation allocates this idle core to the second-stage computation to accelerate classification output generation and reduce inference latency.

[0108] The system load status is low (120 requests, 18% GPU utilization), the target inference model is CNN (second-type model), and the first resource allocation is 2 GPU cores. The system identified 3 idle GPU cores, and the second resource allocation allocated 2 of these idle cores to the second stage of computation, optimizing feature map processing efficiency and improving resource utilization.

[0109] To address the high parallel computing requirements of the second-stage computation of the second type of model (such as CNN or RNN), idle resources are prioritized to ensure the efficient execution of key computations such as feature map processing or classification layers, thereby improving inference performance and response speed.

[0110] In one embodiment, such as Figure 3 As shown, the inference request is calculated based on the second resource allocation to obtain the inference result, including:

[0111] Extract input data from the request parameters;

[0112] The first stage of computation calculates the input data to obtain intermediate data, which is then transferred to the second stage of computation.

[0113] The intermediate data is processed in the second stage to generate inference results;

[0114] Store the inference results in the storage area.

[0115] The storage area refers to the memory or disk space (such as HBM memory) used to store the inference results, ensuring that the results are accessible or transferable.

[0116] Specifically, by extracting input data (such as 64 224×224 images) from inference requests and performing staged computations, the system first generates intermediate data such as feature maps through convolutional layers, and then generates classification results through fully connected layers to ensure accurate matching with task requirements, optimizing data flow and computational efficiency. Idle resources are dynamically allocated, such as allocating one GPU core under high load (1800 requests, GPU utilization 88%) or two cores under low load (120 requests, GPU utilization 18%), accelerating the second-stage computation and reducing inference latency (e.g., from 150ms to 100ms under high load, and from 120ms to 80ms under low load), while avoiding over-allocation of resources, maintaining system stability, and reserving resources for concurrent tasks. Automated resource identification, data extraction, intermediate data transfer, and result storage reduce manual intervention and improve system scalability and reliability. Results are stored in high-bandwidth memory to support fast access, adapting to real-time applications and complex scenarios.

[0117] In one embodiment, the input data is processed through a first-stage computation to obtain intermediate data, which is then transmitted to the second-stage computation, including:

[0118] The input data is split to obtain at least one data sub-batch;

[0119] The first data sub-batch is calculated using the first stage of calculation to generate intermediate data;

[0120] While transferring intermediate data to the second-stage calculation, the second data sub-batch is calculated through the first-stage calculation.

[0121] Among them, micro batches refer to each micro batch containing a portion of the samples in the inference request, which is used to support finer-grained parallel computing.

[0122] Specifically, after the input data is split, the system first sends the first data sub-batch to the first-stage computation, such as a computation node performing self-attention computation. After the first sub-batch is processed in this stage, the system immediately transmits the generated intermediate data (i.e., the output of the first-stage computation) to the second-stage computation, such as a computation node performing feedforward network (FFN) or other operator operations for further processing. Simultaneously, the first-stage computation does not need to wait for the second-stage processing to complete; instead, it immediately receives and processes the second data sub-batch and continues its first-stage computation. Each data sub-batch forms a decoupled asynchronous pipelined processing relationship between the two computation stages; that is, while the first stage continuously produces intermediate data, the second stage processes the intermediate results of the previous sub-batch in parallel. This fully utilizes the high-bandwidth computing resources (suitable for the first stage) and high-performance computing resources (suitable for the second stage) of heterogeneous chips, thereby improving the overall throughput and execution efficiency of the inference task.

[0123] In one embodiment, such as Figure 3 As shown, storing the inference results in the storage area includes:

[0124] The storage area includes: a high-frequency access area and a low-frequency access area;

[0125] A hash value is generated from the reasoning result using a hash algorithm;

[0126] If a hash value corresponding to the inference result exists in the high-frequency access area, the access event of that hash value is recorded, and the access frequency statistics of the high-frequency access area are updated.

[0127] If a hash value corresponding to the inference result exists in the low-frequency access area, the access event of that hash value is recorded, and the access frequency statistics of the low-frequency access area are updated.

[0128] If the access frequency of a hash value in the high-frequency access zone reaches the high-frequency threshold periodically within a preset time, the inference result corresponding to the hash value will be migrated to the low-frequency access zone.

[0129] If the access frequency of a hash value in the low-frequency access zone reaches the low-frequency threshold periodically within a preset time, the inference result corresponding to the hash value will be migrated to the high-frequency access zone.

[0130] In response to insufficient memory in the storage area, the inference result with the lowest access frequency in the low-frequency access area is deleted.

[0131] The high-frequency access area refers to the portion of the storage area used to store frequently accessed inference results, allocated on high-performance storage media. The low-frequency access area refers to the portion of the storage area used to store less frequently accessed inference results, allocated on lower-cost storage media. A hash algorithm is an algorithm that maps inference results to fixed-length hash values ​​(such as MD5 or SHA-1) to uniquely identify the inference results, facilitating fast retrieval and comparison. A hash value is a unique identifier generated by a hash algorithm for the inference results, used to locate and track access to the inference results within the storage area. An access event refers to the operation of reading or querying inference results, recorded as a hash value access record, used to statistically analyze access frequency. The high-frequency threshold is the lower limit for the access frequency of hash values ​​in the high-frequency access area; if the access frequency of a hash value is lower than this threshold within a preset time period, it indicates a decrease in its access popularity. The low-frequency threshold is the upper limit for the access frequency of hash values ​​in the low-frequency access area; if the access frequency of a hash value exceeds this threshold within a preset time period, it indicates an increase in its access popularity. The preset time period is the time window used to statistically analyze access frequency.

[0132] Specifically, by dividing the storage area into high-frequency and low-frequency access zones, inference results are precisely allocated to high-performance storage (such as HBM memory) or low-cost storage (such as ordinary disks), ensuring low-latency reading of frequently accessed data while saving high-cost storage resources. A hash algorithm generates unique hash values, accelerating data retrieval and location, and optimizing management efficiency in large-scale, high-concurrency inference scenarios. By recording access events and updating access frequency statistics within a preset time period, the system can accurately capture data access patterns. Combined with high-frequency and low-frequency thresholds, it dynamically migrates inference results, ensuring that high-frequency data resides in high-performance storage and low-frequency data is transferred to low-cost storage, thus achieving an optimal balance between performance and resource efficiency. When storage area memory is insufficient, the inference results with the lowest access frequency in the low-frequency access zone are deleted, freeing up space without affecting critical data access, ensuring system stability and scalability.

[0133] In one embodiment, during the calculation based on the second resource allocation, the actual resource utilization rate is monitored in real time, and resource reallocation is triggered when the deviation exceeds a preset threshold, including:

[0134] Obtain the actual utilization rate of computing resources;

[0135] The resource allocation based on the second resource allocation achieves the expected utilization rate;

[0136] The deviation value is calculated by comparing the actual utilization rate with the expected utilization rate.

[0137] If the deviation value exceeds the utilization threshold, resources are reallocated.

[0138] Among them, actual utilization rate refers to the actual degree of use of computing resources during the execution of the inference task, expressed as a percentage, and covers the computing load of the GPU cores; expected utilization rate refers to the resource utilization rate of the inference task under ideal conditions according to the resource configuration of the second resource allocation; deviation value refers to the difference between actual utilization rate and expected utilization rate.

[0139] Specifically, assuming the target inference model is a first-class model, based on the second resource allocation, 4 GPU cores and 8GB HBM memory were allocated for this task, with an expected GPU utilization of 75% and a memory utilization of 85%. The actual utilization rate was 92%, and the memory usage was 7.8GB, meaning a memory utilization rate of 97.5%. The memory deviation was:

[0140] ;

[0141] The utilization threshold is set to 10% to 20%, meaning a deviation of <10% or >20% triggers resource reallocation. Since the GPU deviation (22.67%) exceeds the upper limit of the threshold (20%), while the memory deviation (14.71%) is within the threshold range, the system triggers resource reallocation only for the GPU. The system scan found two idle GPU cores available, so the resources were adjusted to five GPU cores, while maintaining 8GB of memory. The second resource allocation configuration was updated and applied to subsequent inference tasks. If no idle resources are available, the system can reduce the batch size (e.g., from 32 to 24) to alleviate GPU load.

[0142] In one embodiment, a model inference performance optimization device is provided, comprising: a receiving module 710, a first allocation module 720, a load level determination module 730, a second allocation module 740, and an execution module 750.

[0143] The receiving module 710 is used to respond to receiving an inference request, identify the structural features of the target inference model, and determine the type of the target inference model based on the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request;

[0144] The first allocation module 720 is used to determine the first resource allocation based on the type of the target inference model and the request parameters;

[0145] The load level determination module 730 is used to acquire system load data and determine the system load status level based on the system load data.

[0146] The second allocation module 740 is used to determine the second resource allocation based on the first resource allocation and the system load status level;

[0147] The execution module 750 is used to calculate the inference request based on the second resource allocation and obtain the inference result.

[0148] In one embodiment, the receiving module 710 is configured to:

[0149] The architecture of the target inference model is read through model structure analysis, and the architectural features in the target inference model architecture are extracted.

[0150] If the architectural features match the preset architectural features, then the target inference model is a first-class model;

[0151] If the architectural features do not match the preset architectural features, the target inference model is a second type of model.

[0152] In one embodiment, the load level determination module 730 is used for:

[0153] System load data includes: number of requests, computing resource utilization, storage resource status, and network latency;

[0154] Obtain historical system load data;

[0155] Create a predictive model based on historical system load data;

[0156] The system load status level is determined by analyzing the number of requests, computing resource utilization, storage resource status, and network latency using a predictive model.

[0157] In one embodiment, the execution module 750 is configured to:

[0158] Extract input data from the request parameters;

[0159] The first stage of computation calculates the input data to obtain intermediate data, which is then transferred to the second stage of computation.

[0160] The intermediate data is processed in the second stage to generate inference results;

[0161] Store the inference results in the storage area.

[0162] It should be understood that, although Figure 2 The steps in the device block diagram are shown sequentially as indicated by the arrows; however, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be executed in other orders. Furthermore, Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0163] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the model inference performance optimization method at runtime.

[0164] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0165] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the model inference performance optimization method.

[0166] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, such as Figure 4 As shown, to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0167] The above provides a detailed description of a model inference performance optimization method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for optimizing model inference performance, characterized in that, include: In response to receiving a reasoning request, the structural features of the target reasoning model are identified, and the type of the target reasoning model is determined based on the structural features. The inference request includes request parameters, the target inference model is used to execute the inference request, and the structural features include: number of layers, parameter size, and type of computational operation; The first resource allocation is determined based on the type of the target inference model and the request parameters; Acquire system load data and determine the system load status level based on the system load data; The second resource allocation is determined based on the first resource allocation and the system load status level; In response to the calculation of the inference request based on the second resource allocation, the actual utilization rate of the computing resources is monitored in real time, the expected utilization rate is obtained based on the resource configuration of the second resource allocation, the deviation value is obtained by comparing the actual utilization rate with the expected utilization rate, and in response to the deviation value exceeding the utilization rate threshold range, resource reallocation is triggered and the second resource allocation is updated, and the inference request is calculated again based on the updated second resource allocation to obtain the inference result. The process of identifying the structural features of the target inference model and determining the type of the target inference model based on the structural features includes: The architecture of the target inference model is read through model structure analysis, and the architectural features in the target inference model architecture are extracted. If the architectural features match the preset architectural features, then the target inference model is a first type model; If the architectural features do not match the preset architectural features, then the target inference model is a second type of model; Determining the first resource allocation based on the type of the target inference model and the request parameters includes: In response to the target inference model being a first type of model, the first resource allocation is determined according to the allocation strategy corresponding to the first type of model and the request parameters; In response to the target inference model being a second type of model, the first resource allocation is determined according to the allocation strategy corresponding to the second type of model and the request parameters, wherein the allocation strategy determines the specific allocation method of computing and memory resources based on the resource allocation rules or algorithms predefined by the model type and in combination with the request parameters; The calculation process of the reasoning request of the target reasoning model is divided into a first-stage calculation and a second-stage calculation, wherein the first-stage calculation is used to process input data and generate intermediate values, and the second-stage calculation is used to generate output results based on the intermediate values. Determine the initial resource allocation based on the request parameters; In response to the target inference model being a first type of model, the resource allocation ratio calculated in the first stage for the inference request of the first type of model is greater than the resource allocation ratio calculated in the second stage. In response to the target inference model being a second type of model, the resource allocation ratio calculated in the first stage for the inference request of the second type of model is less than the resource allocation ratio calculated in the second stage. Obtain system load data, and determine the system load status level based on the system load data, including: The system load data includes: number of requests, computing resource utilization, storage resource status, and network latency; Obtain historical system load data; A predictive model is created based on the historical system load data; The system load status level is determined by analyzing the number of requests, the utilization rate of computing resources, the status of storage resources, and the network latency using the prediction model. The prediction model analyzes the number of requests, the utilization rate of computing resources, the status of storage resources, and the network latency to determine the system load status level, including: The number of requests, the utilization rate of computing resources, the status of storage resources, and the network latency are normalized to obtain a time series dataset. Input the time series dataset into the prediction model to obtain the normalized load index; The normalized load metric is denormalized to obtain readable raw data; If the readable raw data is greater than a first preset load data threshold, then the system load status level is high load; In response to the readable raw data being less than a second preset load data threshold, the system load status level is low load, wherein the first preset load data threshold is greater than the second preset load data threshold.

2. The model inference performance optimization method according to claim 1, characterized in that, Based on the historical system load data, a predictive model is created, including: The inference request and the system load data are correlated to generate structured time data; The structured time data is cleaned and extracted to obtain preset features; The preset features are normalized to obtain a preset feature dataset; The preset feature dataset and the structured time data are converted into a time series input format; The prediction model is set based on the time series input format.

3. The model inference performance optimization method according to claim 1, characterized in that, Determining the second resource allocation based on the first resource allocation and the system load status level includes: In response to the system load status level being high load and the target inference model being the first type of model, idle resources are identified, and high-bandwidth resources among the idle resources are allocated to the first stage of computation. In response to the system load status level being high load and the target inference model being the second type of model, the idle resources are identified, and the high computing power of the idle resources is allocated to the second stage of computation.

4. The model inference performance optimization method according to claim 3, characterized in that, Determining the second resource allocation based on the first resource allocation and the system load status level further includes: In response to the system load status level being low and the target inference model being the first type of model, the idle resources are identified and the high bandwidth resources are allocated to the first stage of computation. In response to the system load status level being low and the target inference model being the second type of model, the idle resources are identified, and the high computing power resources are allocated to the second stage of computation.

5. The model inference performance optimization method according to claim 1, characterized in that, The inference request is calculated based on the second resource allocation to obtain the inference result, including: Extract the input data from the request parameters; The input data is calculated in the first stage to obtain intermediate data, which is then transmitted to the second stage for calculation. The intermediate data is calculated through the second stage of calculation to generate inference results; The reasoning results are stored in the storage area.

6. The model inference performance optimization method according to claim 5, characterized in that, The input data is processed through the first stage of calculation to obtain intermediate data, which is then transmitted to the second stage of calculation, including: The input data is split to obtain at least one data sub-batch; The first data sub-batch is calculated using the first stage calculation to generate the intermediate data; While the intermediate data is being transferred to the second stage of computation, the second data sub-batch is being computed through the first stage of computation.

7. The model inference performance optimization method according to claim 5, characterized in that, Storing the inference results in a storage area includes: The storage area includes: a high-frequency access area and a low-frequency access area; A hash value is generated from the inference result using a hash algorithm; If a hash value corresponding to the inference result exists in the high-frequency access area, the access event of the hash value is recorded, and the access frequency statistics of the high-frequency access area are updated. If a hash value corresponding to the inference result exists in the low-frequency access area, the access event of the hash value is recorded, and the access frequency statistics of the low-frequency access area are updated. If the access frequency of the hash value in the high-frequency access zone reaches a high-frequency threshold periodically within a preset time, the inference result corresponding to the hash value is migrated to the low-frequency access zone. If the access frequency of the hash value in the low-frequency access zone reaches a low-frequency threshold periodically within a preset time, the inference result corresponding to the hash value is migrated to the high-frequency access zone. In response to insufficient memory in the storage area, the inference result with the lowest access frequency in the low-frequency access area is deleted.

8. A performance optimization apparatus for implementing the model inference performance optimization method according to any one of claims 1-7, characterized in that, The device includes: A receiving module is configured to, in response to receiving an inference request, identify the structural features of the target inference model and determine the type of the target inference model based on the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request; The first allocation module is used to determine the first resource allocation based on the type of the target inference model and the request parameters; The load level determination module is used to acquire system load data and determine the system load status level based on the system load data. The second allocation module is used to determine the second resource allocation based on the first resource allocation and the system load status level. The execution module is used to calculate the inference request based on the second resource allocation and obtain the inference result.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images