Model reasoning performance optimization method and device, equipment and storage medium

Abstract

The invention discloses a model reasoning performance optimization method and device, equipment and a storage medium, and the method comprises the steps: responding to a received reasoning request, recognizing the structural features of a target reasoning model, and determining the type of the target reasoning model according to the structural features, the reasoning request comprises request parameters, the target reasoning model is used for executing the reasoning request, and the target reasoning model is used for executing the reasoning request; determining a first resource allocation according to the type of the target inference model and the request parameter, obtaining system load data, determining a system load state level according to the system load data, determining a second resource allocation according to the first resource allocation and the system load state level, and calculating the inference request according to the second resource allocation to obtain an inference result, through the method, the resource utilization rate can be optimized, and the system throughput can be improved.

Description

Technical Field

[0001] This application relates to the technical field of servers, and in particular, to a method, device, equipment, and storage medium for optimizing model inference performance. Background Art

[0002] Currently, large model inference services usually adopt an architecture of hybrid deployment of prefill and decode, that is, the two stages run on the same computing device. The prefill stage is responsible for processing the parallel computing of long sequences and belongs to a compute-intensive operation; while the decode stage needs to frequently read and write the KV cache and belongs to a memory-intensive operation. Due to the significant differences in the computing characteristics of the two stages, the hybrid deployment mode is likely to lead to unbalanced utilization of GPU resources, which in turn causes problems such as high first token latency (TTFT) and limited throughput.

[0003] To optimize the inference efficiency, the PD separation technology is proposed to decouple and deploy the prefill and decode stages on different devices. For example, by separating the computing tasks of the two stages and allocating computing resources specifically: the prefill stage preferentially utilizes high-computing-power devices, and the decode stage is adapted to high-bandwidth memory devices. Open-source solutions such as vLLM improve the throughput by optimizing the KV cache management. By independently scheduling the tasks of the two stages and leveraging dedicated hardware to improve the overall performance, the system throughput is increased.

[0004] Although the PD separation technology significantly improves the inference efficiency by decoupling the prefill and decode stages, the existing PD separation architecture adopts a static resource allocation strategy and cannot dynamically adjust the computing resource ratio according to the real-time load, which is prone to resource mismatch in the multi-task concurrency scenario; the cross-stage data transmission relies on high-overhead serialization operations, resulting in a significant increase in the first token latency. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, equipment, and storage medium for optimizing model inference performance that can optimize resource utilization and improve system throughput for the above technical problems.

[0006] In a first aspect, a method for optimizing model inference performance is provided, including: In response to receiving an inference request, identify the structural features of the target inference model, and determine the type of the target inference model according to the structural features; where the inference request includes request parameters, and the target inference model is used to execute the inference request; Determine a first resource allocation according to the type of the target inference model and the request parameters; Obtain system load data, and determine the system load status level according to the system load data; Determine a second resource allocation according to the first resource allocation and the system load status level; Calculate the inference request according to the second resource allocation to obtain an inference result.

[0007] In a second aspect, there is provided a model inference performance optimization device, which is applied to the model inference performance optimization method described in the first aspect, and includes: A receiving module, configured to, in response to receiving an inference request, identify the structural features of a target inference model, and determine the type of the target inference model according to the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request; A first allocation module, configured to determine a first resource allocation according to the type of the target inference model and the request parameters; A load level determination module, configured to obtain system load data and determine the system load status level according to the system load data; A second allocation module, configured to determine a second resource allocation according to the first resource allocation and the system load status level; An execution module, configured to calculate the inference request according to the second resource allocation to obtain an inference result.

[0008] In a third aspect, there is also provided a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the model inference performance optimization method described in the first aspect is implemented.

[0009] In a fourth aspect, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the model inference performance optimization method described in the first aspect is implemented.

[0010] By implementing the above model inference performance optimization method, device, equipment, and storage medium, the method receives an inference request including a target inference model and request parameters, identifies the structural features of the target inference model to determine its type, formulates a first resource allocation according to the type and request parameters, evaluates the load status level in combination with system load data, dynamically adjusts it to a second resource allocation to adapt to load fluctuations in a multi-task scenario, and performs inference calculations based on the second resource allocation, significantly improving inference performance and system efficiency. Through structural feature analysis, the model type can be accurately identified, ensuring that resource allocation matches the model characteristics, thereby optimizing computing and memory usage and improving inference efficiency. Combining real-time system load data for two-stage resource allocation, dynamically adjusting resources to adapt to load fluctuations, avoiding resource waste or performance bottlenecks caused by fixed allocation, and significantly improving GPU utilization and system throughput. The inference process is executed based on the second resource allocation, ensuring efficient use of computing resources and reducing latency. Significantly improving inference performance, resource utilization efficiency, and system adaptability. Description of the Drawings

[0011] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0012] Figure 1 It is a schematic flowchart of a method for optimizing the model inference performance provided by an embodiment of the present application; Figure 2 It is a structural block diagram of a device for optimizing the model inference performance provided by an embodiment of the present application; Figure 3 It is a timing diagram of a method for optimizing the model inference performance provided by an embodiment of the present application; Figure 4 It is an internal structure diagram of a computer device in an embodiment of the present application. Specific embodiments

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0014] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0015] To enable those skilled in the art of this technology to better understand the solution of the present application, the following will further elaborate on the present application in conjunction with the accompanying drawings and specific embodiments.

[0016] In one embodiment, as Figure 1 shown, a method for optimizing model performance is provided, including: S100: In response to receiving an inference request, identify the structural features of the target inference model, and determine the type of the target inference model according to the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request.

[0017] Among them, the target inference model refers to a large model for performing inference tasks (such as GPT, LLaMA, etc.). The request parameters include: input data (such as user questions), inference configuration (such as maximum generation length, temperature parameter, etc.), and possible resource requirements (such as GPU computing power, memory occupancy, etc.); the target inference model refers to the machine learning model specified in the user request for performing inference tasks; structural features refer to the architectural characteristics of the target inference model, including but not limited to the number of model layers, parameter scale, types of computational operations (such as matrix multiplication, convolution operations), input and output dimensions, and the configuration of the attention mechanism, etc. These features reflect the computational complexity and resource requirements of the model; the model type refers to classifying the target inference model into a specific category according to the structural features.

[0018] Specifically, by accurately analyzing the structural features of the model such as the number of layers, parameter scale, and types of computational operations, the model type can be accurately identified, ensuring that the subsequent resource allocation highly matches the computational and memory requirements of the model, thereby improving the inference efficiency.

[0019] S200: Determine the first resource allocation according to the type of the target inference model and the request parameters.

[0020] Among them, the request parameters refer to the specific parameters related to task execution included in the inference request, such as: the scale of the input data, the length of the generated output, the batch size, the latency requirement, and other user-defined constraints. The first resource allocation refers to the computational resources initially allocated for the inference task according to the type of the target inference model and the request parameters, such as: the number of GPU / CPU cores, the number of computing nodes, and the memory resources.

[0021] Specifically, by clarifying the model type and combining the request parameters, the computational and memory resource requirements can be accurately matched, avoiding over-allocation or under-allocation of resources, thereby improving the inference efficiency and resource utilization rate. Considering the request parameters comprehensively, such as the batch size and the latency requirement, ensures that the resource allocation is aligned with the task objectives, supports diverse inference scenarios, such as the flexible adaptation of real-time inference and batch inference, and improves the system versatility.

[0022] S300: Obtain the system load data and determine the system load status level according to the system load data.

[0023] Among them, the system load data refers to the real-time metric data reflecting the running state of the current computing system, which is collected in real time through system monitoring tools or hardware counters, and comprehensively describes the resource usage and load situation of the system. The system load status level: refers to the classification result of the degree of system resource occupancy obtained by analyzing the system load data, which is used to guide the resource allocation decision.

[0024] Specifically, by collecting system load data in real time, the current operating state of the system can be accurately reflected, resource bottlenecks or idle situations can be discovered in a timely manner, and data-driven decision support for subsequent resource adjustment can be provided. By converting the load data into a hierarchical load status level, the complex system state analysis process is simplified, making the resource allocation decision more efficient and operable. The real-time monitoring of load data and the accurate classification of status levels ensure a high degree of matching between resource allocation and the actual operating state of the system.

[0025] S400: Determine the second resource allocation based on the first resource allocation and the system load status level.

[0026] Among them, the second resource allocation refers to the final resource allocation plan obtained by dynamically adjusting based on the first resource allocation in combination with the system load status level.

[0027] Specifically, by comprehensively considering the first resource allocation and the system load status level, the resource configuration can be dynamically adjusted to ensure a high degree of matching between resource allocation and the current operating state and task requirements of the system. The dynamic optimization mechanism of the second resource allocation enhances the adaptability of the system in multi-task and high-concurrency scenarios, can flexibly handle load fluctuations, and ensures the stability and efficiency of the inference task.

[0028] S500: Calculate the inference request according to the second resource allocation to obtain the inference result.

[0029] Among them, the inference result refers to the output result generated after performing calculations using the target inference model under the second resource allocation according to the inference request.

[0030] Specifically, by receiving an inference request containing the target inference model and request parameters, identifying the structural characteristics of the target inference model to determine its type, formulating the first resource allocation according to the type and request parameters, evaluating the load status level in combination with the system load data, dynamically adjusting to the second resource allocation to adapt to load fluctuations in multi-task scenarios, and performing inference calculations based on the second resource allocation, the inference performance and system efficiency are significantly improved. Through structural feature analysis, the model type can be accurately identified to ensure that resource allocation matches the model characteristics, thereby optimizing the use of computing and memory and improving the inference efficiency. Combining real-time system load data for two-stage resource allocation and dynamically adjusting resources to adapt to load fluctuations avoid resource waste or performance bottlenecks caused by fixed allocation, significantly improving GPU utilization and system throughput. The inference process is executed based on the second resource allocation to ensure the efficient use of computing resources and reduce latency. The inference performance, resource utilization efficiency, and system adaptability are significantly improved.

[0031] In one embodiment, identifying the structural characteristics of the target inference model and determining the type of the target inference model according to the structural characteristics includes: Read the architecture of the target inference model through model structure analysis, and extract the architecture features in the architecture of the target inference model; If the architecture features match the preset architecture features, the target inference model is a first type of model; If the architecture features do not match the preset architecture features, the target inference model is a second type of model.

[0032] Among them, the preset architecture features refer to the predefined feature templates or criteria for classifying model types. The first type of model refers to the model category with highly matching architecture features and a specific type having clear characteristics of computing and memory requirements (such as high memory requirements and suitability for parallel computing). The second type of model refers to the model category with unmatched architecture features, which is other types (such as CNN, RNN) or hybrid architectures, indicating that its computing or memory requirements are significantly different from those of the first type of model.

[0033] Specifically, through model structure analysis technology, parse the target inference model, such as a configuration file or a computational graph, and extract its architecture features, such as key attributes like the number of layers, type of computing units, parameter scale, etc. Compare the extracted architecture features with the preset architecture feature library, and evaluate the matching degree based on feature similarity. If the architecture features highly match the preset features, determine that the target inference model is a first type of model; if not, determine it as a second type of model. This process is achieved through automated analysis and classification algorithms to ensure the accuracy and efficiency of model type recognition. Precise architecture feature extraction and type determination ensure that subsequent resource allocation highly matches the computing and memory requirements of the model, thereby improving inference efficiency. The automated classification mechanism reduces manual intervention, improves the automation level and robustness of the system, and adapts to diverse inference requests.

[0034] In one embodiment, determine the first resource allocation according to the type of the target inference model and the request parameters, including: If the target inference model is a first type of model, determine the first resource allocation according to the allocation strategy corresponding to the first type of model and the request parameters; If the target inference model is a second type of model, determine the first resource allocation according to the allocation strategy corresponding to the second type of model and the request parameters.

[0035] Among them, the allocation strategy refers to the resource allocation rules or algorithms predefined according to the model type, and combined with the request parameters, determine the allocation method of computing and memory resources.

[0036] Let the target inference model be a first - type model. The request parameters include: the input text length is 512 words, the batch size is 32, and the inference accuracy requirement is FP16 (half - precision floating - point operation, balancing performance and memory occupancy). The allocation strategy for the first - type model is to preferentially allocate high - memory - bandwidth resources for the first - stage calculation to support the generation of key - value caches in the first - stage calculation. Suppose the dimension of the target inference model is 1024, FP16 occupies 2 bytes, and the key - value cache occupies approximately 512×32×1024×2≈33.5MB per layer. Suppose the model has 24 layers, with a total of approximately 804MB. According to the allocation strategy, preferentially select GPUs with high memory bandwidth. To meet the parallel computing requirements of a batch size of 32, allocate 4 GPU cores to ensure the efficient execution of the first - stage calculation.

[0037] Let the target inference model be a second - type model. The request parameters include: the input image resolution is 224×224 pixels, the batch size is 64, and the inference accuracy requirement is INT8. The allocation strategy for the second - type model is to preferentially allocate high - parallel - computing resources to support the second - stage calculation; the initial input of each image occupies approximately 224×224×3×1 byte≈150KB. Suppose the second - type model has 50 layers, and the intermediate feature maps and fully - connected layers need to store approximately 64×50×150KB≈480MB of memory. According to the allocation strategy, preferentially select GPU cores with high parallel - computing capabilities. To meet the parallel computing requirements of a batch size of 64, allocate 6 GPU cores to share the computing tasks of the convolutional and fully - connected layers to ensure the efficient execution of the second - stage calculation.

[0038] Specifically, determining the first resource allocation according to the target inference model type and request parameters lays the foundation for resource optimization in the inference process, bringing significant beneficial effects. The targeted allocation strategy ensures that computing and memory resources are accurately matched with model characteristics and task requirements, thereby improving inference efficiency. The automated allocation mechanism reduces manual intervention, improves the automation level and robustness of the system, and adapts to diverse inference tasks. The differentiated allocation based on model type simplifies the resource planning process, improves the configuration efficiency, and provides a reliable basis for subsequent dynamic adjustment. This step ensures a high degree of alignment between resource configuration and task characteristics, significantly improving computing resource utilization, inference performance, and system throughput.

[0039] In one embodiment, determining the first resource allocation according to the type and request parameters of the target inference model further includes: Split the calculation process of the inference request of the target inference model into a first - stage calculation and a second - stage calculation, where the first - stage calculation is used to process input data and generate intermediate values, and the second - stage calculation is used to generate output results based on the intermediate values; Determine the initial resource allocation according to the request parameters; In response to the target inference model being a first - type model, the resource allocation ratio for the first - stage calculation of the inference request for the first - type model is greater than that for the second - stage calculation; In response to the target inference model being a second - type model, the resource allocation ratio for the first - stage calculation of the inference request for the second - type model is less than that for the second - stage calculation.

[0040] Among them, the intermediate value refers to the temporary data generated by the first - stage calculation and serves as the input for the second - stage calculation; the initial resource allocation refers to the resource allocation plan preliminarily determined according to the request parameters and serves as the basis for the first resource allocation.

[0041] Specifically, the calculation process of the inference request is split into a first - stage calculation and a second - stage calculation. The initial resource allocation is determined according to the request parameters. For the type of the target inference model, if it is a first - type model, more resources are preferentially allocated for the first - stage calculation; if it is a second - type model, more resources are allocated for the second - stage calculation. It is realized through predefined allocation strategies and optimization algorithms to ensure that the resource allocation is accurately matched with the model type and task requirements. By splitting the inference request into two - stage calculations and determining the first resource allocation according to the model type and request parameters, this step provides refined support for the resource optimization of the inference process and brings significant beneficial effects. The splitting of the two - stage calculation enables the resource allocation to more accurately match the calculation characteristics of the model, optimizing the calculation efficiency and resource utilization rate. The design of preferentially allocating first - stage resources for the first - type model and second - stage resources for the second - type model ensures that the resource configuration is highly aligned with the calculation requirements of the model, thereby improving the inference performance.

[0042] In one embodiment, system load data is obtained, and the system load status level is determined according to the system load data, including: [[ID=!2]] The system load data includes: the number of requests, the computing resource utilization rate, the storage resource status, and the network latency; Historical system load data is obtained; A prediction model is created according to the historical system load data; Through the prediction model, the number of requests, the computing resource utilization rate, the storage resource status, and the network latency are analyzed to determine the system load status level.

[0043] Among them, the prediction model refers to a machine - learning or statistical model constructed based on historical system load data, such as: time - series prediction model, regression model, which is used to predict future load trends or analyze current load data to determine the system state.

[0044] Specifically, the collection and analysis of real-time load data ensure that the system status assessment accurately reflects the current resource usage. The prediction model utilizes historical data to capture load trends, improving the foresight and accuracy of the status level determination, thereby optimizing the resource allocation efficiency. Based on the comprehensive analysis of the request quantity, computing resource utilization rate, storage resource status, and network latency, the system load status level can comprehensively reflect the system operation status, support targeted resource adjustment, reduce resource waste or performance bottlenecks, and significantly improve the resource utilization rate.

[0045] In one embodiment, creating a prediction model based on historical system load data includes: Associating inference requests with system load data to generate structured time data; Cleaning and extracting the structured time data to obtain preset features; Normalizing the preset features to obtain a preset feature dataset; Converting the preset feature dataset and the structured time data into a time series input format; Setting the prediction model based on the time series input format.

[0046] Among them, the structured time data refers to an ordered dataset generated by associating inference requests with system load data according to timestamps. For example, it includes a time dimension (such as a timestamp) and corresponding request parameters and load metrics (such as the request quantity, resource utilization rate), and is stored in the form of a table or a database. Data cleaning refers to the process of preprocessing the structured time data. For example, it includes removing outliers, filling in missing values, eliminating duplicate data, or standardizing the format to improve the data quality. Normalization refers to the standardization process of the preset features, mapping feature values with different dimensions (such as the request quantity and network latency) to a unified range (such as [0, 1]) to eliminate the dimension difference. The time series input format refers to an input format in which the preset feature dataset and the structured time data are organized in time order, including a time step, a feature vector, and a label (such as the load status level), and is adapted to the time series prediction model.

[0047] Specifically, by using timestamps, inference requests, such as request parameters, and system load data, such as the number of requests and resource utilization rates, are correlated to generate structured time data that includes time dimensions and load metrics. The data is cleaned by removing outliers, filling in missing values, and standardizing the format, and preset features, such as the mean number of requests and the peak resource utilization rate, are extracted. The preset features are normalized to form a preset feature dataset with a unified dimension. The preset feature dataset and the structured time data are converted into a time series input format that includes time steps and feature vectors. Based on the time series input format, a prediction model is configured, and model parameters are set to predict the system load status or request trends. The generation of structured time data ensures that the correlation between inference requests and system load is fully explored, providing a comprehensive data foundation for subsequent analysis. Data cleaning and feature extraction improve data quality and feature representativeness, enhancing the accuracy of the prediction model. The normalization process eliminates dimension differences, thereby optimizing model training efficiency and stability. The conversion to the time series input format adapts to dynamic load prediction requirements and can support the modeling of complex time-dependent relationships. Based on the settings of the prediction model, the system load trend can be analyzed prospectively, guiding the optimization of resource allocation, reducing performance bottlenecks and resource waste.

[0048] In one embodiment, the request quantity, computing resource utilization rate, storage resource status, and network latency are analyzed through a prediction model to determine the system load status level, including: Normalize the request quantity, computing resource utilization rate, storage resource status, and network latency to obtain a time series dataset; Input the time series dataset into the prediction model to obtain normalized load metrics; Denormalize the normalized load metrics to obtain readable raw data; In response to the readable raw data being greater than the first preset load data threshold, the system load status level is high load; In response to the readable raw data being less than the second preset load data threshold, the system load status level is low load, where the first preset load data threshold is greater than the second preset load data threshold.

[0049] Among them, the time series data set refers to the normalized request quantity, computing resource utilization rate, storage resource status, and network latency data, which are organized in chronological order, contain time steps and feature vectors, and are suitable for input into the time series prediction model; the normalized load indicator refers to the normalized form of the load prediction result output by the prediction model, which is generated based on the time series data set and represents the quantitative value of the system load, located in the normalized range (such as [0, 1]); denormalization refers to the process of converting the normalized load indicator back to the original dimension, such as restoring the indicator in the range of [0, 1] to the original request quantity or latency value to generate readable original data for easy interpretation and threshold comparison; the readable original data refers to the load indicator data with the original dimension obtained after denormalization, such as the actual request quantity, percentage utilization rate, or millisecond latency, which is convenient for comparison with the preset threshold; the first preset load data threshold refers to the predefined high-load determination criterion, which is set based on historical data or experience, such as the request quantity being greater than 1000 or the utilization rate being greater than 80%, and is used to identify the high-load state of the system; the second preset load data threshold refers to the predefined low-load determination criterion, which is lower than the first preset load data threshold, such as the request quantity being less than 200 or the utilization rate being less than 30%, and is used to identify the low-load state of the system.

[0050] Specifically, it is achieved through automated data processing and model prediction to ensure the accuracy and real-time nature of the load state determination. The normalization process eliminates the dimension difference, ensures the adaptation of the time series data set to the prediction model, and improves the stability and prediction accuracy of model training. The prediction model generates load indicators based on the time series data, captures the dynamic load trend, and provides a forward-looking basis for state determination. Denormalization generates readable original data, which is convenient for comparison with the preset threshold, ensuring that the determination result is intuitive and operable. The load state classification (high load, low load) based on the threshold simplifies the system state evaluation process.

[0051] In one embodiment, determining the second resource allocation according to the first resource allocation and the system load state level includes: In response to the system load state level being high load and the target inference model being the first type of model, determine the second resource allocation according to the first resource allocation corresponding to the first type of model and the allocation strategy corresponding to high load; In response to the system load state level being high load and the target inference model being the second type of model, determine the second resource allocation according to the first resource allocation corresponding to the second type of model and the allocation strategy corresponding to high load; In response to the system load state level being low load and the target inference model being the first type of model, determine the second resource allocation according to the first resource allocation corresponding to the first type of model and the allocation strategy corresponding to low load; In response to the system load status level being low load and the target inference model being a second - type model, determine the second resource allocation according to the first resource allocation corresponding to the second - type model and the allocation strategy corresponding to low load.

[0052] Specifically, assume that the system load status level is high load (request quantity 1200, GPU utilization rate 85%), the target inference model is Transformer (a first - type model), and the first resource allocation is 4 high - memory - bandwidth GPUs. According to the high - load allocation strategy, the second resource allocation is increased to 6 GPUs, and high - bandwidth memory is preferentially allocated to support key - value cache generation, thereby reducing inference latency. The system load status level is high load, the target inference model is CNN (a second - type model), and the first resource allocation is 3 parallel computing cores. According to the high - load allocation strategy, the second resource allocation is increased to 5 cores to optimize the parallel computing ability to accelerate feature map processing and output generation. The system load status is low load (request quantity 150, GPU utilization rate 25%), the target inference model is Transformer, and the first resource allocation is 4 GPUs. According to the low - load allocation strategy, the second resource allocation is reduced to 2 GPUs to release redundant resources to improve utilization. The system load status is low load, the target inference model is CNN, and the first resource allocation is 3 cores. According to the low - load allocation strategy, the second resource allocation is reduced to 1 core, and the memory bandwidth is reduced to optimize resource usage. By combining the first resource allocation and the system load status level to determine the second resource allocation, this step provides dynamic and refined resource optimization support for the inference process, bringing significant beneficial effects. The allocation strategy that comprehensively considers the model type and the system load status ensures that the resource configuration is highly aligned with the current task requirements and the system operation status, thereby improving the inference performance and resource utilization rate. For high - load scenarios, preferentially allocate more resources to relieve the performance bottleneck; for low - load scenarios, reduce redundant resources to avoid waste and optimize the system efficiency. The automated adjustment mechanism reduces manual intervention and improves the automation degree and robustness of the system.

[0053] In one embodiment, determining the second resource allocation according to the first resource allocation and the system load status level further includes: In response to the system load status level being high load and the target inference model being a first - type model, identify idle resources and allocate the idle resources to the first - stage calculation. In response to the system load status level being low load and the target inference model being a first - type model, identify idle resources and allocate the idle resources to the first - stage calculation.

[0054] Specifically, assume that the system load status level is high load (request quantity 1500, GPU utilization rate 90%), the target inference model is Transformer (the first type of model), and the first resource allocation is 4 high-memory-bandwidth GPUs. At this time, the system identifies 2 idle GPU cores, and the second resource allocation assigns these 2 idle cores to the first-stage calculation to accelerate the key-value cache generation and reduce the inference latency; The system load status level is low load (request quantity 100, GPU utilization rate 20%), the target inference model is Transformer (the first type of model), and the first resource allocation is 2 GPUs. The system identifies 3 idle GPU cores, and the second resource allocation assigns 2 of these idle cores to the first-stage calculation to optimize the calculation efficiency of the pre-filling stage and improve the resource utilization rate.

[0055] For the high resource requirements of the first-stage calculation of the first type of model, preferentially allocate idle resources to ensure the efficient execution of key calculations such as key-value cache generation, thereby improving the inference performance. Whether the system is in high load or low load, dynamically utilizing idle resources can optimize the resource utilization rate and reduce waste. In high-load scenarios, allocate idle resources to relieve the performance bottleneck; in low-load scenarios, make full use of idle resources to improve the system efficiency.

[0056] In one embodiment, determining the second resource allocation according to the first resource allocation and the system load status level further includes: In response to the system load status level being high load and the target inference model being the second type of model, identify the idle resources and allocate the idle resources to the second-stage calculation; In response to the system load status level being low load and the target inference model being the second type of model, identify the idle resources and allocate the idle resources to the second-stage calculation.

[0057] Specifically, assume that the system load status is high load (request quantity 1800, GPU utilization rate 88%), the target inference model is CNN (the second type of model), and the first resource allocation is 3 GPU cores. The system identifies 1 idle GPU core, and the second resource allocation assigns this idle core to the second-stage calculation to accelerate the classification output generation and reduce the inference latency.

[0058] The system load status level is low load (request quantity 120, GPU utilization rate 18%), the target inference model is CNN (the second type of model), and the first resource allocation is 2 GPU cores. The system identifies 3 idle GPU cores, and the second resource allocation assigns 2 of these idle cores to the second-stage calculation to optimize the feature map processing efficiency and improve the resource utilization rate.

[0059] For the high parallel computing requirements in the second stage of the second type of models (such as CNN or RNN), idle resources are preferentially allocated to ensure the efficient execution of key computations such as feature map processing or classification layers, thereby improving the inference performance and response speed.

[0060] In one embodiment, as Figure 3 shown, the inference request is computed according to the second resource allocation to obtain an inference result, including: Extract the input data from the request parameters; Compute the input data through the first stage of computation to obtain intermediate data and transmit it to the second stage of computation; Compute the intermediate data through the second stage of computation to generate an inference result; Store the inference result in the storage area.

[0061] Among them, the storage area refers to the memory or disk space (such as HBM memory) used to store the inference result to ensure that the result is accessible or transmissible.

[0062] Specifically, by extracting the input data (such as 64 images of 224×224) from the inference request and through staged computations, first generate intermediate data such as feature maps through the convolutional layer, and then generate classification results through the fully connected layer to ensure an accurate match with the task requirements, optimizing the data flow and computational efficiency. Dynamically allocate idle resources, such as allocating 1 GPU core under high load (1800 requests, GPU utilization rate 88%) or 2 cores under low load (120 requests, GPU utilization rate 18%), to accelerate the second stage of computation and reduce the inference latency (such as from 150ms to 100ms under high load, from 120ms to 80ms under low load), while avoiding over-allocation of resources, maintaining system stability, and reserving resources for concurrent tasks. Automated resource identification, data extraction, intermediate data transmission, and result storage reduce manual intervention, improving the system scalability and reliability. The result is stored in high-bandwidth memory to support fast access, suitable for real-time applications and complex scenarios.

[0063] In one embodiment, computing the input data through the first stage of computation to obtain intermediate data and transmitting it to the second stage of computation includes: Split the input data to obtain at least one data sub-batch; Compute the first data sub-batch through the first stage of computation to generate intermediate data; While transmitting the intermediate data to the second stage of computation for computation, compute the second data sub-batch through the first stage of computation.

[0064] Among them, the data sub-batch (micro batches) means that each data sub-batch contains some samples in the inference request and is used to support finer-grained parallel computing.

[0065] Specifically, after the input data is split, the system first sends the first data sub-batch to the first-stage calculation, for example, a computing node for performing self-attention calculation. After completing the processing of the first sub-batch in this stage, the system immediately transmits the generated intermediate data (i.e., the output result of the first-stage calculation) to the second-stage calculation, for example, a computing node for performing a feed-forward network (FFN) or other operator operations for further processing. At the same time, the first-stage calculation does not need to wait for the second-stage processing to complete, but immediately receives and processes the second data sub-batch and continues to perform its first-stage calculation. Each data sub-batch forms a decoupled asynchronous pipelining relationship between the two calculation stages, that is, while the first stage continuously produces intermediate data, the second stage processes the intermediate results of the previous sub-batch in parallel. This fully utilizes the characteristics of high-bandwidth computing resources (suitable for the first stage) and high-computing-power computing resources (suitable for the second stage) in heterogeneous chips, thereby improving the throughput and execution efficiency of the overall inference task.

[0066] In one embodiment, as Figure 3 shown, storing the inference result in the storage area includes: The storage area includes: a high-frequency access area and a low-frequency access area; Generating a hash value for the inference result through a hash algorithm; In response to the existence of a hash value corresponding to the inference result in the high-frequency access area, record the hash value access event and update the access frequency statistics of the high-frequency access area; In response to the existence of a hash value corresponding to the inference result in the low-frequency access area, record the hash value access event and update the access frequency statistics of the low-frequency access area; In response to the access frequency of the hash value in the high-frequency access area reaching the high-frequency threshold periodically within a preset time, migrate the inference result corresponding to the hash value to the low-frequency access area; In response to the access frequency of the hash value in the low-frequency access area reaching the low-frequency threshold periodically within a preset time, migrate the inference result corresponding to the hash value to the high-frequency access area; In response to insufficient memory in the storage area, delete the inference result with the lowest access frequency statistics in the low-frequency access area.

[0067] Among them, the high-frequency access area refers to the part of the storage area used to store frequently accessed inference results, which is allocated on high-performance storage media; the low-frequency access area refers to the part of the storage area used to store less frequently accessed inference results, which is allocated on lower-cost storage media; the hash algorithm refers to an algorithm that maps inference results to a fixed-length hash value (such as MD5 or SHA-1), used to uniquely identify inference results for fast retrieval and comparison; the hash value refers to the unique identifier generated for the inference result by the hash algorithm, used to locate and track the access situation of the inference result in the storage area. The access event refers to the operation of reading or querying the inference result, recorded as the access record of the hash value, used to count the access frequency. The high-frequency threshold refers to the lower threshold of the access frequency of the hash value in the high-frequency access area. If the access frequency of a certain hash value is lower than this threshold within the preset time period, it indicates that its access heat has decreased; the low-frequency threshold refers to the upper threshold of the access frequency of the hash value in the low-frequency access area. If the access frequency of a certain hash value exceeds this threshold within the preset time period, it indicates that its access heat has increased; the preset time period refers to the time window used to count the access frequency.

[0068] Specifically, by dividing the storage area into high-frequency and low-frequency access areas, the inference results are accurately allocated to high-performance storage (such as HBM memory) or low-cost storage (such as ordinary disks), ensuring low-latency reading of frequently accessed data while saving high-cost storage resources. The hash algorithm generates a unique hash value to accelerate data retrieval and positioning, optimizing the management efficiency in large-scale and high-concurrency inference scenarios. By recording access events and updating the access frequency statistics within the preset time period, the system can accurately capture the data access pattern, dynamically migrate the inference results in combination with the high-frequency and low-frequency thresholds, making the high-frequency data resident in high-performance storage and the low-frequency data transferred to low-cost storage, thus achieving the best balance between performance and resource efficiency. When the memory in the storage area is insufficient, the inference result with the lowest access frequency in the low-frequency access area is deleted to release space without affecting the access to critical data, ensuring the stability and scalability of the system.

[0069] In one embodiment, during the calculation according to the second resource allocation, the actual resource utilization rate is monitored in real time. When the deviation exceeds the preset threshold, resource reallocation is triggered, including: Obtain the actual utilization rate of the computing resources; Obtain the expected utilization rate based on the resource configuration of the second resource allocation; Perform a comparison calculation based on the actual utilization rate and the expected utilization rate to obtain a deviation value; In response to the deviation value exceeding the utilization threshold range, resource reallocation is performed.

[0070] Among them, the actual utilization rate refers to the actual usage degree of computing resources during the running of the inference task, expressed as a percentage, covering the computing load of GPU cores; the expected utilization rate refers to the resource usage rate of the inference task in an ideal state according to the resource configuration of the second resource allocation; the deviation value refers to the difference between the actual utilization rate and the expected utilization rate.

[0071] Specifically, assume that the target inference model is a first-class model. Based on the second resource allocation, 4 GPU cores and 8GB HBM memory are allocated for this task. The expected GPU utilization rate is 75%, and the memory utilization rate is 85%; the actual utilization rate collected is 92%, and the memory occupancy is 7.8GB, that is, the memory utilization rate is 97.5%. The memory deviation is: ; Assume that the utilization rate threshold range is 10% to 20%, that is, the deviation value <10% or >20% triggers reallocation. Since the GPU deviation (22.67%) exceeds the upper limit of the threshold range of 20%, and the memory deviation (14.71%) is within the threshold range, the system triggers resource reallocation only for the GPU. The system scans and finds that 2 idle GPU cores are available, so the resources are adjusted to 5 GPU cores, keeping the memory at 8GB unchanged, updating the second resource allocation configuration and applying it to subsequent inference tasks. If there is no idle resource, the system can reduce the batch size (such as from 32 to 24) to relieve the GPU load pressure.

[0072] In one embodiment, a model inference performance optimization device is provided, including: a receiving module 710, a first allocation module 720, a load level determination module 730, a second allocation module 740, and an execution module 750: The receiving module 710 is configured to, in response to receiving an inference request, identify the structural features of the target inference model, and determine the type of the target inference model according to the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request; The first allocation module 720 is configured to determine the first resource allocation according to the type of the target inference model and the request parameters; The load level determination module 730 is configured to obtain system load data and determine the system load status level according to the system load data; The second allocation module 740 is configured to determine the second resource allocation according to the first resource allocation and the system load status level; The execution module 750 is configured to perform calculations on the inference request according to the second resource allocation to obtain an inference result.

[0073] In one embodiment, the receiving module 710 is configured to: Read the architecture of the target inference model through model structure analysis, and extract the architecture features in the architecture of the target inference model; In response to the architecture features matching the preset architecture features, the target inference model is a first - type model; In response to the architecture features not matching the preset architecture features, the target inference model is a second - type model.

[0074] In one embodiment, the load level determination module 730 is configured to: The system load data includes: the number of requests, the computing resource utilization rate, the storage resource status, and the network latency; Obtain historical system load data; Create a prediction model based on the historical system load data; Analyze the number of requests, the computing resource utilization rate, the storage resource status, and the network latency through the prediction model to determine the system load status level.

[0075] In one embodiment, the execution module 750 is configured to: Extract the input data from the request parameters; Perform calculations on the input data through the first - stage calculation to obtain intermediate data and transmit it to the second - stage calculation; Perform calculations on the intermediate data through the second - stage calculation to generate an inference result; Store the inference result in the storage area.

[0076] It should be understood that although Figure 2 the steps in the device structure block diagram are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, Figure 2 at least a part of the steps in

[0077] An embodiment of the present application also provides a computer - readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any of the above - mentioned method embodiments for optimizing the model inference performance when running.

[0078] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memory (ROM), random access memory (RAM), external hard drives, magnetic disks, or optical discs that can store computer programs.

[0079] The embodiments of the present application also provide a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the embodiments of the above model inference performance optimization method.

[0080] Those skilled in the art can further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. For example, Figure 4 As shown, to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0081] The above has introduced in detail a model inference performance optimization method provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method for optimizing model inference performance, characterized in that, Including: In response to receiving an inference request, identify the structural features of the target inference model, and determine the type of the target inference model according to the structural features; Wherein the inference request includes request parameters, and the target inference model is used to execute the inference request; Determine the first resource allocation according to the type of the target inference model and the request parameters; Obtain system load data, and determine the system load status level according to the system load data; Determine the second resource allocation according to the first resource allocation and the system load status level; Perform calculations on the inference request according to the second resource allocation to obtain an inference result.

2. The method for optimizing model inference performance according to claim 1, wherein Identifying the structural features of the target inference model and determining the type of the target inference model according to the structural features includes: Read the architecture of the target inference model through model structure analysis, and extract the architecture features in the target inference model; In response to the architecture features matching the preset architecture features, the target inference model is a first type of model; In response to the architecture features not matching the preset architecture features, the target inference model is a second type of model.

3. The method for optimizing the model inference performance according to claim 2, wherein Determining the first resource allocation according to the type of the target inference model and the request parameters includes: In response to the target inference model being a first type of model, determine the first resource allocation according to the allocation strategy corresponding to the first type of model and the request parameters; In response to the target inference model being a second type of model, determine the first resource allocation according to the allocation strategy corresponding to the second type of model and the request parameters.

4. A method for optimizing the performance of model inference according to claim 3, characterized in that Determining the first resource allocation according to the type of the target inference model and the request parameters further includes: Split the calculation process of the inference request of the target inference model into a first-stage calculation and a second-stage calculation, where the first-stage calculation is used to process input data and generate intermediate values, and the second-stage calculation is used to generate output results based on the intermediate values; Determine the initial resource allocation according to the request parameters; In response to the target inference model being a first type of model, the resource allocation ratio for the first-stage calculation of the inference request of the first type of model is greater than that of the second-stage calculation; In response to the target inference model being a second type of model, the resource allocation ratio for the first-stage calculation of the inference request of the second type of model is less than that of the second-stage calculation.

5. A method for optimizing the performance of model inference according to claim 4, characterized in that Obtaining system load data and determining the system load status level according to the system load data includes: The system load data includes: the number of requests, the utilization rate of computing resources, the status of storage resources, and network latency; Obtain historical system load data; Create a prediction model according to the historical system load data; Analyze the number of requests, the utilization rate of computing resources, the status of storage resources, and the network latency through the prediction model to determine the system load status level.

6. The model inference performance optimization method according to claim 5, wherein Creating a prediction model according to the historical system load data includes: Associate the inference request and the system load data to generate structured time data; Clean and extract the structured time data to obtain preset features; Normalize the preset features to obtain a preset feature dataset; Convert the preset feature dataset and the structured time data into a time series input format; Set the prediction model based on the time series input format.

7. A method for optimizing the performance of model inference according to claim 5, characterized in that Analyze the request quantity, the computing resource utilization rate, the storage resource status, and the network latency through the prediction model to determine the system load status level, including: Normalize the request quantity, the computing resource utilization rate, the storage resource status, and the network latency to obtain a time series dataset; Input the time series dataset into the prediction model to obtain a normalized load index; Denormalize the normalized load index to obtain readable original data; If the readable original data is greater than the first preset load data threshold, the system load status level is high load; If the readable original data is less than the second preset load data threshold, the system load status level is low load, where the first preset load data threshold is greater than the second preset load data threshold.

8. A method for optimizing the performance of model inference according to claim 7, characterized in that, Determine the second resource allocation according to the first resource allocation and the system load status level, including: If the system load status level is high load and the target inference model is the first type of model, identify idle resources and allocate high-bandwidth resources in the idle resources to the first-stage calculation; If the system load status level is high load and the target inference model is the second type of model, identify the idle resources and allocate high-computing-power resources in the idle resources to the second-stage calculation.

9. The method for optimizing the model inference performance according to claim 8, wherein Determining the second resource allocation according to the first resource allocation and the system load status level further includes: If the system load status level is low load and the target inference model is the first type of model, identify the idle resources and allocate the high-bandwidth resources to the first-stage calculation; If the system load status level is low load and the target inference model is the second type of model, identify the idle resources and allocate the high-computing-power resources to the second-stage calculation.

10. A method for optimizing the performance of model inference according to claim 4, characterized in that, Calculate the inference request according to the second resource allocation to obtain an inference result, including: Extract the input data from the request parameters; Calculate the input data through the first-stage calculation to obtain intermediate data and transmit it to the second-stage calculation; Calculate the intermediate data through the second-stage calculation to generate an inference result; Store the inference result in the storage area.

11. A method for optimizing the performance of model inference according to claim 10, characterized in that, Calculate the input data through the first-stage calculation to obtain intermediate data and transmit it to the second-stage calculation, including: Split the input data to obtain at least one data sub-batch; Calculate the first data sub-batch through the first-stage calculation to generate the intermediate data; While transmitting the intermediate data to the second-stage calculation for calculation, calculate the second data sub-batch through the first-stage calculation.

12. A method for optimizing the performance of model inference according to claim 10, characterized in that Storing the inference result in the storage area includes: The storage area includes: a high-frequency access area and a low-frequency access area; Generate a hash value for the inference result through a hash algorithm; In response to the existence of a hash value corresponding to the inference result in the high-frequency access area, record the hash value access event and update the access frequency statistics of the high-frequency access area; In response to the existence of a hash value corresponding to the inference result in the low-frequency access area, record the hash value access event and update the access frequency statistics of the low-frequency access area; In response to the access frequency of the hash value in the high-frequency access area reaching the high-frequency threshold periodically within a preset time, migrate the inference result corresponding to the hash value to the low-frequency access area; In response to the access frequency of the hash value in the low-frequency access area reaching the low-frequency threshold periodically within a preset time, migrate the inference result corresponding to the hash value to the high-frequency access area; In response to insufficient memory in the storage area, delete the inference result with the lowest access frequency statistics in the low-frequency access area.

13. An apparatus for optimizing model inference performance, characterized in that, The device includes: A receiving module, configured to, in response to receiving an inference request, identify the structural features of a target inference model, and determine the type of the target inference model according to the structural features; wherein the inference request includes request parameters, and the target inference model is used to execute the inference request; A first allocation module, configured to determine a first resource allocation according to the type of the target inference model and the request parameters; A load level determination module, configured to obtain system load data and determine the system load status level according to the system load data; A second allocation module, configured to determine a second resource allocation according to the first resource allocation and the system load status level; An execution module, configured to calculate the inference request according to the second resource allocation to obtain an inference result.

14. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images