Deployment method of inference model and computing equipment

Abstract

Embodiments of the invention provide an inference model deployment method and configuration method, and a computing device, relate to the technical field of servers, and are used for ensuring high reliability and stability of a deployed inference model. The method comprises the following steps: acquiring hardware features of a computing cluster and model features of an inference model; selecting a to-be-optimized parameter subspace in the deployment configuration parameter space based on the hardware features and the model features; the deployment configuration parameter space is a set of a group of adjustable deployment configuration parameters; testing a value combination generated based on the parameter subspace, and determining a performance index of the value combination during testing; and deploying the reasoning model on the computing cluster based on the value combination corresponding to the performance index under the condition that the performance index meets a preset performance condition.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a method for deploying an inference model and a computing device. Background Technology

[0002] Inference models, with their pre-defined computational logic, can perform complex logical reasoning, thereby accelerating the intelligent transformation of various industries. However, inference performance has become a core bottleneck for their practical implementation. The quality of inference performance not only directly affects user experience such as response latency and interaction smoothness, but is also closely related to deployment costs and commercial value.

[0003] The computing clusters on which inference models are deployed are mostly heterogeneous hardware environments (including CPUs, GPUs, FPGAs, etc.). Differences in hardware architecture and instruction sets make it extremely difficult to accurately determine the optimal configuration parameters (such as hardware allocation, inference engine selection, caching strategies, etc.). Currently, the industry mostly relies on engineers manually traversing inference engine combinations and exploring a vast parameter space. This process takes several weeks and is limited by experience and knowledge, ultimately only managing to obtain a locally optimal solution. This not only results in high time and manpower costs but also lacks reliable guarantees for model performance and may even lead to a sharp performance drop in edge scenarios. Summary of the Invention

[0004] This application provides a method for deploying an inference model and a computing device to ensure the high reliability and stability of the deployed inference model.

[0005] In a first aspect, embodiments of this application provide a method for deploying an inference model, the method comprising: acquiring hardware characteristics of a computing cluster and model characteristics of an inference model; wherein, the hardware characteristics are used to represent the hardware configuration and / or computing power of the computing cluster; the hardware configuration is used to represent the processing power configuration, data transmission configuration, or memory capacity configuration of the computing cluster; the model characteristics are used to represent the computational complexity and / or resource requirements of the inference model; based on the hardware characteristics and model characteristics, determining the target configuration parameters required to deploy the inference model on the computing cluster; testing the combinations of values ​​of the target configuration parameters, and determining the performance indicators of each combination of values ​​during the test; the performance indicators include the throughput and / or latency of the inference model during the test; and, if the performance indicators meet preset performance conditions, deploying the inference model on the computing cluster based on the combination of values ​​corresponding to the performance indicators.

[0006] The inference model deployment method provided in this application first identifies key target configuration parameters that can affect performance based on the hardware characteristics of the computing cluster and the model characteristics of the inference model. This reduces the parameter search space for configuring the inference model, allowing optimization work to focus on high-performance areas and laying an efficiency foundation for subsequent steps. Then, the combinations of target parameter values ​​are tested and their performance indicators are evaluated. Through systematic comparative analysis, combinations of values ​​that meet preset performance conditions are selected, and the model is deployed on the computing cluster accordingly. This feature-analysis-driven targeted optimization deployment process avoids operational risks caused by improper configuration, ensuring that the final deployed inference model has high reliability and stability.

[0007] In one possible implementation, the combination of values ​​of the target configuration parameter is tested, and the performance indicators of each value combination during the test are determined. This includes: performing a combination search within the range of values ​​of the target configuration parameter based on a multi-objective optimization algorithm to obtain a first value combination; the multi-objective optimization algorithm is used to improve the throughput of the inference model and reduce the latency of the inference model; the inference model is tested based on the first value combination, and the throughput and latency of the inference model during the test are obtained.

[0008] In this implementation, the first combination of values ​​determined by the multi-objective optimization algorithm enables the inference model to achieve higher throughput and lower latency on specific hardware, thereby improving the overall performance of the model and better meeting the needs of fast response and efficient processing in practical applications.

[0009] In one possible implementation, the method further includes: if the performance index does not meet the preset performance conditions, re-determine the second value combination based on the multi-objective optimization algorithm, and determine the performance index of the second value combination when conducting the test.

[0010] In this implementation, when the performance metric fails to meet the preset performance conditions, it indicates that the first value combination obtained through the multi-objective optimization algorithm has failed to meet the requirements, suggesting that the current value combination is insufficient in balancing multiple objectives such as throughput and latency of the inference model. The multi-objective optimization algorithm possesses powerful global search and intelligent adjustment capabilities. Reusing this algorithm allows for a more precise exploration of new value combination directions within the parameter subspace based on performance information from previous tests, generating a second value combination. Subsequent performance testing to obtain new metrics not only visually demonstrates the optimization effect of the second value combination compared to the first, but also provides crucial evidence for further adjustments to the optimization strategy. Through continuous iteration of this process, the ideal value combination that meets the preset performance conditions is gradually approximated, ensuring the efficient and stable operation of the inference model in practical applications.

[0011] In one possible implementation, the method further includes: performing parameter importance analysis on the performance indicators of multiple value combinations to determine the degree of influence of each target configuration parameter on the performance indicators; and performing a combination search within the value range of the target configuration parameters based on a multi-objective optimization algorithm, including: performing a combination search within the value range of the target configuration parameters based on the degree of influence of the target configuration parameters according to the multi-objective optimization algorithm to obtain a first value combination.

[0012] In this implementation, the determination of the first combination of values ​​not only considers the needs of multi-objective optimization (such as improving throughput and reducing latency) but also incorporates the results of parameter importance analysis, ensuring a more scientific and reasonable search process. This first combination of values ​​better adapts to specific hardware environments and model requirements, enabling the inference model to achieve a better performance balance, fully leveraging the hardware's potential, and improving the overall operating efficiency of the inference model.

[0013] In one possible implementation, the method further includes: determining the performance score corresponding to the combination of values ​​based on the performance index; the performance score is positively correlated with throughput and negatively correlated with latency; and determining whether the performance index meets the preset performance conditions based on the performance score.

[0014] In this implementation, the performance score is determined by throughput and latency, which can comprehensively consider the number and speed of tasks processed by the system, and comprehensively and accurately evaluate the system's performance under different combinations of values.

[0015] In one possible implementation, determining whether a performance indicator meets a preset performance condition based on a performance score includes: if the improvement in the performance score of the current value combination compared to the performance score of the previous value combination is less than or equal to a preset improvement, then determining that the performance indicator of the current value combination meets the preset performance condition.

[0016] In this implementation, when the performance score of the current value combination does not exceed a preset improvement threshold compared to the previous round, it indicates that the optimization effect has reached saturation. Continuing to adjust the value combination through multi-objective optimization algorithms may not yield significant performance gains. At this point, determining that the performance index meets the preset conditions ensures that the current value combination can guarantee a relatively stable and efficient state for the inference module, while preventing the pursuit of minor improvements from leading to local optima or unnecessary computational costs. This balances the optimization effect and avoids unnecessary waste of resources.

[0017] In one possible implementation, the performance score corresponding to the combination of values ​​is determined based on performance indicators, including: determining a timeout score based on latency when the latency of the inference model exceeds a preset duration during testing; determining the throughput ratio of throughput to target throughput, and the latency ratio of target latency to latency; determining the performance score corresponding to the combination of values ​​based on the throughput ratio, latency ratio, and timeout score; and the timeout score is negatively correlated with the performance score.

[0018] In this implementation, if the latency exceeds the preset duration, it indicates that the latency is too high. This may be due to reasons such as high model complexity, insufficient system resources, or unreasonable resource allocation. To avoid these problems, a timeout score can be determined, and a penalty mechanism can be used to force subsequent optimization processes to optimize in the direction of reducing latency.

[0019] In one possible implementation, the target configuration parameters required to deploy the inference model to the computing cluster are determined based on hardware features and model features. This includes: for each deployment configuration parameter, determining the adaptability of the deployment configuration parameter to the hardware features and its correlation with the model features; and selecting the target configuration parameter to be optimized from multiple deployment configuration parameters in the deployment configuration parameter space based on the adaptability of each deployment configuration parameter to the hardware features and its correlation with the model features.

[0020] In this implementation, the target deployment configuration parameters to be optimized are selected based on model features and hardware features. This approach can adapt the deployment configuration parameters to the hardware capabilities based on the hardware's computing and memory characteristics, fully tapping the hardware's potential, improving resource utilization efficiency, and avoiding resource waste or overload. At the same time, it can combine model features such as model accuracy and complexity to ensure model performance and quality, preventing model accuracy from decreasing or convergence difficulties due to inappropriate parameters. This makes model deployment more adaptable and flexible, and can better meet the needs of different scenarios.

[0021] In one possible implementation, an inference model is deployed on a computing cluster based on the value combinations corresponding to the performance metrics, including: establishing a mapping relationship between the value combinations corresponding to the performance metrics and the computing cluster and the inference model, and the mapping relationship is used to deploy the inference model on the computing cluster.

[0022] In this implementation, by constructing a mapping relationship between value combinations and computing clusters and inference models, we can not only shorten the deployment and configuration time of inference models in subsequent similar scenarios and avoid repeated optimization processes, but also ensure that each deployment is based on verified configuration parameters, fundamentally guaranteeing the high reliability and long-term stability of the inference model.

[0023] In one possible implementation, the method further includes: determining an appropriate inference engine based on hardware characteristics and model characteristics; determining a deployment configuration parameter space based on hardware characteristics, model characteristics, and the inference engine; and determining the target configuration parameters required to deploy the inference model to the computing cluster based on hardware characteristics and model characteristics, including: determining the target configuration parameters required to deploy the inference model to the computing cluster via the inference engine in the deployment configuration parameter space based on hardware characteristics and model characteristics.

[0024] In this implementation, the most suitable inference engine is first matched based on hardware capabilities and model structure characteristics. Then, the appropriate configuration parameter space is determined based on the "hardware-model-engine" ternary feature. This concentrates optimization resources on the appropriate configuration parameter space, avoiding blind searching across the entire parameter space and ensuring deep adaptation between the configuration scheme and the underlying hardware. This improves deployment efficiency while guaranteeing the stability of the inference model and the high efficiency of resource utilization.

[0025] Secondly, embodiments of this application provide a method for deploying an inference model, comprising: in response to an inference model deployment request, obtaining a target inference model indicated in the inference model deployment request; determining a target value combination corresponding to the current computing cluster and the target inference model based on the mapping relationship between the computing cluster, the inference model and the value combination; the value combination being used to set configuration parameters required for deploying the inference model on the computing cluster; and deploying the target inference model on the current computing cluster based on the target value combination.

[0026] In one possible implementation, the mapping relationship between the computing cluster, the inference model, and the combination of values ​​is determined by the method provided by the first aspect and its possible implementation.

[0027] The inference model deployment method provided in this application eliminates the need for manual enumeration of the four-dimensional parameter matrix consisting of the inference engine, concurrency, input length, and computational precision when configuring the target inference model in the current computing cluster. It also eliminates the need for weeks of manual testing to find local optima. Configuration can be completed quickly using a pre-tested mapping relationship between the computing cluster, the inference model, and the combination of values. Furthermore, since this mapping relationship is pre-tested for the current hardware environment and model characteristics, it ensures accurate matching of configuration parameters, thereby guaranteeing the performance of the deployed target inference model.

[0028] Thirdly, embodiments of this application provide a computing device, which includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the methods provided by the first aspect and its possible implementations or the methods provided by the second aspect and its possible implementations.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium comprising: computer software instructions; which, when executed in a computing device, cause the computing device to implement the method provided by the first aspect and its possible implementations or the method provided by the second aspect and its possible implementations.

[0030] Fifthly, embodiments of this application provide a computer program product that, when run on a computing device, causes the computing device to execute the steps of the related method described in the first aspect to implement the method of the first aspect; or causes the computing device to execute the steps of the related method described in the second aspect to implement the method of the second aspect.

[0031] The beneficial effects of the third to fifth aspects mentioned above can be referred to the corresponding descriptions of the first or second aspects, and will not be repeated here. Attached Figure Description

[0032] Figure 1 A flowchart illustrating a deployment method for an inference model provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating a deployment method for an inference model provided in this application embodiment. Figure 2 ; Figure 3 A schematic diagram of a scanner architecture provided in this application embodiment. Figure 1 ; Figure 4 A schematic diagram of a scanner architecture provided in this application embodiment. Figure 2 ; Figure 5 A schematic diagram of a scanner architecture provided in this application embodiment. Figure 3 ; Figure 6 A schematic diagram of a scanner architecture provided in this application embodiment. Figure 4 ; Figure 7 A flowchart illustrating another deployment method for an inference model provided in an embodiment of this application; Figure 8 This is a schematic diagram of the overall process of configuring an inference model provided in an embodiment of this application; Figure 9 This is a schematic diagram of the architecture of a computing device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0036] Currently, large language models are accelerating the intelligent transformation of various industries with their breakthrough capabilities in deep semantic understanding and complex logical reasoning. However, inference performance has become a core bottleneck for their practical implementation. The quality of inference performance not only directly affects user experience such as response latency and interaction smoothness, but is also closely related to deployment costs and commercial value. At the same time, the resource requirements of models of different sizes vary greatly. Distillation models below 10B consume less resources, but their performance is compromised. On the other hand, models with tens of billions of bytes have better accuracy, but their memory / computing power requirements are 5-20 times higher than those of distillation models, making the performance-cost trade-off during deployment more challenging.

[0037] Looking further, the computing clusters on which inference models are deployed are mostly heterogeneous hardware environments (including CPUs, GPUs, FPGAs, etc.). Differences in hardware architecture and instruction sets make it extremely difficult to accurately determine the optimal configuration parameters (such as hardware allocation, inference engine selection, caching strategies, etc.). Currently, the industry largely relies on engineers manually traversing inference engine combinations and exploring a vast parameter space. This process takes several weeks and is limited by experience and knowledge, ultimately only managing to obtain a locally optimal solution. This not only incurs high time and manpower costs but also lacks reliable guarantees for model performance, and may even lead to a sharp performance drop in edge scenarios.

[0038] Based on this, embodiments of this application provide a method for deploying an inference model. This method first identifies key target configuration parameters that can affect performance based on the hardware characteristics of the computing cluster and the model characteristics of the inference model. This reduces the parameter search space for configuring the inference model, allowing optimization efforts to focus on high-performance areas and laying an efficiency foundation for subsequent steps. Furthermore, the combinations of target parameter values ​​are tested and their performance indicators are evaluated. Through systematic comparative analysis, combinations of values ​​that meet preset performance conditions are selected, and the model is deployed on the computing cluster accordingly. This feature-analysis-driven, targeted optimization deployment process avoids operational risks caused by improper configuration, ensuring that the final deployed inference model has high reliability and stability.

[0039] The inference model deployment method provided in this application can be applied to computing devices. This application does not limit the specific form of the computing device. For example, the computing device can be a server. The server can be a single server or a server cluster composed of multiple servers. In some embodiments, the server cluster can also be a distributed cluster, which is not limited in this application. As another example, the computing device can specifically be a terminal device. The terminal device can be referred to as: terminal, user equipment (UE), terminal device, access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent, or user equipment, etc. The terminal device can specifically be a mobile phone, augmented reality (AR) device, virtual reality (VR) device, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc.

[0040] Figure 1 This is a flowchart illustrating a method for deploying an inference model, as provided in an embodiment of this application. Figure 1 As shown, the deployment method of the inference model provided in this application embodiment may include the following steps: S101. Obtain the hardware characteristics of the computing cluster and the model characteristics of the inference model.

[0041] Hardware characteristics are used to represent the hardware configuration and / or computing power of the computing cluster's hardware resources. Hardware configuration refers to the collection of specifications, topology, and interconnection methods of various computing resources in the computing cluster. Specifically, it can include: processing power configuration (such as CPU / GPU model, number of cores, and computing power indicators), data transmission configuration (including interconnection bandwidth inside and outside nodes, network topology, and storage I / O performance), and memory capacity configuration (covering the capacity and bandwidth of host memory and accelerator memory). Together, they constitute the basic physical resource architecture that can support the execution of computing tasks. For example, hardware configuration can include at least one of the following: hardware type (such as heterogeneous components such as CPU, GPU, FPGA, ASIC, etc.), number of CPU cores / threads, GPU memory capacity (such as 8GB / 24GB / 48GB) and memory bandwidth (such as 500GB / s), number of FPGA logic units, storage device type (SSD / HDD) and capacity, and RAM size and frequency. These parameters directly determine the scale of the basic hardware resources.

[0042] Computing power is used to reflect the efficiency of hardware in processing data. It can include at least one of the following: CPU floating-point operation power (such as GFLOPS / TFLOPS), GPU single-precision / half-precision computing performance, FPGA data throughput (such as GB / s), hardware parallel processing power (such as the number of CUDA cores and TensorCores of GPU), and hardware-supported acceleration instruction sets (such as AVX-512 of CPU and TensorRT optimization support of GPU), reflecting the efficiency of hardware in carrying model computing tasks.

[0043] Model features are used to represent the computational complexity and / or resource requirements of an inference model. Computational complexity is directly related to the amount of data the model processes. Core model features may include at least one of the following: model parameter size (e.g., 10B / 32B / hundreds of billions), number of neural network layers (e.g., number of encoder / decoder layers in a Transformer architecture), number of neurons per layer and activation function type (e.g., ReLU, GeLU), computational operation density (e.g., the proportion of matrix multiplication and convolution operations), and sequence length (e.g., the upper limit of the number of tokens in the input text). Resource requirements are used to represent the consumption of hardware resources during model inference. The corresponding model characteristics may include at least one of the following: peak memory usage (e.g., 2GB for the distillation model and 20GB for the billion-model), computing power requirements (e.g., how many TFLOPS of computing power are needed per second for inference), data transmission requirements (e.g., bandwidth requirements for model weight loading and input / output data transmission), and memory usage (e.g., memory consumption of temporary data during model runtime). These directly affect the feasibility and performance of model deployment on hardware.

[0044] This application does not limit the method by which computing devices obtain the hardware characteristics of a computing cluster. As one implementation, when the computing cluster includes computing devices, the computing devices can obtain the hardware configuration and basic operating status of the computing cluster through system-level tools and API calls. For heterogeneous computing clusters containing multiple types of hardware, hardware characteristics can also be collected uniformly through integrated tools (such as open-source unified monitoring tools, cloud platform monitoring components, etc.), and this application does not impose any limitations on this approach.

[0045] This application does not limit the way a computing device obtains model features of an inference model. As one implementation, the computing device can extract basic model features (such as parameter size and number of network layers) from the model file of the inference model using parsing tools. As another feasible implementation, if the model provides official documentation or metadata (such as a model card for a pre-trained model or metadata embedded in a custom model), model features can be determined through model metadata and documentation queries. Furthermore, the computing device can also obtain resource requirement features (such as peak memory usage and computing power requirements) through dynamic monitoring during model inference; this application does not impose any limitations on these aspects.

[0046] S102. Based on hardware and model characteristics, determine the target configuration parameters required to deploy the inference model to the computing cluster.

[0047] Configuration parameters represent the necessary configuration steps for deploying the inference model onto the hardware resources of the computing cluster via the inference engine. These parameters play a crucial role in deploying the inference model to the computing cluster's hardware resources, precisely describing and controlling various configuration settings during the deployment process. The inference engine is a software framework or toolset used to load the inference model onto the hardware devices of the computing cluster (such as CPUs, GPUs, FPGAs, ASICs, etc.) and efficiently execute the model's inference task—generating prediction results based on input data. It shields the complexity of the underlying hardware, providing developers with a unified interface to run the model, enabling rapid deployment and application across different hardware platforms.

[0048] This application does not limit the specific implementation of deployment configuration parameters. For example, deployment configuration parameters may include hardware resource allocation parameters, such as at least one of computing unit allocation, memory allocation, and storage resource allocation. Another example is that deployment configuration parameters may include model execution parameters, such as at least one of batch size, data precision, or parallel computing strategy. Yet another example is that deployment configuration parameters may include parameters related to the inference engine when processing model inference tasks, such as at least one of computational precision parameters, parallel computing parameters, or hardware acceleration parameters. It should be understood that deployment configuration parameters may also include at least one of KV cache optimization, quantization strategy, memory optimization, and structured output; this application does not limit these aspects.

[0049] For example, computation unit allocation parameters are used in systems with multi-core CPUs, GPUs, and dedicated AI acceleration chips (such as NPUs) to determine which computation units the inference model uses and the load ratio of each computation unit. For instance, some computational tasks can be allocated to the GPU, and another part to the NPU, optimizing performance by adjusting the allocation ratio. Memory allocation parameters are the amount of video memory (for GPUs) or system memory allocated to the model. Appropriate memory allocation parameters can prevent performance degradation or deployment failures due to insufficient memory, while also improving memory utilization efficiency. Storage resource allocation determines the model's storage location and storage method on storage devices (such as hard drives and SSDs). Different storage devices have different read / write speeds and capacities; choosing the appropriate storage device and storage method can affect the model's loading time and running efficiency. Batch size parameters refer to the number of input samples processed simultaneously in a single forward propagation. The choice of batch size affects the model's inference speed and memory usage. Data precision includes the precision of model parameters and input data, such as using 32-bit floating-point numbers (FP32), 16-bit floating-point numbers (FP16), or 8-bit integers (INT8). Parallel computing strategies can include data parallelism, model parallelism, or hybrid parallelism. Different parallel strategies are suitable for different models and hardware architectures. Choosing the appropriate strategy can fully leverage the parallel computing capabilities of the hardware and improve inference performance.

[0050] Based on hardware and model characteristics, determining the target configuration parameters required to deploy the inference model to the computing cluster from all deployment configuration parameters can reduce the complexity and computational cost of the optimization process while ensuring optimization effectiveness. Hardware characteristics (such as the number of CPU cores, GPU memory size, and memory bandwidth) and model characteristics (such as the number of model parameters, model structure type, and input / output dimensions) significantly affect the relationship between deployment configuration parameters and model performance. For example, for models with a large number of parameters, the choice of batch size is strictly limited when running on GPUs with limited memory, which is completely different from the situation on GPUs with large memory.

[0051] This application does not impose specific restrictions on the method of selecting target configuration parameters. As an implementation approach, engineers usually have deployment experience on similar hardware environments and model architectures, so they can refer to previously successfully deployed configuration parameters as target configuration parameters.

[0052] As an alternative approach, a surrogate model can be constructed using machine learning algorithms (such as regression models and neural networks). This model can approximate the relationship between deployment configuration parameters and model performance. In practice, experiments are first conducted with a limited number of value combinations to collect corresponding performance data, which is then used to train the surrogate model. Since hardware features (such as the number of CPU cores, GPU memory size, and memory bandwidth) and model features (such as the number of model parameters and model structure type) influence the relationship between parameters and performance, using them as input features for the surrogate model allows the model to more accurately learn the correlation between parameters and performance under different conditions, thereby improving prediction accuracy. Subsequently, parameter sensitivity analysis is performed using the trained surrogate model to predict the model's performance under different value combinations, thus selecting the target configuration parameters. For example, using a random forest algorithm to construct the surrogate model, analyzing the importance of each feature in the random forest can clarify which parameters have a more significant impact on model performance.

[0053] In some embodiments, when determining the target configuration parameter, its corresponding target value range can be determined simultaneously. Specifically, S102 can be implemented as follows: based on hardware characteristics and model characteristics, select the parameter subspace to be optimized in the deployment configuration parameter space. Here, the deployment configuration parameter space refers to a high-dimensional set composed of all adjustable deployment configuration parameters and their corresponding value ranges, where each dimension represents a configurable parameter, and the value range of each dimension defines all possible legal values ​​for that parameter. Selecting the parameter subspace to be optimized in the deployment configuration parameter space means determining the target configuration parameter and its corresponding target value range among all adjustable deployment configuration parameters.

[0054] As one approach, parameter sensitivity analysis can be performed using a proxy model trained with machine learning algorithms to predict the model's performance under different value combinations, thereby selecting the target configuration parameters and their corresponding target value ranges.

[0055] As another implementation method, pruning algorithms can be used to reduce the number of value combinations in the deployment configuration parameter space, resulting in a parameter subspace to be optimized. Pruning algorithms can systematically analyze and filter the deployment configuration parameter space based on certain evaluation criteria and optimization objectives. Specifically, the pruning algorithm first evaluates the performance of each value combination, obtaining its corresponding model performance indicators, such as inference speed, accuracy, and resource utilization, through surrogate model prediction or actual small-scale experimental measurements. Then, based on preset thresholds or optimization strategies, value combinations that do not significantly improve performance or contribute little to overall performance are removed. After this series of filtering and removal operations, the originally large deployment configuration parameter space is streamlined, ultimately resulting in a smaller, more targeted parameter subspace with optimization potential.

[0056] S103. Test the combination of values ​​of the target configuration parameters and determine the performance index of each value combination during the test.

[0057] Generating target configuration parameters eliminates redundant and inefficient combinations, focusing on key areas that can improve performance. Subsequently, combinations of target configuration parameter values ​​are generated, and these combinations are then tested to obtain key performance indicators such as throughput, latency, and memory usage. These indicators directly reflect the inference model's performance on the hardware environment under different value combinations, including efficiency (throughput, latency) and resource utilization (memory usage, CPU / GPU utilization). This facilitates the evaluation and comparison of different combinations, allowing for the selection of high-performance parameter configurations to maximize model performance and achieve efficient use of hardware resources.

[0058] This application does not limit the specific implementation of performance metrics. For example, performance metrics may include throughput, latency, video memory usage, and CPU / GPU utilization. Throughput refers to the amount of data or requests processed by the model per unit of time. Latency is the time elapsed from inputting data into the model to the model producing output results. During operation, the model uses video memory to store model parameters, intermediate results, and other data. Video memory resources are limited, and excessively high video memory usage may cause the model to fail to run or result in performance degradation. CPU / GPU utilization reflects the degree of utilization of hardware computing resources.

[0059] As one implementation method, performance metrics include the throughput and latency of the inference model during testing; S103 includes the following S1031-S1032: S1031. Based on the multi-objective optimization algorithm, a combined search is performed within the range of values ​​of the target configuration parameters to obtain the first combination of values.

[0060] Multi-objective optimization algorithms are algorithms that can simultaneously consider multiple optimization objectives, such as genetic algorithms, particle swarm optimization, and multi-objective evolutionary algorithms. In this embodiment, it can simultaneously improve the throughput and reduce the latency of the inference model. These two objectives are often mutually restrictive; for example, increasing the batch size may increase throughput, but it may also increase latency. The purpose of multi-objective optimization algorithms is to find a balance among these conflicting objectives, to find an optimal combination of values.

[0061] Multi-objective optimization algorithms systematically search for the target configuration parameters within their target value range. They can traverse different combinations of values ​​within this range based on specific search strategies, such as selection, crossover, and mutation operations in genetic algorithms, or particle movement and update rules in particle swarm optimization. By continuously evaluating the model performance (primarily focusing on throughput and latency) under each value combination, the multi-objective optimization algorithm selects one or more value combinations that effectively satisfy both the objectives of increasing throughput and reducing latency, designating these as the first value combination. This first value combination represents the relatively optimal solution found by the multi-objective optimization algorithm within the target value range of the current target configuration parameters. The first value combination determined by the multi-objective optimization algorithm enables the inference model to achieve higher throughput and lower latency on specific hardware, thereby improving the overall performance of the model and better meeting the demands for fast response and efficient processing in practical applications.

[0062] As a feasible implementation method, the method also includes: performing parameter importance analysis on the performance indicators of multiple value combinations to determine the degree of influence of each target configuration parameter on the performance indicators; S1031 can be implemented as: based on a multi-objective optimization algorithm, performing a combination search within the value range of the target configuration parameters according to the degree of influence of each target configuration parameter to obtain the first value combination.

[0063] Before performing parameter importance analysis on performance metrics with multiple value combinations, the impact of target configuration parameters on performance metrics is unknown and complex. By collecting performance metric data under different value combinations and using statistical analysis, machine learning, and other methods to perform parameter importance analysis, the degree of influence of each deployment configuration parameter on performance metrics (such as throughput and latency) can be clarified. Furthermore, multi-objective optimization algorithms can adjust their search strategies based on the degree of influence of each deployment configuration parameter. For parameters with a high degree of influence, the algorithm will pay more attention and conduct a more refined search, such as performing denser sampling and deeper exploration within the value range of these parameters; for parameters with a low degree of influence, the search precision and range can be appropriately reduced. This allows the algorithm to more effectively find the optimal value combination, improve search efficiency, and avoid wasting excessive computational resources on irrelevant parameters.

[0064] This application does not limit the specific methods of parameter importance analysis. For example, the sensitivity of different parameters to performance can be identified through ±10% disturbance testing, performance stability assessment, or safety boundary analysis, thus identifying key parameters and performing robust corrections. The ±10% disturbance test is a method that observes changes in system performance by artificially applying a certain magnitude change to a parameter (increasing or decreasing the original parameter value by 10%). Performance stability assessment evaluates the system's ability to maintain relative performance stability under parameter disturbances. Safety boundary refers to the permissible range of parameter variation while ensuring that system performance and stability requirements are met.

[0065] In this implementation, the determination of the first combination of values ​​not only considers the needs of multi-objective optimization (improving throughput and reducing latency) but also incorporates the results of parameter importance analysis, ensuring a more scientific and reasonable search process. This first combination of values ​​better adapts to specific hardware environments and model requirements, enabling the inference model to achieve a better performance balance, fully leveraging the hardware's potential, and improving the overall operating efficiency of the inference model.

[0066] S1032. Test the inference model based on the first value combination, and obtain the throughput and latency of the inference model during the test.

[0067] After obtaining the first value combination, it is applied to the inference model for testing. The inference model configured with the first value combination can be run in a specific hardware environment, simulating actual input data and request load. This allows for precise measurement of the inference model's throughput when processing data, i.e., the number of requests or the amount of data successfully processed per unit time; simultaneously, the latency from input data to output results is recorded. These measured throughput and latency data intuitively reflect the performance of the first value combination in actual operation, providing important basis for further evaluation and optimization of model parameters.

[0068] As shown in S1031-S1032, the solution provided in this embodiment uses a multi-objective optimization algorithm to perform a combined search of deployment configuration parameters within the parameter subspace to obtain a first combination of values. This multi-objective optimization algorithm can simultaneously address the two key, sometimes conflicting, objectives of improving inference model throughput and reducing latency. Traditional single-objective optimization often focuses on optimizing only one metric while neglecting others, potentially leading to a one-sided improvement in model performance. However, the multi-objective optimization algorithm, through intelligent searching within the parameter subspace, can find a combination of values ​​that achieves a good balance between throughput and latency, resulting in a comprehensive improvement in overall model performance and meeting the dual requirements of efficient processing and rapid response in practical applications. Testing the inference model using the first combination of values ​​obtained from the multi-objective optimization algorithm yields throughput and latency data that more accurately reflect the model's actual performance after optimization.

[0069] S104. If the performance indicators meet the preset performance conditions, deploy the inference model on the computing cluster based on the value combination corresponding to the performance indicators.

[0070] After obtaining performance metrics such as throughput and latency, it is necessary to compare these actually measured performance metrics with pre-defined performance conditions. These pre-defined performance conditions are determined comprehensively based on factors such as the needs of the actual application scenario and the limitations of hardware resources. For example, in offline inference scenarios processing large-scale data, a high throughput level may be required. Only when the tested performance metrics meet these pre-defined conditions will the inference model be deployed on the computing cluster based on the corresponding combination of performance metric values.

[0071] When deploying an inference model, the parameters in the combination of values ​​need to be transformed into specific configuration instructions for a particular inference engine, and an executable deployment instance needs to be generated. This process includes packaging the optimized model file, dependency libraries, and this configuration file into a complete runtime environment, and then scheduling and starting this instance on the target compute node through a cluster management tool (such as Kubernetes), and finally completing the online deployment of the inference model.

[0072] In some embodiments, S104 can be implemented as follows: based on the value combination corresponding to the performance index, establish a mapping relationship between the value combination corresponding to the performance index and the computing cluster and the inference model, and the mapping relationship is used to deploy the inference model on the computing cluster.

[0073] To enable the validated value combinations to be used in multiple scenarios, a mapping relationship can be established between value combinations that meet preset performance conditions and the computing cluster and inference model. By establishing this mapping relationship, optimization results can be rapidly migrated across scenarios. When faced with new deployment requirements, historically validated configuration combinations can be automatically matched based on the target cluster hardware characteristics and the features of the model to be deployed, significantly reducing the cost of repeated tuning and enabling continuous accumulation and intelligent reuse of deployment experience. This not only significantly shortens the deployment and configuration time of the inference model in subsequent similar scenarios and avoids repeated optimization processes, but also ensures that each deployment is based on validated optimal parameters, fundamentally guaranteeing the high reliability and long-term stability of the inference model.

[0074] As one implementation, a combination of values ​​corresponding to performance metrics that meet preset performance conditions can be called a value combination template. This template is used to set the configuration parameters required for deploying inference models on the computing cluster via the inference engine. With the value combination template, when deploying inference models in the computing cluster, there is no need to perform complex parameter searches and performance tests again. The inference engine can directly use the template for deployment, avoiding repeated parameter optimization and testing processes, and improving deployment efficiency and consistency.

[0075] As one implementation method, after obtaining the value combination template, it can be verified and tested. The verified value combination template is then stored in the storage system for later use. The verified value combination template can ensure the stable operation of the inference model on the computing cluster, reduce system failures and performance fluctuations caused by improper parameter configuration, and improve the reliability and stability of the system.

[0076] As can be seen from S101-S104, the deployment method of the inference model provided in this application first determines the key target configuration parameters that can affect performance based on the hardware characteristics of the computing cluster and the model characteristics of the inference model. This reduces the parameter search space for configuring the inference model, allowing optimization work to focus on areas with higher efficiency and laying an efficiency foundation for subsequent steps. Furthermore, the combinations of target parameter values ​​are tested and their performance indicators are evaluated. Through systematic comparative analysis, combinations of values ​​that meet preset performance conditions are selected, and the model is deployed on the computing cluster accordingly. This feature-driven, targeted optimization deployment process avoids operational risks caused by improper configuration, ensuring that the final deployed inference model has high reliability and stability.

[0077] In some embodiments, multi-objective optimization algorithms need to simultaneously satisfy multiple objectives (such as simultaneously increasing throughput and reducing latency), and these two objectives often conflict with each other. For example, increasing the batch processing size may increase throughput, but it will increase latency. Moreover, there are numerous deployment configuration parameters, and there are complex nonlinear relationships between them. Changing one parameter may trigger a chain reaction of other parameters, thus affecting multiple objectives. This conflict between objectives and the complexity of parameter relationships make it difficult for the algorithm to find the optimal combination of values ​​that satisfies all objectives at once.

[0078] As a feasible implementation method, the method provided in this application embodiment further includes: when the performance index does not meet the preset performance conditions, re-determining the second value combination based on the multi-objective optimization algorithm, and determining the performance index of the second value combination when conducting the test.

[0079] When performance metrics fail to meet preset performance conditions, it indicates that the first combination of values ​​obtained through the multi-objective optimization algorithm fails to meet the requirements, suggesting that the current combination of values ​​is insufficient in balancing multiple objectives such as throughput and latency of the inference model. The multi-objective optimization algorithm possesses powerful global search and intelligent adjustment capabilities. Reusing this algorithm, based on performance information from previous tests, can more accurately explore new directions for value combinations and generate a second combination of values. Subsequent performance testing to obtain new metrics not only visually demonstrates the optimization effect of the second combination compared to the first, but also provides crucial evidence for further adjustments to the optimization strategy. Through continuous iteration of this process, the ideal combination of values ​​that meets the preset performance conditions is gradually approximated, ensuring the efficient and stable operation of the inference model in practical applications.

[0080] As a feasible approach, Pareto front analysis can be performed on the first combination of values, and the second combination of values ​​can be determined based on the analysis results and a multi-objective optimization algorithm.

[0081] The Pareto front is the set of all Pareto optimal solutions in a multi-objective optimization problem. A Pareto optimal solution is one that cannot further improve the performance of any objective without sacrificing the performance of other objectives. Taking maximizing throughput and minimizing latency as an example, solutions on the Pareto front represent the best balance that the inference model can achieve on these two objectives under different combinations of values. Pareto front analysis compares the first combination of values ​​with solutions on the Pareto front. If the first combination of values ​​is on the Pareto front, it is a Pareto optimal solution, meaning that the performance of any objective cannot be further improved without sacrificing the performance of other objectives. If the first combination of values ​​is not on the Pareto front, we can analyze the difference between it and solutions on the Pareto front to identify areas for improvement, providing direction for subsequent multi-objective optimization algorithms. Then, the adjusted multi-objective optimization algorithm continues the search in the parameter subspace to generate a second combination of values. During the search, the search can be more targeted towards the Pareto front to improve the performance of the second combination of values.

[0082] In some embodiments, due to the large number of configurable deployment parameters, comprehensive optimization would not only consume enormous computational resources but also struggle to efficiently locate the optimal solution due to an overly broad search space. Therefore, when selecting target configuration parameters for optimization, the compatibility between the deployment parameters and hardware characteristics should be considered. Different hardware architectures (such as CPUs, GPUs, and TPUs) exhibit significant differences in computing power, memory bandwidth, and caching mechanisms. Only by selecting parameters that match the hardware characteristics can the hardware's performance advantages be fully utilized. On the other hand, the correlation between deployment configuration parameters and model characteristics cannot be ignored. The model's structural complexity, parameter size, and computational graph characteristics all significantly impact deployment effectiveness. For models with different characteristics, parameters closely related to them should be selected for optimization to ensure the model can run efficiently in the deployment environment.

[0083] As a feasible implementation method, S102 can be specifically implemented as follows: S1021. For each deployment configuration parameter, determine the compatibility between the deployment configuration parameter and the hardware characteristics, as well as the correlation with the model characteristics.

[0084] For each deployment configuration parameter, actual tests or simulations can be conducted to observe the utilization of hardware resources under different hardware parameter settings. For example, computing resource allocation parameters can be tested to assess CPU and GPU utilization and load balancing under different allocation ratios; data transfer parameters focus on memory bandwidth usage and data transfer latency. Based on these test results, parameter settings that enable efficient use of hardware resources and avoid performance bottlenecks can be selected, thereby determining the compatibility between parameters and hardware characteristics.

[0085] For each deployment configuration parameter, analyze its impact on different features of the model. For example, the model parallelism strategy parameter can be used to study its effect on improving the computational efficiency of models with complex inter-layer connections, as well as its changes in memory usage and communication overhead for models with a large number of parameters. The quantization accuracy parameter needs to be examined for its impact on model accuracy and inference speed, as different model structures have different sensitivities to quantization accuracy. Through extensive experiments comparing the model's performance under different parameter settings, the correlation between parameters and model features is clarified, thereby determining appropriate parameter values.

[0086] S1022. Based on the adaptability of each deployment configuration parameter to hardware features and its correlation with model features, select the target configuration parameter to be optimized from multiple deployment configuration parameters in the deployment configuration parameter space.

[0087] When selecting parameters to optimize from the deployment configuration parameter space, it is necessary to comprehensively consider the adaptability of each parameter to hardware features and its correlation with model features. If a deployment configuration parameter has a high adaptability to hardware computing power, adjusting it can significantly improve hardware resource utilization, such as optimizing batch size to improve GPU computing core utilization; at the same time, if the deployment configuration parameter has a strong correlation with model features and has a significant impact on model accuracy, complexity, or convergence speed, such as adjusting model structure parameters to change the number of model parameters and computational load, thereby affecting model performance, then such deployment configuration parameters should be given priority as optimization targets. This ensures that the optimization work can accurately target key factors and improve model deployment effectiveness and resource utilization efficiency.

[0088] As a feasible implementation method, it also includes: S1023, determining the target value range corresponding to each target configuration parameter based on hardware characteristics and model characteristics, and obtaining the parameter subspace to be optimized. This facilitates the determination of the value combination of the target configuration parameters in the parameter subspace to be optimized in S103.

[0089] After selecting the target configuration parameters, it is necessary to comprehensively consider the interaction between hardware characteristics and model characteristics, and further adjust and optimize the value range of each parameter. The number of computing cores and clock speed of the hardware determine its computing speed and parallel processing capability. If the deployment configuration parameter values ​​are too low, the computing potential of the hardware cannot be fully utilized. Hardware memory (including video memory and system memory) is the space for storing model parameters and intermediate calculation results. If the parameter values ​​are too small, the model may not be able to load completely or may frequently exchange data during operation, reducing efficiency. For example, GPUs have a large number of computing cores suitable for parallel computing. If the batch size parameter is set too small, the GPU computing cores cannot work in parallel sufficiently, resulting in idle resources. Therefore, based on the hardware characteristics of the computing cluster, the minimum parameter value that allows the hardware to operate efficiently can be determined as a lower limit.

[0090] Model accuracy is a key performance indicator. Overly large deployment configuration parameters can disrupt the model structure and parameter relationships, leading to decreased accuracy. Model complexity (number of parameters, computational load, etc.) must be commensurate with hardware processing capabilities. Excessive parameter values ​​can cause model complexity to exceed hardware limits, resulting in slow or unusable operation. Therefore, parameter upper limits can be determined based on model characteristics such as accuracy requirements, model complexity, and hardware communication capabilities.

[0091] Hardware characteristics and model characteristics influence each other, and a comprehensive consideration is needed when determining the range of parameter values. For example, the batch size parameter must take into account both hardware computing power and memory capacity, as well as the model's data requirements and training convergence. By experimenting and analyzing indicators such as hardware resource utilization and model performance under different value combinations, the optimal balance between hardware and model can be found, thereby determining the reasonable value range of each deployment configuration parameter to be optimized, and obtaining the parameter subspace to be optimized.

[0092] The solution provided in this embodiment selects the target configuration parameters to be optimized based on model features and hardware features, and determines the target value range corresponding to each target configuration parameter. This not only adapts the deployment configuration parameters to the hardware capabilities based on the hardware's computing and memory characteristics, fully tapping the hardware's potential, improving resource utilization efficiency, and avoiding resource waste or overload, but also combines model features such as model accuracy and complexity to ensure model performance and quality, preventing model accuracy degradation or convergence difficulties due to inappropriate parameters. At the same time, a reasonable value range narrows the subsequent optimization search space, reduces optimization difficulty, improves optimization efficiency, and makes model deployment more adaptable and flexible, better meeting the needs of different scenarios.

[0093] In some embodiments, performance metrics are a series of standards used to measure the performance of combinations of values ​​in a specific aspect. While these metrics can reflect a certain dimension of performance, they are often difficult to compare and score objectively. Therefore, by establishing a reasonable scoring mechanism, multiple performance metrics can be transformed into a unified performance score, thereby enabling an intuitive evaluation of the overall performance of the system or model under different combinations of values, and thus determining whether it meets the preset performance conditions.

[0094] As one feasible approach, please refer to Figure 2 The method provided in this application embodiment further includes: S201. Determine the performance score corresponding to the combination of values ​​based on the performance indicators.

[0095] Among them, the performance score is positively correlated with throughput and negatively correlated with latency.

[0096] Throughput reflects the number of tasks a system processes per unit of time. Higher throughput means the system can complete more work in the same amount of time, resulting in higher efficiency. Therefore, performance scores are positively correlated with throughput, increasing as throughput rises, highlighting the positive contribution of the system's efficient task processing capabilities to overall performance. Latency, on the other hand, represents the time required for the system to complete a task from receiving it. Lower latency indicates a faster response and more timely processing. Performance scores are negatively correlated with latency, meaning lower latency leads to higher scores, emphasizing the importance of rapid system response for improving performance. This configuration comprehensively considers both the number and speed of tasks processed by the system, providing a comprehensive and accurate evaluation of system performance under different value combinations.

[0097] In some embodiments, excessive latency is often caused by insufficient system resources or unreasonable resource allocation. Therefore, when determining performance scores, additional penalties can be imposed for timeouts to identify resource bottlenecks in the system, such as insufficient CPU, memory, network bandwidth, etc.

[0098] As a feasible implementation method, S201 includes: S2011. If the latency of the inference model exceeds the preset duration during testing, a timeout score is determined based on the latency.

[0099] The preset duration is pre-set and can be determined according to needs in actual applications; this application embodiment does not impose any restrictions on it. As one implementation method, it can be based on the preset duration specified in a Service-Level Agreement (SLA). An SLA is a formal agreement signed between a service provider and a user, which clarifies the service provider's commitment and standards to provide services to the user at a specific quality level within a specific timeframe. This agreement aims to ensure the quality and reliability of the service while providing users with a basis for measuring the service provider's performance.

[0100] If the latency exceeds the preset duration, it indicates that the latency is too high. This may be due to reasons such as high model complexity, insufficient system resources, or unreasonable resource allocation. To avoid these problems, a timeout score can be determined, and a penalty mechanism can be used to force subsequent optimization processes to optimize in the direction of reducing latency.

[0101] One implementation method is to determine the timeout score using the following formula: Violation penalty = N * (Delay - Preset duration). Where N is the preset weight.

[0102] Multiply the portion of the delay exceeding the preset duration by a coefficient N, and directly output a comparable numerical result. Example: If the preset duration is 100ms, the actual delay is 150ms, N=2, then the penalty for the violation = 2 × (150 - 100) = 100 points. A higher score indicates a more severe timeout issue.

[0103] S2012. Determine the throughput ratio of the throughput to the target throughput, and the delay ratio of the target delay to the delay.

[0104] S2013. Determine the performance score corresponding to the combination of values ​​based on the throughput ratio, latency ratio, and timeout score.

[0105] Among them, the timeout score is negatively correlated with the performance score.

[0106] As one implementation method, the performance score can be determined using the following formula:

[0107] in, This represents the performance score corresponding to the combination of values; , These are the weights assigned to throughput and latency, respectively. Assigning weights to throughput and latency quantifies the importance of different metrics, allowing for adjustments to the weights based on specific scenario requirements to balance latency and throughput. For example, as one implementation approach, the overall processing capacity of the system (throughput) is more important than the response speed of a single request (latency); therefore, the weight corresponding to throughput can be higher than the weight corresponding to latency. It is 0.7; It is 0.3.

[0108] S202. Determine whether the performance indicators meet the preset performance conditions based on the performance score.

[0109] After obtaining the performance score, it can be compared with a pre-set threshold or target value to objectively measure whether the system performance meets the standards. If the performance score meets or exceeds the preset conditions, it indicates that the system can meet the performance requirements under the current configuration; otherwise, it indicates that there is a bottleneck or deficiency, and further optimization of parameters or adjustment of resource allocation is needed. This approach not only simplifies the complex performance analysis process but also provides a reliable basis for automated tuning and decision-making.

[0110] As a feasible implementation method, S202 includes: if the improvement of the performance score of the current value combination compared with the performance score of the previous value combination is less than or equal to a preset improvement, determining that the performance index of the current value combination meets the preset performance condition.

[0111] When the performance score of the current value combination does not exceed a preset improvement threshold compared to the previous round, it indicates that the optimization effect has reached saturation. Continuing to adjust the value combination through multi-objective optimization algorithms may not yield significant performance gains. At this point, determining that the performance metric meets the preset condition ensures that the current value combination guarantees a relatively stable and efficient inference module, while preventing the pursuit of minor improvements from leading to local optima or unnecessary computational costs. This balances the optimization effect and avoids unnecessary waste of resources.

[0112] In some embodiments, the inference engine plays a crucial role in deploying a model to a computing cluster, and the model's efficient operation is highly dependent on it. Therefore, when deploying an inference model to a computing cluster, it is also necessary to determine the inference engine to be used.

[0113] As one implementation, the configuration parameter determination method provided in this application embodiment further includes: determining an appropriate inference engine based on hardware features and model features; and determining a deployment configuration parameter space based on hardware features, model features, and the inference engine. S102 can be implemented as: determining the target configuration parameters required to deploy the inference model to the computing cluster through the inference engine in the deployment configuration parameter space based on hardware features and model features.

[0114] The inference engine is a key component for deploying models to computing clusters and achieving efficient inference. It is responsible for reading the model structure, loading model parameters, and performing inference computations using the computing cluster's hardware resources. By selecting an appropriate inference engine based on hardware and model characteristics, the deployment configuration parameter space can be determined more accurately, leading to a more precise selection of target configuration parameters and their corresponding target value ranges.

[0115] Different hardware possesses different computing architectures and performance characteristics. Inference engines need to be optimized for different hardware to fully leverage its performance advantages. Models also exhibit diverse characteristics, including model structure (such as convolutional neural networks, recurrent neural networks, Transformers, etc.), model size, and computational complexity. Different inference engines offer varying degrees of support and optimization effectiveness for different types of models. Once a suitable inference engine is selected based on hardware and model characteristics, the deployment configuration parameter space can be determined based on the engine's functionality and features, combined with the specific requirements of the hardware and model. This parameter space includes all configuration parameters that may affect model inference performance, such as batch size, quantization precision, and memory allocation strategy.

[0116] In this embodiment, the most suitable inference engine is first matched based on hardware capabilities and model structure characteristics. Then, the appropriate configuration parameter space is determined based on the "hardware-model-engine" ternary feature. This concentrates optimization resources on high-performance potential areas, avoiding blind searching across the entire parameter space and ensuring deep adaptation between the configuration scheme and the underlying hardware. This improves deployment efficiency while guaranteeing the stability of the inference service and the high efficiency of resource utilization.

[0117] As an implementation approach, commonly used inference engines include: vllm, sglang, TensorRT, and MindIE. Among them, vllm focuses on efficient inference for large language models. When handling long text generation tasks, its unique pagination attention mechanism and other optimization techniques significantly reduce GPU memory usage and improve inference throughput, making it particularly suitable for scenarios with high requirements for long text interaction. sglang also focuses on the large language model domain, performing excellently in low-latency inference. By optimizing memory access patterns and computation scheduling strategies, it can quickly provide inference results while ensuring generation quality. TensorRT fully utilizes the parallel computing capabilities of GPUs, supports the conversion and optimization of various model formats, and can significantly improve inference speed in computer vision tasks such as image recognition and object detection, as well as some natural language processing tasks. MindIE has efficient model compression and acceleration capabilities, enabling efficient model deployment and inference in key areas such as smart cities and intelligent transportation.

[0118] In some embodiments, due to differences in design philosophy, optimization strategies, and adapted hardware environments among different inference engines, it is necessary to configure matching parameters for each inference engine individually during actual deployment to ensure that the model can achieve high inference performance on the computing cluster. Because of the diversity of inference engines, to facilitate the determination of the value combination template for each inference engine deployed in the computing cluster, the above-described configuration parameter determination method can be executed separately for the model features of each inference model to obtain the value combination template corresponding to each inference model. However, when there are many types of inference engines, executing the above method separately is inefficient and prone to errors. In this case, a dedicated scanner can be developed, utilizing its automation and batch processing capabilities to implement steps S102-S103 of the above configuration parameter determination method, thereby improving overall work efficiency and accuracy.

[0119] As an implementation, when the inference engine is TensorRT-LLM, the TensorRT-LLM scanner can be used to determine the corresponding value combination template.

[0120] For example, please refer to Figure 3The TensorRT-LLM scanner can include: a parameter space generator, a compilation optimization module, a quantization calibrator, batch size, key-value (KV) partitioning strategy, and quantization modes. The parameter space generator generates a series of possible configuration combinations based on the input hardware and model characteristics. These combinations cover multiple aspects such as compilation optimization, batch size, KV partitioning strategy, and quantization modes, providing a range of parameter choices for subsequent optimization processes and ensuring the exploration of optimal configuration combinations. The compilation optimization module adjusts compilation optimization options, such as kernel fusion level, memory copy optimization, code generation strategy, and instruction set optimization, to improve the model's execution efficiency on specific hardware. The quantization calibrator performs quantization on the model, reducing memory usage and computation while maintaining model accuracy as much as possible. The batch size determines the number of input samples processed simultaneously in a single inference operation. The KV partitioning strategy can be used to partition the key-value (KV) cache in large language models to optimize memory access patterns and reduce memory usage. Quantization mode is used to select the quantization method of the model, such as the FP8 scaling factor for FP8 quantization, the INT4 group size for INT4 quantization, etc. Different quantization modes have different trade-offs between accuracy and performance.

[0121] As one implementation method, when the inference engine is TensorRT-LLM, such as Figure 3 As shown, the input to the scanner is model features and hardware features. The combination of values ​​output by the scanner needs to be processed by the performance testing cluster and the result analysis engine before being output as a parameter template.

[0122] As one implementation method, please refer to Table 1. The deployment configuration parameter space corresponding to the TensorRT-LLM inference engine can include the following: Table 1

[0123] As an implementation, when the inference engine is determined to be SGLang, the SGLang scanner can be used to determine the corresponding value combination template.

[0124] For example, please refer to Figure 4 The SGLang scanner can include: Radix tree builder, structured compiler, batch optimizer, Radix block size, tree depth, JSON parsing mode, multi-output configuration, structured batch processing, and load balancing strategy.

[0125] The Radix tree builder is responsible for constructing Radix trees (prefix trees or trie trees). A Radix tree is an efficient data structure used to store and retrieve collections of strings. In the SGLang scanner, it can construct Radix trees from the input model features according to specific rules, enabling subsequent operations such as pattern matching and keyword searching, thus improving scanning efficiency.

[0126] A structured compiler is used to compile the scan rules or patterns defined in SGLang into a structured format. This translates the scan rule descriptions into human-readable representations. Figure 4 The input prompts are converted into an internal representation that the computer can efficiently execute. Through structured compilation, complex scanning logic can be optimized and organized, enabling the scanner to perform accurate and rapid scanning and analysis according to the compiled structured rules when processing input data.

[0127] A batch optimizer is used to optimize the batch processing of input data. During scanning, a large amount of data typically needs to be processed. The batch optimizer can group and batch this data appropriately to improve processing efficiency. It can determine the optimal batch size and processing order based on the characteristics of the data and the needs of the scan, reducing data processing overhead, such as reducing memory accesses and lowering I / O operation frequency.

[0128] The Radix block size defines the size of the data blocks used when processing Radix tree-related operations. Dividing the tree's data into appropriately sized blocks when building, traversing, or searching the Radix tree improves data locality and cache utilization. An appropriate Radix block size can be adjusted based on the system's memory architecture, cache size, and data characteristics to optimize scanner performance.

[0129] Tree depth is used to limit the depth of the Radix tree. Setting the tree depth during Radix tree construction controls the tree's growth range, preventing it from becoming too large and consuming excessive memory resources. At the same time, a reasonable tree depth setting can also affect scanning efficiency and accuracy. An excessively deep tree may complicate and time-consuming the scanning process, while a tree that is too shallow may fail to accurately represent all scanning modes.

[0130] JSON parsing modes specify how the scanner parses JSON data. The SGLang scanner may need to process content containing JSON data; the JSON parsing mode defines how to convert the JSON string into an internal data structure for subsequent scanning and analysis. Different parsing modes can handle JSON data of varying complexity and structure, such as supporting the parsing of nested objects and array processing.

[0131] The multi-output configuration is used to configure the scanner's output method, enabling it to simultaneously output scan results to multiple different targets. For example, scan results can be output to the console, log files, databases, or other applications simultaneously. The multi-output configuration provides a flexible result distribution mechanism, allowing users to further process and analyze scan results according to different needs.

[0132] Structured batch processing combines structured information with batch processing techniques to process input data more efficiently. It can batch process data with similar structures based on their structural characteristics, further improving processing efficiency. For example, when scanning code files, code blocks with the same syntactic structure can be analyzed in batches, reducing repetitive processing steps.

[0133] Load balancing strategies are used to rationally distribute scanning tasks in a multi-threaded or multi-node environment to achieve load balancing. When the scanner needs to process a large amount of data or complex tasks, load balancing strategies can distribute tasks evenly among threads or nodes based on their processing capabilities, avoiding situations where some threads or nodes are overloaded while others are idle, thereby improving the throughput and performance of the entire scanning system.

[0134] As one implementation method, such as Figure 4 As shown, testing the value combinations generated by the SGLang scanner requires both streaming response testing and prefix multiplexing analysis to ultimately obtain the value combination template. Streaming response testing simulates a scenario where data is continuously input into the scanner in a streaming manner during real-world use. Under this test, the scanner needs to process the input data and provide a response in real time, rather than waiting for all data to be input before processing. Streaming response testing primarily focuses on the scanner's overall performance when processing data in real time, while prefix multiplexing analysis focuses on the scanner's efficiency when processing data with specific characteristics (same prefix). Combining the two allows for a comprehensive evaluation of the scanner's performance in different scenarios, avoiding the neglect of other important factors due to focusing only on one aspect.

[0135] As one implementation method, please refer to Table 2. The deployment configuration parameter space corresponding to the SGLang inference engine can include the following: Table 2

[0136] As an implementation, when the inference engine is vLLM, the vLLM scanner can be used to determine the corresponding value combination template.

[0137] For example, please refer to Figure 5 The vLLM scanner can include modules such as a PagedAttention optimizer, a dynamic batch controller, block size, paging strategy, concurrent stress testing, batch size, maximum concurrency, request length distribution, and warm-up strategy. The PagedAttention optimizer manages attention mechanism-related data through paging, optimizes memory access patterns, and improves attention computation efficiency to accelerate model inference. The dynamic batch controller dynamically adjusts the batch size and combination based on real-time request conditions, balancing resource utilization and response latency. Block size defines the basic unit size for data processing or storage, affecting memory allocation, cache utilization, and computational parallelism. Concurrency stress testing simulates scenarios where multiple requests arrive simultaneously, testing the system's performance and stability under high concurrency. Batch size specifies the number of requests processed in each group, significantly impacting computational efficiency and resource usage. Maximum concurrency sets the maximum number of requests the system can handle simultaneously, preventing resource overload and performance degradation. Request length distribution statistically analyzes request length characteristics, providing data for optimizing resource allocation and scheduling strategies. Warm-up strategies are used to bring the system to a stable operating state in a specific way before formally processing requests, thereby reducing performance fluctuations in the initial stage.

[0138] As an implementation, when the inference engine is vLLM, such as Figure 5 As shown, hardware features need to be processed by a memory analyzer before being input into the dynamic batch processing controller. The memory analyzer is responsible for in-depth analysis and processing of the hardware features. It evaluates the hardware's performance under different workloads based on its specific parameters. The dynamic batch processing controller receives the hardware feature information processed by the memory analyzer and formulates a dynamic batch processing strategy accordingly. It adjusts the batch size in real time based on the current remaining hardware resources, the urgency of the inference task, and the characteristics of the model.

[0139] As an implementation, when the inference engine is vLLM, such as Figure 5As shown, when testing the generated value combinations, the relevant environment of the value combinations needs to be monitored first through the fragmentation monitoring module, and the processed value combination template is then input into the parameter recommendation engine. In scenarios where vLLM is used as the inference engine, testing the generated value combinations requires a rigorous and scientific method. Directly using unoptimized value combinations may not fully leverage the performance advantages of vLLM, and may even lead to resource waste or low inference efficiency. By monitoring the relevant environment of the value combinations through the fragmentation monitoring module before inputting the processed value combination template into the parameter recommendation engine, parameters can be precisely adjusted and optimized according to the actual situation, thereby improving the accuracy and effectiveness of the test and ensuring that the final value combinations are better adapted to the vLLM inference engine.

[0140] As one implementation method, please refer to Table 3. The deployment configuration parameter space corresponding to the vLLM inference engine can include the following: Table 3

[0141] As an implementation, when the inference engine is MindIE, the MindIE scanner can be used to determine the corresponding value combination template.

[0142] For example, please refer to Figure 6The MindIE scanner can include modules such as the AscendGraph compiler, tensor parallel controller, dynamic quantizer, graph optimization levels, dynamic shape recompilation, ATP strategy, communication topology, INT8 calibration cycle, and FP8 activation threshold. The AscendGraph compiler converts high-level neural network model descriptions into low-level computation graphs that can be efficiently executed by the chip, fully leveraging the hardware's parallel computing potential to improve model inference performance. The tensor parallel controller is responsible for rationally distributing tensor computation tasks across various computing nodes. The dynamic quantizer dynamically adjusts quantization parameters based on input data characteristics during model inference, further compressing model data size and improving inference speed while maintaining model accuracy. The graph optimization levels set the degree of optimization for the computation graph; different levels employ different optimization strategies, such as operator fusion and constant folding, to balance optimization effectiveness and time cost. Dynamic shape recompilation automatically recompiles the computation graph to adapt to new data shapes when the shape of the input data changes, ensuring the model can correctly handle input data of different dimensions. The ATP strategy, or Automatic Tuning Strategy, uses an intelligent search algorithm to automatically find the optimal configuration from numerous value combinations to maximize the model's inference performance on specific hardware. Communication topology defines the connection methods and data transmission paths for data communication between multiple devices or nodes. Optimizing the communication topology can reduce communication latency and improve the efficiency of distributed inference. The INT8 calibration cycle determines the interval between calibration operations when using INT8 quantization for model inference. A suitable calibration cycle ensures quantization accuracy and avoids accuracy degradation due to prolonged periods without calibration. The FP8 activation threshold sets the threshold value of the activation function when using FP8, controlling the numerical range during quantization and significantly impacting model accuracy and inference performance.

[0143] As an implementation, when the inference engine is MindIE, such as Figure 6As shown, during testing, a customized template can be obtained through a distributed test cluster and linear acceleration analysis. The distributed test cluster consists of multiple computing nodes, simulating a large-scale parallel computing environment. In the testing of the MindIE inference engine, this environment can more realistically reflect performance under large-scale data and high-concurrency requests in actual production. Linear acceleration analysis helps identify performance bottlenecks in the system. For example, if the improvement in inference performance gradually decreases as the number of nodes increases, it may be due to communication latency between nodes becoming a bottleneck. By analyzing the linear speedup ratio, developers can optimize the MindIE inference engine in a targeted manner, such as optimizing communication protocols and improving task scheduling algorithms, to improve system scalability and performance. Combining the performance data collected from the distributed test cluster and the results of linear acceleration analysis, data mining and machine learning algorithms are used to evaluate and filter various value combinations. These parameters include, but are not limited to, batch size, number of threads, and memory allocation strategy. By finding the value combinations that enable the MindIE inference engine to achieve optimal performance in different test scenarios, customized templates are generated.

[0144] As one implementation method, please refer to Table 4. The deployment configuration parameter space corresponding to the MindIE inference engine can include the following: Table 4

[0145] In an exemplary embodiment, this application also provides another method for deploying an inference model. The entity executing this deployment method can be the aforementioned computing device, or other computing devices (such as configuration devices in a computing cluster); this application embodiment does not impose any limitations on this. Please refer to... Figure 7 The deployment method includes the following steps: S701. In response to the inference model deployment request, obtain the target inference model indicated in the inference model deployment request.

[0146] When a company plans to launch new AI-based businesses, or as its business expands, or when the model development team optimizes and improves the existing inference model to enhance its performance, accuracy, or functionality, it needs to send an inference model deployment request to deploy the new model to the production environment so that users can use it.

[0147] As a feasible implementation method, users can initiate inference model deployment requests through an AI cluster software management platform. When deploying inference models on a computing cluster, the models are typically large in size. A computing cluster can be viewed as an AI cluster composed of multiple computing nodes. The platform, acting as an interaction bridge between the user and the AI ​​cluster, provides an entry point for initiating inference model deployment requests. This allows users to efficiently interact with cluster resources, manage the deployment of inference models, and ensure the smooth operation of AI services. When users need to deploy inference models, they don't need complex underlying operations; they can easily initiate deployment requests by following the instructions through the intuitive and easy-to-use management platform interface, lowering the technical threshold and improving the user experience.

[0148] After receiving a deployment request for an inference model, the computing device needs to accurately extract relevant information about the specified target inference model from the request, laying the foundation for a series of deployment operations based on the target inference model.

[0149] S702. Based on the mapping relationship between the computing cluster, the inference model and the value combination, determine the target value combination corresponding to the current computing cluster and the target inference model.

[0150] The combination of values ​​in the mapping relationship is used to set the configuration parameters required for deploying the inference model on the computing cluster via the inference engine. It should be understood that the method for determining this combination of values ​​can refer to the above embodiments, and will not be repeated here.

[0151] As one implementation approach, developers can pre-execute the above method for multiple inference models and multiple computing clusters to obtain multiple value combination templates, and store them in a template library. In this way, S702 can be implemented as follows: based on the target hardware characteristics of the current computing cluster and the target model characteristics of the target inference model, it searches the mapping relationship between computing clusters, inference models and value combinations stored in the template library to obtain the target value combination corresponding to the current computing cluster and the target inference model.

[0152] S703. Deploy the target inference model on the current computing cluster through the inference engine based on the target value combination.

[0153] The target value combination encompasses various configuration parameters required for model deployment, such as computational precision, memory allocation, and parallelization strategies, providing a precise configuration guide for deploying the target inference model. On the current computing cluster, the inference engine first reads the parameter settings from the target value combination, such as computational precision, batch size, and memory allocation strategy. Next, the inference engine adapts the target inference model based on these parameters, converting it into a format that can run efficiently in the current computing cluster's hardware and software environment. Then, the inference engine utilizes the system's computing resources to load the model onto appropriate computing nodes according to the parameter requirements, completing the model deployment and ensuring that the target inference model can respond stably and quickly to inference requests.

[0154] As can be seen, the deployment method of the inference model provided in this application eliminates the need for manual enumeration of the four-dimensional parameter matrix consisting of the inference engine, concurrency, input length, and computational precision when configuring the target inference model in the current computing cluster. It also eliminates the need to spend weeks manually testing to find local optima. Configuration can be completed quickly using pre-tested combinations of target values. Furthermore, since the template is pre-tested for the current hardware environment and model characteristics, it ensures precise matching of configuration parameters, thereby guaranteeing the performance of the deployed target inference model.

[0155] In some embodiments, please refer to Figure 8 Administrators can submit hardware information to the hardware parsing module to determine the hardware characteristics of the computing cluster; they can also submit model information to the model parsing module to obtain the model's feature information. Then, the framework decision tree determines the inference engine corresponding to the model features and hardware features. This information is then sent to the adaptive parameter scanning module, which selects the desired scanner from multiple scanners (as shown in the figure: VLLM scanner, SGlang scanner, TensorRT scanner, and MindIE scanner). It should be understood that... Figure 8 These scanners are merely illustrative examples. In actual applications, more or fewer scanners may be used, and this application does not limit this.

[0156] The scanner can receive model features and hardware features as input, determine the parameter space to be scanned based on the hardware and model features, and then perform automated tests in the parameter space. After collecting performance indicators, the test results are passed to the parameter optimization engine, so that the parameter optimization engine can evaluate the performance of the value combination and determine the final value combination template, which is then stored in the parameter template library.

[0157] In some embodiments, please continue reading Figure 8When a user needs to deploy an inference model on a computing cluster, they can trigger a deployment request on the AI ​​cluster software management platform. The platform then calls AI scheduling software (such as Slurm or K8s) to perform cluster scheduling. Afterward, it uses a template matcher to accurately find a matching target value combination template in the parameter template library and optimizes the deployment of the inference service based on this.

[0158] In an exemplary embodiment, this application also provides a first deployment apparatus for an inference model. This deployment apparatus may be a computing device that performs the aforementioned deployment method for the inference model, or it may be a processor within the computing device. The deployment apparatus may include one or more functional modules for implementing the deployment method of the inference model in the above method embodiments.

[0159] In an exemplary embodiment, this application also provides a second deployment apparatus for an inference model. The configuration apparatus for the inference model can be a computing device that performs the aforementioned deployment method for the inference model, or it can be a processor within the computing device. The deployment apparatus for the inference model can include one or more functional modules for implementing the deployment method of the inference model in the above method embodiments.

[0160] This application also provides a computing device. Figure 9 This is a schematic diagram of the architecture of a computing device provided in an embodiment of this application. Figure 9 As shown, the computing device 100 includes: one or more memories 120, one or more processors 110, a communication bus 140, and a communication interface 130. The processors 110 and memories 120 are connected via the communication bus 140; the one or more memories 120 are used to store computer program code, which includes computer instructions; when the one or more processors 110 execute the computer instructions, the computing device 100 executes the deployment method of the inference model provided in this application embodiment.

[0161] Optionally, the memory 120 may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. The embodiments of this application do not impose any limitations on this.

[0162] The processor 110 may be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof, and the embodiments of this application do not impose any limitations on this.

[0163] The communication bus 140 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This communication bus 140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 It is represented by a single thick line, but this does not mean that there is only one bus or one type of communication bus.

[0164] Communication interface 130 uses any transceiver-like device for communicating with other devices or communication networks, such as control systems, radio access networks (RAN), wireless local area networks (WLAN), etc.

[0165] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0166] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; for example, the related hardware can be a processor of a computing device. The program instructions can be stored in the above-described computer-readable storage medium, and when executed, they can implement the processes of the above method embodiments. The computer-readable storage medium can be memory. The above-described computer-readable storage medium can also be an external storage device, such as a hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Further, the above-described computer-readable storage medium can include both memory and external storage devices. The above-described computer-readable storage medium is used to store the above-described computer program instructions and other programs and data required for the deployment method of the above-described inference model.

[0167] This application also provides a computer program product comprising a computer program that, when run on a computing device, causes the computing device to execute any of the inference model deployment methods provided in the above embodiments.

[0168] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0169] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0170] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for deploying an inference model, the method comprising: The method comprises: obtaining hardware features of a computing cluster and model features of an inference model; wherein the hardware features are used to represent hardware configurations and / or computing capabilities of the computing cluster; the hardware configurations are used to represent at least one of processing capability configurations, data transmission configurations or memory capacity configurations of the computing cluster; and the model features are used to represent computing complexity and / or resource requirements of the inference model; based on the hardware features and the model features, determining target configuration parameters required when deploying the inference model on the computing cluster; testing combinations of values of the target configuration parameters, and determining performance indicators of each value combination when being tested; the performance indicators include throughput and / or latency of the inference model when being tested; in a case where the performance indicators meet preset performance conditions, deploying the inference model on the computing cluster based on a value combination corresponding to the performance indicators.

2. The method of claim 1, wherein, The testing of the combinations of values of the target configuration parameters, and the determination of the performance indicators of each value combination when being tested, comprise: performing combination search within a value range of the target configuration parameters based on a multi-objective optimization algorithm to obtain a first value combination; testing the inference model based on the first value combination, and obtaining the throughput and latency of the inference model when being tested. The method further comprises: in a case where the performance indicators do not meet the preset performance conditions, re-determining a second value combination based on the multi-objective optimization algorithm, and determining performance indicators of the second value combination when being tested.

3. The method of claim 2, wherein, The method further comprises: performing parameter importance analysis on performance indicators of a plurality of the value combinations to determine an influence degree of each target configuration parameter on the performance indicators; The combination search within the value range of the target configuration parameters based on the multi-objective optimization algorithm comprises: performing combination search within the value range of the target configuration parameters based on the multi-objective optimization algorithm according to the influence degree of the target configuration parameters to obtain the first value combination.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: determining a performance score corresponding to the value combination based on the performance indicators; the performance score is positively correlated with the throughput and negatively correlated with the latency; determining whether the performance indicators meet the preset performance conditions based on the performance score.

5. The method of claim 4, wherein, The determination of whether the performance indicators meet the preset performance conditions based on the performance score comprises: in a case where an improvement amount of the performance score of a current value combination compared with a performance score of a previous value combination is less than or equal to a preset improvement amount, determining that the performance indicators of the current value combination meet the preset performance conditions.

6. The method according to claim 4 or 5, characterized in that, The determination of the performance score corresponding to the value combination based on the performance indicators comprises: in a case where the latency of the inference model when being tested exceeds a preset time length, determining an overtime score based on the latency; determining a throughput ratio of the throughput to a target throughput, and a latency ratio of a target latency to the latency; and Determine a performance score corresponding to the value combination based on the throughput ratio, the latency ratio, and the timeout score; the timeout score is negatively correlated with the performance score.

7. The method according to any one of claims 1 to 6, characterized in that, Determine target configuration parameters required when deploying the inference model on the computing cluster based on the hardware features and the model features, including: For each deployment configuration parameter, determine the adaptability of the deployment configuration parameter to the hardware features and the relevance of the deployment configuration parameter to the model features; Select target configuration parameters to be optimized from a plurality of deployment configuration parameters in a deployment configuration parameter space based on the adaptability of each deployment configuration parameter to the hardware features and the relevance of each deployment configuration parameter to the model features.

8. The method according to any one of claims 1 to 7, characterized in that, Deploy the inference model on the computing cluster based on the value combination corresponding to the performance indicator, including: Establish a mapping relationship between the value combination corresponding to the performance indicator and the computing cluster and the inference model based on the value combination corresponding to the performance indicator, and the mapping relationship is used for deploying the inference model on the computing cluster.

9. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Determine an adapted inference engine according to the hardware features and the model features; Determine a deployment configuration parameter space based on the hardware features, the model features, and the inference engine; Determine target configuration parameters required when deploying the inference model on the computing cluster based on the hardware features and the model features, including: Determine target configuration parameters required when deploying the inference model on the computing cluster by the inference engine in the deployment configuration parameter space based on the hardware features and the model features.

10. A computing device, comprising: The computing device includes a processor and a memory; the processor is coupled with the memory; The memory is used to store computer instructions; The computer instructions are loaded and executed by the processor to enable the computing device to implement the method of any one of claims 1-9.

Images