Reasoning resource optimization method, device and electronic equipment

Abstract

The present application provides a method, device and electronic device for optimizing inference resources. The method includes: obtaining inference service response performance data of a neural network model under different hardware resources, as well as hardware cost data under different hardware resources; optimizing and modeling the inference service response performance data and hardware cost data to obtain a hardware resource deployment plan with the lowest hardware cost; generating a configuration file of the neural network model according to the hardware resource deployment plan with the lowest hardware cost; and adaptively scheduling and optimizing the neural network model and different hardware resources according to the configuration file. In the implementation process of the above-mentioned scheme, by optimizing and modeling the inference service response performance data and hardware cost data, and according to the configuration file generated by the hardware resource deployment plan with the lowest hardware cost, the neural network model and different hardware resources are adaptively scheduled and optimized, thereby effectively improving the utilization rate of hardware resources.

Description

Technical Field

[0001] The present application relates to the technical field of large-model reasoning, and more specifically, to a method, device, and electronic device for optimizing reasoning resources. Background Art

[0002] Currently, neural network models are often used for online service reasoning. For example, products such as search engines, chatbots, and virtual assistants often use large language model families or multimodal models as the underlying neural network models for reasoning. In actual practice, it is found that the utilization rate of hardware resources used by these neural network models is low. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide an inference resource optimization method, device and electronic device for improving the problem of low utilization of hardware resources used by the model.

[0004] The embodiment of the present application provides a method for optimizing inference resources, including: obtaining inference service response performance data of a neural network model under different hardware resources, and hardware cost data under different hardware resources; optimizing and modeling the inference service response performance data and hardware cost data to obtain a hardware resource deployment plan with the minimum hardware cost; generating a configuration file of the neural network model according to the hardware resource deployment plan with the minimum hardware cost; and adaptively scheduling and optimizing the neural network model and different hardware resources according to the configuration file. In the implementation process of the above scheme, by optimizing and modeling the inference service response performance data and hardware cost data, and according to the configuration file generated by the hardware resource deployment plan with the minimum hardware cost, the neural network model and different hardware resources are adaptively scheduled and optimized, thereby effectively improving the utilization rate of hardware resources.

[0005] Optionally, in an embodiment of the present application, different hardware resources include: multiple computing resources and multi-level storage resources; optimizing modeling of reasoning service response performance data and hardware cost data, including: constructing an integer linear programming ILP solver through multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources and hardware cost data of multi-level storage resources; using the ILP solver to optimize modeling of reasoning service response performance data and hardware cost data. In the implementation process of the above scheme, by using the ILP solver to perform linear programming optimization modeling on the reasoning service response performance data and hardware cost data, the large model or large language model effectively converts its low hardware cost reasoning demand problem under specific reasoning service response performance requirements into an optimization modeling problem under different hardware resource constraints under specific reasoning service response performance requirements, and can accurately balance the relationship between reasoning performance and hardware cost, ensuring that while meeting specific service response time, throughput and other performance requirements, the hardware deployment and operation costs are minimized, and can automatically select the most cost-effective hardware resource configuration solution, avoiding excessive configuration or waste of resources.

[0006] Optionally, in an embodiment of the present application, the ILP solver includes: an objective function; constructing an integer linear programming ILP solver by using multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources, and hardware cost data of multi-level storage resources, including: determining the objective function by using multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources, and hardware cost data of multi-level storage resources Among them, n_acc represents the total number of types of computing resources, Acc i Indicates the number of computing resources of the i-th computing resource among multiple computing resources, cost_acc i Indicates the hardware cost data of the i-th computing resource among multiple computing resources, n_mem indicates the total number of types of multi-level storage resources, mem j Indicates the storage size of the i-th level storage resource in the multi-level storage resource, cost_mem j Represents the hardware cost data of the i-th level storage resource in the multi-level storage resource. In the implementation process of the above solution, since computing power resources, storage resources and inference service request response performance data (such as parameters of different types of service requests) are closely related to the hardware cost of large-model inference services, setting the above objective function of minimizing the hardware cost of the neural network model can effectively convert the low-hardware cost inference demand problem into an optimization modeling problem under different hardware resource constraints. Specifically, the low-hardware cost hardware resource deployment problem is converted into an integer linear programming problem, which can automatically select the most cost-effective hardware resource configuration solution and avoid over-configuration or waste of resources.

[0007] Optionally, in an embodiment of the present application, the ILP solver further includes: constraints on computing resources; constructing an integer linear programming ILP solver by using multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources, and hardware cost data of multi-level storage resources, and further includes: Calculate various computing resources to obtain the constraints of computing resources; among them, Acc_comp i Indicates the amount of computing required by the i-th computing resource among multiple computing resources, Acc_flops i Indicates the maximum computing power of the i-th computing power resource among multiple computing power resources, T communication "latency_demand" represents the data communication duration between any two computing resources among multiple computing resources, and "latency_demand" represents the maximum latency of the neural network model's inference server. In implementing the above solution, by converting the constraints of multiple computing resources into a mathematical model, the ILP solver can find the global optimal solution under given conditions, ensuring that the resource allocation plan minimizes hardware costs while satisfying all constraints. Furthermore, by directly incorporating the hardware cost of computing resources into the optimization objective, this solution can automatically find the resource configuration solution with the lowest hardware cost while meeting performance requirements (such as low latency).

[0008] Optionally, in an embodiment of the present application, the ILP solver further includes: storage resource constraints; constructing an integer linear programming ILP solver by using multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources, and hardware cost data of multi-level storage resources; and further including: acc +Act acc +KV acc <acc memory and w mem +Act mem +KV mem <mem memory Calculate the multi-level storage resources to obtain the storage resource constraints; where w acc Indicates the data size of the weight parameters of the neural network model in the accelerator memory in the multi-level storage resource, w mem Indicates the data size of the weight parameters of the neural network model in the server memory in the multi-level storage resource, Act acc Indicates the data size of the activation value of the neural network model in the accelerator memory in the multi-level storage resource, Act mem Indicates the data size of the activation value of the neural network model in the server memory in the multi-level storage resource, KV acc Indicates the data size of the KV cache of the neural network model in the accelerator memory in the multi-level storage resources, KVmem Indicates the data size of the KV cache of the neural network model in the server memory in the multi-level storage resource, avv memory Indicates the total memory size of the accelerator memory in the multi-level storage resources, mem memory Represents the total memory size of the server memory in the multi-tier storage resource. In the implementation of the above solution, strict memory resource constraints ensure that the model weights, activation values, and KV cache are appropriately distributed between the accelerator memory and server memory, avoiding overloading of a single storage tier and improving overall memory resource efficiency.

[0009] Optionally, in an embodiment of the present application, the neural network model and different hardware resources are adaptively scheduled and optimized according to the configuration file, including: determining whether the activity value of the model parameters of the neural network model is greater than a preset threshold in the configuration file; if so, moving the model parameters of the neural network model to the accelerator memory in the multi-level storage resource, otherwise, moving the model parameters of the neural network model to the server memory in the multi-level storage resource; wherein the model parameters include: weight parameters and / or KV cache. In the implementation process of the above scheme, moving the model parameters (weights and KV cache) with high activity to the accelerator memory can reduce the access delay of the data, because the accelerator memory is usually closer to the computing unit than the server memory, thereby greatly improving the reading and writing speed of the data, so that frequently accessed data can be processed quickly, improving the reasoning response speed and throughput, further, by distinguishing the high and low activity and reasonably allocating storage resources, the low-activity parameters are placed in the lower-cost server memory, while the high-activity parameters occupy the higher-cost accelerator memory, which can ensure the fast access to critical data while avoiding the waste of expensive resources, and achieving a balance between storage cost and performance.

[0010] An embodiment of the present application also provides an inference resource optimization device, including: a resource data acquisition module, used to obtain the inference service response performance data of the neural network model under different hardware resources, and the hardware cost data under different hardware resources; a deployment plan acquisition module, used to optimize the modeling of the inference service response performance data and the hardware cost data, and obtain the hardware resource deployment plan with the minimum hardware cost; a configuration file generation module, used to generate a configuration file of the neural network model according to the hardware resource deployment plan with the minimum hardware cost; a resource scheduling optimization module, used to adaptively schedule and optimize the neural network model and different hardware resources according to the configuration file.

[0011] Optionally, in an embodiment of the present application, different hardware resources include: multiple computing power resources and multi-level storage resources; the deployment plan acquisition module includes: an ILP solver construction sub-module, which is used to construct an integer linear programming ILP solver through multiple computing power resources, hardware cost data of multiple computing power resources, multi-level storage resources and hardware cost data of multi-level storage resources; a data optimization modeling sub-module, which is used to use the ILP solver to optimize and model the inference service response performance data and hardware cost data.

[0012] Optionally, in an embodiment of the present application, the ILP solver includes: an objective function; an ILP solver construction submodule includes: an objective function determination unit for determining the objective function using multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources, and hardware cost data of multi-level storage resources. Among them, n_acc represents the total number of types of computing resources, Acc i Indicates the number of computing resources of the i-th computing resource among multiple computing resources, cost_acc i Indicates the hardware cost data of the i-th computing resource among multiple computing resources, n_mem indicates the total number of types of multi-level storage resources, mem j Indicates the storage size of the i-th level storage resource in the multi-level storage resource, cost_mem j Indicates the hardware cost data of the i-th level storage resource in the multi-level storage resources.

[0013] Optionally, in the embodiment of the present application, the ILP solver further includes: a constraint condition of computing power resources; an ILP solver construction submodule includes: a computing power constraint obtaining unit for obtaining the computing power constraint through the formula Calculate various computing resources to obtain the constraints of computing resources; among them, Acc_comp i Indicates the amount of computing required by the i-th computing resource among multiple computing resources, Acc_flops i Indicates the maximum computing power of the i-th computing power resource among multiple computing power resources, T communication Indicates the data communication duration between any two computing resources among multiple computing resources. Latency_demand indicates the maximum latency of the inference server of the neural network model.

[0014] Optionally, in the embodiment of the present application, the ILP solver further includes: storage resource constraints; the ILP solver construction submodule includes: a storage constraint obtaining unit for obtaining the storage resource constraints through the formula w acc +Act acc +KV acc <acc memory and w mem +Actmem +KV mem <mem memory Calculate the multi-level storage resources to obtain the storage resource constraints; where w acc Indicates the data size of the weight parameters of the neural network model in the accelerator memory in the multi-level storage resource, w mem Indicates the data size of the weight parameters of the neural network model in the server memory in the multi-level storage resource, Act acc Indicates the data size of the activation value of the neural network model in the accelerator memory in the multi-level storage resource, Act mem Indicates the data size of the activation value of the neural network model in the server memory in the multi-level storage resource, KV acc Indicates the data size of the KV cache of the neural network model in the accelerator memory in the multi-level storage resources, KV mem Indicates the data size of the KV cache of the neural network model in the server memory in the multi-level storage resource, acc memory Indicates the total memory size of the accelerator memory in the multi-level storage resources, mem memory Indicates the total memory size of the server memory in the multi-level storage resource.

[0015] Optionally, in an embodiment of the present application, the resource scheduling optimization module includes: an activity value judgment submodule, used to judge whether the activity value of the model parameters of the neural network model is greater than the preset threshold in the configuration file; a model parameter movement submodule, used to move the model parameters of the neural network model to the accelerator memory in the multi-level storage resources if the activity value of the model parameters of the neural network model is greater than the preset threshold in the configuration file; otherwise, move the model parameters of the neural network model to the server memory in the multi-level storage resources; wherein the model parameters include: weight parameters and / or KV cache.

[0016] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0017] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0018] An embodiment of the present application further provides a computer program product, including: a computer program or computer instructions, which executes the method described above when the computer program or computer instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only illustrate certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flow chart of the inference resource optimization method provided by an embodiment of the present application is shown;

[0021] Figure 2 Schematic diagram of the search space of the integer linear programming ILP solver provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the process of adaptive scheduling optimization provided by an embodiment of the present application is shown;

[0023] Figure 4 The flowchart of the inference resource optimization device provided by the embodiment of the present application is shown;

[0024] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the embodiments of the present application only serve the purpose of illustration and description and are not intended to limit the scope of protection of the embodiments of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the embodiments of the present application illustrate the operations implemented according to some embodiments of the embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the contents of the embodiments of the present application, can add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0026] In addition, the described embodiments are only a portion of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application generally described and shown in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present application, but rather merely represents selected embodiments of the embodiments of the present application.

[0027] It will be understood that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups).

[0028] Before introducing the inference resource optimization method provided in the embodiments of the present application, some concepts involved in the embodiments of the present application are first introduced:

[0029] Linear Programming (LP) is an important branch of operations research that has been studied early, developed rapidly, is widely used, and has relatively mature methods. It is a method to assist people in scientific management.

[0030] Integer Linear Programming (ILP) refers to integer linear programming in which decision variables must take integer values, that is, linear programming problems that require some or all decision variables to take integer values. Among them, integer linear programming includes pure integer linear programming, mixed integer linear programming, and 0-1 type integer linear programming.

[0031] It should be noted that the inference resource optimization method provided in the embodiments of the present application can be executed by electronic devices. Electronic devices herein refer to device terminals or servers capable of executing computer programs, such as smartphones, personal computers, tablet computers, personal digital assistants, or mobile internet devices. Servers refer to devices that provide computing services over a network, such as x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0032] The following is an example of an application scenario to which this inference resource optimization method is applicable: In application scenarios such as search engines, chatbots, and virtual assistants, these application scenarios usually use large models, large language models, or multimodal models with transformer structures as base models to achieve interaction with user terminals. In these application scenarios, most of them face diverse user needs, such as the diversity of single requests and high concurrency requirements. These diverse requirements usually require large-scale clustered deployment of base models with expensive computing power costs (such as hardware accelerators) and storage costs. These computing power costs and storage costs bring a heavy operational burden to enterprise operations. In this case, the inference resource optimization method can be used to optimize the different hardware resources used for the deployment of large models, large language models, or multimodal models to improve the utilization efficiency of these hardware resources, thereby reducing the use cost and operating cost of hardware resources.

[0033] See Figure 1 The flowchart of the reasoning resource optimization method provided by the embodiment of the present application is shown; the embodiment of the present application provides a reasoning resource optimization method, including:

[0034] Step S110: Obtain inference service response performance data of the neural network model under different hardware resources, as well as hardware cost data under different hardware resources.

[0035] The above-mentioned neural network model can be a large model, a large language model or a multimodal model using a transformer structure, such as a Llama model, an open pre-trained Transformer (OPT) model or a general language model (GLM). These large models, large language models or multimodal models can be network models whose model parameters reach a preset value or more. The preset value here can be 2B or 6B, etc.

[0036] Step S120: Optimize and model the inference service response performance data and the hardware cost data to obtain a hardware resource deployment solution with the lowest hardware cost.

[0037] The aforementioned inference service response performance data is the performance data of inference service requests for different neural network models under different hardware resources. This inference service response performance data includes inference latency and throughput performance indicators under different types of computing resources, different levels of storage resources, and different model inference service requirements. It is understood that the inference service corresponding to the inference service response performance data may include different inference requirement parameters. Different inference requirement parameters may include: inference service request frequency, inference requirement service frequency, input text length and output text length of a single request, etc. In specific practice, these different inference requirement parameters can be represented using integers, thereby effectively reducing the search space for optimization modeling and improving the efficiency of obtaining a hardware resource deployment solution with the lowest hardware cost.

[0038] The above-mentioned hardware resource deployment solutions with the lowest hardware cost refer to the deployment solutions with the lowest hardware cost for inference calculations of neural network models under these different hardware resources. These deployment solutions can improve the problem of high inference cost of large models or large language models after deployment.

[0039] It's understandable that since linear programming, nonlinear programming, integer programming, and combinatorial optimization are all optimization modeling methods within optimization theory, in practice, operations research modeling, such as linear programming, nonlinear programming, integer programming, or combinatorial optimization, can be performed on inference service response performance data and hardware cost data to obtain a hardware resource deployment plan that minimizes hardware cost. Considering that linear programming is a fundamental and important part of optimization theory, and integer linear programming is more suitable for inference resource optimization applications, the following focuses on integer linear programming modeling methods.

[0040] Step S130: Generate a configuration file of the neural network model according to the hardware resource deployment solution with the lowest hardware cost.

[0041] It is understandable that the electronic device can count multiple hardware resource deployment schemes with total hardware costs consumed to meet the client's delay requirements, and filter out the hardware resource deployment scheme with the lowest hardware cost from the multiple hardware resource deployment schemes with total hardware costs, and then generate a configuration file for the neural network model based on the hardware resource deployment scheme with the lowest hardware cost.

[0042] Step S140: Adaptively schedule and optimize the neural network model and different hardware resources according to the configuration file.

[0043] In the implementation process of the above solution, by optimizing the modeling of the inference service response performance data and hardware cost data, and generating a configuration file based on the hardware resource deployment solution with the lowest hardware cost, the neural network model and different hardware resources are adaptively scheduled and optimized, thereby effectively improving the utilization of hardware resources.

[0044] As an optional implementation of step S120, the different hardware resources may include: multiple computing resources and multi-level storage resources; and the implementation of optimizing the modeling of the inference service response performance data and the hardware cost data may include:

[0045] Step S121: construct an integer linear programming (ILP) solver using multiple computing resources, hardware cost data of the multiple computing resources, multi-level storage resources, and hardware cost data of the multi-level storage resources.

[0046] The above-mentioned multiple computing resources can be heterogeneous computing resources. Here, heterogeneous computing resources refer to a collection of computing resources composed of computing units with different architectures, designs, and performance characteristics. For example, they can be different types of central processing units (CPUs), field programmable gate arrays (FPGAs), L4, A10G, and different types of graphics processing units (GPUs). The different types of GPUs here include A100, H100, BI100, and MR100, etc. The GPUs here can also be domestically produced GPUs.

[0047] The aforementioned multi-level storage resources may include, for example, accelerator video memory, such as High Bandwidth Memory (HBM). The aforementioned multi-level storage resources may also include server memory, such as Dynamic Random Access Memory (DRAM) or server hard disks. The aforementioned multi-level storage resources may also include server storage hard disks. In practice, storage resources may be calculated using 2GB units. Storage resources less than 2GB may be calculated as 2GB, thereby representing the multi-level storage resources in integer form.

[0048] Step S122: Use the ILP solver to perform linear programming optimization modeling on the inference service response performance data and the hardware cost data to obtain a hardware resource deployment solution with the minimum hardware cost.

[0049] It can be understood that the purpose of constructing the integer linear programming ILP solver mentioned above is to minimize the cost of large-scale inference services while meeting the model inference performance requirements. Therefore, the ILP solver can be used to perform linear programming optimization modeling on the inference service response performance data and hardware cost data to obtain a hardware resource deployment plan with the lowest hardware cost.

[0050] During the implementation of the above solution, by using the ILP solver to perform linear programming optimization modeling on the inference service response performance data and hardware cost data, the large model or large language model can effectively convert its low hardware cost inference demand problem into an optimization modeling problem under different hardware resource constraints under specific inference service response performance requirements. It can accurately balance the relationship between inference performance and hardware cost, ensuring that while meeting specific service response time, throughput and other performance requirements, the hardware deployment and operation costs are minimized, and can automatically select the most cost-effective hardware resource configuration plan, avoiding over-configuration or waste of resources.

[0051] As an optional implementation of step S121, the ILP solver may include: an objective function; and the implementation of constructing the integer linear programming ILP solver may include:

[0052] Step S121a: Determine the objective function by using multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources and hardware cost data of multi-level storage resources The purpose of this objective function is to minimize the inference service cost of a neural network model (such as a large model, a large language model, or a multimodal model) under different hardware resources. The inference service cost here can be the sum of the hardware cost data of multiple computing resources and the hardware cost data of multi-level storage resources.

[0053] Among them, n_acc represents the total number of types of computing resources, Acc i Indicates the number of computing resources of the i-th computing resource among multiple computing resources, cost_acc i Indicates the hardware cost data of the i-th computing resource among multiple computing resources, n_mem indicates the total number of types of multi-level storage resources, mem j Indicates the storage size of the i-th level storage resource in the multi-level storage resource, cost_mem j Indicates the hardware cost data of the i-th level storage resource in the multi-level storage resource. It can be understood that the above storage resource mem j It represents a continuous function. In the specific practice, a discretized form (such as a histogram) can be used for sampling analysis to form the objective function of integer linear programming.

[0054] See Figure 2 A schematic diagram of the search space of the integer linear programming ILP solver provided in an embodiment of the present application is shown; it can be understood that the above-mentioned objective function of minimizing the hardware cost of the neural network model can effectively convert the low hardware cost inference requirement problem into an optimization modeling problem under different hardware resource constraints, specifically, the low hardware cost hardware resource deployment problem is converted into an integer linear programming problem, so that the search space of the integer linear programming ILP solver includes different types of computing resource parameters and multi-level storage resource parameters for large model deployment, large model inference service request frequency, and large model request service parameters (such as the input text length and output text length of a single request).

[0055] In the implementation process of the above solution, since computing power resources, storage resources and inference service request response performance data (such as parameters of different types of service requests) are closely related to the hardware cost of large-model inference services, setting the above objective function of minimizing the hardware cost of the neural network model can effectively convert the low-hardware cost inference demand problem into an optimization modeling problem under different hardware resource constraints. Specifically, the low-hardware cost hardware resource deployment problem is converted into an integer linear programming problem, so that the most cost-effective hardware resource configuration solution can be automatically selected, avoiding excessive configuration or waste of resources.

[0056] As an optional implementation of step S121, the ILP solver may further include: constraints on computing resources; constructing an integer linear programming ILP solver using multiple computing resources, hardware cost data of the multiple computing resources, multi-level storage resources, and hardware cost data of the multi-level storage resources. The implementation may further include:

[0057] Step S121b: Pass Calculate various computing resources to obtain the constraints of computing resources;

[0058] Among them, Acc_comp i Indicates the amount of computing required by the i-th computing resource among multiple computing resources, Acc_flops i Indicates the maximum computing power of the i-th computing power resource among multiple computing power resources, T communication Indicates the data communication duration between any two computing resources among multiple computing resources. Latency_demand indicates the maximum latency of the inference server of the neural network model. The maximum latency of the inference server here can be set according to the specific situation. For example, the maximum latency of the inference server can be set to 50 milliseconds.

[0059] It is understandable that the constraints of the above-mentioned computing power resources can be obtained by offline analysis and calculation through an offline analysis module in the electronic device. The offline analysis here can include: analysis of the weight parameters of the neural network model (such as a large model, a large language model, or a multimodal model), analysis and calculation of different hardware cost requirements, and different reasoning service response performance requirements. In other words, the constraints of such computing power resources are the constraints of the integer linear programming problem formed by analyzing the neural network model (such as a large model, a large language model, or a multimodal model) under different hardware cost requirements and different reasoning service response performance requirements.

[0060] As an optional implementation of step S121, the ILP solver may further include: storage resource constraints; constructing an integer linear programming ILP solver using multiple computing resources, hardware cost data of the multiple computing resources, multi-level storage resources, and hardware cost data of the multi-level storage resources. The implementation may further include:

[0061] Step S121c: By formula w acc +Act acc +KV acc <acc memory and w mem +Act mem +KV mem <mem memory Calculate multi-level storage resources to obtain storage resource constraints;

[0062] Among them, w acc Indicates the data size of the weight parameters of the neural network model in the accelerator memory in the multi-level storage resource, w mem Indicates the data size of the weight parameters of the neural network model in the server memory in the multi-level storage resource, Act acc Indicates the data size of the activation value of the neural network model in the accelerator memory in the multi-level storage resource, Act mem Indicates the data size of the activation value of the neural network model in the server memory in the multi-level storage resource, KV acc Indicates the data size of the KV cache of the neural network model in the accelerator memory in the multi-level storage resources, KV mem Indicates the data size of the KV cache of the neural network model in the server memory in the multi-level storage resource, acc memory Indicates the total memory size of the accelerator memory in the multi-level storage resources, mem memory Indicates the total memory size of the server memory in the multi-level storage resource.

[0063] Optionally, in addition to the constraints on computing resources and storage resources, the above-mentioned ILP solver can also include more constraints, such as: the storage resources at each level need to be greater than the peak storage demand and network throughput of the model inference, and the delay time of large model inference needs to be less than the minimum delay time required by the user service, etc.

[0064] As an optional implementation of step S122, the implementation of adaptively scheduling and optimizing the neural network model and different hardware resources according to the configuration file may include:

[0065] Step S122a: Determine whether the activity value of the model parameter of the neural network model is greater than a preset threshold in the configuration file.

[0066] It can be understood that the activity value of the model parameters of the above-mentioned neural network model can be obtained by statistical analysis of the model parameters of the neural network model. For example, if the model parameters (including weight parameters and / or KV cache) of a neural network model such as a large model, a large language model or a multimodal model are called once during the inference service request, the activity value of the model parameter is added by 1, and the inference service request is completed, the activity value of the model parameter is statistically calculated.

[0067] The implementation of the above step S122a is, for example: by using the calculation formula of activity To calculate the activity value of the model parameters of the neural network model, and then, through the executable program, to determine whether the activity value of the model parameters of the neural network model is greater than the preset threshold in the configuration file. Among them, the above-mentioned activity value can be calculated in the offline statistical analysis stage, tokens_total represents the total number of tokens in the inference process, and tokens_activated represents the number of tokens activated in the nodes of the neural network model during the inference process. Optionally, in the specific practice, the above-mentioned neural network model can be a model using a transformer structure, which usually includes an attention block. In this case, since the cumulative attention scores of all tokens in the attention block follow a power law distribution, the KV cache value of the token with the highest attention score in the attention block (for example, the top 20% of the token) can be directly used as a high-activity KV cache, or the KV cache value of the token with the attention score of the token in the attention block greater than the preset score can be directly used as a high-activity KV cache.

[0068] Step S122b: If the activity value of the model parameter of the neural network model is greater than the preset threshold in the configuration file, the model parameter of the neural network model is moved to the accelerator memory in the multi-level storage resource.

[0069] The implementation method of the above-mentioned step S122b includes: if the activity value of the model parameter of the neural network model is greater than the preset threshold in the configuration file, it means that the activity value of the model parameter is high, then the model parameters of the active part can be placed in the accelerator memory, or the model parameters of the neural network model can be moved to the accelerator memory in the multi-level storage resources, for example, this part of the high-activity parameters can be moved to the high-bandwidth memory (HBM) with fast reading speed.

[0070] Step S122c: If the activity value of the model parameter of the neural network model is less than or equal to the preset threshold in the configuration file, the model parameter of the neural network model is moved to the server memory in the multi-level storage resource.

[0071] For example, the implementation of the above step S122b is as follows: model parameters of different activity levels can be placed in different types of storage resources. If the activity value of the model parameters of the neural network model is less than or equal to the preset threshold in the configuration file, it means that the activity value of the model parameters is low. In this case, the model parameters of the inactive part can be placed in the server memory, and the model parameters of the neural network model can be moved to the server memory in the multi-level storage resources, for example, the model parameters of the inactive part can be moved to the dynamic random access memory (DRAM) or server hard disk with a slower reading speed. The above-mentioned model parameters may include: weight parameters and / or KV cache.

[0072] During the implementation of the above solution, the storage layout is dynamically adjusted according to the actual activity of the model during runtime, allowing the system to automatically adjust under different workloads, improving adaptability to different application scenarios. Especially in the model inference phase, resource allocation can be flexibly optimized according to real-time needs.

[0073] As an optional implementation of the above step S130, for example, after obtaining the hardware resource deployment solution with the minimum hardware cost, a configuration file of the neural network model can be generated according to the hardware resource deployment solution with the minimum hardware cost. It is understandable that since the configuration file is generated according to the hardware resource deployment solution with the minimum hardware cost, the content in the configuration file can be information about the hardware resources that need to be deployed, for example, the configuration file can include model weight parameters and KV cache storage and ratio configuration information in multi-level storage resources, configuration information of various computing resources required for large model neuron calculations, etc.

[0074] See Figure 3A schematic diagram of the process of adaptive scheduling optimization provided by an embodiment of the present application is shown; as an optional implementation method of the above-mentioned step S140, for example: based on the inference demand parameters such as the large model inference service request frequency, the inference demand service frequency, the single request input text length and the output text length, the pending requests of each neural network model are dynamically allocated in real time to the different hardware resources selected in the configuration file to complete the adaptive scheduling optimization of the neural network model and different hardware resources. For example, the electronic device can select appropriate computing power resources from a variety of heterogeneous computing power resources according to the configuration file, and select appropriate accelerator graphics memory and server memory from multi-level storage resources. The above-mentioned adaptive scheduling optimization process can be completed by the real-time scheduling deployment module of the electronic device. The real-time scheduling deployment module can realize the adaptive scheduling optimization allocation function of the pending requests by executing the above inference resource optimization method.

[0075] Optionally, the above-mentioned electronic device can be a cluster control node for multiple computing resources and / or multi-level storage resources. That is to say, the above-mentioned multiple computing resources and / or multi-level storage resources can be operated in a high-availability distributed cluster manner, and the cluster control node can monitor and control the cluster node status of multiple computing resources and / or multi-level storage resources in real time, so as to facilitate adaptive scheduling optimization of the high-availability distributed cluster built with multiple computing resources and / or multi-level storage resources.

[0076] As an optional implementation of the above-mentioned reasoning resource optimization method, an exception handling mechanism module can also be set in the above-mentioned electronic device. The exception handling mechanism module can identify the exception and perform exception handling when an abnormality occurs in the execution process of the above-mentioned reasoning optimization method. For example: in the case of GPU failure or storage device performance degradation, the pending requests of each neural network model can be dynamically allocated to different hardware resources selected in the configuration file, or the hardware resources with abnormalities (including computing resources and storage resources) can be quickly identified, and the pending requests can be reallocated to healthy hardware resources.

[0077] See Figure 4 The flowchart of the inference resource optimization device provided by the embodiment of the present application is shown; the embodiment of the present application provides a inference resource optimization device 200, including:

[0078] The resource data acquisition module 210 is used to obtain the inference service response performance data of the neural network model under different hardware resources, as well as the hardware cost data under different hardware resources.

[0079] The deployment plan obtaining module 220 is used to optimize and model the inference service response performance data and the hardware cost data to obtain a hardware resource deployment plan with the minimum hardware cost.

[0080] The configuration file generation module 230 is used to generate a configuration file of the neural network model according to the hardware resource deployment solution with the minimum hardware cost.

[0081] The resource scheduling optimization module 240 is used to perform adaptive scheduling optimization on the neural network model and different hardware resources according to the configuration file.

[0082] As an optional implementation of the above device, different hardware resources include: multiple computing resources and multi-level storage resources; the deployment plan acquisition module includes:

[0083] The ILP solver construction submodule is used to construct an integer linear programming ILP solver through multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources, and hardware cost data of multi-level storage resources.

[0084] The data optimization modeling submodule is used to optimize and model the inference service response performance data and hardware cost data using the ILP solver.

[0085] As an optional embodiment of the above-mentioned device, an ILP solver includes: an objective function; constructing an integer linear programming ILP solver using multiple computing resources, hardware cost data of the multiple computing resources, multi-level storage resources, and hardware cost data of the multi-level storage resources, including:

[0086] Determine the objective function through multiple computing resources, hardware cost data of multiple computing resources, multi-level storage resources and hardware cost data of multi-level storage resources

[0087] Among them, n_acc represents the total number of types of computing resources, Acc i Indicates the number of computing resources of the i-th computing resource among multiple computing resources, cost_acc i Indicates the hardware cost data of the i-th computing resource among multiple computing resources, n_mem indicates the total number of types of multi-level storage resources, mem j Indicates the storage size of the i-th level storage resource in the multi-level storage resource, cost_mem j Indicates the hardware cost data of the i-th level storage resource in the multi-level storage resources.

[0088] As an optional embodiment of the above-mentioned apparatus, the ILP solver further includes: constraints on computing resources; constructing an integer linear programming ILP solver using multiple computing resources, hardware cost data of the multiple computing resources, multi-level storage resources, and hardware cost data of the multi-level storage resources, and further includes:

[0089] By formula Calculate various computing resources to obtain the constraints of computing resources;

[0090] Among them, Acc_comp i Indicates the amount of computing required by the i-th computing resource among multiple computing resources, Acc_flops i Indicates the maximum computing power of the i-th computing power resource among multiple computing power resources, T communication Indicates the data communication duration between any two computing resources among multiple computing resources. Latency_demand indicates the maximum latency of the inference server of the neural network model.

[0091] As an optional embodiment of the above-mentioned apparatus, the ILP solver further includes: storage resource constraints; constructing an integer linear programming ILP solver using multiple computing resources, hardware cost data of the multiple computing resources, multi-level storage resources, and hardware cost data of the multi-level storage resources, and further includes:

[0092] By formula w acc +Act acc +KV acc <acc memory and w mem +Act mem +KV mem <mem memory Calculate multi-level storage resources to obtain storage resource constraints;

[0093] Among them, w acc Indicates the data size of the weight parameters of the neural network model in the accelerator memory in the multi-level storage resource, w mem Indicates the data size of the weight parameters of the neural network model in the server memory in the multi-level storage resource, Act acc Indicates the data size of the activation value of the neural network model in the accelerator memory in the multi-level storage resource, Act mem Indicates the data size of the activation value of the neural network model in the server memory in the multi-level storage resource, KV acc Indicates the data size of the KV cache of the neural network model in the accelerator memory in the multi-level storage resources, KV mem Indicates the data size of the KV cache of the neural network model in the server memory in the multi-level storage resource, acc memory Indicates the total memory size of the accelerator memory in the multi-level storage resources, mem memory Indicates the total memory size of the server memory in the multi-level storage resource.

[0094] As an optional implementation of the above device, the resource scheduling optimization module includes:

[0095] The activity value judgment submodule is used to judge whether the activity value of the model parameter of the neural network model is greater than the preset threshold in the configuration file.

[0096] The model parameter moving submodule is used to move the model parameters of the neural network model to the accelerator memory in the multi-level storage resources if the activity value of the model parameters of the neural network model is greater than the preset threshold in the configuration file; otherwise, the model parameters of the neural network model are moved to the server memory in the multi-level storage resources; wherein the model parameters include: weight parameters and / or KV cache.

[0097] It should be understood that the device corresponds to the aforementioned inference resource optimization method embodiment and is capable of executing each of the steps involved in the aforementioned method embodiment. The specific functions of the device can be found in the description above, and a detailed description is omitted here. The device includes at least one software functional module that can be stored in a memory in the form of software or firmware or embedded in the device's operating system (OS).

[0098] See Figure 5 The electronic device 300 provided in the embodiment of the present application includes a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0099] The embodiment of the present application further provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program executes the above method when the processor 310 runs. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0100] An embodiment of the present application further provides a computer program product, including: a computer program or computer instructions, which executes the method described above when the computer program or computer instructions are executed by a processor.

[0101] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0102] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly based on the functions involved.

[0103] In addition, the functional modules of each embodiment in the embodiments of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in an appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0104] The above description is only an optional implementation method of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A method for optimizing inference resources, characterized in that: include: Obtaining inference service response performance data of the neural network model under different hardware resources, as well as hardware cost data under the different hardware resources; Optimizing and modeling the inference service response performance data and the hardware cost data to obtain a hardware resource deployment solution with the lowest hardware cost; Generating a configuration file of the neural network model according to the hardware resource deployment solution with the lowest hardware cost; Performing adaptive scheduling optimization on the neural network model and the different hardware resources according to the configuration file; The different hardware resources include: multiple computing resources and multi-level storage resources; the optimization modeling of the reasoning service response performance data and the hardware cost data includes: constructing an integer linear programming (ILP) solver using the multiple computing resources, the hardware cost data of the multiple computing resources, the multi-level storage resources, and the hardware cost data of the multi-level storage resources; and using the ILP solver to optimize the modeling of the reasoning service response performance data and the hardware cost data. The adaptive scheduling optimization of the neural network model and the different hardware resources according to the configuration file includes: judging whether the activity value of the model parameter of the neural network model is greater than the preset threshold value in the configuration file; if so, moving the model parameter of the neural network model to the accelerator memory in the multi-level storage resource; otherwise, moving the model parameter of the neural network model to the server memory in the multi-level storage resource; wherein, the model parameter includes: weight parameter and / or KV cache; the activity value of the model parameter of the neural network model is calculated using the formula Calculated, represents the total number of tokens during the inference process, Represents the number of tokens that are activated in the neural network model during inference.

2. The method according to claim 1, characterized in that The ILP solver includes: an objective function; and constructing an integer linear programming ILP solver using the multiple computing resources, the hardware cost data of the multiple computing resources, the multi-level storage resources, and the hardware cost data of the multi-level storage resources, including: The objective function is determined by the multiple computing resources, the hardware cost data of the multiple computing resources, the multi-level storage resources and the hardware cost data of the multi-level storage resources. ; in, Indicates the total number of types of the various computing resources. Indicates the first of the multiple computing resources The number of computing resources of this type of computing power, Indicates the first of the multiple computing resources Hardware cost data for various computing resources, Indicates the total number of types of the multi-level storage resources, Indicates the first level of the multi-level storage resource The storage size of the level storage resource, Indicates the first level of the multi-level storage resource Hardware cost data for tier-1 storage resources.

3. The method according to claim 2, characterized in that The ILP solver further includes: constraints on computing resources; the integer linear programming ILP solver is constructed by using the multiple computing resources, the hardware cost data of the multiple computing resources, the multi-level storage resources, and the hardware cost data of the multi-level storage resources, further including: By formula Calculating the multiple computing resources to obtain constraints on the computing resources; in, Indicates the first of the multiple computing resources The amount of computing power required by the computing resources, Indicates the first of the multiple computing resources The maximum computing power of the computing resources, Indicates the data communication duration between any two computing resources among the multiple computing resources. Indicates the maximum latency of the inference server for the neural network model.

4. The method according to claim 2, characterized in that The ILP solver further includes: storage resource constraints; the integer linear programming ILP solver is constructed by using the multiple computing resources, the hardware cost data of the multiple computing resources, the multi-level storage resources, and the hardware cost data of the multi-level storage resources, and further includes: By formula and Calculating the multi-level storage resources to obtain constraints on the storage resources; in, represents the data size of the weight parameters of the neural network model in the accelerator memory in the multi-level storage resource, represents the data size of the weight parameters of the neural network model in the server memory in the multi-level storage resource, represents the data size of the activation value of the neural network model in the accelerator memory in the multi-level storage resource, represents the data size of the activation value of the neural network model in the server memory in the multi-level storage resource, Indicates the data size of the KV cache of the neural network model in the accelerator memory in the multi-level storage resource, Indicates the data size of the KV cache of the neural network model in the server memory in the multi-level storage resource, represents the total memory size of the accelerator memory in the multi-level storage resources, Indicates the total memory size of the server memory in the multi-level storage resource.

5. A reasoning resource optimization device, characterized in that: include: A resource data acquisition module is used to obtain the inference service response performance data of the neural network model under different hardware resources, as well as the hardware cost data under the different hardware resources; A deployment plan acquisition module is used to optimize and model the inference service response performance data and the hardware cost data to obtain a hardware resource deployment plan with the lowest hardware cost; A configuration file generating module, configured to generate a configuration file of the neural network model according to the hardware resource deployment solution with the lowest hardware cost; A resource scheduling optimization module, configured to perform adaptive scheduling optimization on the neural network model and the different hardware resources according to the configuration file; The different hardware resources include: multiple computing resources and multi-level storage resources; the optimization modeling of the reasoning service response performance data and the hardware cost data includes: constructing an integer linear programming (ILP) solver using the multiple computing resources, the hardware cost data of the multiple computing resources, the multi-level storage resources, and the hardware cost data of the multi-level storage resources; and using the ILP solver to optimize the modeling of the reasoning service response performance data and the hardware cost data. The adaptive scheduling optimization of the neural network model and the different hardware resources according to the configuration file includes: judging whether the activity value of the model parameter of the neural network model is greater than the preset threshold value in the configuration file; if so, moving the model parameter of the neural network model to the accelerator memory in the multi-level storage resource; otherwise, moving the model parameter of the neural network model to the server memory in the multi-level storage resource; wherein, the model parameter includes: weight parameter and / or KV cache; the activity value of the model parameter of the neural network model is calculated using the formula Calculated, represents the total number of tokens during the inference process, Represents the number of tokens that are activated in the neural network model during inference.

6. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is executed.

8. A computer program product, characterized in that include: A computer program or computer instruction, wherein the computer program or the computer instruction is executed by a processor to perform the method according to any one of claims 1 to 4.

Images