Reasoning service resource configuration method, electronic device and readable storage medium

Abstract

The application discloses a reasoning service resource configuration method, an electronic device and a readable storage medium, relates to the technical field of computers, and comprises the following steps: obtaining a model file of a reasoning service, analyzing the model file to determine minimum storage resource requirements required for running the reasoning service; determining a resource specification that meets the minimum storage resource requirements, wherein the resource specification is used to represent nodes required for running the reasoning service and storage resource configurations corresponding to the nodes; determining a resource configuration strategy matched with the resource specification according to storage resource states of the nodes in a reasoning cluster, and configuring storage resources in the reasoning cluster according to the resource configuration strategy to run the reasoning service, thereby solving the technical problems of low automation and poor utilization of storage resources when resources are configured for the reasoning service, reducing the error risk and cost of manual operation, and improving the utilization of storage resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a reasoning service resource configuration method, an electronic device and a readable storage medium. BACKGROUND

[0002] With the rapid development of computer technology, pre-training models are increasingly widely used in natural language processing, computer vision and other fields. A large amount of storage resources (such as video memory resources) are needed to store data in the model reasoning service process to enable the reasoning service to execute normally. In related technologies, the storage resource information required for reasoning service running is usually manually evaluated and configured before model deployment and reasoning service running, which has low automation and poor utilization of storage resources. SUMMARY

[0003] The present application provides a reasoning service resource configuration method, an electronic device and a readable storage medium to at least solve the problems of low automation and poor utilization of storage resources in related technologies when configuring resources for reasoning services.

[0004] The present application provides a reasoning service resource configuration method, comprising:

[0005] obtaining a model file of a reasoning service, and parsing the model file to determine the minimum storage resource requirement for running the reasoning service;

[0006] determining a resource specification that meets the minimum storage resource requirement, the resource specification being used to represent the node required for running the reasoning service and the storage resource configuration corresponding to the node;

[0007] determining a resource configuration strategy matching the resource specification according to the storage resource state of the nodes in the reasoning cluster, and configuring storage resources in the reasoning cluster according to the resource configuration strategy to run the reasoning service.

[0008] The present application also provides a computer program product, comprising:

[0009] a first processing module for obtaining a model file of a reasoning service, and parsing the model file to determine the minimum storage resource requirement for running the reasoning service;

[0010] a second processing module for determining a resource specification that meets the minimum storage resource requirement, the resource specification being used to represent the node required for running the reasoning service and the storage resource configuration corresponding to the node;

[0011] The third processing module is configured to determine a resource configuration strategy matching the resource specification according to a storage resource state of a node in the inference cluster, and configure storage resources in the inference cluster according to the resource configuration strategy to run the inference service.

[0012] The application further provides an electronic device, including a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the inference service resource configuration methods.

[0013] The application further provides a non-volatile computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any of the inference service resource configuration methods.

[0014] The application further provides another computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of any of the inference service resource configuration methods.

[0015] By the inference service resource configuration method provided by the application, the model file of the inference service is acquired, and the model file is parsed to determine the minimum storage resource requirement required for running the inference service, thereby providing a data basis for subsequent automatic resource configuration. After the minimum storage resource requirement is determined, a resource specification satisfying the minimum storage resource requirement is determined, a resource configuration strategy is determined according to a storage resource state of a node in the inference cluster, the resource configuration strategy is used to indicate how to configure storage resources in the inference cluster to match the resource specification, storage resources are configured in the inference cluster according to the resource allocation strategy to run the inference service. Therefore, the technical problems of low automation degree and poor utilization rate of storage resources in resource configuration of the inference service can be solved, automatic determination of storage resources required for the inference service and automatic execution of storage resource configuration are realized, the error risk and cost of manual operation are reduced, resource waste or deficiency caused by fragmentation can be avoided, and the utilization rate of storage resources is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the application, the drawings required in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 One of the flowcharts of the inference service resource configuration method provided by some embodiments of the application;

[0018] Figure 2Flowchart of the inference service resource configuration method provided by some embodiments of the present application;

[0019] Figure 3 Flowchart of the inference service resource configuration method provided by some embodiments of the present application;

[0020] Figure 4 Flowchart of the inference service resource configuration method provided by some embodiments of the present application;

[0021] Figure 5 Schematic diagram of the computer program product provided by some embodiments of the present application.

[0022] Explanation of reference signs:

[0023] 500: computer program product; 501: first processing module; 502: second processing module;

[0024] 503: third processing module. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0026] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] In order to make those skilled in the art better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0028] The embodiments of the present application provide an inference service resource configuration method, which is described in detail in combination with the execution flow of the inference service resource configuration method.

[0029] Figure 1 Flowchart of the inference service resource configuration method provided by some embodiments of the present application. As Figure 1As shown, the inference service resource configuration method includes steps 110, 120, and 130.

[0030] Step 110, obtaining a model file of an inference service, and parsing the model file to determine the minimum storage resource requirement for running the inference service.

[0031] It can be understood that the inference service refers to deploying a trained machine learning model or a deep learning model into a production environment to make real-time predictions and inferences on new data. The inference process usually involves processing input data and calculating output results through a model. The model file is a file containing the parameters and structure of the model executing the inference service, which is used to load and execute the model in the inference process. It can be considered that the model file contains the information required to perform the inference task.

[0032] Storage resources refer to hardware and software resources used to store data, including but not limited to graphics processing unit (GPU) memory (i.e., video memory), hard disk storage, solid state storage, and memory storage. The performance and capacity of storage resources directly affect the read / write speed and storage capacity of data. The minimum storage resource requirement refers to the minimum amount of storage resources required for the inference service to run normally. It can be understood that complex deep learning models usually require more storage space to store model parameters and intermediate calculation results, and high-frequency real-time inference services require faster storage access speed.

[0033] Parsing the model file to determine the minimum storage resource requirement for running the inference service may, for example, be calculating the minimum storage space required for service running by analyzing the model size, dataset size, or expected load of the model (such as model data volume) contained in the model file. In some embodiments, it can also be automatically evaluating the storage resources required for running the inference service using an automated model analysis tool. If there is historical running data of the target inference service, the resource requirement of the target inference service can also be analyzed based on the historical running data.

[0034] In some embodiments, the obtaining a model file of an inference service, and parsing the model file to determine the minimum storage resource requirement for running the inference service includes:

[0035] Obtaining a model file of an inference service, and parsing the model file to obtain the model parameter quantity and model quantization precision corresponding to the inference service.

[0036] Based on the model parameter quantity and the model quantization precision, determining the minimum storage resource requirement for running the inference service.

[0037] The model parameter quantity refers to the total number of weight and bias parameters in the model, which is usually used to measure the complexity and size of the model and directly affects the storage requirements and computing resource requirements of the model. The model quantization precision refers to the numerical precision used in the model parameters and the model inference process.

[0038] Exemplarily, the model parameter quantity and the quantization precision information can be extracted directly by accessing the model file, or can be extracted by parsing the model file using an automated script or a special tool. Determining the minimum storage resource requirement for running the inference service based on the model parameter quantity and the model quantization precision can be calculating the resource quantity required by the model in the storage and computing process according to the model parameter quantity and the quantization precision, or using simulation or machine learning methods to predict the resource requirement under different parameter quantities and precisions.

[0039] By parsing the model file to determine the model parameter quantity and the quantization precision, and then determining the minimum storage resource requirement for running the inference service, a data basis is provided for subsequent automatic resource allocation, which helps to improve the efficiency and accuracy of resource configuration under the premise that the inference service can normally run.

[0040] An example is given below:

[0041] Obtaining the model parameter quantity by parsing the model file , in units of billions (Billion), for example, the number of tensor arrays can be obtained by loading the model file;

[0042] Obtaining the model quantization precision by parsing the model file , for example, the model quantization precision can be obtained by reading the parameter quantization_config.quant_method (a parameter in the config.json file) in the config.json (a common configuration file format) file, or reading the parameter torch_dtype (a parameter in the config.json file) in the file Common model quantization precisions include 8-bit floating point (8-bit Floating point, FP8), 16-bit floating point (16-bit Floating point, FP16), 16-bit brain floating point (16-bit Brain Floating point, FP16), and 32-bit floating point (32-bit Floating point, FP32), etc.

[0043] Taking the storage resource as the display memory as an example, the minimum storage resource requirement for running the inference service based on the model parameter quantity and the model quantization precision determining the minimum storage resource requirement for running the inference service:

[0044] In the case that the model quantization precision is 8-bit floating point number ( ), ;

[0045] In the case that the model quantization precision is 16-bit floating point number ( ), ;

[0046] In the case that the model quantization precision is 32-bit floating point number ( ), .

[0047] Step 120, determining a resource specification that meets the minimum storage resource requirement, the resource specification being used to represent the node and the storage resource configuration corresponding to the node required for running the inference service.

[0048] It can be understood that the node refers to an independent unit in the cluster. For example, the node can be any kind of device deployed with one or more graphics processing units (GPU). The nodes work together to provide computing and storage capabilities. The inference cluster refers to a group of nodes working together to provide scalable inference capabilities through clustering, supporting large-scale data processing.

[0049] The resource specification is used to guide how to configure the storage resource in the inference cluster to meet the storage requirement of the inference service (at least meet the minimum storage resource requirement). Specifically, the resource specification can be considered as a series of requirements and configurations about the node and its corresponding storage resource defined for running the inference service. In some embodiments, the resource specification can also include requirements for computing resources, storage resources, memory, network bandwidth, etc.

[0050] For the inference service, there can be multiple resource specifications that meet the minimum storage resource requirement.

[0051] Table 1 Resource specification example table

[0052]

[0053] Taking the resource specification example table in Table 1 as an example, for an inference service with a minimum storage resource requirement of about 809G, the resource specification thereof can be a node including 6 GPUs of model A with a memory of 141G, can be a node including 9 GPUs of model B with a memory of 96G, or can be a node including 21 GPUs of model C with a memory of 40G, or if the inference service is deployed in a multi-node distributed manner, the resource specification thereof can be two nodes including 5 GPUs of model B with a memory of 96G.

[0054] The above is only an example to illustrate that there can be multiple resource specifications satisfying the minimum storage resource requirement for an inference service. In actual applications, the resource specification can be used to represent more information, such as the number of nodes, the number of inference service instances of the inference service, the parallel strategy of the inference service, and the like.

[0055] Figure 2 Flowchart two of the inference service resource configuration method provided by some embodiments of the present application. As shown in Figure 2 some embodiments, the inference service resource configuration method further includes:

[0056] receiving an inference service running request; obtaining and parsing a model file of the inference service;

[0057] determining whether there is model information corresponding to the inference service;

[0058] If there is, directly calculate the required video memory (corresponding to the minimum storage resource requirement described above) for running the inference service;

[0059] If there is not, before calculating the required video memory for running the inference service, the model parameter quantity and the model quantization precision also need to be calculated;

[0060] After the required video memory for running the inference service is calculated, obtain the specification information of the existing GPUs in the cluster; and determine the number of different specification GPUs required for running the inference service;

[0061] save the calculated model information (such as model parameter quantity, model quantization precision, etc.).

[0062] The above provides an example of determining the required number of GPUs satisfying the minimum storage resource requirement in the case of storage resource being video memory resource.

[0063] Step 130, according to the storage resource state of the nodes in the inference cluster, determine a resource configuration strategy matching the resource specification, and configure storage resources in the inference cluster according to the resource configuration strategy to run the inference service.

[0064] In actual applications, the inference cluster usually has storage resources of multiple specifications (for example, the GPU memory of each specification of GPU is different), so the storage resources of the nodes in the inference cluster can be managed as a unified resource pool. The storage resource state can be considered to generally refer to the detailed information of the storage resources that each node in the cluster can provide, such as the total amount of available storage resources on each node, the model parameters of each storage device on each node, and the like, or the usage status of the storage resources of each node, such as whether it is idle, whether it has sufficient storage capacity, and the like.

[0065] The resource configuration strategy can be considered as a selection or configuration strategy for the storage resources in the inference cluster according to the resource specifications of the inference service and the storage resource states of the nodes in the cluster. For example, it can be determined whether to deploy the inference service in a single node or in a multi-node distributed manner, it can be determined to preferentially allocate the nodes and corresponding storage resources with lower loads (matching the resource specifications) when configuring the storage resources, to balance the overall load of the cluster, it can be determined to preferentially select the nodes with the same storage resource specifications when deploying the inference service in a multi-node distributed manner, to balance the performance of the inference service running on different nodes, and the like.

[0066] Since the storage resources of the nodes in the inference cluster can be simultaneously used for multiple resource specifications of the inference service, in some embodiments, the different resource specifications also need to be evaluated when determining the resource configuration strategy, such as the impact on the performance of the inference service, the feasibility on each node in the cluster, the scalability when the demand of the inference service grows, and the like.

[0067] After determining the resource configuration strategy, the storage resources are configured in the inference cluster according to the determined resource configuration strategy, to run the inference service. Correctly configuring the storage resources ensures that the inference service can obtain the required storage resources for running, which is the basis for running the inference service.

[0068] The inference service resource configuration method provided by the embodiments of the present application comprises the following steps: obtaining a model file of an inference service, and parsing the model file to determine minimum storage resource requirements required for running the inference service, thereby achieving evaluation of the minimum storage resource requirements required for running the inference service, providing a data basis for subsequent automatic resource configuration, and helping to improve the accuracy of resource configuration and avoid waste or shortage of storage resources; after the minimum storage resource requirements are determined, a resource specification satisfying the minimum storage resource requirements is determined, the resource specification representing the nodes required for running the inference service and the storage resource configuration corresponding to the nodes, thereby providing guidance for subsequent determination of a resource configuration strategy and resource configuration, and ensuring that the resources can meet the running requirements of the inference service; the storage resource state provides comprehensive information of the storage resources of each node for the resource allocation strategy, so that the resource allocation strategy can be determined according to the storage resource state of the nodes in the inference cluster, the resource allocation strategy being used to indicate how to configure the storage resources in the inference cluster to match the resource specification, and the storage resources are configured in the inference cluster according to the resource allocation strategy to run the inference service, thereby achieving automatic determination of the storage resources required for the inference service and automatic execution of the storage resource configuration, reducing the error risk and cost of manual operation, and also avoiding waste or shortage of resources caused by fragmentation and improving the utilization rate of storage resources.

[0069] In some embodiments of the present application, the determination of the resource configuration strategy matching the resource specification according to the storage resource state of the nodes in the inference cluster comprises:

[0070] According to whether the available storage resources of a single node in the inference cluster match the resource specification, it is determined whether to perform single-node deployment or multi-node distributed deployment on the inference service;

[0071] When the single-node deployment is performed, a target single node satisfying the minimum storage resource requirements is automatically selected in the inference cluster to configure the storage resources corresponding to the resource specification on the target single node;

[0072] When the multi-node distributed deployment is performed, a plurality of target nodes satisfying the distributed deployment conditions are automatically selected in the inference cluster to allocate the storage resources corresponding to the resource specification among the plurality of target nodes.

[0073] The available storage resources of a single node can be considered as detailed information of the storage resources that can be provided by a single node, for example, the total amount of available storage resources on the node, the model parameters of each storage device of each node, and the like, or can refer to the usage status of the storage resources of the node, such as whether it is idle or has sufficient storage capacity.

[0074] It can be understood that the single-node deployment refers to deploying the complete inference service on a single node, which is suitable for a scenario where the resource requirement of the inference service can be met by a single node and distributed processing is not required. The multi-node distributed deployment refers to distributing the components of the inference service or the inference service instance on multiple nodes, which is suitable for a scenario where the resource requirement of the inference service cannot be met by a single node or distributed processing is required.

[0075] In the embodiments of the present application, whether the available storage resource of a single node in the inference cluster matches the resource specification is determined to determine whether the inference service is deployed on a single node or distributed on multiple nodes. For example, the storage resource of each single node in the inference cluster can be checked one by one to determine whether it matches the resource specification. If the available storage resource of a single node matches the resource specification, the inference service can be deployed on the single node. The determination can be achieved by evaluating the available storage resource of the node by using a cluster management tool or by manual comparison.

[0076] If the storage resource of a single node in the inference cluster does not match the resource specification, multiple target nodes that meet the distributed deployment condition are selected, and the storage resource corresponding to the resource specification (i.e., the requirement of the node and the corresponding storage resource defined by the resource specification) is allocated among the nodes. For example, the node selection and resource classification can be completed by using the scheduling function of the cluster management tool, or the nodes can be determined by enumeration and traversal.

[0077] Generally, if the available storage resource of a single node matches the resource specification and multi-node distributed deployment is not required, the single-node deployment should be preferred. Specifically, the single-node deployment can simplify the deployment and subsequent management process of the inference service, does not require cross-node communication and data transmission, can simplify the architecture of the inference service and the storage resource, and has lower inference delay than the multi-node distributed deployment.

[0078] However, if the available storage resource of a single node does not match the resource specification, the multi-node distributed deployment can effectively disperse the load of the nodes, avoid overloading of the single node, and enhance the robustness of the inference cluster. In addition, compared with the single-node deployment, the multi-node distributed deployment can effectively improve the scalability of the inference service. When the access frequency or input data of the inference service increases, the storage resource configured for the inference service can be expanded on multiple nodes. In addition, when the inference service is distributed on multiple nodes, the nodes with the same storage resource specification are preferred, which helps to balance the performance of the inference service running on different nodes.

[0079] The inference service resource configuration method provided by the embodiments of the present application determines whether to perform single-node deployment or multi-node distributed deployment of the inference service according to whether the available storage resource of a single node in the inference cluster matches the resource specification, preferentially selects single-node deployment when the available storage resource of a single node matches the resource specification, so as to simplify the inference service architecture and reduce latency, and effectively disperses the load and improves the scalability of the inference service through multi-node distributed deployment when the available storage resource of a single node does not meet the resource specification; in single-node deployment, a node that meets the minimum storage requirement is automatically selected, and the corresponding storage resource is configured on the node, while in multi-node distributed deployment, multiple nodes that meet the condition are selected, and the required storage resource is allocated among the nodes, the inference scheme is deployed through the two modes of single-node deployment and multi-node distributed deployment, so as to make the inference service successfully deployed as much as possible, and the storage resource usage of the inference cluster is optimized.

[0080] In some embodiments of the present application, the storage resource is configured in the inference cluster according to the resource configuration strategy to run the inference service, comprising:

[0081] The storage resource corresponding to the resource specification is respectively configured on the target single node or the multiple target nodes;

[0082] After the storage resource configuration is completed, the inference service instance corresponding to the configured storage resource is started on the configured node to run the inference service;

[0083] In the case of multi-node distributed deployment, a communication link is established between different inference service instances to cooperatively run the inference service.

[0084] It can be understood that the target single node refers to a single node selected for running the inference service in the inference cluster, and the multiple target nodes refer to multiple nodes selected for distributed running of the inference service in the inference cluster.

[0085] The storage resource corresponding to the resource specification is configured on the target single node or the multiple target nodes, that is, the storage resource required for running the inference service according to the resource specification is configured on the storage resource of the target single node or the multiple target nodes (in the case of single-node deployment of the inference service, the storage resource of the target single node is configured, and in the case of multi-node distributed deployment of the inference service, the storage resource of the multiple target nodes is configured). Correctly configuring the storage resource is the basis for the inference service instance to run, so that part of the inference service can be executed.

[0086] A reasoning service instance (also referred to as a Worker instance) generally refers to an independent execution unit for processing a part of the reasoning service. It can be considered as a component of the reasoning service, specifically, it can be a process, a thread, or a container instance, depending on the architecture and implementation of the reasoning service (model).

[0087] Multiple reasoning service instances can work in parallel to improve the processing capacity and throughput of the reasoning service. In addition, by distributing tasks among multiple reasoning service instances, load balancing can be achieved to avoid overloading a single reasoning service instance. In some embodiments, a reasoning task is split into multiple sub-tasks, each of which is processed by a reasoning service instance. By increasing or decreasing the number of reasoning service instances, the processing capacity of the reasoning service can be expanded to adapt to different load requirements.

[0088] In the case of multi-node distributed deployment, communication links are established between different reasoning service instances to allow reasoning service instances in distributed deployment to share data and state. For example, the communication between reasoning service instances can also be simplified by using a service mesh.

[0089] The reasoning service resource configuration method provided by the embodiments of the present application configures the storage resources of the target single node in the case of single-node deployment of the reasoning service, and configures the storage resources of multiple target nodes in the case of multi-node distributed deployment of the reasoning service. By accurately configuring the storage resources, the reasoning service instances can obtain the required storage resources, improving the utilization efficiency of the storage resources. After completing the storage resource configuration, the reasoning service instances corresponding to the configured storage resources are started on the configured nodes to enable the reasoning service to operate normally. In the case of multi-node distributed deployment of the reasoning service, the communication links are established to enable the reasoning service instances in distributed deployment to work cooperatively. By parallel processing between different reasoning service instances, the reasoning service can process more requests, which helps to improve the overall processing capacity and throughput of the reasoning service.

[0090] In some embodiments of the present application, the resource configuration strategy matching the resource specification is determined according to the storage resource state of the nodes in the reasoning cluster, which further comprises:

[0091] In the case of separate deployment of the pre-population phase and the decoding phase of the reasoning service, a first target node set for running the pre-population phase and a second target node set for running the decoding phase are respectively determined according to the resource specification.

[0092] deploy an inference service instance of the prefill phase in the first target node set and deploy an inference service instance of the decoding phase in the second target node set to separate the deployment of the prefill phase and the decoding phase.

[0093] It can be understood that the PD disaggregation deployment divides the inference process into two main phases, namely, prefill and decoding, and deploys the two phases on different hardware resources to improve resource utilization efficiency and inference performance.

[0094] Among them, the prefill phase is mainly used to process the input tokens of the input sequence and store the intermediate calculation results for reuse in the subsequent decoding phase; the decoding phase generates output tokens one by one based on the intermediate calculation results constructed by the prefill phase, and the calculation load of this phase is relatively light.

[0095] The first target node set refers to a set of nodes selected to run the prefill phase. The second target node set refers to a set of nodes selected to run the decoding phase.

[0096] In some embodiments, the number of inference service instances of the prefill phase and the number of inference service instances of the decoding phase of the inference service are both one, and the corresponding first target node set and second target node set each include only one node.

[0097] It can be understood that if the first target node set includes only one node, the prefill phase of the inference service adopts single-node deployment; if the first target node set includes more than one node, the prefill phase of the inference service adopts multi-node distributed deployment.

[0098] Similarly, if the second target node set includes only one node, the decoding phase of the inference service adopts single-node deployment; if the second target node set includes more than one node, the decoding phase of the inference service adopts multi-node distributed deployment.

[0099] Among them, the related content of single-node deployment and multi-node distributed deployment can be referred to the foregoing embodiments.

[0100] According to the resource specifications, a first target node set for running the pre-filling stage and a second target node set for running the decoding stage are determined respectively. Specifically, the pre-filling stage of the inference service is computationally intensive, and the decoding stage of the inference service is video memory intensive. Therefore, in general, high-specification storage resources (such as GPUs) are used to deploy inference service instances of the pre-filling stage, and low-specification storage resources (such as GPUs) are used to deploy inference service instances of the decoding stage. For example, the detailed parameter information of the storage resources of each node can be obtained by scanning through an automated tool or script, and then the first target node set and the second target node set are determined. By assigning tasks to the most suitable nodes, the pre-filling and decoding stages of the inference service are run on the nodes that are most suitable for their resource requirements.

[0101] After determining the first target node set and the second target node set, the inference service instances of the pre-filling stage are deployed on the first target node set, and the inference service instances of the decoding stage are deployed on the second target node set. For example, the automatic deployment and management of the inference service instances can be achieved through an orchestration tool, which realizes the separate deployment of the pre-filling and decoding stages, so that each stage can run efficiently on suitable resources.

[0102] Figure 3 Flowchart III of the inference service resource configuration method provided by some embodiments of the present application. As shown in FIG. 3, in some embodiments, the inference service resource configuration method further comprises: Figure 3

[0103] After starting the inference service deployment, it is determined whether the inference service needs PD separate deployment.

[0104] In the case where the inference service adopts PD separate deployment, it is determined whether there is a deployment template (also referred to as a historical running template) corresponding to the inference service;

[0105] If there is a corresponding deployment template, the deployment template is used for deployment, and if there is not, the number of inference service instances of the pre-filling stage and the number of inference service instances of the decoding stage of the inference service are both set to one.

[0106] It is determined whether the cluster resources are sufficient. In the case where the cluster resources are sufficient, the inference service can be run, and in the case where the cluster resources are insufficient, the inference service waits for idle resources.

[0107] In the case where the inference service does not adopt PD separate deployment, the resource specification list corresponding to the model inference service is obtained, and the cluster idle resource information is obtained.

[0108] ​Determine whether the inference service is single-node running, if yes, the inference service can be run under the condition that the cluster resource is sufficient, if not, the inference service can be multi-node distributed running under the condition that the cluster resource is sufficient.

[0109] The above flow provides an example of the deployment process of the inference service, including the judgment of PD separation deployment, the evaluation of cluster resources, and the final deployment decision, etc.

[0110] The inference service resource configuration method provided by the embodiments of the present application, in the case of separate deployment of the pre-population phase and the decoding phase of the inference service, determines the first target node set for running the pre-population phase and the second target node set for running the decoding phase according to the resource specifications, preferentially uses high-specification storage resources for deploying the inference service instances of the pre-population phase, and uses low-specification storage resources for deploying the inference service instances of the decoding phase, so as to match the resource specifications and the storage devices corresponding to each node, make both phases of the inference service run in a suitable environment, deploy the inference service instances of the pre-population phase in the first target node set, and deploy the inference service instances of the decoding phase in the second target node set, realize the separate deployment of the pre-population phase and the decoding phase, optimize the utilization of storage resources, and help improve the running efficiency and reliability of the inference service.

[0111] In some embodiments of the present application, the method further comprises:

[0112] In the case that the historical running template of the inference service exists, determining the first sub-resource specification and the second sub-resource specification based on the historical running template;

[0113] Based on the historical running template, determining the number of inference service instances of the inference service in the pre-population phase and the number of inference service instances of the inference service in the decoding phase;

[0114] Based on the first sub-resource specification, configuring the storage resources of the pre-population phase to deploy the inference service instances of the inference service in the pre-population phase;

[0115] Based on the second sub-resource specification, configuring the storage resources of the decoding phase to deploy the inference service instances of the inference service in the decoding phase.

[0116] The historical running template can be considered as the performance and resource usage record of the inference service in the past inference cluster deployment and running. In the embodiments of the present application, it can be used to guide the current resource configuration, which includes the first sub-resource specification, the second sub-resource specification, the number of inference service instances of the inference service in the pre-population phase, and the number of inference service instances of the inference service in the decoding phase.

[0117] It can be understood that the first sub-resource specification is used to represent the nodes required for running the pre-filling stage of the inference service and the storage resource configuration corresponding to the nodes. Specifically, the first sub-resource specification is used to guide how to configure the storage resource in the inference cluster to meet the storage requirement of the pre-filling stage of the inference service.

[0118] The second sub-resource specification is used to represent the nodes required for running the decoding stage of the inference service and the storage resource configuration corresponding to the nodes. Specifically, the second sub-resource specification is used to guide how to configure the storage resource in the inference cluster to meet the storage requirement of the decoding stage of the inference service.

[0119] In some embodiments, the first sub-resource specification or the second sub-resource specification can further include the requirement for computing resource, storage resource, memory, network bandwidth, etc.

[0120] For the inference service, there can be multiple first sub-resource specifications or second sub-resource specifications. In some embodiments, the first sub-resource specification and the second sub-resource specification are equal, and each needs to meet the minimum storage resource requirement for running the complete inference service.

[0121] In some embodiments, the inference service instance of the pre-filling stage or the inference service instance of the decoding stage can also be dynamically adjusted in response to real-time load changes.

[0122] In some embodiments, the method further comprises:

[0123] obtaining the identification of the target inference service;

[0124] generating a separate running template of the target inference service based on the number of task execution services of the inference service in the pre-filling stage, the number of task execution services of the inference service in the decoding stage, the first resource specification, and the second resource specification;

[0125] constructing a mapping relationship between the identification and the separate running template.

[0126] The inference service resource configuration method provided in the embodiments of the present application, in the case that there is a historical running template of the inference service, determines a first sub-resource specification and a second sub-resource specification based on the historical running template. The historical data provides an accurate reference for actual runtime resource usage, which helps to more accurately determine the resource requirements of different stages of the inference service, so as to make more reasonable resource configuration. The number of inference service instances of the inference service in the pre-population stage and the number of inference service instances of the inference service in the decoding stage are determined based on the historical running template. The number of instances directly affects the processing capacity and resource consumption of the inference service. According to the historical data, the number of inference service instances is determined, which can ensure that the service meets the performance requirements and avoids resource waste. Based on the historical running template, automatic configuration of storage resources and deployment of instances is realized, manual intervention in the resource configuration process is reduced, and the resource configuration efficiency and accuracy are improved.

[0127] In some embodiments of the present application, the method further comprises:

[0128] When the inference cluster contains heterogeneous nodes, based on the node performance parameters of each node, a candidate node set meeting the resource specification is screened out;

[0129] Based on the running requirements of the inference service, a target node that can carry the inference service is determined from the candidate node set;

[0130] According to the resource configuration strategy, storage resources are configured on the target node, and a corresponding inference service instance is started.

[0131] It can be understood that in the inference cluster, nodes can have different hardware configurations, such as different models of GPUs or different capacities of memories, i.e., heterogeneous nodes. When allocating resources, the heterogeneity of the nodes needs to be considered to ensure that the resource configuration matches the node capability.

[0132] The node performance parameter refers to a parameter describing the performance characteristics of the node, such as computing power, video memory capacity, and storage bandwidth, which can be used to evaluate whether the node meets the resource requirements of a specific inference service.

[0133] The candidate node set can be considered as a set of nodes meeting the resource specification requirements. These nodes are potential candidates for running the inference service. In the embodiments of the present application, the most suitable node is selected from the candidate node set based on the running requirements of the inference service to deploy the inference service. For example, under the premise that the candidate node can carry the inference service, a node with a higher storage resource specification can be preferentially selected as the target node, or a node with the same storage device specification can be preferentially selected as the target node. The present application does not make specific limitations on how to determine the target node from the candidate node set.

[0134] After determining the target node, the storage resource is configured on the target node according to the resource configuration strategy, so as to run the inference service. Correctly configuring the storage resource ensures that the inference service can obtain the required storage resource for running, which is the basis for the running of the inference service.

[0135] The inference service resource configuration method provided in the embodiments of the present application can ensure that the nodes in the candidate node set can meet the resource requirements of the inference service, determine the target node that can carry the inference service from the candidate node set based on the running requirements of the inference service, and select the most suitable node for deployment, so that the utilization of the storage resource can be optimized. The storage resource is configured on the target node, and the inference service instance is started, so that the service can obtain the required resources and run smoothly, thereby improving the reliability and performance of the service.

[0136] In some embodiments of the present application, the method further comprises:

[0137] In the running process of the inference service, the inference performance parameter of the inference service is collected;

[0138] Based on the comparison result of the inference performance parameter and the preset threshold value, the expansion operation of the storage resource is automatically triggered to optimize the inference performance of the inference service.

[0139] It can be understood that the inference performance parameter refers to a key indicator for measuring the inference performance in the running process of the inference service, such as delay, throughput, accuracy, etc. The preset threshold value refers to the target value or upper and lower limit threshold value set for the inference performance parameter, which is used to compare with the actual inference performance parameter to determine whether resource adjustment is needed.

[0140] In the embodiments of the present application, in the running process of the inference service, the inference performance parameter of the inference service is collected. For example, the inference performance parameter of the inference service can be monitored and recorded in real time by a monitoring tool. After collecting the inference performance parameter, the collected performance parameter is compared with the preset threshold value, and whether to trigger the expansion operation of the storage resource is determined based on the comparison result. Through comparison, the performance deficiency or excess can be identified, so as to determine whether expansion is needed. For example, in the case that the inference delay exceeds the preset threshold value, or in the case that the service throughput exceeds the preset threshold value, it can be determined that the expansion operation of the storage resource is triggered. Automatic expansion can quickly respond to load changes, and avoids the delay and inaccuracy caused by manual adjustment.

[0141] The inference service resource configuration method provided by the embodiments of the present application can collect inference performance parameters of the inference service in the running process of the inference service, and through real-time monitoring of the inference performance parameters, the running state of the inference service can be known in time, and a basis for resource adjustment is provided. Based on the comparison result of the inference performance parameters and the preset threshold value, whether resource adjustment is needed is accurately judged through comparison with the preset threshold value, the expansion operation of the storage resource is automatically triggered, manual intervention in the expansion process is reduced, and the resource allocation is more flexible and dynamic, so that the inference performance of the inference service is flexibly optimized, and the inference service can adapt to the changing load demand.

[0142] In some embodiments of the present application, the inference performance parameters at least include time to first token and time per output token; and the automatic triggering of the expansion operation of the storage resource based on the comparison result of the inference performance parameters and the preset threshold value comprises:

[0143] In the case that the time to first token exceeds the time to first token standard value, the number of inference service instances deployed by the inference service in the pre-filling stage is increased to expand the inference service;

[0144] In the case that the time per output token exceeds the time per output token standard value, the number of inference service instances deployed by the inference service in the decoding stage is increased to expand the inference service.

[0145] It should be noted that the time to first token (TTFT), that is, the maximum delay value of the first token generation, refers to the time for the inference service to return the first token response. This index is used to measure the response speed of the model inference service. Shorter time to first token indicates that the inference service can quickly respond to requests.

[0146] The time per output token (TPOT), that is, the average time for generating each output token. This index is used to measure the speed of the model in generating output, that is, the average time interval from generating a token to generating the next token. A lower TPOT value indicates that the model can quickly generate output.

[0147] It can be understood that TTFT mainly measures the initial response speed of the inference service to requests, while TPOT measures the continuous efficiency of the inference service in the output generation process. The combination of the two can comprehensively evaluate the inference performance of the model.

[0148] It can be understood that more inference requests can be processed in parallel and the throughput, i.e., the number of inference tasks completed per unit of time, can be increased by increasing more inference service instances. As the number of instances increases, the inference service can respond to new inference requests more quickly, reducing the waiting time of users. The pre-filling stage is mainly used to process input data and perform preliminary calculations to prepare for generating an output sequence. The time to first token (TTFT) is a key performance indicator of the pre-filling stage, which measures the speed of generating the first token. The decoding stage is mainly used to continue generating the remaining part of the sequence based on the output of the pre-filling stage. The time per token (TPOT) is a key performance indicator of the decoding stage, which measures the average speed of generating each subsequent token.

[0149] A specific example is given below:

[0150] Monitoring the real-time load of the inference service instances of the inference service in the pre-filling stage and the real-time load of the inference service instances of the inference service in the decoding stage to obtain the current time to first token and the current time per token ,

[0151] respectively and the time to first token standard value are compared, respectively and the time per token standard value standard value are compared, and scaling is performed according to the following rules:

[0152] In the case of , the number of inference service instances of the inference service in the pre-filling stage is increased;

[0153] In the case of , the number of inference service instances of the inference service in the decoding stage is increased.

[0154] The inference service resource configuration method provided by the embodiments of the present application, the inference performance parameters mainly include the time to first token and the time per token. In the pre-filling stage of the inference service, input data is mainly processed and preliminary calculations are performed. By increasing the number of inference service instances in the pre-filling stage, the speed of generating the first token can be improved, thereby shortening the response time. In the decoding stage of the inference service, the remaining part of the sequence is mainly generated based on the output of the pre-filling stage. By increasing the number of inference service instances in the decoding stage, the average speed of generating each subsequent token can be improved, thereby improving the overall inference efficiency of the inference service.

[0155] In some embodiments of the present application, in the case where there are multiple inference services, the method further comprises:

[0156] obtain model files of the plurality of inference services, and parse the model files respectively to determine minimum storage resource requirements required for running the plurality of inference services respectively;

[0157] determine one or more candidate resource specifications corresponding to any inference service respectively based on storage resource states of nodes in the inference cluster;

[0158] generate a plurality of global resource configuration strategies based on the candidate resource specifications corresponding to the plurality of inference services respectively;

[0159] select a target global resource configuration strategy capable of running the most inference services from the plurality of global resource configuration strategies;

[0160] configure storage resources according to the target global resource configuration strategy to run the plurality of inference services.

[0161] In an embodiment of the present application, a plurality of different inference services need to be run in the same inference cluster.

[0162] The global resource configuration strategy is a selection or configuration strategy for storage resources in the inference cluster according to resource specifications of the plurality of inference services and storage resource states of nodes in the cluster. It can be considered as a resource configuration scheme generated for resource requirements of the plurality of inference services to be run, to optimize overall resource utilization and enable as many inference services as possible to run.

[0163] In some embodiments, the obtaining of the model files of the plurality of inference services and the parsing of the model files to determine the minimum storage resource requirements required for running the plurality of inference services respectively comprises:

[0164] For each inference service in the plurality of inference services, a model file of the inference service is obtained, and the model file is parsed to determine the minimum storage resource requirements required for running the inference service.

[0165] In some embodiments, the determination of one or more candidate resource specifications corresponding to any inference service respectively based on storage resource states of nodes in the inference cluster comprises:

[0166] For each inference service in the plurality of inference services, a resource specification satisfying the minimum storage resource requirements is determined, and the resource specification is used to represent a node required for running the inference service and a storage resource configuration corresponding to the node.

[0167] It can be understood that for each inference service in the plurality of inference services, a minimum storage resource requirement is determined separately, and one or more corresponding candidate resource specifications are determined separately. The candidate resource specification refers to one or more possible resource configuration options determined for each inference service, which needs to be combined with the candidate resource specifications corresponding to other inference services to generate a global resource configuration strategy.

[0168] Ideally, the global resource configuration strategy should be able to include the respective candidate resource specifications corresponding to each inference service, that is, each inference service can be run based on the global resource configuration strategy. However, due to the limited storage resources of nodes in the inference cluster and other factors, the global resource configuration strategy cannot usually include the respective candidate resource specifications corresponding to each inference service, so the target global resource configuration strategy that can run the most inference services is used to run as many inference services as possible.

[0169] It can be understood that for each inference service included in the global resource configuration strategy and the respective candidate resource specification corresponding to each inference service, the inference service resource configuration can be independently performed based on the resource specification.

[0170] In some embodiments, the method further comprises:

[0171] For each inference service that has a corresponding resource specification in the global resource configuration strategy, a resource configuration strategy matching the resource specification is determined based on the storage resource state of the nodes in the inference cluster, and the storage resources are configured in the inference cluster according to the resource configuration strategy to run the inference service.

[0172] Figure 4 A flowchart of a fourth inference service resource configuration method provided by some embodiments of the present application is shown in FIG. 4. As shown in FIG. 4, in some embodiments, the inference service resource configuration method further comprises: Figure 4

[0173] In the case where there are a plurality of inference services to be scheduled, the cluster idle GPU topology is obtained;

[0174] Based on the creation time of each inference service, the inference services are sorted in ascending order to obtain a list of inference services to be scheduled;

[0175] Each inference service in the list of inference services to be scheduled is traversed, and it is determined whether all inference services have been completed. If the current inference service deployment scheme (corresponding to the global resource configuration strategy described above) is recorded in the case of completion, and the current inference service is deployed in the case of non-completion;

[0176] ​obtaining a resource specification list of a current inference service (consisting of one or more resource specifications corresponding to the inference service), traversing each resource specification in the resource specification list and performing cluster resource application (the cluster resource application can be considered as a process of determining whether the cluster resource meets the resource specification);

[0177] In the case that the cluster resource application corresponding to the current resource specification is successful, performing cluster resource application based on a next resource specification of the current resource specification, and taking a next inference service of the current inference service as the current inference service, returning to the step of determining whether all inference services have been traversed, in the case of completion, recording the current inference service deployment scheme (corresponding to the global resource configuration strategy in the foregoing), and in the case of non-completion, attempting to deploy the current inference service for continuous execution;

[0178] In the case that the cluster resource application corresponding to the current resource specification is not successful, determining whether each resource specification in the resource specification list has been traversed, in the case of completion, terminating and recording the current inference service deployment scheme in the case that there is no cluster resource application corresponding to the resource specification, and in the case of non-completion, performing cluster resource application based on a next resource specification of the current resource specification;

[0179] sorting the inference service deployment schemes, and obtaining a scheme with the largest number of deployed inference services.

[0180] The flowchart provides an example of how to determine an inference service deployment scheme (corresponding to the global resource configuration strategy in the foregoing) to apply for resources for multiple inference services and ensure that as many services as possible can run, including steps such as service sorting, resource application, deployment attempt, and final resource configuration decision.

[0181] The inference service resource configuration method provided by the embodiment of the application obtains and parses model files of multiple inference services to determine minimum storage resource requirements required for running each service, determines one or more candidate resource specifications for each inference service based on storage resource states of nodes in an inference cluster, analyzes resource requirements of each inference service to improve the pertinence and accuracy of resource configuration, which helps to improve resource utilization efficiency, combines multiple global resource configuration strategies based on the candidate resource specifications corresponding to the multiple inference services respectively, and further screens a target global resource configuration strategy capable of running the most inference services from the multiple global resource configuration strategies, and the screening process aims to ensure the global optimality of resource configuration, and preferentially considers a configuration scheme capable of supporting the most services, thereby improving service capability of the cluster, and realizing that as many inference services as possible can run smoothly in the case of resource limitation.

[0182] In some embodiments of the present application, the global resource configuration strategy is generated by combining the candidate resource specifications of the plurality of inference services, including:

[0183] For any candidate resource specification of the inference service, it is determined whether the available storage resource of the node in the inference cluster meets the current candidate resource specification;

[0184] If there is a node that meets the current candidate resource specification, the current candidate resource specification is added to the current global resource configuration strategy;

[0185] The next inference service of the current inference service is taken as a new current inference service, and the step of determining whether the available storage resource of the node in the inference cluster meets the current candidate resource specification for any candidate resource specification of the inference service is continued to be executed until the current global resource configuration strategy contains one candidate resource specification corresponding to each of the plurality of inference services, and for each inference service contained in the strategy, a resource configuration strategy matched therewith is determined according to the storage resource state of the node in the cluster, and the storage resource is configured in the cluster to run the service.

[0186] In the embodiments of the present application, it is determined whether the available storage resource of each node in the cluster meets the candidate resource specification of the inference service currently targeted, which may be, for example, scanned by using an automatic tool and automatically matched with the candidate resource specification. If the available storage resource of the node meets the candidate resource specification, the candidate resource specification is added to the current global resource configuration strategy, and the next inference service is taken as the current service, and the resource matching and configuration steps are repeated.

[0187] A specific example is given below:

[0188] Suppose there are M inference services in total , and each inference service has N corresponding resource specifications (Actually, the resource specifications corresponding to each inference service need to be confirmed according to the actual scene);

[0189] For any candidate resource specification of the inference service , it is determined whether the available storage resource of the node in the inference cluster meets , wherein represents the ith resource specification corresponding to the inference service .

[0190] If it is met, the candidate resource specification is added to the current global resource configuration strategy ​and returns to the step of judging whether the available storage resources of the nodes in the inference cluster satisfy the current candidate resource specification of any of the inference services, and the step continues to be executed until the current global resource configuration strategy comprises one candidate resource specification corresponding to each of the inference services.

[0191] That is, after the inference services candidate resource specification is added to the current global resource configuration strategy , the candidate resource specification of the inference service is judged whether the available storage resources of the nodes in the inference cluster satisfy , if the current candidate resource specification is added to the current global resource configuration strategy , and the above steps are repeatedly executed until the current global resource configuration strategy comprises one candidate resource specification corresponding to each of the inference services.

[0192] The inference service resource configuration method provided by the embodiments of the present application judges whether the available storage resources of the nodes in the cluster satisfy the candidate resource specifications of each inference service, if there is a node satisfying the current candidate resource specification, the current candidate resource specification is added to the current global resource configuration strategy, then the next inference service is processed, and the process is repeated until the candidate resource specifications of all inference services are considered and added to the global resource configuration strategy, by determining the most suitable resource specification for each inference service, as many inference services as possible are implemented, which helps to improve the utilization rate of storage resources.

[0193] In some embodiments of the present application, the method further comprises:

[0194] If there is no node satisfying the current candidate resource specification, taking the current global resource configuration strategy as a global resource configuration strategy, generating a new global resource configuration strategy based on the current global resource configuration strategy and taking the new global resource configuration strategy as the current global resource configuration strategy;

[0195] taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resources of the nodes in the inference cluster satisfy the current candidate resource specification of any of the inference services.

[0196] In this embodiment, it is determined whether the available storage resources of each node in the cluster meet the candidate resource specifications for the current inference service. For example, this can be achieved by using automated tools to scan cluster resources and automatically match them with candidate resource specifications. If the available storage resources of a node do not meet the candidate resource specifications, the generation of the current global resource configuration policy is stopped (i.e., the current global resource configuration policy has already been used as a global resource configuration policy for subsequent filtering). A new global resource configuration policy is generated based on the current global resource configuration policy and used as the current global resource configuration policy. Specifically, this can be achieved by copying the current global resource configuration policy to generate the new global resource configuration policy. The next inference service is then used as the current service, and the resource matching and configuration steps are repeated.

[0197] Here is a specific example:

[0198] There are still M inference services. And set up various reasoning services There are N corresponding resource specifications. (In practical applications, the resource specifications for each inference service need to be confirmed based on the actual scenario.)

[0199] For any reasoning service Candidate resource specifications Determine whether the available storage resources of the nodes in the inference cluster meet the requirements. ,in, Inference service The corresponding i-th resource specification;

[0200] If the conditions are not met, stop generating the current global resource configuration strategy. (that is, at this time) (No longer generating), based on the current global resource configuration strategy. Generate a new global resource configuration strategy And it will be used as the current global resource configuration strategy;

[0201] Current reasoning service The next reasoning service As the new current inference service, and returning to the candidate resource specification for any of the inference services, the step of determining whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification continues.

[0202] That is to say, for Determine whether the available storage resources of the nodes in the inference cluster meet the requirements. And determine the subsequent execution steps based on whether the conditions are met.

[0203] The inference service resource configuration method provided in the embodiments of the present application is used to determine whether the available storage resources of the nodes in the cluster meet the candidate resource specifications of each inference service, if there is no node meeting the current candidate resource specification, the current global resource configuration strategy is stopped from being generated, a new global resource configuration strategy is generated based on the current global resource configuration strategy, the new global resource configuration strategy is taken as the current strategy for continuous execution, the next inference service is taken as the current service, and the resource matching and configuration steps are repeatedly executed, which helps to realize that even in the case of resource limitation, as many inference services as possible can be run, thereby improving the resource utilization efficiency and the service capability of the system.

[0204] In some embodiments of the present application, the method further comprises:

[0205] If there is a node meeting the current candidate resource specification, a new global resource configuration strategy is generated based on the current global resource configuration strategy and taken as the current global resource configuration strategy.

[0206] A next candidate resource specification of the current candidate resource specification is determined.

[0207] It is determined whether the available storage resources of the nodes in the inference cluster meet the next candidate resource specification.

[0208] If there is a node meeting the next candidate resource specification, the next candidate resource specification is added to the current global resource configuration strategy.

[0209] If there is no node meeting the next candidate resource specification, the current global resource configuration strategy is stopped from being generated.

[0210] A new global resource configuration strategy is generated based on the current global resource configuration strategy, the next inference service of the current inference service is taken as a new current inference service, and the step of determining whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification for any candidate resource specification of the inference service is executed.

[0211] It can be understood that, in the case where there is a node meeting the current candidate resource specification, a new global resource configuration strategy is also generated based on the current global resource configuration strategy and taken as the current global resource configuration strategy, and it is determined whether there is a node meeting the next candidate resource specification of the current candidate resource specification, if there is a node meeting the next candidate resource specification, the next candidate resource specification is added to the new current global resource configuration strategy, if there is no node meeting the next candidate resource specification, the new current global resource configuration strategy is stopped from being generated (that is, the new current global resource configuration strategy has been taken as a global resource configuration strategy for subsequent screening).

[0212] A specific example is given below:

[0213] Still set the common M inference services , and set each inference service There are N corresponding resource specifications (In practical applications, the resource specifications corresponding to each inference service need to be confirmed according to the actual scene);

[0214] For any inference service The candidate resource specification , judge whether the available storage resources of the nodes in the inference cluster meet , wherein Indicates the i-th resource specification corresponding to the inference service ;

[0215] If it is satisfied, generate a new global resource configuration strategy Based on the current global resource configuration strategy And as the current global resource configuration strategy;

[0216] Determine the next candidate resource specification of the current candidate resource specification ;

[0217] Judge whether the available storage resources of the nodes in the inference cluster meet the next candidate resource specification ;

[0218] If there is a node that meets the next candidate resource specification , add the next candidate resource specification To the current global resource configuration strategy ;

[0219] If there is no node that meets the next candidate resource specification , stop generating the current global resource configuration strategy (I.e. at this time No longer continue to generate);

[0220] Generate a new global resource configuration strategy Based on the current global resource configuration strategy , and the next inference service of the current inference service As the new current inference service, and return to the step of judging whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification for any of the candidate resource specifications of the inference service. Continue to execute.

[0221] That is, for Judge whether the available storage resources of the nodes in the inference cluster meet , and determine the subsequent steps according to whether it is satisfied.

[0222] The inference service resource configuration method provided in this application, when a node satisfying the current candidate resource specification exists, generates a new global resource configuration policy based on the current global resource configuration policy and uses it as the current policy. Furthermore, it determines the next candidate resource specification and checks whether a node in the cluster satisfies the next candidate resource specification. If a node satisfying the next candidate resource specification exists, that candidate resource specification is added to the current global resource configuration policy; if not, it stops generating the current global resource configuration policy and generates a new global resource configuration policy based on the current policy, continuing to process the next inference service. This iterative approach allows for the judgment of each candidate resource specification for each inference service, helping to run as many inference services as possible and thus improving resource utilization efficiency.

[0223] In some embodiments of this application, the method further includes:

[0224] If no node meets the current candidate resource specifications, stop generating the current global resource configuration strategy;

[0225] A new global resource configuration policy is generated based on the current global resource configuration policy, and this new policy is used as the current global resource configuration policy.

[0226] The next inference service of the current inference service is taken as the new current inference service, and the process of returning to the candidate resource specification for any of the inference services and determining whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification continues.

[0227] Here is a specific example:

[0228] There are still M inference services. And set up various reasoning services There are N corresponding resource specifications. (In practical applications, the resource specifications for each inference service need to be confirmed based on the actual scenario.)

[0229] For any reasoning service Candidate resource specifications Determine whether the available storage resources of the nodes in the inference cluster meet the requirements. ;

[0230] If the conditions are not met, stop generating the current global resource configuration strategy. (that is, at this time) (It has stopped generating).

[0231] Based on the current global resource allocation strategy Generate a new global resource configuration strategy and as the current global resource configuration strategy;

[0232] the current inference service the next inference service as the new current inference service, and returns to the step of judging whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification for any of the inference services, and the step continues to be executed.

[0233] that is, for judging whether the available storage resources of the nodes in the inference cluster meet and determining the subsequent execution step according to whether the condition is met.

[0234] The inference service resource configuration method provided by the embodiments of the present application stops generating the current global resource configuration strategy when it is determined that there is no node meeting the current candidate resource specification. Subsequently, a new global resource configuration strategy is generated based on the existing global resource configuration strategy, and is set as the current strategy. Then, the next inference service is taken as the new current inference service, and the resource specification judgment step is returned to, and the resource matching process continues to be executed, and each inference service is processed iteratively to determine the most suitable resource configuration for each inference service, which helps to implement as many running inference services as possible, thereby improving the utilization efficiency of resources.

[0235] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0236] Figure 5 The schematic diagram of the computer program product provided by some embodiments of the present application. As shown in Figure 5 the present application also provides a computer program product 500, which comprises:

[0237] The first processing module 501 is configured to acquire a model file of an inference service, and parse the model file to determine the minimum storage resource requirement required for running the inference service;

[0238] The second processing module 502 is configured to determine a resource specification meeting the minimum storage resource requirement, the resource specification being used to represent a node required for running the inference service and a storage resource configuration corresponding to the node;

[0239] The third processing module 503 is configured to determine a resource configuration strategy matching the resource specification according to the storage resource state of the nodes in the inference cluster, and configure storage resources in the inference cluster according to the resource configuration strategy to run the inference service.

[0240] In some embodiments, the third processing module 503 is configured to:

[0241] determine whether to perform single-node deployment or multi-node distributed deployment of the inference service according to whether the available storage resource of a single node in the inference cluster matches the resource specification;

[0242] wherein, when performing single-node deployment, a target single node satisfying the minimum storage resource requirement is automatically selected in the inference cluster to configure the storage resource corresponding to the resource specification on the target single node;

[0243] when performing multi-node distributed deployment, a plurality of target nodes satisfying the distributed deployment condition are automatically selected in the inference cluster to allocate the storage resource corresponding to the resource specification among the plurality of target nodes.

[0244] In some embodiments, the third processing module 503 is configured to:

[0245] configure the storage resource corresponding to the resource specification on the target single node or the plurality of target nodes, respectively;

[0246] after completing the storage resource configuration, start an inference service instance corresponding to the configured storage resource on the configured node to run the inference service;

[0247] wherein, in the case of multi-node distributed deployment, a communication link is established between different inference service instances to cooperatively run the inference service.

[0248] In some embodiments, the third processing module 503 is further configured to:

[0249] in the case of separate deployment of the pre-population phase and the decoding phase of the inference service, determine a first set of target nodes for running the pre-population phase and a second set of target nodes for running the decoding phase according to the resource specification, respectively;

[0250] deploy an inference service instance of the pre-population phase in the first set of target nodes and deploy an inference service instance of the decoding phase in the second set of target nodes to separately deploy the pre-population phase and the decoding phase.

[0251] In some embodiments, the computer program product 500 further comprises a fourth processing module configured to:

[0252] in the case where there is a historical running template of the inference service, determine a first sub-resource specification and a second sub-resource specification based on the historical running template;

[0253] determine, based on the historical running template, a number of inference service instances of the inference service in a pre-population stage and a number of inference service instances of the inference service in a decoding stage;

[0254] configure storage resources of the pre-population stage based on the first sub-resource specification to deploy the inference service instances of the inference service in the pre-population stage;

[0255] configure storage resources of the decoding stage based on the second sub-resource specification to deploy the inference service instances of the inference service in the decoding stage.

[0256] In some embodiments, the computer program product 500 further comprises a fifth processing module configured to:

[0257] when the inference cluster comprises heterogeneous nodes, filter a candidate node set satisfying the resource specification based on node performance parameters of the nodes;

[0258] determine, based on running requirements of the inference service, a target node from the candidate node set that can carry the inference service;

[0259] configure storage resources on the target node according to the resource configuration strategy and start corresponding inference service instances.

[0260] In some embodiments, the computer program product 500 further comprises a sixth processing module configured to:

[0261] collect inference performance parameters of the inference service during running of the inference service;

[0262] based on a comparison result of the inference performance parameters and a preset threshold, automatically trigger an expansion operation of the storage resources to optimize inference performance of the inference service.

[0263] In some embodiments, the inference performance parameters at least include a first word element time and a per word element time; and the automatically triggering the expansion operation of the storage resources based on the comparison result of the inference performance parameters and the preset threshold comprises:

[0264] when the first word element time exceeds a first word element time standard value, increasing a number of inference service instances of the inference service in the pre-population stage to expand the inference service;

[0265] when the per word element time exceeds a per word element time standard value, increasing a number of inference service instances of the inference service in the decoding stage to expand the inference service.

[0266] In some embodiments, when there are multiple inference services, the computer program product 500 further comprises a seventh processing module configured to:

[0267] read model files of the multiple inference services, and parse the model files respectively to determine minimum storage resource requirements for running the multiple inference services respectively;

[0268] determine one or more candidate resource specifications corresponding to any inference service respectively based on storage resource states of nodes in the inference cluster;

[0269] generate multiple global resource configuration strategies based on the candidate resource specifications corresponding to the multiple inference services respectively;

[0270] select a target global resource configuration strategy from the multiple global resource configuration strategies, which is capable of running the most inference services;

[0271] configure storage resources according to the target global resource configuration strategy to run the multiple inference services.

[0272] In some embodiments, the generating multiple global resource configuration strategies based on the candidate resource specifications corresponding to the multiple inference services respectively comprises:

[0273] for any candidate resource specification of the inference services, determine whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification;

[0274] if there is a node meeting the current candidate resource specification, add the current candidate resource specification to the current global resource configuration strategy;

[0275] take the next inference service of the current inference service as a new current inference service, and return to the step of determining whether the available storage resources of the nodes in the inference cluster meet the current candidate resource specification, and continue to execute until the current global resource configuration strategy contains one candidate resource specification corresponding to each of the multiple inference services.

[0276] In some embodiments, the computer program product 500 further comprises an eighth processing module configured to:

[0277] if there is a node meeting the current candidate resource specification, generate a new global resource configuration strategy based on the current global resource configuration strategy, and take it as the current global resource configuration strategy;

[0278] determine a next candidate resource specification of the current candidate resource specification;

[0279] determine whether the available storage resources of the nodes in the inference cluster meet the next candidate resource specification;

[0280] if there is a node satisfying the next candidate resource specification, adding the next candidate resource specification to the current global resource configuration strategy;

[0281] if there is no node satisfying the next candidate resource specification, stopping generating the current global resource configuration strategy;

[0282] generating a new global resource configuration strategy based on the current global resource configuration strategy, taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resource of the node in the inference cluster satisfies the current candidate resource specification for any of the inference services.

[0283] In some embodiments, the eighth processing module is further configured to:

[0284] if there is no node satisfying the current candidate resource specification, stopping generating the current global resource configuration strategy;

[0285] generating a new global resource configuration strategy based on the current global resource configuration strategy, and taking the new global resource configuration strategy as the current global resource configuration strategy;

[0286] taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resource of the node in the inference cluster satisfies the current candidate resource specification for any of the inference services.

[0287] In some embodiments, the eighth processing module is further configured to:

[0288] if there is no node satisfying the current candidate resource specification, stopping generating the current global resource configuration strategy, generating a new global resource configuration strategy based on the current global resource configuration strategy, and taking the new global resource configuration strategy as the current global resource configuration strategy;

[0289] taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resource of the node in the inference cluster satisfies the current candidate resource specification for any of the inference services.

[0290] The descriptions of the features in the embodiments of the computer program product can refer to the related descriptions of the embodiments of the inference service resource configuration method, which will not be repeated here.

[0291] Embodiments of the present application also provide an electronic device, including a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the inference service resource configuration method embodiments.

[0292] Embodiments of the present application also provide a non-volatile computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above inference service resource configuration method embodiments when running.

[0293] In an example embodiment, the above non-volatile computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0294] Embodiments of the present application also provide another computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in any of the above inference service resource configuration method embodiments.

[0295] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0296] The above provides a detailed description of the inference service resource configuration method, electronic device and readable storage medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the examples is only applicable to help understand the method and core idea of the present application. It should be noted that for ordinary skilled persons in the technical field, without departing from the principles of the present application, the present application can be improved and modified in several ways. These improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for allocating inference service resources, characterized in that, The method comprises the following steps: In the presence of an inference service, obtaining a model file of the inference service, parsing the model file to determine the minimum storage resource requirement for running the inference service; Determine the resource specification that meets the minimum storage resource requirement, which is used to represent the node required to run the inference service and the storage resource configuration corresponding to the node; According to the storage resource state of the node in the inference cluster, determine the resource configuration strategy matched with the resource specification, and configure the storage resource in the inference cluster according to the resource configuration strategy to run the inference service; In the presence of multiple inference services, obtaining the model files of the multiple inference services, and parsing the model files respectively to determine the minimum storage resource requirement for running the multiple inference services respectively; Based on the storage resource state of the node in the inference cluster, one or more candidate resource specifications corresponding to any inference service are determined respectively; Based on the candidate resource specifications corresponding to the multiple inference services respectively, a plurality of global resource configuration strategies are generated by combination; From the plurality of global resource configuration strategies, a target global resource configuration strategy capable of running the most inference services is selected; According to the target global resource configuration strategy, the storage resource is configured to run the multiple inference services; In the running process of the inference service, the inference performance parameters of the inference service are collected; Based on the comparison result of the inference performance parameters and the preset threshold value, the expansion operation of the storage resource is automatically triggered to optimize the inference performance of the inference service; The inference performance parameters at least include the first word element time and the per word element time; The automatic triggering of the expansion operation of the storage resource based on the comparison result of the inference performance parameters and the preset threshold value comprises: In the case that the first word element time exceeds the first word element time standard value, the number of inference service instances deployed in the pre-filling stage of the inference service is increased to expand the inference service; In the case that the per word element time exceeds the per word element time standard value, the number of inference service instances deployed in the decoding stage of the inference service is increased to expand the inference service.

2. The inference service resource configuration method of claim 1, wherein, The determination of the resource configuration strategy matched with the resource specification according to the storage resource state of the node in the inference cluster comprises: According to whether the available storage resource of a single node in the inference cluster matches the resource specification, it is determined whether to perform single node deployment or multi node distributed deployment on the inference service; Wherein, when performing single node deployment, a target single node meeting the minimum storage resource requirement is automatically selected in the inference cluster to configure the storage resource corresponding to the resource specification on the target single node; When performing multi node distributed deployment, a plurality of target nodes meeting the distributed deployment condition are automatically selected in the inference cluster to distribute the storage resource corresponding to the resource specification among the plurality of target nodes.

3. The inference service resource configuration method of claim 2, wherein, The configuration of the storage resource in the inference cluster according to the resource configuration strategy to run the inference service comprises: Respectively configure the storage resource corresponding to the resource specification on the target single node or the plurality of target nodes; After completing the storage resource configuration, an inference service instance corresponding to the configured storage resource is started on the configured node to run the inference service; In the case of multi-node distributed deployment, a communication link is established between different inference service instances to cooperatively run the inference service.

4. The inference service resource configuration method of claim 1, wherein, The method further comprises: In the case of separate deployment of the pre-filling stage and the decoding stage of the inference service, a first target node set for running the pre-filling stage and a second target node set for running the decoding stage are determined according to the resource specification; In the first target node set, an inference service instance of the pre-filling stage is deployed, and in the second target node set, an inference service instance of the decoding stage is deployed to separate the pre-filling stage and the decoding stage.

5. The inference service resource configuration method of claim 4, wherein, The method further comprises: In the case of the existence of the historical running template of the inference service, a first sub-resource specification and a second sub-resource specification are determined based on the historical running template; Based on the historical running template, the number of inference service instances of the inference service in the pre-filling stage and the number of inference service instances of the inference service in the decoding stage are determined; Based on the first sub-resource specification, the storage resource of the pre-filling stage is configured to deploy the inference service instance of the inference service in the pre-filling stage; Based on the second sub-resource specification, the storage resource of the decoding stage is configured to deploy the inference service instance of the inference service in the decoding stage.

6. The method of claim 1, wherein, The method further comprises: When the inference cluster contains heterogeneous nodes, based on the node performance parameters of each node, a candidate node set that meets the resource specification is screened; Based on the running requirements of the inference service, a target node that can carry the inference service is determined from the candidate node set; According to the resource configuration strategy, the storage resource is configured on the target node, and the corresponding inference service instance is started.

7. The method of claim 1, wherein, The method further comprises: For any candidate resource specification of the inference service, it is determined whether the available storage resource of the node in the inference cluster meets the current candidate resource specification; If there is a node that meets the current candidate resource specification, the current candidate resource specification is added to the current global resource configuration strategy; The next inference service of the current inference service is taken as a new current inference service, and the step of determining whether the available storage resource of the node in the inference cluster meets the current candidate resource specification is returned to continue execution until the current global resource configuration strategy contains one candidate resource specification corresponding to each of the plurality of inference services.

8. The inference service resource configuration method of claim 7, wherein, The method further comprises: If there is a node that meets the current candidate resource specification, a new global resource configuration strategy is generated based on the current global resource configuration strategy and is taken as the current global resource configuration strategy; A next candidate resource specification of the current candidate resource specification is determined; It is determined whether the available storage resource of the node in the inference cluster meets the next candidate resource specification; if there is a node satisfying the next candidate resource specification, adding the next candidate resource specification to the current global resource configuration strategy; if there is no node satisfying the next candidate resource specification, stopping generating the current global resource configuration strategy; generating a new global resource configuration strategy based on the current global resource configuration strategy, taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resource of the node in the inference cluster satisfies the current candidate resource specification for any of the inference services.

9. The inference service resource configuration method of claim 8, wherein, The method further comprises: if there is no node satisfying the current candidate resource specification, stopping generating the current global resource configuration strategy; generating a new global resource configuration strategy based on the current global resource configuration strategy, and taking the new global resource configuration strategy as the current global resource configuration strategy; taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resource of the node in the inference cluster satisfies the current candidate resource specification for any of the inference services.

10. The method of claim 7, wherein, The method further comprises: if there is no node satisfying the current candidate resource specification, stopping generating the current global resource configuration strategy, generating a new global resource configuration strategy based on the current global resource configuration strategy, and taking the new global resource configuration strategy as the current global resource configuration strategy; taking the next inference service of the current inference service as a new current inference service, and returning to the step of judging whether the available storage resource of the node in the inference cluster satisfies the current candidate resource specification for any of the inference services.

11. An electronic device, comprising: comprise: a memory for storing a computer program; a processor for implementing the steps of the inference service resource configuration method according to any one of claims 1 to 10 when executing the computer program.

12. A non-transitory computer readable storage medium, comprising: The non-volatile computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the inference service resource configuration method according to any one of claims 1 to 10.

Images