Resource determination method and apparatus, and program product and storage medium

Abstract

Disclosed in the embodiments of the present application are a resource determination method and apparatus, and a program product and a storage medium. The method in the embodiments of the present application comprises: receiving an inference request; determining a first computing resource required for when a large model performs inference on the inference request; acquiring first resource information of a heterogeneous cluster; and on the basis of the first computing resource and the first resource information, determining a first inference resource from the heterogeneous cluster, wherein the first inference resource is used by the large model to perform inference on the inference request. In this way, for an inference request received in real time, a computing resource required for performing inference on the inference request is determined in real time, which enables dynamic determination of computing resources corresponding to inference requests of different lengths. Thus, an inference resource corresponding to the inference request can be dynamically determined on the basis of the computing resource, and the corresponding inference resource can be more accurately determined from the heterogeneous cluster, so that inference is performed on the inference request on the basis of the inference resource, such that an inference process of the large model can avoid being affected.

Description

Resource determination method, device, program product and storage medium

[0001] The present application claims priority from the Chinese patent application No. 202411395859.7 filed on September 30, 2024, and entitled "A resource determination method, device, program product and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computers, in particular to a resource determination method, device, program product and storage medium. BACKGROUND

[0003] With the increasingly wide application of large models, the demand for corresponding inference based on large models is growing. Among them, the corresponding hardware resources are needed to realize the inference of large models.

[0004] At present, the corresponding hardware resources are determined from the heterogeneous cluster by the resource occupation of the central processing unit (CPU) and / or the resource occupation of the graphic processing unit (GPU) in the heterogeneous cluster, and then the inference of the large model is realized based on the corresponding hardware resources. Among them, the heterogeneous cluster refers to a cluster composed of hardware resources of different types or models, which may be CPU, GPU, field programmable gate array (FPGA) or other types of processors.

[0005] However, the hardware resources determined by this hardware resource determination method are not accurate enough, which will affect the subsequent inference of the large model. SUMMARY

[0006] The embodiments of the present application provide a resource determination method, device, program product and storage medium, which realize more accurate determination of the inference resources corresponding to the large model, and avoid affecting the inference process of the large model.

[0007] Therefore, in a first aspect, the embodiments of the present application provide a resource determination method, which comprises: receiving an inference request; determining a first computing resource required by a large model when inferring the inference request; obtaining first resource information of a heterogeneous cluster; determining a first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information, the first inference resource being used for the large model to infer the inference request.

[0008] In the method provided by the embodiments of the present application, for a real-time received inference request, the computing resources required for inferring the inference request are determined in real time, so that the computing resources corresponding to input requests of different lengths are dynamically determined, the inference resources corresponding to the inference request are dynamically determined based on the computing resources, the corresponding inference resources are more accurately determined from the heterogeneous cluster, and the inference of the inference request based on the inference resources can avoid affecting the inference process of the large model. Moreover, the resource information of the heterogeneous cluster is obtained in real time after the inference request is obtained, so that the actual required resources and the actually available resources are combined to more accurately determine the inference resources corresponding to the large model from the heterogeneous cluster, and the inference process of the large model can be further affected.

[0009] In combination with the first aspect, in a possible implementation of the first aspect, before receiving the inference request, the method further includes: determining second computing resources required by the large model for inferring first inference data in an inference scenario; determining the inference resource from the heterogeneous cluster based on the first computing resource and the first resource information, including: determining second inference resources from the heterogeneous cluster based on the second computing resources; determining the first inference resource from the second inference resource based on the first computing resource and the first resource information. Thus, for different inference scenarios, the corresponding second inference resources can be determined in advance through resource planning, so that more suitable first inference resources can be further determined from the second inference resources planned in advance after receiving the inference request, thereby avoiding affecting the inference process of the large model and meeting different inference needs of users.

[0010] In combination with the first aspect, in a possible implementation of the first aspect, the inference scenario includes a first response scenario, and the second computing resource includes a first computing resource. The second inference resource is determined from the heterogeneous cluster based on the second computing resource, including: the second inference resource is determined from the heterogeneous cluster based on the first computing resource and a computing resource utilization threshold. This embodiment considers that in the inference scenario requiring faster first response, the NPU used by the large model for inference needs higher computing resource to ensure faster first response calculation and guarantee faster first response. Therefore, the second inference resource corresponding to the first response scenario is determined based on the computing resource and the computing resource utilization threshold, so that more suitable second inference resources are determined for subsequent determination of the first inference resource, the inference process of the large model is avoided to be affected, the first response is guaranteed to meet the requirements, and the inference needs of users for the first response scenario are met.

[0011] In a possible implementation of the first aspect, in the first aspect, the inference scenario includes an average latency scenario, the second computing resource includes a first memory resource and a first memory bandwidth resource, and determining the second inference resource from the heterogeneous cluster based on the second computing resource includes: determining the second inference resource from the heterogeneous cluster based on the first memory resource, the first memory bandwidth resource, a memory utilization threshold, and a memory bandwidth utilization threshold. This implementation considers that in the inference scenario requiring smaller average latency, the NPU needs to have greater memory and bandwidth to ensure faster data transmission, faster overall response, and faster average latency. Therefore, the corresponding second inference resource is determined based on the memory resource, the memory bandwidth resource, the memory utilization threshold, and the memory bandwidth utilization threshold, so that a more suitable second inference resource can be determined for subsequent determination of the first inference resource, the impact on the inference process of the large model is avoided, the average latency meets the requirements, and the inference demand of the user for average latency is met.

[0012] In a possible implementation of the first aspect, before receiving the inference request, the method further includes: determining a third computing resource required by the large model for inferring second inference data in the inference scenario, the length of the second inference data being different from the length of the first inference data; and correspondingly, determining the second inference resource from the heterogeneous cluster based on the second computing resource includes: determining the second inference resource from the heterogeneous cluster based on the second computing resource and the third computing resource. In this way, the corresponding computing resource can be determined for inference data of different lengths, the heterogeneous cluster can be more accurately planned based on the computing resource, the corresponding second inference resource can be more accurately determined, and a more suitable first inference resource can be further determined from the second inference resource after receiving the inference request, so that the impact on the inference process of the large model can be avoided, and different inference demands of the user can be met.

[0013] In a possible implementation of the first aspect, the heterogeneous cluster is a heterogeneous neural network processing unit (NPU) cluster, and before receiving the inference request, the method further includes: obtaining second resource information of the heterogeneous NPU cluster, the second resource information including: network interconnection information between a plurality of NPUs and storage and computing capabilities of the NPUs; obtaining model structure information of the large model; determining a model deployment overhead of the large model based on the model structure information; and determining the second inference resource from the heterogeneous cluster based on the second computing resource includes: determining the third inference resource from the heterogeneous NPU cluster based on the second resource information and the model deployment overhead; and determining the second inference resource from the third inference resource based on the second computing resource.

[0014] In this embodiment, the network interconnection information between multiple NPUs and the storage and computing capabilities of the NPUs can be obtained. Thus, by using the relevant information of the NPUs specially designed for accelerating neural network calculation in combination with the model deployment overhead of the large model, the third inference resource that can be used to perform inference on the inference request can be more accurately determined, so that the first inference resource can be more accurately determined subsequently, and the inference process of the model is avoided from being affected. Moreover, the deployment overhead is determined corresponding to the obtained model structure information, so that the inference resource corresponding to the large model of different structures can be more accurately determined, and the inference process of the model is further avoided from being affected.

[0015] With reference to the first aspect, in a possible implementation of the first aspect, the method further includes: combining the multiple NPUs in the heterogeneous NPU cluster to obtain N resource groups, N being a positive integer; and determining the third inference resource from the heterogeneous NPU cluster based on the second resource information and the model deployment overhead, including: determining available resources corresponding to the N resource groups based on the second resource information; and determining the third inference resource from the N resource groups based on the available resources and the model deployment overhead. The third inference resource includes M resource groups, M being a positive integer, and M≤N. In this way, by combining the multiple NPUs in the heterogeneous NPU cluster, the available resources corresponding to each resource group can be determined, so that it is determined for each resource group whether the resource group can be used for inference of the large model. Thus, in the heterogeneous NPU cluster with a large number of NPUs and different models, the third inference resource can be quickly determined, and thus the first inference resource can be quickly determined.

[0016] With reference to the first aspect, in a possible implementation of the first aspect, the method further includes: predicting an inference time required by the large model to perform inference on the inference request based on the first inference resource; if the inference time is less than or equal to a time threshold, performing inference on the inference request by the large model based on the first inference resource; or if the inference time is greater than the time threshold and the inference setting is set to continue in case of inference timeout, performing inference on the inference request by the large model based on the first inference resource. In this embodiment, the inference of the large model is performed when the inference time is less than or equal to the time threshold, or the inference of the large model is performed in case that the user does not care about the inference timeout. Thus, the problem of poor user experience caused by the inference time can be avoided.

[0017] In a second aspect, the present application provides a resource determination apparatus, which comprises:

[0018] The receiving module is configured to receive an inference request.

[0019] The determining module is configured to determine a first computing resource required by a large model to perform inference on the inference request.

[0020] The obtaining module is configured to obtain first resource information of a heterogeneous cluster.

[0021] The determining module is further configured to determine, based on the first computing resource and the first resource information, a first inference resource from the heterogeneous cluster, the first inference resource being used for the large model to perform inference on the inference request.

[0022] The resource determining apparatus has the function of implementing the resource determining method in the first aspect or any possible implementation manner of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0023] The beneficial effects shown in the first aspect are similar to those of the first aspect. For details, see the first aspect, which will not be repeated here.

[0024] In a third aspect, the embodiments of the present application provide a resource determining apparatus. The apparatus can be applied to a terminal device or a circuit, chip, chip system, etc. inside the terminal device, or can be applied to a network device or a circuit, chip, chip system, etc. inside the network device. The apparatus can include at least one processor configured to invoke computer instructions in a memory to cause the resource determining apparatus to perform the resource determining method in the first aspect or any optional implementation manner of the first aspect, or perform the resource determining method in the second aspect or any optional implementation manner of the second aspect.

[0025] In combination with the fourth aspect, in a possible implementation manner of the fourth aspect, the resource determining apparatus can further include a memory.

[0026] In a fifth aspect, the present application provides a computer-readable storage medium, which can include instructions that, when executed on a computer, cause the computer to perform the resource determining method in the first aspect or any optional implementation manner of the first aspect, or perform the resource determining method in the second aspect or any optional implementation manner of the second aspect.

[0027] In a sixth aspect, the present application provides a computer program product including instructions that, when executed on a computer, cause the computer to perform the resource determining method in the first aspect or any optional implementation manner of the first aspect, or perform the resource determining method in the second aspect or any optional implementation manner of the second aspect.

[0028] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting the device to implement the functions involved in the above aspects, such as sending or processing the data and / or information involved in the above methods. In a possible design, the chip system further includes a memory for storing the program instructions and data necessary for the device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0029] In an eighth aspect, the embodiments of the present application provide a chip, which includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to perform the resource determination method in the first aspect or any of the optional implementation manners of the first aspect, or perform the resource determination method in the second aspect or any of the optional implementation manners of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0030] FIG. 1 is a schematic diagram of a scenario for determining inference resources according to an embodiment of the present application;

[0031] FIG. 2 is a schematic diagram of a resource determination method according to an embodiment of the present application;

[0032] FIG. 3 is a schematic diagram of a scenario for starting large model inference according to an embodiment of the present application;

[0033] FIG. 4 is a schematic diagram of another resource determination method according to an embodiment of the present application;

[0034] FIG. 5 is a schematic diagram of a scenario for determining M resource groups according to an embodiment of the present application;

[0035] FIG. 6 is a schematic diagram of a scenario for determining model instances according to an embodiment of the present application;

[0036] FIG. 7 is a schematic diagram of another scenario for starting large model inference according to an embodiment of the present application;

[0037] FIG. 8 is a schematic diagram of a structure of a resource determination apparatus according to an embodiment of the present application;

[0038] FIG. 9 is a schematic diagram of another resource determination apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described below with reference to the accompanying drawings. It is known to those skilled in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0040] The terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence, the term "at least one" should be understood as one or more, and the term "at least one" should be understood as one or more. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a way of distinguishing the objects of the same attribute used in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or equipment containing a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the related concepts involved in the embodiments of the present application are explained below.

[0042] Large model refers to a deep learning model with a very large number of parameters, which is usually used to process large-scale data sets and can learn complex features of data. In the training process of large model, a large amount of data and computing resources are needed, and a long time is needed to train the model. Therefore, large model usually needs to use high-performance computers and large-scale computing clusters for training. At the same time, large model also needs to use various optimization algorithms and techniques to speed up the training process and improve the accuracy of the model. Large model has a wide range of applications, for example, in the field of natural language processing, large model can be used for machine translation, text generation, dialogue system, etc. Tasks; in the field of image processing, large model can be used for image classification, target detection, image generation, etc. Tasks; in the field of speech recognition, large model can be used for speech recognition, speech synthesis, speech conversion, etc. Tasks.

[0043] NPU is used to accelerate the operation of neural network, to solve the problem of low efficiency of traditional chip in neural network operation. The working principle of NPU is to simulate human neurons and synapses at the circuit layer, and directly process large-scale neurons and synapses with deep learning instruction set, and a instruction can complete the processing of a group of neurons. Compared with CPU and GPU, NPU realizes the integration of storage and calculation through highlighting weight, thereby improving the running efficiency.

[0044] Heterogeneous NPU cluster refers to combining multiple NPU of different types into a cluster to achieve more efficient parallel computing and load balancing. Such a cluster can include NPU of different generations, NPU of different computing power, or even NPU of different manufacturers. Through heterogeneous cluster, the advantages of each NPU can be fully utilized, and the performance and energy efficiency ratio can be optimized.

[0045] A heterogeneous GPU cluster refers to combining multiple GPUs of different types into a cluster. Such a cluster can include GPUs of different generations or even GPUs of different manufacturers.

[0046] Currently, the corresponding hardware resources are determined from the heterogeneous cluster based on the resource occupation of the CPU and / or the resource occupation of the GPU in the heterogeneous cluster, and then the inference of the large model is implemented based on the fixed hardware resources. However, the lengths of different inference requests are different, and the inference resources required by inference requests of different lengths are different. Therefore, the inference of inference requests of different lengths based on the same fixed hardware resources will affect the inference process of the large model.

[0047] Therefore, the embodiments of the present application provide a resource determination method, device, program product and storage medium. For the inference request received in real time, the computing resources required for the inference of the inference request are determined in real time. Therefore, the computing resources corresponding to inference requests of different lengths can be dynamically determined, so that the inference resources corresponding to the inference request can be dynamically determined based on the computing resources, the corresponding inference resources can be more accurately determined from the heterogeneous cluster, and the inference of the inference request based on the inference resources can avoid affecting the inference process of the large model.

[0048] Please refer to FIG. 1, which is a scene diagram for determining inference resources provided by an embodiment of the present application. In FIG. 1, there are machine 1, machine 2, machine 3, machine 4, machine 5, machine 6, machine 7 and machine 8 in the heterogeneous cluster. Machine 1 and machine 2 include NPU with model number 910B1, machine 3 and machine 4 include NPU with model number 910B2, machine 5 and machine 6 include NPU with model number 910B3, and machine 7 and machine 8 include NPU with model number 910B4. Server 1, server 2, server 3 and server 4 are determined as the inference resources corresponding to the large model from the heterogeneous NPU cluster, that is, machine 1, machine 2, machine 3 and machine 4 can be used for the inference of the large model. It can be understood that FIG. 1 is only illustrative and should not be construed as limiting the embodiments of the present application.

[0049] Specifically, please refer to FIG. 2, which is a flowchart of a resource determination method provided by an embodiment of the present application. The resource determination method can be executed by a resource determination device. The resource determination method provided by the embodiments of the present application can include:

[0050] S201, receiving an inference request.

[0051] The user inputs the inference request data in the embodiments of the present application, and a corresponding inference request is generated. Thus, the corresponding inference request can be received. For example, the inference request data can be "what does the twenty-four solar terms include", or the inference request data can be "a cat is standing on the table", or the inference request data can be "what is NPU, GPU, CPU", and the like. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0052] S202, determining a first computing resource required by the large model to infer the inference request.

[0053] The required first computing resource can include required memory resource, required memory bandwidth resource, required computing power resource, and the like, and the embodiments of the present application do not limit this.

[0054] In a possible implementation, the first computing resource in the embodiments of the present application can be a memory resource. Accordingly, S202 can include determining a third memory resource required by the large model to infer the inference request. For example, the third memory resource required by the large model to infer the inference request corresponding to the inference request data "what does the twenty-four solar terms include" is 10G, or the third memory resource required by the large model to infer the inference request corresponding to the inference request data "what is NPU, GPU, CPU" is 20G, and the like. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0055] S203, obtaining first resource information of the heterogeneous cluster.

[0056] The heterogeneous cluster in the embodiments of the present application can be a heterogeneous NPU cluster or a heterogeneous GPU cluster, and the like, and the embodiments of the present application do not limit this.

[0057] The first resource information of the heterogeneous cluster can include residual memory resource of the heterogeneous cluster, memory bandwidth resource of the heterogeneous cluster, computing power resource of the heterogeneous cluster, and the like, and the embodiments of the present application do not limit this.

[0058] In a possible implementation, the first resource information in the embodiments of the present application can be residual memory resource. Accordingly, S203 can include obtaining residual memory resource of the heterogeneous cluster. For example, the heterogeneous cluster includes machine 1, machine 2, and machine 3. It is determined that the residual memory resource corresponding to machine 1 is 2G, the residual memory resource corresponding to machine 2 is 12G, and the residual memory resource corresponding to machine 3 is 5G. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0059] S204. Determine, based on the first computing resource and the first resource information, a first inference resource from the heterogeneous cluster.

[0060] The first inference resource is used for the large model to perform inference on the inference request.

[0061] In one possible implementation, if the first computing resource is a third memory resource and the first resource information is a remaining memory, the first inference resource can be determined by combining the remaining memory resources of the plurality of machines in the heterogeneous cluster. For example, the heterogeneous cluster includes machine 1, machine 2, and machine 3, the remaining memory resource corresponding to machine 1 is 2G, the remaining memory resource corresponding to machine 2 is 12G, and the remaining memory resource corresponding to machine 3 is 5G. The third memory resource is 15G. Machine 2 and machine 3 are determined as the first inference resource. It can be understood that the above is only an example and should not be construed as a limitation on the embodiments of the present application.

[0062] It can be seen that, in the embodiments of the present application, the computing resource required for performing inference on the inference request received in real time is determined in real time. Thus, the computing resource corresponding to the inference request of different lengths can be dynamically determined, the inference resource corresponding to the inference request can be dynamically determined based on the computing resource, the corresponding inference resource can be more accurately determined from the heterogeneous cluster, and thus the inference request can be inferred based on the inference resource, which can avoid affecting the inference process of the large model. Moreover, the resource information of the heterogeneous cluster is obtained in real time after the inference request is obtained, which can combine the actual required resource and the actual available resource to more accurately determine the inference resource of the large model from the heterogeneous cluster, and can further avoid affecting the inference process of the large model.

[0063] Please refer to FIG. 3, which is a scene diagram of starting large model inference provided by an embodiment of the present application. In FIG. 3, the resource determination method is further explained on the basis of the resource determination method provided in FIG. 2. In FIG. 3, the user can input the composition information of the heterogeneous cluster that can be used for large model inference, for example, input 910A...910B. This indicates that the series of NPU 910A...910B can be used for large model inference, thereby obtaining the composition information of the heterogeneous cluster. It can be understood that the above is only an example and should not be construed as limiting the embodiments of the present application. After obtaining the composition information of the heterogeneous cluster, the heterogeneous cluster information can be dynamically perceived in real time, the specification configuration of the heterogeneous cluster, the network bandwidth of the machines in the heterogeneous cluster, and the current available resources of all NPU in the heterogeneous cluster, and the like are obtained; using the obtained information, the current loadable resources of each accelerator chip are estimated; according to the model structure information of the large model input by the user in FIG. 3, the on-network calculation overhead of deploying the model is estimated by a theoretical formula, which is the model deployment overhead; then, in combination with the available resources of the current heterogeneous cluster and the deployment model overhead, a plurality of resource groups are determined from the heterogeneous cluster; for each resource group, the number of model instances corresponding to different inference scenarios is found by means of operational optimization, so that the user can decide the final used resource group according to the inference scenario that each group is good at; the inference request is generated according to the inference request data input by the user, and finally the model instance meeting the requirements in the resource group is selected according to the user request and the inference setting, a dynamic scheduling plan with minimum overhead is generated, which is used to start the heterogeneous mixed inference, and finally the inference result corresponding to the inference request is output.

[0064] It should be noted that the user input in input 3 and the user input in input 4 in FIG. 3 can be understood as the inference scenario selected by the user. Prefill and decoding in FIG. 3 are different stages of large model inference, which are processed by different NPU corresponding to the same model instance. Among them, different model instances can be selected for processing according to the inference batch size in combination with the corresponding threshold.

[0065] Please refer to FIG. 4, which is a flow diagram of another resource determination method provided by an embodiment of the present application. FIG. 4 further explains the resource determination method on the basis of the resource determination method provided in FIG. 2 and FIG. 3. The resource determination method in FIG. 4 can be executed by a resource determination device. The embodiments of the present application take a heterogeneous cluster as an example of a heterogeneous NPU cluster. The resource determination method provided by the embodiments of the present application can include:

[0066] S401, obtaining second resource information of a heterogeneous NPU cluster.

[0067] The second resource information includes information of NPU and information of CPU, and the like.

[0068] The information of the NPUs can include network interconnection information between the NPUs, storage and computing capabilities of the NPUs, the number of the NPUs, and the like. The storage and computing capabilities of the NPUs can include scalar computing power of the NPUs, vector computing power of the NPUs, matrix computing power of the NPUs, video memory capacity of the NPUs, and Level 2 (L2) cache capacity of the NPUs. The network interconnection information between the NPUs can include interconnection relationships between the NPUs, network bandwidth of the interconnection, network memory of the interconnection, information of a Huawei Cache Coherency System (HCCS) ring composed of the NPUs, and the like. The information of the HCCS ring can include the number of the rings, network interconnection information within the rings, network interconnection information between the rings, and the like. The HCCS is a protocol for maintaining data consistency of multiple clusters and is used for high-speed interconnection between chips.

[0069] The information of the CPUs can include main memory capacity of the CPUs, Level 1 (L1) cache capacity of the CPUs, and the like.

[0070] It should be noted that the NPUs and the CPUs are included in machines, and each machine can include at least one NPU and at least one CPU. The machines can be servers or the like.

[0071] In the embodiments of the present application, a heterogeneous NPU performance analysis tool of a resource determination apparatus can be run on each machine in a heterogeneous NPU cluster, and commands of a Linux system and a CPU processor and commands of an NPU processor are called to obtain storage and computing capabilities of the NPUs on each machine, the number of the NPUs, and information of the CPUs, and the like. Then, a test tool built in the resource determination apparatus can be used to call hccl commands and commands of the NPU processor to obtain network interconnection information between the NPUs in each machine and network interconnection information between the NPUs in each machine, so as to obtain second resource information of the heterogeneous NPU cluster. The hccl (Huawei Collective Communications Library) commands are a set of communication library commands for distributed parallel computing, and are used to improve the efficiency of distributed training. In this way, the problem that NPU multi-computing resource information and network communication capabilities cannot be obtained in the prior art is solved.

[0072] It should be noted that the real-time resource information of the heterogeneous NPU cluster is obtained based on the heterogeneous NPU performance analysis tool and the test tool in the embodiments of the present application.

[0073] In this embodiment of the present application, the network interconnection information between the NPUs and the storage and computing capabilities of the NPUs can be obtained, so that the inference resource determination can be performed by using the relevant information of the NPU specially designed for accelerating the neural network calculation, thereby more accurately determining the inference resource corresponding to the large model and avoiding affecting the inference process of the model.

[0074] S402, obtain model structure information of the large model.

[0075] In the embodiment of the present application, the model structure information of the large model can be obtained based on the model structure information input by the user. The model structure information can include the total number of layers, the number of hidden layers, the sequence length, the tokenizer size, the model parallelism number, and the like of the large model.

[0076] S403, determine the model deployment overhead of the large model based on the model structure information.

[0077] The model deployment overhead can include the calculation amount, the communication amount, and the memory occupation required for deploying the large model, and the like.

[0078] In the embodiment of the present application, the third inference resource can be determined from the heterogeneous NPU cluster based on the second resource information and the model deployment overhead, so that the first inference resource used for further determining the inference of the inference request can be determined based on the third inference resource. Moreover, the model deployment overhead is determined based on the obtained model structure information, so that the inference resource corresponding to the large model with different structures can be more accurately determined, and the inference process of the model is further avoided from being affected.

[0079] It should be noted that the overhead caused by the deployed model is fixed and is irrelevant to the size of the inference data, i.e., irrelevant to the length of the inference request data input by the user, and is regarded as the static overhead of the large model.

[0080] S404, combine the plurality of NPUs in the heterogeneous NPU cluster to obtain N resource groups.

[0081] N is a positive integer.

[0082] In the embodiment of the present application, the NPUs in the NPU cluster can be combined based on the type of the NPU to obtain N resource groups, or the NPUs in the NPU cluster can be combined based on the number of NPUs on each machine to obtain N resource groups, and the like. The specific combination manner is not limited in the embodiment of the present application.

[0083] S405, determine the available resources corresponding to the N resource groups based on the second resource information.

[0084] In an embodiment of the present application, the available resources corresponding to each resource group can be determined based on the resource information of the heterogeneous NPU cluster, such as the number of available NPUs in each resource group, the total available video memory capacity, and the like, to obtain N groups of available resources corresponding to N resource groups.

[0085] In S406, the third inference resource is determined from the N resource groups based on the available resources and the model deployment overhead.

[0086] The third inference resource includes M resource groups, and M is a positive integer and M≤N.

[0087] In an embodiment of the present application, the N groups of available resources corresponding to the N resource groups and the model deployment overhead are compared, and the N resource groups are filtered according to whether the model deployment overhead can be loaded, and only M resource groups meeting the requirements are retained to complete the planning of the heterogeneous NPU cluster resources.

[0088] Please refer to FIG. 5, which is a scene diagram for determining M resource groups provided by an embodiment of the present application. In FIG. 5, the resource information of the heterogeneous NPU cluster is initially unknown, the resource determination device senses the resources of the heterogeneous NPU cluster in real time to obtain the network topology diagram of the heterogeneous NPU cluster, and then obtains the network interconnection information between NPUs in the heterogeneous NPU cluster, the information of the NPUs, and the information of the CPUs; then, the model structure information input by a user is received, such as the model structure information including the number of layers, the optimizer type, the mixed expert model (MoE), the encoder / decoder, the parameter amount, and the like; then, based on the information of the NPUs and the information of the CPUs, the model structure information is combined to obtain the resource group 1, the resource group 2, and the resource group 3 meeting the requirements. It can be understood that FIG. 5 is only illustrative and should not be understood as a limitation on the embodiments of the present application. It should be noted that Atlas 900, Atlas 800, Atlas 800T, and Atlas 300 in FIG. 5 represent the model of the server, the peripheral component interconnect express (PCIe) is a high-speed serial computer expansion bus standard, the MoE is a model type, and the input value corresponding to the encoder / decoder is used to represent whether only the encoder or the decoder is used, or both the encoder and the decoder are used.

[0089] It can be seen that, in the embodiment of the application, the plurality of NPUs in the heterogeneous NPU cluster are combined, so that the available resources corresponding to each resource group can be determined, thereby determining whether each resource group can be used for inference of the large model, so that in the heterogeneous NPU cluster with a large number of NPUs and different models, the inference resources corresponding to the large model can be determined more quickly.

[0090] S407, determining a second computing resource required by the large model to infer the first inference data in the inference scenario.

[0091] The inference scenario can include at least one of the following: a first response scenario, an average latency scenario.

[0092] It should be noted that, if the inference scenario in the embodiment of the application includes the first response scenario, the second computing resource includes a first sub-computing resource required by the large model to infer the first inference data in the first response scenario; if the inference scenario in the embodiment of the application includes the average latency scenario, the second computing resource includes a second sub-computing resource required by the large model to infer the first inference data in the average latency scenario; if the inference scenario in the embodiment of the application includes the first response scenario and the average latency scenario, the second computing resource includes a first sub-computing resource required by the large model to infer the first inference data in the first response scenario, and a second sub-computing resource required by the large model to infer the first inference data in the average latency scenario.

[0093] It should be noted that the first response scenario requires faster first response, and the average latency scenario requires faster average latency. The first response scenario refers to the time consumption of the first token of the answer obtained after the user inputs the inference request to start inference, also known as the first response time consumption. The average latency scenario refers to the average time consumption of each token except the first token after the user inputs the inference request to start inference and obtains the complete answer.

[0094] In a possible implementation, S407 can include: determining the second computing resource required by the large model to infer the inference data in the inference scenario based on the model structure information.

[0095] S408, determining a third computing resource required by the large model to infer the second inference data in the inference scenario.

[0096] It can be understood that the method for determining the third computing resource in the embodiment of the application is similar to the method for determining the second computing resource in S407 described above, and therefore will not be described again.

[0097] It should be noted that the first inference data and the second inference data are both inference data, the length of the second inference data is different from the length of the first inference data, and the embodiments of the present application are to determine different computing resources required by the large model when performing inference on inference data of different lengths in the inference scenario.

[0098] The length of the inference data is the batch size that can be processed. In the embodiments of the present application, it can be determined according to actual needs which lengths of inference data need to be determined respectively corresponding to the computing resources, and the computing resources required by the large model when performing inference on the inference data of these lengths in the inference scenario are determined. The inference data can be, for example, “What does the twenty-four solar terms include?” and “Heterogeneous NPU cluster”, etc. The embodiments of the present application do not limit this.

[0099] S409, determining the second inference resource from the third inference resource based on the second computing resource and the third computing resource.

[0100] It can be understood that if only the computing resources corresponding to the inference resource of one length need to be determined, the second inference resource is determined from the third inference resource based only on the second computing resource.

[0101] The third inference resource includes i resource groups, i is a positive integer, i≤M, and the second inference resource is determined from the third inference resource, which can also be expressed as i resource groups are determined from the Mth resource group.

[0102] The following describes how to determine the second inference resource from the third inference resource based on the second computing resource, that is, how to determine i resource groups from the Mth resource group based on the second computing resource in different inference scenarios.

[0103] In a possible implementation, the inference scenario is a first response scenario, and the second computing resource includes a first computing resource. The second inference resource is determined from the third inference resource based on the second computing resource, including: determining the second inference resource from the third inference resource based on the first computing resource and a computing resource utilization threshold. The first computing resource includes a first scalar computing resource, a first vector computing resource, a first matrix computing resource, etc.

[0104] It should be noted that in the inference scenario requiring faster first response, the NPU used by the large model for inference needs higher computing resource to ensure faster first response calculation and faster first response, so the inference resource corresponding to the first response scenario is determined based on the computing resource.

[0105] Specifically, the second inference resource is determined from the third inference resource based on the first computing resource and the computing resource utilization threshold, which can include:

[0106] a1, determine K first hardware computing power combinations based on M resource groups.

[0107] wherein K is a positive integer.

[0108] In the embodiments of the present application, the hardware computing power combination corresponding to each resource group in the M resource groups is determined. For example, the M resource groups include resource group 1 and resource group 2. Resource group 1 includes machine 1 and machine 2, and one or more NPUs are arranged on machine 1 and machine 2; resource group 2 includes machine 3, machine 4 and machine 5, and one or more NPUs are arranged on machine 3, machine 4 and machine 5. Based on resource group 1, it is determined that the hardware computing power combination 1 includes machine 1, the hardware computing power combination 2 includes machine 2, the hardware computing power combination 3 includes machine 1 and machine 2, based on resource group 2, it is determined that the hardware computing power combination 4 includes machine 3, the hardware computing power combination 5 includes machine 4, the hardware computing power combination 6 includes machine 5, the hardware computing power combination 7 includes machine 3 and machine 4, the hardware computing power combination 8 includes machine 4 and machine 5, the hardware computing power combination 9 includes machine 3 and machine 5, and the hardware computing power combination 10 includes machine 3, machine 4 and machine 5. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0109] a2, determine the first computing power resource utilization rate corresponding to the hardware computing power combination based on the first computing power resource and the hardware computing power of the hardware computing power combination.

[0110] In the embodiments of the present application, the hardware computing power corresponding to each hardware computing power combination in the K hardware computing power combinations can be determined, and K hardware computing powers corresponding to the K hardware computing power combinations are obtained. Then, based on the first computing power resource and the K hardware computing powers, the first computing power resource utilization rate corresponding to each hardware computing power combination in the K hardware computing power combinations can be determined. Wherein, the first computing power resource utilization rate = first computing power resource / hardware computing power, and the hardware computing power refers to the computing power of NPU for different floating point numbers. Different types of NPUs have different floating point number sizes, and the computing power is also different. The hardware computing power can include the sum of scalar computing power, vector computing power, matrix computing power and the like.

[0111] a3, determine the second inference resource from the third inference resource based on the first computing power resource utilization rate and the computing power utilization rate threshold.

[0112] Wherein, the computing power utilization rate threshold can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0113] It should be noted that the hardware computing power combination with the first computing power resource utilization rate greater than the computing power utilization rate threshold will be included in the second inference resource.

[0114] In the embodiment of the application, for the first response scenario, the first computing resource utilization of each hardware computing power combination is determined, and the second inference resource is determined in combination with the computing power utilization threshold, so that the hardware computing power combination with poor computing power can be excluded, and the inference of the model is performed by the hardware combination with better computing power, thereby avoiding affecting the inference process of the large model.

[0115] It should be noted that the embodiment of the application is not limited to the first response scenario, and the second inference resource can be determined from the M resource groups in various ways. For example, any one of the M resource groups is determined as the second inference resource, or all of the M resource groups are determined as the second inference resource, and the like.

[0116] As can be seen, in the inference scenario requiring faster first response, the NPU used by the large model for inference needs higher computing power resources to ensure faster first response calculation and faster first response. Therefore, the inference resource corresponding to the first response scenario is determined based on the computing power resource, so that a more suitable inference resource can be determined, the inference process of the large model is avoided, the first response meets the requirements, and the inference demand for the first response scenario of the user can be met.

[0117] In a possible implementation, the inference scenario is an average delay scenario, the second computing resource includes a first memory resource and a first memory bandwidth resource, and the second inference resource is determined from the third inference resource based on the second computing resource, including: determining the second inference resource from the third inference resource based on the first memory resource, the first memory bandwidth resource, a memory utilization threshold, and a memory bandwidth utilization threshold.

[0118] It should be noted that in the inference scenario requiring smaller average delay, the NPU needs to have larger memory and memory bandwidth to ensure faster data transmission, faster overall response, and smaller average delay. Therefore, the inference resource corresponding to the average delay is determined based on the memory resource and the memory bandwidth resource.

[0119] Specifically, the second inference resource is determined from the third inference resource based on the first memory resource, the first memory bandwidth resource, the memory utilization threshold, and the memory bandwidth utilization threshold, which can include:

[0120] b1, determining the hardware memory and the hardware memory bandwidth of the resource group in the M resource groups.

[0121] In the embodiment of the application, the hardware memory and the hardware memory bandwidth corresponding to each resource group in the M resource groups can be determined, and M hardware memory resources and M hardware memory bandwidth resources corresponding to the M resource groups are obtained.

[0122] b2, determine a first memory utilization ratio corresponding to a resource group in the M resource groups based on the first memory resource and the hardware memory.

[0123] In the embodiment of the application, the M first memory utilization ratios corresponding to the M resource groups can be determined based on the first memory resource and the M hardware memories. The first memory utilization ratio = the first memory resource / hardware memory.

[0124] b3, determine a first memory bandwidth utilization ratio corresponding to a resource group in the M resource groups based on the first memory bandwidth resource and the hardware memory bandwidth.

[0125] In the embodiment of the application, the M first memory bandwidth utilization ratios corresponding to the M resource groups can be determined based on the first memory bandwidth resource and the M hardware memory bandwidths. The first memory bandwidth utilization ratio = the first memory bandwidth resource / hardware memory bandwidth.

[0126] b4, determine a second inference resource from the third inference resource based on the first memory utilization ratio, the first memory bandwidth utilization ratio, a memory utilization ratio threshold and a memory bandwidth utilization ratio threshold.

[0127] The memory utilization ratio threshold and the memory bandwidth utilization ratio threshold can be set according to actual conditions, and the embodiment of the application does not limit this.

[0128] It should be noted that the second inference resource includes a resource group whose first memory utilization ratio is greater than the memory utilization ratio threshold and whose first memory bandwidth utilization ratio is greater than the memory bandwidth utilization ratio threshold.

[0129] In the embodiment of the application, for the average latency scenario, the memory utilization ratio and the memory bandwidth utilization ratio of each resource group are determined, and the inference resource is determined in combination with the memory utilization ratio threshold and the memory bandwidth utilization ratio threshold. In this way, the resource group with better memory utilization ratio and memory bandwidth utilization ratio can be determined, and the inference of the model is performed with a suitable resource group, thereby avoiding affecting the inference process of the large model.

[0130] It should be noted that the embodiment of the application is not limited to the average latency scenario, and the second inference resource can be determined from the M resource groups in a specific manner. The above is one of the determination methods, and other methods can also be used to determine the second inference resource from the M resource groups, such as determining any one of the M resource groups as the second inference resource, or determining all the M resource groups as the second inference resource, and the like.

[0131] It can be seen that in the embodiment of the present application, considering the inference scene requiring smaller average latency, the NPU requires larger memory and bandwidth to ensure faster data transmission and overall faster response and smaller average latency. Therefore, the corresponding inference resource is determined based on the memory resource and memory bandwidth resource under the average latency, so that more suitable inference resource can be determined, the impact on the inference process of the large model is avoided, the average latency meets the requirements, and the inference demand of the user average latency is met.

[0132] It can be understood that for different inference scenarios, how to determine the second inference resource from the third inference resource based on the second calculation resource and the third calculation resource is similar to how to determine the second inference resource from the third inference resource based on the second calculation resource as described above, and therefore will not be described again. In the embodiment of the present application, the second calculation resource corresponding to the third calculation resource, the second memory resource, the second memory bandwidth resource, the second calculation resource utilization, the second memory utilization and the second memory bandwidth utilization can be determined.

[0133] Referring to FIG. 6, a scene schematic diagram for determining a model instance is provided in the embodiment of the present application. In FIG. 6, one resource group in the M resource groups will determine a corresponding model instance for different inference scenarios. It should be noted that the model instance refers to the number of large models deployed on the hardware resource capable of carrying out inference of the large model. In the first response scenario, one large model can be deployed on one hardware calculation combination, and one hardware calculation combination corresponds to one model instance. In the average latency scenario, one large model can be deployed on one resource group, and one resource group corresponds to one model instance. In FIG. 6, resource group 1 includes machine 1 with model Atlas800, machine 2 with model Atlas900, machine 3 with model Atlas800T and machine 4 with model Atlas800T. For different inference scenarios, calculation modeling, memory and memory bandwidth capacity modeling are performed. In the first response scenario, there are two available model instances, and in the average latency scenario, there is one available model instance. It should be noted that mbz in FIG. 6 refers to the processable batch size, the calculation power 400TFLOPS and the calculation power 282TFLOPS are hardware calculation power, the memory 64GB is hardware memory, and the memory bandwidth 1.6T / S is hardware memory bandwidth. In the first response faster scenario, one large model can be deployed on machine 1 and machine 2, and one large model can also be deployed on machine 3 and machine 4. The two large models correspond to different processable batch sizes. When receiving an inference request, whether to use the model instance 1 with a processable batch size of 128 on machine 1 and machine 2 to perform inference or to select the model instance 2 with a processable batch size of 56 on machine 3 and machine 4 is determined according to the length of the inference request data corresponding to the inference request. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0134] S410, receive resource selection information.

[0135] The resource selection information is used to indicate that j resource groups are selected from i resource groups, that is, fourth inference resources are selected from the second inference resources, the fourth inference resources include j resource groups, j is a positive integer, and j≤i.

[0136] In the embodiment of the application, a user can select j resource groups from i resource groups according to the user's own needs, so as to generate resource selection information, so that the corresponding resource selection information can be received. It should be noted that the j resource groups are for the same inference scene.

[0137] In a possible implementation, in the embodiment of the application, information corresponding to the i resource groups can be displayed on an interface, such as at least one of the processable batch size, the hardware computing power, the hardware memory, and the hardware memory bandwidth, so that the user can determine the j resource groups from the i resource groups.

[0138] S411, receive an inference request.

[0139] It can be understood that S411 is the same as S201 in the above-mentioned embodiments, and therefore will not be described again.

[0140] S412, determine a first computing resource required by the large model to perform inference on the inference request.

[0141] It can be understood that S412 is the same as S202 in the above-mentioned embodiments, and therefore will not be described again.

[0142] S413, obtain first resource information of the fourth inference resource.

[0143] It can be understood that S203 in the above-mentioned embodiments can include: obtaining the first resource information of the fourth inference resource.

[0144] In a possible implementation, S413 can include obtaining the remaining memory of the fourth inference resource. Specifically, the remaining memory of the machine corresponding to each model instance in each of the j resource groups can be obtained.

[0145] S414, determine a first inference resource from the fourth inference resource based on the first computing resource and the first resource information.

[0146] It can be understood that S204 in the above-mentioned embodiments can include: determining the fourth resource from the second resource based on the resource selection information, and determining the first inference resource from the fourth inference resource based on the first computing resource and the first resource information.

[0147] If multiple resource groups in the fourth inference resource meet the inference requirement in the embodiments of the present application, further screening is required. In a possible implementation manner, the resource group corresponding to the batch size greater than the length of the inference request and the minimum length gap can be determined as the first inference resource, so that the most suitable resource group for the large model to perform inference can be determined in combination with the length of the inference request.

[0148] It should be noted that after determining the first inference resource, a corresponding scheduling plan is generated.

[0149] S415, based on the first inference resource, the large model is used to perform inference on the inference request.

[0150] In the embodiments of the present application, the inference time required for the large model to perform inference on the inference request based on the first inference resource can be predicted. If the first inference resource is the resource group corresponding to the first response scenario, the first response time is determined; if the first inference resource is the average time delay scenario, the average time is determined. If the inference time is less than or equal to the time threshold, the large model is used to perform inference on the inference request based on the first inference resource; or if the inference time is greater than the time threshold, and the inference setting is to continue inference timeout, the large model is used to perform inference on the inference request based on the first inference resource. In this way, the inference of the large model is performed when the inference time is less than or equal to the time threshold, or the inference of the large model is performed in the case that the user does not care about the inference timeout, so that the problem of poor user experience caused by the inference time can be avoided.

[0151] The time threshold can be input by the user or predefined, and the embodiments of the present application do not limit this.

[0152] It should be noted that after using the large model to perform inference on the inference request, the corresponding inference result is obtained. For example, if the inference request is "what does the 24 solar terms include", the corresponding inference result is "Lichun, Yushui, Jingzhe, Chunfen, Qingming, Guyu, Lixia, Xiaoman, Mangzhe, Xiazhi, Xiaoshu, Dashu, Liju, Chushu, Bailu, Qiufen, Shangjiang, Xiaoxue, Daxue, Dongzhi, Xiaohan, Dahan". It can be understood that the above is only an exemplary description and should not be understood as a limitation of the embodiments of the present application.

[0153] Please refer to FIG. 7, which is another scene diagram for starting large model inference provided by an embodiment of the present application. In FIG. 7, the resource group selected by the user includes two model instances, the user inputs inference request data with a length of 20, generates a corresponding inference request, determines that the corresponding first resource information is required memory of 10G based on the inference request, obtains that the available memory of model instance 1 is 2G and the available memory of model instance 2 is 12G, determines that model instance 2 is the first inference resource, predicts the inference time consumption of model instance 2, determines that the inference time consumption is 1000ms, which is greater than the maximum time consumption limit 100ms, and since the user sets to continue in timeout, the inference is started using the model instance 2. The maximum time consumption limit is the time consumption threshold. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0154] It can be seen that in the embodiments of the present application, the resource groups of the heterogeneous NPU cluster are divided based on dynamic perception, and a dynamic scheduling plan is generated based on the resource groups, thereby achieving load balancing of the heterogeneous NPU cluster and improving the large model inference efficiency of the heterogeneous NPU cluster. Compared with the homogeneous NPU cluster, the inference acceleration for the heterogeneous NPU cluster is realized, the throughput can be improved by 1.6 times, and the average latency can be reduced by 1.3 times. Moreover, the mixed inference of the heterogeneous NPU cluster is realized, the "resource wall" between different types of chips is broken, the inference service efficiently utilizes different types of NPU in the heterogeneous NPU cluster through dynamic scheduling, and different inference scenarios are predicted for the expense, thereby providing more accurate expense prediction for the customer, realizing efficient utilization of NPU resources according to the inference scenario. Moreover, dynamic scheduling inference acceleration is realized, the resource situation is obtained in real time after the user inputs the inference request data, and a dynamic scheduling plan is generated. Compared with the static scheduling heterogeneous inference technology, the throughput can be improved by 1.3 times, and the average latency can be reduced by 1.2 times.

[0155] Moreover, the embodiments of the present application support real-time resource acquisition of the heterogeneous NPU cluster, solve the technical problems that the prior art cannot acquire NPU multi-element computing power resource information and network communication capability, support subsequent NPU resource group division and dynamic scheduling technology of load balancing, and improve the accuracy of inference resource determination. Suitable NPU resource groups and corresponding expenses in different inference scenarios are provided for the user to select, thereby solving the problem that the prior art is single for the inference scenario, improving the NPU cluster utilization rate according to different inference scenarios, and meeting different inference requirements of the user.

[0156] It should be noted that the embodiments of the present application can be applied to the operation and maintenance scene of the heterogeneous NPU cluster in addition to the large model inference scene of the heterogeneous NPU cluster. Systematic modeling analyzes the performance of the heterogeneous NPU cluster, including cluster interconnection information, network bandwidth between machines, and NPU information inside the heterogeneous NPU cluster, etc., which is helpful for cluster performance fault operation and maintenance analysis. Among them, the network bandwidth between machines includes the speed of transmitting gradient and tensor data between two devices, and the NPU information inside the heterogeneous NPU cluster includes the number of NPUs, scalar computing power, vector computing power, matrix computing power, memory capacity, network communication capability, etc.

[0157] It should be noted that for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0158] In order to better implement the above scheme of the embodiments of the present application, the related device for implementing the above scheme is also provided below.

[0159] Please refer to FIG. 8, the resource determination device 800 provided by the embodiments of the present application can include:

[0160] The receiving module 801 is configured to receive an inference request.

[0161] The determining module 802 is configured to determine a first computing resource required by the large model for inferring the inference request.

[0162] The obtaining module 803 is configured to obtain first resource information of a heterogeneous cluster.

[0163] The determining module 802 is further configured to determine a first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information, and the first inference resource is used for the large model to infer the inference request.

[0164] The device provided by the embodiments of the present application can determine the computing resources required for reasoning on the reasoning request in real time, so as to dynamically determine the computing resources corresponding to input requests of different lengths, dynamically determine the reasoning resources corresponding to the reasoning request based on the computing resources, accurately determine the corresponding reasoning resources from the heterogeneous cluster, and thus avoid affecting the reasoning process of the large model. Moreover, the resource information of the heterogeneous cluster is obtained in real time after the reasoning request is obtained, so that the actual required resources and the actually available resources are combined to accurately determine the reasoning resources corresponding to the large model from the heterogeneous cluster, and the influence on the reasoning process of the large model is further avoided.

[0165] In some embodiments of the present application, the determination module 802 in the resource determination device 800 provided by the embodiments of the present application is further configured to determine, before receiving the reasoning request, a second computing resource required by the large model for reasoning on first reasoning data in a reasoning scenario.

[0166] The determination module 802 is specifically configured to determine, based on the second computing resource, a second reasoning resource from the heterogeneous cluster; and determine, based on the first computing resource and the first resource information, the first reasoning resource from the second reasoning resource.

[0167] In some embodiments of the present application, the reasoning scenario includes a first response scenario, and the second computing resource includes a first computing resource. The determination module 802 in the resource determination device 800 provided by the embodiments of the present application is specifically configured to determine, based on the first computing resource and a computing resource utilization threshold, the second reasoning resource from the heterogeneous cluster.

[0168] In some embodiments of the present application, the reasoning scenario includes an average time delay scenario, and the second computing resource includes a first memory resource and a first memory bandwidth resource. The determination module 802 in the resource determination device 800 provided by the embodiments of the present application is specifically configured to determine, based on the first memory resource, the first memory bandwidth resource, a memory utilization threshold, and a memory bandwidth utilization threshold, the second reasoning resource from the heterogeneous cluster.

[0169] In some embodiments of the present application, the determination module 802 in the resource determination device 800 provided by the embodiments of the present application is further configured to determine, before receiving the reasoning request, a third computing resource required by the large model for reasoning on second reasoning data in a reasoning scenario, the length of the second reasoning data being different from the length of the first reasoning data. The determination module 802 is specifically configured to determine, based on the second computing resource and the third computing resource, the second reasoning resource from the heterogeneous cluster.

[0170] In some embodiments of the present application, the heterogeneous cluster is a heterogeneous neural network processor cluster, and the resource determination apparatus 800 provided by the embodiments of the present application further includes an obtaining module 803, which is configured to obtain second resource information of the heterogeneous neural network processor cluster before receiving the inference request, the second resource information including network interconnection information between a plurality of neural network processors and storage and computing capabilities of the neural network processors.

[0171] The obtaining module 803 is configured to obtain model structure information of the large model.

[0172] The determining module 802 is further configured to determine a model deployment overhead of the large model based on the model structure information.

[0173] The determining module 802 is specifically configured to determine third inference resources from the heterogeneous neural network processor cluster based on the second resource information and the model deployment overhead, and determine second inference resources from the third inference resources based on the second computing resources.

[0174] In some embodiments of the present application, the resource determination apparatus 800 provided by the embodiments of the present application further includes a combination module, which is configured to combine the neural network processors in the heterogeneous neural network processor cluster to obtain N resource groups, N being a positive integer.

[0175] The determining module 802 is specifically configured to determine available resources corresponding to the N resource groups based on the second resource information, and determine the third inference resources from the N resource groups based on the available resources and the model deployment overhead, the third inference resources including M resource groups, M being a positive integer, and M≤N.

[0176] In some embodiments of the present application, the resource determination apparatus 800 provided by the embodiments of the present application further includes a prediction module, which is configured to predict an inference time consumption required by the large model to perform inference on the inference request based on the first inference resources.

[0177] The inference module is configured to perform inference on the inference request by the large model based on the first inference resources if the inference time consumption is less than or equal to the time consumption threshold, or perform inference on the inference request by the large model based on the first inference resources if the inference time consumption is greater than the time consumption threshold and the inference setting is set to continue inference after timeout.

[0178] It should be noted that the information interaction, execution process and the like between the modules / units of the above apparatus are based on the same concept as the method embodiments of the present application, and the technical effects brought by the same are the same as those of the method embodiments of the present application. For details, refer to the description of the method embodiments of the present application described above, which will not be repeated here.

[0179] Next, another resource determination apparatus provided by the embodiment of the present application is introduced. Please refer to FIG. 9, which is a structural schematic diagram of another resource determination apparatus provided by the embodiment of the present application. The resource determination apparatus 900 can include one or more processors 901 and memories 905.

[0180] The memories 905 can include read-only memories and random access memories, and provide instructions and data for the processors 901. A part of the memories 905 can also include non-volatile random access memories (NVRAM). The memories 905 store processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0181] The processors 901 control the operation of the device. In specific applications, various components of the resource determination apparatus 900 are coupled together through a bus system, which can include a data bus in addition to power buses, control buses and state signal buses, etc. However, in order to clearly illustrate, all kinds of buses are referred to as a bus system in the figure.

[0182] The resource determination method disclosed in the embodiments of the present application can be applied to the processor 901 or implemented by the processor 901. The processor 901 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuits in the processor 901 or by instructions in the form of software. The processor 901 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 901 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or the like mature storage medium in the art. The storage medium is located in the storage 905, and the processor 901 reads the information in the storage 905, and combines the hardware to complete the steps of the above resource determination.

[0183] The resource determination apparatus 900 can also include one or more power supplies 902, one or more wired or wireless network interfaces 903, one or more input / output interfaces 904, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0184] In the embodiments of the present application, the processor 901 is configured to execute the resource determination method in the embodiments of FIG. 2 or FIG. 4. It should be noted that the specific manner in which the processor 901 executes the above steps is based on the same concept as the method embodiments corresponding to FIG. 1 to FIG. 7 in the present application, and the resulting technical effects are the same as the method embodiments corresponding to FIG. 1 to FIG. 7 in the present application. For specific content, please refer to the description of the method embodiments in the foregoing description of the present application, which will not be described here.

[0185] The embodiment of the present application also provides a computer readable storage medium, which stores instructions, and when the instructions are executed on a computer, the computer executes the resource determination method described in the foregoing embodiment shown in FIG. 2 or FIG. 4.

[0186] The embodiment of the present application also provides a computer program product, which comprises a program, and when the program is executed on a computer, the computer executes the resource determination method described in the foregoing embodiment shown in FIG. 2 or FIG. 4.

[0187] The processor mentioned in any of the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of the program of the method of the first aspect.

[0188] In addition, it should be noted that the above-described device embodiments are only schematic and that the units mentioned above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the device embodiments provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0189] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CLUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments of the present application.

[0190] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0191] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A resource determination method, characterized by, The method comprises: receiving an inference request; determining a first computing resource required by a large model for inferring the inference request; obtaining first resource information of a heterogeneous cluster; determining a first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information, the first inference resource being used for the large model to infer the inference request.

2. The method of claim 1, wherein, Before the receiving an inference request, the method further comprises: determining a second computing resource required by the large model for inferring first inference data in an inference scenario; determining a second inference resource from the heterogeneous cluster based on the second computing resource; determining the first inference resource from the second inference resource based on the first computing resource and the first resource information. The inference scenario comprises a first response scenario, and the second computing resource comprises a first computing resource. Determining a second inference resource from the heterogeneous cluster based on the second computing resource comprises:

3. The method of claim 2, wherein, determining a second inference resource from the heterogeneous cluster based on the first computing resource and a computing resource utilization threshold. The inference scenario comprises an average time delay scenario, and the second computing resource comprises a first memory resource and a first memory bandwidth resource. Determining a second inference resource from the heterogeneous cluster based on the second computing resource comprises:

4. The method according to any one of claims 2 to 3, characterized in that, determining a second inference resource from the heterogeneous cluster based on the first memory resource, the first memory bandwidth resource, a memory utilization threshold, and a memory bandwidth utilization threshold. Before the receiving an inference request, the method further comprises:

5. The method according to any one of claims 2 to 4, characterized in that, determining a third computing resource required by the large model for inferring second inference data in an inference scenario, the length of the second inference data being different from the length of the first inference data; determining a second inference resource from the heterogeneous cluster based on the second computing resource and the third computing resource. The heterogeneous cluster is a heterogeneous neural network processor cluster. Before the receiving an inference request, the method further comprises: obtaining second resource information of the heterogeneous neural network processor cluster, the second resource information comprising network interconnection information between a plurality of neural network processors and storage and computing capabilities of the neural network processors; 6. The method according to any one of claims 2 to 5, characterized in that, obtaining model structure information of the large model; determining a model deployment overhead of the large model based on the model structure information; determining a third inference resource from the heterogeneous neural network processor cluster based on the second resource information and the model deployment overhead; determining the second inference resource from the third inference resource based on the second computing resource. The method further comprises: combining the plurality of neural network processors in the heterogeneous neural network processor cluster to obtain N resource groups, N being a positive integer; ​ 7. The method of claim 6, wherein, ​ ​ The determining, based on the second resource information and the model deployment overhead, a third inference resource from the cluster of heterogeneous neural network processors includes: determining available resources corresponding to the N resource groups based on the second resource information; determining the third inference resource from the N resource groups based on the available resources and the model deployment overhead, the third inference resource including M of the resource groups, M being a positive integer and M≤N.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: predicting an inference time consumption required by the large model to infer the inference request based on the first inference resource; if the inference time consumption is less than or equal to a time consumption threshold, inferring the inference request by the large model based on the first inference resource; or, if the inference time consumption is greater than the time consumption threshold and an inference setting is set to continue inference timeout, inferring the inference request by the large model based on the first inference resource.

9. A resource determining apparatus, characterized by comprising: The apparatus includes: a receiving module configured to receive an inference request; a determining module configured to determine a first computing resource required by a large model to infer the inference request; an obtaining module configured to obtain first resource information of a heterogeneous cluster; the determining module is further configured to determine, based on the first computing resource and the first resource information, a first inference resource from the heterogeneous cluster, the first inference resource being used by the large model to infer the inference request.

10. A resource determination device, characterized in that, The apparatus includes at least one processor configured to invoke computer instructions in a memory to cause the resource determination apparatus to perform the resource determination method of any one of claims 1 to 8.

11. A computer program product, characterised in that, The apparatus includes instructions that, when executed on a computer, cause the computer to perform the resource determination method of any one of claims 1 to 8.

12. A computer storage medium, characterized in that The apparatus includes a computer program that, when executed on a computer, causes the computer to perform the resource determination method of any one of claims 1 to 8.

13. A chip, characterized by The apparatus includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the resource determination method of any one of claims 1 to 8.

Images