Resource determination method and device, program product and storage medium

Abstract

The embodiment of the invention discloses a resource determination method and device, a program product and a storage medium. The method provided by the embodiment of the invention comprises the following steps: receiving a reasoning request; determining a first computing resource required when the large model performs reasoning on the reasoning request; obtaining first resource information of the heterogeneous cluster; and determining a first reasoning resource from the heterogeneous cluster based on the first computing resource and the first resource information, the first reasoning resource being used for reasoning the reasoning request by the large model. Therefore, for the reasoning request received in real time, the computing resources required for reasoning the reasoning request are determined in real time, so that the computing resources corresponding to the reasoning requests with different lengths can be dynamically determined, and the reasoning resources corresponding to the reasoning request can be dynamically determined based on the computing resources; the corresponding reasoning resource is more accurately determined from the heterogeneous cluster, so that the reasoning request is reasoned based on the reasoning resource, and the influence on the reasoning process of a large model can be avoided.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a resource determination method, apparatus, program product, and storage medium. Background Technology

[0002] As large-scale models become more widely used, the demand for inference based on these models is increasing. This requires corresponding hardware resources to implement inference on large-scale models.

[0003] Currently, the corresponding hardware resources in a heterogeneous cluster are determined by analyzing the resource usage of the Central Processing Unit (CPU) and / or Graphics Processing Unit (GPU). Inference of large models is then performed based on these corresponding hardware resources. A heterogeneous cluster refers to a cluster composed of different types or models of hardware resources, which may include CPUs, GPUs, field-programmable gate arrays (FPGAs), or other types of processors.

[0004] However, the hardware resources determined by this method are not accurate enough, which will affect the inference of subsequent large models. Summary of the Invention

[0005] This application provides a resource determination method, apparatus, program product, and storage medium to more accurately determine the inference resources corresponding to a large model and avoid affecting the inference process of the large model.

[0006] In view of this, in a first aspect, embodiments of this application provide a resource determination method, the method comprising: receiving an inference request; determining a first computing resource required by a large model to perform inference on the inference request; obtaining first resource information of a heterogeneous cluster; and determining a first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information, the first inference resource being used by the large model to perform inference on the inference request.

[0007] The method provided in this application determines the computing resources required for inference of a received inference request in real time. This dynamically determines the computing resources corresponding to input requests of different lengths, enabling the dynamic determination of inference resources for the inference request based on these computing resources. This achieves more accurate determination of the corresponding inference resources from the heterogeneous cluster, allowing inference to be performed on the request based on these resources, thus avoiding impact on the inference process of large models. Furthermore, the method acquires resource information from the heterogeneous cluster in real time after obtaining the inference request. This allows for a more accurate determination of the inference resources corresponding to large models from the heterogeneous cluster, combining the resources actually needed and those actually available, further preventing impact on the inference process of large models.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, before receiving the inference request, the method further includes: determining the second computing resources required by the large model to infer the first inference data in the inference scenario; determining the inference resources from the heterogeneous cluster based on the first computing resources and the first resource information, including: determining the second inference resources from the heterogeneous cluster based on the second computing resources; and determining the first inference resource from the second inference resources based on the first computing resources and the first resource information. In this way, for different inference scenarios, corresponding resource planning can be performed in advance to determine the corresponding second inference resources, so that after receiving the inference request, a more suitable first inference resource can be further determined from the planned second inference resources, thereby avoiding impact on the inference process of the large model and meeting different inference needs of users.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, the inference scenario includes a first-response scenario, and the second computing resource includes the first computing power resource. Based on the second computing resource, determining the second inference resource from a heterogeneous cluster includes: determining the second inference resource from the heterogeneous cluster based on the first computing power resource and a computing power utilization threshold. This implementation considers that in inference scenarios requiring faster first-response, the NPU used for inference by the large model needs higher computing power resources to ensure faster first response computation and thus a faster first-response. Therefore, it determines the corresponding second inference resource for the first-response scenario based on computing power resources and a computing power utilization threshold. This allows for the identification of a more suitable second inference resource for subsequent determination of the first inference resource, avoiding impact on the inference process of the large model, ensuring the first-response meets requirements, and thus satisfying the inference needs of the user's first-response scenario.

[0010] In conjunction with the first aspect, in one possible implementation of the first aspect, the inference scenario includes an average latency scenario, and the second computing resources include first memory resources and first memory bandwidth resources. The step of determining the second inference resource from the heterogeneous cluster based on the second computing resources includes: determining the second inference resource from the heterogeneous cluster based on the first memory resource, the first memory bandwidth resource, a memory utilization threshold, and a memory bandwidth utilization threshold. This implementation considers that in inference scenarios requiring lower average latency, the NPU needs more memory and bandwidth to ensure faster and more data transmission, resulting in a faster overall response and lower average latency. Therefore, it determines the corresponding second inference resource based on memory resources, memory bandwidth resources, a memory utilization threshold, and a memory bandwidth utilization threshold. This allows for the identification of a more suitable second inference resource for subsequent determination of the first inference resource, avoiding impact on the inference process of large models, ensuring that the average latency meets requirements, and thus satisfying the user's inference needs for average latency.

[0011] In conjunction with the first aspect, in one possible implementation of the first aspect, before receiving the inference request, the method further includes: determining a third computing resource required by the large model to infer the second inference data in the inference scenario, wherein the length of the second inference data is different from the length of the first inference data; correspondingly, determining the second inference resource from the heterogeneous cluster based on the second computing resource, including: determining the second inference resource from the heterogeneous cluster based on the second computing resource and the third computing resource. In this way, corresponding computing resources can be determined for inference data of different lengths, and based on these computing resources, more accurate resource planning can be performed on the heterogeneous cluster, and the corresponding second inference resource can be determined more accurately. This allows for the further determination of a more suitable first inference resource from the second inference resource after receiving the inference request, thereby avoiding impact on the inference process of the large model and meeting different inference needs of users.

[0012] In conjunction with the first aspect, in one possible implementation of the first aspect, the heterogeneous cluster is a heterogeneous neural network processing unit (NPU) cluster. Before receiving an inference request, the method further includes: acquiring second resource information of the heterogeneous NPU cluster, the second resource information including: network interconnection information between multiple NPUs and the storage and computing capabilities of the NPUs; acquiring model structure information of a large model; determining the model deployment overhead of the large model based on the model structure information; determining a second inference resource from the heterogeneous cluster based on the second computing resources, including: determining a third inference resource from the heterogeneous PU cluster based on the second resource information and the model deployment overhead; and determining a second inference resource from the third inference resource based on the second computing resources.

[0013] This implementation can acquire network interconnection information and computing power between multiple NPUs. By combining information about NPUs specifically designed to accelerate neural network computation with the deployment overhead of large models, it can more accurately determine the third inference resource that can be used to perform inference requests. This allows for a more accurate determination of the first inference resource subsequently, avoiding any impact on the model's inference process. Furthermore, the deployment overhead is determined based on the acquired model structure information, enabling a more accurate identification of inference resources for large models with different structures, further preventing any disruption to the model's inference process.

[0014] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: combining multiple NPUs in a heterogeneous NPU cluster to obtain N resource groups, where N is a positive integer; determining a third inference resource from the heterogeneous NPU cluster based on second resource information and model deployment overhead, including: determining available resources corresponding to the N resource groups based on the second resource information; and determining the third inference resource from the N resource groups based on the available resources and model deployment overhead. The third inference resource includes M resource groups, where M is a positive integer and M≤N. By combining multiple NPUs in the heterogeneous NPU cluster, available resources can be determined for each resource group, thereby determining whether each resource group can be used for inference of a large model. This allows for faster determination of the third inference resource in heterogeneous NPU clusters with a large number of NPUs of different models, thus enabling faster determination of the first inference resource.

[0015] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: predicting the inference time required for the large model to infer the inference request based on the first inference resource; if the inference time is less than or equal to a time threshold, inferring the inference request using the large model based on the first inference resource; or, if the inference time is greater than the time threshold, and the inference is set to continue after inference timeout, inferring the inference request using the large model based on the first inference resource. This implementation performs inference on the large model when the inference time is less than or equal to the time threshold, or performs inference on the large model when the user does not mind the inference timeout, thus avoiding the problem of poor user experience caused by inference timeout.

[0016] Secondly, this application provides a resource determination apparatus, the apparatus comprising:

[0017] The receiving module is used to receive inference requests;

[0018] The determination module is used to determine the first computational resources required by a large model when it performs inference on an inference request.

[0019] The acquisition module is used to obtain the first resource information of the heterogeneous cluster.

[0020] The determination module is also used to determine the first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information. The first inference resource is used by the large model to perform inference on the inference request.

[0021] The resource determination apparatus has the function of implementing the resource determination method in the first aspect or any possible embodiment of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software, and the hardware or software includes one or more modules corresponding to the above function.

[0022] The beneficial effects shown in this aspect are similar to those in the first aspect, as detailed in the first aspect, and will not be repeated here.

[0023] Thirdly, embodiments of this application provide a resource determination apparatus. This apparatus can be applied to a terminal device or its internal circuits, chips, chip systems, etc., or to a network device or its internal circuits, chips, chip systems, etc. The apparatus may include at least one processor, which is used to call computer instructions in memory to cause the resource determination apparatus to execute the resource determination method in the first aspect or any optional embodiment of the first aspect, or to execute the resource determination method in the second aspect or any optional embodiment of the second aspect.

[0024] In conjunction with the fourth aspect, in one possible implementation of the fourth aspect, the resource determination device may further include a memory.

[0025] Fifthly, this application provides a computer-readable storage medium that may include instructions that, when executed on a computer, cause the computer to perform the resource determination method in the first aspect or any optional embodiment of the first aspect, or to perform the resource determination method in the second aspect or any optional embodiment of the second aspect.

[0026] Sixthly, this application provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the resource determination method in the first aspect or any optional embodiment of the first aspect, or to execute the resource determination method in the second aspect or any optional embodiment of the second aspect.

[0027] Seventhly, this application provides a chip system including a processor for supporting a device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the device. This chip system may be composed of chips or may include chips and other discrete devices.

[0028] Eighthly, embodiments of this application provide a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to execute the resource determination method in the first aspect or any optional embodiment of the first aspect, or to execute the resource determination method in the second aspect or any optional embodiment of the second aspect. Attached Figure Description

[0029] Figure 1 A schematic diagram illustrating a scenario for determining inference resources, provided as an embodiment of this application;

[0030] Figure 2 A flowchart illustrating a resource determination method provided in an embodiment of this application;

[0031] Figure 3 This application provides a schematic diagram of a scenario for enabling large model inference.

[0032] Figure 4 A flowchart illustrating another resource determination method provided in an embodiment of this application;

[0033] Figure 5 This application provides a schematic diagram of a scenario for determining M resource groups.

[0034] Figure 6 A schematic diagram of a scenario for determining a model instance provided in an embodiment of this application;

[0035] Figure 7 A schematic diagram illustrating another scenario for enabling large model inference, provided as an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the structure of a resource determination device provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of another resource determination device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0039] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The term "at least one" should be understood as one or more, and "at least one" should be understood as one or more. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the relevant concepts involved in the embodiments of this application are explained below.

[0041] Large models refer to deep learning models with a very large number of parameters. They are typically used to process large-scale datasets and are capable of learning complex features of the data. Training large models requires a significant amount of data and computing resources, and takes a considerable amount of time. Therefore, large models usually require high-performance computers and large-scale computing clusters for training. Simultaneously, large models also require various optimization algorithms and techniques to accelerate the training process and improve the model's accuracy. Large models have a wide range of applications. For example, in natural language processing, large models can be used for tasks such as machine translation, text generation, and dialogue systems; in image processing, they can be used for tasks such as image classification, object detection, and image generation; and in speech recognition, they can be used for tasks such as speech recognition, speech synthesis, and speech conversion.

[0042] NPUs (Neural Processing Units) are used to accelerate neural network computations, addressing the inefficiency of traditional chips in neural network operations. The NPU works by simulating human neurons and synapses at the circuit level and directly processing large numbers of neurons and synapses using deep learning instruction sets; a single instruction can process a group of neurons. Compared to CPUs and GPUs, NPUs improve operational efficiency by emphasizing weights to integrate storage and computation.

[0043] A heterogeneous NPU cluster refers to combining multiple NPUs of different types into a single cluster to achieve more efficient parallel computing and load balancing. Such a cluster can include NPUs from different generations, with different computing power, or even from different manufacturers. Through heterogeneous clusters, the strengths of each NPU can be fully utilized, optimizing performance and energy efficiency.

[0044] Heterogeneous GPU clusters refer to a cluster that combines multiple GPUs of different types. Such clusters may include GPUs from different generations or even GPUs from different manufacturers.

[0045] Currently, hardware resources are determined from the heterogeneous cluster by analyzing CPU and / or GPU resource usage. Large-scale model inference is then performed based on these fixed hardware resources. However, different inference requests have varying lengths, requiring different inference resources. If all inference requests of different lengths are performed using the same fixed hardware resources, it will negatively impact the inference process of large-scale models.

[0046] To address this, embodiments of this application provide a resource determination method, apparatus, program product, and storage medium. For inference requests received in real time, the computational resources required for inference of that request are determined in real time. This dynamically determines the computational resources corresponding to inference requests of different lengths, enabling the dynamic determination of the corresponding inference resources based on these computational resources. This achieves more accurate determination of the corresponding inference resources from heterogeneous clusters, thus allowing inference requests based on these resources to avoid impacting the inference process of large models.

[0047] Please see Figure 1 This is a schematic diagram of a scenario for determining inference resources provided in an embodiment of this application. Figure 1 The heterogeneous cluster contains machines 1, 2, 3, 4, 5, 6, 7, and 8. Machines 1 and 2 include NPUs of model 910B1, machines 3 and 4 include NPUs of model 910B2, machines 5 and 6 include NPUs of model 910B3, and machines 7 and 8 include NPUs of model 910B4. From this heterogeneous NPU cluster, servers 1, 2, 3, and 4 are identified as the inference resources corresponding to the large model; that is, machines 1, 2, 3, and 4 can be used for inference of the large model. This is understandable. Figure 1 This is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0048] For details, please refer to Figure 2 , Figure 2This application provides a flowchart illustrating a resource determination method, which can be executed by a resource determination device. The resource determination method provided in this embodiment may include:

[0049] S201, Receive inference request.

[0050] In this embodiment of the application, after the user inputs inference request data, a corresponding inference request is generated, thus enabling the receipt of the corresponding inference request. For example, the inference request data could be "What are the 24 solar terms?", or "There is a cat standing on the table...", or "What are NPU, GPU, and CPU?", etc. It is understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0051] S202. Determine the primary computational resources required by the large model when performing inference on the inference request.

[0052] The first computing resources required may include the required memory resources, the required memory bandwidth resources, the required computing power resources, etc., and this application embodiment does not limit them.

[0053] In one possible implementation, the first computing resource in this application embodiment can be a memory resource. Correspondingly, S202 may include determining the third memory resource required when the large model performs inference on the inference request. For example, determining that the third memory resource required when the large model performs inference on the inference request data "What are the Twenty-Four Solar Terms?" is 10G, or determining that the third memory resource required when the large model performs inference on the inference request data "What are NPU, GPU, and CPU?" is 20G, etc. It is understood that the above is merely illustrative and should not be construed as a limitation on the embodiments of this application.

[0054] S203. Obtain the first resource information of the heterogeneous cluster.

[0055] The heterogeneous cluster in this application embodiment can be a heterogeneous NPU cluster or a heterogeneous GPU cluster, etc., and this application embodiment does not limit it.

[0056] The first resource information of the heterogeneous cluster may include: the remaining memory resources of the heterogeneous cluster, the memory bandwidth resources of the heterogeneous cluster, the computing power resources of the heterogeneous cluster, etc., and this application embodiment does not limit this.

[0057] In one possible implementation, the first resource information in this application embodiment may be the remaining memory resources. Accordingly, S203 may include: obtaining the remaining memory resources of the heterogeneous cluster. For example, the heterogeneous cluster includes machine 1, machine 2, and machine 3. It is determined that the remaining memory resources corresponding to machine 1 are 2GB, the remaining memory resources corresponding to machine 2 are 12GB, and the remaining memory resources corresponding to machine 3 are 5GB. It is understood that the above is merely an illustrative description and should not be construed as a limitation on the embodiments of this application.

[0058] S204. Based on the first computing resource and the first resource information, determine the first inference resource from the heterogeneous cluster.

[0059] The first inference resource is used by the large model to perform inference on inference requests.

[0060] In one possible implementation, if the first computing resource is the third memory resource and the first resource information is the remaining memory, the corresponding first inference resource can be determined by combining the remaining memory resources of multiple machines in the heterogeneous cluster. For example, the heterogeneous cluster includes machine 1, machine 2, and machine 3. Machine 1 has 2GB of remaining memory, machine 2 has 12GB of remaining memory, and machine 3 has 5GB of remaining memory. The third memory resource is 15GB. Machine 2 and machine 3 are then determined as the first inference resources. It should be understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0061] As can be seen, in this embodiment, the computing resources required for inference of a received inference request are determined in real time. This dynamically determines the computing resources corresponding to input requests of different lengths, enabling the dynamic determination of the corresponding inference resources based on these resources. This allows for more accurate determination of the corresponding inference resources from the heterogeneous cluster, thus avoiding impact on the inference process of large models when performing inference based on these resources. Furthermore, the resource information of the heterogeneous cluster is obtained in real time after the inference request is acquired. This allows for a more accurate determination of the inference resources corresponding to large models from the heterogeneous cluster, combining the actual required resources with the available resources, further preventing impact on the inference process of large models.

[0062] Please see Figure 3 This is a schematic diagram illustrating a scenario for enabling large-model inference, provided in an embodiment of this application. Figure 3 exist Figure 2 Based on the provided resource determination method, further explanation of the resource determination method is provided. Figure 3The system allows users to input composition information of heterogeneous clusters that can be used for large-scale model inference. For example, inputting "910A...910B" indicates that NPUs in the 910A...910B series can be used for large-scale model inference, thus obtaining the composition information of the heterogeneous cluster. It is understood that the above is merely an illustrative description and should not be construed as a limitation on the embodiments of this application. After obtaining the composition information of the heterogeneous cluster, the system can dynamically perceive the heterogeneous cluster information in real time, acquiring the specifications and configuration of the heterogeneous cluster, the network bandwidth of the machines in the heterogeneous cluster, and the currently available resources of all NPUs in the heterogeneous cluster, etc.; using the acquired information, the current load-bearing resources of each accelerator chip can be estimated; based on... Figure 3 The system uses the model structure information of the large model input by the user to estimate the storage, network, and computing overhead of deploying the model using theoretical formulas. This storage, network, and computing overhead is equivalent to the model deployment overhead. Then, combined with the available resources of the current heterogeneous cluster and the deployment overhead of the model, multiple resource groups are determined from the heterogeneous cluster. For each resource group, operations research optimization is used to find the number of model instances corresponding to different inference scenarios, allowing users to decide on the final resource group to use based on their preferred inference scenarios. Inference requests are generated based on the inference request data input by the user. Finally, based on the user request and inference settings, the system selects the model instances that meet the requirements within the resource group and generates a dynamic scheduling plan that minimizes overhead to start heterogeneous hybrid inference. Finally, the system outputs the inference result corresponding to the inference request.

[0063] It should be noted that, Figure 3 The user input in input 3 and the user input in input 4 can both be understood as the reasoning scenario selected by the user. Figure 3 Prefill and decoding represent different stages of inference for large models, and are processed using different NPUs corresponding to the same model instance. Different model instances can be selected for processing based on the inference batch size and appropriate thresholds.

[0064] Please see Figure 4 This is a flowchart illustrating another resource determination method provided for the implementation of this application. Figure 4 exist Figure 2 , Figure 3 Based on the provided resource determination method, a more detailed explanation of the resource determination method is given. Figure 4 The resource determination method can be executed by a resource determination device. This application embodiment uses a heterogeneous NPU cluster as an example for illustration. The resource determination method provided in this application embodiment may include:

[0065] S401, Obtain the second resource information of the heterogeneous NPU cluster.

[0066] The second resource information includes information about the NPU and the CPU, etc.

[0067] Information about an NPU can include: network interconnection information between multiple NPUs, NPU computing power, the number of NPUs, etc. Specifically, NPU computing power can include: scalar computing power, vector computing power, matrix computing power, video memory capacity, and Level 2 (L2) cache capacity. Network interconnection information between NPUs can include: interconnection relationships between NPUs, interconnection network bandwidth, interconnection network memory, and information about the Huawei Cache Coherency System (HCCS) ring composed of NPUs, etc. HCCS ring information can include: the number of rings, intra-ring network interconnection information, and inter-ring network interconnection information, etc. HCCS is a protocol for maintaining data consistency across multiple clusters and is used for high-speed interconnection between chips.

[0068] CPU information may include: CPU main memory capacity, CPU Level 1 (L1) cache capacity, etc.

[0069] It should be noted that the NPU and CPU are included in the machine, and a machine can include at least one NPU and at least one CPU. The machine can be a server, etc.

[0070] In this embodiment, the heterogeneous NPU performance analysis tool of the resource determination device can be run on each machine in the heterogeneous NPU cluster. It calls commands from the Linux system, CPU processor, and NPU processor to obtain the computing power of the NPUs on each machine, the number of NPUs, and CPU information, etc. Then, through the testing tool built into the resource determination device, it calls the hccl command and NPU processor commands to obtain network interconnection information between NPUs within each machine, as well as network interconnection information between NPUs on different machines, thereby obtaining the second resource information of the heterogeneous NPU cluster. The hccl (Huawei Collective Communications Library) command is a set of communication library commands for distributed parallel computing, designed to improve the efficiency of distributed training. This solves the problem in existing technologies that cannot obtain multi-dimensional computing power resource information and network communication capabilities of NPUs.

[0071] It should be noted that the real-time resource information of the heterogeneous NPU cluster is obtained based on the heterogeneous NPU performance analysis and testing tools in this application embodiment.

[0072] In this embodiment of the application, the network interconnection information between NPUs and the computing power of NPUs can be obtained. In this way, the relevant information of NPUs designed specifically to accelerate neural network computing can be used to determine inference resources, thereby more accurately determining the inference resources corresponding to large models and avoiding affecting the inference process of the model.

[0073] S402. Obtain the model structure information of the large model.

[0074] In this embodiment, the model structure information of a large model can be obtained based on the model structure information input by the user. This model structure information may include: the total number of layers in the large model, the number of hidden layers, the sequence length, the word segmenter size, the model parallelism degree, etc.

[0075] S403. Based on the model structure information, determine the model deployment cost of the large model.

[0076] The overhead of model deployment can include the amount of computation, communication, and memory required to deploy a large model.

[0077] In this embodiment, a third inference resource can be determined from the heterogeneous cluster based on the second resource information and the model deployment overhead. This third inference resource can then be used to further determine the first inference resource ultimately used to perform inference on the inference request. Furthermore, the model deployment overhead is determined by the acquired model structure information, thus more accurately identifying the inference resources corresponding to large models with different structures and further avoiding any impact on the model's inference process.

[0078] It should be noted that the overhead of deploying the model is fixed and is independent of the size of the inference data, that is, it is independent of the length of the inference request data input by the user, and is regarded as the static overhead of a large model.

[0079] S404. Combine multiple NPUs in a heterogeneous NPU cluster to obtain N resource groups.

[0080] Where N is a positive integer.

[0081] In this embodiment, NPUs in the NPU cluster can be combined based on the type of NPU to obtain N resource groups, or based on the number of NPUs on each machine to obtain N resource groups, etc. The specific combination method is not limited in this embodiment.

[0082] S405. Based on the second resource information, determine the available resources corresponding to the N resource groups.

[0083] In this embodiment of the application, the available resources corresponding to each resource group can be determined based on the resource information of the heterogeneous NPU cluster, such as the number of available NPUs in each resource group, the total available video memory capacity, etc., so as to obtain N groups of available resources corresponding to N resource groups.

[0084] S406. Based on available resources and model deployment overhead, determine the third inference resource from N resource groups.

[0085] The third reasoning resource includes M resource groups, where M is a positive integer and M≤N.

[0086] In this embodiment, the available resources and model deployment costs of N resource groups can be compared. Based on whether the model deployment costs can be handled, the N resource groups are filtered, and only M resource groups that meet the requirements are retained, thus completing the planning of heterogeneous NPU cluster resources.

[0087] Please see Figure 5 This is a schematic diagram of a scenario for determining M resource groups provided in an embodiment of this application. Figure 5 The resource information of the heterogeneous NPU cluster is initially unknown. A resource determination device senses the resources of the heterogeneous NPU cluster in real time, obtaining a network topology map of the cluster. Then, it obtains network interconnection information between NPUs, NPU information, and CPU information. Next, it receives user-inputted model structure information, such as the number of layers, optimizer type, Mixed Expert Models (MoE), encoder / decoder, number of parameters, etc. Based on the NPU and CPU information, combined with the model structure information, it obtains resource group 1, resource group 2, and resource group 3 that meet the requirements. This can be understood as... Figure 5 This is merely an illustrative example and should not be construed as limiting the embodiments of this application. It should be noted that... Figure 5 In this context, Atlas900, Atlas800, Atlas800T, and Atlas300 represent server models. Peripheral Component Interconnect Express (PCIe) is a high-speed serial computer expansion bus standard. MoE is a model type. The input values ​​corresponding to Encoder / Decoder indicate whether the large model uses only an encoder or a decoder, or both.

[0088] As can be seen, in this embodiment of the application, by combining multiple NPUs in a heterogeneous NPU cluster, the available resources corresponding to each resource group can be determined, thereby determining whether the resource group can be used for inference of large models. In this way, in a heterogeneous NPU cluster with a large number of NPUs of different models, the inference resources corresponding to large models can be determined more quickly.

[0089] S407. Determine the second computing resources required by the large model when performing inference on the first inference data in the inference scenario.

[0090] The reasoning scenario may include at least one of the following: first-shot scenario and average latency scenario.

[0091] It should be noted that, in the embodiments of this application, if the inference scenario includes the first-shot scenario, the second computing resource includes the first sub-computing resource required by the large model to infer the first inference data in the first-shot scenario; if the inference scenario includes the average latency scenario, the second computing resource includes the second sub-computing resource required by the large model to infer the first inference data in the average latency scenario; if the inference scenario includes both the first-shot scenario and the average latency scenario, the second computing resource includes the first sub-computing resource required by the large model to infer the first inference data in the first-shot scenario, and the second sub-computing resource required by the large model to infer the first inference data in the average latency scenario.

[0092] It should be noted that the first-response scenario requires a faster first response, while the average latency scenario requires a faster average latency. The first-response scenario refers to the time taken to receive the first word (token) of the response after the user inputs a reasoning request and initiates reasoning; it is also called the first response time. The average latency scenario refers to the average time taken for each word remaining after the user inputs a reasoning request, initiates reasoning, and receives a complete response, excluding the first word.

[0093] In one possible implementation, S407 may include: determining, based on model structure information, the second computational resources required by the large model when performing inference on the data to be inferred in the inference scenario.

[0094] S408. Determine the third computing resources required by the large model when it performs inference on the second inference data in the inference scenario.

[0095] It is understood that the method for determining the third computing resource in this embodiment is similar to the method for determining the second computing resource in S407 above, so it will not be described again here.

[0096] It should be noted that both the first inference data and the second inference data are inference data. The length of the second inference data is different from that of the first inference data. The embodiments of this application determine the different computing resources required when a large model performs inference on inference data of different lengths in an inference scenario.

[0097] The length of the inference data refers to the batch size that can be processed. In this embodiment, the required computing resources for different lengths of inference data can be determined based on actual needs, and the computing resources required for the large model to perform inference on these lengths of inference data in the inference scenario can be determined accordingly. The inference data can be things like "what are included in the 24 solar terms" or "heterogeneous NPU clusters," etc., and this embodiment does not limit its scope.

[0098] S409. Based on the second computing resource and the third computing resource, determine the second inference resource from the third inference resource.

[0099] It is understandable that if it is only necessary to determine the computational resources corresponding to a certain length of inference resources, the second inference resource is determined from the third inference resource based solely on the second computational resource.

[0100] The third reasoning resource includes i resource groups, where i is a positive integer and i ≤ M. The second reasoning resource is determined from the third reasoning resource, which can also be expressed as determining i resource groups from the Mth resource group.

[0101] The following explains how to determine the second inference resource from the third inference resource based on the second computing resource in different inference scenarios, that is, how to determine the i resource groups from the Mth resource group based on the second computing resource.

[0102] In one possible implementation, the inference scenario is the initial scenario, and the second computing resource includes the first computing power resource. Based on the second computing resource, the second inference resource is determined from the third inference resource, including: determining the second inference resource from the third inference resource based on the first computing power resource and a computing power utilization threshold. The first computing power resource includes: first scalar computing power, first vector computing power, first matrix computing power, etc.

[0103] It should be noted that in inference scenarios that require faster first response, the NPU used by the large model for inference needs higher computing power to ensure faster first response calculation and thus faster first response. Therefore, the inference resources corresponding to the first response scenario are determined based on computing power resources.

[0104] Specifically, determining the second inference resource from the third inference resource based on the first computing power resource and computing power utilization threshold may include:

[0105] a1. Based on M resource groups, determine K first hardware computing power combinations.

[0106] Where K is a positive integer.

[0107] In this embodiment, the hardware computing power combination corresponding to each of the M resource groups is determined. For example, the M resource groups include resource group 1 and resource group 2. Resource group 1 includes machine 1 and machine 2, each equipped with one or more NPUs; resource group 2 includes machine 3, machine 4, and machine 5, each equipped with one or more NPUs. Based on resource group 1, hardware computing power combination 1 includes machine 1, hardware computing power combination 2 includes machine 2, hardware computing power combination 3 includes machine 1 and machine 2, and based on resource group 2, hardware computing power combination 4 includes machine 3, hardware computing power combination 5 includes machine 4, hardware computing power combination 6 includes machine 5, hardware computing power combination 7 includes machine 3 and machine 4, hardware computing power combination 8 includes machine 4 and machine 5, hardware computing power combination 9 includes machine 3 and machine 5, and hardware computing power combination 10 includes machine 3, machine 4, and machine 5. It should be understood that the above is merely an illustrative description and should not be construed as a limitation on the embodiments of this application.

[0108] a2. Based on the hardware computing power of the first computing power resource and the hardware computing power combination, determine the utilization rate of the first computing power resource corresponding to the hardware computing power combination.

[0109] In this embodiment, the hardware computing power corresponding to each of the K hardware computing power combinations can be determined, resulting in the K hardware computing powers corresponding to the K hardware computing power combinations. Then, based on the first computing power resource and the K hardware computing powers, the utilization rate of the first computing power resource corresponding to each of the K hardware computing power combinations can be determined. Here, the first computing power resource utilization rate = first computing power resource / hardware computing power. Hardware computing power refers to the NPU's computing power for different floating-point numbers; different types of NPUs have different computing powers for different floating-point numbers. Hardware computing power can include the sum of scalar computing power, vector computing power, matrix computing power, etc.

[0110] a3. Based on the first computing power resource utilization rate and the computing power utilization rate threshold, determine the second inference resource from the third inference resource.

[0111] The computing power utilization threshold can be set according to the actual situation, and this application embodiment does not impose any restrictions on it.

[0112] It should be noted that the second inference resource will include hardware computing power combinations where the utilization rate of the first computing power resource is greater than the computing power utilization threshold.

[0113] In this embodiment of the application, for the first-response scenario, the first computing power resource utilization rate corresponding to each hardware computing power combination is determined, and the second inference resource is determined in combination with the computing power utilization rate threshold. In this way, hardware computing power combinations with poor computing power can be excluded, and hardware combinations with better computing power can be used for model inference, so as to avoid affecting the inference process of large models.

[0114] It should be noted that the embodiments of this application are not limited to the first-sound scenario. The above is one way to determine the second inference resource from the M resource groups. Other methods can also be used to determine the second inference resource from the M resource groups, such as determining any one of the M resource groups as the second inference resource, or determining all M resource groups as the second inference resource, etc.

[0115] As can be seen, in this embodiment of the application, considering that in inference scenarios requiring faster first response, the NPU used by the large model for inference needs higher computing power resources to ensure faster first response calculation and thus faster first response, the inference resources corresponding to the first response scenario are determined based on computing power resources. This can identify more suitable inference resources, avoid affecting the inference process of the large model, ensure that the first response meets the requirements, and thus meet the inference needs of the user's first response scenario.

[0116] In one possible implementation, the inference scenario is an average latency scenario, and the second computing resource includes the first memory resource and the first memory bandwidth resource. Based on the second computing resource, the second inference resource is determined from the third inference resource, including: determining the second inference resource from the third inference resource based on the first memory resource, the first memory bandwidth resource, the memory utilization threshold, and the memory bandwidth utilization threshold.

[0117] It should be noted that in inference scenarios requiring lower average latency, the NPU needs to have more memory and memory bandwidth to ensure faster and more data transmission, resulting in a faster overall response and lower average latency. Therefore, the inference resources corresponding to the average latency are determined based on memory resources and memory bandwidth resources.

[0118] Specifically, determining the second inference resource from the third inference resource based on the first memory resource, the first memory bandwidth resource, the memory utilization threshold, and the memory bandwidth utilization threshold can include:

[0119] b1. Determine the hardware memory and hardware memory bandwidth of the resource groups in the M resource groups.

[0120] In this embodiment of the application, the hardware memory and hardware memory bandwidth corresponding to each of the M resource groups can be determined, thereby obtaining the M hardware memory resources and M hardware memory bandwidth resources corresponding to the M resource groups.

[0121] b2. Based on the first memory resource and hardware memory, determine the first memory utilization rate of the resource group in the M resource groups.

[0122] In this embodiment, based on the first memory resource and M hardware memory units, M first memory utilization rates corresponding to the M resource groups can be determined. Wherein, first memory utilization rate = first memory resource / hardware memory.

[0123] b3. Based on the first memory bandwidth resource and hardware memory bandwidth, determine the first memory bandwidth utilization rate of the resource group in the M resource groups.

[0124] In this embodiment, the utilization rates of the M resource groups can be determined based on the first memory bandwidth resources and the M hardware memory bandwidths. Wherein, the first memory bandwidth utilization rate = first memory bandwidth resources / hardware memory bandwidth.

[0125] b4. Based on the first memory utilization rate, the first memory bandwidth utilization rate, the memory utilization rate threshold, and the memory bandwidth utilization rate threshold, determine the second inference resource from the third inference resource.

[0126] The memory utilization threshold and memory bandwidth utilization threshold can be set according to the actual situation, and this application embodiment does not impose any restrictions on them.

[0127] It should be noted that the second inference resource will include resource groups whose first memory utilization rate is greater than the memory utilization rate threshold and whose first memory bandwidth utilization rate is greater than the memory bandwidth utilization rate threshold.

[0128] In this embodiment of the application, for the average latency scenario, the memory utilization rate and memory bandwidth utilization rate of each resource group are determined, and the inference resources are determined by combining the memory utilization rate threshold and the memory bandwidth utilization rate threshold. In this way, the resource group with better memory utilization rate and memory bandwidth utilization rate can be determined, and the model inference is performed with the appropriate resource group to avoid affecting the inference process of the large model.

[0129] It should be noted that the embodiments of this application are not limited to the average latency scenario. The above is one way to determine the second inference resource from the M resource groups. Other methods can also be used to determine the second inference resource from the M resource groups, such as determining any one of the M resource groups as the second inference resource, or determining all M resource groups as the second inference resource, etc.

[0130] As can be seen, in this embodiment of the application, considering that in inference scenarios requiring lower average latency, the NPU needs to have more memory and bandwidth to ensure faster and more data transmission, resulting in a faster overall response and lower average latency, the inference resources corresponding to the average latency are determined based on memory resources and memory bandwidth resources. This allows for the identification of more suitable inference resources, avoids impacting the inference process of large models, ensures that the average latency meets the requirements, and thus satisfies the inference needs of users with lower average latency.

[0131] It is understood that, for different inference scenarios, the specific method of determining the second inference resource from the third inference resource based on the second and third computing resources is similar to the method of determining the second inference resource from the third inference resource based on the second computing resource described above, and therefore will not be repeated here. In the embodiments of this application, the second computing power resource, the second memory resource, the second memory bandwidth resource, the second computing power resource utilization rate, the second memory utilization rate, and the second memory bandwidth utilization rate corresponding to the third computing resource can be determined.

[0132] See Figure 6 This is a schematic diagram of a scenario for determining a model instance, provided in an embodiment of this application. Figure 6 One of the M resource groups will determine the corresponding model instance for different inference scenarios. It should be noted that the model instance refers to the number of large models deployed on the hardware resources that can support large models for inference. In the initial scenario, a large model can be deployed on a hardware computing power combination, with one hardware computing power combination corresponding to one model instance. In the average latency scenario, a large model can be deployed on a resource group, with one resource group corresponding to one model instance. Figure 6 Resource Group 1 includes: Machine 1 (Atlas 800), Machine 2 (Atlas 900), Machine 3 (Atlas 800T), and Machine 4 (Atlas 800T). Computational power modeling, memory modeling, and memory bandwidth modeling are performed for different inference scenarios. In the initial scenario, there are two usable model instances, and in the average latency scenario, there is one usable model instance. It should be noted that... Figure 6In this context, mbz refers to the batch size that can be processed; 400 TFLOPS and 282 TFLOPS represent hardware computing power; 64GB of memory represents hardware memory; and 1.6T / S of memory bandwidth represents hardware memory bandwidth. In scenarios requiring faster response times, a large model can be deployed on machines 1 and 2, and another large model can be deployed on machines 3 and 4, with different batch sizes for each model. When an inference request is received, the length of the inference request data determines whether to use model instance 1 (with a batch size of 128 on machines 1 and 2) or model instance 2 (with a batch size of 56 on machines 3 and 4) for inference. It should be understood that the above is merely illustrative and should not be construed as a limitation on the embodiments of this application.

[0133] S410, Receive resource selection information.

[0134] The resource selection information is used to indicate the selection of j resource groups from i resource groups, that is, the selection of the fourth inference resource from the second inference resource. The fourth inference resource includes j resource groups, where j is a positive integer and j≤i.

[0135] In this embodiment, the user can select j resource groups from i resource groups according to their needs to generate resource selection information, and thus receive the corresponding resource selection information. It should be noted that the j resource groups are for the same inference scenario.

[0136] In one possible implementation, the information corresponding to i resource groups can be displayed on the interface, such as at least one of the following: processable batch size, hardware computing power, hardware memory, hardware memory bandwidth, etc., so that the user can determine j resource groups from i resource groups.

[0137] S411, Receive inference request.

[0138] It is understood that S411 is the same as S201 in the above embodiments, so it will not be described again.

[0139] S412. Determine the first computational resources required by the large model when it performs inference on the inference request.

[0140] It can be understood that S412 is the same as S202 in the above embodiment, so it will not be described again.

[0141] S413, Obtain the first resource information of the fourth reasoning resource.

[0142] It can be understood that S203 in the above embodiment may include: obtaining the first resource information of the fourth inference resource.

[0143] In one possible implementation, S413 may include obtaining the remaining memory of the fourth inference resource. Specifically, it may obtain the remaining memory of the machine corresponding to each model instance in each of the j resource groups.

[0144] S414. Based on the first computing resource and the first resource information, determine the first reasoning resource from the fourth reasoning resource.

[0145] It can be understood that S204 in the above embodiments may include: determining a fourth resource from the second resource based on resource selection information, and determining a first inference resource from the fourth inference resource based on the first computing resource and the first resource information.

[0146] In this embodiment of the application, if multiple resource groups in the fourth inference resource meet the inference requirements, further screening is required. In one possible implementation, the resource group with the corresponding batch size greater than the length of the inference request and the smallest length difference can be identified as the first inference resource. In this way, the most suitable resource group can be determined for inference of large models by combining the length of the inference request.

[0147] It should be noted that after the first inference resource is determined, a corresponding scheduling plan will be generated.

[0148] S415. Based on the first inference resource, use the large model to perform inference on the inference request.

[0149] In this embodiment, the inference time required for the large model to infer the inference request based on the first inference resource can be predicted. Specifically, if the first inference resource is the resource group corresponding to the first-response scenario, the first response time is determined; if the first inference resource is the average latency scenario, the average time is determined. If the inference time is less than or equal to a timeout threshold, the large model is used to infer the request based on the first inference resource; or, if the inference time is greater than the timeout threshold, and the inference is set to continue after a timeout, the large model is used to infer the request based on the first inference resource. This ensures that inference by the large model proceeds when the inference time is less than or equal to the timeout threshold, or when the user is not concerned about inference timeouts, thus avoiding the poor user experience caused by inference timeouts.

[0150] The time consumption threshold can be input by the user or predefined; this application embodiment does not impose any restrictions on this.

[0151] It should be noted that after reasoning the inference request using a large model, the corresponding inference result will be obtained. For example, if the inference request is "What are the 24 solar terms?", the corresponding inference result will be "Beginning of Spring, Rain Water, Awakening of Insects, Spring Equinox, Pure Brightness, Grain Rain, Beginning of Summer, Lesser Fullness, Grain in Ear, Summer Solstice, Lesser Heat, Greater Heat, Beginning of Autumn, End of Heat, White Dew, Autumn Equinox, Cold Dew, Frost's Descent, Beginning of Winter, Lesser Snow, Greater Snow, Winter Solstice, Lesser Cold, Greater Cold". It is understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0152] Please see Figure 7 This is a schematic diagram of another scenario for enabling large model inference provided in an embodiment of this application. Figure 7 The resource group selected by the user includes two model instances. The user inputs inference request data of length 20, generating a corresponding inference request. Based on this inference request, the corresponding first resource information is determined to require 10G of memory. The available memory of model instance 1 is found to be 2G, and the available memory of model instance 2 is found to be 12G. Model instance 2 is determined to be the first inference resource. The inference time of model instance 2 is predicted and determined to be 1000ms, which is greater than the maximum time limit of 100ms. Since the user sets a timeout to continue, inference is started using model instance 2. The maximum time limit is the time threshold. It should be understood that the above is merely an exemplary description and should not be construed as a limitation on the embodiments of this application.

[0153] As can be seen, this embodiment of the application dynamically perceives and divides the resource groups of the heterogeneous NPU cluster, and generates a dynamic scheduling plan based on the resource groups to achieve load balancing of the heterogeneous NPU cluster and improve the inference efficiency of large models of the heterogeneous NPU cluster. Specifically, compared with homogeneous NPU clusters, achieving inference acceleration for heterogeneous NPU clusters can increase throughput by 1.6 times and reduce average latency by 1.3 times. Moreover, achieving hybrid inference of heterogeneous NPU clusters breaks down the "resource wall" between different chip models, allowing the inference service to efficiently utilize different types of NPUs in the heterogeneous NPU cluster through dynamic scheduling, predicting overhead for different inference scenarios, providing customers with more refined and accurate overhead prediction, and achieving efficient utilization of NPU resources according to the inference scenario. Furthermore, achieving dynamic scheduling for inference acceleration, after the user inputs inference request data, real-time acquisition of resource status and generation of a dynamic scheduling plan, this flexible dynamic scheduling strategy, compared with static scheduling heterogeneous inference technology, can increase throughput by 1.3 times and reduce average latency by 1.2 times.

[0154] Furthermore, the embodiments of this application support real-time resource acquisition for heterogeneous NPU clusters, solving the technical problem that existing technologies cannot obtain information on diverse NPU computing resources and network communication capabilities. This supports subsequent dynamic scheduling technologies for NPU resource group partitioning and load balancing, while also improving the accuracy of inference resource determination. It provides suitable NPU resource groups and corresponding overheads for users to choose from in different inference scenarios, addressing the problem of existing technologies being solely focused on inference scenarios. This improves NPU cluster utilization based on different inference scenarios while meeting diverse user inference needs.

[0155] It should be noted that the embodiments of this application, in addition to being applied to large-model inference scenarios for heterogeneous NPU clusters, can also be applied to operation and maintenance scenarios for heterogeneous NPU clusters. Systematic modeling and analysis of heterogeneous NPU cluster performance, including cluster interconnection information, inter-machine network bandwidth, and NPU information within the heterogeneous NPU cluster, is helpful for cluster performance fault operation and maintenance analysis. Among these, inter-machine network bandwidth includes the speed at which gradient and tensor data are transmitted between two devices, and the NPU information within the heterogeneous NPU cluster includes: the number of NPUs, scalar computing power, vector computing power, matrix computing power, memory capacity, network communication capabilities, etc.

[0156] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0157] To facilitate better implementation of the above-described solutions in the embodiments of this application, related apparatus for implementing the above-described solutions is also provided below.

[0158] Please see Figure 8 The resource determination device 800 provided in this application embodiment may include:

[0159] Receiver module 801 is used to receive inference requests;

[0160] Module 802 is used to determine the first computing resources required by the large model when it performs inference on the inference request.

[0161] Module 803 is used to obtain the first resource information of the heterogeneous cluster.

[0162] The determination module 802 is also used to determine the first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information. The first inference resource is used by the large model to perform inference on the inference request.

[0163] In the apparatus provided in this application, for each inference request received in real time, the computing resources required for inference of that request are determined in real time. This dynamically determines the computing resources corresponding to input requests of different lengths, enabling the dynamic determination of the inference resources corresponding to the inference request based on these computing resources. This achieves more accurate determination of the corresponding inference resources from the heterogeneous cluster, thus allowing inference of the request based on these inference resources to avoid affecting the inference process of large models. Moreover, after obtaining the inference request, the resource information of the heterogeneous cluster is obtained in real time. This allows for a more accurate determination of the inference resources corresponding to large models from the heterogeneous cluster by combining the resources actually needed and the resources actually available, further avoiding impact on the inference process of large models.

[0164] In some embodiments of this application, the determination module 802 in the resource determination device 800 provided in this application is further configured to determine the second computing resources required by the large model when performing inference on the first inference data in the inference scenario before receiving the inference request;

[0165] The determination module 802 is specifically used to determine a second inference resource from the heterogeneous cluster based on the second computing resource; and to determine a first inference resource from the second inference resource based on the first computing resource and the first resource information.

[0166] In some embodiments of this application, the inference scenario includes the first-response scenario, and the second computing resource includes the first computing power resource. The determination module 802 in the resource determination device 800 provided in this application embodiment is specifically used to determine the second inference resource from the heterogeneous cluster based on the first computing power resource and the computing power utilization threshold.

[0167] In some embodiments of this application, the inference scenario includes an average latency scenario, and the second computing resource includes a first memory resource and a first memory bandwidth resource. The determination module 802 in the resource determination device 800 provided in this application embodiment is specifically used to determine the second inference resource from the heterogeneous cluster based on the first memory resource, the first memory bandwidth resource, the memory utilization threshold, and the memory bandwidth utilization threshold.

[0168] In some embodiments of this application, the determination module 802 in the resource determination device 800 provided in this application is further configured to determine the third computing resources required by the large model to infer the second inference data in the inference scenario before receiving the inference request, wherein the length of the second inference data is different from the length of the first inference data; the determination module 802 specifically determines the second inference resources from the heterogeneous cluster based on the second computing resources and the third computing resources.

[0169] In some embodiments of this application, the heterogeneous cluster is a heterogeneous neural network processor cluster. The acquisition module 803 in the resource determination device 800 provided in this application embodiment is further used to acquire second resource information of the heterogeneous neural network processor cluster before receiving the inference request. The second resource information includes: network interconnection information between multiple neural network processors and the computing power of the neural network processors.

[0170] Module 803 is used to obtain the model structure information of large models;

[0171] The determination module 802 is also used to determine the model deployment cost of large models based on model structure information;

[0172] The determination module 802 is specifically used to determine the third inference resource from the heterogeneous neural network processor cluster based on the second resource information and model deployment overhead; and to determine the second inference resource from the third inference resource based on the second computing resource.

[0173] In some embodiments of this application, the resource determination device 800 provided in this application embodiment further includes a combination module, used to combine the neural network processors in the heterogeneous neural network processor cluster to obtain N resource groups, where N is a positive integer;

[0174] The determination module 802 is specifically used to determine the available resources corresponding to N resource groups based on the second resource information; and to determine the third inference resources from the N resource groups based on the available resources and model deployment overhead. The third inference resources include M resource groups, where M is a positive integer and M≤N.

[0175] In some embodiments of this application, the resource determination device 800 provided in this application further includes a prediction module for predicting the inference time required for a large model to infer the inference request based on the first inference resource;

[0176] The inference module is used to perform inference on the inference request using a large model based on the first inference resource if the inference time is less than or equal to the time threshold; or, if the inference time is greater than the time threshold and the inference is set to continue after inference timeout, to perform inference on the inference request using a large model based on the first inference resource.

[0177] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0178] Next, we will introduce another resource determination device provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 9This is a schematic diagram of another resource determination device provided in the embodiments of this application. The resource determination device 900 may include one or more processors 901 and memory 905.

[0179] Memory 905 may include read-only memory and random access memory, and provides instructions and data to processor 901. A portion of memory 905 may also include non-volatile random access memory (NVRAM). Memory 905 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0180] The processor 901 controls the operation of the device. In specific applications, the various components of the resource determination device 900 are coupled together through a bus system, which may include not only a data bus but also a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as the bus system in the diagram.

[0181] The resource determination method disclosed in the above embodiments of this application can be applied to or implemented by the processor 901. The processor 901 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 901 or by instructions in software form. The processor 901 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 901 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 905. Processor 901 reads the information in memory 905 and, in conjunction with its hardware, completes the resource determination steps described above.

[0182] The resource determination device 900 may also include one or more power supplies 902, one or more wired or wireless network interfaces 903, one or more input / output interfaces 904, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0183] In this embodiment of the application, the processor 901 is used to execute... Figure 2 or Figure 4 The resource determination method in the corresponding embodiment. It should be noted that the specific manner in which the processor 901 executes the aforementioned steps differs from that in this application. Figures 1 to 7 The various method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figures 1 to 7 The corresponding method embodiments are the same, and for details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0184] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned actions. Figure 2 or Figure 4 The resource determination method described in the illustrated embodiment.

[0185] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 2 or Figure 4 The resource determination method described in the illustrated embodiment.

[0186] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0187] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0189] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0190] The aforementioned computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A resource determination method, characterized by, The method comprises: receiving an inference request; determining a first computing resource required by a large model for inferring the inference request; obtaining first resource information of a heterogeneous cluster; determining a first inference resource from the heterogeneous cluster based on the first computing resource and the first resource information, the first inference resource being used for the large model to infer the inference request.

2. The method of claim 1, wherein, Before the receiving an inference request, the method further comprises: determining a second computing resource required by the large model for inferring first inference data in an inference scenario; determining a second inference resource from the heterogeneous cluster based on the second computing resource; determining the first inference resource from the second inference resource based on the first computing resource and the first resource information. The inference scenario comprises a first response scenario, and the second computing resource comprises a first computing resource. Determining a second inference resource from the heterogeneous cluster based on the second computing resource comprises:

3. The method of claim 2, wherein, determining a second inference resource from the heterogeneous cluster based on the first computing resource and a computing resource utilization threshold. The inference scenario comprises an average time delay scenario, and the second computing resource comprises a first memory resource and a first memory bandwidth resource. Determining a second inference resource from the heterogeneous cluster based on the second computing resource comprises:

4. The method according to any one of claims 2 to 3, characterized in that, determining a second inference resource from the heterogeneous cluster based on the first memory resource, the first memory bandwidth resource, a memory utilization threshold, and a memory bandwidth utilization threshold. Before the receiving an inference request, the method further comprises:

5. The method according to any one of claims 2 to 4, characterized in that, determining a third computing resource required by the large model for inferring second inference data in an inference scenario, the length of the second inference data being different from the length of the first inference data; determining a second inference resource from the heterogeneous cluster based on the second computing resource and the third computing resource. The heterogeneous cluster is a heterogeneous neural network processor cluster. Before the receiving an inference request, the method further comprises: obtaining second resource information of the heterogeneous neural network processor cluster, the second resource information comprising network interconnection information between a plurality of neural network processors and storage and computing capabilities of the neural network processors; 6. The method according to any one of claims 2 to 5, characterized in that, obtaining model structure information of the large model; determining a model deployment overhead of the large model based on the model structure information; determining a third inference resource from the heterogeneous neural network processor cluster based on the second resource information and the model deployment overhead; determining the second inference resource from the third inference resource based on the second computing resource. The method further comprises: combining the plurality of neural network processors in the heterogeneous neural network processor cluster to obtain N resource groups, N being a positive integer; ​ 7. The method of claim 6, wherein, ​ ​ The determining, based on the second resource information and the model deployment overhead, a third inference resource from the cluster of heterogeneous neural network processors includes: determining available resources corresponding to the N resource groups based on the second resource information; determining the third inference resource from the N resource groups based on the available resources and the model deployment overhead, the third inference resource including M of the resource groups, M being a positive integer and M≤N.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: predicting an inference time consumption required by the large model to infer the inference request based on the first inference resource; if the inference time consumption is less than or equal to a time consumption threshold, inferring the inference request by the large model based on the first inference resource; or, if the inference time consumption is greater than the time consumption threshold and an inference setting is set to continue inference timeout, inferring the inference request by the large model based on the first inference resource.

9. A resource determining apparatus, characterized by comprising: The apparatus includes: a receiving module configured to receive an inference request; a determining module configured to determine a first computing resource required by a large model to infer the inference request; an obtaining module configured to obtain first resource information of a heterogeneous cluster; the determining module is further configured to determine, based on the first computing resource and the first resource information, a first inference resource from the heterogeneous cluster, the first inference resource being used by the large model to infer the inference request.

10. A resource determination device, characterized in that, The apparatus includes at least one processor configured to invoke computer instructions in a memory to cause the resource determination apparatus to perform the resource determination method of any one of claims 1 to 8.

11. A computer program product, characterised in that, The apparatus includes instructions that, when executed on a computer, cause the computer to perform the resource determination method of any one of claims 1 to 8.

12. A computer storage medium, characterized in that The apparatus includes a computer program that, when executed on a computer, causes the computer to perform the resource determination method of any one of claims 1 to 8.

13. A chip, characterized by The apparatus includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the resource determination method of any one of claims 1 to 8.

Images