Heterogeneous-NPU-based training load planning method and related device

Abstract

Disclosed in the embodiments of the present application are a heterogeneous-NPU-based training load planning method and a related device, which are used for performing training load planning for a target model in a heterogeneous server cluster. In the present application, a first server acquires neural network processing unit (NPU) resource information of each server in a server cluster, so as to obtain a summary of the NPU resource information. Next, the first server determines a server combination on the basis of the summary of the NPU resource information, wherein the server combination is used for training a target model, and the server combination comprises one or more servers in the server cluster. Thus, the first server can perform training load planning on the target model on the basis of the server combination, so as to obtain a pipeline parallelism strategy, wherein the pipeline parallelism strategy is used for instructing the partitioning of the target model into a plurality of model blocks, each of the plurality of model blocks comprises at least one layer of neurons from the target model, and each of the plurality of model blocks corresponds to each server in the server combination on a one-to-one basis.

Description

A training load planning method based on heterogeneous NPU and related equipment

[0001] The present application claims priority from the Chinese patent application No. 202410868638.0 filed on June 28, 2024, and entitled "A training load planning method based on heterogeneous NPU and related equipment", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of machine learning, and in particular to a training load planning method based on heterogeneous NPU and related equipment. BACKGROUND

[0003] With the increasing of data scale and parameter scale of large models, huge computing and communication overheads are brought. Therefore, the training of large models can be jointly carried out by multiple servers, which can also be referred to as server clusters.

[0004] Each server in the server cluster is built-in with memory, central processing units (CPU) and neural network processing unit (NPU), etc. Currently, the models of each server in the server cluster are not the same, that is, different servers can have different capacity or performance of memory, CPU and NPU, etc. Such a server cluster is also referred to as a heterogeneous server cluster.

[0005] However, for a heterogeneous server cluster composed of multiple servers of different models built-in with NPU, the memory, bandwidth or computing power of NPU between servers of different models are not the same. Currently, there is a lack of a method for training load planning for the differences between NPU in the server when training a model in such a server cluster. SUMMARY

[0006] The embodiments of the present application provide a training load planning method based on heterogeneous NPU and related equipment for training load planning of a target model in a server cluster based on heterogeneous NPU.

[0007] In a first aspect, the present application provides a training load planning based on heterogeneous NPU and related equipment. In the present application, a first server obtains neural network processing unit (NPU) resource information of each server in a server cluster to obtain NPU resource information summary. The first server is one of the servers in the server cluster. Then, the first server determines a server combination based on the NPU resource information summary. The server combination is used for training a target model, and the server combination includes one or more servers in the server cluster. Then, the first server can perform training load planning on the target model based on the server combination to obtain a pipeline parallel strategy. The pipeline parallel strategy is used to indicate that the target model is divided into a plurality of model blocks. Each model block of the plurality of model blocks includes at least one layer of neurons of the target model. Each model block of the plurality of model blocks corresponds to each server in the server combination one by one.

[0008] Therefore, for a heterogeneous server cluster, the first server can select a server combination based on the NPU resource information summary and assign model blocks in the target model to each server in the server combination. Training load planning is achieved for a heterogeneous server cluster composed of multiple servers of different models with built-in NPU. Therefore, the utilization and / or running efficiency of resources in the server cluster can be improved based on demand.

[0009] In some possible implementation manners, the first server can send NPU resource information collection instructions to each server in the server cluster and receive NPU resource information returned by each server in the server cluster based on the NPU resource information collection instructions to obtain NPU resource information of each server in the server cluster. Thus, the first server obtains NPU resource information of each server in the server cluster.

[0010] In some possible implementation manners, the first server can send NPU resource information collection instructions to each server in the server cluster under the control of a user. The NPU resource information collection instructions are used to instruct the servers to collect their own NPU resource information. In some possible implementation manners, the first server can also send NPU resource information collection instructions to each server in the server cluster periodically. The first server can also send NPU resource information collection instructions to each server in the server cluster before training a large model is needed. Here, no limitation is made.

[0011] In some possible implementation manners, the first server obtains NPU resource information of each server in the server cluster, and after obtaining the NPU resource information summary, the first server can obtain training load information of the target model, the training load information being used to indicate resource overhead required for one training iteration of the target model, and determine whether the server cluster supports the resource overhead indicated by the training load information based on the NPU resource information summary. Then, if yes, the first server performs the step of determining a server combination based on the NPU resource information summary; if no, the first server sends alarm information. Thus, the server cluster that does not support the resource overhead indicated by the training load information is alarmed.

[0012] In some possible implementation manners, the alarm information includes a reason why the server cluster does not support the resource overhead indicated by the training load information, and the reason includes insufficient memory, insufficient bandwidth, and / or insufficient computing power. The user is informed of the reason why the server cluster does not support the resource overhead indicated by the training load information.

[0013] In some possible implementation manners, after the first server obtains the pipeline parallel strategy based on the training load planning of the target model based on the server combination, the first server can optimize other parallel strategies and training framework parameters of the target model based on the pipeline parallel strategy, to obtain optimal parameter configurations, so as to improve the utilization rate and / or running efficiency of resources in the server cluster when training the target model.

[0014] In some possible implementation manners, the NPU resource information includes computing power, bandwidth, and memory of each NPU in the server.

[0015] In some possible implementation manners, the first server can split the target model to obtain a plurality of model blocks, each model block in the plurality of model blocks can include one or more layers of neurons in the target model. Then, each model block in the plurality of model blocks is respectively allocated to a server in the server combination, so that the servers in the server combination correspond one-to-one to each model block in the plurality of model blocks, to obtain a pipeline parallel strategy, the pipeline parallel strategy being used to instruct the servers in the server combination to perform data processing on the allocated model blocks.

[0016] The second aspect of the present application provides a server, comprising: a transceiver module, configured to obtain neural network processing unit (NPU) resource information of each server in a server cluster to obtain NPU resource information summary, the first server being one of the servers in the server cluster; a processing module, configured to determine a server combination based on the NPU resource information summary, the server combination being used for training a target model, the server combination comprising one or more servers in the server cluster; and the processing module is further configured to perform training load planning on the target model based on the server combination to obtain a pipeline parallel strategy, the pipeline parallel strategy being used to indicate that the target model is divided into a plurality of model blocks, each model block of the plurality of model blocks comprising at least one layer of neurons of the target model, and each model block of the plurality of model blocks corresponding to each server in the server combination one by one.

[0017] In some possible implementation manners, the transceiver module is specifically configured to: send an NPU resource information collection instruction to each server in the server cluster; and receive NPU resource information returned by each server in the server cluster based on the NPU resource information collection instruction to obtain the NPU resource information of each server in the server cluster.

[0018] In some possible implementation manners, the processing module is further configured to: obtain training load information of the target model, the training load information being used to indicate resource overhead required for one training iteration of the target model; determine whether the server cluster supports the resource overhead indicated by the training load information based on the NPU resource information summary; if yes, perform the step of determining the server combination based on the NPU resource information summary; and if no, send an alarm information.

[0019] In some possible implementation manners, the processing module is further configured to perform optimization on other parallel strategies and training framework parameters of the target model based on the pipeline parallel strategy to obtain optimal parameter configurations.

[0020] The third aspect of the present application provides a computer readable storage medium, the computer readable storage medium storing instructions, when the instructions are executed on a computer, causing the computer to perform the method provided in the first aspect or any possible implementation manner of the first aspect.

[0021] The fourth aspect of the present application provides a computer program product, the computer program product comprising computer execution instructions stored in a computer readable storage medium; at least one processor of a device can read the computer execution instructions from the computer readable storage medium, and the at least one processor executes the computer execution instructions to cause the device to implement the method provided in the first aspect or any possible implementation manner of the first aspect.

[0022] The fifth aspect of the present application provides a communication device, which can include at least one processor, a memory and a communication interface. The at least one processor is coupled with the memory and the communication interface. The memory is configured to store instructions, the at least one processor is configured to execute the instructions, and the communication interface is configured to communicate with other communication devices under the control of the at least one processor. The instructions, when executed by the at least one processor, cause the at least one processor to perform the method in the first aspect or any possible implementation manner of the first aspect.

[0023] The sixth aspect of the present application provides a chip system, which includes a processor configured to support the functions involved in the first aspect or any possible implementation manner of the first aspect.

[0024] In a possible design, the chip system can further include a memory configured to store necessary program instructions and data. The chip system can be composed of a chip, or can include the chip and other discrete devices.

[0025] The technical effects brought by the second to sixth aspects or any possible implementation manner thereof can be referred to the technical effects brought by the first aspect or different possible implementation manners of the first aspect, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0026] FIG. 1 is a schematic diagram of a composition structure of a heterogeneous server cluster to which embodiments of the present application are applied;

[0027] FIG. 2-1 is a schematic diagram of a training load planning method based on a heterogeneous NPU provided by an embodiment of the present application;

[0028] FIG. 2-2 is a schematic diagram of a cluster topology provided by an embodiment of the present application;

[0029] FIG. 2-3 is a schematic diagram of server combination provided by an embodiment of the present application;

[0030] FIG. 2-4 is a schematic diagram of n-layer neurons provided by an embodiment of the present application;

[0031] FIG. 2-5 is a schematic diagram of a pipeline parallel strategy provided by an embodiment of the present application;

[0032] FIG. 3 is a schematic diagram of a structure of a server provided by an embodiment of the present application;

[0033] FIG. 4 is a schematic diagram of a structure of a communication device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] Embodiments of the present application provide a training load planning method based on a heterogeneous NPU and related equipment, which is used for training load planning of a target model in a server cluster based on a heterogeneous NPU.

[0035] The embodiments of the present application will be described below in conjunction with the drawings, obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Those skilled in the art can know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0036] The terms "first", "second", and the like in the specification of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological order. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules does not necessarily limit to those steps or modules clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps appearing in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the steps that have been named or numbered can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0037] The embodiments of the present application relate to many related knowledge, in order to better understand the scheme of the embodiments of the present application, the related terms and concepts that may be involved in the embodiments of the present application will be introduced first. It should be understood that the explanation of related concepts may be limited due to the specific situation of the embodiments of the present application, but it does not mean that the present application can only be limited to this specific situation, and the specific situation of different embodiments may also be different, and the specific situation is not limited here.

[0038] The present application can be applied to a heterogeneous server cluster, which includes a plurality of servers, each server including memory, storage, network card, CPU, NPU and other components, wherein the models of each server in the plurality of servers are not the same.

[0039] The server needs to respond to service requests and process them to provide reliable services. In general, the server should have the ability to carry and guarantee services, and the server needs to have strong processing capability, high stability, high reliability, high security, scalability and manageability. In the embodiments of the present application, the server can be an x86 server, which is also called a complex instruction set computer (CISC) architecture server, i.e. the commonly used personal computer (PC) server. It is based on the PC architecture, uses an Intel or other compatible x86 instruction set processor chip and a Windows operating system server.

[0040] The server can have a large difference due to different configurations or performances, and can include at least one central processing unit (CPU) (for example, at least one processor) and a memory, at least one storage medium (for example, at least one mass storage device) storing application programs or data. The programs stored in the storage medium can include at least one module, and each module can include a series of instruction operations in the server. Further, the central processing unit can be configured to communicate with the storage medium and execute the series of instruction operations in the storage medium on the server. The server can also include at least one power supply, at least one wired or wireless network interface, at least one input / output interface, and / or at least one operating system, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, NetWare, etc.

[0041] Optionally, the CPU can use a general central processing unit, a microprocessor, an application specific integrated circuit (ASIC) or one or more integrated circuits to execute related programs to implement the technical solutions provided in the present application, which is not limited here.

[0042] The memory can be a random access memory (RAM), a read only memory (ROM) or the like storage medium. The memory is used to store data, and the NPU can obtain data from the memory, store data into the memory, so that the CPU calculates the obtained data in the memory, and returns the results obtained after calculation to the memory.

[0043] The memory can be one or more non-volatile memories, such as one or more disk memories, flash memories, or other non-volatile memories. The memory can store program codes, which can be copied into the memory by the CPU, so that the CPU can execute the corresponding application processes or operating systems by executing the program codes in the memory.

[0044] The network card has a media access control (MAC) address, and the network card corresponds to the NPU one-to-one. The network card is used to realize the communication between the NPU and other NPUs in the server cluster.

[0045] The NPU is a processor specially designed for accelerating neural network calculation. The working principle of the NPU is to simulate human neurons and synapses at the circuit layer, to realize the fast processing and analysis of data by simulating the connection and signal transmission process between neurons. Compared with the GPU, the NPU uses its specially designed hardware structure to execute neural network algorithms, and has obvious advantages in the processing efficiency of deep learning. It should be noted that the NPU is built-in with a communication module and a memory module. The communication module enables the NPU to have a certain bandwidth, and the memory module enables the NPU to have a certain capacity of memory.

[0046] It should be noted that multiple NPUs can be built-in in the server, for example, 8 NPUs. Among them, a corresponding network card can be configured for each NPU, which is used to realize the communication between the corresponding NPU and other NPUs outside the server. For example, the NPU can be connected to the switch / router through the corresponding network card, and communicate with the NPU connected to other servers through the switch / router. The different NPUs in the server can communicate through the mainboard built-in in the server.

[0047] As shown in FIG. 1, the heterogeneous server cluster includes a first server and a second server. The first server and the second server are different in model, and the first server includes CPU1, memory 1, storage 1, eight network cards (network cards A1-A8) and eight NPUs (NPUs A1-A8). The eight network cards and the eight NPUs are one-to-one correspondence, and any two NPUs of the eight NPUs communicate through the motherboard built in the first server. Each network card in the eight network cards is used to realize communication between the corresponding NPU and the NPU of other servers except the first server. The second server includes CPU2, memory 2, storage 2, four network cards (network cards B1-B4) and four NPUs (NPUs B1-B4). The four network cards and the four NPUs are one-to-one correspondence, and any two NPUs of the four NPUs communicate through the motherboard built in the second server. Each network card in the four network cards is used to realize communication between the corresponding NPU and the NPU of other servers except the second server. Therefore, the NPU usually has two bandwidths, which are referred to as bandwidth 1 and bandwidth 2. When the NPU communicates with the NPU of other servers through the switch, the bandwidth of the NPU is bandwidth 1. When the NPU communicates with the NPU of the server through the motherboard, the bandwidth of the NPU is bandwidth 2.

[0048] With the increasing data size and parameter size of large models, huge computing and communication overheads are brought. Therefore, the training of large models can be jointly carried out by multiple servers, and the multiple servers can also be referred to as a server cluster.

[0049] Each server in the server cluster is built-in with memory, central processing units (CPUs) and neural network processing units (NPUs), etc. At present, the models of the servers in the server cluster are not the same, that is, different servers can have different capacity or performance of memory, CPUs and NPUs, etc. Such a server cluster is also referred to as a heterogeneous server cluster.

[0050] However, for a heterogeneous server cluster composed of multiple servers of different models built-in with NPUs, the memory, bandwidth or computing power of the NPUs between the servers of different models are not the same. At present, there is a lack of a method for training load planning based on heterogeneous NPUs for the differences between the NPUs in the servers when training a model in such a server cluster.

[0051] Therefore, an embodiment of the present application provides a training load planning based on heterogeneous NPUs and related equipment for training load planning of a target model in a server cluster based on heterogeneous NPUs.

[0052] As shown in FIG. 2-1, the training load planning method based on the heterogeneous NPU provided by the embodiments of the present application mainly includes the following steps:

[0053] 201. The first server sends an NPU resource information collection instruction to the second server.

[0054] In some possible implementation ways, the first server can send the NPU resource information collection instruction to the second server under the control of the user, and the NPU resource information collection instruction is used to instruct the server to collect the NPU resource information of itself. In some possible implementation ways, the first server can also send the NPU resource information collection instruction to the second server regularly, and the first server can also send the NPU resource information collection instruction to the second server before the training of the large model is needed, which is not limited here.

[0055] It should be noted that the first server is one of the servers in the server cluster, that is, each server in the server cluster includes the first server. In the embodiments of the present application, the second server is used to represent each server in the server cluster. Therefore, the second server and the first server can be the same server, and the second server and the first server can be different servers, which are not limited here.

[0056] In some possible implementation ways, the first server can be connected to a switch, and the first server can send the NPU resource information collection instruction to the servers other than the first server in the server cluster through the switch, which is not limited here.

[0057] 202. The second server obtains the NPU resource information of the second server based on the NPU resource information collection instruction.

[0058] In the embodiments of the present application, when the second server receives the NPU resource information collection instruction, the second server can obtain the NPU resource information of the second server based on the NPU resource information collection instruction. In some possible implementation ways, the second server can call the corresponding command in the Linux system to obtain the NPU resource information of the second server.

[0059] In some possible implementation ways, the NPU resource information includes the number of NPUs, the bandwidth (two kinds of bandwidth), the memory capacity and the computing power of each NPU, wherein the computing power includes scalar computing power, vector computing power, matrix computing power, etc., which are not limited here. In some possible implementation ways, after the second server obtains the NPU resource information of the second server, the second server can store the NPU resource information in a designated text.

[0060] 203. The first server obtains the NPU resource information of the second server from the second server.

[0061] In the embodiments of the present application, when the second server obtains the NPU resource information of the second server, if the second server is a server other than the first server, the second server can send the NPU resource information to the first server through the switch. For example, the second server can send a specified text to the first server, and the specified text stores the NPU resource information of the second server. If the second server is the first server, the first server can directly read the specified text from the first server and obtain the NPU resource information of the first server from the specified text.

[0062] It should be noted that, since the second server is used to represent each server in the server cluster, through step 203, the first server obtains the NPU resource information of each server in the server cluster.

[0063] 204. The first server aggregates the NPU resource information of each server in the server cluster to obtain an NPU resource information aggregation.

[0064] The NPU resource information aggregation includes the NPU resource information of each server in the server cluster. In some possible implementation manners, the NPU resource information aggregation can be in the form of a topology graph, i.e., a cluster topology graph. It should be noted that the cluster topology graph includes a plurality of servers, each server includes at least one NPU. Two servers can communicate through a network, two NPUs belonging to the same server can communicate through a motherboard, and two NPUs belonging to different servers can communicate through a network. Therefore, when the server is selected, each NPU of the server can be displayed; when the NPU is selected, the NPU resource information of the NPU can be displayed.

[0065] For example, FIG. 2-2 is a schematic diagram of a cluster topology graph. As shown in FIG. 2-2, the server cluster includes server 1, server 2 and server 3, and server 1-3 are servers of different models and communicate through a network. Among them, each of servers 1-3 is internally provided with 8 NPUs, wherein the bandwidth of each NPU in server 1 for transmitting data through a switch is 100 gigabytes (GB), the bandwidth of each NPU in server 2 for transmitting data through a switch is 200 GB, and the bandwidth of each NPU in server 3 for transmitting data through a switch is 400 GB.

[0066] It should be noted that the NPUs belonging to the same server can be transmitted through the motherboard, and the bandwidth of the motherboard of each server available for transmitting data is not the same. For example, the bandwidth of the motherboard of server 1 is 500 GB, the bandwidth of the motherboard of server 2 is 1000 GB, and the bandwidth of the motherboard of server 3 is 800 GB. Then, if the NPU transmits the operation result to the NPU of the server where the NPU is located, the bandwidth of the motherboard of the server can be used; if the NPU transmits the operation result to the NPU of another server other than the server where the NPU is located, the bandwidth provided by the switch can be used.

[0067] 205. The first server obtains training load information of the target model.

[0068] In some possible implementation manners, the target model is a model input by a user in the first server, or a model obtained by another device in the first server, which is not limited herein. The target model can be a model for visual recognition, a model for language recognition, or a model with other functions, which is not limited herein. In the embodiments of the present application, the target model includes multiple layers of neurons, and each layer of neurons includes multiple neurons.

[0069] It should be noted that the target model obtains input data, processes the input data through each layer of neurons in the multiple layers of neurons, and obtains output data, which is referred to as one training iteration, and the resource consumption (NPU memory consumption, communication consumption, computing power consumption, etc.) required to complete one training iteration is the training load information. In some possible implementation manners, the resource consumption of one training iteration of the target model can be calculated by obtaining the number of operators in the target model and the dimensions of the input data and the output data, and the training load information of the target model is used to indicate the resource consumption of one training iteration of the target model. The computing power of the NPU can include scalar computing power, vector computing power, and matrix computing power.

[0070] 206. The first server determines whether the server cluster supports the resource consumption indicated by the training load information based on the NPU resource information summary.

[0071] In some possible implementation manners, if the server cluster supports the resource consumption indicated by the training load information, the following steps 208-209 or steps 208-210 are continued to be executed; if the server cluster does not support the resource consumption indicated by the training load information, step 207 is executed.

[0072] In some possible implementation manners, the NPU resource information summary includes multiple resources, such as memory, computing power and bandwidth of each NPU. Then, a sum of a value of a certain resource of each NPU in the NPU resource information is calculated to obtain a total sum of the certain resource, such as a total sum of computing power, a total sum of memory, and a total sum of bandwidth. If the total sum of each resource in the NPU resource information summary is greater than the resource consumption of each resource indicated by the training load information, it is determined that the server cluster supports the resource consumption indicated by the training load information. Then, if the total sum of a certain resource in the NPU resource information summary is less than the resource consumption of the certain resource indicated by the training load information, it is determined that the server cluster does not support the resource consumption indicated by the training load information. The total sum of memory is the total sum of memory of each NPU in the NPU resource information summary, the total sum of computing power is the total sum of computing power of each NPU in the NPU resource information summary, and the total sum of bandwidth is the total sum of bandwidth of each NPU in the NPU resource information summary.

[0073] In some possible implementation manners, in order to maintain the redundancy of each resource in the server cluster, the first server can also first multiply the total sum of each resource in the NPU resource information summary by a coefficient (the coefficient is between 0 and 100%, for example, 90%), and then, if the total sum of each resource in the NPU resource information summary multiplied by the coefficient is greater than the resource consumption of each resource indicated by the training load information, it is determined that the server cluster supports the resource consumption indicated by the training load information. Then, if the total sum of a certain resource in the NPU resource information summary multiplied by the coefficient is less than the resource consumption of the certain resource indicated by the training load information, it is determined that the server cluster does not support the resource consumption indicated by the training load information.

[0074] 207. The first server sends alarm information.

[0075] In some possible implementation manners, when the first server determines, based on the NPU resource information summary, that the server cluster does not support the resource consumption indicated by the training load information, the first server can send alarm information. In some possible implementation manners, the alarm information can include the reason why the server cluster does not support the resource consumption indicated by the training load information. For example, the reason can be insufficient memory, insufficient bandwidth and / or insufficient computing power. Herein, the reason is not limited.

[0076] 208. The first server determines, based on the NPU resource information summary, a server combination for training the target model, the server combination including one or more servers in the server cluster.

[0077] In some possible implementation manners, the first server can traverse all server combinations in the server cluster, and obtain a plurality of candidate server combinations, where a sum of resources of each of the server combinations is sufficient to support the resource consumption indicated by the training load information, so that the user selects a suitable server combination from the candidate server combinations. In some possible implementation manners, the suitable server combination can maximize utilization of each resource of each server in the server combination, or the suitable server combination can minimize a time length for training the target model, or the suitable server combination can minimize a cost for training the target model, or the suitable server combination can minimize energy consumption for training the target model, which is not limited herein.

[0078] For example, as shown in FIGS. 2-3, the server combination selected by the user includes server 1, server 2, server 3, and server 4, where server 1 and server 2 are connected to switch 1, server 3 and server 4 are connected to switch 2, switch 1 and switch 2 are communicatively connected through a network, servers 1-4 belong to the server cluster, and the server combination supports the resource consumption indicated by the training load information.

[0079] 209. The first server performs training load planning on the target model based on the server combination, and obtains a pipeline parallel strategy, where the pipeline parallel strategy is used to indicate that the target model is divided into a plurality of model blocks, each model block of the plurality of model blocks includes at least one layer of neurons of the target model, and each model block of the plurality of model blocks corresponds to each server in the server combination in a one-to-one manner.

[0080] In some possible implementation manners, the first server can divide the target model to obtain a plurality of model blocks, where each model block of the plurality of model blocks can include one or more layers of neurons of the target model. Then, each model block of the plurality of model blocks is respectively assigned to a server in the server combination, so that each server in the server combination corresponds to each model block of the plurality of model blocks in a one-to-one manner, thereby obtaining a pipeline parallel strategy, where the pipeline parallel strategy is used to indicate that each server in the server combination performs data processing on the assigned model block.

[0081] For example, the target model includes a plurality of layers of neurons, and each layer of neurons has the same resource consumption. The matrix dimension of input and output of each layer of neurons is (b, s, h), where b represents a batch size, s represents a sequence length, and h represents a hidden size. Then, the first server calculates φ = b * s * h (φ is a computing power consumption) and C = 6 * b * s * h 2(C = communication overhead), M = 3*b*h*s (M is memory overhead), then the resource overhead of the 1 layer neuron in the target model is: φ, C, M.

[0082] Then, solve:

[0083] min{∑ k i＝1 (max Ni ((φ*Ni) / (Si+Vi+Ci))+max Ni (C*Ni) / Oi)}}

[0084] Wherein, Si, Vi, Ci, Oi are the scalar computing power, vector computing power, matrix computing power and bandwidth of server i (i = 1, 2, …, k; k is the number of servers in the server combination) in the server combination respectively. φ*Ni represents the total amount of computing power overhead of Ni model blocks carried by server i, (Si+Vi+Ci) represents the sum of the scalar computing power overhead, vector computing power overhead and matrix computing power overhead of all NPUs in server i, (φ*Ni) / (Si+Vi+Ci) represents the time required to complete the task amount (φ*Ni) with the total amount of computing power resources (Si+Vi+Ci) of all NPUs in server i, max Ni ((φ*Ni) / (Si+Vi+Ci)) represents the value of Ni under the limit of the resource overhead that can be carried by the total amount of computing power resources of all NPUs in server i.

[0085] And C*Ni represents the total amount of communication overhead of Ni layer neurons, (C*Ni) / Oi represents the time required to complete the task amount (C*Ni) with the total amount of bandwidth Oi of all NPUs in server i, max Ni (C*Ni) / Oi) represents the value of Ni under the limit of the resource overhead that can be carried by the total amount of bandwidth of all NPUs in server i.

[0086] Then, min{∑ k i＝1 (max Ni ((φ*Ni) / (Si+Vi+Ci))+max Ni (C*Ni) / Oi)}} represents the minimum time consumption under the maximum utilization of each server in the server combination, so as to obtain Ni (i = 1, 2, …, k), that is, the optimal number of layers Ni (i = 1, 2, …, k; N1+…+Nk=n; n is the number of layers of neurons of the target model, k is the number of servers in the server combination) of neurons allocated to each server in the server combination.

[0087] Then, in the case that the total memory overhead of the Ni layer neurons assigned by the server i cannot exceed the memory mi of the server i (i.e., M Ni≤ mi, M is the memory overhead required by one layer neuron), the optimal number of layers Ni (layer neurons) assigned to the server i can be determined by solving the above formula, so as to obtain the optimal training load planning of the target model in the server combination, and the pipeline parallel strategy is obtained.

[0088] For example, as shown in FIGS. 2-4, the target model includes n layer neurons, and the server combination includes k servers. The pipeline parallel strategy includes: assigning the first to third layer neurons to the server 1, assigning the fourth layer neuron to the server 2, assigning the fifth to seventh layer neurons to the server 3, assigning the eighth to eleventh layer neurons to the server 4, assigning the twelfth to fourteenth layer neurons to the server 5, assigning the fifteenth to seventeenth layer neurons to the server 6, …, assigning the (n-5) to (n-3) layer neurons to the server (k-1), and assigning the (n-2) to n layer neurons to the server k.

[0089] For another example, as shown in FIGS. 2-5, the target model includes 10 layer neurons, and the server combination includes 4 servers. Then, the pipeline parallel strategy includes: assigning the first layer neuron to the server 1, assigning the second layer neuron to the server 2, assigning the third to seventh layer neurons to the server 3, and assigning the eighth to tenth layer neurons to the server 4.

[0090] 210. The first server optimizes other parallel strategies and training framework parameters of the target model based on the pipeline parallel strategy to obtain the optimal parameter configuration.

[0091] In some possible implementation manners, the first server can first determine the number of model parallel (TP) based on the pipeline parallel strategy, i.e., the number of model blocks into which the target model is divided. For example, if the target model is divided into k model blocks, the number of model parallel (TP) is k.

[0092] In some possible implementation manners, after determining the pipeline parallel strategy of the target model, the first server can further select other parallel strategies (such as data parallel (DP), expert parallel (EP), etc., which are not limited herein) and configurations of training framework parameters (such as epoch_size, sink_size, etc., which are not limited herein) in the process of performing operation processing on the assigned model blocks in each server of the server combination.

[0093] In some possible implementations, the first server can combine the resources remaining after each server in the server group carries the split target model to obtain remaining NPU resource information summary. Then, the first server enumerates all possible training framework parameters based on the remaining NPU resource information summary, and other parallel strategies and training framework parameters are referred to as a group of parameter configurations, thereby obtaining candidate multiple groups of parameter configurations. Each group of parameter configurations includes multiple parameters (other parallel strategies and training framework parameters), and the values of the parameters in different groups are different (for example, the value of parameter 1 in parameter configuration 1 is the same as the value of parameter 1 in parameter configuration 2, and the value of parameter 2 in parameter configuration 1 is different from the value of parameter 2 in parameter configuration 2).

[0094] For example, the obtained candidate p groups of parameter configurations include:

[0095] Parameter configuration 1: (EP = X1, TP = Y1,...)

[0096] Parameter configuration 2: (EP = X2, TP = Y2,...)

[0097] …

[0098] Parameter configuration p: (EP = Xp, TP = Yp,...)

[0099] Then, the first server can also use the following steps of the cooperative optimization technique to screen out the best parameter configuration (for example, the parameter configuration with the minimum training time consumption or the best performance) for the candidate multiple groups of parameter configurations generated above.

[0100] Step 1: rule-based screening.

[0101] First, based on the preset rules, suitable values are screened out for a part of parameters.

[0102] Step 2: cost-based screening.

[0103] For all candidate multiple groups of parameter configurations, the cost required for training the target model is calculated. For example, the cost is the total cost of computing power, the total cost of memory, and the total cost of communication, and the comprehensive total cost is obtained by summation, and at least one group of parameter configurations with a smaller comprehensive total cost is screened out.

[0104] Step 3: test-based screening.

[0105] For example, after running a training iteration for each group of parameter configurations of at least one group of parameter configurations screened in step 2, the performance (for example, time consumption) of each group of parameter configurations is recorded, and only the parameter configuration with the best performance (for example, the minimum time consumption) is retained.

[0106] For example, by the above three steps, the best parameter configuration is parameter configuration 2: (EP=X2, TP=Y2,...) for the candidate p-group parameter configurations.

[0107] For example, for the candidate p-group parameter configurations, the best parameter configuration can be selected by sequentially performing rules base optimization (RBO), cost base optimization (CBO) and test base optimization (TBO).

[0108] 1. RBO.

[0109] By RBO, a preset rule can be set for a specified parameter, so that the value of the required parameter is selected by the preset rule. For example, the default value of the parameter configuration is:

[0110] Param 1=0

[0111] Param 2=-1

[0112] …

[0113] Param q=true

[0114] By setting preset rules for Param 1, Param 2 and Param q, respectively, rule1 and rule2 are obtained.

[0115] Rule 1: base on…change Param 1 to 10 / / “…” indicates the specific content of rule1, which may be related to the input value of the parameter, for example, take 0 if less than 0, otherwise take 10

[0116] Rule 2: base on…change Param 2 to 0 / / The specific content of rule2 and the specific content of rule1 can be the same or different, which is not limited here

[0117] Rule q: base on…set param n to false

[0118] Then, the value of the parameter selected by RBO output is:

[0119] Param 1=10

[0120] Param 2=0

[0121] … / / other parameters except Param 1, Param 2 and Param q are not screened

[0122] Param q = false

[0123] 2, CBO.

[0124] Through RBO, the cost of the calculation running under the specified parameter can be specified, so as to screen the value of the parameter with lower cost. For example, the cost of Param 3 and Param 4 is calculated respectively, so as to screen the value of the parameter with lower cost, and the initial value of Param 3 and Param 4 is:

[0125] Param 3 = 1000

[0126] Param 4 = 100

[0127] Setting F1() and F2() to calculate the cost of Param 3 and Param 4 respectively, that is:

[0128] F1(Param 3) = a Param 3 + β / / F1() is used to calculate the cost of Param 3, and a and β can be set

[0129] F2(Param 4) = λ 2 Param 4 + p 2 / / F2() is used to calculate the cost of Param 4, and λ and p can be set

[0130] Therefore, the lower cost of Param 3 and Param 4 is screened as:

[0131] Param 3 = 64 or 128 / / that is, when Param 3 = 64 or Param 3 = 128, the cost is equal, and both are the lowest

[0132] Param 4 = 99 or -99 / / that is, when Param 4 = 99 or Param 4 = -99, the cost is equal, and both are the lowest

[0133] 3, TBO.

[0134] Through TBO, the value of the loss function based on p groups of parameter configurations can be calculated, and the group parameter configuration with the lowest value of the loss function is taken as the best parameter configuration.

[0135] Through RBO and CBO, the parameter configuration obtained is:

[0136] Param 1 = 10

[0137] Param 2 = 0

[0138] Param 3 = 64 or 128

[0139] Param 4 = 99 or -99

[0140] …

[0141] Param q = false

[0142] That is, 4 sets of parameter configurations are obtained, in which the other parameters are equal except for Param 3 and Param 4, and the 4 sets of parameter configurations can be represented as:

[0143] Param set 1 = {Param 3 = 64; Param 4 = 99}

[0144] Param set 2 = {Param 3 = 64; Param 4 = -99}

[0145] Param set 3 = {Param 3 = 128; Param 4 = 99}

[0146] Param set 4 = {Param 3 = 128; Param 4 = -99}

[0147] Then, each set of parameter configurations is trained by model() for one iteration to obtain the corresponding output result output, and the loss function is used to calculate the output result of each set of parameter configurations:

[0148] Model(param set 1) = output1, f loss(output1) / / The lower the value of f loss(), the better

[0149] Model(param set 2) = output2, f loss(output2)

[0150] Model(param set 3) = output3, f loss(output3)

[0151] Model(param set 4) = output4, f loss(output4)

[0152] It should be noted that the lower the value of f loss(), the better the corresponding parameter configuration.

[0153] For example, by TBO, the f loss (output2) of Param set 2 is the lowest, that is, Param set 2 is the optimal parameter configuration, that is, the following is represented:

[0154] Param 1 = 10

[0155] Param 2 = 0

[0156] Param 3 = 64

[0157] Param 4 = -99

[0158] …

[0159] Param q = false

[0160] In the present application, for a heterogeneous server cluster, a first server can select a server combination based on NPU resource information summary and assign each server in the server combination a model block in a target model, thereby realizing training load planning for a heterogeneous server cluster composed of multiple servers of different models with built-in NPUs, so as to improve the utilization and / or running efficiency of resources in the server cluster based on demand.

[0161] It should be noted that, for each method embodiment described above, in order to simplify the description, a series of action combinations are described, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0162] In order to better implement the above-mentioned scheme of the embodiments of the present application, the related device for implementing the above-mentioned scheme is also provided.

[0163] Please refer to FIG. 3, the server 300 provided by the embodiments of the present application can include:

[0164] The transceiver module 301 is configured to obtain the neural network processing unit (NPU) resource information of each server in a server cluster, and obtain an NPU resource information summary, wherein the first server is one of the servers in the server cluster.

[0165] The processing module 302 is configured to determine a server combination based on the NPU resource information summary, wherein the server combination is used for training a target model, and the server combination includes one or more servers in the server cluster.

[0166] The processing module 302 is further configured to perform training load planning on the target model based on the server combination, to obtain a pipeline parallel strategy, where the pipeline parallel strategy is used to indicate that the target model is divided into a plurality of model blocks, each model block of the plurality of model blocks includes at least one layer of neurons of the target model, and each model block of the plurality of model blocks corresponds to each server of the server combination in one-to-one correspondence.

[0167] In some possible implementation manners, the transceiver module 301 is specifically configured to:

[0168] send an NPU resource information collection instruction to each server in the server cluster; and receive NPU resource information returned by each server in the server cluster based on the NPU resource information collection instruction, to obtain NPU resource information of each server in the server cluster.

[0169] In some possible implementation manners, the processing module 302 is further configured to:

[0170] obtain training load information of the target model, where the training load information is used to indicate resource consumption required for one training iteration of the target model; determine whether the server cluster supports the resource consumption indicated by the training load information based on the NPU resource information summary; if yes, perform the step of determining a server combination based on the NPU resource information summary; and if no, send an alarm information.

[0171] In some possible implementation manners, the processing module 302 is further configured to perform optimization on other parallel strategies and training framework parameters of the target model based on the pipeline parallel strategy, to obtain optimal parameter configurations.

[0172] It should be noted that the content of information interaction and execution process between the modules / units of the apparatus is based on the same concept as the method embodiments of the present application, and the technical effects brought by the content are the same as those of the method embodiments of the present application. For details, refer to the description of the method embodiments of the present application.

[0173] The embodiments of the present application further provide a computer storage medium, where the computer storage medium stores a program, and the program performs part or all of the steps recorded in the method embodiments.

[0174] Next, another communication apparatus provided by the embodiments of the present application is introduced. As shown in FIG. 4, the communication apparatus 400 includes:

[0175] A receiver 401, a transmitter 402, a processor 403, and a memory 404. In some embodiments of the present application, the receiver 401, the transmitter 402, the processor 403, and the memory 404 can be connected through a bus or other means, where the connection through a bus is taken as an example in FIG. 4.

[0176] The memory 404 can include a read-only memory and a random access memory, and provide the processor 403 with instructions and data. A part of the memory 404 can also include a non-volatile random access memory (NVRAM). The memory 404 stores an operating system and operation instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, where the operation instructions can include various operation instructions for implementing various operations. The operating system can include various system programs for implementing various basic services and processing hardware-based tasks.

[0177] The processor 403 controls the operation of the communication device 400, and the processor 403 can also be referred to as a central processing unit (CPU). In a specific application, various components of the communication device 400 are coupled together through a bus system, where the bus system can include a data bus in addition to a power bus, a control bus, and a status signal bus, etc. However, in order to clearly illustrate, various buses are referred to as a bus system in the figure.

[0178] The method disclosed in the embodiments of the present application can be applied to the processor 403 or implemented by the processor 403. The processor 403 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 403. The processor 403 described above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 404, and the processor 403 reads the information in the memory 404 and combines the hardware to complete the steps of the above method.

[0179] The receiver 401 can be used to receive input digital or character information, and generate signal input related to relevant settings and function control. The transmitter 402 can include a display device such as a display screen, and the transmitter 402 can be used to output digital or character information through an external interface.

[0180] In the embodiments of the present application, the processor 403 is used to execute the above-mentioned training load planning method based on a heterogeneous NPU.

[0181] In another possible design, when the server 300 or the communication apparatus 400 is a chip, the chip includes a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin, or a circuit, etc. The processing unit can execute computer-executed instructions stored in a storage unit, so that the chip in the terminal performs the wireless reporting information sending method in any one of the first aspects. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the terminal, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0182] The processor mentioned in any one of the above can be a general central processor, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of programs of the above method.

[0183] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0184] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0185] In the above embodiments, the implementation can be wholly or partially by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product.

[0186] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A training load planning method based on heterogeneous NPUs, characterized in that, include: The first server obtains the neural network processing unit (NPU) resource information of each server in the server cluster and obtains the NPU resource information summary. The first server is one of the servers in the server cluster. The first server determines a server combination based on the aggregated NPU resource information. The server combination is used to train the target model and includes one or more servers in the server cluster. The first server performs training load planning based on heterogeneous NPU for the target model based on the server combination to obtain a pipelined parallel strategy. The pipelined parallel strategy is used to instruct the target model to be divided into multiple model blocks. Each of the multiple model blocks includes at least one layer of neurons of the target model. Each of the multiple model blocks corresponds one-to-one with each server in the server combination.

2. The method according to claim 1, characterized in that, The first server obtains NPU resource information from each server in the server cluster, including: The first server sends an NPU resource information collection instruction to each server in the server cluster; The first server receives the NPU resource information returned by each server in the server cluster based on the NPU resource information collection instruction, and obtains the NPU resource information of each server in the server cluster.

3. The method according to claim 1 or 2, characterized in that, The method further includes, after the first server obtains the neural network processing unit (NPU) resource information of each server in the server cluster and summarizes the NPU resource information, the method further includes: The first server obtains the training load information of the target model, which is used to indicate the resource overhead required to perform one training iteration on the target model; The first server determines whether the server cluster supports the resource overhead indicated by the training load information based on the NPU resource information summary; If so, then the first server shall perform the step of determining the server combination based on the NPU resource information summary; If not, the first server issues an alarm message.

4. The method according to claim 3, characterized in that, The alarm information includes the reasons why the server cluster does not support the resource overhead indicated by the training load information, including insufficient memory, insufficient bandwidth, and / or insufficient computing power.

5. The method according to any one of claims 1-4, characterized in that, After the first server performs training load planning based on heterogeneous NPUs on the target model based on the server combination to obtain a pipelined parallel strategy, the method further includes: The first server optimizes other parallel strategies and training framework parameters of the target model based on the pipelined parallel strategy to obtain the optimal parameter configuration.

6. The method according to any one of claims 1-5, characterized in that, The NPU resource information includes the computing power, bandwidth, and memory of each NPU in the server.

7. A server, characterized in that, include: The transceiver module is used to obtain the neural network processing unit (NPU) resource information of each server in the server cluster and obtain the NPU resource information summary. The first server is one of the servers in the server cluster. The processing module is used to determine a server combination based on the NPU resource information aggregation, the server combination being used to train the target model, and the server combination including one or more servers in the server cluster; The processing module is further configured to perform training load planning on the target model based on the server combination to obtain a pipelined parallel strategy. The pipelined parallel strategy is used to instruct the target model to be divided into multiple model blocks. Each model block includes at least one layer of neurons in the target model. Each model block corresponds one-to-one with each server in the server combination.

8. The server according to claim 7, characterized in that, The transceiver module is specifically used for: Send NPU resource information collection instructions to each server in the server cluster; The server receives NPU resource information returned by each server in the server cluster based on the NPU resource information collection instruction, and obtains the NPU resource information of each server in the server cluster.

9. The server according to claim 7 or 8, characterized in that, The processing module is further configured to: Obtain the training load information of the target model, wherein the training load information is used to indicate the resource overhead required to perform one training iteration of the target model; Based on the aggregated NPU resource information, it is determined whether the server cluster supports the resource overhead indicated by the training load information; If so, then proceed with the step of determining the server combination based on the NPU resource information summary; If not, then an alarm message will be issued.

10. The server according to any one of claims 7-9, characterized in that, The processing module is also used to optimize other parallel strategies and training framework parameters of the target model based on the pipelined parallel strategy to obtain the optimal parameter configuration.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that causes a computer device to perform the method as described in any one of claims 1-6.

12. A computer program product, characterized in that, The computer program product includes computer-executable instructions stored in a computer-readable storage medium; at least one processor of the server reads the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to cause the device to perform the method as described in any one of claims 1-6.

13. A communication device, characterized in that, The communication device includes at least one processor, memory, and communication interface; The at least one processor is coupled to the memory and the communication interface; The memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other communication devices under the control of the at least one processor; When the instruction is executed by the at least one processor, it causes the at least one processor to perform the method as described in any one of claims 1-6.

14. A chip system, characterized in that, The chip system includes a processor and a memory, the memory and the processor being interconnected via a circuit, the memory storing instructions, and the processor being used to execute the method as described in any one of claims 1-6.

Images