Model task deployment method and apparatus, and computer system

Abstract

A model task deployment method. The method comprises: obtaining M first mapping relationships corresponding to N layers in a neural network model; on the basis of the M first mapping relationships, obtaining a deployment mode of a model task satisfying a first condition and a second condition on a plurality of compute nodes of a computing device cluster; and deploying the model task according to the deployment mode. In the present application, according to the method, a deployment mode that satisfies matching between the execution time of each stage and matching between the amount of memory used in each stage and a memory resource that can be provided by a compute node in which said stage is deployed can be determined on the basis of relationships between amounts of memory and execution times in different stages of different types of ReC policies configured at different layers in the neural network model corresponding to the model task, so that the model task is deployed according to the deployment mode, and computing resources and memory resources can be fully used in a training process in the stage which is allocated to the compute node, so as to improve the resource utilization rate of the computing device cluster running a training system.

Description

A model task deployment method, device and computer system

[0001] The present application claims priority to the Chinese patent application No. 202411133356.2, filed on August 16, 2024, and entitled "A model task deployment method, device and computer system", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a model task deployment method, device and computer system. BACKGROUND

[0003] With the rapid development of neural network (NN) models, the parameter size, layer number and dataset size of neural network models are exponentially increasing. However, the development of hardware is difficult to keep up with the rapid growth of the size of neural network models. During the model training process, a large amount of memory is often needed to store and process parameters, data and intermediate calculation results, which puts a huge demand on the memory capacity of hardware devices, making the memory of hardware devices a key limitation to the training of neural network models.

[0004] In related technologies, when training a neural network model, pipeline parallelism (PP) and re-computation (ReC) can be used to break through the limitation of the memory of hardware devices. However, due to the high complexity of the strategy for configuring pipeline parallelism and re-computation, it is difficult to configure a strategy that matches the training scenario, resulting in waste of computing resources and memory resources of hardware devices. SUMMARY

[0005] To solve the above problems, in the embodiments of the present application, a model task deployment method is provided, which can improve the resource utilization rate of a computing device cluster running a training system. In addition, the present application also provides a model task deployment device and a computer system corresponding to the model task deployment method.

[0006] To this end, in the embodiments of the present application, the following technical solutions are adopted:

[0007] In a first aspect, a model task deployment method is provided in the embodiments of the present application, including: obtaining M first mapping relationships corresponding to N layers in a neural network model; each of the N layers is applicable to one or more re-computation (ReC) strategies, the first mapping relationship is a mapping relationship between a memory amount and a first execution time, the first execution time is a time for running the layer within a stage using the memory amount based on one of the ReC strategies, the stage is a part of a model task, the model task is training the neural network model or performing inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers of the neural network model is greater than or equal to N; based on the M first mapping relationships, obtaining a deployment manner of the model task on multiple computing nodes of a computing device cluster that satisfies a first condition and a second condition; the deployment manner is used to indicate a computing node corresponding to each stage of the model task and a ReC strategy corresponding to each layer of the neural network model in each stage, each computing node includes one or more computing devices, the first condition is matching between multiple second execution times, the multiple second execution times are respective execution times of multiple stages of the model task under the deployment manner, and the second condition is matching between a memory usage amount of each stage and a memory resource that can be provided by a computing node corresponding to the stage; and deploying the model task according to the deployment manner.

[0008] Each of the N layers is applicable to one or more ReC strategies, which means that one layer can be applicable to one or more ReC strategies. Specifically, one layer applicable to multiple ReC strategies can mean that the whole layer is applicable to multiple ReC strategies, or some operators in the layer are applicable to different ReC strategies, for example, some operators are applicable to one ReC strategy and the other operators are applicable to another ReC strategy; or some operators in the layer are applicable to multiple ReC strategies.

[0009] The matching between the multiple second execution times means that the differences between the multiple second execution times are as small as possible, that is, the execution times of the stages are as uniform as possible. However, due to actual situations, such as differences in resource configuration of the computing cluster, differences in ReC strategies, and differences in structures of the neural networks involved in the deployed tasks, the matching can be manifested in that the execution times of the multiple stages under the deployment manner are the same, or the differences between the execution times are less than a set time threshold, or the differences between the execution times tend to 0 or a minimum.

[0010] The matching between the memory amount used by each stage and the memory resource provided by the computing node corresponding to the stage means that the difference between the memory amount used by each stage under the deployment manner is as small as possible, that is, the memory amount used by each stage is as uniform as possible. However, due to actual situations, such as differences in resource configuration of the computing cluster, differences in ReC strategies, and differences in the structure of the neural network involved in the deployed task, the matching can be manifested in that the proportion of the memory amount used by each stage under the deployment manner to the memory resource provided by the computing node corresponding to the stage is the same, or the difference between them is less than a first set memory threshold, or the difference between them tends to 0 or a minimum. The matching can also be manifested in that the sum of the memory amount used by each layer of each stage under the deployment manner is the same, or the difference between them is less than a second set memory threshold, or the difference between them tends to 0 or a minimum.

[0011] In this embodiment, the method can determine the deployment manner that satisfies the matching between the execution times of each stage and the matching between the memory amount of each stage and the memory resource provided by the computing node to which the stage is deployed according to the relationship between the memory amount and the execution time of different stages of different types of ReC strategies configured in different layers of the neural network model corresponding to the model task, so that the model task is deployed according to the deployment manner, and the stage allocated to the computing node can fully utilize the computing resource and the memory resource in the training process, thereby improving the resource utilization of the computing device cluster running the training system.

[0012] In an embodiment, the obtaining, based on the M first mapping relationships, the deployment manner of the model task on the plurality of computing nodes of the computing device cluster that satisfies the first condition and the second condition specifically includes: obtaining a plurality of third execution times and memory amounts corresponding to the plurality of third execution times according to the M first mapping relationships; each third execution time indicates an execution time of a stage of the model task, and the stage corresponding to each third execution time is different in layers allocated therein and / or different in ReC strategies configured in the layers; selecting the plurality of second execution times from the plurality of third execution times based on the first condition and the second condition; and obtaining the deployment manner based on the layers allocated in the stages corresponding to the plurality of second execution times and the ReC strategies configured in the layers.

[0013] In this embodiment, the method can determine the execution time and memory usage of each layer in the neural network model in different stages with different ReC strategies, and then screen according to the first condition and the second condition to obtain the ReC strategy corresponding to each layer and the stage where the layer is located, so that the deployment mode can make the stage allocated to the computing node fully utilize the computing resources and memory resources in the training process, thereby improving the resource utilization of the computing device cluster running the training system.

[0014] In an embodiment, the obtaining of the M first mapping relationships corresponding to the N layers of the neural network model specifically comprises: obtaining the M first mapping relationships corresponding to the N layers based on historical data; the historical data comprises execution time of different layers of the neural network model with different ReC strategies in different stages of the model task.

[0015] In this embodiment, the method can obtain the first mapping relationship through the types of ReC strategies configured by each layer of the neural network model in history and the stages where the ReC strategies are located, so that the subsequent generated deployment mode is more in line with the historical working mode of the neural network model, thereby more fully utilizing the computing resources of each computing node of the computing device cluster.

[0016] In an embodiment, the obtaining of the M first mapping relationships corresponding to the N layers of the neural network model specifically comprises: profiling the model task to obtain logs of the model task; and obtaining the M first mapping relationships corresponding to the N layers based on the logs of the model task.

[0017] In this embodiment, the method can analyze the model task to obtain the first mapping relationship, so that the subsequent generated deployment mode is more in line with the working mode of the neural network model executing the model task, thereby more fully utilizing the computing resources of each computing node of the computing device cluster.

[0018] In an embodiment, the condition satisfied by the deployment mode further comprises: the memory usage of each stage is less than or equal to the memory resources that can be provided by the computing node corresponding to the stage.

[0019] In this embodiment, the deployment mode generated by the method requires that the memory usage of each stage should be less than or equal to the memory resources that can be provided by the computing node corresponding to the stage, thereby avoiding the situation that the computing node cannot provide sufficient memory usage for the stage, resulting in that the neural network model cannot normally execute the model task.

[0020] In an implementation, the condition that the deployment manner satisfies further includes that the deployment manner of the model task satisfies a pipeline parallel (PP) strategy.

[0021] The pipeline parallel (PP) strategy refers to that a stage where a next layer of each layer is located is a stage where the layer is located, or a previous stage of the stage where the layer is located, or a next stage of the stage where the layer is located, or a parallel stage of the stage where the layer is located, and a stage where a previous layer of each layer is located is the stage where the layer is located, or a next stage of the stage where the layer is located, or a previous stage of the stage where the layer is located, or a parallel stage of the stage where the layer is located.

[0022] In this implementation, the method generates a deployment manner that requires layers of adjacent execution sequences to be allocated to the same stage or adjacent stages, so as to avoid allocation of the layers of adjacent execution sequences to stages whose execution sequences span a large range, thereby reducing the time for the neural network model to execute the model task.

[0023] In an implementation, the deploying the model task according to the deployment manner specifically includes instructing the N layers to execute the model task based on the ReC strategy configured for each layer in the deployment manner and the stage where the layer is located.

[0024] In an implementation, the obtaining the plurality of third execution times according to the M first mapping relationships specifically includes: obtaining computation amounts of the N layers configured with different types of ReC strategies in different stages; obtaining a plurality of execution computation amounts and a plurality of execution time relationships of the N layers according to the computation amounts of the N layers configured with different types of ReC strategies in different stages and the M first mapping relationships; the execution computation amount refers to a relationship between an execution time and a computation amount of a layer configured with a type of ReC strategy in a stage; the execution time relationship refers to a relationship between a plurality of execution times corresponding to different types of ReC strategies configured for a layer in a stage; and obtaining the plurality of third execution times according to the plurality of execution computation amounts and the plurality of execution time relationships of the N layers.

[0025] In this implementation, the method uses the execution computation amount of each layer to optimize the calculation performance of the neural network model, which can eliminate the influence of the calculation / communication superposition degree between layers and the influence on the execution time. The method can configure the most suitable ReC strategy for each layer based on the execution time relationship of each layer configured with different types of ReC strategies.

[0026] In an embodiment, the deployment manner of the model task satisfying the first condition and the second condition on the plurality of computing nodes of the computing device cluster based on the M first mapping relationships specifically comprises: obtaining the deployment manner satisfying the second condition according to the memory usage of the different kinds of ReC strategies of the each layer configuration in different stages, the memory usage of the activation values of the different kinds of ReC strategies of the each layer configuration, the additional memory amount of the different kinds of ReC strategies of the first stage, the additional memory amount of the different kinds of ReC strategies of the second stage, and the memory resources that can be provided by the computing nodes corresponding to the each stage. The first stage refers to the first stage executed in the plurality of stages. The second stage refers to the last stage executed in the plurality of stages. The activation value refers to the output value of a neuron in the neural network model after accepting input and processing through an activation function. The additional memory amount refers to the additional memory generated by the additional operations of the stages relative to the intermediate position. The additional memory amount of the first stage refers to the additional memory consumption generated by the additional operations of the neural network model relative to the stages at the intermediate position. At this time, the additional operations can be data preprocessing, feature extraction and conversion, weight initialization, activation value transmission, etc. The additional memory amount of the first stage refers to the additional memory consumption generated by the additional operations of the neural network model relative to the stages at the intermediate position. At this time, the additional operations can be storage and transmission of activation values, loss calculation, gradient calculation and direction propagation, processing of output results, etc.

[0027] In this embodiment, the method for obtaining the memory amount of the activation values when the different kinds of ReC strategies are configured for the each layer can quantify the influence of the memory amount of the activation values when the different kinds of ReC strategies are configured for the layer on the each layer in the split neural network model. The method for obtaining the additional memory amount of the first stage and the additional memory amount of the last stage can eliminate the influence of the additional memory amount of the first stage and the last stage on the each layer in the split neural network model.

[0028] In a second aspect, an embodiment of the present application provides a model task deployment apparatus, comprising: a first processing unit configured to obtain M first mapping relationships corresponding to N layers in a neural network model; wherein each of the N layers is applicable to one or more ReC strategies, the first mapping relationship is a mapping relationship between a memory amount and a first execution time, the first execution time is a time for running the layer in a stage using the memory amount based on one of the ReC strategies, the stage is a part of a model task, the model task is training the neural network model or performing inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers of the neural network model is greater than or equal to N; a second processing unit configured to obtain, based on the M first mapping relationships, a deployment manner of the model task on a plurality of computing nodes of a computing device cluster that satisfies a first condition and a second condition; the deployment manner is used to indicate a computing node corresponding to each stage of the model task and a ReC strategy corresponding to each layer of the neural network model in each stage, each computing node comprises one or more computing devices, the first condition is matching between a plurality of second execution times, the plurality of second execution times are respective execution times of a plurality of stages of the model task under the deployment manner, and the second condition is matching between a memory usage amount of each stage and a memory resource that can be provided by a computing node corresponding to the stage; and a third processing unit configured to deploy the model task according to the deployment manner.

[0029] In an embodiment, the second processing unit is specifically configured to obtain, according to the M first mapping relationships, a plurality of third execution times and memory usage amounts corresponding to the plurality of third execution times; wherein each third execution time indicates an execution time of a stage of the model task, the stage corresponding to each third execution time is different in allocated layers and / or different in ReC strategies of layers; the plurality of second execution times are selected from the plurality of third execution times based on the first condition and the second condition; and the deployment manner is obtained based on allocated layers and ReC strategies of layers in stages corresponding to the plurality of second execution times.

[0030] In an embodiment, the first processing unit is specifically configured to obtain, based on historical data, the M first mapping relationships corresponding to the N layers; and the historical data comprises execution times of different layers in the neural network model in different stages of the model task, the different layers being configured with different ReC strategies.

[0031] In an implementation, the first processing unit is specifically configured to analyze the model task to obtain a log of the model task; and obtain the M first mapping relationships corresponding to the N layers based on the log of the model task.

[0032] In an implementation, the condition satisfied by the deployment manner further includes that the memory usage of each stage is less than or equal to the memory resource that can be provided by the computing node corresponding to the stage.

[0033] In an implementation, the condition satisfied by the deployment manner further includes that the deployment manner of the model task satisfies a pipeline parallel (PP) strategy.

[0034] In an implementation, the third processing unit is specifically configured to instruct the N layers to execute the model task based on the ReC strategy configured for each layer in the deployment manner and the stage in which the layer is located.

[0035] In an implementation, the second processing unit is specifically configured to obtain the computation amount of different kinds of ReC strategies configured for the N layers in different stages; obtain a plurality of execution computation amounts and a plurality of execution time relationships of the N layers based on the computation amount of different kinds of ReC strategies configured for the N layers in different stages and the M first mapping relationships; the execution computation amount refers to the relationship between the execution time and the computation amount of one layer in one stage; the execution time relationship refers to the relationship between a plurality of execution times of one layer in one stage corresponding to different kinds of ReC strategies; and obtain the plurality of third execution times based on the plurality of execution computation amounts and the plurality of execution time relationships of the N layers.

[0036] In an implementation, the second processing unit is specifically configured to obtain the deployment manner satisfying the second condition based on the memory usage of different kinds of ReC strategies configured for each layer in different stages, the memory usage of activation values of different kinds of ReC strategies configured for each layer, the additional memory amount of different kinds of ReC strategies configured for a first stage, the additional memory amount of different kinds of ReC strategies configured for a second stage, and the memory resource that can be provided by the computing node corresponding to each stage; the first stage refers to the first stage executed in the plurality of stages; and the second stage refers to the last stage executed in the plurality of stages.

[0037] In a third aspect, an embodiment of the present application provides a computer system, the computer system comprising a processing unit and a storage unit; the processing unit of the computer system is configured to execute instructions stored in the processing unit, so that the computer system executes the embodiments of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions, when the computer program instructions are executed by a computer system, the computer system executes the embodiments of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, characterized in that the computer program product stores instructions, when the instructions are executed by a computer system, the computer system executes the embodiments of the first aspect.

[0040] In a sixth aspect, an embodiment of the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the embodiments of the first aspect.

[0041] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the embodiments of the first aspect.

[0042] In an eighth aspect, an embodiment of the present application provides a computer program product comprising instructions, characterized in that the computer program product stores instructions, when the instructions are executed by a computing device cluster, the computing device cluster executes the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0043] The drawings required for use in the embodiments or prior art description are briefly introduced as follows.

[0044] FIGS. 1(a)-1(c) are schematic diagrams of a scenario in which a training system provided in the related art trains a neural network model;

[0045] FIG. 2 is a schematic diagram of a structure of a training system provided in an embodiment of the present application;

[0046] FIG. 3 is a schematic diagram of a scenario in which a user uses a training system provided in an embodiment of the present application;

[0047] FIG. 4 is a flow diagram of a method for deploying a model task according to an embodiment of the present application;

[0048] FIG. 5 is a structural diagram of a device for deploying a model task according to an embodiment of the present application;

[0049] FIG. 6 is a structural diagram of a computing device according to an embodiment of the present application;

[0050] FIG. 7 is an architectural diagram of a computing device cluster according to an embodiment of the present application;

[0051] FIG. 8 is an architectural diagram of another computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0053] The term "and / or" herein is used to describe an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " herein represents an or relationship of associated objects, for example, A / B represents A or B.

[0054] The terms "first" and "second" and the like in the description and claims herein are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0055] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0056] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0057] Before introducing the technical solutions protected by the present application, several professional terms involved in the technical solutions protected by the present application are explained in advance, which are as follows:

[0058] (1) Neural network model

[0059] A neural network model is a computational model inspired by the structure of the human brain, which processes information by mimicking the way neurons in the human brain connect and interact. In a neural network model, layers are the basic units that make up the model, which is a hierarchical structure for processing data. Each layer can be composed of multiple neurons, which are responsible for receiving input, performing calculations, and passing the results to the next layer. In addition, each layer can be responsible for performing specific computational tasks. The layers in a neural network model can include one or more of the following: input layer, hidden layer, output layer, fully connected layer, convolutional layer, pooling layer, self-attention layer, normalization layer, etc. For example, a neural network model can be, but is not limited to, a deep neural network (DNN) model, a convolutional neural network (CNN) model, or a transformer model, etc.

[0060] (2) Operator

[0061] An operator is a function that performs a specific mathematical operation, such as matrix multiplication, addition, activation function, etc. In a neural network model, operators are used to process data and implement the functions of layers. In a neural network model, the functions of layers can be implemented through a series of operators. For example, a fully connected layer can be implemented through a matrix multiplication operator (dot product of weight matrix and input vector) and an addition operator (add bias). By combining different operators, layers with different functions can be designed to meet different task requirements. Among them, in a neural network model, layers are a higher level of abstraction, which defines how data flows and transforms in the network. Operators are specific mathematical operations that implement these transformations.

[0062] (3) PP

[0063] PP is an efficient parallel computing method that breaks down complex tasks into multiple stages that depend on each other. These stages can be executed in parallel and pass the results to the next stage when completed. This pipeline processing allows data to flow between stages without waiting for the previous stage to complete, significantly improving computing efficiency.

[0064] When the neural network model is trained, the PP technology can be used to split the neural network model to divide the neural network model into multiple parts. Each part includes at least one layer. Each part can be a stage, and thus multiple stages are obtained. The layers of each part (i.e., each stage) are respectively deployed to different computing nodes. Each computing node can be responsible for a stage. When deploying the corresponding layer on the computing node, the related parameters of the layer to be deployed can be configured to the corresponding computing node to complete the deployment of the layer on the computing node.

[0065] In the process of splitting by the PP technology, the stage where the next layer of each layer is located is the stage where the layer is located, or the previous stage of the stage where the layer is located, or the next stage of the stage where the layer is located, or the parallel stage of the stage where the layer is located, and the stage where the previous layer of each layer is located is the stage where the layer is located, or the next stage of the stage where the layer is located, or the previous stage of the stage where the layer is located, or the parallel stage of the stage where the layer is located.

[0066] (4) ReC

[0067] ReC is a computing optimization technology that re-executes part of the computation in the calculation to reduce memory consumption and performance. In ReC, some computation results can be discarded temporarily as needed, and these results are recalculated in the later calculation instead of being saved in the memory. The main advantage of ReC technology is that it can reduce memory consumption, especially in the case of limited memory usage, it can save a lot of memory. When some computation results are only used temporarily in the subsequent calculation and do not need to be retained, ReC technology can help the system avoid long-term storage of these results, thereby improving memory utilization.

[0068] The types of ReC strategies can include non-repeated computation strategy, full repeated computation strategy, and selective repeated computation strategy. The non-repeated computation strategy means that when computing results, it is possible to avoid repeating the computation of the part that has already been computed. The full repeated computation strategy means that every time the computation is needed, all the computation steps are performed from the beginning, regardless of whether similar results have been computed before. The selective repeated computation strategy is between non-repeated computation and full repeated computation, which means that the performance is optimized by intelligently identifying and selecting the part that needs to be recalculated.

[0069] Each layer can perform computation with one ReC strategy. At this time, all operators corresponding to the layer can perform computation with the same ReC strategy. Each layer can perform computation with multiple ReC strategies at the same time. At this time, a part of operators corresponding to the layer can perform a part of computation with one ReC strategy, and another part of operators can perform another part of computation with another ReC strategy.

[0070] When training a neural network model by using the ReC technology, some intermediate computation results, such as the output or activation value of a layer, can be avoided to be cached or saved. When these intermediate results are needed again during the model training process, instead of being read from the cache, the necessary computation steps are re-executed to generate these intermediate results. By using the ReC technology, the memory usage can be significantly reduced because the storage requirement is reduced, but the computation time can be increased because the same computation needs to be repeatedly executed.

[0071] Next, the technical solutions provided in the present application are introduced.

[0072] Generally, when training a neural network model, the two technologies of PP and ReC can be comprehensively used. As shown in FIG. 1(a), taking a neural network model including 14 layers as an example, the topological structure order of the neural network model is that the output of layer 1 and the output of layer 2 are the input of layer 3, the output of layer 3 and the output of layer 4 are the input of layer 5, the output of layer 5 is the input of layer 6, the output of layer 6 is the input of layer 7 and the input of layer 8, the output of layer 7 is the input of layer 8, the output of layer 8 is the input of layer 9 and the input of layer 10, the output of layer 9 is the input of layer 11 and the input of layer 12, the output of layer 10 is the input of layer 11 and the input of layer 12, the output of layer 11 is the input of layer 13, the output of layer 12 is the input of layer 13, and the output of layer 13 is the input of layer 14. When training the neural network model, each layer of the neural network model can be trained in the topological structure order.

[0073] As shown in FIG. 1(b), taking the number of computing nodes as 4 as an example. By using the PP technology, the 14 layers of the neural network model are divided into 4 stages. Among them, stage 1 includes 5 layers, stage 2 includes 3 layers, stage 3 includes 4 layers, and stage 4 includes 2 layers. Then, according to the layer division result, the training data of the neural network model and the related parameters of the layers are divided, and the training data and the related parameters of each layer in one stage are divided together as a micro batch, so as to obtain 4 stage corresponding micro batch data.

[0074] As shown in FIG. 1(c), when the four stages are deployed to the corresponding computing nodes, the grouped data of stage 1 can be transmitted to computing node 1, the grouped data of stage 2 can be transmitted to computing node 2, the grouped data of stage 3 can be transmitted to computing node 3, and the grouped data of stage 4 can be transmitted to computing node 4, so as to realize the deployment of stages 1, 2, 3 and 4 to computing nodes 1, 2, 3 and 4 respectively.

[0075] Taking the reception of the grouped data of stage 1 by computing node 1 as an example, when computing node 1 receives the grouped data of stage 1, computing node 1 can construct layers 1, 2, 3, 4 and 5 locally according to the related parameters of the layers in the grouped data. Computing node 1 can distribute the training data for training layer 1 to layer 1, the training data for training layer 2 to layer 2, the training data for training layer 3 to layer 3, and the training data for training layer 4 to layer 4 according to the training data in the grouped data.

[0076] Before computing node 1 trains the five layers, computing node 1 can configure a ReC strategy for each layer respectively. When computing node 1 trains the five layers, according to the order of the topology structure of the neural network model, layer 1 and layer 2 can be trained in parallel, and then the calculation result of layer 1 and the calculation result of layer 2 are input to layer 3 as the input of layer 3. Layer 3 can be trained according to the training data allocated by itself, the calculation result of layer 1 and the calculation result of layer 2, and then the calculation result is input to layer 5 as the input of layer 5. Layer 4 can be trained in parallel with layer 3, and then the calculation result is input to layer 5 as the input of layer 5. Layer 5 can be trained according to the training data allocated by itself, the calculation result of layer 13 and the calculation result of layer 4, and then the calculation result is input to the next computing node 2 as the input of layer 6 in computing node 2.

[0077] During the training of the five layers in computing node 1, the five layers can discard some calculation results according to the ReC strategy configured by themselves, so as to reduce the memory consumption in computing node 1.

[0078] Generally, the calculation amount and the occupied memory amount used by different types of layers during training are not the same, and all the layers in the stage cut by the PP technology are difficult to fully use the computing resources and memory resources of the computing node to which the stage is deployed. When the same layer uses different kinds of ReC strategies for training, the calculation amount and the occupied memory amount are not the same. If each layer is configured with different kinds of ReC strategies, all the layers in the stage cut by the PP technology are more difficult to fully use the computing resources and memory resources of the computing node to which the stage is deployed.

[0079] In view of this, the present application provides a training system of a neural network model. The training system can determine a deployment manner that satisfies matching of execution time of each stage and matching of memory usage of each stage and memory resources provided by a computing node deployed in the stage according to a relationship between memory amount and execution time of different kinds of ReC strategies in different stages in a neural network model corresponding to a model task, so that the model task is deployed according to the deployment manner, and the stage allocated to the computing node can fully utilize computing resources and memory resources in a training process, thereby improving resource utilization of a computing device cluster running the training system.

[0080] FIG. 2 is a structural schematic diagram of a training system provided in an embodiment of the present application. As shown in FIG. 2, the training system 200 can include a model task deployment apparatus 210 and a training apparatus 220.

[0081] The training system 200 is generally deployed on a computing device cluster for training and inference. The computing device cluster can include a plurality of computing devices. The computing device cluster can be divided into a plurality of computing nodes, so that the plurality of computing nodes perform model training or model inference. Each computing node includes one or more computing devices. Each computing node can be responsible for a stage.

[0082] The model task deployment apparatus 210 and the training apparatus 220 in the training system 200 can be executed on any one or several computing nodes in the computing device cluster, or on any one or several computing devices in the computing device cluster. The model task deployment apparatus 210 and the training apparatus 220 can be deployed on the same computing node or computing device, or can be deployed on different computing nodes or computing devices.

[0083] The model task deployment apparatus 210 is configured to obtain a deployment manner of the model task on the plurality of computing nodes according to historical data and device parameters of each computing node running the neural network model. The model task deployment apparatus 210 can divide data collection units 211, computing analysis units 212, memory analysis units 213, and strategy calculation units 214 according to the executed tasks.

[0084] The data collection unit 211 is configured to collect historical data and device parameters of the plurality of computing nodes. Generally, the neural network model can be deployed on different computing nodes respectively, and then trained or inferred by the different computing nodes. When the computing device cluster trains or infers the neural network model, the performance analyzer can be used to obtain the stage corresponding to the computing node where each layer of the neural network model is located, the memory usage of each layer, the type of ReC strategy configured for each layer, the execution time of each layer, operator information of each layer, and the like. The operator information can include the identifier of each operator, the execution time of each operator, the layer where each operator is located, the computation amount of each operator, and the like. The computing device cluster can store each time the obtained basic data in a public storage area, so as to be collected by the data collection unit 211.

[0085] After the data collection unit 211 collects the basic data, the data collection unit 211 can calculate higher-level data according to the relationship between each data in the basic data, which is referred to as historical data hereinafter. Alternatively, the data collection unit 211 can select the basic data related to the N layers participating in the execution of the model task in the neural network model based on the model task, so as to calculate the historical data of the N layers participating in the execution of the model task, thereby reducing the computation amount. The number of layers of the neural network model is greater than or equal to N, and N is an integer greater than 1.

[0086] In one embodiment, the data collection unit 211 can obtain the execution time of each layer of the neural network model in different stages when each layer is configured with different types of ReC strategies, according to the stage corresponding to the computing node where each layer is located, the type of ReC strategy configured for each layer, and the execution time of each layer when the neural network model is trained or inferred each time.

[0087] In another embodiment, the data collection unit 211 can obtain the memory usage of each layer of the neural network model in different stages when each layer is configured with different types of ReC strategies, according to the stage corresponding to the computing node where each layer is located, the type of ReC strategy configured for each layer, and the memory usage of each layer when the neural network model is trained or inferred each time.

[0088] In another embodiment, the data collection unit 211 can obtain M first mapping relationships of each layer, according to the first execution time of each layer in different stages when each layer is configured with different types of ReC strategies and the memory usage of each layer in different stages when each layer is configured with different types of ReC strategies. Each first mapping relationship is a mapping relationship between the memory usage and the first execution time of one layer in the same stage when the layer is configured with one type of ReC strategy. M is an integer greater than or equal to N.

[0089] In another embodiment, the data collection unit 211 can obtain the operator information of each layer of the neural network model configured with different kinds of ReC strategies in different stages according to the stages corresponding to the computing nodes where each layer is located, the kinds of ReC strategies configured for each layer, and the operator information of each layer each time the neural network model is trained or inferred.

[0090] Optionally, the historical data can be generated when the training system 200 is pre-trained. For example, the training system 200 can pre-train the neural network model to obtain the historical data of the neural network model before the deployment device 210 obtains the deployment manner of the model task on the plurality of computing nodes.

[0091] Alternatively, the training system 200 analyzes the model task to obtain a log of the model task. The log usually includes the execution time of each layer or operator in the layer. Based on the log, the data collection unit 211 can obtain the first execution time of each layer configured with different kinds of ReC strategies in different stages, the M first mapping relationships of each layer, and other data.

[0092] The device parameters include the memory amount of each computing node and the computing amount of each computing node. The memory amount refers to the memory usage or the idle memory amount of all the memories in a computing node when the computing node provides the neural network model for training or inference. The computing amount refers to the computing amount occupied or the idle computing amount of all the processors in a computing node when the computing node provides the neural network model for training or inference.

[0093] The computing analysis unit 212 is configured to obtain the execution computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages and the execution time relationship of each layer configured with different kinds of ReC strategies in different stages according to the historical data.

[0094] The computing amount of different types of layers of the neural network model is not the same. The computing amount refers to the consumption of computing resources required to complete one forward propagation or backward propagation. The computing amount can include any one of floating point operations (FLOPs), parameter quantity, and other computing hardware resource consumption. The FLOPs is an index for measuring the computing cost of the neural network model, which represents the number of floating point operations required from input to output of the model. The FLOPs can be estimated by counting the number of multiplication and addition operations of each layer, and can be used as an important index for measuring the computing complexity and efficiency of the model. The parameter quantity can be used as an index for measuring the computing amount of the model. The more the parameter quantity, the more computing operations are required to update and adjust the parameters, thereby increasing the computing amount.

[0095] Exemplarily, the computing and analyzing unit 212 can obtain the running time interval (micro step) of the data packet of the micro batch of the single pipeline from the performance analysis data. The performance analysis data refers to the data obtained by analyzing and optimizing the neural network model using the profiler tool in the deep learning framework. The profiler tool can provide various information about the resource usage and performance bottlenecks in the training process of the neural network model, helping users better understand the behavior of the neural network model and perform optimization. The profiler tool can provide functions such as computing resource usage monitoring, performance analysis, memory allocation analysis, training process visualization, etc. In the embodiment of the present application, the training system 200 can use the profiler tool to analyze the performance bottlenecks of the neural network model, identify which parts of the code run slower or occupy more resources, and detect the memory allocation in the model training process to obtain the performance analysis data.

[0096] The running time interval of the pipeline includes a start time (ms.begin) and an end time (ms.end). In one embodiment, since a single operator appears only once in an iteration of a training, the interval of two appearances of the same operator is the running time interval of a pipeline. After obtaining the operator information of the operators in different stages of each layer configuration with different kinds of ReC strategies from the historical data, the computing and analyzing unit 212 can traverse all the operators, record the identifier of the first matrix multiplication (MATMUL) operator (such as MATMUL-0124) and the time of the appearance of the operator as the start time T1. When the MATMUL operator with the identifier MATMUL-0124 appears again, the computing and analyzing unit 212 records the time of this appearance as the end time T2. The computing and analyzing unit 212 can obtain the running time interval (T1, T2) of a pipeline according to the time of the two consecutive appearances of the MATMUL operator with the identifier MATMUL-0124.

[0097] After obtaining the running time interval of a pipeline, the computing and analyzing unit 212 can obtain the operator information of all the operators in the running time interval of the pipeline from the operator information of the operators in different stages of each layer configuration with different kinds of ReC strategies. In one embodiment, after obtaining the execution time of all the operators, the computing and analyzing unit 212 screens out the operators that satisfy the conditions that the start time of the operator is greater than the start time of the running time interval of the pipeline and the end time of the operator is less than the end time of the running time interval of the pipeline, thereby obtaining all the operators in the running time interval of the pipeline.

[0098] The computing analysis unit 212 can calculate the computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages according to the operators in the runtime interval of the pipeline, the layer in which each operator is located, and the computing amount of each operator. The computing amount of each layer of the neural network model is an important step for optimizing the performance of the neural network model. For example, after the computing analysis unit 212 screens out the operators in the runtime interval of the pipeline, the computing analysis unit 212 can determine all operators of each layer according to the layer in which each operator is located. The computing analysis unit 212 can obtain the computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages according to all operators of each layer and the computing amount of each operator.

[0099] Different kinds of layers in the neural network model have different computing and communication characteristics, and therefore the degree of computing / communication overlap between layers and the influence of operations of different layers on execution time need to be considered. In the embodiments of the present application, the computing analysis unit 212 can calculate the execution computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages according to the computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages and the first execution time of each layer of the neural network model configured with different kinds of ReC strategies in different stages obtained from historical data. The deployment apparatus 210 for model tasks optimizes the computing performance of the neural network model by using the execution computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages, and can eliminate the influence of the degree of computing / communication overlap between layers and the influence on execution time.

[0100] In one embodiment, after the computing analysis unit 212 obtains the first execution time of each layer of the neural network model configured with different kinds of ReC strategies in different stages from historical data, the computing analysis unit 212 can calculate the first execution time / computing amount ratio of each layer of the neural network model configured with different kinds of ReC strategies in different stages according to the first execution time of each layer of the neural network model configured with different kinds of ReC strategies in different stages and the computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages. The computing analysis unit 212 can select a most representative layer from the plurality of layers of the neural network model according to a set rule. The set rule refers to a rule that the number of repetitions is the largest, the computing amount is the highest, and the like. The computing analysis unit 212 can obtain the execution computing amount of each layer of the neural network model configured with different kinds of ReC strategies in different stages according to the ratio of the first execution time / computing amount ratio of each layer of the neural network model configured with different kinds of ReC strategies in different stages to the first execution time / computing amount ratio of the most representative layer.

[0101] The types of ReC strategies executed by the training system under different AI frameworks are different, and the execution times of different types of ReC strategies are also different. In the embodiment of the present application, the computing and analyzing unit 212 can obtain the first execution time of each layer from the historical data, and obtain the execution time relationship of each layer in the neural network model configured with different types of ReC strategies. In the process of optimizing the computing performance of the neural network model, the model task deployment device 210 can be based on the execution time relationship of each layer configured with different types of ReC strategies, so that the model task deployment device 210 can configure the most suitable type of ReC strategy for each layer in combination with the execution time relationship.

[0102] In one embodiment, the computing and analyzing unit 212 can obtain the types of ReC strategies supported by the AI framework currently used by the training system. The types of ReC strategies can include non-repeated calculation, complete repeated calculation, and selective repeated calculation, etc. The computing and analyzing unit 212 can obtain the first execution time of each layer configured with different types of ReC strategies in different stages from the historical data, and calculate the ratio between the first execution time of each layer in the neural network model configured with different types of ReC strategies in a stage to obtain the execution time relationship of each layer in the neural network model configured with different types of ReC strategies in different stages.

[0103] The memory analyzing unit 213 is used to obtain the memory usage of each layer of the neural network model configured with different types of ReC strategies in different stages according to the historical data. The memory usage refers to the peak value of the memory occupied by completing one forward propagation and back propagation.

[0104] Exemplarily, after the memory analyzing unit 213 obtains the memory usage of each layer configured with different types of ReC strategies in different stages from the historical data, the memory usage of the layers in the same stage and configured with the same ReC strategy can be summed up to obtain the memory usage of all layers configured with one ReC strategy in each stage, so as to obtain the memory usage of each stage configured with different types of ReC strategies.

[0105] The memory analysis unit 213 can analyze the number of layers of each stage configured with the same type of ReC policy and the type of ReC policy after obtaining the memory usage of each stage configured with different types of ReC policies. The memory analysis unit 213 can calculate the memory usage of each layer configured with different types of ReC policies in different stages and the memory usage of the activation values of each layer configured with different types of ReC policies according to the memory usage of each stage configured with different types of ReC policies, the number of layers of each stage configured with different types of ReC policies, and the type of ReC policy, except for the first stage and the last stage. The activation value refers to the output value of a neuron in a neural network model after accepting input and processing through an activation function.

[0106] In one embodiment, the memory analysis unit 213 can select any three stages from the plurality of stages, except for the first stage and the last stage, and input the stage id of the selected three stages, the number of layers when the three stages are configured with the same type of ReC policy, the memory usage of the three stages configured with different types of ReC policies, and the plurality of types of ReC policies into formula (1) to calculate the memory usage M P of each layer configured with different types of ReC policies in different stages of the neural network model and the memory usage M A r of the activation values of each layer configured with different types of ReC policies r. Wherein formula (1) is specifically:

[0107] Wherein L represents a set of layers l of a stage, R represents a set of ReC policy types r, S is the number of stages, i represents the stage id, M P represents the memory usage of the layer, M A r represents the memory usage of the activation values configured with the ReC policy of type r, l i r represents the layer configured with the ReC policy of type r in the i-th stage.

[0108] The model task deployment apparatus 210 can configure different kinds of ReC strategies in the memory usage of each stage based on the optimization of the memory performance of the neural network model, optimize each layer in the split neural network model, and configure the most suitable memory amount of the layer for the computing node running the stage. The model task deployment apparatus 210 can quantify the influence of the memory usage of the activation value of the layer configured with different kinds of ReC strategies on each layer in the split neural network model based on the memory usage of the activation value of the layer configured with different kinds of ReC strategies.

[0109] The memory analysis unit 213 can calculate the memory usage of the first stage configured with different kinds of ReC strategies based on the number of layers configured with the same kind of ReC strategy in the first stage of the neural network model, the kind of ReC strategy configured in the layer of the first stage, the memory usage of each layer configured with different kinds of ReC strategies in the first stage, and the memory usage of the activation value of each layer configured with different kinds of ReC strategies.

[0110] The memory analysis unit 213 can calculate the memory usage of the last stage configured with different kinds of ReC strategies based on the number of layers configured with the same kind of ReC strategy in the last stage of the neural network model, the kind of ReC strategy configured in the layer of the last stage, the memory usage of each layer configured with different kinds of ReC strategies in the last stage, and the memory usage of the activation value of each layer configured with different kinds of ReC strategies.

[0111] The memory analysis unit 213 can obtain the memory usage of the first stage configured with different kinds of ReC strategies and the memory usage of the last stage configured with different kinds of ReC strategies from the memory usage of each stage configured with different kinds of ReC strategies, and refer to the obtained memory usage of the first stage configured with different kinds of ReC strategies and the obtained memory usage of the last stage configured with different kinds of ReC strategies as the measured memory usage of the first stage configured with different kinds of ReC strategies and the measured memory usage of the last stage configured with different kinds of ReC strategies. The memory analysis unit 213 can calculate the additional memory amount of the first stage configured with different kinds of ReC strategies by subtracting the calculated memory usage of the first stage configured with different kinds of ReC strategies from the measured memory usage of the first stage configured with different kinds of ReC strategies. Also, the memory analysis unit 213 can calculate the additional memory amount of the last stage configured with different kinds of ReC strategies by subtracting the calculated memory usage of the last stage configured with different kinds of ReC strategies from the measured memory usage of the last stage configured with different kinds of ReC strategies.

[0112] The additional memory amount refers to additional memory generated by additional operations of each stage relative to the intermediate position. The additional memory amount of the first stage refers to additional memory consumption generated by the neural network model when the first stage (e.g., an embedding layer or other preliminary layer) is additionally operated relative to each stage of the intermediate position. At this time, the additional operation can be data preprocessing, feature extraction and conversion, weight initialization, activation value transmission, etc. The additional memory amount of the last stage refers to additional memory consumption generated by the neural network model when the last stage (e.g., a query layer, an output layer, etc.) is additionally operated relative to each stage of the intermediate position. At this time, the additional operation can be storage and transmission of activation values, loss calculation, gradient calculation and direction propagation, processing of output results, etc.

[0113] In one embodiment, the memory analysis unit 213 can input the memory usage amount of the first stage configured with different kinds of ReC policies, the kind of ReC policy configured for the layer of the first stage, the memory usage amount of the last stage configured with different kinds of ReC policies, and the kind of ReC policy configured for the layer of the last stage into Equation (2), and calculate the additional memory amount E0 of the first stage configured with different kinds of ReC policies and the additional memory amount E S-1 of the last stage configured with different kinds of ReC policies.

[0114] wherein M i represents the memory amount of the i-th stage, i = 0 represents the first stage, i = -1 represents the last stage, E0 represents the additional memory amount of the first stage configured with different kinds of ReC policies, and E S-1 represents the additional memory amount of the last stage configured with different kinds of ReC policies.

[0115] The model task deployment apparatus 210 can eliminate the influence of the additional memory amount of the first and last stages configured with different kinds of ReC policies on each layer in the split neural network model, upon obtaining the additional memory amount of the first stage configured with different kinds of ReC policies and the additional memory amount of the last stage configured with different kinds of ReC policies.

[0116] The policy calculation unit 214 is configured to obtain the execution amount of each layer configured with different kinds of ReC policies in different stages, the execution time relationship of each layer configured with different kinds of ReC policies in different stages output by the calculation analysis unit 212, and the memory usage amount of each layer configured with different kinds of ReC policies in different stages output by the memory analysis unit 213, to obtain the deployment mode of the model task in the computing device cluster.

[0117] For example, the policy calculation unit 214 can obtain the number of layers in the neural network model from the definition file of the neural network model, and then determine the number constraint condition of layers in all stages according to the number of stages and the number of layers in the neural network model. The number constraint condition means that the number of layers allocated to all stages is equal to the number of layers in the neural network model.

[0118] The policy calculation unit 214 can obtain the execution order of each layer in the neural network model from the definition file of the neural network model, and then determine the order constraint condition of each layer in each stage according to the execution order of each layer in the neural network model. The order constraint condition satisfies the pipeline parallel PP policy. The pipeline parallel PP policy means that the stage where the next layer of each layer is located is the stage where the layer is located, or the previous stage of the stage where the layer is located, or the next stage of the stage where the layer is located, or the parallel stage of the stage where the layer is located, and the stage where the previous layer of each layer is located is the stage where the layer is located, or the next stage of the stage where the layer is located, or the previous stage of the stage where the layer is located, or the parallel stage of the stage where the layer is located.

[0119] The policy calculation unit 214 can obtain the memory constraint condition of layers in each stage according to the memory usage amount of each layer configured with different kinds of ReC policies in different stages, the memory usage amount of the activation value of each layer configured with different kinds of ReC policies, the additional memory amount of the first stage configured with different kinds of ReC policies, the additional memory amount of the last stage configured with different kinds of ReC policies, and the memory amount of the computing node corresponding to each stage. The memory constraint condition satisfies that the sum of the memory amounts of all layers in each stage is less than or equal to the memory amount of the computing node deployed in the stage.

[0120] In one embodiment, the policy calculation unit 214 can input the memory usage of each layer configuration different kinds of ReC policies in different stages, the memory usage of the activation value of each layer configuration different kinds of ReC policies, the additional memory amount of the first stage configuration different kinds of ReC policies and the additional memory amount of the last stage configuration different kinds of ReC policies calculated by the memory analysis unit 213 into formula (3) to calculate the memory constraint condition M of each stage of layer i≤C . Wherein formula (3) is specifically:

[0121] Wherein, M i represents the memory amount of the i-th stage, i=0 represents the first stage, i=-1 represents the last stage, C represents the machine memory limit, L represents the set of layers l of a stage, R represents the set of ReC policy kinds, S is the number of stages, i represents the serial number of the stage, M P represents the memory amount of the layer, represents the memory usage of the activation value of the ReC policy of kind r, represents the layer in the i-th stage and configuration ReC policy of kind r, E0 represents the additional memory amount of the first stage configuration different kinds of ReC policies, E S-1 represents the additional memory amount of the last stage configuration different kinds of ReC policies.

[0122] The policy calculation unit 214 can calculate a plurality of execution times of each stage allocating different layers and each layer configuration different kinds of ReC policies (for the sake of distinction, the execution time at this time is called "second execution time") according to the number of stages, the kind of ReC policy configured by each layer, the first mapping relationship of each layer, the execution calculation amount of each layer configuration different kinds of ReC policies in different stages, and the execution time relationship of each layer configuration different kinds of ReC policies. The policy calculation unit 214 can calculate a plurality of memory usages of each stage allocating different layers and each layer configuration different kinds of ReC policies according to the number of stages, the kind of ReC policy configured by each layer, the first mapping relationship of each layer, and the memory usage of each layer configuration different kinds of ReC policies in different stages.

[0123] The policy calculation unit 214 can utilize a solving algorithm to solve the plurality of second execution times of each stage and the plurality of memory usages of each stage under the quantity constraint condition, the sequence constraint condition and the memory constraint condition, to obtain the ReC policy type of each layer and the layer configuration of each stage that satisfy the first condition and the second condition, and thus obtain the deployment manner.

[0124] The first condition can refer to matching between the plurality of second execution times of the plurality of stages. The matching between the plurality of second execution times of the plurality of stages refers to that the difference between the plurality of second execution times is as small as possible, i.e., the execution times of the stages are as uniform as possible. However, due to actual situations, such as resource configuration differences of the computing cluster, differences of the ReC policy, differences of the neural network itself structure involved in the deployed task, etc., the matching can be manifested in that the execution times of the plurality of stages under the deployment manner are the same, or the difference between the execution times is less than a set time threshold, or the difference between the execution times tends to 0 or a minimum.

[0125] The second condition can refer to matching between the memory usages of the plurality of stages and the memory resources that the computing nodes corresponding to each stage can provide. The matching between the memory usages of the plurality of stages and the memory resources that the computing nodes corresponding to each stage can provide refers to that the difference between the memory usages of the plurality of stages under the deployment manner is as small as possible, i.e., the memory usages of the stages are as uniform as possible. However, due to actual situations, such as resource configuration differences of the computing cluster, differences of the ReC policy, differences of the neural network itself structure involved in the deployed task, etc., the matching can be manifested in that the proportions of the memory usages of the plurality of stages under the deployment manner to the memory resources that the computing nodes corresponding to the stages can provide are the same, or the difference between the proportions is less than a first set memory threshold, or the difference between the proportions tends to 0 or a minimum. The matching can also be manifested in that the sum of the memory usages of each layer of the plurality of stages under the deployment manner is the same, or the difference between the sums is less than a second set memory threshold, or the difference between the sums tends to 0 or a minimum.

[0126] The solving algorithm can be integer linear programming (ILP). When solving an ILP problem, a specific algorithm is usually needed to find an integer value solution. An ILP problem is an extension of a linear programming problem, in which variables need to take integer values, rather than real values. Illustratively, the strategy calculation unit 214 can use ILP linear programming to solve the quantity constraint condition, the order constraint condition, the memory constraint condition, the plurality of second execution times of each stage, and the plurality of memory usages of each stage, with the total execution time of the neural network model being the shortest as the objective function, to calculate each stage in which the second execution time tends to be the same, the ratio of the memory usage of each stage to the memory amount of the computing node deployed in the stage tends to be the same, or the memory usage of each stage tends to be the same, thereby obtaining the deployment manner.

[0127] The training device 220 is configured to deploy the model task to the computing nodes corresponding to the plurality of stages according to the deployment manner. The training device 220 can divide the PP splitting unit 221 and the ReC strategy configuration unit 222 according to the executed task.

[0128] After obtaining the deployment manner, the PP splitting unit 221 can split the layers of the neural network model using the PP splitting strategy, and split each layer to the computing node deployed in the corresponding stage according to the deployment manner. The PP splitting unit 221 can split the model task and the related parameters of the layer of the neural network model according to the split stage, divide the model task and the related parameters of each layer in a stage together as a grouping data, and distribute the grouping data to the computing node corresponding to the corresponding stage.

[0129] After obtaining the deployment manner, the ReC strategy configuration unit 222 can use the ReC configuration strategy to configure the ReC strategy configuration scheme according to the deployment manner, and send the ReC strategy configuration scheme to each computing node to configure different types of ReC strategies for each layer constructed locally by the computing node.

[0130] When the computing node trains or infers the local layer based on the model task, the computing node can discard some calculation results according to the ReC strategy configured by itself to reduce the memory consumption in the computing node. The plurality of computing nodes in the computing device cluster can fully use the local computing resources and memory resources according to the split and distributed layers and the types of ReC strategies, so as to improve the resource utilization of the computing device cluster running the training system.

[0131] It should be understood that the functional modules, functional devices, etc. involved in the training system 200 can also be realized by software or hardware, and the specific implementation can be determined according to actual conditions, which is not limited here. In addition, the functional modules, functional devices, etc. involved in the training system 200 can be arranged separately or integrated, which is not limited here.

[0132] The above is the introduction of the training system 200 provided by the embodiment of the application. It can be understood that the training system 200 described above can be configured on a cloud computing platform, for example, deployed on at least one virtual machine or container instance, so that the cloud computing platform can provide model training services. Of course, the training system 200 can also be configured on nodes other than the cloud computing platform, for example, can be deployed in at least one data center, or deployed on at least one server, and the specific implementation can be determined according to actual conditions, which is not limited here. Among them, the cloud computing platform can provide a page related to the public cloud service for users to remotely access the public cloud service. In this embodiment, the user can purchase the model training service provided by the training system 200 in advance on the cloud computing platform. For ease of understanding, the interaction mode between the user and the cloud computing platform is described below.

[0133] As shown in FIG. 3, the interaction between the user and the cloud computing platform mainly includes: the user logs in to the cloud computing platform 300 through the client web page, selects and purchases the cloud service (i.e. the model training service) related to the training system 200 in the cloud computing platform 300, and after the purchase, the user can generate the training system 200 on the cloud computing platform 300 based on the functions provided by the model training service. Among them, the cloud computing platform 300 is mainly used to manage the infrastructure running the model training service. For example, the infrastructure of the model training service can include multiple data centers set in different regions, and each data center includes multiple servers. The data center can provide basic resources such as computing resources and storage resources for the model training service. Therefore, when the user purchases and uses the model training service, the user mainly pays for the resources used. When the user uses the model training service, the user can input the demand for model training through the configuration interface, the application program interface (API) or the interface for user interaction provided by the cloud computing platform 300, and the cloud computing platform 300 can generate a model training matched with the user's demand according to the demand input by the user (or other software / hardware, etc.).

[0134] In addition, the modules in the training system 200 can also be partially configured on the cloud side and partially configured on the terminal side, so as to realize model training through end-cloud collaboration. In addition, the training system 200 can also be entirely configured on the terminal side, and the specific implementation can be determined according to actual conditions, which is not limited here.

[0135] The above is the introduction of the training system provided by the embodiments of the present application. Next, based on the above, the deployment method of the model task provided by the embodiments of the present application is introduced.

[0136] For example, FIG. 4 shows a flowchart of a model task deployment method provided by an embodiment of the present application. It can be understood that the method can be executed by the training system 200 described above, and the specific implementation process is as follows:

[0137] Step S401, M first mapping relationships corresponding to N layers in the neural network model are obtained.

[0138] Generally, the neural network model can deploy multiple layers thereof on different computing nodes respectively, and then train or infer by different computing nodes. When the computing device cluster trains or infers the neural network model, the performance analyzer can be used to obtain the stage corresponding to the computing node where each layer of the neural network model is located, the memory usage of each layer, the type of ReC strategy configured for each layer, the execution time of each layer, operator information of each layer, and the like. The operator information can include the identifier of each operator, the execution time of each operator, the layer where each operator is located, the computation amount of each operator, and the like. The computing device cluster can store each time the obtained basic data in a common storage area for collection.

[0139] After the training system 200 collects the basic data, it can calculate higher-level data, referred to as historical data, according to the relationship between each data in the basic data.

[0140] In one embodiment, the training system 200 can obtain the execution time of each layer of the neural network model configured with different types of ReC strategies in different stages (for distinction, the execution time at this time is referred to as "first execution time") according to the stage corresponding to the computing node where each layer of the neural network model is located, the type of ReC strategy configured for each layer, and the execution time of each layer when the neural network model is trained or inferred each time.

[0141] In another embodiment, the training system 200 can obtain the memory usage of each layer of the neural network model configured with different types of ReC strategies in different stages according to the stage corresponding to the computing node where each layer of the neural network model is located, the type of ReC strategy configured for each layer, and the memory usage of each layer when the neural network model is trained or inferred each time.

[0142] In another embodiment, the training system 200 can obtain a plurality of first mapping relationships of each layer according to the first execution time of each layer configuration different kinds of ReC strategies in different stages and the memory usage of each layer configuration different kinds of ReC strategies in different stages. Each first mapping relationship is a mapping relationship between the memory usage and the first execution time of one layer configuration one kind of ReC strategy in the same stage.

[0143] In another embodiment, the training system 200 can obtain the operator information of each layer configuration different kinds of ReC strategies in different stages in the neural network model according to the stage corresponding to the computing node where each layer is located, the kind of ReC strategy configured for each layer and the operator information of each layer when the neural network model is trained or inferred each time.

[0144] The historical data can be generated when the training system is pre-trained. For example, the training system can pre-train the neural network model to obtain the historical data of the neural network model before obtaining the deployment manner of the model task on the plurality of computing nodes by the deployment device 210 using the model task.

[0145] Alternatively, the training system 200 analyzes the model task to obtain a log of the model task. The log usually includes the execution time of each layer or operator in the layer. Based on the log, the training system 200 can obtain the first execution time of each layer configuration different kinds of ReC strategies in different stages, the M first mapping relationships of each layer and other data.

[0146] The device parameters include the memory amount of each computing node and the computing amount of each computing node. The memory amount refers to the memory usage or the idle memory amount of all memories in a computing node when the computing node provides the neural network model for training or inference. The computing amount refers to the computing amount or the idle computing amount of all processors in a computing node when the computing node provides the neural network model for training or inference.

[0147] The above step S401 can be performed by the data acquisition unit 211 in the training system 200.

[0148] In step S402, the deployment manner of the model task on the plurality of computing nodes of the computing device cluster that satisfies the first condition and the second condition is obtained based on the M first mapping relationships.

[0149] The training system 200 can obtain the running time interval of the data packet of the micro-batch of a single pipeline from the performance analysis data. The running time interval of the pipeline includes a start time and an end time. After obtaining the running time interval of a pipeline, the training system 200 can obtain the operator information of all operators in the running time interval of the pipeline from the operator information of the operators in different stages of each layer configuration of different kinds of ReC strategies. The training system 200 can calculate the computation amount of each layer configuration of different kinds of ReC strategies in different stages of the neural network model according to the operators in the running time interval of the pipeline, the layer in which each operator is located, and the computation amount of each operator.

[0150] The training system 200 can calculate the execution computation amount of each layer configuration of different kinds of ReC strategies in different stages of the neural network model according to the computation amount of each layer configuration of different kinds of ReC strategies in different stages of the neural network model and the first execution time of each layer configuration of different kinds of ReC strategies in different stages obtained from the historical data. The model task deployment apparatus 210 optimizes the computation performance of the neural network model by using the execution computation amount of each layer configuration of different kinds of ReC strategies in different stages of the neural network model, which can eliminate the influence of the computation / communication overlapping degree between layers and the influence on the execution time.

[0151] The kinds of ReC strategies executed by the training system under different AI frameworks are different, and the execution time of different types of ReC strategies is also different. In the embodiment of the present application, the training system 200 can obtain the first execution time of each layer from the historical data to obtain the execution time relationship of each layer configuration of different kinds of ReC strategies in the neural network model. During the optimization of the computation performance of the neural network model, the model task deployment apparatus 210 can configure the most suitable kind of ReC strategy for each layer based on the execution time relationship of each layer configuration of different kinds of ReC strategies, so that the model task deployment apparatus 210 can configure the most suitable kind of ReC strategy for each layer in combination with the execution time relationship.

[0152] After the training system 200 obtains the memory usage of each layer configuration of different kinds of ReC strategies in different stages from the historical data, the training system 200 can sum the memory amounts of the layers in the same stage and configured with the same ReC strategy to obtain the memory usage of all layers configured with one kind of ReC strategy in each stage, so as to obtain the memory usage of each stage configuration of different kinds of ReC strategies.

[0153] After obtaining the memory usage of each stage configured with different kinds of ReC policies, the training system 200 can analyze the number of layers of each stage configured with the same kind of ReC policy and the kind of ReC policy. The training system 200 can calculate the memory usage of each layer configured with different kinds of ReC policies in different stages and the memory usage of the activation values of each layer configured with different kinds of ReC policies according to the memory usage of each stage configured with different kinds of ReC policies except for the first stage and the last stage, the number of layers of each stage configured with different kinds of ReC policies except for the first stage and the last stage, and the kind of ReC policy.

[0154] The training system 200 can calculate the memory usage of the first stage configured with different kinds of ReC policies according to the number of layers of the first stage configured with the same kind of ReC policy, the kind of ReC policy configured by the layer of the first stage, the memory usage of each layer configured with different kinds of ReC policies in the first stage, and the memory usage of the activation values of each layer configured with different kinds of ReC policies.

[0155] The training system 200 can calculate the memory usage of the last stage configured with different kinds of ReC policies according to the number of layers of the last stage configured with the same kind of ReC policy, the kind of ReC policy configured by the layer of the last stage, the memory usage of each layer configured with different kinds of ReC policies in the last stage, and the memory usage of the activation values of each layer configured with different kinds of ReC policies.

[0156] The training system 200 can obtain the memory usage of the first stage configured with different kinds of ReC policies and the memory usage of the last stage configured with different kinds of ReC policies from the memory usage obtained by the training system 200 from each stage configured with different kinds of ReC policies, and refer to the measured memory usage of the first stage configured with different kinds of ReC policies and the measured memory usage of the last stage configured with different kinds of ReC policies. The training system 200 can subtract the measured memory usage of the first stage configured with different kinds of ReC policies from the calculated memory usage of the first stage configured with different kinds of ReC policies to calculate the additional memory amount of the first stage configured with different kinds of ReC policies. In addition, the training system 200 can subtract the measured memory usage of the last stage configured with different kinds of ReC policies from the calculated memory usage of the last stage configured with different kinds of ReC policies to calculate the additional memory amount of the last stage configured with different kinds of ReC policies.

[0157] The training system 200 can obtain the number of layers in the neural network model from the definition file of the neural network model, and then determine the number-of-layers constraint condition of all stages according to the number of stages and the number of layers in the neural network model. The number-of-layers constraint condition refers to a condition that the number of layers allocated to all stages is equal to the number of layers in the neural network model.

[0158] The training system 200 can obtain the execution order of each layer in the neural network model from the definition file of the neural network model, and then determine the order constraint condition of each layer in each stage according to the execution order of each layer in the neural network model. The order constraint condition satisfies the pipeline parallel PP policy. The pipeline parallel PP policy refers to that the stage in which the next layer of each layer is located is the stage in which the layer is located, or the previous stage of the stage in which the layer is located, or the next stage of the stage in which the layer is located, or the parallel stage of the stage in which the layer is located, and the stage in which the previous layer of each layer is located is the stage in which the layer is located, or the next stage of the stage in which the layer is located, or the previous stage of the stage in which the layer is located, or the parallel stage of the stage in which the layer is located.

[0159] The training system 200 can obtain the memory constraint condition of each stage according to the memory usage of each layer configured with different kinds of ReC strategies in different stages, the memory usage of the activation value of each layer configured with different kinds of ReC strategies, the additional memory amount of the first stage configured with different kinds of ReC strategies, the additional memory amount of the last stage configured with different kinds of ReC strategies, and the memory amount of the computing node corresponding to each stage.

[0160] The training system 200 can calculate the multiple execution times of each stage allocated with different layers and each layer configured with different kinds of ReC strategies (for the sake of distinction, the execution time at this time is referred to as “second execution time”) according to the number of stages, the kind of ReC strategy configured for each layer, the first mapping relationship of each layer, the execution amount of each layer configured with different kinds of ReC strategies in different stages, and the execution time relationship of each layer configured with different kinds of ReC strategies.

[0161] The training system 200 can use a solving algorithm to solve the multiple second execution times of each stage and the multiple memory usages of each stage under the condition of satisfying the quantity constraint condition, the sequence constraint condition, and the memory constraint condition, to obtain each layer in each stage and the kind of ReC strategy configured for each layer that satisfy the first condition and the second condition, and thus obtain the deployment mode.

[0162] The first condition can mean that the second execution times of each stage are the same or tend to be the same. The first condition can mean that the difference between the second execution times of each stage is less than a set time threshold. The first condition can mean that the second execution times of each stage or the difference between them tends to 0 or a minimum.

[0163] The second condition can mean that the proportion of the memory usage of each stage to the memory amount of the computing node deployed in the stage is the same or tends to be the same. The second condition can mean that the difference between the proportion of the memory usage of each stage to the memory amount of the computing node deployed in the stage is less than a first set memory threshold. The second condition can mean that the difference between the proportion of the memory usage of each stage to the memory amount of the computing node deployed in the stage tends to 0 or a minimum.

[0164] The second condition can refer to the memory usage of each stage being the same or tending to be the same. The second condition can refer to the difference between the memory usage of each stage being less than a second set memory threshold. The second condition can refer to the difference between the difference between the memory usage of each stage tending to 0 or being minimal. The step S402 can be performed by the calculation analysis unit 212, the memory analysis unit 213, and the strategy calculation unit 214 in the training system.

[0165] At step S403, the model task is deployed according to the deployment mode.

[0166] After the training system 200 obtains the deployment mode, the PP partition strategy can be used to partition the layers of the neural network model, and each layer can be partitioned into the corresponding computing node of the stage according to the deployment mode. The training system 200 can partition the model task and the related parameters of the layers of the neural network model according to the partitioned stages, and divide the model task and the related parameters of each layer in a stage together as a group of data and distribute them to the corresponding computing node of the corresponding stage.

[0167] After the training system 200 obtains the deployment mode, the ReC configuration strategy can be used to configure the ReC strategy configuration scheme according to the deployment mode, and the ReC strategy configuration scheme can be sent to each computing node to configure different types of ReC strategies for each layer constructed locally by the computing node.

[0168] When the computing node trains or infers the local layer based on the model task, some calculation results can be discarded according to the ReC strategy configured by the computing node to reduce the memory consumption in the computing node. The multiple computing nodes in the computing device cluster can fully use the local computing resources and memory resources according to the partitioned layers and the types of ReC strategies to improve the resource utilization of the computing device cluster running the training system.

[0169] The step S403 can be performed by the PP partition unit 221 and the ReC strategy configuration unit 222 in the training system.

[0170] FIG. 5 is a structural schematic diagram of a model task deployment device provided in an embodiment of the present application. As shown in FIG. 5, the model task deployment device 500 can be divided into a first processing unit 510, a second processing unit 520, and a third processing unit 530 according to the execution function. The model task deployment device 500 specifically implements the following process:

[0171] The first processing unit 510 is configured to obtain M first mapping relationships corresponding to N layers in a neural network model. Each of the N layers is configured with one or more ReC strategies. The first mapping relationship is a mapping relationship between a memory amount and a first execution time. The first execution time is a time for running a layer in a stage using the memory amount based on a ReC strategy. The stage is a part of a model task. The model task is training the neural network model or performing inference using the neural network model. N is an integer greater than 1. M is an integer greater than or equal to N. The number of layers of the neural network model is greater than or equal to N. The second processing unit 520 is configured to obtain, based on the M first mapping relationships, a deployment manner of the model task on a plurality of computing nodes of a computing device cluster that satisfies a first condition and a second condition. The deployment manner is used to indicate a computing node corresponding to each stage of the model task and a ReC strategy corresponding to each layer of each neural network model in each stage. Each computing node includes one or more computing devices. The first condition is that a plurality of second execution times match. The plurality of second execution times are respective execution times of a plurality of stages of the model task under the deployment manner. The second condition is that a memory usage of each stage matches a memory resource that can be provided by a computing node corresponding to the stage. The third processing unit 530 is configured to deploy the model task according to the deployment manner.

[0172] In an embodiment, the second processing unit 520 is specifically configured to obtain, according to the M first mapping relationships, a plurality of third execution times and a memory usage corresponding to each of the plurality of third execution times. Each of the plurality of third execution times indicates an execution time of a stage of the model task. The stage corresponding to each of the plurality of third execution times is different in allocation of layers and / or different in configuration of ReC strategies of the layers. The second processing unit 520 is specifically configured to select, based on the first condition and the second condition, the plurality of second execution times from the plurality of third execution times. The second processing unit 520 is specifically configured to obtain, based on the allocation of layers and the configuration of ReC strategies of the layers in the stage corresponding to each of the plurality of second execution times, the deployment manner.

[0173] In an embodiment, the first processing unit 510 is specifically configured to obtain the M first mapping relationships corresponding to the N layers based on historical data. The historical data includes execution times of different layers in the neural network model configured with different ReC strategies in different stages of the model task.

[0174] In an embodiment, the first processing unit 510 is specifically configured to analyze the model task to obtain a log of the model task. The first processing unit 510 is specifically configured to obtain the M first mapping relationships corresponding to the N layers based on the log of the model task.

[0175] In an implementation, the condition met by the deployment manner further includes that the memory usage of each stage is less than or equal to the memory resource that can be provided by the computing node corresponding to each stage.

[0176] In an implementation, the condition met by the deployment manner further includes that the deployment manner of the model task meets a pipeline parallelism (PP) strategy.

[0177] In an implementation, the third processing unit 530 is specifically configured to instruct the N layers to perform the model task based on the ReC strategy configured by each layer in the deployment manner and the stage in which the layer is located.

[0178] In an implementation, the second processing unit 520 is specifically configured to obtain the computation amount of different kinds of ReC strategies configured by the N layers in different stages. The second processing unit 520 is specifically configured to obtain a plurality of execution computation amounts and a plurality of execution time relationships of the N layers according to the computation amount of different kinds of ReC strategies configured by the N layers in different stages and the M first mapping relationships. The execution computation amount refers to the relationship between the execution time and the computation amount of one layer in one stage. The execution time relationship refers to the relationship between a plurality of execution times corresponding to different kinds of ReC strategies configured by one layer in one stage. The second processing unit 520 is specifically configured to obtain a plurality of third execution times according to the plurality of execution computation amounts and the plurality of execution time relationships of the N layers.

[0179] In an implementation, the second processing unit 520 is specifically configured to obtain the deployment manner meeting the second condition according to the memory usage of different kinds of ReC strategies configured by each layer in different stages, the memory usage of the activation value of different kinds of ReC strategies configured by each layer, the additional memory amount of different kinds of ReC strategies configured by the first stage, the additional memory amount of different kinds of ReC strategies configured by the second stage, and the memory resource that can be provided by the computing node corresponding to each stage. The first stage refers to the first stage executed in the plurality of stages. The second stage refers to the last stage executed in the plurality of stages.

[0180] The following results of simulation experiments prove the technical effects of the technical solutions protected by the present application.

[0181] It is assumed that the neural network model is an LLAMA2-70B neural network model. The deep learning framework is a mindspore framework. The training system is deployed on a computing device cluster composed of 60 computing devices, and the memory of each computing device is 60 GB. The neural network model is divided into 8 stages. The grouping data number (micro_batch_num) is 128.

[0182] The training system analyzes the calculation amount of each layer from the performance analysis data as {'LlamaEmbedding': 10840, 'LlamaRMSNorm': 3470, 'Linear': 614, 'DecodeLayer': 15633}. The training system calculates the execution time of each layer from the performance analysis data as {'LlamaEmbedding': 26814, 'LlamaRMSNorm': 5178, 'Linear': 703, 'DecodeLayer': 26788}. The training system calculates the execution calculation of each layer according to the calculation amount and the execution time, as {'LlamaEmbedding': 15657, 'LlamaRMSNorm': 3028, 'Linear': 409, 'DecodeLayer': 15633}. The training system obtains 2 kinds of re-computation strategies supported by the mindspore framework for the LLAMA2-70B neural network model, which are non-re-computation and full re-computation. The training system calculates the execution time ratio of the 2 kinds of re-computation strategies as {1:1.4}.

[0183] The training system obtains the memory consumption, the number of re-computation strategies, and the number of layers of each pipeline stage. As shown in Table 1, the memory amount of the pipeline stage (stage 0) is 56975 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 1) is 34933 MB, the number of re-computation strategies is 9 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 2) is 45114 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 3) is 40457 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 4) is 35801 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 5) is 31145 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 6) is 26488 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers. The memory amount of the pipeline stage (stage 7) is 24293 MB, the number of re-computation strategies is 8 layers, and the number of layers is 10 layers.

[0184] Table 1 is the memory amount, the number of re-computation strategies, and the number of layers of each pipeline stage

[0185] The training system selects the pipeline stage (stage 1), the pipeline stage (stage 2), and the pipeline stage (stage 3) from the 8 pipeline stages and inputs them into equation (1), and substitutes ∑l∈L|l i = 10, S = 8 into equation (1), to obtain the following equation group:

[0186] Stage 1, i = 1:

[0187] Stage 2, i = 2:

[0188] Stage 3, i = 3:

[0189] The training system calculates the memory amount M P = 1717 of the activation values of the non-recompute strategy The memory amount of the activation values of the recompute strategy

[0190] The training system inputs the first pipeline stage (stage 0) and the last pipeline stage (stage 7) into equation (2), and substitutes ∑l∈L|l i = 10, S = 8, M P = 1717, into equation (2), to obtain the following equation group:

[0191] Stage 0, i = 0:

[0192] Stage 7, i = 7:

[0193] The training system calculates the memory amount E0= 2547 of each layer of the first pipeline stage and the memory amount E S7 = 1763 of each layer of the last stage.

[0194] The training system determines that the order constraint condition is LlamaEmbedding→DecodeLayer→LlamaRMSNorm→Linear.

[0195] The training system determines that the total number of DecodeLayer in the quantity constraint pipeline stage is equal to 80: x_0+x_1+x_2+x_3+x_4+x_5+x_6+x_7+y_0+y_1+y_2+y_3+y_4+y_5+y_6+y_7=80. Among them, the number represents the stage of the pipeline stage, x represents the number of layers configured for recalculation, and y represents the number of layers configured for non-recalculation.

[0196] The training system calculates the memory constraint condition of each pipeline stage as: 2045*x_0+19005*y_0<=57453 2004*x_1+16844*y_1<=60000 1963*x_2+14683*y_2<=60000 1922*x_3+12522*y_3<=60000 1881*x_4+10361*y_4<=60000 1840*x_5+8200*y_5<=60000 1799*x_6+6039*y_6<=60000 1758*x_7+3878*y_7<=58237

[0197] Among them, 2045=M in the first equation P That is, That is, 41, 19005=M P That is, That is, 2161, 57453=machine memory limit 60000-Additional memory amount E of the first stage S0 That is, 2547. The values of the respective equations in the middle are derived in the same way as the first equation. In the last equation, 58237=machine memory limit 60000-Additional memory amount E of the stage S7 That is, 1763.

[0198] The training system determines the relationship between the total execution time of the LLAMA2-70B neural network model, the relationship between each layer, each stage in the LLAMA2-70B neural network model, and the ReC strategy configured by each layer as:

[0199] Among them, T is the total execution time of the pipeline, t_(S-1) is the execution time of the last stage of the pipeline, is the sum of the execution time of all pipeline stages (stage) of the pipeline training a single micro_batch, max(t i ) is the maximum execution time of training a single micro_batch in all stages, t iExecution time of training a single micro batch for stage with index i, l i r Execution time of layer at stage with index i and recompute policy r, dur exec Execution computation of layer, r.speed is execution time relationship of recompute policy r.

[0200] According to formula (4), the training system can split the total pipeline execution time into three parts: pipeline_total_time (T in the formula) is composed of three parts, lastFP_BP (t S-1 ), max_stage_time (max(t i )) and var_sum_FPi_BPi (in the formula ), and generate the relationship between the layers and recompute policies on the pipeline stage and each part, and then generate the relationship with the total pipeline execution time: lastFP_BP-21886.2*x_7-15633*y_7>=3437

[0201] Wherein, 21886.2 is the execution time of 'DecodeLayer' obtained by the training system 15633 multiplied by the full recompute execution time relationship 1.4, and 3437 is the sum of the execution computation of 'LlamaRMSNorm' and 'Linear'. sum_FPi_BPi+var_sum_FPi_BPi-21886.2*x_0-21886.2*x_1 21886.2*x_2-21886.2*x_3-21886.2*x_4-21886.2*x_5-21886.2*x_6 21886.2*x_7-15633*y_0-15633*y_1-15633*y_2-15633*y_3-15633*y_4-15633*y_5-15633*y_6 -15633*y_7>=30251

[0202] Wherein, 30251 is the sum of the execution calculation amount of 'LlamaRMSNorm', 'Linear' and 'LlamaEmbedding' obtained by the training system. max_stage_time-21886.2*x_0-15633*y_0>=26814 max_stage_time-21886.2*x_1-15633*y_1>=0 max_stage_time-21886.2*x_2-15633*y_2>=0 max_stage_time-21886.2*x_3-15633*y_3>=0 max_stage_time-21886.2*x_4-15633*y_4>=0 max_stage_time-21886.2*x_5-15633*y_5>=0 max_stage_time-21886.2*x_6-15633*y_6>=0 max_stage_time-21886.2*x_7-15633*y_7>=3437 lastFP_BP-126*max_stage_time+pipeline_total_time-var_sum_FPi_Bpi

[0203] Wherein, 126 is the result of micro_batch_num, that is, 128-2.

[0204] The training system outputs the optimal PP splitting strategy and ReC configuration strategy according to the above constraint conditions and time relationships, which are as follows: stage0: x=6, y=2 stage1: x=7, y=2 stage2: x=7, y=3 stage3: x=7, y=3 stage4: x=6, y=4 stage5: x=4, y=6 stage6: x=2, y=9 stage7: x=0, y=12

[0205] The training system can train the LLAMA2-70B neural network model based on the calculation graph generated based on the above PP splitting strategy and ReC configuration strategy. The training system trains using the model task deployment device 210 and not using the model task deployment device 210, and the memory consumption of each pipeline stage obtained is shown in Table 2.

[0206] Table 2 Memory consumption of each pipeline stage obtained by the training system using the model task deployment device 210 and not using the model task deployment device 210

[0207] The training system uses the deployment device 210 of the model task and the deployment device 210 without the model task to train, and the total execution time of the completion is shown in Table Three. Compared with the scheme without the deployment device 210 of the model task, the total execution time of the training system using the deployment device 210 of the model task to train is accelerated by 120%.

[0208] Table Three Total execution time of the training system using the deployment device 210 of the model task and the deployment device 210 without the model task to train

[0209] Figure 6 is a structural schematic diagram of a computing device provided in the embodiments of the present application. As shown in Figure 6, the computing device 700 includes a bus 710, a processor 720, a memory 730 and a communication interface 740. The processor 720, the memory 730 and the communication interface 740 communicate through the bus 710. The computing device 700 can be a server, a computer, a portable notebook, a cabinet, etc. It should be understood that the number of processors and memories in the computing device 700 is not limited in the present application.

[0210] The bus 710 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one line is shown in Figure 6, but it does not mean that there is only one bus or one type of bus. The bus 710 can include a path for transmitting information between various components (for example, the processor 720, the memory 730, the communication interface 740) of the computing device 700.

[0211] The processor 720 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0212] The memory 730 can include volatile memory (such as random access memory (RAM)) and non-volatile memory (such as read-only memory (ROM), floppy, mechanical, solid state, or electrical medium memories, or a combination of these memories).

[0213] The memory 730 stores executable program code that is executed by the processor 720 to implement the functions of the aforementioned modules, such as the first processing unit 510, the second processing unit 520, and the third processing unit 530, to implement the model task deployment method.

[0214] Alternatively, the memory 730 stores executable program code that is executed by the processor 720 to implement the functions of the aforementioned modules to implement the model task deployment method. That is, the memory 730 stores instructions for executing the model task deployment method.

[0215] The communication interface 740 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 700 and other devices or communication networks.

[0216] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0217] As shown in FIG. 7, the computing device cluster includes at least one computing device 700. The memory 730 in one or more computing devices 700 in the computing device cluster can store the same instructions for executing the model task deployment method.

[0218] In some possible implementations, the memory 730 of one or more computing devices 700 in the computing device cluster can also respectively store partial instructions for executing the model task deployment method. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for executing the model task deployment method.

[0219] It should be noted that the memories 730 in different computing devices 700 in the computing device cluster can store different instructions for respectively performing part of the functions of the first processing unit 510, the second processing unit 520 and the third processing unit 530 described above. That is, the instructions stored in the memories 730 in different computing devices 700 can implement the functions of one or more modules in the first processing unit 510, the second processing unit 520 and the third processing unit 530 described above.

[0220] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 8 shows a possible implementation manner. As shown in FIG. 8, two computing devices are respectively a computing device 700A and a computing device 700B, which are connected through a network. Specifically, the computing devices are connected to the network through communication interfaces in the computing devices. In this kind of possible implementation manner, the memory 730 in the computing device 700A stores instructions for performing the functions of part of the modules in the first processing unit 510, the second processing unit 520 and the third processing unit 530 described above. Meanwhile, the memory 730 in the computing device 700B stores instructions for performing the functions of another part of the modules in the first processing unit 510, the second processing unit 520 and the third processing unit 530 described above.

[0221] The connection manner between the computing device cluster shown in FIG. 8 can be that, considering that the deployment method of the model task provided in the present application needs to store a large amount of data, it is considered to assign the functions implemented by another part of the modules in the first processing unit 510, the second processing unit 520 and the third processing unit 530 to the computing device 700B to perform.

[0222] It should be understood that the functions of the computing device 700A shown in FIG. 8 can also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be completed by multiple computing devices 700.

[0223] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to FIG. 6 and FIG. 8. The difference is that the memories 730 in one or more computing devices 700 in the computing device cluster can store the same instructions for performing the deployment method of the model task.

[0224] In some possible implementation manners, the memories 730 of one or more computing devices 700 in the computing device cluster can also respectively store part of the instructions for performing the deployment method of the model task. In other words, the combination of one or more computing devices 700 can collectively execute the instructions for performing the deployment method of the model task.

[0225] It should be noted that the memories 730 in different computing devices 700 in the computing device cluster can store different instructions for performing part of the functions of the computing devices 700. That is, the memories 730 in different computing devices 700 store instructions that can implement the functions of one or more modules of the first processing unit 510, the second processing unit 520, and the third processing unit 530 described above.

[0226] Embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the deployment method of the model task.

[0227] Embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the deployment method of the model task.

[0228] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model task deployment method, characterized in that, The method comprises: obtaining M first mapping relationships corresponding to N layers in a neural network model; each of the N layers applies one or more ReC strategies, the first mapping relationship is a mapping relationship between a memory amount and a first execution time, the first execution time is a time for running the layer in a stage based on one of the ReC strategies using the memory amount, the stage is a part of a model task, the model task is training the neural network model or performing inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, the number of layers of the neural network model is greater than or equal to N; based on the M first mapping relationships, obtaining a deployment manner of the model task on multiple computing nodes of a computing device cluster that satisfies a first condition and a second condition; the deployment manner is used to indicate a computing node corresponding to each stage of the model task and a ReC strategy corresponding to each layer of each neural network model in the stage, each computing node includes one or more computing devices, the first condition is matching between multiple second execution times, the multiple second execution times are respective execution times of multiple stages of the model task under the deployment manner; the second condition is that the memory usage of each stage matches the memory resources that can be provided by the computing node corresponding to the stage; deploying the model task according to the deployment manner.

2. The method of claim 1, wherein, The method comprises: obtaining multiple third execution times and memory usage corresponding to the multiple third execution times according to the M first mapping relationships; each third execution time indicates an execution time of a stage of the model task, the stage corresponding to each third execution time is allocated with different layers and / or different ReC strategies of the layers; based on the first condition and the second condition, selecting the multiple second execution times from the multiple third execution times; based on the layers allocated in the stages corresponding to the multiple second execution times and the ReC strategies of the layers, obtaining the deployment manner.

3. The method according to claim 1 or 2, characterized in that, The method comprises: obtaining the M first mapping relationships corresponding to the N layers based on historical data; the historical data includes execution times of different layers in the neural network model configured with different ReC strategies in different stages of the model task.

4. The method according to claim 1 or 2, characterized in that, The method comprises: profiling the model task to obtain logs of the model task; obtaining the M first mapping relationships corresponding to the N layers based on the logs of the model task.

5. The method according to any one of claims 1 to 4, characterized in that, The condition met by the deployment manner further includes that the memory usage of each stage is less than or equal to the memory resources that can be provided by the computing node corresponding to the stage.

6. The method according to any one of claims 1 to 5, characterized in that, The condition met by the deployment manner further includes that the deployment manner of the model task meets a pipeline parallel (PP) strategy.

7. The method according to any one of claims 1 to 6, characterized in that, The deployment of the model task according to the deployment manner specifically includes: Based on the ReC strategy configured by each layer in the deployment manner and the stage, the N layers are instructed to execute the model task.

8. The method according to any one of claims 2 to 7, characterized in that, The plurality of third execution times are obtained according to the M first mapping relationships, specifically including: The calculation amount of different types of ReC strategies configured by the N layers in different stages is obtained; According to the calculation amount of different types of ReC strategies configured by the N layers in different stages and the M first mapping relationships, a plurality of execution calculation amounts and a plurality of execution time relationships of the N layers are obtained; the execution calculation amount refers to the relationship between the execution time and the calculation amount of one layer in one stage; the execution time relationship refers to the relationship between a plurality of execution times corresponding to different types of ReC strategies configured by one layer in one stage; The plurality of third execution times are obtained according to the plurality of execution calculation amounts and the plurality of execution time relationships of the N layers.

9. The method according to any one of claims 1 to 8, characterized in that, Based on the M first mapping relationships, the deployment manner of the model task on the plurality of computing nodes of the computing device cluster that meets the first condition and the second condition is obtained, specifically including: According to the memory usage of different types of ReC strategies configured by each layer in different stages, the memory usage of the activation value of different types of ReC strategies configured by each layer, the additional memory amount of different types of ReC strategies configured by the first stage, the additional memory amount of different types of ReC strategies configured by the second stage, and the memory resources that can be provided by the computing node corresponding to each stage, the deployment manner that meets the second condition is obtained; the first stage refers to the first executed stage in the plurality of stages; the second stage refers to the last executed stage in the plurality of stages.

10. A model task deployment apparatus characterized by comprising: It includes: The first processing unit is configured to obtain M first mapping relationships corresponding to N layers in a neural network model; each layer in the N layers applies one or more ReC strategies, the first mapping relationship is a mapping relationship between a memory amount and a first execution time, the first execution time is a time based on a ReC strategy, using the memory amount to run the layer in a stage, the stage is a part of a model task, the model task is to train the neural network model or to perform inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers of the neural network model is greater than or equal to N. a second processing unit, configured to obtain, based on the M first mapping relationships, a deployment manner of the model task on multiple computing nodes of a computing device cluster, the deployment manner being used to indicate a computing node corresponding to each stage of the model task and a ReC strategy corresponding to each layer of each neural network model in the each stage, and the each computing node comprising one or more computing devices, the first condition being matching between multiple second execution times, and the multiple second execution times being respective execution times of the multiple stages of the model task under the deployment manner, and the second condition being matching between memory usage of the each stage and memory resources provided by the computing node corresponding to the each stage; a third processing unit, configured to deploy the model task according to the deployment manner.

11. The apparatus of claim 10, wherein the second processing unit is specifically configured to obtain, based on the M first mapping relationships, multiple third execution times and memory usage corresponding to the multiple third execution times, wherein each third execution time indicates an execution time of a stage of the model task, and the each third execution time corresponds to stages in which layers are different and / or ReC strategies configured for the layers are different; select the multiple second execution times from the multiple third execution times based on the first condition and the second condition; and obtain the deployment manner based on layers allocated in stages corresponding to the multiple second execution times and ReC strategies configured for the layers.

12. The apparatus of any of claims 10 or 11, wherein the first processing unit is specifically configured to obtain the M first mapping relationships corresponding to the N layers based on historical data, and the historical data comprises execution times of different ReC strategies configured for different layers in the neural network model in different stages of the model task.

13. The apparatus of any of claims 10 or 11, wherein the first processing unit is specifically configured to analyze the model task to obtain a log of the model task; and obtain the M first mapping relationships corresponding to the N layers based on the log of the model task.

14. The apparatus of any one of claims 10-13, wherein, The conditions satisfied by the deployment manner further comprise that memory usage of the each stage is less than or equal to memory resources provided by the computing node corresponding to the each stage.

15. The apparatus of any of claims 10-14, wherein, The conditions satisfied by the deployment manner further comprise that the deployment manner of the model task satisfies a pipeline parallel (PP) strategy.

16. The apparatus of any of claims 10-15, wherein the third processing unit is specifically configured to instruct the N layers to execute the model task based on the ReC strategy configured for each layer in the deployment manner and a stage in which the each layer is located.

17. The apparatus of any of claims 11-16, wherein The second processing unit is specifically configured to obtain calculation amounts of different kinds of ReC strategies in different stages of the N layers; According to the calculation amounts of different kinds of ReC strategies in different stages of the N layers and the M first mapping relationships, a plurality of execution calculation amounts and a plurality of execution time relationships of the N layers are obtained; the execution calculation amount refers to a relationship between an execution time and a calculation amount of one layer with one kind of ReC strategy in one stage; the execution time relationship refers to a relationship between a plurality of execution times of one layer with different kinds of ReC strategies in one stage; According to the plurality of execution calculation amounts and the plurality of execution time relationships of the N layers, the plurality of third execution times are obtained.

18. The apparatus of any one of claims 10-17, wherein The second processing unit is specifically configured to obtain the deployment mode satisfying the second condition according to memory usage amounts of different kinds of ReC strategies in different stages of each layer, memory usage amounts of activation values of different kinds of ReC strategies of each layer, additional memory amounts of different kinds of ReC strategies of a first stage, additional memory amounts of different kinds of ReC strategies of a second stage, and memory resources that can be provided by a computing node corresponding to each stage; the first stage refers to a first executed stage in the plurality of stages; The second stage refers to a last executed stage in the plurality of stages.

19. A computer system, characterized in that The computer system comprises a processing unit and a storage unit; The processing unit of the computer system is configured to execute instructions stored in the processing unit, so that the computer system executes the method of any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, The computer program product stores instructions, which, when executed by a computer system, cause the computer system to implement the method of any one of claims 1-9.

21. A computer program product comprising instructions, wherein: The computer program product stores instructions, which, when executed by a computer system, cause the computer system to implement the method of any one of claims 1-9.

Images