Model task deployment method and device and computer system

Abstract

The invention discloses a model task deployment method. The method comprises the steps of obtaining M first mapping relationships corresponding to N layers in a neural network model; based on the M first mapping relationships, obtaining a deployment mode of the model task meeting the first condition and the second condition on a plurality of computing nodes of the computing device cluster; and deploying the model task according to the deployment mode. According to the method, the relationship between the memory amount and the execution time of different types of ReC strategies in different stages can be configured according to different layers in a neural network model corresponding to a model task; according to the method and the system, a deployment mode which satisfies matching between execution time of each stage and matching of the memory usage amount of each stage and the memory resource which can be provided by the computing node deployed by the stage is determined, so that a model task is deployed according to the deployment mode, the stage allocated to the computing node can fully utilize the computing resource and the memory resource in the training process, and the training efficiency of the model task is improved. The resource utilization rate of a computing device cluster running a training system is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus and computer system for deploying model tasks. Background Technology

[0002] With the rapid development of neural network (NN) models, the parameter size, number of layers, and dataset size of these models have grown exponentially. However, current hardware development struggles to keep pace with this rapid increase in the size of neural network models. Because model training often requires a large amount of memory to store and process parameters, data, and intermediate computation results, this places enormous demands on the memory capacity of hardware devices, making memory a key constraint hindering neural network model training.

[0003] In related technologies, pipeline parallelism (PP) and re-computation (ReC) can be used to overcome the memory limitations of hardware devices when training neural network models. However, due to the high complexity of configuring these two techniques, it is difficult to configure strategies that match the training scenario, thus wasting the computing and memory resources of the hardware devices. Summary of the Invention

[0004] To address the aforementioned problems, embodiments of this application provide a method for deploying model tasks, which can improve the resource utilization of the computing device cluster running the training system. Furthermore, this application also provides a model task deployment apparatus and computer system corresponding to this model task deployment method.

[0005] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0006] In a first aspect, embodiments of this application provide a method for deploying a model task, comprising: obtaining M first mapping relationships corresponding to N layers in a neural network model; wherein each of the N layers applies one or more recomputation ReC strategies, the first mapping relationship is a mapping relationship between memory amount and first execution time, the first execution time is the time for the layer in a stage to run using the memory amount based on one of the ReC strategies, the stage is a part of the model task, the model task is to train the neural network model or to perform inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers in the neural network model is greater than or equal to N; based on the M first mapping relationships... The relationship is used to obtain the deployment method of the model task on multiple computing nodes of the computing device cluster that satisfies the first and second conditions; the deployment method is used to indicate the computing node corresponding to each stage of the model task, and the ReC strategy corresponding to each layer of the neural network model in each stage, wherein each computing node includes one or more computing devices, the first condition is the matching between multiple second execution times, wherein the multiple second execution times are the respective execution times of the multiple stages of the model task under the deployment method; the second condition is the matching between the memory usage of each stage and the memory resources that the computing node corresponding to each stage can provide; the model task is deployed according to the deployment method.

[0007] The phrase "each of the N layers applies one or more ReC recomputation strategies" means that a layer can apply one or more ReC strategies. Specifically, applying multiple ReC strategies to a layer can mean that the entire layer applies multiple ReC strategies, or that some operators in a layer apply different ReC strategies (e.g., some operators apply one ReC strategy while others apply another); or that some operators in a layer apply multiple ReC strategies.

[0008] The aforementioned matching between multiple second execution times refers to minimizing the differences between the multiple second execution times, i.e., making the execution times of each stage as uniform as possible. However, due to practical considerations such as differences in computing cluster resource configuration, different ReC strategies, and different neural network structures involved in the deployed tasks, the matching results may manifest as the multiple stages having the same execution time under the aforementioned deployment method, or the difference between them being less than a set time threshold, or the difference between them tending towards 0 or minimum.

[0009] The aforementioned matching of the memory usage of each stage with the memory resources provided by the corresponding computing node means minimizing the difference in memory usage among the multiple stages under the deployment method, i.e., making the memory usage of each stage as uniform as possible. However, due to practical factors such as differences in computing cluster resource configuration, different ReC strategies, and different neural network structures involved in the deployed tasks, the matching result may be that the proportion of memory usage of each stage to the memory resources provided by its corresponding computing node is the same, or the difference is less than a first set memory threshold, or the difference tends to be 0 or minimal. The matching result may also be that the total memory usage of each layer of each stage is the same, or the difference is less than a second set memory threshold, or the difference tends to be 0 or minimal.

[0010] In this embodiment, the method can determine the relationship between the amount of memory and the execution time in different stages of different layers of the neural network model corresponding to the model task, based on the configuration of different types of ReC strategies. This determines a deployment method that matches the execution time of each stage and the memory usage of each stage with the memory resources that the computing nodes deployed in that stage can provide. This allows the model task to be deployed according to the deployment method, enabling the stages allocated to the computing nodes to make full use of computing and memory resources during the training process, thereby improving the resource utilization of the computing device cluster running the training system.

[0011] In one implementation, the step of obtaining the deployment method of the model task on multiple computing nodes of the computing device cluster based on the M first mapping relationships, satisfying the first and second conditions, specifically includes: obtaining multiple third execution times and the memory usage corresponding to the multiple third execution times according to the M first mapping relationships; wherein each third execution time indicates the execution time of a stage of the model task, and the layers allocated within the stage corresponding to each third execution time are different, and / or the ReC strategies configured for the layers are different; selecting the multiple second execution times from the multiple third execution times based on the first and second conditions; and obtaining the deployment method based on the layers allocated within the stages corresponding to the multiple second execution times and the ReC strategies configured for the layers.

[0012] In this embodiment, the method can utilize multiple first mapping relationships to determine multiple execution times and memory usages of each layer in the neural network model when configuring different ReC strategies in different stages. Then, it can filter according to the first and second conditions to obtain the ReC strategies and stages corresponding to each layer that meet the first and second conditions. This deployment method allows the stages allocated to the computing nodes to fully utilize computing and memory resources during the training process, thereby improving the resource utilization of the computing device cluster running the training system.

[0013] In one implementation, obtaining the M first mapping relationships corresponding to the N layers in the neural network model specifically includes: obtaining the M first mapping relationships corresponding to the N layers based on historical data; the historical data includes the execution time of different layers in the neural network model with different ReC strategies configured in different stages of the model task.

[0014] In this implementation, the method can obtain a first mapping relationship by the types of ReC policies configured in each layer of the neural network model when it performs model tasks in the past and the stage in which they are located. This makes the subsequent generation and deployment methods more consistent with the historical working methods of the neural network model, thereby making fuller use of the computing resources of each computing node in the computing device cluster.

[0015] In one implementation, obtaining the M first mapping relationships corresponding to the N layers in the neural network model specifically includes: profiling the model task to obtain the log of the model task; and obtaining the M first mapping relationships corresponding to the N layers based on the log of the model task.

[0016] In this implementation, the method can analyze the model task to obtain a first mapping relationship, making the subsequent generation and deployment methods more consistent with the working mode of the neural network model in executing the model task, thereby making fuller use of the computing resources of each computing node in the computing device cluster.

[0017] In one implementation, the deployment method further includes the condition that the memory usage of each stage is less than or equal to the memory resources that the computing node corresponding to each stage can provide.

[0018] In this implementation, the deployment method generated by this method requires that the memory usage of each stage should be less than or equal to the memory resources that the computing node corresponding to that stage can provide, so as to avoid the computing node being unable to provide enough memory for the stage, which would cause the neural network model to be unable to execute the model task normally.

[0019] In one implementation, the deployment method further includes the condition that the deployment method of the model task satisfies the pipelined parallel (PP) strategy.

[0020] The pipelined parallel (PP) strategy refers to the following: the stage of the next layer of each layer is the stage of the current layer, or the stage above the current layer's stage, or the next stage of the current layer's stage, or the parallel stage of the current layer's stage; and the stage of the previous layer of each layer is the stage of the current layer, or the next stage of the current layer's stage, or the previous stage of the current layer's stage, or the parallel stage of the current layer's stage.

[0021] In this embodiment, the deployment method generated by this method requires that layers with adjacent execution sequences should be assigned to the same stage or stages with adjacent execution sequences, so as to avoid assigning layers with adjacent execution sequences to stages with large execution sequences, which would cause the neural network model to take too long to execute the model task.

[0022] In one implementation, deploying the model task according to the deployment method specifically includes: instructing the N layers to execute the model task based on the ReC policy configured for each layer in the deployment method and the stage in which it is located.

[0023] In one implementation, obtaining multiple third execution times based on the M first mapping relationships specifically includes: acquiring the computational amount of the N layers configuring different types of ReC strategies in different stages; obtaining multiple execution computational amounts and multiple execution time relationships for the N layers based on the computational amounts of the N layers configuring different types of ReC strategies in different stages and the M first mapping relationships; wherein the execution computational amount refers to the relationship between the execution time and computational amount of a layer configuring one type of ReC strategy in a stage; the execution time relationship refers to the relationship between multiple execution times corresponding to a layer configuring different types of ReC strategies in a stage; and obtaining the multiple third execution times based on the multiple execution computational amounts and multiple execution time relationships of the N layers.

[0024] In this implementation, the method optimizes the computational performance of the neural network model by leveraging the computational load of each layer, thereby eliminating the impact of computational / communication overlap between layers and its effect on execution time. This method can configure the most suitable ReC strategy for each layer based on the execution time relationships of different ReC strategies configured for each layer.

[0025] In one implementation, based on the M first mapping relationships, a deployment method for the model task satisfying the first and second conditions on multiple computing nodes of a computing device cluster is obtained. Specifically, this includes: obtaining the deployment method satisfying the second condition based on the memory usage of different types of ReC strategies configured for each layer in different stages, the memory usage of activation values ​​of different types of ReC strategies configured for each layer, the additional memory amount of different types of ReC strategies configured for the first stage, the additional memory amount of different types of ReC strategies configured for the second stage, and the memory resources that the computing nodes corresponding to each stage can provide; the first stage refers to the first stage executed among the multiple stages; the second stage refers to the last stage executed among the multiple stages. The activation value refers to the output value of a neuron in the neural network model after receiving input and processing it through an activation function. The additional memory amount refers to the additional memory generated by additional operations relative to intermediate stages. The additional memory amount of the first stage refers to the additional memory consumption generated by the neural network model during additional operations in the first stage (e.g., an embedding layer or other primary layers) relative to intermediate stages. Here, the additional operations can be data preprocessing, feature extraction and transformation, weight initialization, activation value passing, etc. The additional memory required for the first stage refers to the extra memory consumption incurred by the neural network model during the additional operations performed in the last stage (e.g., query layer, output layer, etc.) relative to the intermediate stages. These additional operations can include storing and passing activation values, calculating loss, calculating gradients and propagating directions, and processing output results.

[0026] In this implementation, the method obtains the memory amount of activation values ​​for each layer when configuring different types of ReC policies, which quantifies the impact of the memory amount of activation values ​​for each layer when configuring different types of ReC policies on each layer in the segmented neural network model. Furthermore, by obtaining the additional memory amount of the first stage and the additional memory amount of the last stage, the method can eliminate the impact of the additional memory amount of the first and last stages on each layer in the segmented neural network model.

[0027] Secondly, this application provides a model task deployment apparatus, comprising: a first processing unit, configured to obtain M first mapping relationships corresponding to N layers in a neural network model; wherein each of the N layers applies one or more recomputation ReC strategies, the first mapping relationship is a mapping relationship between memory amount and first execution time, the first execution time is the time for the layer in a stage to run using the memory amount based on one of the ReC strategies, the stage is a part of the model task, the model task is to train the neural network model or to perform inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers in the neural network model is greater than or equal to N; a second processing unit, configured to obtain M first mapping relationships corresponding to N layers in a neural network model; wherein each of the N layers applies one or more recomputation ReC strategies, the first mapping relationship is a mapping relationship between memory amount and first execution time, the first execution time is the time for the layer in a stage to run using the memory amount based on one of the ReC strategies, the first execution time is the time for the layer in a stage to run using the memory amount ... A first mapping relationship is obtained to determine the deployment method of the model task on multiple computing nodes of the computing device cluster, satisfying the first and second conditions. The deployment method is used to indicate the computing node corresponding to each stage of the model task, and the ReC strategy corresponding to each layer of the neural network model in each stage. Each computing node includes one or more computing devices. The first condition is the matching between multiple second execution times, which are the execution times of the multiple stages of the model task under the deployment method. The second condition is the matching between the memory usage of each stage and the memory resources that the computing node corresponding to each stage can provide. A third processing unit is used to deploy the model task according to the deployment method.

[0028] In one implementation, the second processing unit is specifically configured to obtain multiple third execution times and the memory usage corresponding to the multiple third execution times based on the M first mapping relationships; wherein each third execution time indicates the execution time of a stage of the model task, and the layers allocated within the stage corresponding to each third execution time are different, and / or the ReC strategies configured for the layers are different; based on the first condition and the second condition, the multiple second execution times are selected from the multiple third execution times; and based on the layers allocated within the stage corresponding to the multiple second execution times and the ReC strategies configured for the layers, the deployment method is obtained.

[0029] In one implementation, the first processing unit is specifically used to obtain the M first mapping relationships corresponding to the N layers based on historical data; the historical data includes the execution time of different ReC strategies configured in different layers of the neural network model in different stages of the model task.

[0030] In one implementation, the first processing unit is specifically used to analyze the model task to obtain the log of the model task; and based on the log of the model task, to obtain the M first mapping relationships corresponding to the N layers.

[0031] In one implementation, the deployment method further includes the condition that the memory usage of each stage is less than or equal to the memory resources that the computing node corresponding to each stage can provide.

[0032] In one implementation, the deployment method further includes the condition that the deployment method of the model task satisfies the pipelined parallel (PP) strategy.

[0033] In one implementation, the third processing unit is specifically used to instruct the N layers to execute the model task based on the ReC policy configured for each layer in the deployment method and the stage in which it is located.

[0034] In one embodiment, the second processing unit is specifically used to obtain the computational amount of the N layers configuring different types of ReC strategies in different stages; based on the computational amount of the N layers configuring different types of ReC strategies in different stages and the M first mapping relationships, to obtain multiple execution computational amounts and multiple execution time relationships of the N layers; the execution computational amount refers to the relationship between the execution time and computational amount of a layer configuring a ReC strategy in a stage; the execution time relationship refers to the relationship between multiple execution times corresponding to a layer configuring different types of ReC strategies in a stage; based on the multiple execution computational amounts and multiple execution time relationships of the N layers, to obtain the multiple third execution times.

[0035] In one implementation, the second processing unit is specifically used to obtain the deployment method that satisfies the second condition based on the memory usage of different types of ReC strategies configured in different stages for each layer, the memory usage of the activation values ​​of different types of ReC strategies configured in each layer, the additional memory amount of different types of ReC strategies configured in the first stage, the additional memory amount of different types of ReC strategies configured in the second stage, and the memory resources that the computing node corresponding to each stage can provide; the first stage refers to the first stage executed among the multiple stages; the second stage refers to the last stage executed among the multiple stages.

[0036] Thirdly, this application provides a computer system comprising a processing unit and a storage unit; the processing unit of the computer system is configured to execute instructions stored therein, such that the computer system performs the various possible implementations of the first aspect.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computer system, perform the various possible implementations of the first aspect.

[0038] Fifthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a computer system, cause the computer system to implement various possible implementations of the first aspect.

[0039] In a sixth aspect, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the various possible implementations of the first aspect.

[0040] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the various possible implementations of the first aspect.

[0041] Eighthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement various possible implementations of the first aspect. Attached Figure Description

[0042] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0043] Figures 1(a)-1(c) This is a schematic diagram illustrating a scenario where a training system, provided in related technologies, trains a neural network model.

[0044] Figure 2 This is a schematic diagram of the structure of a training system provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram illustrating a user's use of the training system as provided in the embodiments of this application;

[0046] Figure 4 This is a flowchart illustrating a method for deploying a model task provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of a model task deployment device provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of another computing device cluster architecture provided in the embodiments of this application. Detailed Implementation

[0051] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0052] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0053] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0056] Before introducing the technical solution protected by this application, several technical terms involved in the technical solution protected by this application will be explained in advance, namely:

[0057] (1) Neural Network Model

[0058] A neural network model is a computational model inspired by the structure of the human brain. It processes information by mimicking the connections and interactions of neurons in the brain. In a neural network model, layers are the basic units that constitute the model, forming a hierarchical structure for data processing. Each layer can consist of multiple neurons responsible for receiving input, performing calculations, and passing the results to the next layer. Furthermore, each layer can perform a specific computational task. Layers in a neural network model can include one or more of the following: input layer, hidden layer, output layer, fully connected layer, convolutional layer, pooling layer, self-attention layer, and normalization layer. For example, a neural network model can be, but is not limited to, a deep neural network (DNN) model, a convolutional neural network (CNN) model, or a transformer model.

[0059] (2) Operator

[0060] Operators are functions that perform specific mathematical operations, such as matrix multiplication, addition, and activation functions. In neural network models, operators are used to process data and implement the functions of layers. The functions of layers in a neural network model can be achieved through a series of operators. For example, a fully connected layer can be implemented using matrix multiplication (the dot product of the weight matrix and the input vector) and addition (adding bias). By combining different operators, layers with different functions can be designed to adapt to different task requirements. In neural network models, layers are higher-level abstractions that define how data flows and transforms within the network. Operators are the specific mathematical operations that implement these transformations.

[0061] (3)PP

[0062] Parallel processing (PP) is a highly efficient method for parallel computing that breaks down complex tasks into multiple interdependent stages. These stages can execute in parallel, passing the results to the next stage upon completion. This pipelined approach allows data to flow between stages without waiting for the previous stage to finish, significantly improving computational efficiency.

[0063] During neural network model training, Prototype Processing (PP) technology can be used to segment the neural network model into multiple parts. Each part includes at least one layer. Each part can also serve as a stage, resulting in multiple stages. The layers of each part (i.e., each stage) are deployed on different computing nodes. Each computing node can be responsible for one stage. When deploying the corresponding layer on a computing node, the relevant parameters of the layer to be deployed can be configured on the corresponding computing node to complete the deployment of the layer on that computing node.

[0064] In the process of segmentation using PP technology, the stage of the next layer of each layer is either the stage of the current layer, the stage above the current layer, the next stage of the current layer, or a parallel stage of the current layer. Similarly, the stage of the previous layer of each layer is either the stage of the current layer, the next stage of the current layer, the stage above the current layer, or a parallel stage of the current layer.

[0065] (4)ReC

[0066] ReC is a computational optimization technique that reduces memory consumption and performance by re-executing parts of the computation. In ReC, some computation results can be temporarily discarded as needed and recalculated in later calculations, instead of being stored in memory. The main advantage of ReC is its reduced memory consumption, especially in memory-constrained scenarios where it can save significant amounts of memory. When certain computation results are only used temporarily in subsequent calculations and do not need to be retained, ReC helps the system avoid storing these results for extended periods, thereby improving memory utilization.

[0067] Recalculation strategies can be categorized into three types: non-repetitive computation strategies, fully repetitive computation strategies, and selective repetitive computation strategies. Non-repetitive computation strategies aim to avoid recalculating parts that have already been calculated. Fully repetitive computation strategies execute all computation steps from scratch each time a calculation is needed, regardless of whether similar results have been calculated previously. Selective repetitive computation strategies fall between non-repetitive and fully repetitive computation, optimizing performance by intelligently identifying and selecting parts that need to be recalculated.

[0068] Each layer can use one ReC strategy to perform computation. In this case, all operators corresponding to the layer can use the same ReC strategy to perform computation. Each layer can also use multiple ReC strategies to perform computation simultaneously. In this case, some operators corresponding to the layer can use one ReC strategy to perform part of the computation, while other operators can use a different ReC strategy to perform other parts of the computation.

[0069] When training neural network models using ReC (Recursive Conversion) technology, it's possible to avoid caching or storing intermediate computation results, such as layer outputs or activation values. When these intermediate results are needed again during model training, they are not read from the cache; instead, the necessary computation steps are re-executed to generate them. ReC significantly reduces memory usage because it reduces storage requirements, but it can increase computation time because the same calculations need to be performed repeatedly.

[0070] Next, the technical solution provided in this application will be introduced.

[0071] Generally, when training a neural network model, both PP (Programming Point) and ReC (Recursive Conversion) techniques can be used in combination. As shown in Figure 1(a), taking a neural network model with 14 layers as an example, the topological order of the neural network model is as follows: the outputs of layer 1 and 2 are the inputs of layer 3; the outputs of layer 3 and 4 are the inputs of layer 5; the output of layer 5 is the input of layer 6; the output of layer 6 is the input of layer 7 and layer 8; the output of layer 7 is the input of layer 8; the output of layer 8 is the input of layer 9 and layer 10; the output of layer 9 is the input of layer 11 and layer 12; the output of layer 10 is the input of layer 11 and layer 12; the output of layer 11 is the input of layer 13; the output of layer 12 is the input of layer 13; and the output of layer 13 is the input of layer 14. When training the neural network model, each layer can be trained according to the topological order.

[0072] As shown in Figure 1(b), taking a scenario with 4 computational nodes as an example, the 14 layers of the neural network model are divided into 4 stages using the PP (Programmable Pixel) technique. Stage 1 consists of 5 layers, stage 2 consists of 3 layers, stage 3 consists of 4 layers, and stage 4 consists of 2 layers. Then, based on the layer segmentation results, the training data and relevant parameters of the network neural model are further segmented. The training data and relevant parameters of each layer within a stage are grouped together as a micro-batch to obtain the micro-batch data corresponding to the 4 stages.

[0073] As shown in Figure 1(c), when the four stages are deployed to the corresponding computing nodes, the group data of stage 1 can be transferred to computing node 1, the group data of stage 2 can be transferred to computing node 2, the group data of stage 3 can be transferred to computing node 3, and the group data of stage 4 can be transferred to computing node 4, so as to deploy stages 1, 2, 3 and 4 to computing nodes 1, 2, 3 and 4 respectively.

[0074] Taking the case where compute node 1 receives the grouped data from stage 1 as an example, upon receiving the grouped data from stage 1, compute node 1 can locally construct layers 1, 2, 3, 4, and 5 based on the relevant parameters of the layers in the grouped data. Compute node 1 can then allocate the training data for training layer 1 to layer 1, the training data for training layer 2 to layer 2, the training data for training layer 3 to layer 3, and the training data for training layer 4 to layer 4, based on the training data in the grouped data.

[0075] Before training the five layers, computation node 1 can configure a ReC policy for each layer. When training the five layers, according to the topological order of the neural network model, layers 1 and 2 can be trained in parallel, and the results are then fed into layer 3 as its input. Layer 3 can be trained using its assigned training data, the results from layers 1 and 2, and the results are then fed into layer 5 as its input. Layer 4 can be trained in parallel with layer 3, and the results are then fed into layer 5 as its input. Layer 5 can be trained using its assigned training data, the results from layers 1 and 2, and the results are then fed into the next computation node, layer 2, as the input to layer 6.

[0076] During training, the five layers in compute node 1 can discard some computation results according to their configured ReC strategy to reduce memory consumption in compute node 1.

[0077] Typically, different types of layers use varying amounts of computation and memory for training. Therefore, it's difficult for all layers within a stage created using PP (Programming Processing) technology to fully utilize the computational and memory resources of the compute nodes deployed within that stage. Furthermore, the computational and memory usage varies when the same layer is trained using different ReC (Resource Counting) strategies. If each layer is configured with a different ReC strategy, it becomes even more difficult for all layers within a stage created using PP technology to fully utilize the computational and memory resources of the compute nodes deployed within that stage.

[0078] In view of this, this application provides a training system for a neural network model. This training system can determine the relationship between the amount of memory and the execution time in different stages of different layers of the neural network model corresponding to the model task, based on the configuration of different types of ReC strategies. It can also determine a deployment method that matches the execution time of each stage and the memory usage of each stage with the memory resources that the computing nodes deployed in that stage can provide. This allows the model task to be deployed according to the deployment method, enabling the stages allocated to the computing nodes to make full use of computing and memory resources during the training process, thereby improving the resource utilization of the computing device cluster running the training system.

[0079] Figure 2 This is a schematic diagram of the structure of a training system provided in an embodiment of this application. Figure 2 As shown, the training system 200 can be a model task deployment device 210 and a training device 220.

[0080] Training systems 200 are typically deployed on a cluster of computing devices for training and inference. A computing device cluster can include multiple computing devices. This cluster can be divided into multiple computing nodes, allowing multiple nodes to perform model training or inference. Each computing node includes one or more computing devices. Each computing node can be responsible for one stage.

[0081] The model task deployment device 210 and the training device 220 in the training system 200 can be executed on any one or more computing nodes in the computing device cluster, or on any one or more computing devices in the computing device cluster. The model task deployment device 210 and the training device 220 can be deployed on the same computing node or computing device, or they can be deployed on different computing nodes or computing devices.

[0082] The model task deployment device 210 is used to determine the deployment method of the model task on multiple computing nodes based on historical data and the device parameters of each computing node running the neural network model. The model task deployment device 210 can be divided into a data acquisition unit 211, a computational analysis unit 212, a memory analysis unit 213, and a strategy computation unit 214 according to the task being executed.

[0083] The data acquisition unit 211 is used to acquire historical data and device parameters of multiple computing nodes. Typically, a neural network model may deploy its multiple layers on different computing nodes, which are then used for training or inference. When the computing device cluster trains or infers the neural network model, it can use a performance analyzer to obtain basic data such as the stage corresponding to each layer of the neural network model, the memory usage of each layer, the type of ReC policy configured for each layer, the execution time of each layer, and operator information for each layer. Operator information may include the identifier of each operator, the execution time of each operator, the layer in which each operator is located, and the computational cost of each operator. The computing device cluster can store the acquired basic data in a common storage area for collection by the data acquisition unit 211.

[0084] After acquiring basic data, the data acquisition unit 211 can calculate higher-level data, referred to as historical data, based on the relationships between the various data points within the basic data. Optionally, the data acquisition unit 211 can select basic data related to N layers participating in the execution of the model task within the neural network model, based on the model task, to calculate historical data for those N layers, thereby reducing computational load. The number of layers in the neural network model is greater than or equal to N, and N is an integer greater than 1.

[0085] In one embodiment, the data acquisition unit 211 can obtain the execution time of each layer in the neural network model in different stages based on the stage corresponding to the computing node where each layer is located, the type of ReC policy configured for each layer, and the execution time of each layer during each training or inference of the neural network model (for easy distinction, the execution time at this time is called the "first execution time").

[0086] In another embodiment, the data acquisition unit 211 can obtain the memory usage of each layer in the neural network model in different stages based on the stage corresponding to the computing node where each layer is located, the type of ReC policy configured for each layer, and the memory usage of each layer during each training or inference of the neural network model.

[0087] In another embodiment, the data acquisition unit 211 obtains M first mapping relationships for each layer based on the first execution time of different ReC strategies configured for each layer in different stages and the memory usage of different ReC strategies configured for each layer in different stages. Each first mapping relationship is a mapping relationship between the memory usage and the first execution time of a layer with one ReC strategy configured in the same stage. M is an integer greater than or equal to N.

[0088] In another embodiment, the data acquisition unit 211 can obtain the operator information of each layer in the neural network model in different stages by using the stage corresponding to the computing node where each layer is located during each training or inference of the neural network model, the type of ReC policy configured for each layer, and the operator information of each layer.

[0089] Optionally, historical data can be generated during pre-training by the training system 200. For example, the training system 200 can pre-train the neural network model to obtain historical data of the neural network model before obtaining the deployment method of the model task on multiple computing nodes using the model task deployment device 210.

[0090] Alternatively, the training system 200 analyzes the model task to obtain a log of the model task. This log typically includes the execution time of a layer or the operators within a layer. Based on this log, the data acquisition unit 211 can obtain data such as the first execution time of different ReC strategies configured for each layer in different stages, and the M first mapping relationships of each layer.

[0091] The device parameters include the amount of memory and the computational load of each compute node. Memory refers to the amount of memory available for training or inference of the neural network model within a compute node, or the amount of free memory. Computational load refers to the amount of computation available for training or inference of the neural network model from all processors within a compute node, or the amount of free computation.

[0092] The computational analysis unit 212 is used to obtain, based on historical data, the computational cost of configuring different types of ReC strategies for each layer of the neural network model in different stages, and the execution time relationship of configuring different types of ReC strategies for each layer in different stages.

[0093] The computational cost varies depending on the type of layer in a neural network model. Computational cost refers to the amount of computational resources required to complete one forward or backward propagation. Computational cost can include any of the following: floating-point operations (FLOPs), the number of parameters, and other computational hardware resource consumption. FLOPs are a metric for measuring the computational cost of a neural network model, representing the number of floating-point operations required to move from input to output. FLOPs can be estimated by counting the number of multiplication and addition operations in each layer and can be used as an important indicator of the model's computational complexity and efficiency. The number of parameters can also be used as a metric for computational cost. A larger number of parameters typically requires more computational operations to update and adjust these parameters, thus increasing the computational cost.

[0094] For example, the computational analysis unit 212 can obtain the runtime interval (microstep) of a micro-batch data group (micro-batch) of a single pipeline from the performance analysis data. Performance analysis data refers to data obtained by analyzing and optimizing neural network models using a profiler tool in a deep learning framework. The profiler tool can provide various information about resource usage and performance bottlenecks during neural network model training, helping users better understand the behavior of the neural network model and optimize it. The profiler tool can provide functions such as monitoring computational resource usage, performance analysis, memory allocation analysis, and visualization of the training process. In this embodiment, the training system 200 can use the profiler tool to analyze the performance bottlenecks of the neural network model, identify which parts of the code run slowly or consume more resources, and detect memory allocation during model training to obtain performance analysis data.

[0095] The pipeline's runtime interval includes a start time (ms.begin) and an end time (ms.end). In one embodiment, since a single operator appears only once in a training iteration, the interval between two occurrences of the same operator constitutes the pipeline's runtime interval. After obtaining operator information for different ReC strategies configured for each layer in different stages from historical data, the computational analysis unit 212 can iterate through all operators, recording the identifier (e.g., MATMUL-0124) of the first matrix multiplication (MATMUL) operator and the time of its appearance, designated as the start time T1. When the MATMUL operator with identifier MATMUL-0124 appears again, the computational analysis unit 212 records the time of this appearance, designated as the end time T2. The computational analysis unit 212 can obtain a pipeline's runtime interval, (T1, T2), based on the times of two consecutive occurrences of the MATMUL operator with identifier MATMUL-0124.

[0096] After obtaining the runtime interval of a pipeline, the computational analysis unit 212 can retrieve the operator information of all operators within that pipeline runtime interval from the operator information of different stages configured with different types of ReC strategies at each layer. In one embodiment, after obtaining the execution time of all operators, the computational analysis unit 212 filters out operators that satisfy the condition that the start time of the operator is greater than the start time of the pipeline runtime interval and the end time of the operator is less than the end time of the pipeline runtime interval, thereby obtaining all operators within the pipeline runtime interval.

[0097] The computational analysis unit 212 can calculate the computational cost of configuring different types of ReC strategies for each layer of the neural network model at different stages, based on the operators within the pipeline's runtime interval, the layer in which each operator resides, and the computational cost of each operator. The computational cost of each layer in the neural network model is a crucial step in optimizing its performance. For example, after filtering out the operators within the pipeline's runtime interval, the computational analysis unit 212 can determine all operators in each layer based on their layer location. The computational analysis unit 212 can then obtain the computational cost of configuring different types of ReC strategies for each layer of the neural network model at different stages, based on all operators in each layer and the computational cost of each operator.

[0098] Different types of layers in a neural network model have different computational and communication characteristics, so it is necessary to consider the degree of computational / communication overlap between layers and the impact of operations on execution time. In this embodiment, the computational analysis unit 212 can calculate the computational load of each layer in the neural network model configured with different types of ReC strategies in different stages based on the computational load of each layer configured with different types of ReC strategies in different stages, and the first execution time of each layer configured with different types of ReC strategies in different stages obtained from historical data. The model task deployment device 210 uses the computational load of each layer configured with different types of ReC strategies in different stages to optimize the computational performance of the neural network model, which can eliminate the impact of the degree of computational / communication overlap between layers and the impact on execution time.

[0099] In one embodiment, after obtaining the first execution time of each layer with different types of ReC strategies in different stages from historical data, the computational analysis unit 212 can calculate the ratio of the first execution time to the computational cost of each layer with different types of ReC strategies in different stages based on the first execution time and computational cost of each layer with different types of ReC strategies in different stages. The computational analysis unit 212 can select the most representative layer from multiple layers of the neural network model according to predefined rules. These rules include the most frequent repetitions and the highest computational cost. The computational analysis unit 212 can obtain the computational cost of each layer with different types of ReC strategies in different stages based on the ratio of the first execution time to the computational cost of each layer with different types of ReC strategies in different stages to the ratio of the first execution time to the computational cost of the most representative layer.

[0100] The types of ReC strategies executed by the training system differ under different AI frameworks, and the execution time for different types of ReC strategies also varies. In this embodiment, the computational analysis unit 212 can obtain the first execution time of each layer from historical data, thus obtaining the execution time relationship of different types of ReC strategies configured for each layer in the neural network model. During the optimization of the computational performance of the neural network model, the model task deployment device 210 can, based on the execution time relationship of different types of ReC strategies configured for each layer, configure the most suitable type of ReC strategy for each layer by combining the execution time relationship.

[0101] In one embodiment, the computational analysis unit 212 can obtain the types of ReC policies supported by the AI ​​framework currently in which the training system operates. The types of ReC policies may include non-repeated computation, fully repeated computation, and selectively repeated computation. The computational analysis unit 212 can obtain the first execution time of each layer configured with different types of ReC policies in different stages from historical data, and calculate the ratio between the first execution times of each layer in a stage configured with different types of ReC policies, thereby obtaining the execution time relationship of each layer in the neural network model configured with different types of ReC policies in different stages.

[0102] The memory analysis unit 213 is used to obtain the memory usage of each layer of the neural network model in different stages with different types of ReC strategies configured, based on historical data. Here, memory usage refers to a measure of the peak memory occupied during one forward and backward propagation cycle.

[0103] For example, after the memory analysis unit 213 obtains the memory usage of each layer configured with different types of ReC strategies in different stages from historical data, it can sum the memory usage of layers in the same stage that are configured with the same ReC strategy to obtain the memory usage of all layers in each stage configured with one type of ReC strategy, so as to obtain the memory usage of each stage configured with different types of ReC strategies.

[0104] After obtaining the memory usage of different types of ReC strategies configured in each stage, the memory analysis unit 213 can analyze the number of layers with the same type of ReC strategy configured in each stage and the types of ReC strategies. Based on the memory usage of different types of ReC strategies configured in each stage (excluding the first and last stages), the number of layers with different types of ReC strategies configured in each stage (excluding the first and last stages), and the types of ReC strategies, the memory analysis unit 213 can calculate the memory usage of each layer in the neural network model configured with different types of ReC strategies in different stages, as well as the memory usage of the activation values ​​of different types of ReC strategies configured in each layer. Activation values ​​refer to the output values ​​of neurons in the neural network model after receiving input and processing it through an activation function.

[0105] In one embodiment, the memory analysis unit 213 can arbitrarily select three stages from multiple stages, excluding the first and last stages, and input the stage ID (stageid) of the selected three stages, the number of layers when the three stages are configured with the same type of ReC strategy, the memory usage of the three stages with different types of ReC strategies, and the multiple types of ReC strategies into formula (1) to calculate the memory usage M of each layer of the neural network model configured with different types of ReC strategies in different stages. P And the memory usage of the activation values ​​of different types of ReC policies (r) configured in each layer. Formula (1) is specifically as follows:

[0106]

[0107] Where L represents the set of layers l of a stage, R represents the set of ReC policy types r, S is the number of stages, i represents the stage index, and M... P Memory usage of the presentation layer This indicates the memory usage of the activated value of the ReC policy for configuration type r. This represents the layer in the i-th stage that is configured with a ReC strategy of type r.

[0108] In optimizing the memory performance of the neural network model, the model task deployment device 210 can configure different types of ReC strategies for each layer of the neural network model in different stages to optimize the memory usage of each layer in the segmented neural network model, and configure the layers corresponding to the most suitable amount of memory for the computing nodes running the stages. After obtaining the memory usage of the activation values ​​of different types of ReC strategies configured for each layer, the model task deployment device 210 can quantify the impact of the memory usage of the activation values ​​of different types of ReC strategies configured for each layer on the segmentation of each layer in the neural network model.

[0109] The memory analysis unit 213 can calculate the memory usage of different types of ReC strategies in the first stage based on the number of layers with the same type of ReC strategy in the first stage of the neural network model, the types of ReC strategies configured in the layers of the first stage, the memory usage of each layer with different types of ReC strategies in the first stage, and the memory usage of the activation values ​​of each layer with different types of ReC strategies.

[0110] The memory analysis unit 213 can calculate the memory usage of different types of ReC strategies in the last stage based on the number of layers with the same type of ReC strategy configured in the last stage of the neural network model, the types of ReC strategies configured in the layers of the last stage, the memory usage of each layer with different types of ReC strategies in the last stage, and the memory usage of the activation values ​​of each layer with different types of ReC strategies.

[0111] The memory analysis unit 213 can obtain the memory usage of the first stage with different ReC strategies and the memory usage of the last stage with different ReC strategies from the memory usage of each stage, and refer to them as the measured memory usage of the first stage with different ReC strategies and the measured memory usage of the last stage with different ReC strategies. The memory analysis unit 213 can subtract the calculated memory usage of the first stage with different ReC strategies from the measured memory usage of the first stage with different ReC strategies to calculate the additional memory amount of the first stage with different ReC strategies. Similarly, the memory analysis unit 213 can subtract the calculated memory usage of the last stage with different ReC strategies from the measured memory usage of the last stage with different ReC strategies to calculate the additional memory amount of the last stage with different ReC strategies.

[0112] Additional memory refers to the extra memory required relative to the additional operations performed in the intermediate stages. The additional memory for the first stage refers to the extra memory consumption incurred by the neural network model during additional operations in the first stage (e.g., the embedding layer or other primary layers) relative to the intermediate stages. These additional operations can include data preprocessing, feature extraction and transformation, weight initialization, activation value propagation, etc. The additional memory for the last stage refers to the extra memory consumption incurred by the neural network model during additional operations in the last stage (e.g., the query layer, output layer, etc.) relative to the intermediate stages. These additional operations can include storing and propagating activation values, loss calculation, gradient calculation and direction propagation, and processing the output results, etc.

[0113] In one embodiment, the memory analysis unit 213 can input the memory usage of different ReC strategies configured in the first stage, the number of ReC strategies configured in the first stage, the memory usage of different ReC strategies configured in the last stage, and the number of ReC strategies configured in the last stage into formula (2) to calculate the additional memory E0 of different ReC strategies configured in the first stage and the additional memory E of different ReC strategies configured in the last stage of the neural network model. S-1 Formula (2) is specifically as follows:

[0114]

[0115] Among them, M i This represents the memory amount of the i-th stage, where i = 0 represents the first stage, i = -1 represents the last stage, and E0 represents the additional memory amount required to configure different types of ReC strategies in the first stage. S-1 This indicates the amount of additional memory required to configure different ReC strategies in the last stage.

[0116] The model task deployment device 210, by obtaining the additional memory amount for configuring different types of ReC strategies in the first stage and the additional memory amount for configuring different types of ReC strategies in the last stage, can eliminate the impact of the additional memory amount for configuring different types of ReC strategies in the first and last stages on the layers in the segmented neural network model.

[0117] The strategy calculation unit 214 is used to obtain the deployment method of the model task in the computing device cluster based on the execution computation amount of different types of ReC strategies configured for each layer in different stages, the execution time relationship of different types of ReC strategies configured for each layer in different stages, and the memory usage of different types of ReC strategies configured for each layer in different stages, as output by the calculation analysis unit 212, and the memory usage of different types of ReC strategies configured for each layer in different stages, as output by the memory analysis unit 213.

[0118] For example, the policy calculation unit 214 can obtain the number of layers in the neural network model from the definition file of the neural network model, and then determine the layer quantity constraint for all stages based on the number of stages and the number of layers in the neural network model. The quantity constraint stipulates that the number of layers allocated to all stages is equal to the number of layers in the neural network model.

[0119] The strategy computation unit 214 can obtain the execution order of each layer in the neural network model from the definition file of the neural network model, and then determine the sequence constraints of each layer in each stage based on the execution order of each layer in the neural network model. The sequence constraints satisfy the pipelined parallel (PP) strategy. The pipelined parallel (PP) strategy means that the stage containing the next layer of each layer is either the stage containing the current layer, or the stage preceding the current layer, or the next stage of the current layer, or a parallel stage of the current layer; and the stage containing the previous layer of each layer is either the stage containing the current layer, or the next stage of the current layer, or the previous stage of the current layer, or a parallel stage of the current layer.

[0120] The strategy calculation unit 214 can obtain the memory constraints for each stage based on the memory usage of different ReC strategies configured for each layer in different stages, the memory usage of the activation values ​​of different ReC strategies configured for each layer, the additional memory required for different ReC strategies configured for the first stage, the additional memory required for different ReC strategies configured for the last stage, and the memory of the computing nodes corresponding to each stage. The memory constraints satisfy the condition that the sum of the memory of all layers in each stage is less than or equal to the memory of the computing nodes deployed in that stage.

[0121] In one embodiment, the policy calculation unit 214 can input the memory usage of different types of ReC policies configured for each layer in different stages, the memory usage of the activation values ​​of different types of ReC policies configured for each layer, the additional memory of different types of ReC policies configured for the first stage, and the additional memory of different types of ReC policies configured for the last stage, calculated by the memory analysis unit 213, into formula (3) to calculate the memory constraint M of each stage. i≤c Formula (3) is specifically as follows:

[0122]

[0123] Among them, M i Let represent the memory amount of the i-th stage, where i = 0 represents the first stage and i = -1 represents the last stage. C represents the machine memory limit, L represents the set of layers l of a stage, R represents the set of ReC policy types r, S is the number of stages, and i represents the stage index. P The amount of memory required for the presentation layer. This indicates the memory usage of the activated value of the ReC policy with configuration type r. E0 represents the layer in the i-th stage that configures a ReC strategy of type r, and E0 represents the additional memory required to configure different types of ReC strategies in the first stage. S-1 This indicates the amount of additional memory required to configure different ReC strategies in the last stage.

[0124] The strategy calculation unit 214 can calculate multiple execution times (for clarity, these execution times are referred to as "second execution times") for each stage with different layers and each layer with different ReC strategies, based on the number of stages, the types of ReC strategies configured in each layer, the first mapping relationship of each layer, the computational cost of different ReC strategies configured in different stages for each layer, and the execution time relationship of different ReC strategies configured in each layer. The strategy calculation unit 214 can also calculate multiple memory usages for each stage with different layers and each layer with different ReC strategies, based on the number of stages, the types of ReC strategies configured in each layer, the first mapping relationship of each layer, and the memory usage of different ReC strategies configured in different stages for each layer.

[0125] The strategy calculation unit 214 can use a solution algorithm to solve for multiple second execution times and multiple memory usages of each stage under the conditions of quantity constraints, sequence constraints and memory constraints, so as to obtain the types of ReC strategies configured in each layer of each stage that satisfy the first and second conditions, thereby obtaining the deployment method.

[0126] The first condition can refer to the matching of the second execution times of multiple stages. Matching the second execution times of multiple stages means that the differences between the second execution times are as small as possible, that is, the execution times of each stage are as uniform as possible. However, due to practical reasons, such as differences in the resource configuration of the computing cluster, different ReC strategies, and different neural network structures involved in the deployed tasks, the matching result may be that the execution times of the multiple stages are the same under the deployment method, or the difference between them is less than a set time threshold, or the difference between them tends to be 0 or minimal.

[0127] The second condition can refer to matching the amount of memory used by multiple stages with the memory resources provided by the computing nodes corresponding to each stage. Matching the amount of memory used by multiple stages with the memory resources provided by the computing nodes corresponding to each stage means minimizing the difference in memory usage among the various stages under different deployment methods, i.e., making the memory usage of each stage as uniform as possible. However, due to practical factors such as differences in computing cluster resource configuration, different ReC strategies, and different neural network structures involved in the deployed tasks, the matching result can be expressed as the proportion of memory used by each stage to the memory resources provided by its corresponding computing nodes being the same, or the difference being less than the first set memory threshold, or the difference tending towards 0 or minimum. The matching result can also be expressed as the total amount of memory used by each layer of each stage being the same, or the difference being less than the second set memory threshold, or the difference tending towards 0 or minimum.

[0128] The solution algorithm can be integer linear programming (ILP). Solving ILP problems typically requires specific algorithms to find integer solutions. ILP problems are an extension of linear programming problems where variables need to take integer values, not real values. For example, the policy computation unit 214 can use the shortest total execution time of the neural network model as the objective function, and utilize ILP linear programming to solve for quantity constraints, order constraints, memory constraints, multiple second execution times for each stage, and multiple memory usages for each stage. It calculates each stage where the second execution times tend to be the same, and where the ratio of memory usage for each stage to the memory of the computing nodes deployed in that stage tends to be the same, or where the memory usage for each stage tends to be the same, thereby obtaining the deployment method.

[0129] The training device 220 is used to deploy model tasks to computing nodes corresponding to multiple stages according to the deployment method. The training device 220 can divide the tasks to be executed into PP splitting units 221 and ReC policy configuration units 222.

[0130] After obtaining the deployment method, the PP segmentation unit 221 can use the PP segmentation strategy to segment the layers of the neural network model, and allocate each layer to the corresponding computing nodes of the stage according to the deployment method. The PP segmentation unit 221 can segment the model tasks and related parameters of the network neural model according to the segmented stages, group the model tasks and related parameters of each layer in a stage together as a group of data, and allocate them to the corresponding computing nodes of the corresponding stage.

[0131] After obtaining the deployment method, the ReC policy configuration unit 222 can use the ReC configuration policy to send the ReC policy configuration scheme to each computing node according to the ReC policy configuration scheme configured in the deployment method, so that the computing node can configure different types of ReC policies for each layer built locally.

[0132] When a compute node trains or infers its local layers based on a model task, it can discard some computation results according to its configured ReC policy to reduce memory consumption. Multiple compute nodes in a compute cluster can fully utilize their local computing and memory resources according to the layers allocated after partitioning and the type of ReC policy, thereby improving the resource utilization of the compute cluster running the training system.

[0133] It should be understood that the functional modules and devices involved in the training system 200 can all be implemented through software or hardware, depending on the actual situation, and are not limited here. Furthermore, the functional modules and devices involved in the training system 200 can be arranged individually or integrated, and are not limited here.

[0134] The above is a description of the training system 200 provided in this application embodiment. It is understood that the training system 200 can be configured on a cloud computing platform, for example, deployed on at least one instance such as a virtual machine or container, so that the cloud computing platform can provide model training services. Of course, the training system 200 can also be configured on nodes other than the cloud computing platform, for example, deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud computing platform can provide pages related to public cloud services for users to remotely access public cloud services. In this embodiment, the user can pre-purchase the model training services provided by the training system 200 on the cloud computing platform. For ease of understanding, the interaction between the user and the cloud computing platform is described below.

[0135] like Figure 3 As shown, the interaction between the user and the cloud computing platform mainly includes: the user logs into the cloud computing platform 300 through a client webpage, selects and purchases cloud services (i.e., model training services) related to the training system 200 within the cloud computing platform 300. After purchase, the user can generate the training system 200 on the cloud computing platform 300 based on the functions provided by the model training service. The cloud computing platform 300 is primarily used to manage the infrastructure for running the model training service. For example, the infrastructure for the model training service may include multiple data centers located in different regions, each data center including multiple servers. The data centers can provide basic resources for the model training service, such as computing resources and storage resources. Therefore, when purchasing and using the model training service, the user mainly pays for the resources used. When using the model training service, the user can input their model training requirements through the configuration interface, application programming interface (API), or user interaction interface provided by the cloud computing platform 300. The cloud computing platform 300 can then generate a model training system matching the user's (or other software / hardware, etc.) input requirements.

[0136] Alternatively, some modules in the training system 200 can be configured on the cloud side and others on the edge side, thereby achieving model training through edge-cloud collaboration. Furthermore, the training system 200 can also be entirely configured on the edge side, depending on the specific circumstances; no limitation is made here.

[0137] The above is an introduction to the training system provided in the embodiments of this application. Next, based on the above content, the deployment method of the model task provided in the embodiments of this application will be introduced.

[0138] For example, Figure 4 This diagram illustrates a flowchart of a model task deployment method provided in an embodiment of this application. It can be understood that this method can be executed by the aforementioned training system 200, and the specific implementation process is as follows:

[0139] Step S401: Obtain the M first mapping relationships corresponding to the N layers in the neural network model.

[0140] Typically, a neural network model can deploy its multiple layers on different computing nodes, which then perform training or inference. When a computing device cluster trains or infers a neural network model, a performance analyzer can be used to obtain basic data such as the stage corresponding to each layer of the computing node, the memory usage of each layer, the types of ReC policies configured for each layer, the execution time of each layer, and operator information for each layer. Operator information can include the identifier of each operator, the execution time of each operator, the layer in which each operator resides, and the computational cost of each operator. The computing device cluster can store the basic data obtained each time in a shared storage area for future collection.

[0141] After the training system 200 collects basic data, it can calculate higher-level data based on the relationships between the various data in the basic data, which will be referred to as historical data.

[0142] In one embodiment, the training system 200 can obtain the execution time of different ReC policies configured for each layer in different stages of the neural network model based on the stage corresponding to the computing node where each layer is located during each training or inference of the neural network model, the type of ReC policy configured for each layer, and the execution time of each layer (for easy distinction, the execution time at this time is called the "first execution time").

[0143] In another embodiment, the training system 200 can obtain the memory usage of each layer in the neural network model in different stages based on the stage corresponding to the computing node where each layer is located, the type of ReC policy configured for each layer, and the memory usage of each layer during each training or inference of the neural network model.

[0144] In another embodiment, the training system 200 obtains multiple first mapping relationships for each layer based on the first execution time of different ReC strategies configured for each layer in different stages and the memory usage of different ReC strategies configured for each layer in different stages. Each first mapping relationship is a mapping relationship between the memory usage and the first execution time of a layer configured with a certain ReC strategy in the same stage.

[0145] In another embodiment, the training system 200 can obtain the operator information of each layer in the neural network model configured with different types of ReC policies in different stages based on the stage corresponding to the computing node where each layer is located during each training or inference of the neural network model, the type of ReC policy configured for each layer, and the operator information of each layer.

[0146] Historical data can be generated during pre-training by the training system. For example, the training system can pre-train the neural network model to obtain historical data of the neural network model before obtaining the deployment method of the model task on multiple computing nodes using the model task deployment device 210.

[0147] Alternatively, the training system 200 analyzes the model task to obtain a log of the model task. This log typically includes the execution time of a layer or the operators within a layer. Based on this log, the training system 200 can obtain data such as the first execution time of each layer with different types of ReC policies configured in different stages, and the M first mapping relationships of each layer.

[0148] The device parameters include the amount of memory and the computational load of each compute node. Memory refers to the amount of memory available for training or inference of the neural network model within a compute node, or the amount of free memory. Computational load refers to the amount of computation available for training or inference of the neural network model from all processors within a compute node, or the amount of free computation.

[0149] The above step S401 can be performed by the data acquisition unit 211 in the training system 200.

[0150] Step S402: Based on M first mapping relationships, obtain the deployment method of the model task satisfying the first and second conditions on multiple computing nodes of the computing device cluster.

[0151] The training system 200 can obtain the runtime intervals of micro-batch data groups for a single pipeline from performance analysis data. The pipeline runtime interval includes the start and end times. After obtaining the runtime interval of a pipeline, the training system 200 can retrieve the operator information of all operators within that pipeline runtime interval from the operator information of different ReC policies configured for each layer in different stages. Based on the operators within the pipeline runtime interval, the layer in which each operator is located, and the computational cost of each operator, the training system 200 can calculate the computational cost of configuring different ReC policies for each layer of the neural network model in different stages.

[0152] The training system 200 can calculate the computational cost of configuring different ReC policies for each layer of the neural network model in different stages based on the computational cost of configuring different ReC policies for each layer in different stages, and the first execution time of configuring different ReC policies for each layer in different stages obtained from historical data. The model task deployment device 210 optimizes the computational performance of the neural network model by utilizing the computational cost of configuring different ReC policies for each layer in different stages, which can eliminate the influence of the degree of computation / communication superposition between layers and the impact on execution time.

[0153] The types of ReC policies executed by the training system differ under different AI frameworks, and the execution time for different types of ReC policies also varies. In this embodiment, the training system 200 can obtain the first execution time of each layer from historical data, thus obtaining the execution time relationship of different types of ReC policies configured for each layer in the neural network model. During the optimization of the computational performance of the neural network model, the model task deployment device 210 can, based on the execution time relationship of different types of ReC policies configured for each layer, configure the most suitable type of ReC policy for each layer by combining the execution time relationship.

[0154] After obtaining the memory usage of each layer configured with different types of ReC strategies in different stages from historical data, the training system 200 can sum the memory usage of layers in the same stage that are configured with the same ReC strategy to obtain the memory usage of all layers in each stage configured with one type of ReC strategy, so as to obtain the memory usage of each stage configured with different types of ReC strategies.

[0155] After obtaining the memory usage of different types of ReC policies configured in each stage, the training system 200 can analyze the number of layers with the same type of ReC policy configured in each stage and the types of ReC policies. Based on the memory usage of different types of ReC policies configured in each stage except the first and last stages, the number of layers with different types of ReC policies configured in each stage except the first and last stages, and the types of ReC policies, the training system 200 can calculate the memory usage of each layer in the neural network model with different types of ReC policies configured in different stages, as well as the memory usage of the activation values ​​of each layer with different types of ReC policies configured.

[0156] The training system 200 can calculate the memory usage of different types of ReC strategies in the first stage based on the number of layers with the same type of ReC strategy configured in the first stage of the neural network model, the types of ReC strategies configured in the layers of the first stage, the memory usage of each layer with different types of ReC strategies in the first stage, and the memory usage of the activation values ​​of each layer with different types of ReC strategies.

[0157] The training system 200 can calculate the memory usage of different types of ReC strategies in the last stage based on the number of layers with the same type of ReC strategy configured in the last stage of the neural network model, the types of ReC strategies configured in the layers of the last stage, the memory usage of each layer with different types of ReC strategies in the last stage, and the memory usage of the activation values ​​of each layer with different types of ReC strategies.

[0158] The training system 200 can obtain the memory usage of the first stage with different ReC strategies and the memory usage of the last stage with different ReC strategies from the memory usage of each stage, and refer to them as the measured memory usage of the first stage with different ReC strategies and the measured memory usage of the last stage with different ReC strategies. The training system 200 can subtract the calculated memory usage of the first stage with different ReC strategies from the measured memory usage of the first stage with different ReC strategies to calculate the additional memory amount of the first stage with different ReC strategies. Similarly, the training system 200 can subtract the calculated memory usage of the last stage with different ReC strategies from the measured memory usage of the last stage with different ReC strategies to calculate the additional memory amount of the last stage with different ReC strategies.

[0159] The training system 200 can obtain the number of layers in the neural network model from the model definition file, and then determine the layer number constraint for each stage based on the number of stages and the number of layers in the neural network model. The layer number constraint stipulates that the number of layers allocated to all stages is equal to the number of layers in the neural network model.

[0160] The training system 200 can obtain the execution order of each layer in the neural network model from the definition file of the neural network model, and then determine the sequence constraints of each layer in each stage based on the execution order of each layer in the neural network model. The sequence constraints satisfy the pipelined parallel (PP) strategy. The pipelined parallel (PP) strategy means that the stage containing the next layer of each layer is either the stage containing the current layer, or the stage preceding the current layer, or the next stage of the current layer, or a parallel stage of the current layer; and the stage containing the previous layer of each layer is either the stage containing the current layer, or the next stage of the current layer, or the stage preceding the current layer, or a parallel stage of the current layer.

[0161] The training system 200 can determine the memory constraints for each stage by considering the memory usage of different ReC strategies configured for each layer in different stages, the memory usage of activation values ​​for different ReC strategies configured for each layer, the additional memory required for different ReC strategies configured for the first stage, the additional memory required for different ReC strategies configured for the last stage, and the memory size of the compute nodes corresponding to each stage. The memory constraints satisfy the condition that the sum of the memory sizes of all layers in each stage is less than or equal to the memory size of the compute nodes deployed in that stage.

[0162] The training system 200 can calculate multiple execution times (for clarity, these execution times are referred to as "second execution times") for each stage with different layers and each layer with different ReC strategies, based on the number of stages, the types of ReC strategies configured in each layer, the first mapping relationship of each layer, the computational cost of different ReC strategies configured in different stages for each layer, and the execution time relationship of different ReC strategies configured in each layer. The training system 200 can also calculate multiple memory usages for each stage with different layers and each layer with different ReC strategies, based on the number of stages, the types of ReC strategies configured in each layer, the first mapping relationship of each layer, and the memory usage of different ReC strategies configured in different stages for each layer.

[0163] The training system 200 can use a solution algorithm to solve for multiple second execution times and multiple memory usages of each stage under the conditions of quantity constraints, order constraints, and memory constraints. This allows it to obtain the types of ReC strategies configured in each layer of each stage that satisfy the first and second conditions, thereby obtaining the deployment method.

[0164] The first condition can refer to the second execution time of each stage being the same or approaching the same. The first condition can also refer to the difference between the second execution times of each stage being less than a set time threshold. Finally, the first condition can refer to the second execution times of each stage or the difference between them approaching 0 or being minimal.

[0165] The second condition can refer to the ratio of memory usage of each stage to the memory of the compute nodes deployed in that stage being the same or approaching the same. Alternatively, the second condition can refer to the difference between the ratio of memory usage of each stage to the memory of the compute nodes deployed in that stage being less than the first set memory threshold. Finally, the second condition can refer to the difference between the ratio of memory usage of each stage to the memory of the compute nodes deployed in that stage approaching 0 or being minimal.

[0166] The second condition can refer to the memory usage of each stage being the same or tending to be the same. The second condition can refer to the difference in memory usage between each stage being less than a second set memory threshold. The second condition can refer to the difference in memory usage between each stage tending to 0 or being minimal. Step S402 can be jointly executed by the computational analysis unit 212, memory analysis unit 213, and policy calculation unit 214 in the aforementioned training system.

[0167] Step S403: Deploy the model task according to the deployment method.

[0168] After obtaining the deployment method, the training system 200 can use the PP splitting strategy to split the layers of the neural network model, and allocate each layer to the corresponding computing nodes of the stage according to the deployment method. The training system 200 can further split the model tasks and related parameters of the network neural model according to the split stages, grouping the model tasks and related parameters of each layer in a stage together as a group of data, and then allocating them to the corresponding computing nodes of the corresponding stage.

[0169] After the training system 200 obtains the deployment method, it can use the ReC configuration strategy to send the ReC strategy configuration scheme to each computing node according to the ReC strategy configuration scheme of the deployment method, so that the computing node can configure different types of ReC strategies for each layer built locally.

[0170] When a compute node trains or infers its local layers based on a model task, it can discard some computation results according to its configured ReC policy to reduce memory consumption. Multiple compute nodes in a compute cluster can fully utilize their local computing and memory resources according to the layers allocated after partitioning and the type of ReC policy, thereby improving the resource utilization of the compute cluster running the training system.

[0171] The above step S403 can be jointly executed by the PP splitting unit 221 and the ReC policy configuration unit 222 in the above training system.

[0172] Figure 5 This is a schematic diagram of the structure of a deployment device for a model task provided in an embodiment of this application. Figure 5 As shown, the deployment device 500 for this model task can be divided into a first processing unit 510, a second processing unit 520, and a third processing unit 530 according to its execution function. The specific implementation process of the deployment device 500 for this model task is as follows:

[0173] The first processing unit 510 is used to obtain M first mapping relationships corresponding to N layers in the neural network model. Each of the N layers applies one or more Recomputation ReC (Reload Control) strategies. The first mapping relationship is a mapping relationship between memory usage and first execution time. The first execution time is the time taken to run a layer within a stage based on a ReC strategy and the amount of memory used. A stage is a part of the model task, which is either training the neural network model or performing inference using the neural network model. N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers in the neural network model is greater than or equal to N. The second processing unit 520 is used to obtain, based on the M first mapping relationships, a deployment method for the model task satisfying the first and second conditions on multiple computing nodes in the computing device cluster. The deployment method indicates the computing node corresponding to each stage of the model task, and the ReC strategy corresponding to each layer of the neural network model in each stage. Each computing node includes one or more computing devices. The first condition is matching between multiple second execution times, which are the execution times of the multiple stages of the model task under the respective deployment method. The second condition is matching the memory usage of each stage with the memory resources that the computing node corresponding to each stage can provide. The third processing unit 530 is used to deploy model tasks according to the deployment method.

[0174] In one implementation, the second processing unit 520 is specifically used to obtain multiple third execution times and the memory usage corresponding to the multiple third execution times based on M first mapping relationships. Each third execution time indicates the execution time of a stage of the model task, and the layers allocated within each stage corresponding to each third execution time are different, and / or the ReC strategies configured for the layers are different. The second processing unit 520 is specifically used to select multiple second execution times from the multiple third execution times based on first and second conditions. The second processing unit 520 is specifically used to obtain a deployment method based on the layers allocated within the stages corresponding to the multiple second execution times and the ReC strategies configured for the layers.

[0175] In one implementation, the first processing unit 510 is specifically used to obtain M first mapping relationships corresponding to N layers based on historical data. The historical data includes the execution time of different ReC strategies configured for different layers in the neural network model at different stages of the model task.

[0176] In one implementation, the first processing unit 510 is specifically used to analyze the model task and obtain the log of the model task. Specifically, the first processing unit 510 is used to obtain M first mapping relationships corresponding to N layers based on the log of the model task.

[0177] In one implementation, the deployment method also satisfies the condition that the memory usage of each stage is less than or equal to the memory resources that the computing node corresponding to each stage can provide.

[0178] In one implementation, the deployment method also includes satisfying the following conditions: the deployment method of the model task satisfies the pipelined parallel (PP) strategy.

[0179] In one implementation, the third processing unit 530 is specifically used to instruct N layers to perform model tasks based on the ReC policy configured for each layer in the deployment mode and the stage in which it is located.

[0180] In one implementation, the second processing unit 520 is specifically used to obtain the computational amount of N layers configuring different types of ReC strategies in different stages. Specifically, the second processing unit 520 is used to obtain multiple execution computational amounts and multiple execution time relationships for the N layers based on the computational amounts of the N layers configuring different types of ReC strategies in different stages and M first mapping relationships. Execution computational amount refers to the relationship between the execution time and computational amount of a layer configuring one type of ReC strategy in a stage. Execution time relationship refers to the relationship between multiple execution times corresponding to different types of ReC strategies configured in a layer in a stage. Specifically, the second processing unit 520 is used to obtain multiple third execution times based on the multiple execution computational amounts and multiple execution time relationships of the N layers.

[0181] In one implementation, the second processing unit 520 is specifically used to obtain a deployment method that satisfies the second condition based on the memory usage of different types of ReC strategies configured for each layer in different stages, the memory usage of the activation values ​​of different types of ReC strategies configured for each layer, the additional memory amount of different types of ReC strategies configured for the first stage, the additional memory amount of different types of ReC strategies configured for the second stage, and the memory resources that the computing nodes corresponding to each stage can provide. The first stage refers to the first stage executed among multiple stages. The second stage refers to the last stage executed among multiple stages.

[0182] The following simulation experiment results demonstrate the technical effectiveness of the technical solution protected in this application.

[0183] Assume the neural network model is the LLAMA2-70B neural network model. The deep learning framework is the MindSpore framework. The training system is deployed on a cluster of 60 computing devices, each with 60GB of memory. The neural network model is divided into 8 stages. The micro-batch number is 128.

[0184] The training system analyzed the performance analysis data and determined the computational cost of each layer to be {'LlamaEmbedding':10840,'LlamaRMSNorm':3470,'Linear':614,'DecodeLayer':15633}. The system also calculated the execution time of each layer to be {'LlamaEmbedding':26814,'LlamaRMSNorm':5178,'Linear':703,'DecodeLayer':26788}. Based on the computational cost and execution time of each layer, the system calculated the total computational cost to be {'LlamaEmbedding':15657,'LlamaRMSNorm':3028,'Linear':409,'DecodeLayer':15633}. The training system obtained the two recomputation strategies supported by the MindSpore framework for the LLAMA2-70B neural network model: non-recomputation and full recomputation. The training system calculated the execution time ratio of these two recomputation strategies to be {1:1.4}.

[0185] The training system obtains the memory consumption, number of recomputed strategies, and number of layers for each pipeline stage. As shown in Table 1, pipeline stage 0 has a memory footprint of 56975MB, 8 recomputed strategies, and a total of 10 layers. Pipeline stage 1 has a memory footprint of 34933MB, 9 recomputed strategies, and a total of 10 layers. Pipeline stage 2 has a memory footprint of 45114MB, 8 recomputed strategies, and a total of 10 layers. Pipeline stage 3 has a memory footprint of 40457MB, 8 recomputed strategies, and a total of 10 layers. Pipeline stage 4 has a memory footprint of 35801MB, 8 recomputed strategies, and a total of 10 layers. Pipeline stage 5 has a memory footprint of 31145MB, 8 recomputed strategies, and a total of 10 layers. The pipeline stage (stage 6) has a memory footprint of 26488MB, with 8 recompute strategies and a total of 10 recompute layers. The pipeline stage (stage 7) has a memory footprint of 24293MB, with 8 recompute strategies and a total of 10 recompute layers.

[0186] Table 1 shows the memory usage, number of recompile strategies, and number of layers for each pipeline stage.

[0187] assembly line stage Memory (MB) Recomputation strategy number of floors stage0 56975 8-layer recalculation 10th floor stage1 34933 9-layer recalculation 10th floor stage2 45114 8-layer recalculation 10th floor stage3 40457 8-layer recalculation 10th floor stage4 35801 8-layer recalculation 10th floor stage5 31145 8-layer recalculation 10th floor stage6 26488 8-layer recalculation 10th floor stage7 24293 8-layer recalculation 10th floor

[0188] The training system selects pipeline stage (stage 1), pipeline stage (stage 2), and pipeline stage (stage 3) from the eight pipeline stages and inputs them into formula (1), and sets ∑l∈L|l i Substituting S = 10 and S = 8 into formula (1), we obtain the following system of equations:

[0189]

[0190] The training system calculates the memory requirement M for each layer. P =1717, the memory usage of activation values ​​in the non-recompute strategy. Memory usage of the recalculation strategy for activation values

[0191] The training system inputs the first pipeline stage (stage 0) and the last pipeline stage (stage 7) into formula (2), and sets ∑l∈L|l i =10, S=8, Substituting into formula (2), we obtain the following system of equations:

[0192] Stage0, when i = 0:

[0193] Stage 7, when i = 7:

[0194] The training system calculates the memory usage E0 = 2547 for each layer of the first pipeline stage, and the memory usage E for each layer of the last stage. S7 =1713.

[0195] The training system determined the sequential constraints as LlamaEmbedding→DecodeLayer→LlamaRMSNorm→Linear.

[0196] The training system determined that the total number of DecodeLayers in the pipeline stage is equal to 80 due to the quantity constraint: x_0 + x_1 + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + y_0 + y_1 + y_2 + y_3 + y_4 + y_5 + y_6 + y_7 = 80. Here, the numbers represent the pipeline stage, x represents the number of layers configured for recomputation, and y represents the number of layers configured for non-recomputation.

[0197] The training system calculates the memory constraints for each pipeline stage as follows:

[0198] 2045*x_0+19005*y_0<=57453

[0199] 2004*x_1+16844*y_1<=60000

[0200] 1963*x_2+14683*y_2<=60000

[0201] 1922*x_3+12522*y_3<=60000

[0202] 1881*x_4 + 10361*y_4 <= 60000

[0203] 1840*x_5 + 8200*y_5 <= 60000

[0204] 1799*x_6+6039*y_6<=60000

[0205] 1758*x_7+3878*y_7<=58237

[0206] In the first equation, 2045 = M P Right now That is, 41,19005 = M P Right now That is, 2111,57453 = machine memory limit 60000 - additional memory for the first stage E S0 That is, 2547. The values ​​of the middle equations are derived similarly to the first equation. In the last equation, 58237 = the additional memory E required to meet the machine's memory limit of 60000-stage. S7 That is, 1713.

[0207] The training system determined the following relationships between the total execution time of the LLAMA2-70B neural network model and the relationships between the various layers and stages within the LLAMA2-70B neural network model, as well as the ReC policies configured for each layer:

[0208]

[0209] Where T is the total execution time of the pipeline, and t_(S-1) is the execution time of the last stage of the pipeline. The sum of the execution times of all pipeline stages for training a single microbatch of data in the pipeline, max(t) i ) represents the maximum execution time for training a single micro-batch across all stages, t i The execution time of a single micro-batch is trained for stage with index i. For a layer at index i with a recomputation strategy of r configured, dur exec r represents the computational cost of layer execution, and r.speed represents the execution time relationship of the recomputation strategy r.

[0210] According to formula (4), the training system can divide the total pipeline execution time into three parts: pipeline_total_time (T in the formula) consists of three parts, namely lastFP_BP (t in the formula). S-1 ), max_stage_time (max(t) in the formula) i )) and var_sum_FPi_Bpi (in the formula) This process generates the relationships between layers and recomputation strategies on the pipeline stage and their respective parts, thereby generating a relationship with the total pipeline execution time.

[0211] lastFP_BP-21886.2*x_7-15633*y_7>=3437

[0212] Among them, 21886.2 is obtained by multiplying the execution time of 'DecodeLayer' (15633) obtained by the training system with the execution time of the complete recomputation (1.4), and 3437 is the sum of the execution computation of 'LlamaRMSNorm' and 'Linear'.

[0213] sum_FPi_BPi+var_sum_FPi_BPi-21886.2*x_0-21886.2*x_1

[0214] 21886.2*x_2-21886.2*x_3-21886.2*x_4-21886.2*x_5-21886.2*x_6

[0215] 21886.2*x_7-15633*y_0-15633*y_1-15633*y_2-15633*y_3-15633*y_4-15633*y_5-15633*y_6-15633*y_7>=30251

[0216] Where 30251 is the sum of the execution computation of 'LlamaRMSNorm', 'Linear', and 'LlamaEmbedding' obtained by the training system.

[0217] max_stage_time-21886.2*x_0-15633*y_0>=26814

[0218] max_stage_time-21886.2*x_1-15633*y_1>=0

[0219] max_stage_time-21886.2*x_2-15633*y_2>=0

[0220] max_stage_time-21886.2*x_3-15633*y_3>=0

[0221] max_stage_time-21886.2*x_4-15633*y_4>=0

[0222] max_stage_time-21886.2*x_5-15633*y_5>=0

[0223] max_stage_time-21886.2*x_6-15633*y_6>=0

[0224] max_stage_time-21886.2*x_7-15633*y_7>=3437

[0225] lastFP_BP-126*max_stage_time+pipeline_total_time-var_sum_FPi_Bpi

[0226] Where 126 is the result of micro_batch_num, which is 128-2.

[0227] Based on the above constraints and time considerations, the training system outputs the optimal PP splitting strategy and ReC configuration strategy, which are as follows:

[0228] stage0: x = 6, y = 2

[0229] stage1: x = 7, y = 2

[0230] stage2: x = 7, y = 3

[0231] stage3: x = 7, y = 3

[0232] stage4: x = 6, y = 4

[0233] stage5: x = 4, y = 6

[0234] stage6: x = 2, y = 9

[0235] stage7: x = 0, y = 12

[0236] The training system can train the LLAMA2-70B neural network model based on the computation graph generated by the PP splitting strategy and ReC configuration strategy described above. The training system uses both the deployment device 210 with and without model tasks for training, and the memory consumption of each pipeline stage is shown in Table 2.

[0237] Table 2 shows the memory consumption of each pipeline stage obtained by the training system using the model task deployment device 210 and not using the model task deployment device 210.

[0238]

[0239] The training system was trained using both the deployment device 210 with and without model tasks, and the total execution time is shown in Table 3. Compared to the scheme without using the deployment device 210 with model tasks, the training system trained using the deployment device 210 with model tasks achieved a 120% speedup in total execution time.

[0240] Table 3 shows the total execution time of the training system using the deployment device 210 with and without model tasks.

[0241]

[0242] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 6 As shown, the computing device 700 includes a bus 710, a processor 720, a memory 730, and a communication interface 740. The processor 720, memory 730, and communication interface 740 communicate with each other via the bus 710. The computing device 700 can be a server, computer, laptop, server rack, etc. It should be understood that this application does not limit the number of processors and memory in the computing device 700.

[0243] The bus 710 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 710 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 710 may include a path for transmitting information between various components of the computing device 700 (e.g., processor 720, memory 730, communication interface 740).

[0244] The processor 720 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0245] The memory 730 may include volatile memory, such as random access memory (RAM). The memory 730 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0246] The memory 730 stores executable program code, which the processor 720 executes to implement the functions of the aforementioned modules, such as the first processing unit 510, the second processing unit 520, and the third processing unit 530, thereby realizing the deployment method for the model task. In other words, the memory 730 stores instructions for executing the deployment method for the model task.

[0247] Alternatively, the memory 730 stores executable code, which the processor 720 executes to implement the functions of the aforementioned modules, thereby realizing the deployment method for the model task. That is, the memory 730 stores instructions for executing the deployment method for the model task.

[0248] The communication interface 740 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0249] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0250] like Figure 7 As shown, the computing device cluster includes at least one computing device 700. The memory 730 in one or more computing devices 700 in the computing device cluster may store the same instructions for a deployment method to execute model tasks.

[0251] In some possible implementations, the memory 730 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the deployment method of the model task. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the deployment method of the model task.

[0252] It should be noted that the memories 730 in different computing devices 700 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the first processing unit 510, the second processing unit 520, and the third processing unit 530. That is, the instructions stored in the memories 730 of different computing devices 700 can implement the functions of one or more modules of the first processing unit 510, the second processing unit 520, and the third processing unit 530.

[0253] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, two computing devices, computing device 700A and computing device 700B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 730 in computing device 700A stores instructions for performing the functions of some modules of the first processing unit 510, the second processing unit 520, and the third processing unit 530. Meanwhile, the memory 730 in computing device 700B stores instructions for performing the functions of another portion of the modules of the first processing unit 510, the second processing unit 520, and the third processing unit 530.

[0254] Figure 8 The connection method between the computing device clusters shown can be based on the fact that the deployment method of the model task provided in this application requires a large amount of data storage. Therefore, it is considered that the functions implemented by another part of the modules in the first processing unit 510, the second processing unit 520 and the third processing unit 530 are handed over to the computing device 700B for execution.

[0255] It should be understood that Figure 8 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0256] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 6 and Figure 8 The connection method of the computing device cluster is different in that the memory 730 of one or more computing devices 700 in the computing device cluster can store the same instructions for the deployment method of executing model tasks.

[0257] In some possible implementations, the memory 730 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the deployment method of the model task. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the deployment method of the model task.

[0258] It should be noted that the memory 730 in different computing devices 700 within the computing device cluster can store different instructions for executing some functions of the computing device 700. That is, the instructions stored in the memory 730 of different computing devices 700 can implement the functions of one or more modules of the first processing unit 510, the second processing unit 520, and the third processing unit 530 described above.

[0259] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform a deployment method for a model task.

[0260] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a deployment method for a model task.

[0261] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for deploying a model task, characterized in that, The method includes: M first mapping relationships are obtained corresponding to N layers in the neural network model; wherein, each of the N layers applies one or more recomputation ReC strategies, the first mapping relationship is a mapping relationship between memory amount and first execution time, the first execution time is the time of the layer running in a stage based on one of the ReC strategies and using the memory amount, the stage is a part of the model task, the model task is to train the neural network model or to use the neural network model to perform inference, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers of the neural network model is greater than or equal to N; Based on the M first mapping relationships, a deployment method for the model task on multiple computing nodes of the computing device cluster that satisfies the first and second conditions is obtained. The deployment method is used to indicate the computing node corresponding to each stage of the model task, and the ReC strategy corresponding to each layer of the neural network model in each stage. Each computing node includes one or more computing devices. The first condition is the matching between multiple second execution times, which are the execution times of the multiple stages of the model task under the deployment method. The second condition is the matching between the memory usage of each stage and the memory resources that the computing node corresponding to each stage can provide. Deploy the model task according to the deployment method described.

2. The method according to claim 1, characterized in that, The deployment method of the model task on multiple computing nodes of the computing device cluster, based on the M first mapping relationships and satisfying the first and second conditions, specifically includes: Based on the M first mapping relationships, multiple third execution times and the memory usage corresponding to the multiple third execution times are obtained; wherein, each third execution time indicates the execution time of a stage of the model task, and the layers allocated within the stage corresponding to each third execution time are different, and / or the ReC strategies configured for the layers are different; Based on the first condition and the second condition, the plurality of second execution times are selected from the plurality of third execution times; The deployment method is obtained based on the layers allocated within the stages corresponding to the multiple second execution times and the ReC strategy configured in those layers.

3. The method according to claim 1 or 2, characterized in that, The process of obtaining the M first mapping relationships corresponding to the N layers in the neural network model specifically includes: The M first mapping relationships corresponding to the N layers are obtained based on historical data; the historical data includes the execution time of different ReC strategies configured in different layers of the neural network model in different stages of the model task.

4. The method according to claim 1 or 2, characterized in that, The process of obtaining the M first mapping relationships corresponding to the N layers in the neural network model specifically includes: Analyze the model task to obtain its logs; Based on the logs of the model task, the M first mapping relationships corresponding to the N layers are obtained.

5. The method according to any one of claims 1-4, characterized in that, The deployment method also requires that the memory usage of each stage is less than or equal to the memory resources that the computing node corresponding to each stage can provide.

6. The method according to any one of claims 1-5, characterized in that, The deployment method also satisfies the following conditions: the deployment method of the model task satisfies the pipelined parallel (PP) strategy.

7. The method according to any one of claims 1-6, characterized in that, Deploying the model task according to the deployment method specifically includes: Based on the ReC policy configured for each layer in the deployment method and the stage it is in, the N layers are instructed to execute the model task.

8. The method according to any one of claims 2-7, characterized in that, The process of obtaining multiple third execution times based on the M first mapping relationships specifically includes: Obtain the computational cost of configuring different types of ReC strategies for the N layers in different stages; Based on the computational cost of configuring different types of ReC strategies in different stages across the N layers and the M first mapping relationships, multiple execution computational costs and multiple execution time relationships are obtained for the N layers. The execution computational cost refers to the relationship between the execution time and computational cost of configuring one ReC strategy in a stage for a layer. The execution time relationship refers to the relationship between multiple execution times corresponding to configuring different types of ReC strategies in a stage for a layer. Based on the multiple execution computation quantities and multiple execution time relationships of the N layers, the multiple third execution times are obtained.

9. The method according to any one of claims 1-8, characterized in that, Based on the M first mapping relationships, the deployment method of the model task satisfying the first and second conditions on multiple computing nodes of the computing device cluster is obtained, specifically including: Based on the memory usage of different ReC strategies configured in different stages for each layer, the memory usage of the activation values ​​of different ReC strategies configured in each layer, the additional memory amount of different ReC strategies configured in the first stage, the additional memory amount of different ReC strategies configured in the second stage, and the memory resources that the computing nodes corresponding to each stage can provide, the deployment method that satisfies the second condition is obtained; the first stage refers to the first stage executed among the multiple stages; the second stage refers to the last stage executed among the multiple stages.

10. A deployment device for a model task, characterized in that, include: The first processing unit is configured to obtain M first mapping relationships corresponding to N layers in the neural network model; wherein each of the N layers applies one or more recomputation ReC strategies, the first mapping relationship is a mapping relationship between memory amount and first execution time, the first execution time is the time to run the layer in a stage using the memory amount based on one of the ReC strategies, the stage is a part of the model task, the model task is to train the neural network model or to perform inference using the neural network model, N is an integer greater than 1, M is an integer greater than or equal to N, and the number of layers in the neural network model is greater than or equal to N; The second processing unit is configured to, based on the M first mapping relationships, obtain a deployment method for the model task on multiple computing nodes of the computing device cluster that satisfies the first and second conditions; the deployment method is used to indicate the computing node corresponding to each stage of the model task, and the ReC strategy corresponding to each layer of the neural network model in each stage, wherein each computing node includes one or more computing devices, the first condition is matching between multiple second execution times, wherein the multiple second execution times are the respective execution times of the multiple stages of the model task under the deployment method; the second condition is matching the memory usage of each stage with the memory resources that the computing node corresponding to each stage can provide; The third processing unit is used to deploy the model task according to the deployment method.

11. The apparatus according to claim 10, characterized in that, The second processing unit is specifically used to obtain multiple third execution times and the memory usage corresponding to the multiple third execution times according to the M first mapping relationships; wherein, each third execution time indicates the execution time of a stage of the model task, and the layers allocated within the stage corresponding to each third execution time are different, and / or the ReC strategies configured in the layers are different; Based on the first condition and the second condition, the plurality of second execution times are selected from the plurality of third execution times; The deployment method is obtained based on the layers allocated within the stages corresponding to the multiple second execution times and the ReC strategy configured in those layers.

12. The apparatus according to claim 10 or 11, characterized in that, The first processing unit is specifically used to obtain the M first mapping relationships corresponding to the N layers based on historical data; the historical data includes the execution time of different ReC strategies configured in different layers of the neural network model in different stages of the model task.

13. The apparatus according to claim 10 or 11, characterized in that, The first processing unit is specifically used to analyze the model task and obtain the log of the model task; Based on the logs of the model task, the M first mapping relationships corresponding to the N layers are obtained.

14. The apparatus according to any one of claims 10-13, characterized in that, The deployment method also requires that the memory usage of each stage is less than or equal to the memory resources that the computing node corresponding to each stage can provide.

15. The apparatus according to any one of claims 10-14, characterized in that, The deployment method also satisfies the following conditions: the deployment method of the model task satisfies the pipelined parallel (PP) strategy.

16. The apparatus according to any one of claims 10-15, characterized in that, The third processing unit is specifically used to instruct the N layers to execute the model task based on the ReC policy configured for each layer in the deployment method and the stage it is in.

17. The apparatus according to any one of claims 11-16, characterized in that, The second processing unit is specifically used to obtain the computational cost of configuring different types of ReC strategies in different stages of the N layers; Based on the computational cost of configuring different types of ReC strategies in different stages across the N layers and the M first mapping relationships, multiple execution computational costs and multiple execution time relationships are obtained for the N layers. The execution computational cost refers to the relationship between the execution time and computational cost of configuring one ReC strategy in a stage for a layer. The execution time relationship refers to the relationship between multiple execution times corresponding to configuring different types of ReC strategies in a stage for a layer. Based on the multiple execution computation quantities and multiple execution time relationships of the N layers, the multiple third execution times are obtained.

18. The apparatus according to any one of claims 10-17, characterized in that, The second processing unit is specifically used to obtain the deployment method that satisfies the second condition based on the memory usage of different types of ReC strategies configured for each layer in different stages, the memory usage of the activation values ​​of different types of ReC strategies configured for each layer, the additional memory amount of different types of ReC strategies configured for the first stage, the additional memory amount of different types of ReC strategies configured for the second stage, and the memory resources that the computing node corresponding to each stage can provide; the first stage refers to the first stage executed among the multiple stages; The second stage refers to the last stage that is executed among the plurality of stages.

19. A computer system, characterized in that, The computer system includes a processing unit and a storage unit; The processing unit of the computer system is used to execute instructions stored in the processing unit so that the computer system performs the method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computer system, cause the computer system to perform the method as described in any one of claims 1-9.

21. A computer program product containing instructions, characterized in that, The computer program product stores instructions that, when executed by a computer system, cause the computer system to perform the method described in any one of claims 1-9.

Images