Model processing method and system, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a model processing method and system, electronic equipment, a storage medium and a program product, and relates to the field of artificial intelligence. The method is applied to a distributed training system, the distributed training system comprises a management node and a plurality of training nodes, and the method comprises the following steps: the management node obtains training configuration information of a model and resource information of each training node in the plurality of training nodes, and generates a model deployment strategy according to the obtained information, the model deployment strategy is used for indicating a division mode adopted for fragmenting the model and a node distribution strategy corresponding to the model fragmentation obtained based on the division mode; and the management node obtains a plurality of model fragments corresponding to the model according to the model deployment strategy, and deploys each model fragment in the plurality of model fragments to the corresponding training node.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a model processing method, a distributed training system, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] A model refers to a neural network structure constructed using deep learning technology, has tens of thousands or even hundreds of millions of parameters, and can train a large amount of data, and exhibits strong language understanding, generation and reasoning capabilities in various tasks and fields. The more parameters a model has, the more processor (such as a graphics processing unit, GPU) memory it needs for training and inference. Currently, data centers capable of aggregating vast amounts of resources are extremely scarce, making large model training limited by computing power resources, and only limited computing power can be used to complete the training and inference of small and medium-sized models. Therefore, how to effectively use the distributed computing power resources in the network, especially how to use the processors distributed in different regions for distributed and efficient collaboration, to provide computing power support for large model training, has become a problem to be solved in the field of artificial intelligence.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The present disclosure provides a model processing method, a distributed training system, an electronic device, a computer readable storage medium, and a computer program product.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a model processing method is provided, applied to a distributed training system including a management node and a plurality of training nodes, the method is executed by the management node, and the method includes: obtaining training configuration information of a model and resource information of each training node in the plurality of training nodes; generating a model deployment strategy according to the training configuration information of the model and the resource information of each training node; the model deployment strategy is used to indicate a division manner adopted for model slicing, and a node allocation strategy corresponding to a model slice obtained based on the division manner; obtaining a plurality of model slices corresponding to the model according to the model deployment strategy, and deploying each model slice in the plurality of model slices to a corresponding training node.

[0007] In some embodiments of the present disclosure, the training configuration information of the model comprises a division strategy of training data and a hierarchical structure of the model, and the resource information of each training node comprises computing resource of each training node and communication delay information between each training node and other training nodes; wherein, the generating of the model deployment strategy according to the training configuration information of the model and the resource information of each training node comprises: generating the model deployment strategy according to the division strategy of the training data, the hierarchical structure of the model, the computing resource of each training node and the communication delay information between each training node and other training nodes; and the model deployment strategy comprises one or more pipeline parallel groups, one or more data parallel groups, and a corresponding relationship between model shards obtained based on the pipeline parallel groups and the data parallel groups and the plurality of training nodes.

[0008] In some embodiments of the present disclosure, the model deployment strategy is generated according to one or more of the following principles: minimizing the number of training nodes required for model deployment; adjusting the division strategy of the training data if the overall computing resource of the plurality of training nodes is scattered; minimizing the communication delay between different training nodes in each data parallel group in the one or more data parallel groups and the communication delay between different training nodes in each data parallel group being similar; and minimizing the communication delay between different data parallel groups in the one or more data parallel groups and the communication delay between different data parallel groups being similar.

[0009] In some embodiments of the present disclosure, the obtaining of the plurality of model shards corresponding to the model according to the model deployment strategy and the deployment of each model shard in the plurality of model shards to a corresponding training node comprise: replicating and fragmenting the model according to the one or more pipeline parallel groups and the one or more data parallel groups in the model deployment strategy to obtain the plurality of model shards; assigning each model shard in the plurality of model shards to a corresponding training node according to the corresponding relationship between the model shards and the plurality of training nodes in the model deployment strategy; and receiving a configuration completion message sent by the training node corresponding to each model shard to determine that the model deployment is completed.

[0010] In some embodiments of the present disclosure, the method further comprises: creating a training task corresponding to the model deployment strategy and generating an identifier of the training task.

[0011] In some embodiments of the present disclosure, the assigning each model shard in the plurality of model shards to a corresponding training node according to the correspondence between the model shards in the model deployment strategy and the plurality of training nodes comprises: assigning, according to the correspondence, an identifier of the training task, a hierarchical structure of the each model shard and parameter information of the each model shard to the training node corresponding to the each model shard, so that the training node corresponding to the each model shard performs resource configuration.

[0012] In some embodiments of the present disclosure, the method further comprises: establishing a node relationship between the training node corresponding to the each model shard and other training nodes; the node relationship is used to indicate adjacent nodes in a pipeline parallel group to which the training node corresponding to the each model shard belongs and other training nodes in a data parallel group to which the training node corresponding to the each model shard belongs; and assigning the node relationship between the training node corresponding to the each model shard and other training nodes to the training node corresponding to the each model shard, so that the training node corresponding to the each model shard performs node relationship configuration.

[0013] In some embodiments of the present disclosure, the training configuration information of the model comprises model training quality requirement information; and before the model deployment strategy is generated according to the training configuration information of the model and the resource information of the each training node, the method further comprises: filtering the plurality of training nodes according to the resource information of the each training node and the model training quality requirement information, and filtering out training nodes in the plurality of training nodes that do not meet the model training quality requirement information.

[0014] In some embodiments of the present disclosure, the method further comprises: starting the training node corresponding to the each model shard to execute the training task according to the identifier of the training task.

[0015] In some embodiments of the present disclosure, the method further comprises: recording, in each round of training of the training task, a training state of each batch of data in the current round of training.

[0016] In some embodiments of the present disclosure, the method further comprises: recording, in each round of training of the training task, a parameter update state of the each model shard; and in response to the each model shard completing parameter update, updating a training state of the model and determining to execute a next round of training.

[0017] According to a further aspect of the present disclosure, a model processing method is provided, applied to a distributed training system including a management node and a plurality of training nodes, the method being performed by a training node in the plurality of training nodes, and the method comprising: deploying a model shard allocated by the management node according to a model deployment strategy; wherein the model deployment strategy is generated according to training configuration information of a model and resource information of each training node in the plurality of training nodes, and the model deployment strategy is used to indicate a division manner for dividing the model and a node allocation strategy corresponding to the model shard obtained based on the division manner.

[0018] In some embodiments of the present disclosure, the deploying the model shard allocated by the management node according to the model deployment strategy comprises: obtaining an identification of a training task, a hierarchical structure of the model shard corresponding to the training node, and parameter information of the model shard corresponding to the training node; establishing a training task processing unit, and associating the training task processing unit with the identification of the training task; and performing resource configuration according to the hierarchical structure of the model shard corresponding to the training node and the parameter information of the model shard corresponding to the training node.

[0019] In some embodiments of the present disclosure, the method further comprises: obtaining a node relationship between the training node and other training nodes; the node relationship is used to indicate adjacent nodes in a pipeline parallel group to which the training node belongs and other training nodes in a data parallel group to which the training node belongs; and performing node relationship configuration according to the node relationship between the training node and the other training nodes.

[0020] In some embodiments of the present disclosure, the method further comprises: executing the training task by the training task processing unit according to the identification of the training task.

[0021] In some embodiments of the present disclosure, the executing the training task by the training task processing unit comprises: in each round of training of the training task, receiving, by the training task processing unit, a forward activation message sent by an upstream adjacent node in a pipeline parallel group to which the training node belongs; the forward activation message includes forward activation data output by the upstream adjacent node; performing calculation on the forward activation data output by the upstream adjacent node to obtain forward activation data output by the training node; and packaging the forward activation data output by the training node to obtain a new forward activation message, and sending the new forward activation message to a downstream adjacent node in the pipeline parallel group to which the training node belongs.

[0022] In some embodiments of the present disclosure, the executing the training task by the training task processing unit comprises: in each round of training of the training task, receiving, by the training task processing unit, a backward gradient message sent by a downstream adjacent node in a pipeline parallel group to which the training node belongs; the backward gradient message comprises backward gradient data output by the downstream adjacent node; performing calculation on the backward gradient data of the downstream adjacent node to obtain backward gradient data output by the training node; and packaging the backward gradient data output by the training node to obtain a new backward gradient message, and sending the new backward gradient message to an upstream adjacent node in the pipeline parallel group to which the training node belongs.

[0023] In some embodiments of the present disclosure, the method further comprises: in each round of training of the training task, sending, to other training nodes in a data parallel group to which the training node belongs, backward gradient data output by the training node, and receiving backward gradient data output by other training nodes in the data parallel group to which the training node belongs; in response to obtaining the backward gradient data of all batch data in the current round of training in the data parallel group to which the training node belongs, performing gradient data aggregation calculation to obtain aggregated gradient data, and updating parameter information of a model shard corresponding to the training node by using the aggregated gradient data.

[0024] In some embodiments of the present disclosure, the method further comprises: packaging the aggregated gradient data to obtain an aggregated gradient message; and sending the aggregated gradient message to other training nodes in the data parallel group to which the training node belongs.

[0025] In some embodiments of the present disclosure, the method further comprises: in response to receiving an aggregated gradient message sent by other training nodes in the data parallel group to which the training node belongs, determining not to perform gradient data aggregation calculation; the aggregated gradient message sent by the other training nodes comprises the aggregated gradient data; and updating parameter information of a model shard corresponding to the training node by using the aggregated gradient data. According to still another aspect of the present disclosure, a distributed training system is provided, which comprises a management node and a plurality of training nodes; wherein the management node is configured to: obtain training configuration information of a model and resource information of each training node in the plurality of training nodes; generate a model deployment strategy according to the training configuration information of the model and the resource information of each training node; the model deployment strategy is used to indicate a division manner adopted for model slicing, and a node allocation strategy corresponding to a model shard obtained based on the division manner; obtain a plurality of model shards corresponding to the model according to the model deployment strategy, and deploy each model shard in the plurality of model shards to a corresponding training node.

[0026] According to a further aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device configured to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the model processing method as described in the above embodiments.

[0027] According to a further aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon a computer program that, when executed by a processor, implements the model processing method as described in the above embodiments.

[0028] According to a further aspect of the present disclosure, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the model processing method as described in the above embodiments.

[0029] The model processing method provided by the embodiments of the present disclosure is applied to a distributed training system composed of a management node and a plurality of training nodes. The management node is responsible for obtaining training configuration information of a model and resource information of each training node, and generating a model deployment strategy according to the information. The model deployment strategy indicates how to shard the model and how to deploy the model shards to the corresponding training nodes. The management node obtains a plurality of model shards according to the model deployment strategy, and deploys each model to the corresponding training node. It can be seen that the model processing method can bring the dispersed training nodes in the network into the training process across geographical restrictions. The management node is responsible for collecting and integrating the resource information of the training nodes, providing sufficient computing power support for large model training. The management node dynamically adjusts the model deployment strategy according to the training configuration information of the model and the real-time resource information of each training node, so as to adapt to the model training requirements of different scales and complexities. When generating the model deployment strategy, the resource situation of each training node is considered to ensure that the model shards can be deployed to the appropriate training nodes, avoid performance bottlenecks due to excessive load of the training nodes, and also ensure that the resources of the training nodes can be fully utilized.

[0030] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing undue limitations on the disclosure.

[0032] Figure 1 A structural schematic diagram of a neural network model to which the embodiments of the present disclosure are applicable is shown;

[0033] Figure 2A schematic diagram showing deployment of a model on different nodes after sharding of the model according to an embodiment of the present disclosure is shown.

[0034] Figure 3 A schematic diagram showing pipeline parallel processing and data parallel processing of a model according to an embodiment of the present disclosure is shown.

[0035] Figure 4 A flowchart of a model processing method according to an embodiment of the present disclosure is shown.

[0036] Figure 5 A schematic diagram of a model deployment scenario according to an embodiment of the present disclosure is shown.

[0037] Figure 6 A flowchart of a model processing method according to yet another embodiment of the present disclosure is shown.

[0038] Figure 7 A system architecture diagram of a distributed training system according to an embodiment of the present disclosure is shown.

[0039] Figure 8 A flowchart of model deployment by a management node and a training node of a distributed training system according to an embodiment of the present disclosure is shown.

[0040] Figure 9 A diagram showing forward activation propagation in each round of training of a training task according to an embodiment of the present disclosure is shown.

[0041] Figure 10 A diagram showing backward gradient propagation in each round of training of a training task according to an embodiment of the present disclosure is shown.

[0042] Figure 11 A flowchart of parameter updating by a training node according to an embodiment of the present disclosure is shown.

[0043] Figure 12 A flowchart of parameter updating by a management node according to an embodiment of the present disclosure is shown.

[0044] Figure 13 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0045] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Features described in the description, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0046] In addition, the accompanying drawings are only schematic and are non-limiting illustrative of the disclosure. Identical components have been given the same reference numerals in the various drawings and the same reference designators have been maintained where possible in the non-limiting figures of the drawings and explicit descriptions of the same will not be repeated in each drawing. Some of the schematic drawings shown are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0047] It should be noted that the ordinal numbers such as "first", "second", etc. mentioned in the embodiments of the present disclosure are used to distinguish a plurality of objects, and are not used to limit the order, time sequence, priority or importance of the plurality of objects, and the description of "first", "second" does not necessarily mean that the objects are different.

[0048] In order to effectively utilize the distributed computing resources in the network, especially to utilize the processors distributed in different regions for distributed efficient cooperation, and to provide computing power support for large-scale model training, based on the model slicing, pipeline parallelism and data parallelism in distributed training, the embodiments of the present disclosure provide a model processing method and a distributed training system. In order to facilitate understanding, the model slicing, pipeline parallelism and data parallelism are explained and described.

[0049] Model slicing: a model with a neural network architecture is divided into multiple parts in series, each part is called a model slice, and each model slice contains one or more layers of neural units.

[0050] Pipeline parallelism: after the model is divided into multiple serial model slices from input to output, the front and rear model slices can process different data respectively, so as to achieve the effect of simultaneous training. This training method is called pipeline parallelism.

[0051] Data parallelism: when the training data is divided into multiple equal parts and input into multiple identical models respectively, then after the neural network calculation of each model, multiple equal outputs are obtained. This training method is called data parallelism.

[0052] The existing model is mainly constructed based on the structure of the neural network, and different neural networks are divided into standard networks, recurrent networks, convolutional networks, autoencoders, etc. due to slight differences in structure and different application scenarios, which basically include input layer, hidden layer and output layer, and the hidden layer is usually composed of multiple layers of neurons. The structure of the neural network model to which the model processing method provided by the embodiments of the present disclosure is applicable is not related to the structure of different neural network models. In order to facilitate understanding, the model processing method and the distributed training system of the embodiments of the present disclosure are described taking a typical feedforward neural network model as an example.

[0053] Figure 1A structure diagram of a neural network model to which embodiments of the present disclosure are applicable is shown. Referring to Figure 1 The neural network model is a typical feedforward neural network model, including an input layer, a hidden layer, and an output layer, wherein the hidden layer includes four layers, i.e., a hidden layer 1, a hidden layer 2, a hidden layer 3, and a hidden layer 4, each of the hidden layers includes four neural units, and the input layer and the output layer each include three neural units. The neural units in each layer are interconnected in a full connection manner.

[0054] Figure 2 A diagram showing deployment of the model on different nodes after being split into pieces according to embodiments of the present disclosure is shown. Since the resources of a single data center / region are limited, to implement distributed training of the model, the complete model structure is split into multiple model pieces, and each model piece is deployed in a node in a different region. Considering that too much splitting when splitting the model will bring massive data information synchronization between the model pieces, and the instability of the asynchronous network greatly affects the time delay of the massive data information synchronization, which will lengthen the time of training the large model. Based on this, when splitting the complete model structure, vertical splitting is performed, and horizontal splitting is not performed. As shown in Figure 2 Figure 1 The model shown in is split into four model pieces, which are deployed in nodes in four different cities, and each split stage includes one or more complete layers.

[0055] After the complete model is split into multiple model pieces, multiple nodes can work together to bear the computing power required by the entire model, and at the same time, different batches of training data in a round of training can simultaneously and in parallel occupy different model pieces to perform calculation. For example, when batch 3 data is calculated on the city C node in Figure 2 , batch 2 data is calculated on the city B node, and batch 1 data is calculated on the city A node. This training calculation of different batches of data in parallel on different model pieces is called pipeline parallel, and the combination of multiple model pieces of a complete model in series is called a pipeline parallel group.

[0056] In model training, to improve training efficiency, multiple pipeline parallel groups (i.e., multiple complete models) are replicated and deployed to perform parallel training, and the gradient information calculated by each pipeline parallel group is aggregated to obtain new model parameters. To ensure parallel training of multiple pipeline parallel groups, the training data in each round is divided into multiple data subsets, and different data subsets are given to different pipeline parallel groups for training.

[0057] Figure 3 A diagram showing pipeline parallel processing and data parallel processing of the model according to embodiments of the present disclosure is shown. As shown in Figure 3 ​As shown, two pipeline parallel groups A and B are deployed in the network, corresponding to input data subsets A and B respectively, and each data subset is further divided into batches 1, 2, 3, 4, etc. according to the order. In the training process, the same batch data of different data subsets may or may not be associated in training. For example, the batch 1 data of data subset A and the batch 1 data of data subset B may or may not be associated in training. This way of dividing training data into data subsets for training according to the number of pipeline parallel groups is called data parallelism. In data parallel training, to ensure synchronization of activation data and gradient data between model shards, the division method is consistent, for example, the way of dividing model shards in pipeline parallel groups A and B is consistent. Generally, the combination of model shards with the same arrangement position in different pipeline parallel groups is called a data parallel group, for example, Figure 3 Data parallel groups W, X, Y, and Z in FIG. 1.

[0058] Next, the model processing method and distributed training system of the embodiments of the present disclosure are explained and described.

[0059] Figure 4 A flowchart of a model processing method according to an embodiment of the present disclosure is shown. Figure 4 The model processing method provided by the embodiments is applied to a distributed training system, which includes a management node and a plurality of training nodes.

[0060] The management node serves as the management center of the entire model training or computing task, and undertakes important responsibilities such as task scheduling, resource allocation, resource scheduling, global training management, and state monitoring. The management node can be an independent entity server with sufficient computing power and storage resources to manage the entire training process. The management node can also be composed of a server cluster, and these servers are connected through high-speed networks to cooperate to effectively manage large-scale training tasks. In addition, the management node can also be deployed in the cloud, using the elastic scaling capability of cloud services to cope with training tasks of different scales. In terms of naming, in addition to "management node", it can also be called "management device", "management center" or "model training management center" according to specific responsibilities and functions.

[0061] Multiple training nodes are model training nodes distributed across various regions. As units that execute actual computational tasks, training nodes are responsible for handling tasks such as data computation and parameter updates during model training. A training node can be an independent physical server with sufficient computing resources to support the execution of training tasks. Training nodes can also be a server cluster composed of multiple servers, or virtual computing resources deployed in the cloud. In terms of naming, in addition to "training node," it can also be called "computing node," "computing device," "worker node," or "model training node," depending on the specific application scenario.

[0062] Management nodes and training nodes, as well as different training nodes, are connected via the internet or a private network. In some scenarios, management nodes and training nodes may be located in different geographical locations, spanning different data centers or cloud service providers, thus requiring an internet connection. Furthermore, to improve data transmission speed, stability, and security, a private network can be established to connect management nodes and training nodes.

[0063] Reference Figure 4 The model processing method is executed by the management node, and specifically includes the following steps S410 to S430.

[0064] Step S410: Obtain the training configuration information of the model and the resource information of each training node among multiple training nodes.

[0065] The model training configuration information refers to the parameter information required during model training, which may include model architecture (such as the number of layers and parameters per layer), training objectives (such as accuracy and loss function), training cycle, and batch size of training data. This training configuration information can be provided by the user requesting model training.

[0066] Multiple training nodes can be distributed across GPU training nodes in different regions. The resource information for each training node includes details such as the number and model of GPUs, memory size, storage capacity, and computing power. This information is used to evaluate the computing capabilities of the training node. The resource information for each training node can be obtained through various methods, including directly querying the node's resource information, collecting real-time data using monitoring tools, or periodically collecting and summarizing resource reports from each node through a management node.

[0067] Step S420: Generate a model deployment strategy based on the model's training configuration information and the resource information of each training node; wherein, the model deployment strategy is used to indicate the partitioning method used to partition the model and the node allocation strategy corresponding to the model partitions obtained based on the partitioning method.

[0068] The purpose of the model deployment strategy is to determine how to split the model into multiple model shards and how to allocate these model shards to appropriate training nodes to maximize resource utilization and training efficiency.

[0069] The division manner can be determined based on the architecture and resource requirements of the model, and can be based on the layers, parameter blocks or other logical units of the model for division, with the purpose of ensuring that the training nodes can process a part of the model while minimizing the communication overhead and data transmission delay between nodes.

[0070] After determining the division manner, a node allocation strategy is formulated, i.e., determining which training node each model shard is deployed to. The node allocation strategy can be formulated based on the resource information of each training node to ensure that the model shards can fully utilize the computing power of the training nodes and avoid resource overload or idleness.

[0071] In step S430, according to the model deployment strategy, multiple model shards corresponding to the model are obtained, and each model shard in the multiple model shards is deployed to a corresponding training node.

[0072] After generating the model deployment strategy, the model is processed according to the model deployment strategy, multiple model shards are obtained, and each model shard is deployed to a corresponding training node.

[0073] The model processing method provided by the embodiments of the present disclosure can bring the dispersed training nodes in the network into the training process across geographical restrictions, and the management node is responsible for collecting and integrating the resource information of these training nodes to provide sufficient computing power support for large model training. The management node dynamically adjusts the model deployment strategy according to the training configuration information of the model and the real-time resource information of each training node, so as to adapt to the model training requirements of different scales and complexities, and when generating the model deployment strategy, the resource situation of each training node is considered to ensure that the model shards can be deployed to appropriate training nodes, avoid performance bottlenecks due to excessive load of the training nodes, and also ensure that the resources of the training nodes can be fully utilized.

[0074] In some embodiments of the present disclosure, the training configuration information of the model includes model training quality requirement information. Before generating the model deployment strategy according to the training configuration information of the model and the resource information of each training node, the method further includes: filtering the multiple training nodes according to the resource information of each training node and the model training quality requirement information, and filtering out the training nodes in the multiple training nodes that do not meet the model training quality requirement information.

[0075] The model training quality requirement information refers to a constraint condition, a quality standard or a performance index that needs to be met during model training. For example, a geographical requirement of a training node, which limits that training nodes deployed during model training are located in the same region to reduce network delay and improve data transmission efficiency; for another example, a time limit requirement of a training node, which limits a minimum computing power value required by a single model shard to ensure that training can be completed within a specified time; for another example, a model accuracy, recall rate and the like obtained by training.

[0076] The resource information of each training node refers to the computing resources available to each training node, such as GPU, memory size, network bandwidth and the like, which determine the ability of the training node to process data and perform model calculation.

[0077] Before generating the model deployment strategy, the multiple training nodes are filtered according to the resource information of each training node and the model training quality requirement information, so as to ensure that only those training nodes that can meet the model training quality requirement are selected for training, and the nodes that do not meet the requirement are filtered out, so as to avoid resource waste and ensure the efficiency and quality of the training process. After the filtering of the training nodes is completed, the model deployment strategy can be generated according to the resource situation of the remaining training nodes and the specific requirement of model training.

[0078] In the embodiments of the present disclosure, by filtering out the training nodes that do not meet the model training quality requirement, it can be ensured that the nodes participating in the training have sufficient computing power and meet the constraint conditions, which helps to guarantee the performance and accuracy of the final model, meet the user's expectations and requirements; by filtering the nodes, the number of nodes that need to be considered by the model deployment strategy can be reduced, thereby simplifying the deployment process, which helps to reduce the complexity and error rate of the deployment strategy and improve the reliability and stability of the deployment; the filtering of the nodes can be dynamically adjusted according to the model training quality requirement, so that the system is more flexible and scalable.

[0079] In some embodiments of the present disclosure, the training configuration information of the model includes a division strategy of training data and a hierarchical structure of the model, and the resource information of each training node includes computing power resources of each training node and communication delay information between each training node and other training nodes. Wherein, the model deployment strategy is generated according to the training configuration information of the model and the resource information of each training node, including: generating the model deployment strategy according to the division strategy of the training data, the hierarchical structure of the model, the computing power resources of each training node and the communication delay information between each training node and other training nodes.

[0080] The division strategy of the training data refers to the subdivision rule of the training data, including the number of rounds of training data, the number of data subsets contained in the data of each round of training, the number of batches, the size of each data batch, etc. The hierarchical structure of the model refers to the hierarchical structure of the model itself, including the total number of layers of the model, the number of neurons of each layer, and the hierarchical structure of the model is important for determining which levels can be calculated in parallel and how to allocate these levels on different training nodes. When generating the model deployment strategy, model parameter information can also be used, including the connection relationship of all neural units and all parameters.

[0081] The computing resource of each training node is used to represent the ability of each training node to process data and perform computing tasks. The communication delay information between each training node and other training nodes is used to represent the time required for data transmission from the training node to other training nodes, and the communication delay information includes multiple types, such as propagation delay, transmission delay, and processing delay.

[0082] The generation of the model deployment strategy is a complex process that needs to consider the division strategy of the training data, the hierarchical structure of the model, the computing resources of each training node, and the communication delay information between nodes. The specific implementation can be: analyzing the division strategy of the training data, determining how the training data will be divided into different subsets and batches, determining the number of data subsets and the number of batches required for each round of training, and the amount of data in each batch; analyzing the hierarchical structure of the model to obtain the total number of layers and the number of neurons in each layer, identifying which levels can be calculated in parallel and the dependency relationship between different levels; evaluating the computing resources of each training node to identify nodes with strong computing power and potential computing bottlenecks; examining the communication delay information between nodes to identify node pairs with high communication efficiency and potential communication bottlenecks; determining the subdivision rule of the training data according to the division strategy of the training data, determining which levels will be calculated on which training nodes based on the hierarchical structure of the model and the computing resources of the training nodes, and optimizing the data synchronization and model parameter update strategy considering the communication delay between nodes.

[0083] In the embodiments of the present disclosure, the model deployment strategy includes one or more pipeline parallel groups, one or more data parallel groups, and the correspondence between the model shards obtained based on the pipeline parallel groups and the data parallel groups and the plurality of training nodes.

[0084] Parallel computing of different batches of data on different model shards is called pipeline parallel, and the combination of multiple model shards that form a complete model in series is called a pipeline parallel group. The model deployment strategy includes one or more pipeline parallel groups, that is, in the model deployment strategy, it can be decided to copy a model into one or more pipeline parallel groups, and training is performed simultaneously through one or more pipeline parallel groups, which can accelerate the training of the model.

[0085] The data parallel group in the model deployment strategy refers to a strategy of deploying the same model copy to multiple training nodes, each node processing a different subset of data, so as to realize parallel computing to improve training efficiency. In the embodiment of the present disclosure, the structure of the model shards contained in each data parallel group is the same, and the number of model shards contained in each data parallel group is the same as the number of pipeline parallel groups (i.e., the number of model copies).

[0086] The correspondence between the model shards obtained based on the pipeline parallel group and the data parallel group and the multiple training nodes is used to indicate on which training node the model shard obtained according to the model deployment strategy is deployed.

[0087] In the embodiment of the present disclosure, the model deployment strategy is generated according to one or more of the following principles:

[0088] (1) Minimize the number of training nodes required for model deployment. In order to reduce the huge communication overhead caused by too many training nodes, as much as possible, use fewer training nodes to complete the distributed training of the model under the condition of not exceeding the cost, that is, the product of the number P of pipeline parallel groups and the number Q of data parallel groups in the model deployment strategy should be as small as possible.

[0089] (2) If the overall computing power resources of the multiple training nodes are scattered, adjust the training data division strategy. When the computing power distribution of the multiple training nodes in the network is relatively scattered, that is, the total sum of the idle computing power resources of all training nodes meets the demand, but the idle computing power resources of a single training node is small, it is considered to further subdivide the L data batches of each round of training, for example, to 2L data batches with smaller size.

[0090] (3) Minimize the communication delay between different training nodes in each data parallel group, and the communication delay between different training nodes in the group is similar.

[0091] In each data parallel group, frequent communication is needed between different training nodes to synchronize gradient information. By optimizing the communication delay between nodes in the data parallel group, the waiting time can be reduced and the training speed can be improved. At the same time, keeping the communication delay between different training nodes in the group similar can avoid the impact of communication delay on the overall training efficiency of some nodes.

[0092] (4) Minimize the communication delay between different data parallel groups in one or more data parallel groups, and the communication delay between different data parallel groups is similar.

[0093] When there are multiple data parallel groups, communication is also needed between different data parallel groups to coordinate the training process. By minimizing the communication latency between different data parallel groups and keeping these latencies similar, the synchronization and stability of the entire training process can be ensured, further improving the training efficiency.

[0094] In some embodiments of the present disclosure, according to the model deployment strategy, obtaining a plurality of model shards corresponding to the model, deploying each model shard in the plurality of model shards to a corresponding training node, comprises: according to one or more pipeline parallel groups and one or more data parallel groups in the model deployment strategy, replicating and fragmenting the model to obtain a plurality of model shards; according to the correspondence between the model shards in the model deployment strategy and the plurality of training nodes, distributing each model shard in the plurality of model shards to a corresponding training node; receiving the configuration completion message sent by the training node corresponding to each model shard, and determining that the model deployment is completed.

[0095] After generating the model deployment strategy, the model is replicated according to the number P of pipeline parallel groups in the model deployment strategy, and the replicated model is fragmented according to the number Q of data parallel groups. The number of model shards contained in each pipeline parallel group is the same as the number Q of data parallel groups, that is, each pipeline parallel group contains Q model shards, and the number of model shards contained in each data parallel group is the same as the number of pipeline parallel groups, that is, each data parallel group contains P model shards.

[0096] After obtaining P*Q model shards, according to the correspondence between the model shards and the plurality of training nodes, each model shard is distributed to a corresponding training node, and a configuration completion message is sent after the training node corresponding to each model shard deploys the distributed model shard. The management node receives the configuration completion message sent by the training node corresponding to each model shard, and determines that the model deployment is completed.

[0097] Figure 5 A model deployment scenario schematic diagram of an embodiment of the present disclosure is shown. Figure 5 In the figure, the hierarchical structure of the model is simplified, and the model shard obtained by dividing a complete model is simplified and identified by a single rectangle, which blurs the internal neural units and the connection relationship between the neural units, and shows the parallel and serial relationship between the model shards.

[0098] Referring to Figure 5 The model is replicated into 2 pipeline parallel groups, namely pipeline parallel group 1 and pipeline parallel group 2, each pipeline parallel group contains 3 model shards, pipeline parallel group 1 contains model shard 1, model shard 2 and model shard 3, and pipeline parallel group 2 contains model shard 4, model shard 5 and model shard 6.

[0099] Figure 5In the example, model shard 1 is assigned to node 1 and model shard 4 is assigned to node 4, model shard 1 and model shard 4 are model shards in different two pipeline parallel groups and the model hierarchy structures contained in model shard 1 and model shard 4 are the same, belonging to the same data parallel group. Similarly, model shard 2 is assigned to node 2 and model shard 5 is assigned to node 5, model shard 2 and model shard 5 are model shards in different two pipeline parallel groups and the model hierarchy structures contained in model shard 2 and model shard 5 are the same, belonging to the same data parallel group; model shard 3 is assigned to node 3 and model shard 6 is assigned to node 6, model shard 3 and model shard 6 are model shards in different two pipeline parallel groups and the model hierarchy structures contained in model shard 3 and model shard 6 are the same, belonging to the same data parallel group.

[0100] Model shard 1 is assigned to corresponding node 1, and after node 1 deploys model shard 1, a configuration completion message is sent. Similarly, nodes 2 to 6 deploy the model shards assigned to themselves, and then send configuration completion messages to the management node. In this way, the management node determines that the model deployment is complete.

[0101] In some embodiments of the present disclosure, the method further includes: creating a training task corresponding to the model deployment strategy, and generating an identifier of the training task.

[0102] In the embodiments of the present disclosure, a training task corresponding to the model deployment strategy can be created, and a unique identifier is generated for the training task. The identifier is used to track and manage the training task in subsequent model deployment and model training processes.

[0103] In some embodiments of the present disclosure, according to the correspondence between the model shards in the model deployment strategy and the plurality of training nodes, each model shard in the plurality of model shards is assigned to a corresponding training node, including: according to the correspondence, assigning the identifier of the training task, the hierarchical structure of each model shard and the parameter information of each model shard to the training node corresponding to each model shard, so that the training node corresponding to each model shard performs resource configuration.

[0104] The model deployment strategy has defined how to divide the model into a plurality of model shards and how to assign these model shards to different training nodes. According to the correspondence in the model deployment strategy, the identifier of the training task, the hierarchical structure of each model shard and the parameter information of each model shard are assigned to the training node corresponding to each model shard. In this way, the training node corresponding to each model shard can perform resource configuration according to the received information, including memory allocation, computing resource allocation, etc., so as to efficiently execute the training task.

[0105] In some embodiments of the present disclosure, the method further comprises: establishing a node relationship between the training node corresponding to each model shard and other training nodes; wherein the node relationship is used to indicate adjacent nodes in the pipeline parallel group to which the training node corresponding to each model shard belongs, and other training nodes in the data parallel group to which the training node corresponding to each model shard belongs; and distributing the node relationship between the training node corresponding to each model shard and other training nodes to the training node corresponding to each model shard, so that the training node corresponding to each model shard is configured with the node relationship.

[0106] In distributed training, model shards are distributed to different training nodes, and these training nodes need to communicate with each other to coordinate the training process. In the embodiments of the present disclosure, for each training node corresponding to a model shard, the management node establishes a node relationship between the training node and other training nodes. This node relationship is used to describe the connection between the training node and other training nodes, in particular, adjacent nodes in the pipeline parallel group to which the training node belongs and other training nodes in the data parallel group to which the training node belongs.

[0107] The pipeline parallel group refers to a set of training nodes divided into different stages in model parallel training, and each stage is responsible for a part of the model calculation. The adjacent nodes refer to the upstream adjacent node and the downstream adjacent node of the current training node in the same pipeline parallel group. For example, Figure 5 For example, the upstream adjacent node of node 2 is node 1, and the downstream adjacent node is node 3.

[0108] The data parallel group refers to a set of training nodes responsible for the same model shard but processing different data batches in data parallel training. For example, Figure 5 For example, the other training nodes in the data parallel group to which node 2 belongs are nodes 5.

[0109] The node relationship between the training node corresponding to each model shard and other training nodes is distributed to the training node corresponding to each model shard, so that each training node knows which other nodes it needs to communicate with in order to correctly perform the training task.

[0110] For example, Figure 5 After receiving the node relationship between itself and other training nodes, nodes 1 to 6 can communicate with other nodes. Specifically, in the forward propagation process of training data and activation data, nodes 1 and 4 respectively calculate and process the input data, and then transmit the calculated output activation data to nodes 2 and 5 for further processing, and then to nodes 3 and 6; on the contrary, in the backward propagation process of gradient data, nodes 3 and 6 respectively calculate and process the input gradient data, and then transmit the calculated output gradient data to nodes 2 and 5 for further processing, and then to nodes 1 and 4.

[0111] In some embodiments of the present disclosure, further comprising: according to the identification of the training task, starting the training node corresponding to each model slice to execute the training task.

[0112] In the embodiments of the present disclosure, after the training node corresponding to each model slice completes the configuration of the allocated model slice, according to the identification of the training task, the training node corresponding to each model slice is started to execute the training task, forward propagation and back propagation calculation are performed according to the model slice, and the model parameters are updated.

[0113] Taking Figure 5 as an example to illustrate the training process of each round of training of the training task. In each round of training of the training task, it is assumed that each data batch input is identified by B i,j , i is the ID of the data subset and the ID of the corresponding pipeline parallel group, such as 1 and 2 in Figure 5 , j is the ID of the data batch in the data subset i, Figure 5 , and examples are 1, 2, and 3. After each B i,j input is calculated by different module slices, the gradient data generated is identified by G i,j,k , i is the ID of the pipeline parallel group where it is located, j is the ID of the corresponding data batch, and k is the number of the data parallel group where the model slice is located.

[0114] In data training, the gradient data G 1,1,3 and G 2,1,3 calculated by node 3 and node 6 are respectively transmitted to node 2 and node 5, and G 1,1,3 and G 2,1,3 are also transmitted to all other nodes in the same data parallel group, that is, mutual transmission between node 3 and node 6. After node 3 and node 6 in the same data parallel group collect the gradient data sent by all other nodes in the group, a temporary gradient data of the model slice of the data parallel group is calculated according to the algorithm defined in advance, such as G’ 1,3 =(G 1,1,3 +G 2,1,3 ) / 2, where G’ 1,3 is a temporary gradient data, 1 represents the data batch ID of the corresponding input data in the data subset, 3 represents the ID of the data parallel group to which it belongs, and the algorithm for calculating it is predefined, which may not be a linear average way.

[0115] Next, node 3 and node 6 will continue to calculate G 1,2,3 and G 2,2,3 according to the gradient data transmitted in the backward direction, and calculate G’ 2,3 in the same way. Next, G’ 3,3and finally calculate the final gradient data G"3 of the data parallel group after this round of training, where 3 in G"3 represents the number of the data parallel group, i.e. 3, and G"3 = (G 2,3 + G' 3,3 ) / 2, and the calculation method is also predefined and can not be linear. Node 3 and node 6 update the local model shard parameters P3 according to the value of G"3.

[0116] In the manner of node 3 and node 6, node 1 and node 4, node 2 and node 5 also calculate the final gradient data G"1 and G"2 of this round of training, and finally obtain the updated parameters P1 and P2.

[0117] As can be seen from the typical process exemplified above, in the process of gradient data back propagation, because each node in the data parallel group needs to collect the gradient data of all other nodes, each node needs to cache the newly calculated gradient data while performing back propagation, to ensure that all nodes in the group can calculate the final parameters according to all the gradient data.

[0118] In some embodiments of the present disclosure, further comprising: recording the training state of each batch of data in the current round of training in each round of training of the training task.

[0119] In each round of training of the training task, the training state of all data batches of all data subsets in the current round of training is recorded, for example, the data batch B 1,3 After being input into the model, all forward activation processes have been completed, and the current state is the back propagation gradient calculation state of the data parallel group with ID 2. When a network or node fails, the batches of data affected by the failure that have not completed training can continue to complete the remaining training from the failure.

[0120] In some embodiments of the present disclosure, further comprising: recording the parameter update state of each model shard in each round of training of the training task; in response to each model shard completing parameter update, updating the training state of the model and determining to execute the next round of training.

[0121] In each round of training of the training task, the state of whether all model shard parameters are updated is recorded, and the next round of data training is performed after all model shard parameters are updated.

[0122] In the embodiments of the present disclosure, when the network fails, the data communication state between nodes is monitored to locate which nodes have failures or bottlenecks. If it is found that the upstream and downstream communication of a certain pipeline parallel group of a certain training task is blocked, the training of this pipeline parallel group will be interrupted in time, and the training data that has not been completed by the round training will be rescheduled to the corresponding position of other pipeline parallel groups for training. If it is found that a certain data parallel group has a communication congestion problem, the model shard causing the congestion will be redeployed, and the replacement of the node where the model shard is located will be completed after the round training ends and before the next round training starts.

[0123] In the embodiments of the present disclosure, the management node collects the computing power of each training node and the delay between different nodes in real time, and determines the optimal model deployment strategy according to these resource conditions and the hierarchical structure of the model itself, that is, the optimal pipeline parallel and data parallel scheme, determines how to divide the model shards of the model and which nodes to use to deploy these model shards. In addition, the management node also manages the deployed training nodes and training tasks, including merging, replacing and other management operations on certain model training nodes due to network or other reasons, and monitoring the progress state of different training tasks.

[0124] Figure 6 A flowchart of a model processing method according to another embodiment of the present disclosure is shown. Figure 6 The model processing method provided by the embodiments is applied to a distributed training system including a management node and a plurality of training nodes. Referring to Figure 6 The model processing method is performed by the training node, and specifically includes the following step S610.

[0125] In step S610, the model shard allocated by the deployment management node according to the model deployment strategy; wherein the model deployment strategy is generated according to the training configuration information of the model and the resource information of each training node in the plurality of training nodes, and the model deployment strategy is used to indicate the division method adopted for the model, and the node allocation strategy corresponding to the model shard obtained based on the division method.

[0126] In the distributed training system, the management node is responsible for formulating a model deployment strategy according to the training configuration information of the model and the resource information of each training node, which determines how to shard the model and to which training nodes the shards should be allocated. It should be noted that the distributed training system includes multiple training nodes, and some training nodes may not receive the model shards allocated by the management node, which may be because the resource status of these training nodes is temporarily not suitable for processing the current model shards, or because these training nodes do not meet the model training quality requirement information in the training configuration information of the model, or because the management node makes a strategic choice to optimize the overall training efficiency.

[0127] The model processing method shown in the embodiments of the present disclosure is executed by receiving the model shards allocated by the management node. In addition, whether the training node receives the model shard or not, the training node will report the resource information of the training node to the management node, which includes the computing power resource of the training node and the communication delay information between the training node and other training nodes. Each training node in the plurality of training nodes reports its own computing power resource and communication delay information with other training nodes to the management node, so that the management node generates a model deployment strategy, divides the model into multiple model shards, and allocates each model shard to a corresponding training node.

[0128] In some embodiments of the present disclosure, deploying the model shards allocated by the management node according to the model deployment strategy includes: obtaining an identification of a training task, a hierarchical structure of a model shard corresponding to the training node, and parameter information of the model shard corresponding to the training node; establishing a training task processing unit, and associating the training task processing unit with the identification of the training task; and performing resource configuration according to the hierarchical structure of the model shard corresponding to the training node and the parameter information of the model shard corresponding to the training node.

[0129] The identification of the training task is used to ensure that each training task can be uniquely identified and tracked. The training node determines the specific composition and computing requirements of the model shard according to the hierarchical structure and parameter information of the allocated model shard, so as to perform resource configuration.

[0130] In the embodiments of the present disclosure, the training node creates a training task processing unit, and identifies the training task processing unit through the received identification of the training task, so that the training task can be correctly allocated to the corresponding training task processing unit, and the training task is executed by the training task processing unit.

[0131] In some embodiments of the present disclosure, the method further comprises: obtaining a node relationship between the training node and other training nodes; wherein the node relationship is used to indicate adjacent nodes in a pipeline parallel group to which the training node belongs and other training nodes in a data parallel group to which the training node belongs; and performing node relationship configuration according to the node relationship between the training node and other training nodes.

[0132] The training node can obtain the node relationship between the training node and other training nodes assigned by the management node, which is used to describe the connection between the training node and other training nodes, in particular, the adjacent nodes in the pipeline parallel group to which the training node belongs and the other training nodes in the data parallel group to which the training node belongs. The training node records the upstream and downstream adjacent nodes in the pipeline parallel group to which the training node belongs and the other training nodes in the data parallel group to which the training node belongs. In this way, the training node knows which other nodes it needs to communicate with in order to correctly perform the training task.

[0133] In some embodiments of the present disclosure, the method further comprises: executing the training task by the training task processing unit according to the identification of the training task.

[0134] The training node is a basic computing unit responsible for distributed training, and the training task processing unit in the training node is a unit that specifically undertakes model slice deployment. The training node can contain multiple different training task processing units responsible for completing different training tasks. In the embodiments of the present disclosure, the training node executes the training task by the corresponding training task processing unit according to the identification of the training task.

[0135] In some embodiments of the present disclosure, the execution of the training task by the training task processing unit comprises: in each round of training of the training task, receiving, by the training task processing unit, a forward activation message sent by an upstream adjacent node in the pipeline parallel group to which the training node belongs; the forward activation message includes forward activation data output by the upstream adjacent node; performing calculation on the forward activation data output by the upstream adjacent node to obtain forward activation data output by the training node; packing the forward activation data output by the training node to obtain a new forward activation message, and sending the new forward activation message to a downstream adjacent node in the pipeline parallel group to which the training node belongs.

[0136] In each round of training of the training task, the training node receives a forward activation message from an upstream adjacent node in the pipeline parallel group to which the training node belongs. The forward activation message contains forward activation data output by the upstream adjacent node, i.e., data calculated by the upstream adjacent node in the forward propagation process. In addition to the forward activation data, the forward activation message also includes description information of the forward activation data, which contains <training task identification r, training round i, data subset ID j, data batch k>.

[0137] After receiving the forward activation data, the training node performs a computation on the data, which can include applying an activation function, performing forward propagation of a neural network layer, etc. After the computation is completed, the training node outputs its own forward activation data.

[0138] The training node packages its output forward activation data into a new forward activation message, which is sent to the next downstream node in the pipeline parallel group to which the training node belongs. The new forward activation message also includes a description of the training node's output forward activation data, which includes <training task ID r, training round i, data subset ID j, data batch k>. The next downstream node receives this message and continues the similar process of processing and passing.

[0139] In some embodiments of the present disclosure, performing a training task by a training task processing unit includes: in each training round of the training task, receiving, by the training task processing unit, a backward gradient message sent by a next downstream node in the pipeline parallel group to which the training node belongs; the backward gradient message including backward gradient data output by the next downstream node; performing a computation on the backward gradient data of the next downstream node to obtain backward gradient data output by the training node; and packaging the backward gradient data output by the training node to obtain a new backward gradient message, and sending the new backward gradient message to a previous upstream node in the pipeline parallel group to which the training node belongs.

[0140] In each training round of the training task, the training node receives a backward gradient message from a next downstream node in the pipeline parallel group to which the training node belongs. The backward gradient message includes backward gradient data output by the next downstream node, i.e., data calculated by the next downstream node in the backward propagation process. In addition to the backward gradient data, the backward gradient message also includes a description of the backward gradient data, which includes <training task ID r, training round i, data subset ID j, data batch k>.

[0141] After receiving the backward gradient data, the training node performs a computation on the data, which involves combining the backward gradient of the next downstream node with the parameters or activation values of the current node to calculate the backward gradient data that the current node should output. After the computation is completed, the training node outputs its own backward gradient data.

[0142] The training node packages the backward gradient data of its output to form a new backward gradient message, which will be sent to the upstream neighboring node in the pipeline parallel group to which the training node belongs. The new backward gradient message also includes description information of the backward gradient data of the output of the training node, which contains <training task ID r, training round i, data subset ID j, data batch k>. The upstream neighboring node will receive this message and continue to perform similar processing and transmission processes.

[0143] In some embodiments of the present disclosure, further comprising: in each training round of the training task, sending the backward gradient data of the output of the training node to other training nodes in the data parallel group to which the training node belongs, and receiving the backward gradient data of the output of other training nodes in the data parallel group to which the training node belongs; in response to obtaining the backward gradient data of all batch data in the current training round in the data parallel group to which the training node belongs, performing gradient data aggregation calculation to obtain aggregated gradient data, and updating the parameter information of the model shard corresponding to the training node using the aggregated gradient data.

[0144] In each training round of the training task, the training node not only exchanges backward gradient data with neighboring training nodes in the pipeline parallel group, but also exchanges backward gradient data with other training nodes in the data parallel group to which it belongs. This means that in addition to the gradient transmission brought by pipeline parallelism, there is an additional gradient aggregation step for synchronizing gradient information within the data parallel group.

[0145] When the training node obtains the backward gradient data of all batch data in the current training round in the data parallel group to which it belongs, it performs gradient data aggregation calculation, which is usually to aggregate the backward gradient data of all batch data in the data parallel group to which it belongs to obtain a unified, aggregated gradient data. Then, the training node updates the parameter information of the model shard corresponding to it using the aggregated gradient data.

[0146] In some embodiments of the present disclosure, further comprising: packaging the aggregated gradient data to obtain an aggregated gradient message; and sending the aggregated gradient message to other training nodes in the data parallel group to which the training node belongs.

[0147] After completing the gradient data aggregation calculation, the training node will package the aggregated gradient data to form an aggregated gradient message, which contains the gradient data calculated by aggregating the backward gradient data of all batch data in the current round of training in the data parallel group to which the training node belongs. After packaging, the training node will send this aggregated gradient message to other training nodes in the data parallel group. In this way, all nodes in the data parallel group can receive the aggregated gradient message containing the aggregated gradient data and perform corresponding model shard parameter updates.

[0148] In this way, it is ensured that all participating nodes can synchronize the latest gradient information, thereby maintaining the consistency and accuracy of model training. Through this cross-node gradient synchronization and update mechanism, the distributed training system can parallelly process large-scale data sets and complex models on multiple nodes, significantly improving training efficiency.

[0149] In some embodiments of the present disclosure, it further includes: in response to receiving the aggregated gradient message sent by other training nodes in the data parallel group to which the training node belongs, determining not to perform gradient data aggregation calculation; the aggregated gradient message sent by other training nodes includes aggregated gradient data; and updating the parameter information of the model shard corresponding to the training node using the aggregated gradient data.

[0150] The training node will receive the aggregated gradient message sent by other training nodes in the data parallel group, which contains the gradient data aggregated by other nodes. In response to receiving this aggregated gradient message, the training node will determine that it does not need to perform gradient data aggregation calculation again, because the aggregated gradient message already contains all the necessary information that can be directly used to update the parameters of the model shard. Then, the training node will use the aggregated gradient data contained in the aggregated gradient message to update the parameter information of the model shard corresponding to it, ensuring that the model shard can be updated based on the latest, globally aggregated gradient information, thereby maintaining the consistency and accuracy of model training.

[0151] In the embodiments of the present disclosure, the training node can efficiently use the aggregated gradient information from other nodes in the data parallel group, avoid unnecessary repeated calculation, and ensure smooth model training, which helps to improve training efficiency.

[0152] In the embodiments of the present disclosure, the process of each round of training performed by the training node combines pipeline parallelism and data parallelism, fully utilizes multi-node computing resources, and improves training efficiency; pipeline parallelism allows different parts of the model to be processed in parallel on different nodes, while data parallelism allows different data batches of the same model part to be processed in parallel on multiple nodes. This combination enables efficient training on larger data sets and more complex models.

[0153] Figure 7 A system architecture diagram of a distributed training system according to an embodiment of the present disclosure is shown. As shown in the figure, the distributed training system includes a management node and a plurality of training nodes. It should be understood that the number of training nodes and management nodes shown is exemplary, and the corresponding number of training nodes and management nodes can be set according to actual needs. Figure 7 Figure 7 It should be understood that the number of training nodes and management nodes shown is exemplary, and the corresponding number of training nodes and management nodes can be set according to actual needs.

[0154] The management node is configured to: obtain training configuration information of a model and resource information of each training node in the plurality of training nodes; generate a model deployment strategy according to the training configuration information of the model and the resource information of each training node, wherein the model deployment strategy is used to indicate a division manner adopted for model slicing and a node allocation strategy corresponding to a model slice obtained based on the division manner; and obtain a plurality of model slices corresponding to the model according to the model deployment strategy, and deploy each model slice in the plurality of model slices to a corresponding training node.

[0155] In the embodiment of the present disclosure, the management node is configured to: collect idle computing power resources of each training node and delay conditions between different nodes in real time, and calculate and determine an optimal pipeline parallelism and data parallelism scheme according to these resource conditions and the hierarchical structure of the model itself, that is, to determine how to divide the model slicing of the model and which nodes to use to deploy these model slices. In addition, the management node is also configured to: manage the deployed training nodes and training tasks, including management operations such as merging, replacing, etc. of certain model training nodes due to network or other reasons; and monitor the progress state of different training tasks.

[0156] As shown in the figure, the management node includes three functional units: a network resource management unit, a model deployment orchestration model, and a training task management unit. Figure 7 The network resource management unit is configured to: collect resource information reported by all training nodes, including idle computing power resources of each training node and delay information between different training nodes.

[0157] The network resource management unit maintains two database information. One database includes idle computing power resources of each model training node at different times, which is a data set composed of three-element arrays after one-dimensional expansion, represented as <node number, statistical time, idle computing power value>. Another database includes delay data between each training node and other training nodes, which is a data set composed of multiple four-element arrays after one-dimensional expansion, represented as <upstream node number, downstream node number, statistical time, transmission delay value>.

[0158]

[0159] ​​The network resource management unit is configured to periodically collect the idle computing power and time delay information sent by each training node and update the above database in real time. When the model deployment arrangement unit issues a resource query request, the network resource management unit provides real-time database information to the model deployment arrangement unit.

[0160] The model deployment arrangement unit is the core algorithm unit of the entire distributed training system, and is configured to optimize the replication and segmentation of the model according to the idle computing power resources and time delay of each training node in the network, and distribute the replicated and segmented model shards to different training nodes.

[0161] When a user submits a training request to the training task management unit, the user provides the training data shard rule, such as dividing the training data into several rounds for training, how many batches of data each round contains, and the hierarchical structure of the model. The training task management unit transmits this information to the model deployment arrangement unit, and the model deployment arrangement unit obtains the model deployment strategy as input by combining the computing power resources and time delay information obtained from the network resource management unit.

[0162] Specifically, each user of the model distributed training system may have their own requirements for how data is trained according to the scenario. Taking a typical data training method as an example, the entire training data set is divided into N*L data batches, L data batches are taken out for training on the model each round, a total of N rounds of training, and each data batch produces a complete set of model gradient information after training. After aggregating and calculating all L sets of model gradient information, a new set of gradient information is obtained, and the parameters of the entire model are updated using the new gradient information, thereby completing a round of training. Next, L data batches are taken out from the data set, and a round of training is performed on the model whose parameters are updated in the previous round, and finally the parameter update is completed. In this way, N rounds of training are completed.

[0163] In order to make the computing power resources of multiple training nodes be utilized as much as possible and reduce the idle waiting time of computing power, the model deployment arrangement unit optimizes the generation of the model deployment strategy according to the available idle computing power resources and time delay of each training node, and finally obtains that the model is replicated and deployed into P pipeline parallel groups, and the total is divided into Q data parallel groups, each pipeline parallel group contains Q model shards, and each data parallel group contains P model shards. The model deployment strategy can be generated according to the following principles.

[0164] (1) In order to reduce the huge communication overhead caused by too many nodes, as much as possible, the number of nodes used to complete the distributed training of the model should be reduced under the condition that the cost does not exceed the user's cost, that is, the value of P*Q should be as small as possible.

[0165] (2) When the idle computing resources of each training node in the network are scattered, that is, the sum of the idle computing resources of all training nodes meets the demand, but the idle computing resources of a single training node are small, the L data batches of each round of training should be further subdivided, such as being converted into 2L data batches with smaller size.

[0166] (3) The maximum latency between nodes in different data parallel groups should be as small as possible and close to each other.

[0167] (4) The maximum latency between different data parallel groups should be as small as possible and close to each other.

[0168] The training task management unit is used to manage the deployment of training tasks and distributed models, mainly with the following functions. (1) According to user demand, a new training task is established, and the training task ID is assigned; (2) For a new training task, model shards are specifically deployed on different nodes to form the required pipeline parallel group and data parallel group; (3) The logical relationship between nodes in each training task is maintained, such as which nodes belong to the same data parallel group and which nodes have adjacent relationships in the same pipeline parallel group; (4) In each round of training of each training task, the status of all data batches of all data subsets being trained is recorded, for example, data batch B 1,3 After being input into the model, all forward activation processes have been completed, and the current state is the backward gradient calculation of the data parallel group with ID 2; (5) When a network or node fails, the training of each batch of data affected by the failure is continued from the breakpoint to complete the remaining training; (6) In each round of training of each training task, the status of whether all model shard parameters are updated is recorded, and the user is notified to proceed to the next round of data training after all model shard parameters are updated.

[0169] The training task management unit is also used to locate which nodes have failures or bottlenecks by monitoring the data communication state between nodes when a network fails. Specifically, if it is found that a pipeline parallel group of a certain training task has upstream and downstream communication blocked, the training of this pipeline parallel group will be interrupted in time, and the training of the data that has not completed the training in the current round will be rescheduled to the corresponding position of other pipeline parallel groups for training; if it is found that a data parallel group has a communication congestion problem, the model shard causing the congestion will be redeployed, and the replacement of the node where the model shard is located will be completed after the current round of training ends and before the next round of training begins.

[0170] The training task management unit maintains five database information. The first database is the model layer information contained in each data parallel group in each training task, which is a data set composed of four-element arrays after one-dimensional expansion, and the four-element array is <training task ID, data parallel group ID, model start layer number, model end layer number>. For example, in training task 189, the four-element array of the data parallel group containing the input layer and the next two layers of neural units is <189, 0, 0, 2>.

[0171] The third database is the forward downstream node information of each node in the same pipeline parallel group, which is also a data set composed of three-element arrays after one-dimensional expansion, and the three-element array is <training task ID, node ID, forward downstream node ID>.

[0172] The fourth database is the training completion state of each batch data, which is a data set composed of four-element arrays after one-dimensional expansion, and the four-element array is <training task ID, data batch ID, direction, data parallel group ID>. Here, the direction represents the forward activation transmission process and the backward gradient transmission process through 0 and 1, and the data parallel group ID indicates which model slice of the data parallel group the calculated activation (or gradient) data is.

[0173] The fifth database is whether each model slice has completed the required parameter update in this round of training, which is a data set composed of four-element arrays after one-dimensional expansion, and the four-element array is <training task ID, pipeline parallel group ID of the model slice, data parallel group ID of the model slice, number of updated parameters>. If the model slice has not completed the parameter update in this round of training, the "number of updated parameters" here will be the value of the last round.

[0174] The training node in the distributed training system is the basic computing unit responsible for distributed training. The training node is used to periodically provide the idle computing resources of the node and the time delay information between other nodes to the management node, and also needs to cooperate with the management node to deploy training task processing units for different training tasks, and report various state information of each training task processing unit in real time.

[0175] As Figure 7As shown, the training node includes a local idle computing power statistics unit, a remote communication delay statistics unit, a training task local management unit, and a training task processing unit. The training task processing unit is a unit that specifically undertakes model slice deployment. Each training node can contain multiple different training task processing units, which are responsible for completing different training tasks. The training task processing unit further includes four modules, namely, forward activation processing, backward gradient processing, parameter update processing, and model slice calculation.

[0176] The local idle computing power statistics unit is configured to periodically count the idle computing power resources in the training node and report the same to the network resource management module of the management node. For example, the local idle computing power statistics unit counts how many idle GPUs in the training node and corresponding models and computing power.

[0177] The training node initiates a heartbeat process to test the communication delay information between the training node and other nodes in the distributed training system through the remote communication delay statistics unit. Once heartbeat process includes sending an inquiry message and a response message. The inquiry message contains the training node ID and the sending time in the message body. The remote training node returns a response message upon receiving the inquiry message. The response message contains the remote node ID (i.e., the sender of the response message), the inquiry message receiving time, and the response message sending time. The training node records the response message receiving time upon receiving the response message and calculates the bidirectional delay between the training node and the remote node. The remote communication delay statistics unit records and updates the bidirectional delay in real time and periodically sends the information to the network resource management module of the management node.

[0178] The training task processing unit is the most basic training unit in the distributed training system and is responsible for the calculation of a specific model slice. Each training task processing unit includes a forward activation processing module, a backward gradient processing module, a parameter update processing module, and a model slice calculation module. Next, the functions of each module in the training task processing unit are described.

[0179] The forward activation processing module is configured to receive activation data sent by an upstream adjacent node in the same pipeline parallel group, pass the activation data to the model slice calculation module for calculation, and then pack the new activation data output by the calculation into a message and send the same to a downstream adjacent node in the same pipeline parallel group. The message contains the following key information: <training task ID, downstream adjacent node address, node address, data subset ID to which the activation data belongs, and data batch ID to which the activation data belongs (i.e., batch in the data subset)>.

[0180] The backward gradient processing module is configured to receive gradient data sent by a downstream adjacent node in the same pipeline parallel group, and send the gradient data to the model slice calculation module for calculation and processing, and then pack the new gradient data output by the calculation into a message and send the message to an upstream adjacent node in the same pipeline parallel group. The message contains key information such as <training task ID, upstream adjacent node address, node address, data subset ID to which the gradient data belongs, and data batch ID to which the gradient data belongs>. In addition, the backward gradient processing module will copy a copy of the received gradient data and send it to the parameter update processing module.

[0181] The parameter update processing module is configured to obtain the latest gradient data from the backward gradient processing module, and pack the gradient data into a message and broadcast the message to other training nodes in the same data parallel group. The parameter update processing module is also configured to receive gradient data messages sent by other training nodes in the data parallel group, and perform aggregation calculation after receiving all the gradient data sent by other training nodes, and finally update the parameters of the model slice according to the aggregated gradient data.

[0182] It should be understood that there are various ways for the parameter update processing module to calculate the aggregated gradient, and the calculation method is determined by the training method. Figure 5 For example, the temporary aggregated data of the gradient data generated by the data in the same batch ID in each data subset can be calculated first, and then the second aggregation calculation is performed on all batch temporary aggregated data to obtain the final aggregated gradient data. If the two-layer aggregation calculation required by the training method is only a simple linear calculation, the batch aggregation calculation is not required before the batch aggregation calculation, but the calculation can be mixed together, which can reduce the synchronization waiting time of each gradient data, thereby saving the aggregation time of the entire gradient data.

[0183] The parameter update processing module also has a cache function, which is configured to cache single-batch gradient data sent by the training node and other training nodes in the data parallel group, so as to facilitate the final unified aggregation calculation. The gradient data message contains key information such as <training task ID, address of other nodes in the group, node address, data subset ID to which the gradient data belongs, and batch ID to which the gradient data belongs>.

[0184] The model slice calculation module is a calculation unit for data training, and is a specific deployment and implementation of a single model slice of a model divided as a whole on a node. The model slice calculation module is configured to forwardly output new activation data according to input activation data, and backwardly output new gradient data according to input gradient data; and when obtaining the final aggregated gradient data from the parameter update processing module, the parameter update of each neural unit in the module is completed according to the aggregated gradient data.

[0185] The training task local management unit is configured to manage each training task processing unit, including: cooperating with the training task management unit of the management node to create a new training task in the node, deploying a new training task processing unit, including allocating computing power resources and creating each module in the training task processing unit; obtaining a training task ID from the training task management unit, which is used to identify and manage the newly created training task processing unit.

[0186] The training task local management unit is further configured to: cooperate with the training task management unit of the management node to realize deletion and resource release of the training task processing unit in the node; cooperate with the training task management unit of the management node to realize capacity expansion and resource increase of the training task processing unit in the node; and report the running status of each training task processing unit to the training task management unit in real time, including the time delay of active data and gradient data processing, the time delay of gradient synchronization and aggregation calculation, etc.

[0187] Next, based on the distributed training system shown in FIG. 1, the coordination mode and workflow between the management node and the training node in the distributed training system are described. Figure 7

[0188] Figure 8 A flowchart of model deployment of the management node and the training node of the distributed training system of the embodiment of the present disclosure is shown. As shown in FIG. 2, the model deployment can include the following steps. Figure 8

[0189] Step S801: The training node periodically sends the idle computing power resources of the node and the communication time delay information between the node and other training nodes to the network resource management unit of the management node, and the network resource management unit updates the computing power resources and the communication time delay information of the whole network.

[0190] Step S802: The user sends a request for model training to the training task management unit of the management node, and sends the subdivision rule of the training data, the hierarchical structure of the model, and the complete model.

[0191] The subdivision rule of the training data includes the number of rounds of data training, the number of subsets and the number of batches of data contained in each round of training, and the size of each data batch; the hierarchical structure of the model includes the total number of layers of the model and the number of neural units of each layer; and the complete model includes the connection relationship and all parameters of all neural units contained in the model.

[0192] Step S803: The training task management unit requests the model deployment orchestration unit to calculate the optimal model deployment strategy, and provides key constraint information, including the subdivision rule of the training data, the hierarchical structure of the model, and the overall quality requirement of the training, etc.

[0193] ​​Step S804, after receiving the request from the training task management unit, the model deployment orchestration unit requests the network resource management unit to obtain appropriate resource information. This request contains preliminary constraints set according to user requirements, for example, limiting the minimum computing power value required by a single model shard because the model size is large and the user time limit is high, or limiting all training nodes to be within the same geographic area.

[0194] Step S805, after receiving the request, the network resource management unit filters out the node resource set that meets the user's requirements and sends it to the model deployment orchestration unit. The node resource set includes the idle computing power resources of each training node in the set and the communication delay information between different training nodes.

[0195] Step S806, the model deployment orchestration unit calculates the optimal model deployment strategy according to the node resource set sent by the network resource management unit, the previously obtained training data subdivision rule, and the hierarchical structure of the model. The model deployment strategy includes establishing how many pipeline parallel groups, how to divide the model shards in each pipeline parallel group (i.e., the construction of data parallel groups), and which node to deploy each model shard.

[0196] Step S807, the model deployment orchestration unit sends the calculated model deployment strategy to the training task management unit.

[0197] Step S808, after receiving the model deployment strategy, the training task management unit creates a new training task, sets the training task ID, sets the corresponding pipeline parallel group and data parallel group according to the model deployment strategy, divides the complete model into multiple model shards, maps each model shard to a different node, and assigns each node to a specific pipeline parallel group and data parallel group.

[0198] Step S809, the training task management unit sends configuration instructions for the model shard to the training task local management unit of the training node, which contains the training task ID and the complete model shard corresponding to the node, including the connection relationship of all neural units in the shard and the complete parameters.

[0199] Step S810, after confirming that the local idle computing power resources meet the requirements, the training task local management unit of the training node creates a new training task, configures the model shard, establishes a new training task processing unit, and uses the received training task ID to identify this training task processing unit.

[0200] Step S811, the training task local management unit of the training node reports the result of the model shard configuration to the training task management unit of the management node.

[0201] Step S812, after the training task management unit collects the messages that all training nodes send that the model shards are configured, the training task management unit establishes the model management library of this training task, and records the information of the data parallel group and the pipeline parallel group to which all participating nodes belong in this training task.

[0202] Step S813, the training task management unit returns a message to the training task local management unit of each training node, indicating the node relationship between each node and other nodes in this round of training task. The returned message includes the training task ID, the other node ID and address in the data parallel group to which the node belongs, and the upstream and downstream adjacent node ID and address in the pipeline parallel group to which the node belongs.

[0203] Step S814, after the training task local management unit of the training node receives the message of the node relationship with other nodes, the remaining part of the training task processing unit is configured, for example, the address of the upstream and downstream adjacent nodes in the pipeline parallel group to which the node belongs in this training task is written into the backward gradient processing module and the forward activation processing module.

[0204] Step S815, after the training task processing unit is fully configured, the training task local management unit of the training node sends a message to the training task management unit of the management node.

[0205] Step S816, after receiving the configuration completion messages from all nodes of this training task, the training task management unit of the management node sends a message to the user to inform that the distributed training system of this training task has completed deployment.

[0206] Figure 9 The figure shows the process of forward activation transmission in each round of training of the training task of the embodiment of the present disclosure. Figure 9 The embodiment is described by taking node 2 in Figure 5 as an example, the upstream adjacent node in the pipeline parallel group to which node 2 belongs is node 1, and the downstream adjacent node in the pipeline parallel group to which node 2 belongs is node 3.

[0207] As shown in Figure 9 , the process of forward activation transmission of node 2 in a single round of training of training task r includes the following steps.

[0208] Step S901, the training task local management unit of node 2 receives the forward activation transmission message sent by the upstream adjacent node in the same pipeline parallel group, i.e. node 1. This message contains the forward activation data (compressed and encoded) output by node 1 and the description information of this data, and the description field is represented as <training task ID r, training round i, data subset ID j, data batch k>.

[0209] Step S902, after receiving the forward activation delivery message, the training task local management unit of node 2 determines the corresponding training task processing unit according to the training task ID.

[0210] Step S903, the forward activation data and the description information in the message are delivered to the forward gradient processing module of the corresponding training task processing unit.

[0211] Step S904, after receiving the forward activation data, the forward activation processing module decodes and decompresses the forward activation data.

[0212] Step S905, the forward activation processing module inputs the decoded and decompressed forward activation data into the model shard computing module of the same training task processing unit, and the model shard computing module calculates new forward activation data.

[0213] Step S906, the model shard computing module delivers the calculated new forward activation data to the forward activation processing module.

[0214] Step S907, the forward activation processing module compresses and encodes the new forward activation data, and packs the data together with the description information into a new forward activation message. The description information in the new forward activation message includes <training task ID r, training round i, data subset ID j, data batch k>.

[0215] Step S908, the forward activation processing module sends the new forward activation message to the downstream adjacent node, i.e. node 3, in the same pipeline parallel group. After receiving the message, node 3 will perform the same operation as node 2.

[0216] Figure 10 The process diagram of the backward gradient delivery in each training round of the training task of the embodiment of the present disclosure is shown. Figure 10 The embodiment is described by taking node 2 in Figure 5 as an example. The upstream adjacent node of node 2 in the pipeline parallel group to which node 2 belongs is node 1, and the downstream adjacent node of node 2 in the pipeline parallel group to which node 2 belongs is node 3.

[0217] As shown in Figure 10 , the process of node 2 in the backward gradient delivery in a single training round of training task r includes the following steps.

[0218] Step S1001, the downstream adjacent node of node 2 in the pipeline parallel group to which node 2 belongs, i.e. node 3, sends the output backward gradient message to node 2. The message contains the backward gradient data calculated and output by node 3 and the description information for the gradient data, which includes <training task ID r, training round i, data subset j, data batch k>.

[0219] Step S1002, after receiving the backward gradient message, the training task local management unit of node 2 determines the corresponding training task processing unit according to the training task ID.

[0220] Step S1003, the training task local management unit of node 2 transmits the backward gradient data and the description information in the message to the backward gradient processing module in the corresponding training task processing unit.

[0221] Step S1004, after receiving the backward gradient data, the backward gradient processing module decodes and decompresses the backward gradient data.

[0222] Step S1005, the backward gradient processing module inputs the decoded and decompressed backward gradient data into the model shard computing module in the same training task processing unit.

[0223] Step S1006, the model shard computing module calculates new backward gradient data and transmits the calculated new backward gradient data to the backward gradient processing module.

[0224] Step S1007, the backward gradient processing module copies and transmits the new backward gradient data to the parameter update processing module in the same training task processing unit.

[0225] Step S1008, the parameter update processing module aggregates and calculates the new backward gradient data with other batches of backward gradient data to obtain aggregated gradient data, and updates the parameters of the model shard using the aggregated gradient data.

[0226] Step S1009, the backward gradient processing module compresses and encodes the new backward gradient data, and packs it into a backward gradient message together with the description field of the data.

[0227] Step S1010, the backward gradient processing module transmits the packed backward gradient message to the upstream adjacent node, i.e., node 1, in the same pipeline parallel group. The description information in the gradient message includes <training task ID r, training round i, data subset ID j, data batch k>. After receiving the message, node 1 performs the same operations as node 2.

[0228] The backward gradient processing module in the training task processing unit of the training node copies and transmits the new backward gradient data to the parameter update module. The parameter update module collects all related backward gradient data, performs gradient data aggregation calculation and model shard parameter update, and finally synchronizes the parameter update result to the training task management unit of the management node. After collecting the parameter update results of all model shards in the current training, the training task management unit notifies the user to start the next round of data training.

[0229] Figure 11 A flowchart of the parameter updating of the training node of the embodiment of the present disclosure is shown. As shown in the figure, the parameter updating flow of the training node includes the following steps. Figure 11

[0230] Step S1101, the parameter updating processing module starts in a waiting state, waiting to receive the backward gradient data calculated by all batch data in the current training in the data parallel group to which it belongs, including all the backward gradient data calculated by the node itself.

[0231] Step S1102, if the backward gradient data calculated by the backward gradient processing module of the node itself is received, the backward gradient data is packaged into a backward gradient message, and the message is sent to the parameter updating processing module of the other training nodes in the data parallel group to which it belongs.

[0232] Step S1103, if the backward gradient data sent by the other training nodes in the data parallel group to which it belongs is received, it is judged whether the backward gradient data of all batch data in the current training in the data parallel group is received, if yes, step S1104 is executed, if not, step S1101 is executed.

[0233] Step S1104, the parameter updating processing module aggregates and calculates all the collected backward gradient data to obtain the aggregated gradient data.

[0234] Step S1105, the parameter updating processing module updates the parameters in the model shard calculation module of the node according to the aggregated gradient data, and after the update is completed, the result is notified to the global model management module of the management node through a message. It should be noted that the global model management module can be a submodule in the training task management module of the management node.

[0235] Step S1106, the aggregated gradient data is packaged into a message and sent to all other nodes in the data parallel group to which it belongs, except those nodes that have already obtained the aggregated gradient data, for example, in the case of directly moving from step S1101 to step S1105, the aggregated gradient data is directly obtained from another node, so the aggregated gradient data does not need to be sent to this node.

[0236] Step S1107, the parameter updating processing module directly enters a new round of data training state, i.e., comes to a state similar to step S1101, waiting to receive the gradient data of each batch in the next round of training.

[0237] Figure 12 A flowchart of the parameter updating of the management node of the embodiment of the present disclosure is shown. As shown in the figure, Figure 12 ​As shown, the parameter updating process of the global model management module of the management node includes the following steps.

[0238] Step S1201, enter the waiting state, wait for the notification that the parameter updating of all model shards in this round of training is completed, if the message that the parameter updating of a model shard is completed is received from a certain training node, enter step S1202.

[0239] Step S1202, judge whether the parameters of all model shards have been updated, if not, return to step S1201, if yes, enter step S1203.

[0240] Step S1203, update the state of model training, identify the end of this round of training under the current training task, and then send a message to the user, informing the user that the next round of data training can be started, enter step S1204.

[0241] Step S1204, similar to step S1201, start waiting for the notification that the parameter updating of all model shards in the next round of training under the current training task is completed.

[0242] The model processing method and the distributed training system proposed in the embodiments of the present disclosure are designed for the model shard, pipeline parallelism and data parallelism in distributed training, and a two-level management and control architecture of management nodes and training nodes is designed, and key function modules are deployed between the two-level architecture to realize the optimized deployment and scheduling of computing resources, and the effective utilization of training data activation and gradient is also realized, so that the data can be trained in an orderly and efficient manner.

[0243] Figure 13 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. It should be noted that Figure 13 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0244] As Figure 13 As shown, the electronic device 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1302 or programs loaded from a storage portion 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 are also stored. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0245] The following components are connected to the I / O interface 1305: an input part 1306 including a keyboard, a mouse, etc.; an output part 1307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1308 including a hard disk, etc.; and a communication part 13013 including a network interface card such as a LAN card, a modem, etc. The communication part 13013 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as necessary. A removable media 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1310 as necessary, so that a computer program read out therefrom is installed in the storage part 1308 as necessary.

[0246] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 13013, and / or installed from the removable media 1311. When the computer program is executed by the central processing unit (CPU) 1301, the above-described functions defined in the system of the present disclosure are executed.

[0247] It should be noted that the computer-readable medium shown in the disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, terminal device or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, terminal device or device. While in the disclosure, the computer-readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagating data signal can take many forms, including but not limited to electromagnetic signals, optical signals or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, terminal device or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0248] The flowcharts and block diagrams in the drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the disclosure. In this regard, each block in the flowchart or block diagram can represent a module, program segment or part of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the function involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0249] The units described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. The described units can also be arranged in a processor, for example, a processor can be described as including an address acquisition unit, an identifier generation unit, and a route publishing unit. Among them, the names of these units do not constitute a limitation on the units themselves in some cases. For example, the address acquisition unit can also be described as "a unit that acquires an Internet Protocol address of the first network device".

[0250] As another aspect, the present disclosure also provides a computer-readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments. For example, the electronic device can implement the steps shown in the above embodiments. Figure 4

[0251] According to an aspect of the present disclosure, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in various optional implementation manners of the above embodiments.

[0252] It should be understood that the number of any elements in the present disclosure is used by way of example only, not limitation, and any naming is only for distinction, and does not have any limiting meaning.

[0253] Other embodiments of the present disclosure will be apparent to those skilled in the art upon consideration of the specification and practice of the disclosed application. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known, accepted, or customary practice in the art to which the present disclosure pertains. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0254] It should be understood that the present disclosure is not limited to the precise structures described and shown in the above embodiments and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.​

Claims

1. A model processing method characterized by comprising: The method is applied to a distributed training system including a management node and a plurality of training nodes, and is performed by the management node, and includes: obtaining training configuration information of a model and resource information of each training node in the plurality of training nodes; generating a model deployment strategy according to the training configuration information of the model and the resource information of each training node, the model deployment strategy being used to indicate a division manner adopted for model slicing and a node allocation strategy corresponding to model slices obtained based on the division manner; obtaining a plurality of model slices corresponding to the model according to the model deployment strategy, and deploying each model slice in the plurality of model slices to a corresponding training node.

2. The method of claim 1, wherein, The training configuration information of the model includes a division strategy of training data and a hierarchical structure of the model, and the resource information of each training node includes computing resource of each training node and communication latency information between each training node and other training nodes; wherein the generating of the model deployment strategy according to the training configuration information of the model and the resource information of each training node includes: generating the model deployment strategy according to the division strategy of the training data, the hierarchical structure of the model, the computing resource of each training node and the communication latency information between each training node and other training nodes, the model deployment strategy including one or more pipeline parallel groups, one or more data parallel groups and a correspondence between model slices obtained based on the pipeline parallel groups and the data parallel groups and the plurality of training nodes.

3. The method of claim 2, wherein, The model deployment strategy is generated according to one or more of the following principles: minimizing the number of training nodes required for model deployment; if the overall computing resource of the plurality of training nodes is scattered, adjusting the division strategy of the training data; minimizing the communication latency between different training nodes in each data parallel group in the one or more data parallel groups and making the communication latency between different training nodes in each data parallel group similar; minimizing the communication latency between different data parallel groups in the one or more data parallel groups and making the communication latency between different data parallel groups similar.

4. The method of claim 2, wherein, The obtaining of the plurality of model slices corresponding to the model according to the model deployment strategy and the deploying of each model slice in the plurality of model slices to a corresponding training node include: performing replication and slicing processing on the model according to the one or more pipeline parallel groups and the one or more data parallel groups in the model deployment strategy to obtain the plurality of model slices; allocating each model slice in the plurality of model slices to a corresponding training node according to the correspondence between the model slices in the model deployment strategy and the plurality of training nodes; receiving a configuration completion message sent by the training node corresponding to each model slice to determine that model deployment is completed.

5. The method of claim 4, wherein, The method further includes creating a training task corresponding to the model deployment strategy and generating an identifier of the training task.

6. The method of claim 5, wherein, The step of allocating each model shard in the plurality of model shards to a corresponding training node according to the correspondence between the model shards and the plurality of training nodes in the model deployment strategy includes: Based on the correspondence, the identifier of the training task, the hierarchical structure of each model segment, and the parameter information of each model segment are assigned to the training nodes corresponding to each model segment, so that the training nodes corresponding to each model segment can perform resource configuration.

7. The method of claim 6, wherein, The method further includes: Establish node relationships between the training nodes corresponding to each model slice and other training nodes; the node relationships are used to indicate the adjacent nodes in the pipeline parallel group to which the training nodes corresponding to each model slice belong, and other training nodes in the data parallel group to which the training nodes corresponding to each model slice belong. The node relationships between the training nodes corresponding to each model segment and other training nodes are assigned to the training nodes corresponding to each model segment, so that the training nodes corresponding to each model segment can be configured with node relationships.

8. The method of claim 1, wherein, The training configuration information of the model includes model training quality requirements; Before generating a model deployment strategy based on the model's training configuration information and the resource information of each training node, the method further includes: Based on the resource information of each training node and the model training quality requirements, the multiple training nodes are filtered out, and the training nodes that do not meet the model training quality requirements are filtered out.

9. The method of claim 5, wherein, The method further includes: activating the training nodes corresponding to each model slice to execute the training task based on the identifier of the training task.

10. The method of claim 9, wherein, The method further includes: In each round of training for the training task, the training status of each batch of data in the current round is recorded.

11. The method of claim 9, wherein, The method further includes: In each round of training for the training task, the parameter update status of each model slice is recorded; In response to the completion of parameter updates for each model slice, the training state of the model is updated, and the next round of training is determined.

12. A model processing method characterized by comprising: Applied to a distributed training system, the distributed training system including a management node and multiple training nodes, the method is executed by one of the multiple training nodes, and the method includes: Deploy the model shards allocated by the management node according to the model deployment strategy; The model deployment strategy is generated based on the model's training configuration information and the resource information of each training node among the multiple training nodes. The model deployment strategy is used to indicate the partitioning method used to partition the model and the node allocation strategy corresponding to the model partitions obtained based on the partitioning method.

13. The method of claim 12, wherein, The deployment of model shards allocated by the management node according to the model deployment strategy includes: Obtain the identifier of the training task, the hierarchical structure of the model slice corresponding to the training node, and the parameter information of the model slice corresponding to the training node; Establish a training task processing unit and associate the training task processing unit with the identifier of the training task; According to the hierarchical structure of the model shard corresponding to the training node and parameter information of the model shard corresponding to the training node, resource configuration is performed.

14. The method of claim 13, wherein, The method further comprises: obtaining a node relationship between the training node and other training nodes; the node relationship is used to indicate adjacent nodes in a pipeline parallel group to which the training node belongs and other training nodes in a data parallel group to which the training node belongs; According to the node relationship between the training node and other training nodes, node relationship configuration is performed.

15. The method of claim 13, wherein, The method further comprises: according to the identification of the training task, performing the training task by the training task processing unit.

16. The method of claim 15, wherein, The performing of the training task by the training task processing unit comprises: In each round of training of the training task, the training task processing unit receives a forward activation message sent by an upstream adjacent node in the pipeline parallel group to which the training node belongs; the forward activation message comprises forward activation data output by the upstream adjacent node; The forward activation data output by the upstream adjacent node is calculated to obtain forward activation data output by the training node; The forward activation data output by the training node is packaged to obtain a new forward activation message, and the new forward activation message is sent to a downstream adjacent node in the pipeline parallel group to which the training node belongs.

17. The method of claim 15, wherein, The performing of the training task by the training task processing unit comprises: In each round of training of the training task, the training task processing unit receives a backward gradient message sent by a downstream adjacent node in the pipeline parallel group to which the training node belongs; the backward gradient message comprises backward gradient data output by the downstream adjacent node; The backward gradient data of the downstream adjacent node is calculated to obtain backward gradient data output by the training node; The backward gradient data output by the training node is packaged to obtain a new backward gradient message, and the new backward gradient message is sent to an upstream adjacent node in the pipeline parallel group to which the training node belongs.

18. The method of claim 17, wherein, The method further comprises: In each round of training of the training task, the training node sends backward gradient data output by the training node to other training nodes in the data parallel group to which the training node belongs, and receives backward gradient data output by other training nodes in the data parallel group to which the training node belongs; In response to obtaining the backward gradient data of all batch data in the current round of training in the data parallel group to which the training node belongs, gradient data aggregation calculation is performed to obtain aggregated gradient data, and the aggregated gradient data is used to update the parameter information of the model shard corresponding to the training node.

19. The method of claim 18, wherein, The method further comprises: The aggregated gradient data is packaged to obtain an aggregated gradient message; The aggregated gradient message is sent to other training nodes in the data parallel group to which the training node belongs.

20. The method of claim 18, wherein, The method further comprises: In response to receiving an aggregated gradient message sent by other training nodes in the data parallel group to which the training node belongs, it is determined that gradient data aggregation calculation is not performed; the aggregated gradient message sent by the other training nodes comprises the aggregated gradient data; In response to receiving an aggregated gradient message sent by other training nodes in the data parallel group to which the training node belongs, it is determined that gradient data aggregation calculation is not performed; the aggregated gradient message sent by the other training nodes comprises the aggregated gradient data; Update parameter information of the model shard corresponding to the training node according to the aggregated gradient data.

21. A distributed training system, comprising: The distributed training system comprises a management node and a plurality of training nodes; wherein, The management node is configured to: obtain training configuration information of a model and resource information of each training node in the plurality of training nodes; generate a model deployment strategy according to the training configuration information of the model and the resource information of each training node; the model deployment strategy is used to indicate a division manner adopted for model slicing and a node allocation strategy corresponding to a model shard obtained based on the division manner; obtain a plurality of model shards corresponding to the model according to the model deployment strategy, and deploy each model shard in the plurality of model shards to a corresponding training node.

22. An electronic device, comprising: Comprise: One or more processors; A storage device configured to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the model processing method of any one of claims 1 to 11, or implement the model processing method of any one of claims 12 to 20.

23. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the model processing method of any one of claims 1 to 11, or implement the model processing method of any one of claims 12 to 20.

24. A computer program product comprising a computer program which, when executed by a processor, implements the model processing method of any one of claims 1 to 11, or implements the model processing method of any one of claims 12 to 20.

Images